Martin Jaggi

Gradient-Normalized Smoothness for Optimization with Approximate Hessians

Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining

Weight Decay may matter more than µP for Learning Rate Transfer in Practice

Benchmarking Optimizers for Large Language Model Pretraining

A Split-Client Approach to Second-Order Optimization

$\alpha$-LoRA: Effective Fine-Tuning via Base Model Rescaling

MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training Data Processing to Every Language

Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs

Effective Interplay between Sparsity and Quantization: From Theory to Practice

GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining

Attention with Markov: A Curious Case of Single-layer Transformers

HyperINF: Unleashing the HyperPower of Schulz's Method for Data Influence Estimation

Intrinsic User-Centric Interpretability through Global Mixture of Experts

URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training

Towards Fully FP8 GEMM LLM Training at Scale

On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists

CoTFormer: A Chain of Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference

On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists

HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimation