David Grangier

Compute-Optimal Quantization-Aware Training

Pretraining with hierarchical memories: separating long-tail and common knowledge

The data-quality illusion: Rethinking Classifier-based Quality Filtering for LLM Pretraining

Partial Parameter Updates for Efficient Distributed Training

No Need to Talk: Asynchronous Mixture of Language Models

Scaling Laws for Optimal Data Mixtures

The AdEMAMix Optimizer: Better, Faster, Older

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

Need a Small Specialized Language Model? Plan Early!

Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging

Dynamic Gradient Alignment for Online Data Mixing