Dan Busbridge

Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training

Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration

Distillation Scaling Laws

Scaling Laws for Optimal Data Mixtures

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

Theory, Analysis, and Best Practices for Sigmoid Self-Attention