Joel Hestness

Scaling with Collapse: Efficient and Predictable Training of LLM Families

Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs

GQA-$\mu$P: The Maximal Parameterization Update for Grouped Query Attention and Fully Sharded Data Parallel

Don't be lazy: CompleteP enables compute-efficient deep transformers

Power Lines: Scaling laws for weight decay and batch size in LLM pre-training

Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs

BLIMEY: Towards Better Routing Methods in Sparse Mixture of Experts