Nikhil Vyas

The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton

Adam or Gauss-Newton? — A Comparative Study In Terms of Basis Alignment and SGD Noise

Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants

Mixture of Parrots: Experts improve memorization more than reasoning

How Does Critical Batch Size Scale in Pre-training?

SOAP: Improving and Stabilizing Shampoo using Adam for Language Modeling

A New Perspective on Shampoo's Preconditioner

Deconstructing What Makes a Good Optimizer for Autoregressive Language Models