Sham M. Kakade

Cognitive models can reveal interpretable value trade-offs in language models

The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton

Any-Order Flexible Length Masked Diffusion

Fine-Tuning Masked Diffusion for Provable Self-Correction

Seesaw: Accelerating Training by Balancing Batch Size and Learning Rate Scheduling

Adam or Gauss-Newton? — A Comparative Study In Terms of Basis Alignment and SGD Noise

In Good GRACES: Principled Teacher Selection for Knowledge Distillation

Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning

Parameter-Efficient Reinforcement Learning using Prefix Optimization

The Emergence of Complex Behavior in Large-Scale Ecological Environments

LOTION: Smoothing the Optimization Landscape for Quantized Training

Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants

Understanding the Design Space and Cross-Modality Transfer for Vision-Language Models

A Mechanistic Analysis of Low-Precision Instabilities in Microscaling Formats

Random Scaling of Emergent Capabilities

Selective Underfitting in Diffusion Models

Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions

EvoLM: In Search of Lost Language Model Training Dynamics

Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining

The Role of Sparsity for Length Generalization in LLMs

Interpreting the linear structure of vision-language model embedding spaces

Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models

Mixture of Parrots: Experts improve memorization more than reasoning

Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond

How Does Critical Batch Size Scale in Pre-training?

Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems

Eliminating Position Bias of Language Models: A Mechanistic Approach

A New Perspective on Shampoo's Preconditioner

SOAP: Improving and Stabilizing Shampoo using Adam for Language Modeling

Deconstructing What Makes a Good Optimizer for Autoregressive Language Models

Universal Length Generalization with Turing Programs

Multi-Agent Reinforcement Learning from Human Feedback: Data Coverage and Algorithmic Techniques

Universal length generalization with Turing Programs

Soup to go: mitigating forgetting during continual learning with model averaging