Hemanth Saratchandran

Conditioned Initialization for Attention

Procedural Pretraining: Warming Up Language Models with Abstract Data

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

Stable Forgetting: Bounded Parameter-Efficient Unlearning in LLMs

Cutting the Skip: Training Residual-Free Transformers

Leaner Transformers: More Heads, Less Depth

Polynomial Alternatives to Softmax in Transformers

Spectral Conditioning of Attention Improves Transformer Performance

Structured Initialization for Vision Transformers

Efficient Learning with Sine-Activated Low-Rank Matrices

RandLoRA: Full rank parameter-efficient fine-tuning of large models

Sampling Theory and Overparameterization: Shaping Loss Landscapes in $\ell^2$ Regression

Rethinking Softmax: Self-Attention with Polynomial Activations