Nathan Godey

Headless Language Models: Learning without Predicting with Contrastive Weight Tying

Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck