Kaiyue Wen

QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation

Fantastic Pretraining Optimizers and Where to Find Them

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

PaTH Attention: Position Encoding via Accumulating Householder Transformations

Overtrained Language Models Are Harder to Fine-Tune

From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency

Weight ensembling improves reasoning in language models

Task Generalization with Autoregressive Compositional Structure: Can Learning from $D$ Tasks Generalize to $D^T$ Tasks?

Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View

RNNs are not Transformers (Yet): The Key Bottleneck on In-Context Retrieval