Songlin Yang

Log-Linear Attention

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

Distilling to Hybrid Attention Models via KL-Guided Layer Selection

Test-Time Training Done Right

HybridCoT: Interleaving Latent and Text Chain-of-Thought for Efficient Reasoning

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

PaTH Attention: Position Encoding via Accumulating Householder Transformations

Radial Attention: $\mathcal O(n \log n)$ Sparse Attention for Long Video Generation

Gated Delta Networks: Improving Mamba2 with Delta Rule

Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study

A Controlled Study on Long Context Extension and Generalization in LLMs

Test-Time Training Done Right

A Controlled Study on Long Context Extension and Generalization in LLMs