Yutao Zeng

UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

Scaling Law for Quantization-Aware Training

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models

Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Models

HSA: Head-wise Sparse Attention for Efficient and Accurate Long-context Inference

Stepsize anything: A unified learning rate schedule for budgeted-iteration training

Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling

Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models

HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization

Hyper-Connections

Ultra-Sparse Memory Network