zhou Xun

UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

Scaling Law for Quantization-Aware Training

Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Models

Towards Simple and Provable Parameter-Free Adaptive Gradient Methods

Conda: Column-Normalized Adam for Training Large Language Models Faster

MARS: Unleashing the Power of Variance Reduction for Training Large Models

Stepsize anything: A unified learning rate schedule for budgeted-iteration training

Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling

Model Merging in Pre-training of Large Language Models

HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization

Ultra-Sparse Memory Network

Investigating the Overlooked Hessian Structure: From CNNs to LLMs