Siyuan Qiao

Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss

SeeDNorm: Self-Rescaled Dynamic Normalization

UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models

Balanced Actor Initialization: Stable RLHF Training of Distillation Reasoning Models

HSA: Head-wise Sparse Attention for Efficient and Accurate Long-context Inference

SensorLM: Learning the Language of Wearable Sensors