Jin Ma

Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models

Scaling Law for Quantization-Aware Training

HSA: Head-wise Sparse Attention for Efficient and Accurate Long-context Inference

INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats

Model Merging in Pre-training of Large Language Models