Kamil Adamczewski

$\mu$-Parameterization for Mixture of Experts

Projected Compression: Trainable Projections for Efficient Transformer Compression

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies

Joint or Disjoint: Mixing Training Regimes for Early-Exit Models

Different Rates for Different Weights: Decoupled Relative Learning Rate Schedules