SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation

Thank you for your thoughful comments! Please see our response to your concerns below:

(W1) The method is only evaluated on the Phi-MoE family of models. This calls into question if the conclusions from Sec 4.2 and 4.3 are specific to Phi-MoE or can be applied to other model families.

We appreciate this important concern about the generalizability of our conclusions. We remark that our multi-stage approach is architecturally agnostic and contains no Phi-specific design choices. To address the generalizability of our multi-stage approach (Section 4.2), we are currently conducting experiments on Mixtral 8x7B to provide additional evidence beyond the Phi-MoE family. Given the computational requirements of these large-scale experiments, we will do our best to provide these results before the discussion phase ends.

Regarding Section 4.3's conclusion about MoE models being easier to prune, we believe using Phi series leads to more reliable conclusions than using other model families. Phi-medium and Phi-MoE use the same pre-training data and similar training settings. Both of them are trained from-scratched and well-converged, and achieve similar performance in the end. In contrast, other model families do not disclose the differences in their MoE and dense training and often have large performance gaps. Hence, using them for experiment may not be able to get a fair comparison and draw meaningful conclusion. In our revision, we will present these findings as well-supported conclusions within the scope of our experimental evidence while noting the need for broader validation across diverse architectures.

(W2, Q2) Distillation at ~400B tokens seems quite resource intensive even though it's 10% of the original pretraining data. It is not explained whether the token horizon is a crucial factor for the method to work. In comparison, Minitron studied distillation using 90B tokens. Would multi-stage pruning/distillation still outperform single-stage at lower token budgets (E.g. <100B tokens)?

Thank you for raising this practical concern. Our method remains effective with significantly reduced token budgets. To demonstrate this, we evaluated our approach using only 90B total tokens (with 25B for the first stage) on Phi-mini-MoE:

We can see that our multi-stage approach notably outperforms the one-stage baseline, demonstrating the method's effectiveness across different training horizons.

We use a larger token budget than Minitron mainly due to our significantly more aggressive compression ratios: 42B→3.8B (9% retention) and 42B→7.6B (18% retention) with 400B tokens, compared to Minitron's 15B→8B (53% retention)/8B→4B (50% retention) with 90B total tokens. Higher compression ratio often requires a larger token budget for recovery to maintain performance. Furthermore, practitioners can adjust the training horizon based on available resources, though with some performance trade-offs.

(Q1) Does multi-stage distillation outperform one-stage distillation and are MoE models easier to prune for any other model family than Phi/Phi-MoE? I believe additional evidence OR discussion on the limitations of the existing evidence would help readers understand the take-aways of Sec 4.2 and 4.3.

We are working to provide Mixtral experiments to strengthen the evidence for our multi-stage approach's general applicability. For the conclusion that MoE models are easier to prune (Section 4.3), we believe this finding has broader validity because our experimental design controls for most confounding factors—both Phi-medium and Phi-MoE share the same pretraining data and similar procedures, isolating the architectural differences. Using other model families for experiment may not be able to get a fair comparison and draw meaningful conclusion. However, we acknowledge that general claims require more evidence. In our revision, we will provide clearer discussion regarding the generalizability.

Thank you for the valuable feedback! Please see our response per each of your concerns below:

1. The authors mentioned the compressed models remain competitive with larger models while achieving lower latency. But the latency figure is only shown in the Appendix. It would be a lot better to me if this figure is promoted to the main body to show a full picture of the serving performance vs. the model accuracy.

Thank you for this valuable suggestion! We agree that including the latency analysis in the main paper would provide a more comprehensive view of our models' performance-efficiency trade-offs.We have moved Figure 4 to Section 4.5 of the main paper to improve readability.

2. Latency performance of Phi-3-Small is not found in Figure 4. What is the rationale behind selecting these particular models? Could the authors elaborate the performance of Phi-3-small and Phi-mini-MoE?

In figure 4, we select models with similar performance levels (Llama 3.1 8B, Phi 3 mini) or MoE models with similar activated parameter counts (Qwen 1.5 MoE, DeepSeek V2 Lite) with Phi-mini-MoE. We aim to show that our model is more efficient compared to models with similar performance, and that it maintains efficiency advantages over other MoEs with similar activated parameter counts.

To address your concern, We added latency profiling on Phi-3-small in the plot (results available at https://anonymous.4open.science/r/SlimMoE-rebuttal-8F3F/latency_phismall.pdf). The results confirm that Phi-mini-MoE (2.4B activated, 7.6B total) achieves substantially lower latency than Phi-3-small (7.4B total), demonstrating the efficiency benefits of sparse activation when compared to dense models of similar total parameter count.

3. I failed to find the rationale behind comparing to Llama-3 series. Why the authors pick Llama-3 series model as a baseline in the abstract. Isn't Phi-3 series a better baseline and worth highlight in the abstract?

Thanks for pointing this out! You are absolutely correct that Phi-3 series represents a better baseline as our comparison between Phi-3 series models is fair and satisfies the criteria – 1) same training data for the MoE and dense models, 2) only architectural difference between the MoE and dense. For 3), our results show that Phi-mini-MoE (7.6B total/2.4B activated) achieve similar or better performance compared to Phi-3-mini (3.8B) with ⅔ activated parameters. Phi-tiny-MoE (3.8B total/1.1B activated) keeps the same total parameters as Phi-3-mini while activating ⅓ of them, demonstrating significant efficiency gains with controlled performance trade-offs.

We compare to the Llama-3 series in the abstract mainly to provide readers with a reference point. As Llama3 is one of the most popular and widely-recognized open-source models, comparing with it can give the reader a clear sense of the relative position of our developed models. We acknowledge that this distinction could be clearer. In the revision, we will emphasize the Phi-3 series as our primary technical baseline in the abstract, and strengthen the discussion of Phi-3 comparisons throughout the paper.

4. I am not sure how practical the proposed approach is when the model gets larger, as training a large model for few hundred billion tokens sounds like an expensive process.

We appreciate this practical concern about computational costs. While 400B tokens represents a substantial amount, we'd like to highlight several factors that make our approach more efficient even when dealing with large models:

1. Our multi-stage framework allocates most training tokens to the smallest target model size. Since the longest training stage operates on the heavily pruned model (E.g., Phi-tiny-MoE has 3.8B as the target size, which is significantly smaller than the original 42B parameters).
2. Training the target size models from scratch would require the full ~4T tokens. Using only 400B tokens (10% of original training data) to achieve competitive performance via compression represents a substantial efficiency gain.
3. We can pre-compute and cache teacher model logits as a one-time operation, significantly reducing memory requirements during distillation stages.
4. Our method allows users to adjust the token budget based on available resources. To demonstrate this flexibility, we evaluated a reduced budget variant using only 90B total tokens (25B for the first stage) on Phi-mini-MoE: | Method | MMLU | Arc-C | Winograde | Hellaswag | | ----------- | ----- | ----- | --------- | --------- | | One-stage | 64.11 | 59.13 | 72.34 | 73.23 | | Multi-stage | 65.59 | 62.54 | 72.74 | 73.38 |

Even with this significantly reduced budget, our multi-stage approach notably outperforms the one-stage baseline, demonstrating the method's effectiveness across different training horizon.

5. The authors only showed the experiments on Phi- series model only. Would the proposed method generalize to other model architectures?

Our method is architecturally agnostic and contains no Phi-specific design choices. To better support the generalization ability of the proposed method, we are conducting experiments on Mixtral 8x7B models. Due to the large size and computational resources needed. We will try our best to provide the results before the discussion phase ends. We choose Phi-3.5-MoE for main experiments for several reasons:

1. Phi-3.5-MoE (42B parameters) represents the largest MoE model we could comprehensively study given our computational resources, while still being substantial enough to demonstrate meaningful compression benefits. Larger models like DeepSeek-V3 exceed our current experimental capacity.
2. Phi-3.5-MoE achieves strong benchmark performance, which may be more sensitive to pruning. This can better show the difference between compression techniques.

6. How did the authors choose these 400B tokens in the dataset? Would a continued pre-trained Phi-3 series on the same 400 billion tokens form a stronger baseline?

We randomly sampled 400B tokens from the same pre-training dataset used for Phi-3 series. We also observed that continued pre-training the baseline on these same repeated tokens would not improve performance. As we conduct large-ratio pruning on the models, we believe using additional tokens for recovery is reasonable and is much cheaper than training a smaller model from scratch.

Thank you for a detailed explanation. That clear things up and I would like to raise my score.

Thank you for your positive review! Please see our response to each of your concerns below:

1. The paper is primarily empirical. While effective, the SlimMoE method lacks a theoretical framework to explain why staged slimming and intra-expert pruning outperform other approaches (e.g., optimization landscape analysis, convergence guarantees).

We agree that SlimMoE is primarily empirical. However, this approach aligns with the current state of MoE compression research, where theoretical analysis remains exceptionally challenging due to the complex routing dynamics and deep architectures involved. Recent MoE compression methods such as Seer-MoE [1] and [2] are similarly empirical and do not offer convergence guarantees or formal optimization analyses. Even in the broader context of dense LLM compression, established works like LLM-Pruner [4] and Minitron [5] rely primarily on empirical validation rather than theoretical guarantees. The complexity of analyzing MoE models stems from their non-trivial routing mechanisms and the interactions between multiple experts across deep layers. To our knowledge, only one recent work [3] provides theoretical guarantees for MoE expert pruning, but their analysis is limited to simplified single-layer MoE architectures on binary classification tasks, which does not extend to the deep MoE LLMs we studied. Given this landscape, our contribution lies in providing comprehensive empirical analysis and demonstrating the practical effectiveness of SlimMoE across multiple benchmarks and compression ratios. We will include this discussion of the theoretical challenges inherent to MoE analysis in our revision.

Reference

[1] Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts

[2] Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models.

[3] A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts

[4] LLM-Pruner: On the Structural Pruning of Large Language Models

[5] Compact language models via pruning and knowledge distillation.

2. Although well-executed, the framework combines known techniques (pruning and distillation) rather than introducing fundamentally new ideas. A deeper theoretical analysis. For example, explaining why expert slimming better preserves model capacity than expert dropping, would strengthen the conceptual contribution.

While SlimMoE builds upon established pruning and distillation techniques, our systematic integration of multi-stage design with intra-expert slimming represents a novel approach specifically designed for MoE architectures. We can provide more insight into why expert slimming outperforms expert dropping through sensitivity analysis. Using Taylor expansion, when parameters change by Δw, the loss function expands as:

For pruning (setting $\Delta w_i = -w_i$), the loss increase becomes:

In our experiment, we use the sensitivity scores $|\frac{\partial \mathcal{L}}{\partial w_i} w_i|$ as pruning criterion and observe that scores are similarly distributed across experts, indicating that representational knowledge is distributed across all experts rather than concentrated in few experts. Expert dropping removes all parameters of an expert simultaneously, potentially eliminating unique sub-functions critical for certain token distributions, resulting in large cumulative $\Delta \mathcal{L}$. In contrast, expert slimming selectively removes only the least sensitive weights within each expert, preserving high-impact neurons and maintaining expert specialization. This provides a more principled, fine-grained approach that better maintains model capacity through smaller overall loss changes. As no existing rigorous theoretical framework fully captures how deep MoE models retain knowledge and how pruning affects, our analysis provides intuitive and empirically-supported insights that advance understanding in this domain.