Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations

Thank you for your insightful suggestions!

W1: limited structural novelty

We appreciate the reviewer’s concern about the structural novelty of our approach and the reliance on similarity- or norm-based measures.

(1) First, our goal is not to introduce architectural changes but to provide a practical pruning framework tailored to ultra-large MoE models. In this context, structural changes in the sense of redesigning the MoE architecture would require retraining from scratch, which is infeasible at the scale of DeepSeek-R1 (671B). Instead, we focus on a post-hoc pruning strategy that is lightweight, training-free, and easily deployable, which we argue is a critical form of innovation for models of this magnitude.

(2) Second, selecting an effective metric for expert pruning is non-trivial， especially for large MoE models. Through our analysis on DeepSeek-R1, we observed a counterintuitive phenomenon: with ultra-large MoEs, a small number of calibration samples is sufficient to consistently identify domain-specialized experts. We term this the few-shot expert localization phenomenon. Building on this, our integration of output-aware expert importance (gating value × output norm) with token-level contribution estimation enables us to capture both the strength and the influence of expert activations in a single forward pass, making accurate and efficient pruning feasible at this unprecedented scale.

(3) Third, our scoring strategy is theoretically grounded. Beyond empirical observations, we show that using the L2 norm of the gated expert output and similarity of the input and output of MoE modules can be interpreted as an approximation of the gradient-based contribution to the loss. This ensures that our pruning strategy is not heuristic but aligned with the underlying optimization objective. We show the analysis below.

Optimization Objective:

Our pruning goal is to minimize the expected loss on a calibration dataset $D_{\text{calib}}$, under a constraint that only $M$ experts are retained per layer. Formally, we solve:

Here, $m_{il} \in \{0, 1\}$ is a binary mask indicating whether expert $i$ in layer $l$ is retained.

This is a combinatorial problem and is NP-hard. To solve it approximately, we adopt a greedy strategy that relies on estimating the importance of each expert.

Gradient-Based Expert Importance:

We approximate the change in loss when removing expert $j$ via a first-order Taylor expansion:

Applying the chain rule, we obtain:

$\frac{\partial \mathcal{L}}{\partial m_{j,l}} = \sum_{t=1}^T \left( \frac{\partial \mathcal{L}}{\partial \tilde{h}{t,l}} \right)^T \left( g_{j,t,l} e_{j,t,l} \right)$

This shows that the importance of expert $j$ is proportional to the projection of the gradient of the loss onto the expert’s output.

L2 Norm in Output-Aware Expert Importance Assessment:

While we cannot compute the gradient $\partial \mathcal{L} / \partial \tilde{h}\_{t,l}$ during a forward-only pass, we can compute the expert’s contribution term $g\_{j,t,l} e\_{j,t,l}$.

This upper bound shows that the L2 norm of $g_{j,t,l} e_{j,t,l}$ is a direct proxy for the expert’s contribution to the loss-sensitive direction in the model. Since the gradient component is unavailable during a forward pass, we approximate expert importance using this measurable term.

Thus, using the expert’s output L2 norm as part of the importance score is not ad hoc—it follows directly from a first-order gradient-based analysis of our objective.

Additional analysis of Expert-Level Token Contribution Estimation

For expert-token contribution estimation, we need to compute $\partial \mathcal{L} / \partial \tilde{h}_{t,l}$. However, it is very costly to conduct gradient computation. Thus, we emply $1-Sim(\tilde h_t^l,h_t^l)$ as a approximation.

W2: Expert capacities and token-routing distributions across different domains with different methods.

In Figures 2, 6, and 9, we present a comparison of the similarities between the gating scores and EASYEP experts across different domains, as well as the similarities among EASYEP, gating scores, and frequency. It can be observed that our method exhibits greater variability in expert importance compared to the others, allowing it to better capture the relative importance among experts. On the other hand, our method shows higher similarity across domains than the gating scores, which to some extent indicates that our approach captures expert importance more effectively, rather than merely domain-specific characteristics, thereby maintaining a stronger overall capability.

W3: Weak statistical reliability

We have provided the mean and standard deviation for computing error bars of DeepSeek-R1 on the AIME2024 dataset (32 samples), as shown in the table below. The results demonstrate that our method achieves better performances and comparable robustness compared to the full model baseline.

W4: Experiments with more models

We provide the experiments results on Qwen3-30B-A3B, the results are shown in the following Table. We can observe that for the MoE models without shared experts (which appears in DeepSeekMoEs), expert pruning is much more hard. For example, performances of other methods on AIME is very poor. On the contory, our method can achieve over 80% performances on AIME with only 50% experts, further demonstrating the effectiveness and universality.

Q1: Analysis of performance drops of task-specific pruning and methods to mitigate it.

We agree that pruning with a single-domain focus can lead to performance degradation when testing on different domains, and we have performed both qualitative and quantitative analyses to better understand and address this limitation of EASY-EP.

Low Cross-Domain Expert Overlap: Figure 2 (A) and (B) in our paper provide direct evidence. When examining the top 128 experts ranked by gating scores, the overlap between the Math and Code domains is only 74%. For the top 16 experts, the overlap is almost negligible, with nearly disjoint sets.

Essential Role of Domain Experts: Removing Science-specific experts causes GPQA accuracy to drop by 11.82 points. In contrast, the same pruning leads to only a 2.25-point decrease on LiveCodeBench, a Code task. This asymmetry highlights that domain experts are critical within their respective domains but have limited transferability.

Ways to mitigate it:

● Mixed-Domain Pruning: By pruning with mixed-domain data, we can preserve a broader range of important experts. As shown in our paper, it can achieve a great performance preservation on multiple tasks.

● Task-Aware Expert Activation: using a lightweight classifier or predictor to identify the domain or task of the input, and then selectively activating the corresponding experts.

Q2: In line 121, could the authors properly define "demonstrations"?

``Demonstrations'' is a common term in In-Context Learning (ICL). Specifically, it refers to examples or sample inputs that are provided to an LLM. These samples serve as references that guide the model's responses or predictions.

Dear reviewer,

This is a kind reminder as the discussion period will end soon. We have made every effort to thoroughly address your comments in our responses, including conducting additional experiments and providing further analysis. If you have any remaining questions or concerns, please let us know at your earliest convenience.

Thank you for your insightful suggestions!

W1/Q1/L1: Statistical Significance Reporting

We have provided the mean and standard deviation for computing error bars of DeepSeek-R1 on the AIME2024 dataset (32 samples), as shown in the table below. The results demonstrate that our method achieves better performance and comparable robustness compared to the full model baseline.

W2/Q2/L2: Reproducibility of Results

We have provided all the experiment details in Appendix E and Section 5.1 to ensure the reproducibility of results.

W3/Q3/L3: Impact of Expert Pruning on Long-term Generation Abilities

In our paper, we analyze the gap in long-term generation abilities between the pruned reasoning and non-reasoning models. For reasoning models, the long-term generation abilities are easily harmed after pruning for baselines. However, our method can better preserve the long-term generation abilities with only half the experts in DeepSeek-R1

W4: comprehensive comparison with existing expert pruning method

In our paper, we conduct a comparison with existing methods, including expert pruning and merging. Our method can achieve significantly better performance than these methods, demonstrating the effectiveness of our method.

In addition, there exist some perturbation-based methods that are not evaluated due to the huge computational costs. As analyzed in Appendix F, it will take over 10^7 for pruning one model, which is not useful for the large MoEs.

Q4: Generalization Across Domains

We analyze the generalization both in-domain and out-of-domain.

In-domain generalization: To verify that our method is not only effective for a specific dataset, we conducted in-domain cross-dataset experiments in Section 3.2. As shown in Figure 2 (D), we found that within the mathematics domain, the overlap rate of important experts identified using different datasets (AIME-2023, AIME-2024, AIME-2025, HMMT-Feb 2025) exceeded 84%. This indicates that the experts identified by our method have strong in-domain generality and are not merely adapted to a specific calibration dataset.

Out-of-domain generalization: We also evaluated the out-of-domain generalization capability. In Section 5.3.2 and Table 4, we show the performance on other unrelated domains after pruning with data from a specific domain. The experimental results show that the model exhibits a certain degree of generalization ability, especially between similar domains (e.g., mathematics/science, code/operating system agents). Meanwhile, we also point out that for multi-task scenarios, we recommend using a "mixed-domain pruning" strategy, as its cross-domain performance is superior to single-domain pruning.

Q5/L5: Clarity and Structure of the Paper and Method

We will improve the structure of our paper in the final version.

L4: Potential Negative Social Impacts

Our work aims at the memory efficiency of large MoE models through a pruning technique. Thus, our method itself does not generate or exacerbate bias. Its impact is inherently linked to the societal impact of the base model being pruned, such as DeepSeek-R1.

However, we fully agree that any technology that makes large models easier to deploy could indirectly amplify both the positive and negative societal impacts of these models themselves. In the final version, we will add a statement to the limitations section to clarify that while our method is technically neutral, its application should adhere to the responsible usage guidelines for the underlying large language models.

Thank you for your insightful suggestions!

W1&Q2: Experiments on more models

We provide the experimental results on Qwen3-30B-A3B, the results are shown in the following Table. We can observe that for the MoE models without shared experts (which appear in DeepSeekMoEs), expert pruning is much harder. For example, the performance of other methods on AIME is very poor. On the contrary, our method can achieve over 80% performance on AIME with only 50% experts, further demonstrating the effectiveness and universality.

W2: Comparison with complex baselines

As shown in our paper, we present M-SMoE (MC-SMoE is a further pruning version of M-SMoE, and its performance is worse than it). As shown in Table 2 in our paper and the following table, although the method is more complex, the performance of M-SMoE is significantly worse than our method, which demonstrates the effectiveness of our method.

Q1: Relationship between similarities and variance in gating scores

To investigate the relationship between the token-level representational change (measured by our similarity-based token contribution metric) and the properties of the activated experts, we computed the Pearson correlation between token similarity and the variance of the corresponding gating scores. The result was −0.3119, indicating a weak negative correlation. Though tokens with larger representational changes (i.e., lower similarity) tend to have slightly higher variance in their gating scores, this relationship is not strong.

Q3: Potential for employment on dynamic expert selection during inference

Yes, I think our method can also be employed in the task. We infer that the dynamic expert selection during inference aims to identify important experts in the MoE models via the initial tokens and to prune the MoE models for other tokens in the long generation. We analyze the top experts in the first 1000 tokens and all the tokens and observe that there are over 90% overlap ratios. Thus, we can dynamically prune experts according to the initial 1000 tokens for the long-generation tasks.

Dear Reviewer,

As the discussion period will conclude tomorrow, we would like to kindly remind you that we have provided detailed responses to your latest comments, including additional experiments and extended analyses. If there are any remaining questions or concerns, we would greatly appreciate it if you could share them at your earliest convenience.

Thank you for your insightful suggestions!

W1: Most of these metrics have been used before in literature

We appreciate the reviewer’s observation that several of the metrics employed in our work have appeared in prior literature. We would like to clarify two key points regarding novelty and contribution.

1. For the issue of simplicity being mistaken for lack of novelty， our method’s simplicity is not due to insufficient innovation but is a deliberate design decision. For ultra-large MoE models such as DeepSeek-R1 (671B), overly complex pruning schemes are impractical. A lightweight scoring mechanism that requires only a single forward pass is crucial to ensure efficiency and deployability. Thus, simplicity is a necessity for practical impact rather than a limitation.
2. Regarding the reviewer’s request to clarify our core novelty beyond metric reuse, our primary contribution lies in uncovering the counterintuitive phenomenon of “few-shot expert localization”. We demonstrate, for the first time, that in ultra-large MoE models, identifying domain-specialized experts actually becomes easier with only a few demonstrations. This sheds light on the clear functional partitioning inside very large MoEs and enables effective domain-specific pruning.
3. In addition, compared to prior metric-based approaches, our method provides a stronger theoretical grounding. Specifically, we propose using the L2 norm of the gated expert output and the similarity between the input and output of MoE modules as the importance score, which can be interpreted as an approximation of the gradient-based contribution to the loss. Under this theoretical framework, our method serves as a principled proxy rather than a heuristic, ensuring that our pruning strategy aligns with the optimization objective. A detailed derivation is provided in our response to W2.

W2: Why is the contribution to the final output’s L2 norm a good measure in theory?

We provide a principled derivation from our optimization objective and gradient analysis, showing how the L2 norm is a good measure.

Optimization Objective:

Our pruning goal is to minimize the expected loss on a calibration dataset $D_{\text{calib}}$, under a constraint that only $M$ experts are retained per layer. Formally, we solve:

Here, $m_{il} \in \{0, 1\}$ is a binary mask indicating whether expert $i$ in layer $l$ is retained.

This is a combinatorial problem and is NP-hard. To solve it approximately, we adopt a greedy strategy that relies on estimating the importance of each expert.

Gradient-Based Expert Importance:

We approximate the change in loss when removing expert $j$ via a first-order Taylor expansion:

Applying the chain rule, we obtain:

$\frac{\partial \mathcal{L}}{\partial m_{j,l}} = \sum_{t=1}^T \left( \frac{\partial \mathcal{L}}{\partial \tilde{h}{t,l}} \right)^T \left( g_{j,t,l} e_{j,t,l} \right)$

This shows that the importance of expert $j$ is proportional to the projection of the gradient of the loss onto the expert’s output.

L2 Norm in Output-Aware Expert Importance Assessment:

While we cannot compute the gradient $\partial \mathcal{L} / \partial \tilde{h}\_{t,l}$ during a forward-only pass, we can compute the expert’s contribution term $g\_{j,t,l} e\_{j,t,l}$.

This upper bound shows that the L2 norm of $g_{j,t,l} e_{j,t,l}$ is a direct proxy for the expert’s contribution to the loss-sensitive direction in the model. Since the gradient component is unavailable during a forward pass, we approximate expert importance using this measurable term.

Thus, using the expert’s output L2 norm as part of the importance score is not ad hoc—it follows directly from a first-order gradient-based analysis of our objective.

Additional analysis of Expert-Level Token Contribution Estimation

For expert-token contribution estimation, we need to compute $\partial \mathcal{L} / \partial \tilde{h}_{t,l}$. However, it is very costly to conduct gradient computation. Thus, we emply $1-Sim(\tilde h_t^l,h_t^l)$ as a approximation.

W3: Comparison with perturbation-based methods and router-norm-based methods

Due to the enormous parameter size of DeepSeek-R1 and the high complexity required by this method, it is difficult for us to implement it directly. As analyzed in our paper, NAEE and CD-MOE require 10^75 and 24,768 evaluations per layer, respectively. On a single H200 GPU, performing one round of pruning would require over 10^75 evaluations and approximately 9,976 days to complete (as described in Appendix E), which is unacceptable for current frameworks. Therefore, this method is, in any case, impractical to apply to models as large as DeepSeek-R1.

For the router-norm-based method, we require the models before and after fine-tuning to evaluate the change in router. Thus, we select DeepSeek-V3-base and DeepSeek-R1 to select experts with the maximum L2 Norm of the difference of router weights and remove other experts. As shown in the following Table, this method can not achieve comparable performance to our method, which further demonstrates the effectiveness of our method.

Q1: How do you use the tokens of the pruned experts?

In our framework, tokens originally routed to the pruned experts are not discarded. Instead, they are re-routed to the remaining experts based on the gating mechanism. Specifically, after pruning, the router operates over the retained experts only, and each token is dynamically assigned to the most suitable expert(s) among this reduced set. This ensures that every token continues to receive expert processing without additional overhead.

Dear Reviewer,

As the discussion period will conclude tomorrow, we would like to kindly remind you that we have provided detailed responses to your latest comments, including additional experiments and extended analyses. If there are any remaining questions or concerns, we would greatly appreciate it if you could share them at your earliest convenience.

Dear Reviewer,

As the discussion period will soon conclude, we are writing to follow up on our submitted response. We hope the extended analyses and experiments have clarified our work, and we would be happy to address any final questions you may have.

We appreciate your valuable feedback.