Discovering Important Experts for Mixture-of-Experts Models Pruning Through a Theoretical Perspective

Thanks for your careful review and comments!

W1: The designed metric (Shapley value) does not seem completely novel to me. Although the metric considers all possible combinations of unpruned experts when evaluating the importance of a given expert, it still measures importance based on the change in model performance when that expert is pruned versus retained, using a calibration dataset—similar to the approach in [1]. The consideration of all possible combinations of unpruned experts while evaluating the metric is to capture the interdependency of the experts. However, no empirical evidence is provided to justify that there exists a sufficient level of interdependency. Moreover, it is not clear whether the interdependency is inter-layer or intra-layer.

Thank you for your thoughtful feedback. We appreciate the opportunity to clarify and strengthen our contributions.

1. Novelty of Shapley value metric and comparison with [1]:
   • Although both NAEE and our method measures importance based on the change in model performance when that expert is pruned versus retained, NAEE [1] enumerates all possible combinations of retained experts and selects the combination that minimizes the Frobenius norm of the output difference before and after pruning. For fine-grained large-scale MoE with hundreds of experts, NAEE [1] method are impractical in practice, as they require tens of thousands or even trillions of enumerations, which is obviously infeasible.
   • On the contrary, our method does not require enumerating all possible expert combinations and can obtain a high-performance pruned MoE in just tens of minutes. By quantifying the contribution of each expert through Shapley value, thereby effectively identifying those experts that are highly relevant to task performance. In addition, we use Monte Carlo sampling to approximate Shapley value. Crucially, we have improved the accuracy and efficiency of this approximation through novel early truncation and Router-Guided Importance Sampling (RGIS) techniques. These constitute the core algorithmic innovations we propose. Therefore, our method has significant novelty compared to previous approaches.
2. Empirical evidence for expert interdependencies: We acknowledge that direct evidence for the level of interdependency was not explicitly quantified in our paper. We believe that the experts in MoE naturally have interdependent relationships, which include both intra-layer and inter-layer relationships. Intra-layer interdependency occur when experts within a layer compete/cooperate via top-k routing, while inter-layer ones arise from sequential processing (e.g., an expert in layer $l$ relying on outputs influenced by experts in layer $l-1$). Therefore, our formulation treats the expert set $\mathcal{E}$ globally (across all layers), naturally capturing inter-layer and intra-layer interdependency. For empirical evidence, please see our superior performance over baselines that do not model interdependencies (e.g., Frequency and Gating Score in Tables 1, 2 and 3). This gap indirectly evidences interdependencies, as these metrics underperform by pruning "important" experts in isolation without considering model interdependencies.

W2: The theoretical results are only to support the sampling strategy to approximate the metric. However, no theoretical support is provided to show that the metric is itself effective.

The effectiveness of Shapley value itself as a metric rests on its axiomatic guarantees from cooperative game theory, which we have shown in Section C. These properties ensure it is uniquely suited for expert contribution assessment in MoE:

1. Efficiency ensures the sum of Shapley values equals the total performance gain achievable by the expert set. Pruning low Shapley value experts thus minimizes degradation, as their absence has little impact on the overall utility.
2. Symmetry ensures equitable treatment of experts with identical impacts, preventing bias in pruning decisions.
3. Null Player Property identifies experts with zero marginal impact (Shapley value is 0), enabling safe pruning without performance loss.
4. Linearity preserves the metric's validity under combinations of games, which is relevant for multi-layer MoE where performance can be viewed as additive across layers.

These axioms collectively establish Shapley Value as a theoretically grounded metric for fair attribution in cooperative settings and ensure its effectiveness.

W3: The paper compared the proposed method’s performance against some baselines. Similar to the proposed method, all those baselines are calibration data dependent, either evaluating the sensitivity of the individual experts over the calibration dataset or using the routing statistics (e.g., Gating score, Frequency) on the dataset. Comparison with data-free approach as in [2] would better evaluate the effectiveness of the proposed method.

1. Comparison with [2].

   • [2] proposes to prune experts based on the change in the $l_2$ norm of the router in a fine-tuned MoE compared to that in pre-trained MoE. For example, when pruning a vision MoE fine-tuned on CIFAR-10, the pruning metric is $l_2$ norm difference of the router between this fine-tuned model and the model pre-trained on ImageNet-21k. It should be noted that although this pruning process does not require a dataset, the process of obtaining the fine-tuned model using a fine-tuning dataset does require a dataset. Therefore, strictly speaking, the method proposed in [2] is not a data-free method. In addition, the method proposed in [2] is not applicable to most scenarios of pruning Mixture-of-Experts LLMs, because there is no corresponding pre-trained model for pruning in such cases. In contrast, our method can directly prune any MoE without comparing the $l_2$ norm difference with the pre-trained model.

   • The method proposed in [2] cannot be used to prune MoE model mentioned in our paper, so it cannot be directly compared with our method. We migrated our method to prune the V-MoE model mentioned in [2]. We used the training set of CIFAR-10 as the calibration dataset, evaluated CIFAR-10 test set accuracy, and pruned 50% experts without fine-tuning after pruning. The results are as follows: |Method|Accuracy| |-|-| |Method in [2]|98.13%| |Ours|98.32%|

     Our method outperforms the method proposed in [2], which once again demonstrates the effectiveness of our method. We will include the above comparison results with the data-free approach [2] in the final version of our paper.

Furthermore, we would like to make the following clarifications regarding the calibration dataset.

1. The use of a calibration dataset is a common practice in expert pruning for MoE. Our method does require the use of a calibration dataset, but the use of a calibration dataset is a common practice in expert pruning for MoE. All the expert pruning methods compared in our paper, including Gating Score, Frequency, NAEE, Expert Trim, and CD-MoE, adopt calibration datasets.

2. The use of calibration datasets is also a common practice in compressing LLMs. Many previous SOTA methods for LLMs pruning (e.g., SparseGPT [A1], and LLM-Pruner [A2]) and quantization (e.g., AWQ [A3] and SmoothQuant [A4]) have employed calibration datasets. By using calibration dataset and running the forward propagation of the model, we can measure the weights that have a significant impact on the final output of the model, thereby reducing compression error.

[A1] SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. ICML 2023.

[A2] LLM-Pruner: On the Structural Pruning of Large Language Models. NeurIPS 2023.

[A3] AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. MLSys 2024.

[A4] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023.

Q1: With $\tau→0$ (i.e., no early truncation), the first term on the right side of equation (6) diminishes, i.e., we have a tighter bound. Then, how does the early truncation help, as we are having a looser bound for the case?

1. It is correct that the first term in the error bound on the right side of equation 6 approaches zero as $\tau→0$ (approaching no truncation), resulting in a tighter theoretical bound that reduces to the standard Monte Carlo sampling error term. However, the primary purpose of Theorem 1 is not to claim that early truncation tightens the bound compared to no truncation, but rather to provide a rigorous guarantee that the truncated estimator remains accurate (with bounded error) despite skipping certain low-information sampling steps.
2. The key benefit of early truncation is computational efficiency, which indirectly leads to improved estimation accuracy in practice. Without truncation, each Monte Carlo sample requires evaluating the full permutation of $N$ experts, leading to $\mathcal{O}(MN)$ complexity that is often prohibitive. Truncation allows us to early-exit permutations where performance has collapsed, significantly reducing the average number of evaluations per sample. This enables us to afford a larger number of samples $M$ within the same computational budget, which in turn reduces the second term in the bound (the sampling error). As shown in our ablation studies (Section 4.6), this technique reduces the overall pruning cost by nearly half while improving accuracy.

Q2: In equation (7), over which distribution is $\mathbb{E}(G(\boldsymbol{x}))$ calculated?

Thank you for your careful reading and this was an oversight on our part. $\mathbb{E}(G(\boldsymbol{x}))$ is not calculated over any distribution and the expectation symbol $\mathbb{E}$ is redundant, and we will remove $\mathbb{E}$ in the final version of the paper.

We hope that our response has addressed your concerns, and we kindly ask for a more favorable evaluation of our work.

Thank you for your rebuttal. My replies are given below:

1. Regarding Q1: Thank you for your clarification. I was confused (didn't count this as a weakness) about the results of Theorem 1. Your answer clarifies my confusion.

2. Regarding W1, W2, and W3: I am fully aware of using the calibration dataset in previous works for compressing LLMs in general and pruning experts in MoE in particular. For the same reason, evaluating the sensitivity of the experts on the calibration dataset (certainly, in a different way using the Shapley value to improve the performance and then sampling strategy to reduce the computational overhead) still seems to lack novelty. One of the major claims of this work is to reduce the computational overhead of finding prunable experts. That is why I asked the authors to compare the results with the router-norm-based method. The authors claim that the router-norm-based method is not data-free, as it uses finetuning dataset. However, my view regarding this work is different from the authors', as here the fine-tuning dataset is used for training (which indeed requires a sufficient amount of data), not for identifying the sensitivity of the experts. The authors provided a comparison with the specified method. However, improving the results by only 0.20% with the additional complexity and computational overhead does not seem a major advantage. Also, the authors did not compare for other pruning ratios. Finally, I still believe that providing some sort of theoretical performance guarantee of the pruned MoE model itself while using the Shapley value would greatly benefit the work.

I greatly appreciate the effort of the authors during the rebuttal.

Finally, I still believe that providing some sort of theoretical performance guarantee of the pruned MoE model itself while using the Shapley value would greatly benefit the work.

Thank you for your valuable suggestion. According to your suggestion, we have provided the theoretical performance guarantee of the pruned MoE model itself while using the Shapley value.

Theorem [Performance Preservation Bound]

Let $\mathcal{E}$ denote the complete set of experts in the MoE model, and $\mathcal{E}\_{\text{pruned}} = \mathcal{E} \setminus \\{E_i^l \mid \hat{\phi}\_{E_i^l} < \epsilon\\} (l\in\\{1,2,\ldots,L\\}, i\in\\{1,2,\ldots,n\\})$ be the pruned expert set with Shapley value threshold $\epsilon > 0$. Under the following conditions:

1. The Shapley value estimator satisfies $|\hat{\phi}\_{E_i^l} - \phi\_{E_i^l}| \leq \eta$ for all $E_i^l \in \mathcal{E}$ with probability $\geq 1 - \delta$.
2. The cooperative game is additive, i.e., $V(\mathcal{S}) = \sum_{E_i^l \in \mathcal{S}} \phi\_{E_i^l}$ for any $\mathcal{S} \subseteq \mathcal{E}$.

Then with probability at least $1 - \delta$, the performance degradation after pruning satisfies:

where $N = |\mathcal{E}|$ is the total number of experts.

Proof.

Under the additivity assumption,

Therefore, the performance degradation is

For each pruned expert $E_i^l \notin \mathcal{E}\_{\text{pruned}}$ (i.e., $\hat{\phi}\_{E_i^l} < \epsilon$), we have

with probability at least $1 - \delta$. Let $R = |\\{E_i^l \notin \mathcal{E}\_{\text{pruned}}\\}|$ be the number of pruned experts. Then,

Therefore we complete the proof.

We believe that the aforementioned theory has further enhanced the theoretical contribution of our work. We will incorporate the above content into the final version of our paper and would like to express our sincere gratitude again for your valuable suggestion.

We hope that the above response has addressed your concerns and look forward to your further response.

Thanks for your careful review and comments!

W1 & Q1: Core idea lacks novelty. The use of Shapley values for model pruning has appeared in prior work on neurons and convolutional filters, and extending this to MoE experts is a conceptually straightforward adaptation. The main contributions—early truncation and RGIS—are useful heuristics, but they are presented as practical add‑ons rather than fundamental algorithmic innovations.

We appreciate the reviewer's insightful comment regarding prior applications of Shapley value in neurons or convolutional filters pruning. The similarity between these works and ours is that both determine the importance of model components based on the Shapley value, thereby pruning the unimportant model components.

However, we respectfully argue that extending Shapley values to MoE pruning is far from a straightforward adaptation due to the unique structural and computational characteristics of MoE architectures. Unlike dense neural networks or convolutional layers, where components (e.g., neurons or filters) are always active and contribute uniformly, MoE experts operate in a sparse, conditionally activated manner governed by gating mechanisms. This introduces interdependencies among experts that create a fundamentally different "cooperative game" compared to prior settings. Directly applying existing Shapley-based pruning methods would overlook these dynamics, leading to inefficient or inaccurate importance estimation. The results of our ablation experiments in Table 4 show that our method has higher accuracy and lower pruning cost compared with the original Shapley value pruning method based on Monte Carlo sampling.

Moreover, our main contributions—early truncation and Router-Guided Importance Sampling (RGIS)—are not merely practical "add-ons" but principled innovations designed to address the exponential computational challenges unique to large-scale MoE. Early truncation leverages the observation of performance collapse in low-expert coalitions and is backed by a novel error bound theorem (Theorem 1), which theoretically justifies its use and demonstrates bounded estimation error. Similarly, RGIS exploits MoE's gating probabilities as a natural prior for importance sampling via a Plackett-Luce model, with proven unbiasedness and variance reduction (Theorem 2}). These are not generic heuristics; they are tailored to MoE's router-driven sparsity, reducing computational complexity while improving accuracy, as validated in our ablation studies and experiments, where Shapley-MoE outperforms baselines by significant margins.

In essence, while building on the foundational idea of Shapley value, our work introduces a novel framework that adapts and extends them to the distinct challenges of MoE pruning, with theoretically grounded estimate accelerations that enable scalability to MoE. We believe this represents a meaningful advancement in efficient MoE deployment.

W2: Theoretical analysis also has limitations. Theorem 2 only provides a one‑sided variance bound, without quantifying expected gains, making practical benefit of RGIS hard to predict.

Thank you for your insightful feedback on Theorem 2. However, we would like to make the following clarification.

1. Expected gains of RGIS: We agree that a more precise quantification of variance reduction would strengthen the analysis. The one-sided bound was designed to offer a rigorous guarantee: under the reasonable assumption of positive correlation between activation frequencies and true Shapley value, RGIS ensures variance is no worse than standard Monte Carlo sampling. However, we also acknowledge that this bound does not specify the magnitude of improvement, which depends on factors such as strength of correlation, specific models/datasets, etc., and this increases the difficulty of our theoretical analysis.
2. Practical benefits of RGIS: To address above, our ablation studies in Table 4 provide empirical quantification of RGIS's benefits: RGIS improves the accuracy of our Shapley-MoE method. Removing RGIS would lead to an increase of 1.08 in perplexity. This means that RGIS can accelerate Monte Carlo sampling, as Monte Carlo sampling requires more sampling steps to achieve the same level of accuracy.

W3: Theorem 1 ignores additional variance introduced by importance weights in RGIS, which may lead to large hidden constants.

We appreciate reviewer's astute observation regarding potential impact of importance weights on variance in Theorem 1, which indeed focuses on early truncation mechanism and does not explicitly account for RGIS-induced variance, which may lead to large hidden constants. However, we would like to make the following clarification.

1. Clarify the theorem's scope: Theorem 1 specifically bounds the estimation error for early truncation in standard Monte Carlo sampling, without incorporating RGIS. Its purpose is to justify truncation mechanism's efficiency and bounded error independently.
2. Theoretical mitigation of variance: Theorem 2 demonstrates that RGIS reduces variance compared to standard Monte Carlo under premise of assuming a complete sampling process, which mitigates additional variance introduced by importance weights in RGIS as well as magnitude of hidden constants.
3. Empirical validation: Table 4 provides empirical validation for early truncation and RGIS. When both early truncation and RGIS are removed, perplexity increases by 3.49. Our ablation studies further demonstrate that RGIS with truncation yields accurate Shapley estimates and better pruning results than naive Monte Carlo, suggesting combined approach is robust empirically even if hidden constants are not fully minimized.

W4: There is no theoretical justification that Shapley values computed with respect to 1/PPL will transfer to downstream tasks.

While Shapley values in our method are computed using V=1/PPL (an intrinsic measure of language modeling proficiency), we justify its transferability via both theory and empirics:

1. Theoretically, perplexity is a standard, intrinsic evaluation metric for LLMs, directly measuring a model's ability to predict held-out data and reflecting its overall language modeling quality. By using V=1/PPL, we quantify expert contributions to the model's core generative capabilities, which form the foundation for downstream performance. This choice aligns with prior work in LLM pruning and quantization, where perplexity is commonly used as proxies for compressed LLM performance.
2. Empirically, we evaluate zero-shot performance on seven downstream tasks and knowledge reasoning, arithmetic, and code generation capabilities in experimental section. experiments show that the Shapley value calculated based on 1/PPL reflects the expert's contribution well, and the MoE model with a higher Shapley value has higher accuracy on downstream datasets.

W5: Despite reduced cost, the method still relies on Monte Carlo estimates.

The exact computation of Shapley values has time complexity $\mathcal{O}(2^N)$, which is infeasible for MoE where $N$ typically exceeds $10^3$. Monte Carlo sampling approximates it, reducing complexity to $\mathcal{O}(MN)$, which $M$ is the number of samples. Thus, Monte Carlo estimation is essential to significantly lower computational costs. In addition, in our response to Weakness 1 from Reviewer ovkS, we also demonstrated that error between Shapley value estimated using Monte Carlo sampling and ground-truth Shapley value is very small.

Q2: Providing some intuition or guidance on sample complexity would make it easier to assess the method’s practicality.

The Monte Carlo-based Shapley value estimation has time complexity $\mathcal{O}(MN)$, where $M$ is the sample number and $N$ is the total number of experts in MoE. Figure 2 shows pruning costs under different $M$ settings; increasing $M$ raises costs. Thus, balancing search efficiency and model performance, we set $M=20$. Our method's end-to-end time is only 36 minutes, indicating good practicality.

Q3: It would also be useful to offer theoretical or empirical justification for assumption that activation frequency reflects an expert’s true contribution.

The relationship between activation frequency and expert contribution is justified as follows:

1. Theoretically, the gating network in MoE routes inputs to the most appropriate experts, prioritizing those that contribute most to performance. Frequently activated experts are those the model deems essential for handling inputs. Therefore, the above theoretical evidence supports the assumption that activation frequency reflects an expert’s true contribution.
2. Empirically, we present extensive results of pruning experts in MoE models based on activation frequency in Section 4. Frequency-based pruning outperforms random-based pruning, which empirically validates that activation frequency can indeed reflect an expert’s true contribution.

Q4: It remains unclear whether expert rankings based on 1/PPL objective are indicative of broader model importance.

Extensive experiments demonstrate that expert rankings based on 1/PPL and Shapley value effectively reflect importance in broader MoE models. This includes results of our method on popular open-source MoEs (Qwen, DeepSeek and Mixtral series) in Sections 4.2 and 4.3, and multimodal MoE in Section I.1.

Q5: Some insight into how often early truncation occurs and the extent to which it may bias perplexity estimates would be appreciated.

Early truncation is applied during each Monte Carlo sampling when catastrophic performance collapse occurs. Table 4 presents ablation results: removing it increases pruning cost (time rises from 36 to 62 minutes) and degrades pruned MoE performance (perplexity increases by 1.67).

We hope that our response has addressed your concerns, and we kindly ask for a more favorable evaluation of our work.

Many thanks to the authors for your thorough response and clear clarifications.

We appreciate the time and effort you put in reviewing our work, as well as your detailed comments and valuable questions. We understand the concerns you raised and we're pleased to address these concerns.

Weakness 1: The differences between the ground-truth and estimated value: The authors use Monte Carlo sampling to approximate the Shapley value, and calculating the ground-truth Shapley value is an NP-hard problem. Despite this, can the authors measure the differences between the ground-truth Shapley value and the estimated value?

Thank you for your suggestions. We have conducted experimental measurements below. Considering that the time complexity of calculating the ground-truth Shapley value using Equation. 3 is $\mathcal{O}(2^N)$, where $N$ is the total number of experts in the MoE model, and the number of experts in each MoE layer of the Qwen series and DeepSeek series MoE models exceeds 60. Therefore, we used Equation. 3 in our paper to calculate the ground-truth Shapley value of the 8 experts in the first MoE layer of the Mixtral-8x7B model, and adopted our proposed Shapley-MoE method to calculate the estimated Shapley value. The experimental results are as follows:

From the above experimental results, we can draw the following conclusions:

1. The difference between the ground-truth Shapley value and the estimated value is very small, which proves that using our Shapley-MoE method to estimate the Shapley value is highly effective. In addition, since calculating the ground-truth Shapley value is an NP-hard problem, it is not feasible in practice. However, our method provides an effective reference scheme for estimating Shapley value in practice.
2. Both the ground-truth and estimated Shapley values follow the descending order of 2/4/3/5/6/7/1/0. This means that the experts of MoE pruned based on the estimated Shapley value are the same as those pruned based on the ground-truth Shapley value. In this case, the accuracy of the pruned model obtained by Shapley-MoE is the same as that of the pruned model based on the ground-truth Shapley value. This proves that the Shapley value estimated by our Shapley-MoE method will not affect the final pruning accuracy.

Thank you again for your insightful opinions. We will incorporate the above experimental results into the final version of our paper.

Weakness 2 & Question 1: More experimental results: The experiments are conducted in the setting of 25% and 50% pruning, what about 75% pruning or else? Can you provide a performance trend in terms of different percentages of pruning?

Thank you for your suggestion. We have shown the experimental results under different pruning rates in Section 4.3. In Table 3, we show the WikiText-2 perplexity performance of the pruned Mixtral-8x7B model under a wider range of pruning rates, from 12.5% to 75%. The experimental results show that:

1. As the pruning rate increases, the perplexity of the pruned model shows an upward trend, which indicates that the performance of the pruned model shows a downward trend with the increase of the pruning rate.
2. Across different pruning rate settings, the perplexity of the pruned models obtained by our Shapley-MoE method consistently remains lower than that of pruned models obtained by other methods, which fully demonstrates the robustness and effectiveness of our Shapley-MoE method.

Question 2: Please illustrate the motivation for using Monte Carlo sampling to approximate the Shapley estimation.

Thank you for your question. The motivation for using Monte Carlo sampling to approximate the Shapley estimation is to make the calculation of Shapley value feasible for large-scale MoE models. While the Shapley value theoretically provides a metric for quantifying expert contributions, exact computation through Eq. 3 requires evaluating all $2^N$ expert subsets, which is a prohibitive proposition for modern MoE architectures where total expert count $N$ often exceeds $10^3$. This exponential complexity motivates the development of efficient approximation methods. Therefore, we employ the Monte Carlo sampling approach to estimate Shapley value through empirical expectation over expert permutations. By averaging over $M$ permutations, Monte Carlo sampling reduces the computational complexity from $\mathcal{O}(2^N)$ to $\mathcal{O}(MN)$, making the calculation of Shapley value feasible for large-scale MoE models.

Question 3: Can this method be employed in other LLM architectures? If yes, how to achieve this goal?

Thank you for your question.

1. Our method is applicable to expert pruning for various MoE model architectures. In Sections 4.2 and 4.3, we present the experimental results of our method on various popular open-source MoE architectures, including the Qwen series (Qwen1.5-MoE-A2.7B, Qwen2-57B-A14B, and Qwen3-30B-A3B), the DeepSeek series (DeepSeekMoE-16B and DeepSeek-V2-Lite), and the Mixtral series (Mixtral-8x7B). In addition, our method is also applicable to expert pruning for multimodal MoE models. In Section I.1, we present the experimental results of the MoE-LLaVA-Qwen-1.8B-4e model.
2. Our method is applicable to expert pruning for all models with mixture-of-experts architectures that include experts. However, for models with other dense LLM architectures, such as the LLaMA-1/2 series, our method is not applicable because these models do not contain experts.

Finally, we hope that our response has addressed your concerns, and we kindly ask for a more favorable evaluation of our work. We will incorporate the above clarification into the final version of our paper. Thank you!

We greatly appreciate your time, detailed comments, and valuable suggestions. We understand the concerns you raised and we're pleased to address these concerns.

Weakness & Question 1: The paper lacks a thorough discussion on the necessity of expert pruning. Given that the core motivation is to reduce memory usage and improve inference speed, it would be more convincing to include comparisons with alternative approaches such as channel pruning and quantization. These methods may achieve similar resource reductions with potentially smaller performance degradation. Although the authors suggest that Shapley-MoE can be combined with such methods, it remains unclear whether expert pruning lies on the Pareto frontier when considering the trade-off between performance and efficiency. How do expert pruning, channel pruning, quantization, and their combinations compare in terms of memory-performance and speed-performance trade-off curves?

Thank you for your thoughtful and constructive feedback. According to your suggestions, we demonstrate the compression effects of expert pruning, channel pruning, and quantization on the Qwen2-57B-A14B model. Specifically, we pruned 25% of the experts using our method, pruned 25% of the channels in each expert using magnitude pruning, and performed 4-bit quantization for the Qwen2-57B-A14B model using the GPTQ method. The experimental results are as follows:

From the data in the table, we can observe the following:

(1) Our method has advantages over channel pruning: Our Shapley-MoE method outperforms the channel pruning method, with a perplexity that is 1.21 lower and an average zero-shot accuracy that is 1.65% higher. We believe that although channel pruning operates at a finer-grained level to prune the model, it disrupts important experts, leading to a more severe performance degradation. In contrast, our method preserves the full capabilities of relatively important experts, which enables it to achieve better performance.

(2) Compared with the original model, our method has only a slight performance drop, while it can significantly reduce the memory usage and improve the inference speed: Compared with the original model, our method only increases the perplexity by 0.92 and decreases the average zero-shot accuracy by merely 2.34%. Meanwhile, it can achieve a 19.79% reduction in memory usage and a 1.26$\times$ improvement in inference speed. Compared with quantization, the aforementioned significant reduction and inference speed improvement can be obtained on any device without the need for a specific inference engine.

(3) Our method still has advantages over quantization: The perplexity of our method is only 0.64 higher than that of the quantization method, and the average zero-shot accuracy is only 1.96% lower. Furthermore, combining our method with the quantization method can further advance the Pareto frontier of MoE model compression. Shapley-MoE with AWQ quantization only increases the perplexity by 1.26 and decreases the average zero-shot accuracy by 2.61% compared with the original model, while achieving a 77% reduction in memory usage and a 1.51$\times$ improvement in inference speed. This further demonstrates the superiority of our method.

Furthermore, we would also like to make the following clarifications regarding the necessity of expert pruning and its comparison with channel pruning and quantization methods.

1. On the necessity of expert pruning: We agree that a more explicit discussion of why expert pruning is essential for MoE would enhance the paper. MoE architectures, as noted in our methodology section, feature a large number of sparsely activated experts (often exceeding 1000 in models like DeepSeekMoE-16B or Qwen2-57B-A14B), leading to significant memory and computational overhead during inference. Unlike dense models, MoE's conditional activation means many experts contribute marginally or redundantly, making expert-level pruning a natural and efficient way to reduce parameters while preserving the model's sparse computation benefits. This is particularly crucial for deployment on resource-constrained devices.
2. Comparison with pruning and quantization method: You are correct that methods like channel pruning (e.g., removing less important channels in FFNs) and quantization (e.g., 4-bit or 8-bit weight compression) can achieve similar resource reductions. However, these approaches are orthogonal to expert pruning: channel pruning operates at a finer granularity within experts, potentially leading to higher performance degradation in sparse MoE settings due to disruption of expert specialization, while quantization reduces bit-width but specific quantized kernels are required to achieve model memory reduction and improved inference speed. In contrast, expert pruning is hardware-agnostic and can reduce model memory usage and enhance inference speed on any device.

Thank you again for your constructive suggestions. We will include the above results in the final version of our paper.

Weakness & Question 2: The paper acknowledges that different tasks may require different calibration datasets for expert pruning, which aligns with the conceptual understanding of MoE experts. However, this raises questions about the fairness of comparisons with baseline methods like Random or Frequency-based pruning. Specifically, under randomly selected datasets and tasks, does Shapley-MoE still demonstrate consistent advantages over prior approaches? How does Shapley-MoE perform against baseline methods (e.g., Random, Frequency) under random datasets and task distributions, where its calibration strategy may not align with task-specific characteristics?

We appreciate the reviewer’s insightful comment regarding the role of calibration datasets in expert pruning. According to your suggestions, we verify the effectiveness of our method on random datasets, we randomly selected total 128 examples from C4, Pile, and PTB to form the calibration dataset. These datasets are also commonly used as calibration datasets in previous compression methods (such as Wanda [3], SparseGPT [4], and OmniQuant [5]). We pruned 25% of the experts in Qwen1.5-MoE-A2.7B model. The experimental results are as follows:

Our Shapley-MoE method outperforms other baseline methods. For example, compared with the best Frequency baseline, Shapley-MoE has a perplexity that is 0.37 lower and an average zero-shot accuracy that is 1.52% higher. We believe Shapley-MoE maintains advantages due to its game-theoretic foundation: it quantifies experts' marginal contributions to overall model performance, capturing inter-expert collaborations that random or frequency-based methods overlook. Frequency-based approach only consider isolated activation counts and gating score-based approach only consider isolated gating weight magnitude, which may not reflect true importance in diverse coalitions, while random pruning ignores any structure and task. Therefore, our method has advantages in any data distribution.

Furthermore, we would also like to make the following clarifications regarding the use of the calibration dataset in our paper.

1. To ensure fairness in our comparisons, all baseline methods (including Random and Frequency-based pruning) were evaluated using the same calibration dataset as Shapley-MoE. We follow the settings of Expert pruning methods for MoE models (such as NAEE [1] and CD-MoE [2] method), randomly sampling 128 examples from the C4 training dataset as calibration data.

2. In addition, the calibration datasets used in our paper are all datasets frequently employed in previous compression works. C4 dataset has been widely utilized in methods such as Wanda [3], SparseGPT [4] and OmniQuant [5].

3. Furthermore, NAEE [1] found that using the MATH dataset for calibration during the expert pruning process of MoE models can improve the accuracy of the pruned models on mathematical tasks, as this enables the identification of experts that are more important for mathematical tasks. Our paper follows this setup and employs domain-specific calibration datasets to improve the accuracy of the pruned model on domain-specific tasks.

[1] Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models. ACL 2024.

[2] Condense, don’t just prune: Enhancing efficiency and performance in moe layer pruning. arXiv.

[3] A Simple and Effective Pruning Approach for Large Language Models. ICLR 2024.

[4] SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. ICML 2023.

[5] Omniquant: Omnidirectionally calibrated quantization for large language models. ICLR 2024.

Finally, we hope that our response has addressed your concerns, and we kindly ask for a more favorable evaluation of our work. We will incorporate the above clarification into the final version of our paper. Thank you!

Dear Reviewer ysJ4,

We have put in many efforts to address the raised questions and concerns. As the NeurIPS rebuttal period is approaching its end, we kindly remind you to review our submitted response. Your feedback is essential for finalizing our work.

Thank you for your attention.

Best regards,

Submission20 Authors