Fantastic Experts and How to Find Them: A Multi-Dimensional Study for Experts-Level Sparsification in Mixture-of-Experts

1. Methodological Foundation: 1.1. Lacks theoretical justification for the proposed criteria in MC-Suite: The activation-based criteria (EAS, EAE) lack clear connection to expert importance; The weight-based metrics (EWS, RWN) need theoretical grounding for why they indicate expert redundancy; No justification for why gradient-based measures reliably indicate expert importance; 1.2. The iterative pruning and fine-tuning approach appears ad-hoc without strong motivation: No clear explanation why k rounds is better than one-shot pruning; The choice of pruning schedule (amount pruned per round) seems arbitrary;The relationship between pruning rounds and fine-tuning steps needs justification. 1.3. Many design choices seem empirically driven without proper justification, some of them are listed as follows: Selection of 32 experts per round for pruning; Choice of task-agnostic fine-tuning dataset; Hyperparameter settings for reconstruction compensation.

2. Technical Presentation: 2.1. Missing a dedicated Related Work section in the main body, making it difficult to understand the paper's positioning 2.2. Introduction is unnecessarily long and unfocused. 2.3. The methodology section lacks clear organization and proper explanation of key components. 2.4. Mathematical notations and formulations are not well explained (e.g., equations 3-18).

3. Experimental Design: 3.1. Limited comparison with state-of-the-art baselines, especially on some key metrics like perplexity. The comparison with following ones are necessary: Recent expert pruning methods like SEER-MoE; Standard model compression techniques adapted for MoE; Performance on more established benchmarks. 3.2. Experiments on larger MoE models (>8x7B parameters) are missing. 3.3. Implementation details are insufficient for reproducibility, some of them are crucial: Exact hyperparameters for fine-tuning (learning rate schedule, batch size, etc.); Details of the calibration process for different criteria; Computational infrastructure requirements. 3.4. Ablation studies don't fully justify the necessity of all proposed components, the following ones should be specifically examined: Individual contribution of each criterion in MC-Suite; Impact of different pruning schedules; Effect of fine-tuning duration;

4. Paper Organization and Clarity: 4.1. Figures lack proper resolution and detailed captions (e.g., Figure 1). 4.2. Inconsistent font sizes and styles in figures (particularly Figure 2). 4.3. Writing style is verbose with low information density, some parts need more concision: Literature review can be shortened by 40%; Experimental setup description is repetitive;Results discussion contains redundant information. 4.4. Poor organization makes it difficult to follow the main contributions, some suggestions can be considered: The authors can move technical background to preliminaries, consolidate criteria description into a unified framework, and present results in a more structured manner.

1. Most criteria in MC-Suite lack originality and proper justification. EWS, EAN, and other metrics are direct adaptations from traditional pruning methods without convincing evidence of their applicability to expert importance estimation. The gradient- based criteria introduce significant computational overhead without demonstrating clear benefits.
2. The iterative pruning framework has multiple unaddressed issues such as lack of stopping criteria and potential catastrophic forgetting. The task-agnostic fine-tuning could potentially harm expert specialization. The linear combination assumption in reconstruction compensation oversimplifies the complex relationships between experts.
3. The experimental evaluation is incomplete. Critical comparisons with state-of-the-art methods (e.g., ExpertPrune, DynamicMoE) are missing. The paper doesn't examine zero-shot generalization, impact on load balancing, or changes in expert specialization patterns. The largest tested model is only 8×7B parameters. The papers for ExpertPrune and DynamicMoE are listed as follows: ExpertPrune: Lu, Xudong, et al. "Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models." arXiv preprint arXiv:2402.14800 (2024); DynamicMoE: Huang, Quzhe, et al. "Harder Tasks Need More Experts: Dynamic Routing in MoE Models." arXiv preprint arXiv:2403.07652 (2024). There are actually many related works in recent years, which should be compared in the experiments.
4. Implementation details crucial for reproducibility are missing in the main body. The paper doesn't specify key hyperparameters for fine-tuning, details of the calibration process, or computational requirements.
5. The analysis of expert behavior changes during and after pruning is superficial. There's no examination of how pruning affects routing mechanisms, token distribution, or model robustness. A more detailed case study should be provided.
6. The figures have severe presentation issues: Figure 1 has poor resolution and lacks the necessary caption for explaining the figure content. Figure 2's text uses inconsistent fonts and sizes. Figure 4 contains too much context. Its organization can also be further improved.
7. The writing style in this paper is not easy to read and understand. Some proposed concepts, like 'MOE Lottery Subnetworks', have vague meanings. The core technique motivation and details, however, are not clarified in detail. The organization of this paper, especially Section 2 about methodology, is over-scattered, and it is hard to capture the logical structure and the most significant contribution. The presentation lacks a clear separation between methodology and experimental validation, making it challenging to distinguish the proposed method from its empirical evaluation. A more structured organization with clearly demarcated methodology and experiment sections would significantly improve readability. The proposed equations in this paper lack corresponding explanations or clarifications.
8. Several language and formatting issues throughout the paper: Typo: "idetified" instead of "identified" in footnote 3; Misplaced comma in "Our experiments found that a non-uniform dropping of experts per layer by estimating c globally creates bottleneck layers, with some layers having significantly high sparsity while some remain unpruned, leading to diminished finetuning benefits and sharding simplicity."; Missing commas or periods after equations 1-18; Missing articles: "by gating function" should be "by the gating function" (line 232)

I am open to discussion and willing to reconsider my score if my major concerns are resolved.

• Some experimental details are missing or hard to find. Please refer to the questions for more details.
• The empirical studies, particularly those concerning the identification of the most effective expert-pruning criteria (e.g., Table 2), were conducted on a single pre-trained MoE model, specifically the Mistral 8x7B Base. This limitation could potentially undermine the validity of the empirical findings in Section 3.1, as it remains unclear whether these conclusions are generally applicable.
• The expert pruning criteria appear suitable only for considering expert pruning within each layer independently. As stated in line 375, non-uniform dropping of experts leads to poorer performance.