Retraining-Free Merging of Sparse Mixture-of-Experts via Hierarchical Clustering

C2. O-prune [1] also conducts experiments on domain-specific tasks (e.g., GSM8K, Math). I am interested in the performance of HC-SMoE on these datasets.

Response: We appreciate the reviewer's interest in HC-SMoE's performance on domain-specific datasets.

Our experimental results indicate that HC-SMoE achieves comparable performance to O-prune on GSM8K dataset, which substantiates the effectiveness of our methodology. HC-SMoE accomplishes these results with significant computational efficiency, and requires less than 1/10 of the runtime compared to O-prune when performing on Mixtral models. Furthermore, when applied to models with increased expert counts, such as Qwen, HC-SMoE maintains optimal performance within practical timeframes, which demonstrates its scalability and effectiveness for complex MoE architectures. The detailed experimental results are presented in Table r11 for Qwen models and Table r12 for Mixtral models.

Table r11. Comparison of 5-shot GSM8k accuracy between HC-SMoE and O-Prune-100 on Qwen1.5-MoE-A2.7B-Chat using strict and flexible matching across different calibration datasets. Please note that due to Qwen's large expert count, evaluation of O-prune was limited to 100 expert combinations per layer, as exhaustive computation would require prohibitive runtime for practical deployment.

Table r12. Comparison of 5-shot GSM8k accuracy between HC-SMoE and O-Prune on Mixtral 8x7B using strict and flexible matching across different calibration datasets. Please note that the computational requirements of O-prune exceed other methods by a factor of ten to thirty, which presents significant practical limitations for model compression tasks.

Rebuttal:

We appreciate the reviewer’s valuable feedback and effort spent on the review and would like to respond to the reviewer’s questions as follows.

C1. O-prune [1] requires enumerating all possible combinations of experts, resulting in significant time overhead. I would like to know how HC-SMoE compares to other approaches in terms of runtime and resource consumption.

Response: We appreciate the reviewer's interest in the runtime and resource consumption comparison of HC-SMoE with other approaches. The following sections provide a detailed analysis of the computational requirements.

The experimental evaluation of HC-SMoE utilized five L48 GPUs for Mixtral and four V100 GPUs for Qwen. The reported runtimes exclude model loading and dataset preparation overhead to focus on algorithmic efficiency. The results are presented in Tables r9 and r10.

O-prune requires exhaustive evaluation of expert combinations, which induces substantial computational overhead. This limitation becomes more pronounced as the number of experts increases in MoE architectures. The computational constraints restricted our evaluation of O-prune on Qwen to 100 random combinations per layer for performance estimation.

In contrast, HC-SMoE completes its operations in under ten minutes for comparable settings. HC-SMoE exhibits slightly longer runtimes than F-prune, S-prune, and M-SMoE due to the hierarchical clustering process, which requires iterative distance recalculations and cluster groupings. Nevertheless, HC-SMoE demonstrates substantial efficiency advantages over O-prune. For example, O-prune requires approximately one hour to achieve a 50% pruning ratio on Mixtral, whereas HC-SMoE completes the same task in under two minutes.

Table r9. Computational costs of pruning Mixtral 8x7B to average six (25% pruning ratio) and four (50% pruning ratio) experts per layer for HC-SMoE and four baselines.

Table r10. Computational costs of pruning Qwen1.5-MoE-A2.7B-Chat to average 45 (25% pruning ratio) and 30 (50% pruning ratio) experts per layer for HC-SMoE and four baselines.

The following equations formulate the Silhouette score and Dunn index:

Silhouette Score

For each data point $i$ in cluster $C_I$, let $a(i)$ denote the mean distance between $i$ and all other data points in the same cluster, and let $b(i)$ represent the smallest mean distance of $i$ to all points in any other cluster, defined as:

$

a(i) = \frac{1}{|C_I|-1}\sum_{j\in C_I, i\neq j}d(i, j), \quad b(i) = \min_{J\neq I}\frac{1}{|C_J|}\sum_{j\in C_J}d(i,j),

$

where $d(i, j)$ represents the distance function between data points $i$ and $j$. The Silhouette score of a single data point $i$ is formulated as follows:

$s(i)=\frac{b(i)-a(i)}{\max(a(i), b(i))},\quad \text{if }|C_i| > 1 , \quad s(i)=0, \quad \text{if }|C_I|=1$

To calculate a single Silhouette score for all experts, the mean of the Silhouette scores from all data points is used as the representative:

$

S = \frac{1}{n}\sum_{i=1}^n s(i).

$

The range of Silhouette scores is from +1 to -1, with +1 indicating well-separated clusters.

Dunn Index

The Dunn Index is an internal evaluation scheme for measuring the quality of clustering algorithms. Let $C_I$ be a cluster of vectors and $x, y$ be any two feature vectors. The intra-cluster distance is defined as follows:

$

\Delta_I = \max_{x,y\in C_I}d(x, y)

$

The inter-cluster distance is defined as follows:

$

\delta(C_I,C_J) = \min_{x\in C_I, y\in C_J}d(x, y)

$

The Dunn Index equals the ratio of the minimum inter-cluster distance to the maximum intra-cluster distance for $m$ clusters:

$

DI_m = \frac{\min_{1\leq i < j \leq m}\delta(C_i, C_j)}{\max_{1\leq k\leq m}\Delta_k}

$

These formulations provide a mathematical foundation for evaluating the quality of clustering results in the context of the ablation study.

C4. In Fig. 1, why did you not compare the methods on the 14B model?

Response: Fig. 1 presents a comparison of HC-SMoE with other baseline methods on Qwen1.5-MoE-A2.7B-Chat. The original Qwen1.5-MoE-A2.7B-Chat model comprises 14B parameters, which serves as the reference point denoted by a gray star to indicate its average accuracy across eight language tasks. The comparison excludes pruning methods for the 14B configuration since it represents the original, unmodified model architecture. The 11B and 8B model variants correspond to Qwen 45x2.7B and Qwen 30x2.7B respectively, which are derived through expert pruning at rates of 25% and 50%.

The updated manuscript incorporates additional data points at 9.5B parameter count. These configurations result from expert pruning at rates of 37.5%, and provide more extensive comparative results.

---

C5. Section 3.2.1 presents the method of similarity metric but contains a lot of discussions about related work.

Response: We appreciate the reviewer's feedback. The rationale for incorporating related literature discussion in Section 3.2.1 stems from the necessity to establish context for the proposed similarity metrics through comparison with established methodologies. This contextualization reveals the foundations that motivated our design decisions and demonstrates the capabilities of the proposed approach to address specific limitations in existing methods. Moreover, we have refined the section to emphasize the mathematical formulations and core concepts that form the basis of the similarity metrics.

We hope these revisions address the reviewer's question about the structure and content of Section 3.2.1.

---

C6. In line 299/300, is alpha_i fixed or not? If it is fixed, will it also suffer from the issue that you mentioned in line 199-203 about frequency-based methods?

Response: We appreciate the reviewer's question. In frequency-weighted merging, $\alpha_i$ is determined based on the usage frequency of the corresponding experts on the calibration dataset. We use frequency as a guide to weigh the experts that contribute to the merged expert.

Regarding the question about lines 199-203, we had noted two key issues with M-SMoE's frequency-based approach for determining expert retention per layer. First, the number of experts per layer serves as a crucial hyperparameter for MoE, as it determines the capacity to store knowledge at that layer, and different tasks require varying amounts of knowledge across layers. HC-SMoE addresses this by decoupling the number of experts in each layer from frequency compared to M-SMoE. This ensures that this parameter remains independent of task-specific data.

Moreover, in M-SMoE's merging strategy, experts with high usage frequency are never merged together due to an assumption that frequent activation indicates dissimilarity. However, high frequency alone does not guarantee that two experts produce dissimilar outputs. Our method addresses this through hierarchical clustering based on expert outputs. This enables accurate groupings of experts that exhibit similar behavior.

The proposed frequency-weighted merging therefore avoids these two drawbacks. To demonstrate the robustness of the proposed method, we evaluated merging results across different datasets through validation experiments using average merging, as presented in Table 7 in our manuscript. The experimental results demonstrate that the choice of merging strategy produces minimal impact relative to the influence of the clustering algorithm and similarity metric.

We hope the explanation clarifies the distinctions between our approach and M-SMoE's frequency-guided merging.

---

C7. In Table 4, the best performance of 'ARC-c' should be the Average linkage using the Weight setting, right?

Response: We appreciate the reviewer's attention to our typo. We have revised Table 4 in the updated manuscript to highlight the highest score of ARC-c in bold.

C3. The choice of the calibration dataset. I did not see any ablation study about the choice of the calibration dataset, and I think the performance of the proposed method should highly depend on the calibration dataset. If the calibration dataset is not comprehensive enough, e.g., not covering enough domain specific data, the clustering may not be very informative, which may lead to poor performance. For example, if we want the LLM to perform well on a law-related or medical-related task, can you also rely on the same calibration dataset used in the experiments?

Response: We appreciate the reviewer for raising the question about the choice of the calibration dataset. To address this concern, we conducted two sets of experiments to demonstrate that the calibration dataset choice has minimal impact on the eight language tasks presented in the original manuscript and the additional experiments on the medical-related MedMCQA task [1].

Calibration Dataset Comparison

We evaluated our approach using three different calibration datasets: C4 [2], MATH [3], and CodeQA [4]. These datasets vary in their domain focus:

1. C4: General-purpose and closely aligned with language tasks.
2. MATH: Focused on math question answering.
3. CodeQA: Addresses Python code question answering.

The results presented in Table r8 demonstrate that the performance across the eight evaluated language tasks remains consistent, regardless of the calibration dataset used. Although C4 aligns most closely with general language tasks, the clustering results remain stable even when using domain-specific datasets such as MATH or CodeQA. This finding indicates that for general language tasks, the choice of calibration dataset has only a negligible impact on the effectiveness of our method.

Domain-Specific Evaluation

To further evaluate domain-specific scenarios, we conducted experiments on MedMCQA, a challenging medical-related task. MedMCQA is a large-scale Multiple-Choice Question Answering (MCQA) dataset specifically designed to address real-world medical entrance exam questions. It contains over 194,000 high-quality multiple-choice questions sourced from the AIIMS and NEET PG entrance exams, covering 2,400 healthcare topics across 21 medical subjects. In the evaluations, a two-shot prompt is utilized as follows:

“Please choose one option among A,B,C,D to answer the question.

Question: Chronic urethral obstruction due to benign prismatic hyperplasia can lead to the following change in kidney parenchyma

Options:

A. Hyperplasia

B. Hyperophy

C. Atrophy

D. Dyplasia

Ans:C

 

Question: All of the following are surgical options for morbid obesity except -

Options:

A. Adjustable gastric banding

B. Biliopancreatic diversion

C. Duodenal Switch

D. Roux en Y Duodenal By pass

Ans:D

 

Question: {data.question}

Options:

A. {data.opa}

B. {data.opb}

C. {data.opc}

D. {data.opd}

Ans:”

The model's output is assessed based on whether it generates {A, B, C, D} in the following three tokens.

For these experiments, we used the validation set of MedMCQA for evaluation and its training set as the calibration dataset. Given that MedMCQA has an imbalanced answer distribution (950/735/617/514 for answers A/B/C/D) in the validation dataset, we provide a detailed evaluation using precision, recall, and F1-score metrics to ensure a comprehensive analysis.

Our method demonstrated robust performance while maintaining its effectiveness even with datasets tailored to specialized domains. In particular, our approach outperformed all three baselines, which highlights its capability to handle tasks spanning diverse domains.

We hope these additional experiments and analyses adequately address the reviewer's questions regarding the choice of the calibration dataset.

[1] Ankit Pal et al., MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering, Proceedings of the Conference on Health, Inference, and Learning, 2022

[2] Colin Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, Journal of machine learning research, 2020

[3] Dan Hendrycks et al., Measuring mathematical problem solving with the math dataset, NeurIPS 2021.

[4] Chenxiao Liu et al., CodeQA: A Question Answering Dataset for Source Code Comprehension, EMNLP 2021.

C2. Besides, the paper proposed to use a "hard" hierarchical clustering, and I am wondering if it is more effective to use "soft" hierarchical clustering or simply "soft" clustering without hierarchies.

Response: We appreciate the reviewer for raising the question of considering soft clustering in the MoE merging framework. To explore this, we implemented the Fuzzy C-Means (FCM) algorithm [1], which allows each expert to belong to multiple clusters with varying degrees of membership.

Let $E = \{e_1, \ldots, e_n\}$ denote the $n$ experts to be clustered. The FCM algorithm outputs $c$ cluster centers $C = \{c_1, \ldots, c_c\}$ and a partition matrix. The degree of membership of expert $e_i$ in cluster $c_j$ is denoted by $u_{ij} \in [0, 1]$. FCM minimizes the following objective function:
$

J_m = \sum_{i=1}^N \sum_{j=1}^C u_{ij}^m |e_i - c_j|^2 \quad \ldots \quad (1),
$

where the membership degrees and cluster centers are updated iteratively as follows:
$

u_{ij} = \left(\sum_{k=1}^C \left(\frac{|e_i - c_j|}{|e_i - c_k|}\right)^{\frac{2}{m-1}}\right)^{-1}, \quad c_j = \frac{\sum_{i=1}^N u_{ij}^m \cdot e_i}{\sum_{i=1}^N u_{ij}^m} \quad \ldots \quad (2)
$

For our experiments, we set the hyperparameter $m = 2$.

Nevertheless, applying soft clustering introduces ambiguity in our frequency-weighted merging method. To address this, we modified the merging process. For each cluster, the final merged expert is computed as a weighted sum of the experts, where the membership degree serves as the weight:
$

e^{c_j} = \sum_{i=1}^n u_{ij} e_i \quad \ldots \quad (3)
$

Since HC-SMoE does not modify the router weights (as presented in Fig. 2(c) of the manuscript), input tokens are routed to the corresponding merged expert based on their original assignment. In the FCM setting, this direct routing becomes infeasible, as every expert belongs to all clusters to some degree. To adapt, we merged the router weights using the same weighted formula as above.

Table r6 and r7 compare HC-SMoE with FCM. The results indicate a significant accuracy degradation when using FCM, which is likely due to interference in the router weights. This performance decline can be attributed to the fundamental differences between hard and soft clustering. In hard clustering, each expert belongs to exactly one cluster, which allows for a clear and unambiguous assignment of input tokens to merged experts. In contrast, soft clustering assigns each expert to multiple clusters with varying degrees of membership, which could lead to a more complex and potentially less effective routing process. Moreover, the weighted merging of router weights in the FCM setting may introduce noise and dilute the specialized knowledge captured by individual experts. This dilution can hinder the model's ability to effectively route input tokens to the most relevant experts, resulting in suboptimal performance.

These findings highlight that applying soft clustering methods in the MoE merging framework requires a more sophisticated design to handle router weights, particularly in retraining-free settings. We consider it an interesting direction for future exploration to develop novel techniques that can leverage the benefits of soft clustering while mitigating the challenges associated with routing and merging in the context of MoE models.

Table r6. Comparison of HC-SMoE and Fuzzy-Cmeans on Qwen1.5-MoE-A2.7B-Chat.

[1]: Bezdek et al., FCM: The fuzzy c-means clustering algorithm, 1984

Rebuttal: We appreciate the reviewer’s valuable feedback and effort spent on the review and would like to respond to the reviewer’s questions as follows.

C1. The motivation of using the "Hierarchical" clustering is not clear to me. I cannot intuitively get the idea of why hierarchical clustering is better than simple K-means clustering, although the results confirmed that K-means clustering is less effective.

Response: We appreciate the reviewer for raising the question about the motivation behind using hierarchical clustering instead of K-means. We elaborate on this by providing a detailed explanation structured into two parts: theoretical justification and experimental validation.

Theoretical Justification

We selected hierarchical clustering over K-means based on two primary reasons: its stability and determinism. The initialization of cluster centroids in K-means is often random, which can result in different clustering outcomes across multiple runs on the same dataset [1]. This non-deterministic behavior of K-means makes it less suitable for tasks where the reproducibility and consistency of clustering results are crucial, such as in the evaluation of downstream tasks. In contrast, hierarchical clustering generates deterministic results for a given dataset and linkage method. This deterministic property ensures that the clustering results are reproducible and consistent across different runs. Furthermore, hierarchical clustering employs a systematic approach to merge clusters based on a specified linkage criterion. This linkage criterion determines the distance between clusters and governs the merging process. By optimizing the linkage criterion, hierarchical clustering guarantees the formation of stable clusters that minimize the intra-cluster distance and maximize the inter-cluster distance. This optimization ensures that the resulting clusters are compact and well-separated, which is desirable for effective pruning and merging of experts in the context of model compression.

Experimental Validation

In addition to the theoretical justification, we further validated the effectiveness of hierarchical clustering compared to K-means through experiments. We conducted experiments using both methods across three similarity metrics (i.e., router logits, expert outputs, and expert weights) and evaluated six clustering criteria:

1. L2 distance: $||T(x)−S(x)||_2$, where $T(x)$ and $S(x)$ represent the outputs of the original and pruned models, respectively. Lower values are better.
2. Cosine similarity: $\text{cosine-similarity}(T(x),S(x))$. Higher values are better.
3. Silhouette score (Euclidean): Measures how similar an object is to its cluster compared to other clusters, using Euclidean distance. Higher values are better.
4. Dunn index (Euclidean): Evaluates cluster compactness and separation, using Euclidean distance. Higher values are better.
5. Silhouette score (Cosine): Similar to (3) but based on cosine similarity. Higher values are better.
6. Dunn index (Cosine): Similar to (4) but based on cosine similarity. Higher values are better.

Silhouette score evaluates clustering quality at the data point level ( i.e., expert level), while the Dunn index evaluates it at the cluster level. The Dunn index considers maximum intra-cluster and minimum inter-cluster distances, while the Silhouette score uses mean distances. Both metrics highlight clustering compactness and separability. We excluded evaluations involving the cosine similarity of expert weights due to the high computational cost of processing concatenated weight tensors, which would require excessive GPU memory. The detailed formulation is provided at the bottom of this response.

Table r5 summarizes the results. Hierarchical clustering with expert outputs achieves the lowest L2 error and the highest cosine similarity with the original model outputs at 25% and 50% pruning ratios. Moreover, hierarchical clustering consistently outperforms K-means across most clustering metrics, demonstrating better clustering quality. These results substantiate the stability and effectiveness of hierarchical clustering in producing compact, well-separated clusters. Furthermore, the zero-shot performance on eight language tasks, as demonstrated in Table 5 in our manuscript, further supports the superiority of hierarchical clustering. Across all tasks, hierarchical clustering consistently outperforms K-means, and achieves better and more stable accuracy.

Based on the above reasons, hierarchical clustering is preferable for its deterministic nature, superior clustering quality, and consistent performance across similarity metrics and benchmarks. We hope this explanation clarifies the advantages of hierarchical clustering over K-means.

Rebuttal: We appreciate the reviewer’s valuable feedback and effort spent on the review and would like to respond to the reviewer’s questions as follows.

C1. While this work demonstrates competitive accuracy, it lacks a comprehensive assessment of efficiency metrics, such as speedup and memory usage. Given that efficiency is a key contribution, this aspect of the experimental results is essential.

Response: We appreciate the reviewer for bringing attention to the importance of efficiency metrics in evaluating our proposed method. To address this, we have included an assessment of throughput (token per ms), latency (s), FLOPs, memory usage, and model size for both the Mixtral 8x7B and Qwen1.5-MoE-A2.7B-Chat models, as well as their pruned versions, in Table 10 of the updated manuscript. Furthermore, we have provided an additional description and discussion of these efficiency metrics in Appendix B.

In the newly introduced experiments, we utilized the same calibration dataset as the main experiments, which consists of 32 sequences of 2,048 tokens sampled from the C4 corpus. As revealed in Table 10, HC-SMoE achieves significant reductions in memory usage and FLOPs by decreasing the number of experts while maintaining the same throughput and latency. Although reducing the number of experts may inherently decrease router latency by reducing output channels, we have chosen to retain the original router weights to avoid compromising accuracy. As a result, the router operates as if the original number of experts is preserved, while experts within the same group produce identical outputs through the corresponding merged expert.

We hope that this evaluation clarifies the computational and memory costs of the proposed model. Below is the table we provided in the manuscript's Table 10.

Table 10 (in our manuscript). Evaluation of computational and memory efficiency across multiple models. For Mixtral: Mixtral 8x7B (original), Mixtral 6x7B (25% pruned), and Mixtral 4x7B (50% pruned). For Qwen1.5-MoE-A2.7B-Chat: Qwen 60x2.7B (original), Qwen 45x2.7B (25% pruned), and Qwen 30x2.7B (50% pruned). All measurements use identical input sequences and include throughput (tokens per ms), latency (s), GFLOPs, model memory, and model size (number of parameters).

C3. Although HC-SMoE is validated on eight zero-shot language tasks, its effectiveness may vary in more complex tasks or domains, potentially limiting its broader applicability.

Response: To evaluate HC-SMoE on more complex domain-specific tasks, we conducted additional experiments on MedMCQA [1], a challenging medical question-answering dataset. Table r4 presents the experimental results, which demonstrate HC-SMoE's robust performance in domain-specific applications. The performance of HC-SMoE exceeds all three baseline methods in this specialized domain.

For the reviewer's reference, MedMCQA comprises a large-scale Multiple-Choice Question Answering (MCQA) dataset designed for real-world medical entrance examinations. The dataset contains over 194,000 high-quality multiple-choice questions from AIIMS and NEET PG entrance exams, which span 2,400 healthcare topics across 21 medical subjects. Our experiments utilize a two-shot prompt format:

“Please choose one option among A,B,C,D to answer the question.

Question: Chronic urethral obstruction due to benign prismatic hyperplasia can lead to the following change in kidney parenchyma

Options:

A. Hyperplasia

B. Hyperophy

C. Atrophy

D. Dyplasia

Ans:C

 

Question: All of the following are surgical options for morbid obesity except -

Options:

A. Adjustable gastric banding

B. Biliopancreatic diversion

C. Duodenal Switch

D. Roux en Y Duodenal By pass

Ans:D

 

Question: {data.question}

Options:

A. {data.opa}

B. {data.opb}

C. {data.opc}

D. {data.opd}

Ans:”

The evaluation protocol assesses whether the model outputs {A, B, C, D} in the subsequent three tokens.

The experimental methodology employs the MedMCQA validation set for evaluation and its training set for calibration. Due to the imbalanced answer distribution in the validation dataset (950/735/617/514 for answers A/B/C/D), we present a comprehensive analysis through precision, recall, and F1-score metrics.

Table r4. Experimental results of HC-SMoE and three baselines on MedMCQA with Mixtral 8x7B and the compressed version of reducing experts to six and four per layer.

[1] Ankit Pal et al., MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering, Proceedings of the Conference on Health, Inference, and Learning, 2022

C2. A theoretical analysis of the effectiveness of expert merging and HC-SMoE would enhance the understanding of the method's performance.

Response: We are thankful to the reviewer for paying attention to the theoretical analysis of HC-SMoE and expert merging. Our response addresses the theoretical foundations and benefits of expert merging in our approach.

The effectiveness of expert merging originates from fundamental principles in neural network optimization and representation learning. Multiple experts that exhibit similar output patterns likely encode redundant information or perform comparable transformations on the input data. The identification and merging of these functionally similar experts enables model redundancy reduction while it preserves essential computational capabilities.

The weighted merging process is preferred since it preserves model performance through two fundamental mechanisms. First, experts with similar outputs operate within the same functional subspace, which indicates their capture of related features and transformations. The merged expert retains this shared functionality through the elimination of redundant parameters. Second, the weighted merging approach maintains the relative order of row vectors in the expert network weights, which ensures minimal perturbation to the learned feature representations. Compared to weighted merging, fix-dominant merging, the third merging method we proposed, disturbs the feature order inside the weight matrices of the expert network, please refer to appendix A.2 for detailed explanation.

The experimental evidence in Table 7 of our manuscript demonstrates that effective clustering serves as the primary determinant of model performance. This finding emerges from the observation that even basic average merging strategies achieve superior results. The theoretical foundation for this phenomenon rests in the functional equivalence among experts within each cluster, which manifests through their consistent output patterns. This fundamental property ensures robust model performance across different merging methodologies.

Table r3. Zero-shot performance evaluation of HC-SMoE and four baseline methods on Mixtral 8x7B-v0.1 with expert reduction to three and two per layer. The runtime for each algorithm is provided in seconds. Please note that the computational requirements of O-prune exceed other methods by a factor of five to ten, which presents significant practical limitations for model compression tasks.

C2. I am wondering about the results or analysis for more extreme expert reduction scenarios, such as reducing to 25% or 10% of the original experts. This would give insight into how the method performs under more aggressive compression.

Response: We appreciate the reviewer for raising this question. To evaluate extreme pruning scenarios, we conducted additional experiments at substantial compression rates of 62.5% and 75%. Our analysis compares HC-SMoE against four established baselines: F-prune, S-prune, O-prune and M-SMoE. The results for the Qwen1.5-MoE-A2.7B-Chat and Mixtral 8x7B models are presented in Table r2 and r3, respectively.

Several benchmark tasks employ multiple-choice formats. ARC-c, ARC-e, HellaSwag, MMLU, and OBQA require selection from four options, with 0.25 as the random-guess baseline. BoolQ, RTE, and Winogrande utilize binary choices, with 0.5 as the random-guess baseline. Scores in proximity to these baselines indicate a substantial deterioration of model capabilities.

Under extreme reduction settings, our experiments reveal that all baselines experience significant accuracy degradation, with performance often falling below random-guess baselines. In contrast, HC-SMoE maintains competitive accuracy even at a 75% reduction through its output-based clustering and merging strategy. This finding can be attributed to the fact that experts with similar outputs are likely to capture related features or patterns in the data. Merging these experts allows the model to preserve essential information while reducing redundancy, which enables a more compact representation without compromising performance.

The evaluation excludes 90% reduction scenarios because such extreme pruning would leave only one expert in the Mixtral model, which conflicts with its top-2 routing policy. Nevertheless, the results obtained at 62.5% and 75% pruning rates provide substantial evidence that HC-SMoE can effectively operate under extreme compression settings.

Table r2. Zero-shot performance evaluation of HC-SMoE and three baseline methods on Qwen1.5-MoE-A2.7B-Chat with expert reduction to 23 and 15 per layer. We exclude O-prune for this experiment due to the large search space.

Rebuttal:

We appreciate the reviewer’s valuable feedback and effort spent on the review and would like to respond to the reviewer’s questions as follows.

C1. There is a lack of theoretical analysis on the performance of expert clustering using different similarity metrics.

Response: Our response addresses this through three parts: formal metric analysis, empirical evaluation, and experimental results.

Formal Analysis of Similarity Metrics

We begin by defining three similarity metrics evaluated in our study. For an input token $x$:

• Router logits $R(x)$: Calculated as $R(x) = x \cdot W_R$, where $W_R$ represents the router weight.
• Expert weights $W$: Concatenated expert weights, $W = \text{flatten}(W_{gate}||W_{down}||W_{up})$, which remain independent of the input $x$.
• Expert output $E(x)$: Defined as the output of each expert without multiplying routing scores, expressed as $E(x) = (\sigma(x\cdot W_{gate}) \odot (x\cdot W_{up}))\cdot W_{down}$.

Router logits primarily reflect the assignment preference of tokens and depend on input distributions. These logits capture local routing decisions rather than global functional similarity between experts in an MoE layer. Similar router logits may indicate shared importance for certain tokens but fail to reflect functional equivalence.

Expert weights present two limitations. First, their processing requires substantial computational overhead. Second, the high dimensionality of weight spaces relative to input/output dimensions obscures functional relationships. Different weight configurations can produce similar outputs, and similar weights may generate functionally distinct behaviors due to the complex dependencies in high-dimensional parameter spaces.

Expert outputs directly reflect the functional behavior of experts in an MoE layer [1-3]. The average expert outputs over a calibration dataset with $T$ tokens, computed as $\frac{1}{T}\sum^T_{i=1}E(x_i)$, capture this functional behavior. This establishes expert output as the most appropriate similarity metric for clustering.

Experimental Validation

Our analysis of layer outputs uses two quantitative criteria:

1. L2 distance: $|| T(x) - S(x) ||_2$​, where $T(x)$ and $S(x)$ represent the outputs of the original and pruned models. Lower values indicate better performance.
2. Cosine similarity: $\text{cosine-similarity}(T(x), S(x))$. Higher values indicate better alignment.

Table r1 shows that hierarchical clustering based on expert outputs achieves the lowest L2 error and highest cosine similarity with original model outputs across 25% and 50% pruning ratios. As demonstrated in the literature [1,2,3], outputs that closely resemble the original model correlate strongly with improved performance in reduced model configurations. These results validate that experts with similar outputs maintain similar functionality and can serve as strong candidates for merging with minimal performance impact.

Table r1. The L2 error and cosine similarity of last layer output between the original model Qwen1.5-MoE-A2.7B-Chat and the model produced by hierarchical clustering with three different metrics at 25% and 50% pruning ratio.

Zero-Shot Evaluation Results

The zero-shot evaluation results on eight language tasks (Tables 4 and 6 in the manuscript) demonstrate that output-based clustering achieves higher average accuracy compared to weights or router logits across clustering methods. This improved performance follows from the direct relationship between expert outputs and model functionality, which preserves essential computational patterns after merging.

We thank the reviewer for the question and hope this analysis provides sufficient clarification.

[1] Liang et al., Less is More: Task-aware Layer-wise Distillation for Language Model Compression, ICML 2023

[2] Romero et al., FitNets: Hints for Thin Deep Nets, ICLR 2015

[3] Hinton et al., Distilling the Knowledge in a Neural Network, NeurIPS 2014