We appreciate the reviewer’s valuable feedback and effort spent on the review, and would like to respond to the reviewer’s questions as follows.

Q1. Theoretical Gaps: No formal analysis of clustering quality or merging stability.

Response: We appreciate the reviewer's suggestion regarding theoretical analysis. We would like to direct the reviewer to refer to the general response section. Please refer to the theoretical justification link below.

• theoretical justification

Q2. Experiments are comprehensive but lack latency/throughput comparisons (Table 19 only reports FLOPs and memory). Inference efficiency gains from merging are unclear.

Response: We thank the reviewer for this observation. Our primary objective focuses on reducing parameter count in MoE models without retraining while maintaining computational efficiency. Table 19 demonstrates significant reductions in FLOPs and memory usage, which highlights the efficiency gains achieved through our approach.

It is important to note that our method preserves the top-K routing mechanism, wherein each token continues to be assigned to $K$ experts per MoE layer, the same as the original model. As a result, when we reduce the number of experts in each layer to $r$ while ensuring $r \ge K$, the inference cost remains equivalent to that of the original model. While merging reduces storage and computational requirements per expert, the routing mechanism dictates that inference cost per token primarily depends on K rather than the total number of experts.

Q3. Does HC-MoE cause a huge impact on the semantic space of the original model? It is a pity that neither of these issues is explicitly explained in the text.

Response: To address concerns regarding potential semantic shifts, we have incorporated t-SNE visualizations that compare expert representations before and after merging. These results demonstrate that HC-SMoE effectively preserves the model's semantic coherence, as the expert distributions maintain substantial consistency after the merging process.

Specifically, we visualize the first MoE layer of Mixtral-8x7B, utilizing the same calibration dataset described in Section 4.1. We collect 65,536 output vectors per expert, each with a hidden dimension of 4,096, by directing identical MoE input tokens to all experts. Each point in the t-SNE plot represents the average of 128 token outputs, which then undergoes projection into two dimensions using sklearn.manifold.TSNE with n_components = 2. The analysis yields several key observations:

• With perplexity = 8, the t-SNE visualization of the original model reveals a clear eight-cluster structure, which corresponds to the eight experts.
• After applying HC-SMoE to reduce the expert set to six experts, the resulting t-SNE plot maintains a well-defined six-cluster structure, despite some overlap. This indicates that HC-SMoE maintains expert specialization to a significant extent, and preserves the output distribution of the original model.

It is important to note that t-SNE visualization at the single-token level would appear as random noise without discernible cluster structure. This occurs because individual token embeddings exist in high-dimensional space and do not inherently form clusters without appropriate aggregation.

• t-SNE of each expert’s output of the original Mixtral8x7B's first layer.
• t-SNE of each expert’s output of first layer after HC-SMoE which reduced each layer in Mixtral8x7B to 6 experts.
• t-SNE of each expert’s output of the original model layer where each point indicates a single output token.

Due to space limitations, we can only provide concise responses here. We have more detailed and comprehensive answers regarding the concerns on ethical statement, recommend related works on cluster-based routing MoE as well as figure 2's enhancement and other questions, which we look forward to discussing thoroughly with the reviewer in the next phase of the review process.

We appreciate the reviewer’s valuable feedback and effort spent on the review, and would like to respond to the reviewer’s questions as follows.

Q1. It was not entirely clear to me why Li et al. (2024) proposal is one-shot and why HC is iterative.

Response: The fundamental distinction between these approaches lies in their underlying methodologies. Li et al.'s expert pruning operates through a one-shot mechanism, wherein only the top $r$ experts with the highest routing scores are retained based on a single evaluation of the data. This process involves computing routing scores by averaging results from a forward pass across the entire calibration dataset, followed by sorting experts according to these averaged scores to identify the top rr r candidates. This methodology proceeds in a direct, non-iterative manner.

In contrast, our HC-SMoE method employs an iterative approach. Hierarchical clustering necessitates multiple sequential steps to systematically merge experts into clusters. Each step selects the optimal pair of clusters (or experts) to combine based on minimizing the average intra-cluster distance. This iterative procedure continues until precisely $r$ clusters are established, ensuring that each clustering decision optimizes the process by minimizing clustering error at every iteration.

Q2. It would have been nice to include an algorithm in the paper.

Response: We acknowledge the reviewer's recommendation regarding a more detailed description of the algorithm, and we have provided the link to algorithm. Specifically, in Algorithm 1, we outline the steps of hierarchical clustering and expert merging in our methodology.

Q3. When using HC, are the experts merged at every step when creating the dendrogram? Or the experts are merged only at the desired step of say 25% sparsity?

Response: To clarify the merging process: The procedure initially involves clustering experts through the construction of a dendrogram, which provides a hierarchical representation of expert relationships. Upon completion of the dendrogram, we proceed to merge the experts at the final stage to achieve the specified sparsity level. This means that experts are not merged at every step of the dendrogram creation. Instead, they are merged only once the final clustering decision is made, based on the desired sparsity or number of clusters.

We hope this explanation adequately addresses the reviewer’s concerns.

We appreciate the reviewer’s valuable feedback and effort spent on the review, and would like to respond to the reviewer’s questions as follows.

Q1. Please consider adding baseline results to see if HC-SMoE still outperforms the baselines when the calibration dataset and the evaluation dataset have larger distribution differences.

Response: We acknowledge the reviewer's valuable inquiry regarding HC-SMoE's performance under distribution shifts. To evaluate the robustness of HC-SMoE against calibration dataset distribution shifts, we conducted additional experiments with MATH and CodeQA as calibration datasets. The results demonstrate the following:

• HC-SMoE consistently maintains superior or equivalent performance compared to all baselines across all configurations on Qwen1.5-MoE-A2.7B-Chat, which demonstrates its capacity to generalize effectively across diverse calibration distributions.
• When implementing 25% pruning with MATH calibration on Mixtral8x7B, S-prune exhibits marginally superior performance compared to HC-SMoE. However, S-prune demonstrates significantly inferior performance in all other experimental scenarios.
• F-prune exhibits substantial performance degradation when utilizing MATH and CodeQA, which indicates that pruning methodologies based solely on frequency or routing scores lack stability, whereas HC-SMoE maintains robust performance regardless of the calibration dataset employed.

The experimental results with calibration datasets MATH and CodeQA are presented as follows. The red color indicates performance decreases relative to calibration dataset C4, while the green color signifies performance improvements. We excluded O-prune from these experiments as its original publication [4] provides comprehensive analysis regarding the impact of calibration datasets.

• Qwen on MATH
• Qwen on CodeQA
• Mixtral on MATH
• Mixtral on CodeQA

Q2. The proposed method requires a calibration dataset. How large is this dataset in the experiments? How sensitive is the performance of the proposed algorithm to the size of this dataset?

Response: Section 4.1 of our original manuscript provides comprehensive details regarding the calibration dataset size. To further evaluate the sensitivity of our method to dataset size, we conducted an additional experiment using Qwen with 50% expert parameter pruning while varying the calibration dataset size (16, 32 (original), and 64 examples). The results indicate that the average performance across eight zero-shot benchmarks remains remarkably consistent, which highlights the robustness of HC-SMoE with respect to calibration dataset size.

• Different size of calibration dataset

Q3. Related Works on EEP

Response: To provide a clear comparison, we include a table that summarizes the performance and runtime of EEP and HC-SMoE. It merits mention that the reported accuracy for BoolQ and RTE in the EEP paper differs significantly from our results, likely due to differences in evaluation protocols—we use the EleutherAI Language Model Evaluation Harness.

• EEP comparison table
• Mixtral8x7B-Instruct result

Q4. Fix-Dominant Merging

Q4.1 What is the definition of "dominant" in Figure 4?

Response: The definition of “dominant” in fixed-dominant merging refers to a “concept” of dominant expert within a cluster, since it would be the only one expert within the cluster to preserve the original weight vector order. In HC-SMoE, the dominant expert refers to the expert that exhibits the closest proximity to the cluster center.

Q4.2 Could you explain in more detail how the merging is done?

Response: Our default fix-dominant merging methodology implements simple average merging, wherein all experts within a cluster contribute equally to the merged representation.

Due to space limitations, we can only provide concise responses here. We have more detailed and comprehensive answers regarding the concerns on section 3.2, section 4.3 and table 4, fix-dominant merging, explanation on table 4 and 5, as well as EEP comparison, which we look forward to discussing thoroughly with the reviewer in the next phase of the review process.

We appreciate the reviewer’s valuable feedback and effort spent on the review, and would like to respond to the reviewer’s questions as follows.

Q1. Theoretical analysis is still lacking: the theoretical advantages of hierarchical clustering are described vaguely.

Response: Please refer to theoretical justification part in link.

• Theoretical justification

Q2. However, the generalization capability to multimodal domains is not covered, and details on calibration dataset sampling are missing, which may introduce bias.

Response: We appreciate the question from the reviewer. This paper concentrates on the MoE-based language domain. Extension to multimodal domains represents an interesting direction for future research but exceeds the current scope of this work. Future investigations could profitably explore the application of HC-SMoE to additional modalities.

Q3. Comparison of Memory Consumption, Computational Cost, and Overheads We have incorporated a table that compares the runtime and memory usage of HC-SMoE against various baselines. The results demonstrate that HC-SMoE achieves competitive runtime and memory efficiency across different models while maintaining superior performance on benchmarks.

• Mixtral model experiments
• Qwen model experiments

Q4. Choice of Euclidean Distance as Distance Metric for Expert Outputs

Response: In response to the reviewer’s question about the choice of Euclidean distance as the distance metric for expert outputs in hierarchical clustering, we selected the Euclidean distance due to its effectiveness in measuring the similarity between averaged expert outputs, which are high-dimensional vectors. This choice aligns with prior work such as [3][4], where the Frobenius norm was used to measure pruning error between the original model and the pruned model. In our case, the Euclidean distance is a natural fit, given that the expert outputs are vectors in Euclidean space.

In addition, we show in Table 20 of the paper that even with Euclidean distance, our approach yields high cosine similarity in the final layer output, demonstrating that the choice of Euclidean distance does not hinder performance. We also present empirical results indicating that HC-SMoE produces the best cluster quality, as measured by the silhouette score and Dunn index, when compared to K-means. Please note that in Table 20 the similarity scores between different metrics cannot be directly compared, as the silhouette score and Dunn index are computed on different bases.

Q5. What is the specific sampling scheme for the calibration dataset?

Response: Regarding the calibration dataset, we exactly follow the protocol in [4], using a sampling scheme where we randomly select 32 sentences from the C4 dataset, with each sentence containing 2,048 tokens. The same sampling scheme is used across all experiments to ensure fairness and reproducibility.

Q6. Related Work on Model Soup

Response: We acknowledge the related work on Model Soup and its connection to our approach. Our average merging technique shares similarities with the concept of uniform model soup, where multiple models are averaged to create a unified model. However, while uniform model soup typically involves the combination of multiple models into a single entity, HC-SMoE focuses on merging a set of experts into $r$ clusters, where $r<n$. Our approach exhibits greater complexity, as it incorporates expert clustering based on similarity metrics and a frequency-weighted merging process. We will include citations to the relevant work on uniform model soup in the paper to highlight this connection and differentiate our methodology.

Due to space limitations, we can only provide concise responses here. We have more detailed and comprehensive answers regarding the concerns on limitations discussion of HC-SMoE and other questions, which we look forward to discussing thoroughly with the reviewer in the next phase of the review process.