MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition

Thanks for the valuable feedback and recognition. We have tried our best to address all concerns in the last few days. Please see our below responses to your concerns and questions one by one.

---

Q1: More Examples of Matrix Similarity Across MoE Models

A1:

(1) We've conducted additional analysis on Phi-3.5-MoE and DeepSeekMoE (within inherent expert group) that shows similar patterns of V-matrix redundancy across architectures. These models exhibit 0.92 and 0. 81 average CKA similarity among V-matrices individuals, confirming that this property generalizes across different MoE architectures.

(2) Table 3 provides empirical validation that our V-matrix sharing approach works effectively across multiple architectures, confirming that matrix similarity is a general property of MoE models rather than specific to Mixtral.

(3) The theoretical basis for V-matrix redundancy relates to how MoE models are trained. Since all experts process similar input distributions but specialize in different aspects, their output projections (captured by V-matrices) share substantial structure while their internal representations (captured by U-matrices) diverge more significantly.

Q2: Refining U-Matrix Trimming Strategy

A2:

(1) Table 6 shows perplexity trade-offs on Mixtral-8×7B: 4.86 (no trimming) → 5.98 → 7.11 → 10.25 as k increases. k=2 balances performance and speed at high compression.

(2) This value aligns with the standard top-k routing mechanism in MoE architectures, which typically activates 2 experts per token. By retaining 2 U-matrices, we maintain consistency with the model's inherent routing structure.

(3) We further propose Adaptive k Selection Framework to improve U-matrix trimming:

$k_l^* = \arg\min_k {\mathcal{L}(k) + \lambda \cdot \mathcal{Q}(k)}$

where $\mathcal{L}(k)$ represents the estimated performance loss from trimming to k matrices, $\mathcal{Q}(k)$ denotes the parameter count, and $\lambda$ controls the tradeoff. The performance loss is approximated using the information coverage of retained matrices:

$\mathcal{L}(k) \approx 1 - \frac{\sum_{i=1}^k f_i \cdot \sigma_i}{\sum_{i=1}^N f_i \cdot \sigma_i}$

where $f_i$ is the expert sampling frequency and $\sigma_i$ is the sum of singular values, jointly capturing the expert's contribution to model performance.

Using Automatic k Determination, k is computed automatically during compression based on calibration data statistics, without manual tuning. For each layer, the algorithm calculates the marginal utility of increasing k and stops when additional U-matrices provide diminishing returns relative to their parameter cost. Our experiments with Mixtral-8×7B demonstrate the effectiveness of this approach:

• Layer-specific k Distribution: Analysis of the automatically determined k values reveals an intuitive pattern: early and late layers (which our sensitivity metric identifies as more critical) receive higher k values (typically 2-3), while middle layers receive lower values (typically 1-2). This aligns with our understanding of information flow in transformer architectures and confirms that the algorithm captures meaningful layer-wise differences.

These results show that adaptive k selection consistently outperforms fixed k=2, with Adaptive k providing the best performance by preserving more information from the original expert space, particularly at higher compression ratios but U-matrix merging can find the best tradeoff between Runtime and performance.

Q3: Minor Typos and Readability of Table 1

A3:

(1) We'll correct the typo "TThese" in Figure 1's caption and conduct a thorough proofreading of the entire manuscript to ensure clarity and consistency in revision.

(2) Table 1 will be redesigned for improved readability. We'll use a clearer layout with better spacing and more consistent formatting to highlight the key differences between methods.

(3) We'll ensure that method descriptions are concise yet precise, using a consistent terminology throughout the table and the rest of the paper.

---

Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments.

Thank you so much for constructive comments, and recognition. Please see our below responses:

---

Q1: Performance Degradation at Higher Compression Ratios

A1:

(1) MoE-SVD outperforms alternatives at all compression levels. At 60% compression: MoE-SVD=13.52 perplexity; ASVD/SVD-LLM>10,000 perplexity on WikiText-2.

(2) Lightweight LoRA fine-tuning (MoE-SVD†) mitigates degradation at 1% of full training cost. At 50% compression: improves accuracy from 0.43 to 0.49, recovering 9.5% original performance.

(3) Following the general phenomenon of performance loss in almost high-ratio compression, MoE-SVD significantly improves trade-off curve vs. existing approaches.

(4) Task-specific analysis shows reasoning tasks like ARC-e maintain performance at 60% compression, while MathQA shows higher sensitivity.

Q2: Theoretical Justification for the Sensitivity Metric

A2:

(1) Our sensitivity metric integrates three complementary components with theoretical foundations:

(a) Sampling frequency ($f_i$): expert utilization rate determined by router.

(b) Principal rank ($p_i$): effective dimensionality of expert's weight matrix from matrix approximation theory.

(b) Activation outliers ($a_i$): functional distinctiveness of experts.

(2) Information-Theoretic Interpretation of Sensitivity Metric: We have derived our sensitivity metric ( $S_L = Σ f_i · p_i · a_i$) from an information-theoretic perspective please see A2 in responsing to Reviewer Cmem.

We show that under certain assumptions, our metric approximates the expected information loss from decomposition:

$E[I(W_i; Y|X)] - E[I(W̃_i; Y|X)] ∝ f_i · p_i · a_i$

where $I(W_i; Y|X)$ represents the mutual information between expert weights $W_i$ and model output $Y$ given input $X$.

(3) Table 5 validates that combining these components ($f_i · p_i · a_i$) achieves optimal performance (8.67 perplexity) vs. individual components (9.27-12.65 perplexity).

Q3: Fixed k=2 in U-Matrix Trimming and more principled methodology for K selection

A3:

(1) Table 6 shows perplexity trade-offs on Mixtral-8×7B: 4.86 (no trimming) → 5.98 → 7.11 → 10.25 as k increases. k=2 balances performance and speed at high compression.

(2) This value aligns with the standard top-k routing mechanism in MoE architectures, which typically activates 2 experts per token. By retaining 2 U-matrices, we maintain consistency with the model's inherent routing structure.

(3) We further propose Adaptive k Selection Framework to improve U-matrix trimming:

$k_l^* = \arg\min_k {\mathcal{L}(k) + \lambda \cdot \mathcal{Q}(k)}$

where $\mathcal{L}(k)$ represents the estimated performance loss from trimming to k matrices, $\mathcal{Q}(k)$ denotes the parameter count, and $\lambda$ controls the tradeoff. The performance loss is approximated using the information coverage of retained matrices:

$\mathcal{L}(k) \approx 1 - \frac{\sum_{i=1}^k f_i \cdot \sigma_i}{\sum_{i=1}^N f_i \cdot \sigma_i}$

where $f_i$ is the expert sampling frequency and $\sigma_i$ is the sum of singular values, jointly capturing the expert's contribution to model performance. Using Automatic k Determination, k is computed automatically during compression based on calibration data statistics, without manual tuning. For each layer, the algorithm calculates the marginal utility of increasing k and stops when additional U-matrices provide diminishing returns relative to their parameter cost. Our experiments with Mixtral-8×7B demonstrate the effectiveness of this approach:

Q4: Alternative V-Matrix Sharing Strategies

A4:

Yes. For DeepSeek-MoE and Qwen2 (with inherented organized expert groups), we actually employ clustering strategy following architecture setting. We will emphasize this point and add more analysis in the revision.

Q5: About size and layout

A5:

We will carefully revise all size and layout of some tables and figures in the revision. Thanks.

---

Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments.

Thank you very much for your detailed and constructive feedback. We have tried our best to address all concerns in the last few days. Please see our responses below one by one：

---

Q1: About Theoretical Justification

A1:

(1) Theoretical Foundation of Decomposition Sensitivity: We have formalized the relationship between expert sensitivity and model performance by analyzing how SVD decomposition affects information flow in MoE architectures. In Section 3.2, we present a detailed theoretical analysis that demonstrates why initial and final layers are particularly sensitive to decomposition due to their role in translating between embedding space and MoE-specific representations. This explains the empirical observations in Figure 1 (left), where decomposing these layers leads to disproportionate performance degradation.

(2) Information-Theoretic Interpretation of Sensitivity Metric: We have derived our sensitivity metric ( $S_L = Σ f_i · p_i · a_i$) from an information-theoretic perspective please see A2 in responsing to Reviewer Cmem.

We show that under certain assumptions, our metric approximates the expected information loss from decomposition:

$E[I(W_i; Y|X)] - E[I(W̃_i; Y|X)] ∝ f_i · p_i · a_i$

where $I(W_i; Y|X)$ represents the mutual information between expert weights $W_i$ and model output $Y$ given input $X$.

(3) Matrix Redundancy Theory: In Section 3.3, we have added theoretical analysis explaining why V-matrices exhibit higher redundancy than U-matrices in MoE architectures. This analysis builds on prior work in Similarity of Neural Network Representations Revisited (Kornblith et al., 2019), showing that the shared output space constraints in MoE models naturally lead to similar output transformations (V-matrices) while maintaining diverse input transformations (U-matrices) for specialization.

(4) Error Bounds for Matrix Sharing: We have derived error bounds for our V-matrix sharing approach based on matrix approximation theory. For an expert weight $W_i = U_iΣ_iV_i^T$ with selection rank-r approximation $W_i^{r} = U_i^{r}Σ_i^{r}(V_i^{r})^T$, sharing the V-matrix introduces additional error bounded by: $||W_i - U_iΣ_iV_s^T||_F^2 ≤ ||W_i - W_i^r||_F^2 + ||Σ_i||_F^2 · ||V_i^T - V_s^T||_F^2$ . $V_s$ is the shared V-matrix selected based on router sampling frequency as defined in Equation 5 in the paper.This bound becomes tighter as the similarity between V-matrices increases, aligning with our empirical observations of high CKA similarity (average >0.8) between expert V-matrices.

(5) Compression-Performance Tradeoff Analysis: We have added a theoretical framework that characterizes the relationship between compression ratio, parameter efficiency, and model performance. This analysis explains why selective decomposition based on sensitivity metrics achieves a better Pareto frontier than uniform compression approaches. We derive performance bounds that predict deterioration patterns at different compression ratios, validating our experimental findings in Table 2.

Q2: About minimal performance loss

A3:

(1) The term "minimal" is comparative rather than absolute. We will revise our abstract: " For 40-60% compression, MoE-SVD achieves .... while maintaining relative performance superior to other compression methods." in the revision.

(1) Our claim of "minimal performance loss" is relative to other compression methods at the same compression ratio. As shown in Table 2, at 20% compression, MoE-SVD maintains 92% of the original model's average accuracy (0.58 vs. 0.63), while competing methods maintain less than 85%.

Q3: Novelty Relative to Prior Work

A3:

(1) In Table 1 and Appendix D, we have already compared and analyzed of MoE-SVD to provide insights into why our approach succeeds where standard SVD methods (ASVD, SVD-LLM) fail on MoE architectures.

(2) Our V-matrix sharing and U-matrix trimming strategies represent a novel approach to exploiting the unique redundancy patterns in MoE models. We emphasize that these observations and designs not present in existing MoE and SVD methods.

(3) Compared to expert pruning methods (MoE-Compression, MoE-I²), our approach maintains the sparse activation mechanism while making each expert more efficient. This avoids the significant performance drops associated with expert elimination (e.g., 23% accuracy drop when pruning 25% of experts in Mixtral-8×7B).

(4) MoE-SVD requires no retraining and is hardware-independent, making it more practical for real-world deployment than methods requiring extensive fine-tuning or specialized hardware.

---

Q4: Typos

A4: Thanks for the suggestion. We will fix "TThese" and double check and revise all typos in the revision.

---

Finally, we genuinely hope that our explanations and efforts can improve the overall evaluation of our work. We thank the reviewer again for the helpful comments.

Thanks for the valuable feedback. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, we would really appreciate it if the reviewer considers raising the score. Please see our responses below one by one：

---

Q1: Explanation of Figure 3

A1: We clarify that we already provided detailed explanations in Section 3.2 [Lines 220-239] Decomposable Expert Layer Analysis. Figure 3 shows our decomposition strategy results across different compression ratios, visualising which expert layers are selected for decomposition at each ratio is the best strategy based on formulation 4, revealing how compression targets different layers as the ratio increases. We will revise the figure caption and add clarifications in Section 3.3 in the revision.

Q2: About a_i in Equation 4

A2: As detailed in Section 3.2 [Line 207-211], Appendix D.1 [Line 850-859] and pyhon pseudo-code Algorithm 1 [Line 980-985], the $a_i$ signifies the activation outliers for the $i$-th expert, defined as the outlier ratio within the expert's weight matrix. Outliers are identified as values exceeding a predefined threshold, which is set as a multiple of the mean value of the matrix. Mathematically, the outlier ratio is calculated as:

$$a_i = \frac{\sum_{a \in A_i} \mathbb{I}(|a| > \tau \cdot \operatorname{Mean}(|A_i|))}{|A_i|}$$

where $|A_i|$ denotes the total number of activations for the $i$-th expert, $\operatorname{Mean}(|A_i|)$ is the mean absolute value of these activations, and $\tau$ is a user-defined threshold. This metric highlights the presence of outlier activations indicative of the expert's contribution to the model's capacity.

Algorithm: Pyhon pseudo-code for $a_i$.

    # Compute activation outliers (a_i)
    a_i = torch.zeros(num_experts, device=device)
    for i, activation in enumerate(activations):
        mean_abs_act = torch.mean(torch.abs(activation))
        outliers = torch.sum(torch.abs(activation) > tau * mean_abs_act)
        a_i[i] = outliers / activation.numel()
    
    # Compute final sensitivity metric S_L
    S_L = torch.sum(f_i * r_i * a_i)
    

Q3: About Performance Improvements

A3:

(1) As shown in Table 2. At 20% compression ratio, MoE-SVD achieves an average accuracy of 0.58 across reasoning tasks compared to 0.51 for ASVD, 0.44 for SVD-LLM and 0.33 for MC-SMoE. This represents a 23% reduction in performance degradation relative to the original model (0.63). At higher compression ratios (50-60%), where other methods experience catastrophic collapse (e.g., ASVD and SVD-LLM reach perplexities >10,000 on WikiText-2), MoE-SVD maintains reasonable performance with a perplexity of 13.52 on WikiText-2.

(2) The real-world utility is validated by significant efficiency gains: 1.5-1.8× inference speedup on Mixtral-8×7B at 60% compression making deployment feasible on resource-constrained devices.

(3) Ablation studies in Tables 4-6 confirm that each component of our approach contributes meaningfully to its success, with selective decomposition, V-matrix sharing, and U-matrix trimming all playing crucial roles in balancing efficiency and performance.

Q4: Generalizability of the Shared V-Matrix

A4:

The shared V-matrix is not universal but dynamically determined for each model architecture. To demonstrate this, we conducted comprehensive layer-wise CKA similarity analysis across different MoE architectures:

Layer-wise Analysis Within Models: this table shows the layer-wise CKA similarity patterns for U and V matrices within three diverse MoE architectures:

These findings provide strong empirical evidence that our approach of dynamically selecting V-matrices based on layer-specific router statistics is well-justified.

---

Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We genuinely hope that our explanations and efforts can improve the overall evaluation of our work.