Delta Decompression for MoE-based LLMs Compression

Dear Reviewer 4qJS

Thank you for your insightful comments and for acknowledging the strengths of D²-MoE in terms of efficiency, accuracy, and practical applicability. Below, we address your concerns in detail.

Q1: Motivation and Theoretical Justification of the D²-MoE Framework

A1:

As illustrated by the CKA similarity analysis in Figure 1, MoE models exhibit substantial similarities among different experts. Motivated by this observation, we propose merging experts into a shared representation. During this merging process, we utilize Fisher-weighted merging to emphasize critical expert weights effectively. Furthermore, to retain task-specific details and enhance model performance, we introduce a delta branch by applying Singular Value Decomposition (SVD) to the delta weights.

Q2: The dynamic reallocation of delta weights might introduce additional implementation complexity—can this be analyzed further?

A2:

(1) While delta weight reallocation adds computational overhead, Table 4 shows that D²-MoE achieves up to 1.5× inference speedup even with dynamic structured pruning.

(2) Figure 2 demonstrates that delta weights exhibit strong low-rank properties, allowing for efficient compression without excessive reallocation overhead.

(3) We plan to further refine caching or partial updates to reduce overhead, but for now the two-phase pruning strategy offers a favorable trade-off between complexity and improved compression.

Q3: The theoretical foundation of truncation-aware SVD should be elaborated on.

A3:

(1) As shown in Table 6, naively truncating singular values leads to performance degradation and model collapse. In contrast, truncation-aware SVD significantly outperforms both vanilla and activation-aware SVD, achieving a lower WikiText-2 perplexity (5.28) compared to standard SVD (6.22).

(2) Therefore, we leverage the activation Gram matrix to compute a scale matrix represented as $\text{Cholesky}(X \cdot X^T)$. By introducing this scale matrix, truncating fewer singular values allows us to preserve more essential information.

(3) Why we represent the scale matrix as $\text{Cholesky}(X \cdot X^T)$ can be proven as follows: When the scaling matrix $S$ is the Cholesky decomposition of $X \cdot X^T$, we have $S \cdot S^T = X \cdot X^T$. Under this condition, the compression loss $L_i$ caused by truncating singular values equals precisely to the singular value $sigma_i$ itself. Consequently, truncating the smallest singular values results in minimal compression loss, theoretically justifying our use of the scale matrix defined as $\text{Cholesky}(X \cdot X^T)$.

Q4: The discussion on combining D²-MoE with other compression techniques (e.g., quantization, knowledge distillation) should be expanded.

A4:

(1) We integrate quantization to further reduce memory footprints in the following Table, following approaches like GPTQ that already show how delta weights can be quantized effectively. Additionally, we apply the mixed-precision quantization method from MC-MoE to our D²-MoE.

Table D²-MoE+quantization

(2) Knowledge distillation can further enhance D²-MoE by transferring knowledge from the full model to its compressed counterpart, effectively preserving generalization capabilities. As demonstrated in the following table, applying advanced distillation methods indeed improves our approach's performance.

Table D²-MoE+KD

Reviewer V5e9

Thank you for your detailed review and for recognizing the novelty of D²-MoE and its strong empirical performance. Below, we address your concerns in depth.

Q1: The criteria for setting the SVD truncation threshold should be explained.

A1:

(1) We set an overall compression ratio (e.g., 40%), and to enhance performance, we prune 10% of the base weights. Under this setting, we can calculate the truncation threshold in D²-MoE accordingly. Detail can be seen in Table 7.

(2) We integrate activation statistics by computing a scaling matrix from the activation Gram matrix (see Section3.3). This ensures that truncating fewer singular values preserves more essential information. And the scale matrix can be represented as $\text{Cholesky}(X \cdot X^T)$. And Table 6 demonstrates that truncation-aware SVD significantly outperforms vanilla and activation-aware SVD, reducing WikiText-2 perplexity to 5.28 compared to 6.22 for standard SVD.

(3) Why we represent the scale matrix as $\text{Cholesky}(X \cdot X^T)$ can be proven as follows: When the scaling matrix $S$ is the Cholesky decomposition of $X \cdot X^T$, we have $S \cdot S^T = X \cdot X^T$. Under this condition, the compression loss $L_i$ caused by truncating singular values equals precisely to the singular value $sigma_i$ itself. Consequently, truncating the smallest singular values results in minimal compression loss, theoretically justifying our use of the scale matrix defined as $\text{Cholesky}(X \cdot X^T)$.

Q2: The theoretical justification for Fisher-weighted merging should be expanded—why is it better than simpler merging strategies?

A2:

(1) We draw inspiration from model merging methods, where typically a base model is merged with various task-specific vectors. However, standard model merging techniques are ineffective for MoE expert merging since there is no explicit base expert weight. In contrast, Fisher merging directly operates on expert weights, emphasizing parameters with higher gradient norms relative to the likelihood. This effectively identifies and retains the most critical expert weights.

(2) Our experiments in Table 5 confirm that Fisher-weighted merging consistently outperforms mean averaging and frequency-based merging. Fisher merging achieves a 5.28 perplexity on WikiText-2, compared to 7.66 for mean averaging and 6.42 for frequency-based merging.

Q3: Does D²-MoE generalize to non-MoE architectures? Could this delta compression approach be applied to dense transformers?

A3:

(1) We design D²-MoE specifically for multi-expert redundancy, but the principle of extracting a shared base plus compressed deltas is not limited to specialized experts.

(2) We are now carrying out similar approach apply to large dense transformers by factoring out common subspaces from multiple layers and storing residual differences.

(3) We leave systematically exploring dense variants for future work, as the our D²-MoE currently focuses on MoE layers.

Dear Reviewer LEQk

Q1: Careful discussion on relation with LoRA

A1:

(1) Our framework structurally builds a multi-LoRA setup for MoE compression, consisting of a single base branch and multiple delta low-rank branches, enabling us to leverage existing LoRA research for further fine-tuning and efficient multi-LoRA inference.

(2) Unlike standard LoRA methods, our approach is a post-training compression framework that does not require additional training.

(3) Compared with similar methods such as [LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation(ICML 2023)] and [Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (MC-sMoE, ICLR 2024 Spotlight)], which also apply pruning and SVD but involve further training and both come from top-tier machine learning conferences, our method significantly outperforms these approaches in performance and requires no training, as illustrated in the table.

Table D²-MoE vs LoSparse(ICML23), MC-sMoE(ICLR24 Spotlight)

(4) Additionally, further fine-tuning and knowledge distill to effectively recover performance, as shown in the table

Table D²-MoE+further training

Q2: Discussion on MoE quantization methods.

A2:

(1) Quantization and D²-MoE are orthogonal approaches: Quantization aims to reduce the model's precision, primarily speeding up inference and reducing memory usage rather than parameter count, while our D²-MoE targets parameter reduction explicitly through delta decomposition, it does acknowledge quantization as a complementary approach in the related work section, mentioning techniques like BitDelta that successfully quantize delta weights.

(2) Existing quantization methods can be easily integrated into the D²-MoE framework.

Table D²-MoE+quantization

Q3: The scalability of D²-MoE to larger models

A3:

(1) Table 2 show results on DeepSeekMoE-16B-Base at up to 60% compression. While not the exact DeepSeek V3 variant, these 16B-parameter experiments reveal consistent advantages, indicating D²-MoE scales to large MoE.

(2) Due to limited experimental resources, we plan to integrate advanced DeepSeek V3 configurations in future work. To demonstrate the scalability of D²-MoE to larger models, we conduct experiments on Mixtral-8x22B. Experimental results are summarized in the table below:

Table D²-MoE on larger scalability

Dear Reviewer 73q5

Thank you for your thoughtful review and constructive feedback. We appreciate your recognition of the novelty of D²-MoE and its strong empirical performance. Below, we address your concerns in detail.

Q1: The relationship between compression ratio and performance degradation is not analyzed in detail.

A1:

(1) Table 2 demonstrates D²-MoE's robustness at various compression ratios. For example, on Mixtral-8×7B, performance declines gradually with increased compression (from 20% to 60%) without model collapse. Specifically, at 40% compression, D²-MoE achieves an average accuracy of 0.57, significantly outperforming NAEE (0.48) and MoE-I² (0.49). Even at 60% compression, D²-MoE maintains 0.52 accuracy, whereas NAEE drops sharply to 0.36.

(2) Figure 4 shows how delta weight trimming affects D²-MoE's performance. As trimming increases, perplexity gradually rises, yet our method maintains stable performance even at extreme compression ratios of 81%. Specifically, at 43% compression (trimming 1 delta weight), D²-MoE achieves a WikiText-2 perplexity of 6.43. Remarkably, even under extreme 81% compression, the model does not collapse, demonstrating the robustness and effective trade-off between compression ratio and performance.

Q2: The interaction of D²-MoE with other compression techniques, particularly quantization, is not explored.

A2:

(1) Our current approach combines SVD decomposition and pruning, orthogonal to quantization. We focus on decomposing experts into Fisher-weighted bases and SVD-compressed delta weights without formally integrating quantization yet. However, our design allows straightforward quantization of low-rank delta factors (Section “Delta compression in MoE LLMs”).

(2) Existing quantization methods can be easily integrated into the D²-MoE framework: We plan to integrate quantization techniques to further reduce memory footprint, as demonstrated in the following table. Approaches such as GPTQ have shown effective quantization of delta and base-merged weights. Additionally, we apply the mixed-precision quantization method from MC-MoE to our D²-MoE.

Table D²-MoE+quantization

Q3: The introduction should better separate the motivation from the proposed solution.

A3:

(1) Problems with existing approaches: We will revise the introduction to juxtapose the storage/memory challenges of MoE (motivation) with our delta decomposition approach (solution), referencing new Table 1 (Section “Related Work”) to highlight how D²-MoE differs from pure pruning or merging.

(2) D²-MoE's motivation: We emphasize that the moderate overlap (CKA 0.3–0.5) between experts to extract a shared base weight while preserving expert-specific diversity in compressed delta form.

Q4: Figures (e.g., CKA similarity, singular value energy retention) need more detailed captions explaining their implications.

A4:

(1) We will expand the captions of Figures 3 and 4 to explicitly describe what each metric represents.

(2) For Figure 3 (CKA similarity), we will clarify that lower similarity values indicate that merging all experts directly would lead to performance loss, justifying the need for delta decomposition.

(3) For Figure 4 (singular value energy retention), we will highlight that delta weights exhibit strong low-rank properties, making them well-suited for SVD-based compression.

Q5: Minor typos need correction.

A5:

(1) We will correct all typos, including those in page 1 ("successful compact" → "successfully compacts") and page 3 ("experts delta decomposition" → "expert delta decomposition").

(2) We will conduct a thorough proofreading pass to ensure consistency and clarity throughout the paper.