Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition

Table: More comparisons with other structured compression methods under around 20% compression ratio.

(3) In addition to accuracy preservation, our superiority in inference acceleration also deserves to be considered for efficiency performance across many mobile or speed-demanding scenarios. Moreover, our MoE-SVD are hardware-independent and successfully compact LLM to high compression ratios without additional training.

Q2: Comparison with fine-tuning compression methods like MC-SMoE [1].

A2: (1) As shown in Table below, our approach differs from MC-SMoE [1] in comparison methods, algorithmic pipeline, decomposition and training approach, evaluating Models: Positioned as a model merge technique, MC-SMoE [1] only addresses small-scale MoE by first merging experts and then applying vanilla decomposition before fine-tuning. In contrast, our MoE-SVD designs customized strategies like selective decomposition for large-scale MoEs without expert merging and fine-tuning.

Table: Comparisons with MC-SMoE [1].

(1) As shown in Table below, our method MoE-SVD demonstrates superior speed improvements compared to MC-SMoE, achieving a 1.2x speedup and significantly higher performance across various benchmark datasets, including WikiText-2, PTB, and C4. These performance gains highlight the enhanced efficiency and scalability of MoE-SVD in processing large-scale MoE tasks, contributing to improved model performance and throughput.

Table: More comparisons with MC-SMoE [1] under around 20% compression ratio.

Dear Reviewer nYc5,

Thanks for the valuable feedback. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, we would really appreciate it if the reviewer considers raising the score. Please see our responses below one by one：

Q1: About accuracy degradation.

A1: (1) Our MoE-SVD performance can continue improving with more calibration samples. Our original setting of calibration sample size is 64 the same as that of SVD-LLM for general LLM. During rebuttal, we observed that existing MoE compression methods like MC-SMoE [4] employ larger calibration sample sizes. We have aligned the calibration samples (512) accordingly, and the updated results for MoE-SVD (512) are presented in the following Table. The results indicate that our MoE-SVD method, without fine-tuning, exhibits only ~3% reduction in performance at a 20% compression ratio respectively on Phi3.5 MoE and Mixtral 8x7B, which is clearly superior to other compression methods. Additionally, our MoE-SVD with LoRA fine-tuning achieves performance levels close to that of the original dense MoE model.

Table A: Performance of Mixtral-8×7B and Phi-3.5-MoE compressed by MoE-SVD with 512 calibration samples under 20% compression ratios.

(2) Model compression methods (including ours and compression methods) make tradeoffs of performance-efficiency and always introduces the performance degradation to reduce model size, which is common phenomenon in this field. Our benefits are that our MoE-SVD effectively reduces performance drops than other methods: noticeable performance-efficiency improvements compared to other structured compression methods (see Table below), significant performance gains (>100 lower perplexity and 5%~20% accuracy gains) than to other SVD methods (see Table 2).

(3) Analysis of trade-off between computation time and communication time:

Our SVD-MoE methodology introduces a sophisticated approach to matrix decomposition in expert systems, fundamentally transforming the computational architecture while preserving mathematical integrity. The core operation involves decomposing the expert weight matrix $\mathbf{W} \in \mathbb{R}^{H \times H}$ (with hidden dimension $H = 4096$) into constituent matrices $\mathbf{U} \in \mathbb{R}^{H \times r}$ and $\mathbf{V}^\top \in \mathbb{R}^{r \times H}$, where the reduced rank $r = 1024$ represents a quarter of the hidden dimension ($r = H/4$). For input activations $\mathbf{X} \in \mathbb{R}^{B \times H}$ with batch size $B = 128$, the computational pathway evolves from the direct multiplication $\mathbf{Y = XW}$ to a two-stage process $\mathbf{Y = XUV}^\top$, introducing an intermediate activation $\mathbf{Z = XU} \in \mathbb{R}^{B \times r}$.

The computational efficiency manifests in quantifiable metrics:

$\text{Computations}_{\text{before}} = B \times H^2 = 128 \times 4096^2 = 2.15 \times 10^{11}$

$\text{Computations}_{\text{after}} = 2 \times B \times H \times r = 2 \times 128 \times 4096 \times 1024 = 1.07 \times 10^{11}$

Which achieving a 50% reduction.

This efficiency gain, however, introduces additional complexity in tensor parallelism operations. The communication volume expands from $B \times H = 128 \times 4096 = 524,288$ elements to $B \times (r + H) = 128 \times (1024 + 4096) = 655,360$ elements, representing a 25% increase.

Performance metrics on A100 GPUs (600 GB/s NVLink bandwidth, FP32 precision):

Communication time, calculated as $\text{Communication Time} = \frac{\text{Total Data (bytes)}}{\text{Bandwidth (bytes/s)}}$, increases from $3.5 \times 10^{-6}$ seconds to $4.37 \times 10^{-6}$ seconds.

Computation-Communication Tradeoff:

Computation time, based on 19.5 TFLOPS peak performance, decreases substantially from $\frac{2.15 \times 10^{11}}{19.5 \times 10^{12}} \approx 11 \times 10^{-3}$ seconds to $\frac{1.07 \times 10^{11}}{19.5 \times 10^{12}} \approx 5.5 \times 10^{-3}$ seconds. The aggregate processing time improves from $11.0035 \times 10^{-3}$ seconds to $5.50437 \times 10^{-3}$ seconds, yielding a net optimization of $5.49913 \times 10^{-3}$ seconds.

The marginal increase in communication overhead ($\Delta \text{Communication Time} = 0.87 \times 10^{-6}$ seconds) proves negligible against the substantial computational savings ($\Delta \text{Computation Time} = 5.5 \times 10^{-3}$ seconds).

This favorable ratio demonstrates that the SVD-MoE architecture successfully navigates the trade-off between computational efficiency and communication overhead. The high-bandwidth interconnect infrastructure effectively mitigates the increased communication volume, ensuring that the additional synchronization requirements do not compromise the overall performance gains achieved through matrix decomposition.

(4) We had considered adding experiments. However, we were unable to perform experiments within the rebuttal period. PyTorch's tensor parallelism currently supports only official models. Adapting it to non-official models like ours would require modifications to the underlying library, which was not feasible in the given timeframe. Nonetheless, our analysis strongly indicates that the benefits of reduced computation outweigh the additional communication costs. We will explore this in future work.

Finally, we genuinely hope that our explanations and efforts can improve the overall evaluation of our work. We are glad to discuss further comments and suggestions.

(2) Following the suggestion, we involve extra experiment on Qwen2-57B-A14B as in following Table. The results demonstrate consistent performance and speed improvements across these diverse tasks, suggesting our method's generalizability. Thanks for the suggestion, we already added this experiment in the revision [Line 857-863].

Table: Performance of Qwen2-57B-A14B compressed by MoE-SVD under 20% compression ratios.

Q5: Tensor parallelism and potential communication overhead.

A5: (1) Our MoE-SVD can reduce the computation time because of the reduced size of the weight matrix via decomposition. As shown in Table below, our method can get 1.2x~1.5x speedup in inference. These speedup results already outperform other structured compression at algorithmic level, which confirms the effectiveness of our method.

Table: Metrics (model size, TFLOPs, runtime) of MoE-SVD. Runtime denotes runtime throughput (Tokens/sec).

(2) Combining with particular hardware system optimization methods like tensor parallelism, all the low-rank decomposition approaches (including our methods. MC-SMoE [1], ASVD and SVD-LLM, etc) will introduce communication overheads. This is a common phenomenon for this kind of techniques in specific scenarios like tensor parallelism. But we highlight that the total overhead optimization represents the trade-off between computation time and communication time, which also depends on the individual hardware device.

Dear Reviewer eFn1,

Thanks for constructive comments. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, we would appreciate it if the reviewer considers raising the score. Please see our responses below one by one：

Q1: About "Hardware Dependency" for semi-structured sparse methods.

A1: Up to now, only NVIDIA Ampere and NVIDIA Hopper architecture GPUs include new sparse Tensor Cores that can support semi-structured sparse methods [1]. Commonly used NVIDIA Volta (e.g., V100), Turing (e.g., RTX 2080 Ti), Pascal, Maxwell, and Kepler series architecture GPUs do not support semi-structured sparse methods. Thanks for the suggestion, we have added this in the revision [Line 53].

[1] NVIDIA Corporation. (2020). NVIDIA Ampere Architecture In-Depth. NVIDIA Developer Blog.

Q2 & Q5: About variables in Equation 4 and more clarify on Section 3.2.

A2: (1) The variable $r_i$ denotes the principal rank of the $i$-th expert, which is the number of dominant singular values in the diagonal matrix $\Sigma_i$ obtained from the SVD of the expert's weight matrix $W_i$. The principal rank $r_i$ is our self-defined as the number of singular values in $\Sigma_i$ that exceed a given threshold, effectively capturing the dimensionality of the weight matrix's significant components. This rank reflects the structural complexity of the expert's weight representation, with higher values of $r_i$ indicating more complex and information-rich weights. Note that $r_i$ does not denote vanilla rank of the matrix (number of non-zero singular values).

(2) The $f_i$ represents the sampling frequency of the $i$-th expert, quantifying the frequency with which this expert is selected by the router during inference. This metric also reflects the number of selections made by the expert within the calibration dataset. It is computed based on the routing decisions of the gating network $G(x)$ across a representative dataset $\mathcal{X}$ :

where $\mathbb{I}[\cdot]$ is the indicator function, and $k$ is the number of experts selected by the top-$k$ gating mechanism.

(3) The $a_i$ signifies the activation outliers for the $i$-th expert, defined as the outlier ratio within the expert's weight matrix. Outliers are identified as values exceeding a predefined threshold, which is set as a multiple of the mean value of the matrix. Mathematically, the outlier ratio is calculated as:

where $|A_i|$ denotes the total number of activations for the $i$-th expert, $\operatorname{Mean}(|A_i|)$ is the mean absolute value of these activations, and $\tau$ is a user-defined threshold. This metric highlights the presence of outlier activations indicative of the expert's contribution to the model's capacity.

(4) Ablation studies of sensitivity metric. Our metric of $r_i$ and $a_i$ weighted by $f_i$ provides a comprehensive metric for identifying which expert layers are most suitable for decomposition. Results in Table 3 show that our metrics can obtain better performance (8.67 vs.16.57 ) than vanilla $a_i$ (Non-uniform SVD [OWL]) . Our additional experiments show that our metrics with $r_i$ and $f_i$ obtained 9.85 and 12.65, respectively, demonstrating the importance of these designs.

Thanks for the suggestion, we have clarified Section 3.2 [Line 256-233], added more details in Appendix D.2 [Line 941-965] and pyhon pseudo-code Algorithm 1 [Line 1080-1118] in the revision .

Q4: About  decomposition process of the expert matrix. 

A4: Yes. our decomposition process applies SVD individually to each fully connected layer within each expert, rather than concatenating the layers. Thanks for the suggestion, we have clarified this in Section 3.1 [line 196-202] and provided more explanations and pyhon pseudo-code Algorithm 2 [Line 1134-1174] in the revision.

Q5: About  calculation of the sensitivity score. 

A5: (1)The $a_i$ signifies the activation outliers for the $i$-th expert, defined as the outlier ratio within the expert's weight matrix. Outliers are identified as values exceeding a predefined threshold, which is set as a multiple of the mean value of the matrix. Mathematically, the outlier ratio is calculated as:

where $|A_i|$ denotes the total number of activations for the $i$-th expert, $\operatorname{Mean}(|A_i|)$ is the mean absolute value of these activations, and $\tau$ is a user-defined threshold. This metric highlights the presence of outlier activations indicative of the expert's contribution to the model's capacity.

(2) The $f_i$ represents the sampling frequency of the $i$-th expert, quantifying the frequency with which this expert is selected by the router during inference. This metric also reflects the number of selections made by the expert within the calibration dataset. It is computed based on the routing decisions of the gating network $G(x)$ across a representative dataset $\mathcal{X}$ :

where $\mathbb{I}[\cdot]$ is the indicator function, and $k$ is the number of experts selected by the top-$k$ gating mechanism.

(3) Generalizability. Our extra tests show that frequency values obtained from WikiText-2 dataset indeed can work well with other datasets (PTB and C4) with only ~0.1 perplexity variations. In addition, Our experiments test results in multiple datasets in Table 2 and ablation analysis in Table 5 also confirm the generalizability of our Moe-SVD.

(4) Ablation studies of sensitivity metric. Our metric of $r_i$ and $a_i$ weighted by $f_i$ provides a comprehensive metric for identifying which expert layers are most suitable for decomposition. Results in Table 3 show that our metrics can obtain better performance (8.67 vs.16.57 ) than vanilla $a_i$ (Non-uniform SVD [OWL]) . Our additional experiments show that our metrics with $r_i$ and $f_i$ obtained 9.85 and 12.65, respectively, demonstrating the importance of these designs.

Thanks for the suggestion, we have clarified Section 3.2 [Line 256-233], added more details in Appendix D.2 [Line 941-965] and pyhon pseudo-code Algorithm 1 [Line 1080-1118] in the revision .

Algorithm: Pyhon pseudo-code for sensitivity metric of MoE-SVD.

def compute_layer_sensitivity(experts_weights, activations, gating_outputs, calibration_data, top_k=2, tau=2.0):
    """
    Compute layer-wise sensitivity metric S_L for MoE compression
    
    Args:
        experts_weights (list of torch.Tensor): Weight matrices for each expert
        activations (list of torch.Tensor): Activation values for each expert
        gating_outputs (torch.Tensor): Router outputs for calibration data
        calibration_data (torch.Tensor): Calibration dataset
        top_k (int): Number of experts to select per token
        tau (float): Threshold for activation outliers
    
    Returns:
        float: Layer sensitivity score S_L
    """
    num_experts = len(experts_weights)
    device = experts_weights[0].device
    
    # Compute sampling frequency (f_i)
    top_k_indices = torch.topk(gating_outputs, top_k, dim=-1).indices
    expert_counts = torch.zeros(num_experts, device=device)
    for indices in top_k_indices:
        expert_counts[indices] += 1
    f_i = expert_counts / len(calibration_data)
    
    # Compute principal rank (r_i) using SVD
    r_i = torch.zeros(num_experts, device=device)
    for i, weight in enumerate(experts_weights):
        U, S, V = torch.linalg.svd(weight)
        # Count singular values above threshold
        threshold = torch.max(S) * 1e-2  # Threshold
        r_i[i] = torch.sum(S > threshold)
    
    # Compute activation outliers (a_i)
    a_i = torch.zeros(num_experts, device=device)
    for i, activation in enumerate(activations):
        mean_abs_act = torch.mean(torch.abs(activation))
        outliers = torch.sum(torch.abs(activation) > tau * mean_abs_act)
        a_i[i] = outliers / activation.numel()
    
    # Compute final sensitivity metric S_L
    S_L = torch.sum(f_i * r_i * a_i)
    
    return S_L

Dear Reviewer ghaz,

Thanks for constructive comments. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, we would appreciate it if the reviewer considers raising the score. Please see our responses below one by one：

Q1: About  comparisons with structured compression methods [1]~[6].

A1: (1) Following the suggestions, we have conducted additional experiments to compare our MoE-SVD method with these structured compression methods in Table below. Note that we were unable to include STUN [5] in our comparisons, as it was recently published in September on arXiv without available code. We already involve this comparisons [Line 833-845] in the revision.

Table: More comparisons with other structured compression methods under around 20% compression ratio.

(2) The result shows that our MoE-SVD has significant performance gains over LoSparse [3] and MC-SMoE [4] in both speedup and performance metrics. This improvement stems from our customized strategies for MoE compression, such as selective decomposition, which extend beyond the straightforward low-rank decomposition utilized by LoSparse and MC-SMoE.

(3) Compared to prunning methods in structured settings in Wanda [1] and SparseGPT [2], our approach achieves significant speedups and consistent performance gains.

(4) We have discussed the drawbacks of these pruning methods, like Wanda and and SparseGPT, in original manuscript [Line 46-69], noting their dependence on specific hardware and their limited speedup ratios. Our MoE-SVD designs customized strategies like selective decomposition for large-scale MoEs without expert merging and fine-tuning, in contrast to MC-SMoE [4] , which only addresses small-scale MoE by first merging experts and then applying vanilla decomposition before fine-tuning (see addition discussion in the revision [Line 156-158]).

Q2: About metrics (model size, TFLOPS, runtime). 

A2: (1) We clarify that we already include some metrics in the original manuscript like runtime throughput (tokens/sec) in Figure 5 and Memory usage (GB) in Figure 6. We also present analysis about these metrics in the Experiment Section [Line 430-466]. For example, our method obtains 1.53 × acceleration in runtime throughput.

(2) Following the suggestions to provide a comprehensive assessment, we organize key metrics in the following Table, which has been added to the revision [Line 900-915]. These results specifically details runtime metrics across various compression ratios, with explicit measurement of speedups ranging from 1.2× to 1.5×.

Table: Metrics (model size, TFLOPs, runtime) of MoE-SVD. Runtime denotes runtime throughput (Tokens/sec) on a single H800 GPU.

Table A: Performance of MoE-SVD with 512 calibration samples under 20% compression ratios.

Table B: More comparisons with other structured compression methods under around 20% compression.

Table C: The centered kernel alignment (CKA) similarity before/after decomposition for decomposed layer [3,5,6,7,9,12,23,24,25], in Mixtral-8×7B on WikiText-2.

Table D: Performance of Qwen2-57B-A14B compressed by MoE-SVD under 20% compression ratios.

Algorithm: Pyhon pseudo-code for sensitivity metric of MoE-SVD.

def compute_layer_sensitivity(experts_weights, activations, gating_outputs, calibration_data, top_k=2, tau=2.0):
    """
    Compute layer-wise sensitivity metric S_L for MoE compression
    
    Args:
        experts_weights (list of torch.Tensor): Weight matrices for each expert
        activations (list of torch.Tensor): Activation values for each expert
        gating_outputs (torch.Tensor): Router outputs for calibration data
        calibration_data (torch.Tensor): Calibration dataset
        top_k (int): Number of experts to select per token
        tau (float): Threshold for activation outliers
    
    Returns:
        float: Layer sensitivity score S_L
    """
    num_experts = len(experts_weights)
    device = experts_weights[0].device
    
    # Compute sampling frequency (f_i)
    top_k_indices = torch.topk(gating_outputs, top_k, dim=-1).indices
    expert_counts = torch.zeros(num_experts, device=device)
    for indices in top_k_indices:
        expert_counts[indices] += 1
    f_i = expert_counts / len(calibration_data)
    
    # Compute principal rank (r_i) using SVD
    r_i = torch.zeros(num_experts, device=device)
    for i, weight in enumerate(experts_weights):
        U, S, V = torch.linalg.svd(weight)
        # Count singular values above threshold
        threshold = torch.max(S) * 1e-2  # Threshold
        r_i[i] = torch.sum(S > threshold)
    
    # Compute activation outliers (a_i)
    a_i = torch.zeros(num_experts, device=device)
    for i, activation in enumerate(activations):
        mean_abs_act = torch.mean(torch.abs(activation))
        outliers = torch.sum(torch.abs(activation) > tau * mean_abs_act)
        a_i[i] = outliers / activation.numel()
    
    # Compute final sensitivity metric S_L
    S_L = torch.sum(f_i * r_i * a_i)
    
    return S_L

Algorithm: Pyhon pseudo-code for SVD Expert Decomposition of MoE-SVD.

class MoESVDCompression:
    def __init__(self, truncate_k=None, top_k_experts=2):
        """
        Initialize Activation-Weighted SVD with Matrix Sharing and Trimming
        
        Args:
            truncate_k (int): Number of singular values to keep
            top_k_experts (int): Number of top experts to select for U-matrix trimming
        """
        self.truncate_k = truncate_k
        self.top_k_experts = top_k_experts
        
    def compute_activation_weights(self, X):
        """
        Compute activation-weighted scaling matrix S using Cholesky decomposition
        
        Args:
            X (torch.Tensor): Activation matrix [batch_size,feature_dim]
            torch.mm(X, X.t()) is Cumulative activation matrix, representing the sum of processed activation data.
        """
        # Compute Gram matrix
        gram = torch.mm(X, X.t())
        
        # Cholesky decomposition
        S = torch.linalg.cholesky(gram)
        return S
    
    def decompose_expert(self, W_original, X):
        """
        Perform activation-weighted SVD on single expert
        
        Args:
            W_original (torch.Tensor): Original weight matrix
            X (torch.Tensor): Activation matrix
        """
        # Compute activation-weighted matrix
        S = self.compute_activation_weights(X)
        W_aw = torch.mm(W_original, S)
        
        # Perform SVD
        U, sigma, V = torch.linalg.svd(W_aw, full_matrices=False)
        
        # Truncate if specified
        if self.truncate_k is not None:
            U = U[:, :self.truncate_k]
            sigma = sigma[:self.truncate_k]
            V = V[:self.truncate_k, :]
            
        return U, torch.diag(sigma), V, S

Algorithm: Pyhon pseudo-code for Matrix Sharing & Trimming of MoE-SVD.

class MoESVDCompression:
    def __init__(self, truncate_k=None, top_k_experts=2):
        """
        Initialize Activation-Weighted SVD with Matrix Sharing and Trimming
        
        Args:
            truncate_k (int): Number of singular values to keep
            top_k_experts (int): Number of top experts to select for U-matrix trimming
        """
        self.truncate_k = truncate_k
        self.top_k_experts = top_k_experts
    def __init__(self, truncate_k=None, top_k_experts=2):
        """
        Initialize Activation-Weighted SVD with Matrix Sharing and Trimming
        
        Args:
            truncate_k (int): Number of singular values to keep
            top_k_experts (int): Number of top experts to select for U-matrix trimming
        """
        self.truncate_k = truncate_k
        self.top_k_experts = top_k_experts
        
    def compress_moe(self, expert_weights, activations, routing_frequencies):
        """
        Compress MoE using V-matrix sharing and U-matrix trimming
        
        Args:
            expert_weights (list): List of expert weight matrices
            activations (list): List of activation matrices for each expert
            routing_frequencies (torch.Tensor): Expert selection frequencies
        """
        num_experts = len(expert_weights)
        compressed_experts = []
        
        # Decompose all experts
        decomposed = []
        for i in range(num_experts):
            U, Sigma, V, S = self.decompose_expert(expert_weights[i], activations[i])
            decomposed.append((U, Sigma, V, S))
            
        # Select shared V-matrix based on highest routing frequency
        max_freq_idx = torch.argmax(routing_frequencies)
        V_shared = decomposed[max_freq_idx][2]
        
        # Sort experts by routing frequency for U-matrix trimming
        sorted_indices = torch.argsort(routing_frequencies, descending=True)
        
        # Perform U-matrix trimming and construct compressed experts
        for i in range(num_experts):
            # Find top-k U-matrices from more frequently used experts
            more_frequent = [j for j in sorted_indices if routing_frequencies[j] > routing_frequencies[i]]
            top_k_indices = more_frequent[:self.top_k_experts]
            
            if len(top_k_indices) < self.top_k_experts:
                # If not enough more frequent experts, use own U-matrix
                top_k_indices = top_k_indices + [i]
            
            # Combine selected U-matrices and corresponding Sigma matrices
            U_combined = torch.zeros_like(decomposed[i][0])
            Sigma_combined = torch.zeros_like(decomposed[i][1])
            
            for idx, expert_idx in enumerate(top_k_indices[:self.top_k_experts]):
                U_combined += decomposed[expert_idx][0]
                Sigma_combined += decomposed[expert_idx][1]
            
            # Reconstruct compressed expert
            W_compressed = torch.mm(torch.mm(U_combined, Sigma_combined), 
                                 torch.mm(V_shared, torch.inverse(decomposed[i][3])))
            compressed_experts.append(W_compressed)
            
        return compressed_experts