Olica: Efficient Structured Pruning of Large Language Models without Retraining

We really appreciate the time and efforts you extended in reviewing our paper. Below please find our responses regarding your concerns.

Q1: SVD and AWSVD treat each of $W_v$ and $W_o$ independently, which might provide more accurate estimates than treating the product of the two matrices, $W_vW_o^T$.

A1: Thanks for your question. The reconstruct loss $|| XW_{vo} - X\hat{W_{vo} }||$ relies primarily on $W_{vo}=W_vW_o^{\top}$, rather than each $W_v$ and $W_o$ individually. Although $W_v$ and $W_o$ can be individually estimated accurately by SVD and AWSVD, the reconstruction loss of the product $W_vW_o^T$ can not be guaranteed, as it may incur the error accumulation issue introduced by the product of separately estimated $\hat{W_v}$ and $\hat{W_o}$.

Q2: Is the gate projection pruned in the same manner as the up projection? What are the differences compared to LoRAP?

A2: Yes, the gate projection of the FFN layer is pruned in the same manner as the up projection and this is coincide with LoRAP. Our purpose here is to design the linear calibration, i.e., approximating the residual errors $E=f(X)-\hat{f}(X) \approx X\hat{W}$ by a linear model, such that $f(X)\approx \hat{f}(X)+ X\hat{W}$, no matter how $f$ is pruned to $\hat{f}$. In principle, any efficient pruning methods for $f$ can be adopted here.

Q3: Could you provide additional experimental results on higher pruning ratio?

Table 1: Results of larger models with 50% sparsity ratios.

A3: We have demonstrated the results of 50% sparsity ratio in the Appendix D of our paper, including 7B and 13B models. Here, we further scale the model parameters up to 30B and 70B. From Table 1, we can observe that our proposed method still works better than LoRAP under higher pruning ratio and larger models. As for the fine-tuned results of these pruned larger models, we leave to future works due to the resource constraints.

Q4: Results of other models (e.g., OPT, Qwen, and Phi) and MoE-based architectures.

Table 2: Results of 20% sparsity ratio.

A4: We further extended the evaluation results to LLMs Phi-2 (A), Qwen2.5-14B (B), and Mixtral-8x7B-v0.1 (C), where Mixtral-8x7B-v0.1 is a MoE-based model. As shown in Table 2, we see that our approach consistently outperforms LoRAP, showing high applicability and superiority.

Q5: What are the details of calculating the latency? Can this method achieve speedups with a batch size of 1 for both prefilling and decoding stages?

A5: Indeed, following the evaluation protocols of works LLM-Pruner and LoRAP, we mainly tested the speedup of the prefilling state, where the vanilla HuggingFace is used and the batch size is set to 2. As for the decoding stage, please see Table 3 in the Q4 raised by Reviewer 9AzP, we can see that under 50% sparsity, the throughput in tokens of pruned model can enhance about 30%.

Q6: Could you explain why the proposed calibration method is considered superior to A, B, and C?

A6: Both A and B propose to adjust the combination of the remaining filters to minimize the reconstruction error. However, if the pruned filters contain information not carried by any remaining filters, this adjustment is insufficient. Our proposed approach directly models the residual errors, complementing the remaining filters by introducing new information they do not capture. As for C, it uses a constant baseline to compensate for the pruned filters, which may reduce the accuracy. As shown in Table 3 presented in A7, FLAP performs significantly worse than the proposed Olica.

Q7: Comparisons with Shortened LLaMA, FLAP, and SLEB (LLaMA-7B).

Table 3: Comparison results.

A7: From Table 3, we see that our proposed Olica consistently outperforms these baselines. We do not compare with methods Sheared LLaMA and Minitron due to the unfairness, as they require huge computation and data resources to retrain the pruned model. For example, to retrain a 2.7B model, Sheared LLaMA requires 50B tokens and 16 GPUs. In contrast, our proposed method is highly efficient, enabling pruning models with more than 70B parameters on a single NVIDIA GeForce RTX 4090 GPU with less than an hour runtime.

We greatly thank you for the detailed reviews and helpful suggestions. We reply point-by-point here.

Q1: The claim of "no retraining" is misleading.

A1: Thanks for your question. In deep learning, "training'' or "fine-tuning'' generally require a series of forward and backward processes to compute the gradients so as to update the parameters of neural networks. However, Our Olica only takes one forward process and does not involve the backward process. In this sense, our method does not require retraining. In the literature [1, 2], they also refer this paradigm as no retraining.

[1] Sparsegpt: Massive language models can be accurately pruned in one-shot. ICML, 2023.

[2] A simple and effective pruning approach for large language models. ICLR, 2024.

Q2: Comparions to one-shot magnitude-based or Taylor expansion-based structured pruning.

A2: Sorry for the confusion. We clarify that the baselines LLM-Pruner and LoRAP are one-shot Taylor expansion-based and magnitude-based methods, respectively. Their results have been included in Table 2 and Table 3 of our paper.

Q3: The performance of 30B+ model and higher pruning ratio are unknown.

Table 1: Results of larger models and higher sparsity ratios (SR), where A=LLaMA-30B, B=LLaMA-2-70B.

A3: We further conducted experiments on models LLaMA-30B and LLaMA-2-70B. As shown in Table 1, we see that our proposed Olica consistently achieves better performance with different pruning raios and model scales, which clearly demonstrates the scalability of our porposed Olica.

Q4: No evaluation on challenging datasets. Throughput in tokens per second is not benchmarked.

Table 2: Results (accuracy) on MMLU of Llama-2-13b-chat (20% sparsity).

Table 3: Throughput in tokens per second of pruned LLaMA-30B tested on 3 RTX 4090 GPUs using vanilla HuggingFace.

A4: We exend the evaluation results to Massive Multi task Language Understanding (MMLU) task, which is a quiz bank covering 57 subjects, presenting a greater challenge compared to the Commonsense Reasoning datasets. From Table 2, we can still obtain better performance. We further tested the throughput in tokens of pruned LLaMA-30B. As shown in Table 3, under 50% sparsity, the throughput in tokens of pruned model can enhance about 30%.

Q5: Several key ideas, e.g., how ridge regression is applied, are underexplained.

A5: As for the ridge regression, since our target is to estimate the the pruning errors by a linear model: $E=f(X)-\hat{f}(X) \approx X\hat{W}$, where $\hat{f}$ is the pruned version of $f$ and $\hat{W}$ is the parameter matrix to estimate, a natural solution is the least-square estimation: $\hat{W}=(X^{\top}X)^{-1}E$. However, in the modern LLMs, the input matrix $X$ is extremely high-dimensional, leading to the irreversibility of $X^{\top}X$. To solve the problem, following the ridge regression, we add a $l_2$ penalty to the regression loss: $||E - XW|| + \lambda || W ||^{2}_{F}$. This leads a closed-form solution: $\hat{W}=(X^{\top}X+\lambda I)^{-1}E$, where $I$ is an identity matrix.

Q6: Confidence intervals or significance tests should be provided.

Table 4: t-test of comparison experiments of our paper.

A6: To conduct significance tests, we set the null hypothesis as "the baseline methods and the proposed Olica have the same accuracy $\mu$". We conduct five random experiments by independently selecting calibration datasets, over which we record the mean accuracy, denoted as $\hat{\mu}$. The significance tests of p-values are reported in Table 4, and we can see that all the null hypotheses are rejected when p-value $<$ 0.01.

Q7: Some recent works are missing. For example, [1] Search for Efficient Large Language Models.

A7: We directly cite the results from [1]. Under the 20% sparsity, our average accuracies are: 61.10% (LLaMA-7B) and 63.73% (LLaMA-13B), whereas [1] are 59.71% (LLaMA-7B) and 62.10% (LLaMA-13B).

Thank you for providing the insightful comments. We will try our best to address your concerns as follows.

Q1: The key observation of this article is that MHA depends on $W_{q_i}W_{k_i}^{\top}$, which is not true on llama, because of RoPE. However, all experiments in this article are done on llama. This means that all the discussion in Section 3.2 doesn’t apply to llama. In Appendix A, authors claim that they apply PCA separately on $W_q$ and $W_k$. This part is quite unclear. And If PCA is separately applied on $W_q$ and $W_k$, what is the innovation of this compared with SVD-LLM [1], ASVD [2]?

A1: Despite the separate estimation of $W_q$ and $W_k$, it cannot diminish our contribution to the estimation of $W_v$ and $W_o$. Our approach is different from both SVD-LLM and ASVD in the following aspects. First, both SVD-LLM and ASVD treat the matrices $W_v$ and $W_o$ independently and propose different variants of SVD to reconstruct $W_v$ and $W_o$ individually. Considering that the reconstruction loss $||XW_{vo} - X\hat{W_{vo}}||$ relies on $W_{vo}=W_vW_o^{\top}$, rather than each $W_v$ and $W_o$ individually, the proposed Olica regards the product $W_{vo}=W_vW_o^{\top}$ as an unified entity. Although SVD-LLM and ASVD can gain more accurate estimations of $W_v$ and $W_o$, the reconstruction loss of the product $W_{vo}$ can not be guaranteed, because it may incur the error accumulation issue induced by the product of separately estimated $\hat{W_{v}}$ and $\hat{W_{o}}$. As evidenced by Table 5 of our paper, the proposed method can achieve significantly better performance than separate estimations of $W_v$ and $W_o$. Moreover, we compare the proposed Olica with SVD-LLM based on the same evaluation settings. We obtain the following performance on three datasets: ARC_easy (69%), PIQA (78%) and WinoG (70%) using Olica, whereas the performance of SVD-LLM(W) is: ARC_easy (62%), PIQA (71%), and WinoG (61%). These results clearly demonstrated the advantages of our proposed method.

[1] Wang, Xin, et al. "Svd-llm: Truncation-aware singular value decomposition for large language model compression." arXiv preprint arXiv:2403.07378 (2024).

[2] Yuan, Zhihang, et al. "Asvd: Activation-aware singular value decomposition for compressing large language models." arXiv preprint arXiv:2312.05821 (2023).

Q2: The linear calibration turns the skip connection into weighted low-rank layer, which is quite similar to SliceGPT. SliceGPT also converts the skip connection into a multiplication of two low-rank matrices. What is the contribution of this part?

A2: Thanks for your insightful comments. The proposed linear calibration (LC) is designed to calibrate the residual information of pruned layers by linear models, that is, $E=f(X)-\hat{f}(X)\approx X\hat{W}$, such that the pruned layer $\hat{f}$ can be calibrated to approximate its original version: $f(X) \approx \hat{f}(X) + X\hat{W}$. To further reduce the number of parameters of $\hat{W}$, we performed SVD on $\hat{W}$. The SliceGPT inserts an identity matrix $I=UU^{\top}$ into the intermediate layers of transformer so that to reduce its hidden dimension, where the orthogonal matrix $U$ is obtained by perform SVD on the feature matrix $X$. We emphasize that the SVD is a general technique and can be used for different purposes. Here, we use it for reducing the number of parameters of $\hat{W}$, whereas SliceGPT use it for reducing the feature dimension of $X$.

Q3: In paragraph of “Fast OND”, authors claim that Figure 2 shows that the low-rank structure of the product $W_v W_o^T$ can be determined by one of the $W_v$ and $W_o$. But Figure 2 only shows that the trend of singular value in $W_v$ and $W_o$ but doesn’t show that they have similar singular vectors. So the motivation of performing SVD on one of $W_v$ and $W_o$ seems not very solid.

A3: Thanks for your insightful comments. Since $rank(W_v W_o^T)< min(rank(W_v), rank(W_o))$, the low-rank property of $W_v W_o^T$ is upper bounded by the ranks of $W_v$ and $W_o$. If either $W_v$ or $W_o$ exhibits the low-rank structure, we immediately know that $W_v W_o^T$ is also a low-rank matrix. As a result, we only need to examine the distribution of the singular values of $W_v$ and $W_o$ to determine their low-rank structures. We will state these more clearly in the final manuscript.

Many thanks for your time and efforts in reviewing our paper. We will fully address your concerns below.

Q1: The product of $W_{q_i}W_{k_i}^{\top}$ has already shown the low-rank property since $d_h$ is smaller than $d$. Based on this, are you trying to reduce $d_h$ into an even smaller $r$ using PCA ?

A1: Thanks for your question. Indeed, our orthogonal neuron decomposition (OND) method is to reduce the dimensionality of each attention head, i.e, reduce $d_h$ into a smaller $r$ using PCA.

Q2: I am confused about Eq.7. Is $f(X)=XW$ in Eq.7? If so, this means that your target is to learn $\hat{f}(X)=0$, which is $\hat{W}=0$. If not, what are you trying to learn in Eq.7? The authors should further explain this.

A2: Sorry for the confusion. In Eq.7, the $f(X)$ is denoted as the original FFN layer. Our objective is to recover the residual errors $E$ of pruned layers $\hat{f}$, that is, $E=f(X)-\hat{f}(X)$, where $f$ is the original version of $\hat{f}$. We approximate $E$ by a linear model: $E\approx X\hat{W}$, and thus the pruned layer $\hat{f}$ can be calibrated to approximate its original version: $f(X) \approx \hat{f}(X) + X\hat{W}$. The $\hat{f}$ is obtained by the algorithm detailed in the FFN Pruning section of our paper and both $f(X)$ and $\hat{f}(X)$ are fixed in Eq.7. So the objective of Eq.7 is to learn the parameters $\hat{W}$ of the liner model given both $f$ and $\hat{f}$ fixed.

Q3: There is a lack of some ablation studies. For example, adjusting the pruning ratio of FFN layers and MHA layers.

Table1: Ablation study of different pruning ratios of FFN layers and MHA layers (LLaMA-7B).

A3: Thanks for this valuable suggestion. We conducted ablation studies with different pruning ratios of FFN layers and MHA layers in the Table 1. We observe that the MHA layers can tolerate a larger pruning ratio than the FFN layers, meaning that the core design of Transformers, i.e., the multi-head attention layer, is particularly redundant. This observation coincides with existing works such as [1].

[1] What matters in transformers? not all attention is needed. https://arxiv.org/abs/2406.15786.

Q4: There is a lack of comparison to other SOTA methods, such as OSSCAR and FLAP.

Table2: Comparisons with FLAP (Accuracy ($\uparrow$)).

Table3: Comparisons with OSSCAR (PPL ($\downarrow$) on WikiText).

A4: We compare Olica with FLAP and OSSCAR in Table 2 and Table 3 in terms of accuracy and PPL, respectively. We can see that the proposed Olica consistently outperforms FLAP with large margins. Besides, our approach surpasses OSSCAR on the larger models, i.e., OPT-2.7B and OPT-6.7B.