SlimLLM: Accurate Structured Pruning for Large Language Models

Q1: Proposed metric for MHA

A1: We design similarity-based approach to enhance linear fitting by maximizing output similarity. To verify the effectiveness of our similarity-based approach, we employed the fluctuation metric from FLAP to evaluate the importance of heads, while aligning other configurations. From the results, It can be observed that our similarity-based strategy achieves a lower PPL.

Q2: Computations in Algorithm 1

A2: The computational cost of greedy search is acceptable. The time spent on greedy search is approximately 2 seconds for each layer on Llama-7B.

Q3: Importance metric for the FFN

A3: Yes, there is another reason. When pruning the input channels of the down projection, each channel's input activation only affects the magnitude of its output vector, while the direction is determined by the weights. For up and gate projection, their outputs correspond to the input activations of the down projection, and thus, we have not considered their impact on the direction of the final output matrix.

Q4: Linear regression

A4: A and B can be integrated into the output projection. Here, the weight matrix of the output projection is $A \cdot W_{o}$, and the bias term is $B + B_{o}$.

Q5: r_0​ and alpha

A5: Under the pruning ratio $r$ r_0 is calculated according to the following formula: $r_{0} = r *$ pruned_layers / total_layers

For alpha, we provide results for various values. Other models can use these results for adjustment.

Q6: Performance

A6: The experimental results show that the advantages of our model generally decrease after fine-tuning. This is mainly because the fine-tuning strategies of the two methods cannot be fully aligned. During compression, LORAP uses low-rank decomposition for MHA's linear layers, while LORA fine-tuning does not. LORAP reconstructs the decomposed matrices into a single matrix to keep MHA's parameter count unchanged. This increases the parameter count during LORA training.

Q7: Calibration samples

A7: We have added experiments to investigate the impact of the calibration set size on model performance.

Q8: Experiments

A8: To demonstrate the generalizability of our strategy, we have included experimental results on the Vicuna-7B.

Q9: 13B model

A9: Thank you for your suggestion. We have added experiments on LLaMA-13B.

Q10: Latency

A10: The framwork is Vanilla HuggingFace. To test decoding latency, we set the batch size to 8 and measured the latency for generating 256 tokens. The results are as follows:

Others: Thank you for comments on writing, we will revise it carefully.

Q1: This paper should be carefully proofread. For example, what is $O_{-h_{i}}$ in Figure 1? $S_{-p}$ is not easy to understand in Algorithm 1 and should be replaced by better equation format.

A1: Thank you for your suggestion. $O_{-h_{i}}$ denotes the output excluding the $i-th$ head. $S_{-p}$ represents the set of unpruned attention head. $S_{-p}$={$head_{i}$, $i \in$ unpruned head index}. The detailed descriptions will be added in the paper.

Q2: What is exactly the latency reported in Table 3? Prefill latency or Decoding latency? More details should be included.

A2: Thank you for your suggestion. We show the prefill latency in Table 3 and we will add details in paper. Additionally, we conducted experiments to measure the decoding latency. The table below shows the latency results for generating 256 tokens with a batch size of 8.

Q3: The proposed method is evaluated for LLMs. Is it possible that this method could also prune vision-language models?

A3: Yes, it is. For VLM (Visual-Language Model) models, the pruning strategy needs to be adapted, such as the proportion of visual tokens to text tokens. We are currently conducting preliminary experiments on Qwen-VL-7B, and the experimental results are shown in the table. We will present more details in our future work.

Q1: Computational complexity of iterative head pruning (Algorithm 1) is unaddressed, raising concerns for larger models.

A1: The computational complexity of Algorithm 1 is acceptable. First, the number of heads is generally small, which limits the number of iterations in the algorithm. Second, the outputs of each head can be precomputed and each iteration only involves the calculation of similarity, which reduces the number of redundant calculations during the iterations. For each layer, the time spent on greedy search is approximately 2 seconds when pruning Llama-7B.

Q2: Linear regression is applied only to output matrices. Is it possible for extending it to intermediate layers for further reducing accuracy loss.

A2: We compensate for the pruning error by performing linear fitting between the outputs before and after pruning. This requires that the outputs of the two stages maintain the same dimensionality. Currently, the output matrices of MHA and FFN meet this requirement. However, for other linear layers, since the output channels have been pruned, this method cannot be directly applied at present.

Q1: Although the proposed sub-methods have been proven effective, the relation between FFN pruning and attention layers pruning is no so clear. Are they independent methods?

A1: In this method, both the head pruning and channel pruning strategies are designed to increase the linear correlation between the outputs before and after pruning, which benefits our linear regression strategy. Meanwhile, considering the significant difference in the number of heads and channels, we employ the PCA method in the FFN to make the pruned output as close as possible to the principal direction. In the MHA, we directly maximize the Pearson similarity to bring the similarity closer.

Q2: The paper mainly focuses on LLaMA models. It is encouraged to conduct more experiments on other models.

A2: Thank you for your suggestion. We have added the experimental results on Vicuna-7B.

Q3: The performance gain appears to be not so significant at a 20% pruning ratio, as shown in Table 5. This suggests that SlimLLM might be more beneficial for higher compression levels but less impactful for moderate pruning. A deeper investigation into why performance varies across different pruning ratios would be valuable.

A3: We posit that the impact on model performance is comparatively minimal at a pruning ratio of 20%, whereas the pruning error becomes significantly more pronounced at a pruning ratio of 50%. Our proposed strategy is capable of effectively mitigating this error, thereby yielding more substantial performance improvements.

Q4: The layer-wise pruning ratio relies on empirical hyperparameter α, lacking theoretical justification and more ablation studies.

A4: Thank you for your suggestion. We have added ablation experiments on Llama-7B, and the results are as follows. When the value of alpha increases to 7, the model achieves optimal performance. Further increment of alpha leads to a decline in model performance.

Q1: The paper focuses on pruning-based acceleration of LLMs but lacks a comparison with quantization-based acceleration methods. Including such a comparison would better demonstrate the proposed approach's advantages of LLM compressing techniques.

A1: Thank you for your suggestion. Quantization is also a highly effective method for LLM compression. Since the directions of pruning and quantization as compression strategies are different and they can yield cumulative benefits, we have not compared our method with quantization at present.

Q2: Table 2 only compares the proposed method with LoRAP. Expanding the evaluation to include more state-of-the-art pruning or compression baselines would strengthen the demonstration of the method's effectiveness and generality.

A2: Thank you for your suggestion. In Table 2, we compared our method with the LoRAP method, which has excellent compression performance. However, due to memory limitations, we were unable to conduct experiments on LLMPruner with Llama2-7B. Below are the relevant results reproduced from other works, which show that our pruning strategy can better preserve model performance.