Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models

Dear Reviewer hn9p,

Thank you for your insightful feedback and positive evaluation of our manuscript. We appreciate your recognition of the strengths of our work, particularly the novelty of our channel permutation technique and the comprehensiveness of our experimental evaluation. We address your concerns and questions below:

Weakness 1: The first technique, relative importance, and activation, seems to be an incremental improvement.

Reply: Thank you for your comments. We acknowledge that the improvement of RIA over prior work is modest, around 0.1 - 0.2 points. However, it is important to establish the baseline for this evaluation. If the baseline is zero, an improvement of 0.1-0.2 points may seem negligible. Yet, our objective is to recover the performance of the dense model after pruning. Therefore, the relevant improvement should be measured in relation to the performance drop from the dense model. As reported in our article: "Notably, our method achieves a 50% improvement in mitigating the performance drop of the Dense model compared to SparseGPT (16% in LLaMA and LLaMA2 model family), and a 17% improvement compared to Wanda (13% in LLaMA and LLaMA2 model family)." We believe that any improvement over 10% should not be considered incremental, and this improvement is consistent across all models in our study.

We apologize for not providing a detailed formula to compute this percentage, which may have led to confusion. To address the reviewer's concern, we have added an explanation of this formula in Appendix J of the newly uploaded manuscript. Here is a simplified version for clarity:

Given two post-training pruning methods, A and B, with perplexities P(A) and P(B), and the baseline perplexity P(D) of the dense network, the percentage by which method B prevents the performance drop compared to method A can be computed as $\frac{P(A) - P(B)}{P(A) - P(D)}$. This formula provides a fair evaluation of the newly proposed method compared to previous methods, considering the dense model's performance in the post-training pruning field. Using this calculation, the improvement of RIA in preventing performance drops compared to Wanda is 17% across all models, and 50% compared to SparseGPT across all models.

Weakness 2: In Section 5.3, which discusses N:M sparsity, I was anticipating experimental results on smaller LLMs such as Llama2-7b, and a higher sparsity ratio than 50%. This would potentially highlight the advantages of the proposed method over SparseGPT and Wanda more effectively. Could the authors provide their insights on this?

Reply: Thank you for your suggestion. In response, we have conducted additional experiments with varying sparsity levels of N:M sparsity — 25% (1:4 sparsity), 50% (2:4 sparsity), and 75% (3:4 sparsity) — on the Llama2-7b model with the default setting in our article.

To explore this further, we compared two pruning strategies: pruning from layer 4 to layer 32 which is the solution proposed by LLM-pruner to prevent the performance drop and pruning all layers. Our findings confirm that preserving the first and last layers indeed mitigates performance degradation.

Our results indicate that at a 75% sparsity level, there is a significant decline in performance. Although in comparison to Wanda, there is a significant improvement in RIA, the perplexity goes too high to be implemented in real cases.

Here is a summary of our results:

Pruning from layer 4 to layer 30:

Pruning all the layers:

[1] LLM-Pruner: On the Structural Pruning of Large Language Models, NeurIPS 2023

Weakness 3: Regarding the inference latency under n:m sparsity, I would like to suggest that instead of providing layer-wise speedup, the authors could consider providing end-to-end latency for the pruned LLMs. My reason for this suggestion is that I am curious about whether n:m sparsity is indeed an effective structured pruning pattern within LLMs, especially when compared to pure structured pruning methods such as LLM-Pruner. Could the authors provide their perspective on this?

Reply: Thank you for your valuable suggestions and questions. We separated your question into two parts:

1. The performance of N:M sparsity and structured sparsity. We adopted LLM-pruner as the representative structured pruning method to compare with our methods. The results are shown in the above reply. We use the detailed hyperparameter setting: C4 with 128 samples as calibration data, and Wikitext2 as the evaluation data. The PPL of structured pruning (LLM-pruner) increases significantly with just 25% sparsity. This suggests that structured pruning is not effective in post-training pruning scenarios, as the pruned information from the entire channels cannot be recovered without retraining or fine-tuning. Also, To further address the reviewer's concern, we add Appendix E to explain this phenomenon in detail.

2. We here offer the inference latency of the structured pruned model with 50% sparsity and 2:4 sparsity model. Our experiment is on LLaMA2-7b, the sequence length of the input is 12. The deployment is on 2 Nvidia A100 80GB in parallel on Ubuntu 22.04.1 system. Here, we offer the inference latency with varying batch sizes.

The table shows that structural 50% sparsity marginally outperforms the 2:4 sparsity model as batch sizes increase. It's important to note that when dealing with smaller input batches, both structured 50% sparsity and 2:4 sparsity models do not significantly boost inference speed. However, as the batch size grows, the acceleration for 2:4 sparsity approximates 1.5x, whereas the structured 50% sparsity tends to reach about 1.7x acceleration.

In conclusion, given the performance and the inference latency analysis, although structured pruning could provide a faster acceleration than N:M sparsity with the same sparsity level, the performance drop is unignorable even when the sparsity is very low. Therefore, in the current situation, the N:M constraint sparsity is the only sparsity pattern that strikes the balance between performance and acceleration speed.

We thank the suggestion the Reviewer gave and we added this part in Appendix E to offer a comprehensive comparison of N:M sparsity and structured sparsity.

Dear Reviewer,

We are happy to hear that your concerns have been addressed. Thanks for your support!

Best wishes,

Authors

Dear Reviewer hn9p,

Thank you for your thoughtful evaluation and constructive feedback on our manuscript. We appreciate your recognition of the strengths of our work, including its simplicity, clarity, and practicality. Your insights have been instrumental in guiding our revisions and enhancements. We address your concerns and questions as follows:

Weakness 1: The observation that encouraging a more uniform sparsity pattern is beneficial was also made by Wanda, RIA seems to be an extension of that (also across columns). Similarly, that permutation reordering can be helpful for n:m sparsity was found by [1], this paper only introduces a simpler but more scalable heuristic for finding such a permutation, based on average activation values. While there is some novelty, it is not particularly high.

Reply: We are grateful for your insightful feedback and the opportunity to clarify the unique contributions of our work. Our approach distinguishes itself in two critical aspects.

Firstly, RIA incorporates a consideration of the relative importance of weights. As written in the article, "In practice, we find similar issues also exist in other prevalent pruning metrics, e.g., Wanda prunes around 600 channels out of 5120 channels, with more than 10% channels corrupted. Given that well-trained LLMs contain unique information in the input and output channels, it is critical to avoid channel corruption in post-training pruning." The proposal of relative importance adopts a more uniformly pruning structure that prevents this issue. To address the reviewer's concern, we created Appendix E in the newly uploaded manuscript that explains well the motivation to introduce the relative importance. The performance across all the tests involved in this article, for instance, PPL across all the models on both unstructured and semi-structured sparsity, the sensitivity test of sparsity and number of samples, zero-shot performance, suggests that incorporating relative importance into the pruning metrics offers a consistent advantage against Wanda.

Secondly, the proposal of the channel permutation in this article is to offer a solution to introduce N:M sparsity in large language models. As we introduced in Appendix G: "The greedy method [1] is challenging to implement in LLMs due to its prohibitive running time, as shown in Table 1. We conducted only a single experiment for performance comparison on LLaMA2-13b, which took us 3 days. We tested the greedy method with 100 escape iterations to handle the permutation, the PPL on Wikitext-2 with a permuted 2:4 constraint is 8.01, which is comparable to our CP method (7.99). However, the CP's execution time was just 30 minutes, making it possible to be applied in LLMs." This is a big improvement since no previous works of post-training pruning really stepped into the N:M sparsity field because of the big drop in performance. However, in this article, with an acceptable channel permutation time, the case of RIA (2:4+CP) performs and surpasses the dense model with a large amplitude on BoolQ, MNLI, and RTE datasets of zero-shot performance. We want to draw more attention to the application of how to utilize the N:M constraint since it is currently the best way to retain performance while also offering inference speedup in real cases because it is hardware-friendly. Our method offers a solution to make N:M sparsity possible to be adopted in deploying LLMs.

In addition, as commented in the Reviewer's content, it is true that separating the channels into groups is heuristic, but it has its nature. To better explain our motivation for introducing this and to address the reviewer's concern, we added context in Appendix A that shows the reason why our method can have better results. In short, the post-channel-permutation distribution (2:4 + CP) of input channels’ retained degrees can be seen in Figure 5, closely mirroring the results of unstructured pruning which is the global optimal of the specific pruning metrics (for instance RIA, Sparsegpt, and RIA).

In summary, while our work builds on existing concepts in the field, the refinements and innovations we introduce, particularly in terms of scalability and efficiency, contribute significantly to the practical application of N:M sparsity in large model training.

[1] Channel permutations for N: M sparsity, NeurIPS 2021

Questions 1: what is the channel here in the Transformer models? Transformer models does not have channel or column.

Reply: Thank you for your question. In our study, when we refer to 'channels' in the context of Transformer models, we are specifically referring to the dimensions of the input or output weights in the model's layers, which aligns with a concept frequently used in the context of MLP.

In more detail, the term 'channel' in our usage can be understood as analogous to rows or columns of the weight matrices in Transformer layers. For example, in a fully connected layer of an MLP or a linear layer of a Transformer, each row (or column, depending on the implementation) of the weight matrix can be thought of as a 'channel'. These 'channels' represent different learned features or filters that the model applies to the input data.

This terminology, while more commonly associated with CNNs, is also applicable to Transformer models, especially when discussing the dimensions of weights and their transformations. The use of the term 'channel' in this context allows us to discuss the architecture and optimization of these models more precisely, particularly in relation to weight sparsity and efficiency improvements, which are central themes in our research.

Our adoption of this terminology is consistent with established literature, including the NVIDIA article [1], which discusses channel permutations as a method for implementing sparsity in neural networks. It's also important to note that similar terminologies are used in other works in the field [2].

[1] Channel permutations for N: M sparsity, NeurIPS 2021

[2] A simple and effective pruning approach for large language models

Question 2: for ""We employ 128 samples from the C4 dataset", is it possible/worth to do the experiments on a larger and more diverse calibration set (128 might be limited)?

Reply: Thanks for the question. We have done the experiments and included them in the second response.

Dear Reviewer ee73,

Thank you for acknowledging the contributions of our manuscript and for providing valuable feedback to further enhance its impact. We are pleased to address the highlighted concerns and questions.

Weakness 1: Overall the manuscript has solid contributions, but expanding the variety of models, tasks, and languages could strengthen the demonstrated effectiveness. Testing scalability and comparing to other recent techniques would also help round out the evaluation. But within the chosen scope, the paper delivers valuable advancements for efficient LLM pruning.

Reply: We thank the suggestion of the reviewer and we agree that we need to test the scalability and comparing to other recent techniques. To address the reviewer's concern, we added experiments on LLM-pruner and pruning only based on the activation which can be seen as a comparison between structured pruning, N:M pruning and unstructured pruning. In addition, we also included the experiments of combining post-training pruning and post-training quantization into consideration. Please refer to the general comments.

Weakness 2: For "We employ 128 samples from the C4 dataset": using only 128 samples from C4 as the calibration data is quite limited. With so few samples, the activation statistics may not sufficiently capture the full distribution.

Reply: Thank you for your insightful suggestion. In response, we have conducted a comprehensive comparison using varying numbers of samples from 2 to 512. Please refer to Figure 9 in the new uploaded manuscript. The experiments are conducted on LLaMA2-7b. The calibration dataset used in this experiment is C4 and evaluation is assessed on Wikitext2. The sparsity is always fixed at 50%.

PPL changes regarding the increase of number of samples.

Pruning time (s) changes regarding the increase of number of samples.

Our findings indicate that increasing the number of calibration samples does not significantly enhance performance. However, it does lead to a substantial increase in the pruning time of all the methods.

We believe this outcome is due to the distinct nature of calibration samples as compared to training samples. Calibration data is primarily utilized for calculating the activation distribution, rather than for shaping the loss curve. This distinction suggests that a large dataset is not as critical for calibration as it is for training.

Furthermore, this approach aligns with methodologies used in other notable works, such as SparseGPT and Wanda, which also employ 128 calibration data samples.

It's noteworthy that the performance of RIA remains stable as the number of samples increases, achieving a notably low Perplexity (PPL) with as few as two calibration samples.

Weakness 4: What is the relevance of the experiments in Figure 2 to the claim that activation outliers exist independently of the dataset and model's parts? The reviewer thinks that even if activation values exhibit a high correlation between two datasets, it is possible that activation outliers can be eliminated. Therefore, it would be helpful to clarify the connection between Figure 2 and the claim about activation outliers.

Reply:

• Thanks for the reviewer raising this question. To address the reviewer’s concern, we added the below content in Section 3.3 in the updated file.

  “We offer evidence that the Spearman Rank correlation between pairs of the activations of different datasets is positive. This positivity is a necessary condition to incorporate the activation into our RIA formula. And indeed it is always satisfying.”

Weakness 5: Can we expect additional performance improvements when combined with post-training quantization methods such as Smooth Quant or AWQ?

Reply:

• Thank you for your insightful question. Because there is not a remarkable decrease in performance using quantization, it indicates that our method could also be applied in combination with quantization. This will be a new and valuable point to comment on in the article.

• We tested on two quantization methods: GPTQ [4] and AWQ [5]. Here are the results for pruning + GPTQ and pruning + AWQ on LLaMA2-7b, weight bit (wbit) = 4, using C4 for calibration and Wikitext2 for evaluation. We still get the best performance on two quantizations among all pruning methods:

  	Magnitude	SparseGPT	Wanda	RIA
  Unstructured (50% sparsity)	16.02	6.99	6.92	6.81
  Unstructured (50% sparsity) With GPTQ	15.21	7.59	7.41	7.28
  Unstructured (50% sparsity) With AWQ	17.12	7.15	7.10	6.97

• We have identified two primary strategies for merging post-training pruning with quantization: first pruning then quantizing (a), and first quantizing then pruning (b.)

  • Our findings indicate a preference for (a) pruning before quantizing.
  • This is potentially because for (b), conventional block-wise quantization relies merely on the min and max weights within a block, and pruning can lead to a reduction in these extremes, thereby potentially aiding quantization.
  • In contrast, (a) quantizing prior to pruning adversely affects the computation of weight importance for pruning, resulting in diminished performance.

• The discussion is added as Appendix I in our manuscript.

Minor: Why is the title "plug-and-play"?

Reply:

• The reason we call it plug-and-play is because of 3 reasons.

  • Ignorable Pruning time.
  • No need for any additional finetuning or retraining.
  • With channel permutation, our method can perform well in the zero-shot experiments when adopting N:M sparsity pattern while currently other post-training pruning methods cannot.

• To further address the reviewer’s concern, we introduce in the discussion: “We decide to call our method “plug-and-play” because, in the zero-shot experiment, it demonstrated performance similar to the original dense model with no need for any additional finetuning or retraining whereas, the other post-training pruning method such as Sparsegpt and Wanda are not “Plug-and-play” because they don’t have channel permutation.”

Minor: The hyperlink in the 6-page appendix seems to be incorrect.

Reply: Thank you. We have fixed that in our updated version of the paper.

Reference:

[1] Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning, NeurIPS 2022

[2] SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot, ICML 2023

[3] LLM-Pruner: On the Structural Pruning of Large Language Models, NeurIPS 2023

[4] GPTQ: ACCURATE POST-TRAINING QUANTIZATION FOR GENERATIVE PRE-TRAINED TRANSFORMERS, ICLR 2023

[5] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Weakness 2: According to AWQ [1], activation-aware weight quantization, which selects important weights based on activation distribution rather than weight distribution, outperforms traditional weight-based quantization. Inspired it, the reviewer suggests that it would be meaningful to consider baseline methods based on activation-based weight pruning for comparison. Therefore, the authors might incorporate and compare unstructured pruning based solely on activations in Table 2.

Reply:

• Thanks for the question. Pruning solely on activation means the whole channel will be removed together, thus pruning on activation actually belongs to structured pruning. We have applied this method on LLaMA2-7b, LLaMA1-13b, and LLaMA1-30b. The results are included in the Table on the last response and also in updated Table 2 as an ablation test in the newly uploaded version of the manuscript.

• The results indicate that relying solely on activation, the PPL increases significantly with just 25% sparsity, reaching over 5000 at 50% sparsity on each model. This suggests that structured pruning is not effective in post-training pruning scenarios, as the pruned information from the entire channels cannot be recovered without retraining or fine-tuning.

Weakness 3: In addition to the comparisons with N:M sparsity methods in Table 4 and Table 5, it is advisable to include a comparison with other structured pruning techniques in terms of performance and inference speed improvement. For instance, including a method like Dejavu [2] in the comparison would enhance the comprehensiveness of the evaluation.

Reply:

• To enhance the comprehensiveness of our evaluation, we have included additional results from the LLM-pruner [2], a state-of-the-art structured pruning method in our article, as detailed in our first response. These results demonstrate the superiority of our N:M semi-structured pruning.

• As for Dejavu [3], we would like to clarify that Dejavu cannot be fairly considered for comparison in this study because it differs fundamentally from our method in terms of its operational framework.

  • Dejavu is not a post-training pruning method as it introduces additional parameters and requires the predictor to be trained for several epochs (50, as per the provided code).
  • Consequently, Dejavu requires computational time resources, while our method is designed to be applied without retraining, offering a 'plug-and-play' solution for practical scenarios.

• To address this distinction and your concern, we have added the following statement in the related work section of our manuscript.

  “Note that the method proposed in this article is designed for application without retraining and finetuning, which differentiates it from techniques like Dejavu [3] that require additional training steps. Consequently, our comparisons are focused on methods that do not involve retraining, such as Magnitude, Wanda, and SparseGPT.”

• Regarding inference speed improvement, we offered the acceleration rate of the semi-structured pruning in Section 5.4. For the end-to-end inference acceleration of structured sparsity and semi-structured sparsity, we here offer their inference latency. Our experiment is on LLaMA2-7b, the sequence length of the input is 12. The deployment is on 2 Nvidia A100 80GB in parallel on Ubuntu 22.04.1 system. Here, we offer the inference latency with varying batch sizes.

  	1	8	16	64
  dense	281.13 ms	547.82 ms	1011.81ms	3742.76 ms
  structured 50% sparsity	238.92 ms	326.14 ms	616.13 ms	2181.69 ms
  2:4 sparsity	225.60 ms	357.48 ms	731.81 ms	2495.41 ms

  The table shows that structural 50% sparsity marginally outperforms the 2:4 sparsity model as batch sizes increase. It's important to note that when dealing with smaller input batches, both structured 50% sparsity and 2:4 sparsity models do not significantly boost inference speed. However, as the batch size grows, the acceleration for 2:4 sparsity approximates 1.5x, whereas the structured 50% sparsity tends to reach about 1.7x acceleration.

Dear Reviewer NPvK,

Thank you for your review of our paper. We appreciate your positive evaluation and also the questions to our submission which aims at improving our presentation and encouraging us to process more empirical evidence to support our methods.

We will reply to each of the weakness you mentioned and the questions you raised.

Weakness 1: Insufficient experimental support for motivation: This paper argues the existence of 'channel corruption' asserting that removing input/output channels results in decreased performance as observed in prior works. However, the paper lacks empirical evidence to substantiate this claim. It would be valuable if the authors could include preliminary experiments to provide a basis for their motivation.

Reply: Thank you for your feedback. We understand the need for empirical evidence to support our assertion of 'channel corruption' and have taken steps to address this in our revised manuscript Appendix E (Evidence of Channel Corruption).

1. Evidence of Channel Corruption: We have updated our manuscript to include new empirical data demonstrating channel corruption. As illustrated in Figure 6, our analysis of the Wanda [1] model shows that approximately 10% of channels exhibit corruption. This evidence directly supports our claim and provides a clearer understanding of the phenomenon.

2. Detrimental Impact of channel corruption on Model Performance:

• We have conducted experiments using LLM-Pruner [2] which is a SOTA structured pruning method used in LLMs. We also compare our method with the baseline structured pruning method of pruning merely based on the activation aligning with the experiment you proposed in Weakness 2. We utilize this comparison to further illustrate the effects of channel corruption.

• We separated the experiments to two different Tables. One utilized the strategy in LLM-Pruner that only pruned the layers from 4-30 layers and the other one pruned all the layers.

• Our results indicate: 1) The performance of LLM-Pruner, perplexity, notably increases at already a low sparsity level of 25%. 2) Using LLM-Pruner's approach, which involves skipping 6 out of 32 layers, led to lower perplexity scores (PPLs). However, the balance between performance and the increased inference speed due to sparsity requires further exploration. We aim to continue investigating this issue promptly for this study.

3. Additional Section in Appendix E: To further address your concerns, we have added a dedicated section in Appendix E of our manuscript. This section explicitly discusses the influence of channel corruption on post-training pruning methods. Due to time constraints, our current experiments are limited to the LLaMA2-7b model. We plan to extend our investigation to more models to provide comprehensive evidence of channel corruption and its implications.

Prune from layer 4 to layer 32

Prune all the layers