The LLM Surgeon

Thank you for your feedback and help to improve the paper. In particular, we thank the reviewer for noting the paper is well-organised and easy to comprehend and deem our fundamental contributions reasonable and innovative.

1. compute and memory

We agree with the reviewer that we should include a quantitative comparison between methods. Compared to other methods (e.g. SparseGPT), we are able to obtain better compression results but only at the expense of longer compression time. We will make sure this is absolutely clear from the main text. The table below shows compression time on Llama-v2 7B and 13B models. We will add these results to the paper appendices:

The method is highly parallelizable and more GPUs can be traded for a linear speed-up in time, if desired. Compression time measured using the required memory:

Further engineering efforts may further speed up compression, especially for unstructured pruning. As mentioned, our method does come with a higher computational cost compared to other approaches. Importantly, compression of LLMs in general only needs to happen once after which a pruned model can be deployed infinitely many times without further cost. This motivates our method which takes longer to run but provides much better final test performance.

2. Larger models

We agree with the reviewers that it would be interesting to assess performance on even larger models. The model sizes were chosen to demonstrate compression on OPT models with millions of parameters but also on the Llama architecture, demonstrating pruning of models in the order of billions of parameters. Following the suggestions by the reviewers (zzzW, 2GDn), we now also have results for the larger Llama-v2 13B model and evaluated its performance on downstream tasks (see results below). We find that our method also outperforms baselines on this larger model, as well as on downstream tasks.

3. Global rank ordering

We thank the reviewer for raising this point. Our proposed improved curvature approximations use gradient information that relates curvatures of individual layers to the final global objective. The gradient terms can be interpreted as the relative influence layer outputs have on the final objective. For this reason, our improved curvature estimates are especially useful in conjunction with global thresholding, which allocates weight to be pruned globally rather than for layers separately. Nevertheless, we will run an ablation study on this and include it in the appendix.

4. Structured pruning related work

We thank the reviewer for bringing other structured sparsification works to our attention. Both works are going to be published at NeurIPS 2023 and appeared a few months before the ICLR deadline on arXiv so should be considered concurrent work. Further, [2] does not seem to include any experiments on structured pruning of rows and columns and only considers semi-structured pruning (e.g. 2:4). The other paper [1] does include such experiments, but for -20% sparsity on Llama-v2 reports a very large degradation in performance from 12.62 to 74.63, whereas our method on the same model, sparsity and data achieves test perplexity from 5.12 to 6.18. We would like to emphasise that structured pruning is a very hard task on which our method outperforms any existing approach by a large margin. Regardless, we deem [1, 2] relevant prior literature and will add them to the related work section.

[1] LLM-Pruner: On the Structural Pruning of Large Language Models. In NeurIPS, 2023

[2] A Simple and Effective Pruning Approach for Large Language Models. In Arxiv, 2023

** Edit 20 Nov: added timing results for Llama-v2 13B and GPU memory

We are very glad that most of the concerns have been addressed. In this reply, we hope to take away the remaining hesitation regarding model efficiency.

Overall, our method was developed for structured pruning, where it shines. Here we are roughly as fast as the baseline (K-OBD) and much better (17h08m versus 16h58m on Llama-v2). For unstructured, it is also better but indeed a bit slow, but this is not the main point of the method. We included this comparison to have a well-established baseline to compare to and because the method outperformed pruning performance on all structure types.

• why not replace costly pruning with finetuning

This is indeed an important question. However, we would like to push back on the possibility of using finetuning as an alternative for our approach. It is indeed that for unstructured and semi-structured pruning finetuning can be beneficial and can sometimes work [1], although not always reliably. The reason for this is that limited data used for compression is not always effective in preserving the capabilities of an existing pre-trained model. Empirically, we found that first-order updates (e.g. LoRA) can not always make up for removals of weights (see also Sec. 4.2), a problem which gets much worse in the case of structured pruning, in which the degradation in loss can be very large if weights are removed without updates that take weight correlations into account.

Compared to finetuning, our approach has two big benefits, namely: (a) it is hyperparameter free, (b) allows dynamic allocation which weights to be removed, and (c) correlated weight updates which allow retaining as much information as possible from the existing pre-trained model, even with limited data. In experiments, we found that finetuning to not sufficient to recover performance and are not aware of literature using finetuning to work for structured pruning. We believe this difficulty to be the reason why relatively few papers even consider structured pruning in their experiments. As such, our method has important benefits compared to finetuning approaches. We do think we could have made this point more clear in the paper and will improve this.

• computational cost

Our method takes longer to run but achieves higher performance.

We very much admire efforts such as [1] or SparseGPT as these works demonstrate how much compression can be achieved only using activations and weights. Nevertheless, only relying on activations and weight strength implies strong assumptions on the curvature of the loss and makes these methods unsuitable for more constrained structure types. Our improved curvature allows for dynamic allocation of weight removal and improved correlated weight updates. As a result, our method can be used for structured pruning and results in higher overall performance.

Although we are proud of the efficiencies that allow scaling very rich Kronecker structures to LLMs, our resulting method does remain much more computationally expensive compared to existing pruning methods in relative terms. From a practical perspective, however, we argue that the compute time may not be 'excessive' at all! The reason for this is that a model only needs to be pruned once after which it can be deployed infinitely many times. Therefore, it can very well be economically advantageous to temporarily spend multiple GPU resources on compression to achieve a better performance after pruning. In absolute terms, the use of multiple large GPUs is very common practice in the field of large language models and many more GPUs are typically used to train and deploy large language models. Moreover, our method can be trivially parallelized in case further speed-ups or larger models are required. We hope this provides a better context and emphasises the trade-off that should be made by the practitioner.

Further, the trade-off between performance and computational cost might very well be fundamental (therefore something that we may not be able to fix). Given that, we sincerely hope that the computational cost is not a ground for rejection, as our method offers significant benefits including structured pruning and state-of-the-art pruning performance.

We thank the reviewer and do agree that the trade-off should be communicated clearly to the reader and practitioners who may wish to use the method. We promise to make an effort to ensure this is very clear from the main text.

Thank you for your feedback and help to improve the paper. We thank the reviewer for finding the paper is well written, clearly presented and theoretically principled. We further value that the reviewer appreciated the 'careful derivations of correlated update rules' and noted the strength of the work and state-of-the-art pruning results, especially on structured pruning.

Use of approximations

Due to the high dimensionality of LLM models, the constraint optimization problem considered in this work is inherently intractable. This means that it can be mathematically proven that the problem can not be solved in an exact manner and the sheer use of an approximation should thus not be considered a 'weakness' but rather a necessity. In fact, most other methods (even magnitude pruning) can be seen as some sort of crude approximation to the Hessian of the global loss and are strictly more approximate than ours. Thus, compared to other methods we use the most accurate curvature approximation. We expect this to be a misunderstanding and will make this more clear in the main text to prevent confusion.

Irregular sparsity patterns

We thank the reviewer for raising this as the computational benefits of the considered pruning structures might not have been emphasised enough in the paper. It is not true that structured pruning results in irregular sparsity patterns. Rather the opposite, it provides the most regular sparsity patterns - hence the name 'structured' - and is very easy to accelerate. As complete rows and columns are removed, structured pruning of X% of weights means that matrix dimensions can directly be reduced by X%. The resulting smaller matrices have simply lower dimensions and are thus directly amenable to existing standard CUDA/and hardware-accelerated kernels. We recognize that we only very briefly mention this in Sec. 2. We thank the reviewer and will make sure this point is made more clear in the main text.

Savings in terms of FLOPS / compute time

We are not exactly sure what is meant by 'updating offsets gains', but will assume the question is whether reported weight reduction relates to actual computational and memory savings at deployment. Unlike most other sparsity patterns, structured pruning allows for a direct reduction in the dimensionality of weight matrices by removing complete rows and columns. Therefore, a reduction of weights will result in the same reduction in the number of FLOPS. This is not generally true for most other sparsity patterns, and in fact an important benefit and in part the motivation for structured sparsity patterns. Below, we provide compression time of compressed models which we will add to the paper appendices.

Inference time overhead

A major advantage of our method is that there is no overhead at inference time. All savings are absorbed into the weight of the neural network and there are no separate offsets to be separately stored. We have not mentioned this benefit explicitly and will make sure it is clear in the paper.

Related work

We thank the reviewers for pointing out this work. We find the reference relevant and will add it to an updated related work section.

Questions / additional experiments

1. Performance

The table below shows compression runtimes on the Llama-7b model. We will add these results to the paper appendices.

Our method is most efficient for structured pruning, but it must be noted that engineering efforts may further improve speed for unstructured pruning. The focus of the paper is structured pruning, on which we achieve state-of-the-art compression rates. Importantly, compression of LLMs only needs to happen once after which a pruned model can be deployed infinitely many times without further cost. This motivates our method which takes longer to run but provides much better final test performance. We will add the timing results to the paper and make the last point clear from the text.

2. Large model sizes

We agree that it would be interesting to assess performance on even larger models. Following the suggestions by the reviewers (zzzW, 2GDn), we now also include results for the larger Llama-v2 13B model and evaluated performance on downstream tasks (see reply to Reviewer 2GDn). We find that our method also outperforms baselines on this larger model, as well as on downstream tasks.

Thank you for your feedback and help to improve the paper. We appreciate that the reviewer found the approach sound and well presented, further recognising our contributions to Hessian-based pruning amenable to LLMs.

Training with C4 instead of wikitext2

Unlike SparseGPT which used the C4 dataset, we used the wikitext2 dataset for compression in our experiments. Although the amount of data is the same, C4 can be regarded slightly more general in that it contains multiple languages. We agree with the reviewer that it would be interesting to evaluate performance with C4. Following these comments, we have therefore also evaluated the use of the C4 dataset as reference data for compression in the case of structured pruning. We find that our model in this case still outperforms our baselines.

The additional results on downstream tasks can be found in our reply to Reviewer zzzW (also includes training with suggested C4 data)

Further, we would like to stress that all results in our paper (including the SparseGPT) were trained on wikitext2 to enable fair comparison. To do so, we used the same dataloader and evaluation script as the official SparseGPT repo and reran all SparseGPT results to be trained on wikitext2. This even resulted in better scores for our SparseGPT baseline than the C4 trained results from the original SparseGPT paper. Yet, we find that our method using improved curvature estimates still outperformed the baselines. We are aware that these efforts might not have been made clear enough and will ensure this is clearly stated in the final version.

Downstream benchmarks

The reviewer raised the need for downstream benchmarks 'for a fair comparison and acceptance of the work'. Although perplexity scores in prior work tend to correlate well with downstream performance, we recognise the need for such additional experiments to offer a complete comparison between methods. We have performed downstream benchmarks on structured pruning compressed using wikitext-2 but also the C4 dataset.

The additional results on downstream tasks can be found in our reply to Reviewer zzzW

We find that our method not only performs well in terms of test perplexity but also correlates well with downstream performance, outperforming the baselines on these downstream tasks.

Related work

We agree with the suggestion by the reviewer to extend the related work so that all important pruning work on large language model compression is included. In particular, we will include related work on structured pruning and other LLM pruning works that use a curvature-based approach.

This is indeed a typo. Typos will be fixed.