SliceGPT: Compress Large Language Models by Deleting Rows and Columns

Hi reviewer jAXk, thanks for taking the time to review our work.

experiment section does not comprehensively address the comparison between SliceGPT and SparseGPT

We are working on this, see main rebuttal.

The absence of inference time data for SparseGPT in the experiments makes it challenging to convincingly demonstrate the superior efficiency of SliceGPT. (and Q2)

This was a challenge for us: the SparseGPT authors do not provide a way to verify and kind of speedup using their method. See for example this github issue. The issue is that pytorch does not provide off-the-shelf support for structured sparsity. This means that at the moment, it’s not possible to verify end-to-end speedup for SparseGPT. Our work-around is to provide end-to-end results for SliceGPT, and apples-to-apples comparisons with SparseGPT on their terms. Please see main rebuttal for more details and forthcoming experiment.

lack of comparative analysis with state-of-the-art pruning methods such as low-rank approximation, unstructured sparsity, and block sparsity.

You’re right – we should add some rationale here.

1. different sparsities. We compared to SparseGPT 2:4 sparsity because this is the SOTA method for LLMs. SparseGPT also provides unstructured sparsity and other sparsity structures, but these provide little to no speedup, which is the motivation the developers of 2:4 sparsity. This is why we chose SparseGPT 2:4 as a baseline, because it’s the strongest competitor. If we’re doing better than this, we’re doing well.

2. Low rank approximations. We did make reference to low-rank works, which take each weight matrix and replace it with a pair of low-rank matrices. To our knowledge, no one has applied this idea to LLMs, and as we discussed in the text, it seems unlikely that they could improve the performance of LLMs. Of course, the LoRA method has been very influential for fine-tuning, but we’re not aware of a method to use low-rank approximations in improving deployment of LLMs.

3. Other works. We’ve added reference to LLM-pruner, a modern method for pruning LLMs. LLM-pruner provides less speedup than SliceGPT (see their table 3), requires fine-tuning post sparsity, and does not retain perplexity as well as our SliceGPT method (see their table 1). We’ve also added more context around other pruning methods, see response to reviewer 96nz.

Can you show the performance (perplexity, inference time) of SliceGPT at 50% sparsity? (Q1)

Yes, of course. We ran a small experiment for you just now: on OPT66B with 50% removal, SliceGPT achieves a wikitext-2 ppl of 11.39 (against a dense baseline of 9.34). Fixing the sequence length to 128, we were able to achieve a batchsize of 32 on 1x H100 card, leading to a throughput of 440 tokens/s. For the dense model, we required 2x H100 cards and could only fit a batchsize of 16, leading to 141 tokens/s. That’s a 3.12x speedup, and we only use one card! Please read our main rebuttal to understand the nuance of comparing SliceGPT with SparseGPT, as well as a more comprehensive experiment on the effect of batchsize.

layernorm transformers can be converted to RMSnorm" is not well motivated.

Apologies if this wasn’t clear. SliceGPT depends on the computational-invariance trick. In turn, this depends on the commutation property of equation (2) – you can apply an orthogonal transform before-and-after RMSNorm without changing the model output. The trick does not work for LayerNorm, so we have to convert a network to RMSNorm before we apply SliceGPT.

Do the authors plan to release the code?

100%, we feel the same as you. It’s difficult for us to release pre-publication without de-anonymizing ourselves, but our code will be on github with an MIT license as soon as we’re published. If you would like to inspect the code this week, we’re happy to arrange that (though it will take a while for us to go through the code, to be certain that we’re not de-anonymizing ourselves).

recompile the pdf

Will do! We blame overleaf 😊

use of a random projection

We could indeed use a random projection, but it would break eq. 2. It’s not clear to us that random projections could help here, but we hope that you experiment with our code and let us know on release!

not all layers may be equivalent signal-wise

You’re right! We’re preparing such an experiment, see main rebuttal.

what do the authors mean by "using dense kernels in our compressed models"?

Ah, this is something we haven’t communicated very well. Sparse models, like those produced by SparseGPT, are actually not easy to run. Pytorch does not have off-the-shelf support (kernels) for 2:4 structured sparsity yet. In fact, those authors decline to provide actually accelerated models in this Github issue: https://github.com/IST-DASLab/sparsegpt/issues/15. We simply meant that the models produced by SliceGPT are easy to run in standard pytorch. We were trying to be diplomatic about their (very cool) contribution, whilst letting you know that our code is easy to run.

small proof of Equation (2)

Sure, we can add one to the appendix.

"Theorem" is too strong for Theorem 1, I suggest "Proposition" or "Remark".

Sure, fixed.

Typo

Fixed, thanks.

Hi Reviewer fXjP,

We hope we managed to give satisfying answers to your questions in our reply above. One of your questions which we did not fully answer was about varying the dimension of each layer: we added experiment (3) in the main rebuttal which demonstrates the feasibility of this, with performance (perplexity) gains for OPT models but degradation in Llama models. We believe varying sparsity/slicing by layer is an important area of research, and hope that a future work can devise a such a scheme for SliceGPT that works for more models than just OPT.

Should you have any further questions, please let us know. If not, we would be very grateful if you were to consider increasing your score.

Hi Reviewer 96nz, thanks so much for your comments. Here are our replies to your queries.

(1) don’t acknowledge prior work on neuron pruning

You’re right, this is our bad, and we’ve added a new section to the text including those references and more. One thing to note is that LLMs are so much bigger than convnets, that some previous method don’t apply. For example, LeCun’s original OBS method requires O(N^5) operations to prune an NxN weight matrix (an N^3 matrix inversion for each element you want to prune). Completely infeasible on LLMs, where N might be 12k. Nonetheless, we should have cited more work in the pruning literature, even though we didn’t build on it directly.

(2) Second, the results in Table 1 suggest to me that 2:4 sparsity is preferable to the proposed technique?

See our main rebuttal on comparison with SparseGPT.

Hi Reviewer 96nz,

You were concerned about the performance of SliceGPT compared with SparseGPT in terms of inference runtime and accuracy. We think our main rebuttal provides a clearer comparison of the two methods: we show in experiment (1) that SliceGPT (25%) is the same speed as SparseGPT (50%), before we take improved data movement (and increased possible batch size) into account. Experiment (2) demonstrates the increase in batch size possible with SliceGPT leads to throughput of up to 6.26x that of the dense model, due to memory savings.

Should you have any further questions, please let us know. If not, we would be very grateful if you were to consider increasing your score.

Hi Reviewer gLsa, thanks so much for your comments. Here are our replies to your queries.

The accuracy loss is not neglibible

We agree that this isn’t the right word – we’ve changed the text. We like to point out though that the accuracy loss is less than for SparseGPT (and we have an apples-to-apples latency comparison coming in the next day or so).

The speedup is not impressive.

We respectfully disagree! For sparsity/pruning methods, our speedup is pretty good. For example, see the recent LLM-pruner paper. (2305.11627.pdf (arxiv.org) At 20% pruning of LLama 7b, their wikitext ppl is 17.39 at best (up from a baseline of12.62, see their Table 1), and the speedup is less than 15% (see their Table 3). We’re also adding experiments on batch-size (see main rebuttal), where we expect to see much larger throughput increases.

Compared to the quantization-based method, there is no advantage (and Q2)

We agree that quantization is absolutely key for deploying LLMs, and contributes a huge speedup. But pruning and quantization can work together to make LLM deployment cheaper and faster. Most pipelines use pruning and quantization, along with finetuning/continued pre-training. See for example Olive.

There is nothing to prevent a user from applying quantization on top of SliceGPT, and we anticipate that this will give the best speedup. We’d like to emphasize a key point of SliceGPT: that the activations are smaller, leading to less data movement on/between devices, which is something that quantization cannot do. The experiment in our main rebuttal will show this clearly.

it is not fair to multiply the number of GPUs by the total latency

You’re right. If we were to deploy using a more sophisticated deployment that allowed continuous batching, we’d see a speedup of by a factor of up-to the number of GPUs used. Since part of our speedup comes from reduced number of GPUs, the speedup will be less impressive. However, the total energy consumed will still be in proportion to the numbers in our text. We’ve updated the text to reflect this nuance, here’s what we added:

Our HuggingFace-based testing does not enjoy continuous batching, where the next batch can be loaded onto the first GPU whilst the second GPU processes the current batch. This means that in terms of inference time, the dense-model could be improved more than our sliced model in terms of GPUms. Nonetheless, our measurements do reflect the energy-usage per token in such a deployment.

The "computational invariance" trick is similar to a trick in SmoothQuant

You’re right, SmoothQuant also modifies the model, before quantizing. We’ve added a reference to this effect, thanks! Again, we're not competing with SmoothQuant, or any other quantization method. We fully intend for SliceGPT to be used in pipelines where quantization is also applied.

I've read the rebuttal and appreciate the authors' improvements to their experiments. However, they still note key issues like accuracy loss and limited benefits compared to quantization. So, I'll stick with my initial score.

Hi Reviewer 7EAi, thanks so much for your comments. Here are some replies to your queries.

insights of how the rows and columns are sparsified (Q2)

We should have made this clear – we delete the rows / columns that correspond to the smallest eigenvalues. By convention, these are the last rows/columns. Since each deletion (D in equation 9) happens between a pair of blocks, the same number of rows must be deleted from an input matrix as columns from the preceding output matrix. Figure 4 shows the deletions as hatched areas.

Since we use the eigenvalues of the covariance matrix of X as the rotation, after applying Q the network remains invariant (as per thm1) but the variance of each column of X will be equal to the eigenvalues. You can see a plot of some example eigenvalue decays in the supplementary material. If the variance of a column is small, we can assume it is zero always: this is the same as deleting the corresponding column of the preceding weight matrix and the corresponding row of the subsequent weight matrix.

does not provide head-to-head comparison with SparseGPT (Q1)

One of the premises of the paper is memory saving…

which operation/layer is more sensitive to sparsification? (Q3)

We are addressing each of these with a new small experiment, see main rebuttal.

Hi Reviewer 7EAi,

You were concerned that we hadn’t provided a head-to-head comparison with SparseGPT, and that the memory savings over SparseGPT weren't clear. We think our main rebuttal fixes this: we show in experiment (1) that 25% slicing is the same speed as 50% sparsity, before we take improved data movement (and increased possible batch size) into account. Experiment (2) demonstrates the increase in batch size possible with SliceGPT leads to throughput of up to 6.26x that of the dense model, due to memory savings.

We also demonstrate the feasibility of applying varying slicing by layer - with performance improvements in OPT but the opposite in Llama models, in experiment (3).

Should you have any further questions, please let us know. If not, we would be very grateful if you were to consider increasing your score.

We would like to share this experiment to address point (2) in the above: SliceGPT has significant memory benefits since the activations (signal, X) are sliced, as well as the weight matrices.

We ran some tests on 80GB H100 GPUs to compare the token throughput of our models sliced at 25% and 50% compared to the dense model. We set the sequence length to 128, and found the maximum throughput by doubling the batch size until the GPUs ran out of memory or the throughput dropped off. We see that the 25% sliced models achieve up to 1.55x throughput improvement. At 50% slicing we can run the largest models on one GPU instead of two, with large increases in throughput: 3.13x and 1.87x. This means that for a fixed number of GPUs, these models achieve 6.26x and 3.75x throughput of a dense model (last column). We note that the WikiText2 perplexity of SliceGPT at 50% is worse than SparseGPT (yet still not catastrophic!), but the throughput is much higher than could be achieved with a sparse method that does not slice the activations.

For models of size 13B, the performance increase from batch-size increasing is less pronounced because the models take up little of the H100 GPU memory, so we’re already able to get a huge batchsize of 512 with the dense model. On consumer grade GPUs (with less memory) we would expect to see improved throughput for these smaller models too.

Thank you for helping us make this advantage of SliceGPT clearer. We hope this improves your view of SliceGPT: we will be in touch shortly with details on (1) and (3) in the main rebuttal above, as well as a memory/batchsize experiment on smaller devices.

As we discussed above, it is not currently straightforward to run end-to-end evaluations of sparse LLMs. In the SparseGPT paper, those authors provided the relative timing of each operation involved in an OPT transformer layer. This is not a full picture of the relative speedup, yet this is the only way that we can run an apples-to-apples experiment. Above, we demonstrated the advantages of SliceGPT in terms of being able to use bigger batches; here we replicate the experiment in the SparseGPT paper, extended to multiple model sizes, and for Llama-2 architecture.

We used the CuSparseLT 0.5 library to run sparse matmuls on an A100 card, replicating the size of the matrix-matrix multiplications in three different-sized Llama-2 models. We used pytorch to run similar matmuls for the dense equivalent, and for SliceGPT (which is also straightforward dense matmul, but smaller). We chose a sequence length of 2048, and took the matrix sizes from the HuggingFace config files. We took the median runtime over 100 attempts.

Each Llama-2 layer requires a gated FFN with one up projection, one down projection, and a gated projection. In attention, the architecture of the model means that the query matmul is a different size to the key and value matmuls. The following table shows the time taken in ms to run each matmul in the model, plus a “total” time and a relative speedup.

We see that the SparseGPT 2:4 scheme is far from providing a 2x throughput improvement, even though the weights require half the original storage. For the biggest model, SparseGPT 2:4 gives a 1.32x relative speedup, exactly the same as SliceGPT 25%, which has superior perplexity! For the smaller models, the speedup of SparseGPT is a little better than SliceGPT (1.48 vs 1.33). As expected, SliceGPT 50% gives a huge speedup in all cases near 1.9x (though admittedly with worse perplexity).

We also benchmarked the OPT architecture in the same way. In this case, the matmuls associated with Key, Value, Query and out are all the same size, and there are just two matmuls in the MLP section (fc1 and fc2).

For the OPT case, we see that SliceGPT 25% is actually slightly faster than SparseGPT 2:4 for the largest model size. On the smallest models, the results are closer than for the Llama-2 case (1.43x vs 1.31x).

Overall, the speedup of SliceGPT at 25% is competitive with SparseGPT 2:4. We note that this benchmark is very much on “SparseGPT’s terms” in that data movement is not accounted for, which our above experiment suggests is superior in SliceGPT due to the reduced activation (signal) size.

SliceGPT also has the advantage that you can run it right now, end-to-end, using only existing pytorch code 😊

We hope these experiments clarify the situation regarding relative performance of SliceGPT and SparseGPT.

We would like to share this experiment to address point (3) in the above: varying slicing level by layer.

The reviewers suggested inspecting the impact of varying the level of slicing by block; suggesting that some parts of the network might be more sensitive than others. A natural metric to use for deciding on how much to delete from each block is the eigenvalue decay: Instead of specifying the slicing level upfront, we set the fraction of the total variance (=eigenvalues) to discard during each PCA calculation, which sets the number of rows and columns to slice off from each matrix. For each model, we ran three experiments with varying target variances to obtain a total reduction on the network close to 25%.

The results are shown in the table below: we show the perplexity level with uniform slicing; with the new slicing scheme and the improvement. Varying the slicing level by layer improves the WikiText2 perplexity in OPT models, but has the opposite effect in Llama-2 models. The schedules themselves can be seen in the existing supplementary, where we have provided boxplots of the eigenvalues.

There have been a few previous works on designing a sparsity scheme, including OWL. We’ve added our small experiment to the supplementary material, and we hope that a future work can devise a layerwise slicing scheme for SliceGPT that works for more models than just OPT.

We trust this sates the reviewer’s curiosity regarding layerwise slicing 😊.

We look forward to engaging in further fruitful discussions with the reviewers in light of the above experiments.

Hello Reviewers,

Thanks for all your comments that have improved our manuscript. We've uploaded a revised pdf, and a new supplementary material that includes the results of experiments that you suggested.

We hope that we've clarified the relative performance of SliceGPT and our baseline SparseGPT in these experiments. In particular, we emphasize that SliceGPT-25% is as performant or better than SparseGPT 2:4 in terms of latency in many cases.

Best wishes, SliceGPT authors.