SqueezeLLM: Dense and Sparse Quantization

W1. References

First, here are some missing references. Regarding sparse encoding of the outliers, [1] propose an approach that bears some similarities and would be worth mentioning. Similarly, regarding the measurement of the importance using the Hessian matrix, pruning techniques have previously used similar estimates [2,3] and in particular [4]. I think the paper would benefit from these elements. Second, I think the results are great, so there is no need to not highlight other methods when they perform on par with the proposed SqueezeLLM.

We appreciate the reviewer’s valuable feedback and the key references. We will include these references and comparisons in the final version of our paper to provide a more comprehensive context.

---

Q1. Ideas on Fine-tuning

I wonder how would the authors fine-tune such compressed model without losing the benefits of the compression technique, i.e. how to fold a LoRA or else.

This is indeed an interesting question given that finetuning LLMs is emerging as an interesting research field. We envision that SqueezeLLM could be effectively employed as a backbone quantization method in parameter-efficient finetuning scenarios similar to QLoRA [1] – where the main model would be quantized for the forward pass, and only the smaller adapter modules would undergo weight updates – without losing the benefits of our compression technique. While it's common in practice [1, 2] to retain the LoRA module after fine-tuning, how to fold them will be an interesting avenue for future research.

[1] Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." arXiv preprint arXiv:2305.14314 (2023).

[2] Chai, Yuji, et al. "INT2. 1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation." arXiv preprint arXiv:2306.08162 (2023).

We appreciate the reviewer's insightful comments and the references provided. We have carefully reviewed the mentioned works and emphasize our novelty as follows. We will include this comparison in the paper to acknowledge the prior works and clarify the novelty of our approach.

---

W1. Comparisons with Other Clustering Methods

Regarding the two approaches, there needs to be more discussion on prior works. K-means clustering for Quantization is a popular technique used in signal processing. For neural networks, works such as DeepCompression [Han et al., ICLR 2016], SLQ/MLQ [Xu et al., AAAI 2018], HPTQ [Xu et al., IEEE Signal Processing Letters 2023] employ k-means clustering for quantization. It would be great to acknowledge such and similar works in literature and compare them to justify the novelty claimed in Sec 4.1. Of course, LLMs as a target application is new, but the technique may need to be more novel in its current presentation.

That is a fair question. About the comparison with prior k-means methods, we kindly emphasize that our approach is different. In particular, SqueezeLLM incorporates a sensitivity-aware weighted k-means clustering method combined with the dense-and-sparse decomposition, which allows for effective low-bit LLM quantization.

In particular, our sensitivity-aware weighted clustering is different from the non-weighted (i.e. sensitivity-agnostic) clustering approaches in the prior works and allows minimal performance degradation without retraining. As discussed in Section 4.1, our method ensures minimal loss degradation under widely accepted assumptions, and therefore significantly reduces performance degradation in contrast to the sensitivity-agnostic approach. This is demonstrated in Figure 1, with a considerable improvement from 18.09 (sensitivity-agnostic) to 7.75 (sensitivity-aware). Furthermore, such a precise allocation of LUT entries through the sensitivity-aware approach is especially critical for LLM quantization, where retraining or finetuning can be highly restricted. This is in contrast with methodologies like DeepCompression or SLQ/MLQ, which performs post-clustering retraining to recover performance.

---

W2. Comparisons with Other Decomposition Methods

A similar treatment is observed in Dense and Sparse Decomposition (Sec 4.2), where similar techniques exist in the literature, not necessarily for LLMs. The author must acknowledge and discuss the prior works, such as DSD [Han et al., ICLR 2017], Scatterbrain [Chen et al., NeurIPS 2021], Vitality [Dass et al., HPCA 2023], etc to bring out the novelty.

DSD is a significant paper that introduces a novel way of training neural networks where the term "sparse" represents post-pruned weights, and "dense" denotes weights that have been recovered post-pruning. However, in our approach, “sparse” refers to isolated outlier weights to enhance post-training quantization performance. Furthermore, DSD operates as a training-time technique, which trains the post-pruned (so-called sparse) weights and the post-recovery (so-called dense) components separately before merging them into a single matrix, whereas our method is tailored for inference, with a major focus on enhancing inference-time efficiency and latency. While DSD and ours use the same terms of “dense” and “sparse”, their implications are dissimilar.

With regards to Scatterbrain and Vitality, our method proposes a direct solution to tackle the outlier issue in LLMs, which has been a major obstacle for low-bit LLM quantization. In contrast, both Scatterbrain and Vitality introduce sparsity to represent attention maps as a combination of large attention values and the low-rank approximation of the residual matrix. Additionally, our approach further incorporates “sensitive” values within the sparse matrix, which significantly mitigates distortion in non-uniform quantization LUT entries, leading to a considerable improvement in post-quantization performance (7.67 vs 7.56 in Figure 1).

W3. Novelty and Comparison with SmoothQuant

Overall, it seems the LLMs provide opportunities to employ a novel combination of existing techniques. If so, the current presentation of Sec 4 does not paint an accurate picture and needs to be rewritten to acknowledge similar works and present the novelty. In addition, discussing SmoothQuant [Xiao et al., arXiv 2022, ICML 2023] in the context of post-training quantization for LLMs might be worthwhile.

LLMs indeed present unique behavior and challenges compared to previous models, necessitating novel approaches to enable effective low-bit quantization. SmoothQuant, as the reviewer mentioned, exemplifies one such novel approach that offers a quantization method that tackles new challenges in LLMs and facilitates more accurate quantization. We would also like to kindly emphasize that SqueezeLLM has introduced two novel methods, the sensitivity-aware weighted k-means clustering method combined with the dense-and-sparse decomposition, to tackle the unique challenges in effective low-bit LLM quantization.

We acknowledge the reviewer’s suggestion, and we will include SmoothQuant in the paper. At the same time, we kindly emphasize that our approach is fundamentally different from SmoothQuant. While SmoothQuant is a quantization framework for both weights and activations with a specific focus on mitigating the activation outliers by reweighting the associated weight channels, SqueezeLLM is a weight-only quantization method that focuses on weight outliers.

We appreciate the reviewer's insightful comments, and here we address your comments in detail:

W1. Kernel Concerns and Evaluation on A100

Implementation and Kernel Concerns: I have reservations about the kernel implementation. Introducing such a custom kernel, as described in your method, appears to disrupt the established high-performance computing framework. Although an unstructured or non-uniform structure might enhance model performance, it could also negatively impact acceleration performance or complicate it. Hence, a more detailed exposition about the kernel is essential to substantiate the novelty of your method. While your results indicate an improved model performance, the absence of a dedicated kernel might undermine its effectiveness. Additionally, it's worth pondering why there was no effort to implement and optimize the kernel specifically for the A100. Even though the A100/H100 might be a costly choice, they can serve as a foundational hardware for large language models, particularly in tandem with NVIDIA's inference software. It's noteworthy that the memory bandwidth for the RTX series is below 1TB/s.

This is a fair point. We had originally focused on lower-grade GPUs partly due to the need for enabling inference on more affordable GPUs, and partially due to our limited access to A100 systems for our experiments. However, in light of the reviewer’s insightful comments, we have conducted additional experiments using our kernels on an A100 system to run the linear layers of LLaMA model for generating a sequence length of 128. As demonstrated in the table below, our kernel implementation still attains 1.5-2.5x performance speedups relative to the fp16 matrix-vector multiply kernel across different model sizes, similar to what we have observed with our previous experiments. This is despite any additional optimizations or tuning.

Therefore, our evaluation results indicate that SqueezeLLM does not introduce any negative impact on acceleration performance due to our unstructured or non-uniform structure. Furthermore, we would like to highlight that the implementation of SqueezeLLM does not introduce complexity. The core of SqueezeLLM is non-uniform quantization which only involves look-up-table operations, which can be implemented as simply and efficiently as those in uniform quantization.

Matrix-Vector Multiply Kernel Runtime for Generating 128 Tokens

---

W2. Feedback on Model Performance

Feedback on Model Performance: The majority of the model performance results focus on PPL outcomes. I believe it would be beneficial to include MMLU or CSR results for larger models within the main content, rather than relegating it to the appendix.

Thank you for your constructive feedback. We acknowledge your point regarding the inclusion of MMLU and CSR results for larger models in the main body of the content. As can be seen in Table A.4 and A.7, our method consistently outperforms other methods both in terms of perplexity as well as MMLU accuracy on larger models, showing the strong transferability of the results in the smaller models to the larger models. The decision to place these results in the appendix was due to the page constraints, and we will include it in the main content in the final version. Reviewer 1 also asked us to perform 5-shot MMLU results which are included in response to their question (please see W3 of Reviewer U5Sb)

W3. 5-shot MMLU Performance

It would be beneficial to display five-shot performance results on the MMLU benchmark using LLaMA, as this would offer a more comprehensive comparison with other methodologies.

That is an excellent question. We have included five-shot performance results on the MMLU benchmark for Vicuna-v1.1 in the table below and also included a comparison with AWQ with the same model size. Similar to the zero-shot results that we had included in the paper, we can see that SqueezeLLM consistently achieves higher-quality results. At the moment, we are also running a comparison with GPTQ but we expect GPTQ to have a similar performance as AWQ. We will include these results in the final version of the paper beside Table 2 where we compared instruction following capabilities of the different models.

MMLU 5-shot Accuracy

---

Q1. Outliers vs. Sensitive Values

Is there a relationship between outliers and sensitive weights? Specifically, are most of the sensitive weights outliers?

From our extensive experiments conducted on Vicuna, LLaMA, and LLaMA2 models, we found no direct correlation between outliers and sensitive weights. In fact, we frequently observed that weights with smaller values can also be sensitive (as in Figure 2, Left). This is expected since the sensitivity measured by the Fisher information is not necessarily correlated with the magnitude of the weight value, but more with the curvature of the loss function with respect to that weight value.

---

Q2. Uniform Quantization + Dense-and-Sparse Decomposition

Have you experimented with combining a uniform quantization scheme with the sparse decomposition concept?

Yes, we have indeed explored integrating uniform quantization with the dense-and-sparse decomposition while developing SqueezeLLM. Specifically, we applied dense-and-sparse decomposition in combination with channel-wise uniform quantization on LLaMA 7B and 13B models. In the table below, we compared the performance of uniform quantization and our sensitivity-aware non-uniform quantization, both paired with the dense-and-sparse decomposition. The results indicate that non-uniform quantization based on our sensitivity-aware K-means method yields significantly better perplexity. It is worth noting that the performance of uniform quantization could potentially be improved through grouped quantization strategies, where a small set of weights (e.g. 128 as in the table below) are grouped together to have their own scaling factor. While grouped quantization, as demonstrated in techniques like GPTQ and AWQ, is a viable approach, the SqueezeLLM approach offers a much simpler implementation by avoiding the complexities of fine-grained grouping and it still outperforms uniform quantization methods with grouping.

Perplexity on C4

---

Q3. Comparison with AWQ kernels

In Table 1, there's a comparison with AWQ's latency performance. However, such a comparison is missing in Table 3 (section 5.4). Since the AWQ kernel is well showcased, it would be more convincing to show a comparison of the proposed kernel with AWQ.

We appreciate the suggestion to include a comparison with AWQ's public kernel. However, as we outlined in Appendix A.3, we chose not to compare our kernels directly with those from AWQ for a couple of key reasons: (1) The 3-bit kernels of AWQ are not publicly available; and (2) AWQ has incorporated optimizations that are unrelated to quantization, such as LayerNorm and positional embedding. These optimizations, while effective, are universally applicable to various quantization methods including SqueezeLLM and GPTQ.

We appreciate the reviewer's constructive feedback, and here we address your comments in detail:

W1. Resource Requirements

SqueezeLLM uses a small sample of the training dataset to execute end-to-end forward and backward passes for gradient computations. This process appears to be more resource-intensive than other methods like RTN, GPTQ, or AWQ. It would be beneficial to understand the time and resources required for SqueezeLLM's dense-and-sparse quantization.

That is correct and SqueezeLLM involves an end-to-end forward and backward pass over a small set of sampled data. This method differs from other methods such as RTN, which does not require any forward or backward pass, and GPTQ or AWQ, which requires isolated gradient calculation over individual layers, resulting in lower overhead. However, it is important to note that the overhead of SqueezeLLM is quite manageable and, crucially, our method consistently outperforms RTN, GPTQ, and AWQ as shown in the paper (Table 1). To provide a clearer understanding of SqueezeLLM’s resource requirements, we conducted a comprehensive analysis of the time and memory requirements for (1) Fisher information computation and (2) sensitivity-aware k-means clustering for nonuniform quantization as below.

1. Fisher Information Computation:

We assessed the end-to-end latency and peak memory usage for computing Fisher information on an A100 system across different LLaMA models. The results indicate that the latency for calculating Fisher information (i.e. the sum of squares of gradients) is pretty fast and quantizing the 65B model can be performed on 4 A100 GPUs to address the peak memory usage.

2. Sensitivity Aware K-means Clustering:

Another key component of SqueezeLLM is the sensitivity-aware K-means clustering applied to each weight matrix. This process can be executed efficiently in parallel across different layers, possibly using multiple cloud CPU instances. Our latency measurements on 8 Intel Xeon Platinum 8380 systems show that the time overhead for this clustering process is quite manageable. We will include these data in the final version of the paper to offer a comprehensive view of SqueezeLLM's time and resource demands.

---

W2. Comparison with FlexRound

Relatedly, in the discussion about the need to minimize overall perturbations for the final loss term in section 4.1's "Sensitivity-Based K-means Clustering," the paper should also compare its performance with methods like AdaRound [1] or FlexRound [2]. These methods utilize a small calibration set and employ layer-wise or block-wise post-training quantization techniques. Notably, since FlexRound reports results on LLaMA using uniform weight-only quantization, it would strengthen the paper's claim about optimization with the final loss.

That is a great question. Below in the table, we provide a comparison with AdaRound and FlexRound. We first have to kindly emphasize that it is unfortunately not feasible to do a direct comparison with FlexRound due to their closed-source code and models, as well as a lack of experimental details for their perplexity evaluation on WikiText. However, to address the reviewer’s question to the best of our ability, we have utilized the results published in the FlexRound paper (for AdaRound and FlexRound) to assess the amount of post-quantization perplexity degradation of each method across different LLaMA models. As one can see, SqueezeLLM consistently exhibits a smaller drop in perplexity compared to AdaRound and FlexRound.

We should also kindly emphasize that we have performed a comprehensive comparison with the latest weight-only quantization schemes designed for LLMs, such as AWQ and SpQR, which share the same goal of minimizing output activation perturbation as FlexRound. In these comparisons, our method consistently demonstrates superior performance, both with 4-bit and 3-bit. Therefore, we maintain that our sensitivity-based k-means clustering that optimizes the final loss provides a more effective post-training quantization solution compared to other existing quantization strategies.

Perplexity on Wikitext

Dear Author,
Thank you for responding to my previous concerns.
I have an additional question regarding Equation (4) in your paper. It appears that Equation (4) requires a more explicit derivation. The interpretation of Equation (4) as the minimization problem involving the sum of the Fisher Information for the weight difference between the original and quantized weights is somewhat unclear to me. Specifically, I am struggling to understand the connection between this concept and “minimizing the overall perturbation with respect to the final loss term.” which is the term on the paper. To the best of my knowledge, as demonstrated in the BRECQ paper [1], the derivation of Equation (3) should conclude with an expression similar to:

$argmin$ $\sum_{i=1}^{N}$ $\mathscr{F}$ $(y-\hat{y})^2$

where y is an original output of each layer, $\hat{y}$ is an output of the quantized layer. For additional context, I refer you to Appendix A.1, “Proof of Theorem 3.1,” in the BRECQ paper on page 12 [1].

[1] https://openreview.net/pdf?id=POWv6hDd9XH

I understand Eq. (2) in your paper that is the same as Eq. (4) in the BRECQ paper. In particular, it is worth noting that $H$ in Eq. (2) in your paper is exactly the same as $\bar{\text{H}}^{(w)}$ in Eq. (4) in the BRECQ paper.

In the BRECQ paper, $\Delta w \bar{\text{H}}^{(w)} \Delta w$ is approximated to $\Delta z \bar{\text{H}}^{(z)} \Delta z$ by using Theorem 3.1 in the BRECQ paper, where $\bar{\text{H}}^{(z)}$ is the output Hessian that is totally different from $\bar{\text{H}}^{(w)}$. Then, the output Hessian $\bar{\text{H}}^{(z)}$ is approximated to the Fisher Information Matrix $\mathcal{F}$ as shown in Eq. (10) in the BRECQ paper.

On the contrary, in your paper, $H$ in both Eq. (2) and Eq. (3) is essentially $\bar{\text{H}}^{(w)}$, but then $\bar{\text{H}}^{(w)}$ is approximated to the Fisher Information Matrix $\mathcal{F}$ in Eq. (4) without any explanation. So, I would like to point out how the authors can derive the Fisher Information Matrix $\mathcal{F}$ from $\bar{\text{H}}^{(w)}$.

The authors mentiond [1], but to the best of my knowledge, like the BRECQ paper, [1] also handled the pre-activation Hessian $\partial^2E / \partial a_i^2$ as seen in Eq. (7) and (8) in [1].

[1] Optimal brain damage. Yann LeCun, John Denker, and Sara Solla. NeurIPS 1989.