QuantEase: Optimization-based Quantization for Large Language Models

We thank the reviewer for their comments. Here are our responses:

1. On the inference speed of outlier-aware QuantEase - Please note that the correct algorithm to compare outlier-aware QuantEase with is SpQR. As we note in Remark 2 of the paper, the setup of dividing weights into a set of quantized weights and a few outliers is standard and similar to the setting studied by SpQR. Therefore, our outlier-aware version enjoys the same speed-ups shown by SpQR using the implementation provided by them.

2. On perplexity vs. zero-shot performance - Please note Figures 1 and 3 in the paper already measure zero-shot performance on the LAMBADA benchmark. From the figures, it is clear that QuantEase outperforms other methods on zero-shot tasks. Note that this is the setup also used by the GPTQ paper and related papers.

3. On experiments conducted for Llama - We tried to get access to Llama weights by asking Meta, but they did not respond to our request. We note that OPT and BLOOM are publicly-available and cover a wide range of model sizes, making them suitable for our comparisons. Moreover, the GPTQ paper also reports results using the same models.

Based on these clarifications, we would like to ask the reviewer to kindly consider increasing their score.

1. At the following url: https://openreview.net/forum?id=Q1u25ahSuy, some reviewers raised questions about the inference speed of SpQR, and those questions seems not to be dealt with yet. For this reason, until now, I cannot expect the speed-up of the outlier-aware version of QuantEase like a normal 3-bit or 4-bit weight quantization approach.

2. Furthermore, the accuracy of QuantEase seems not to be better than existing methods based on experimental results that authors provided. First of all, many LLM papers like Llama papers [1, 2] measures the zero-shot performance of common sense reasoning tasks including BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA. In addition, from my experience, the zero-shot performance of common sense reasoning tasks and the five-shot accuracy of MMLU are weakly correlated to the perplexity of WikiText2. In this sense, measuring the perplexity of WikiText2 and the accuracy of LAMBADA only seems not to be sufficient at all.

[1] LLaMA: Open and Efficient Foundation Language Models

[2] Llama 2: Open Foundation and Fine-Tuned Chat Models

3. Using OPT and BLOOM only is too outdated because there are a couple of ways to use Llama models such as [3] and [4]. I cannot understand why the authors should have access to Llama weights by asking Meta, which appears to make no sense.

[3] https://huggingface.co/huggyllama

[4] https://huggingface.co/meta-llama

Accordingly, I keep my score.

We thank the reviewer for their encouraging and insightful comments. Below, we provide a point-by-point response to your comments.

1. Quantization with grouping - Our claim that grouping can lead to slower inference is based on Table 3 of [3], which shows GPTQ with 128 grouping can be about an order of magnitude slower (in terms of inference latency) compared to ordinary GPTQ. Nevertheless, we had explored the performance of QuantEase with grouping, before deciding to not use grouping due to its overhead. These results are reported below. In these experiments, we use 3-bit quantization with either 128 or 256 grouping. Here, we report perplexity on wikitext2 datasets. The first table below shows the results for the OPT family, and the second one for the BLOOM family.

We note that for the OPT family, except for the case of OPT-2.7b with 128-grouping, QuantEase outperforms GPTQ.

For the BLOOM family, we note that QuantEase outperforms GPTQ in all cases.

2. CD vs. Hessian-based optimization- We use cyclic CD where we optimize one coordinate at a time instead of all variables in one shot. Our single coordinate optimization does involve the hessian. CD is well-known to perform very well compared to gradient/hessian based methods when the optimization along a single coordinate is cheap, which is the case for our problem (see section 3.2 of the paper for the efficiency of our CD updates). In this sense, QuantEase is similar to, and motivated by Liblinear [1] and Improved Glmnet [2] which are classic methods from machine learning where CD works extremely effectively.

3. Why does Coordinate Descent (CD) outperform GPTQ - This is an interesting but difficult question. Here we provide some points that differentiate the operation characteristics of CD-based QuantEase and GPTQ, which helps us to better understand the difference in performance.

• First, as we note in Section 2, CD forms a descent procedure that decreases the objective value (i.e., the quantization error) over iterations, while ensuring the solution remains feasible (i.e., quantized). Specifically, after we complete a pass of CD over all coordinates, we achieve a feasible and quantized solution However, as CD updates preserve feasibility of the solution, we can perform another pass of CD on the coordinates, which by construction of CD, decreases (or at least, does not increase) the objective. This allows us to further decrease the quantization error over multiple passes, which is what we do. On the other hand, we note that GPTQ, by construction, stops after looping once over all columns of weights. GPTQ updates can also break the feasibility of the solution which shows GPTQ is not a descent procedure.
• Second, GPTQ is based on Optimal Brain Surgeon (OBS)[4] and Optimal Brain Compression (OBC)[5] updates that were originally derived for neural network sparsification and pruning, rather than quantization. However, due to the discrete quantization structure, GPTQ uses an approximation to the exact OBS updates that have been derived before (this is discussed in details in Appendix A.1 of the paper). This can partially explain why GPTQ can lose in optimization performance. We note that as a by-product of these OBS-based updates, GPTQ requires the inversion of the Hessian, while our CD-based does not need that.

In the end, we would like to ask the reviewer to kindly revisit their evaluation and consider increasing the score.

[1] C.J. Hsieh et al, A Dual Coordinate Descent Method for Large-scale Linear SVM, ICML 2008.

[2] G.X. Yuan et al, An Improved GLMNET for L1-regularized Logistic Regression, JMLR 2012.

[3] S. Kim et al., (2023). SqueezeLLM: Dense-and-Sparse Quantization.

[4] Hassibi, Babak, David G. Stork, and Gregory J. Wolff. "Optimal brain surgeon and general network pruning." IEEE international conference on neural networks. IEEE, 1993.

[5] Frantar, Elias, and Dan Alistarh. "Optimal brain compression: A framework for accurate post-training quantization and pruning." Advances in Neural Information Processing Systems 35 (2022): 4475-4488.

I thank authors for the rebuttal.

I agree that grouping can introduce extra latency penalties. However, the introduced latency is often smaller on 4-bits rather than 3-bits, due to the instructions needed for decoding 4-bits are much simpler due to its power-of-2 alignment.

However, in the new benchmark authors demonstrated with grouping, the performance gain is negligibly small. Comparing this work to novel low bit work such as "QuIP: 2-Bit Quantization of Large Language Models With Guarantees", I decided to lower my rating from 6 to 3.

Thank you for reviewing our paper and your encouraging comments. Below, we provide responses to the two points you raised.

• AWQ/GPTQ memory issues - As mentioned in the paper, both AWQ and GPTQ run out of memory on V100 GPUs when quantizing the OPT-66b model, owing to a limited memory of 32GB. Based on feedback from the reviewer, we re-ran the experiments on OPT-66b using A100 GPUs (we have limited availability of these GPUs, which made running these experiments difficult). Following are the results. As we can see, QuantEase outperforms both GPTQ and AWQ for this model.

• Comparison to QuIP and 2-bit quantization - Thanks for bringing our attention to this interesting paper. It seems the best results in [1] are achieved when incoherence processing is used, for example, in Table 1 of [1], the best performance is always achieved by an algorithm that uses incoherence processing. This is especially true for 2-bit quantization, where non-incoherence methods seem to perform poorly. However, incoherence processing needs projection steps---see line 3 of Algorithm 2 of [1] where quantized weights $W$ are rescaled using two orthogonal matrices $U,V$. As authors note in Section 4 of [1], producing and applying such projections requires a delicate bookkeeping of $U,V$ and possible computational overheads in the inference time. As far as we could tell, the runtime implications of such overheads have not been explored numerically. This makes the practical applicability of incoherence processing for 2-bit quantization less clear. On the other hand, QuantEase performs standard quantization, which is commonly used in practice. However, we note that it seems incoherence processing can be treated as a module and integrated to any existing quantization method, as it is applied to GPTQ in [1]. As we see in [1], incoherence processing helps with boosting the performance of GPTQ, and as we show here, QuantEase can perform better than GPTQ. Therefore, we believe adding incoherence projection to QuantEase can lead to better quantization accuracy, although such a study is out of the scope of our paper.

Based on our response here, we would like to kindly ask the reviewer to revisit their score. We will be happy to include snippets of this discussion in the final version of the paper, if the paper were to be accepted.

[1]: QuIP: 2-Bit Quantization of Large Language Models With Guarantees