CDQuant: Accurate Post-training Weight Quantization of LLMs using Greedy Coordinate Descent

We thank the reviewer for their feedback. Below, we address some of the key concerns raised.

Comparisons to GPTQ: As pointed out in our paper, the arbitrary order used by GPTQ usually leads to drop in quality, especially in extreme bit quantization. Our work tries to address this drawback of GPTQ via efficient greedy coordinate descent strategies. As pointed out by the reviewer, while OBQ also performs CD, its update rule is drastically different from ours and is very inefficient. This is because OBQ involves updating all the un-quantized coordinates in each iteration, which is very inefficient on accelerators such as GPUs, TPUs. As a result of this, the OBQ paper only managed to show experiments on ResNet models with a few million parameters. Our work addresses this drawback by providing efficient coordinate descent strategies which can scale to models with 100B parameters. Our algorithm is simple and only involves updating a single coordinate in each step.

Computational cost: We would like to first note that on a multiple GPU setting, GPTQ and the CD variant of CDQuant have comparable runtimes. For FFN1 (FFN2) quantization, CD is 5x (2x) slower than GPTQ on 8 H100 GPUs. For the single GPU setting, the gap between their runtimes is more pronounced. However, as shown in Table 22 in Appendix F.4, CDQuant, even with 1/8th of the iterations, achieves better perplexity than GPTQ. Also, in Figure 1 in Appendix F.4 (please look at the updated paper), CD’s L2 activation reconstruction error converges in roughly 1/8th of the iterations across several settings. This reduction in iterations makes the runtime of CDQuant comparable to GPTQ. Moreover, in Table 21, in Appendix F.4, we show that replacing BCD with CD for FFN2 quantization does not lead to a significant performance drop. Since quantizing FFN2 is a bottleneck for BCD, this substitution significantly speeds it up. With the above two modifications, BCD can be considerably sped up, and CD could even run faster than GPTQ. As future work, we will work on writing kernels to implement the gathers and scatters in our algorithm efficiently.

Gains on model sizes: We would like to highlight that our technique indeed shows significant gains on extreme quantization of large models (INT2 INT3 quantization). For instance, for INT2 quantization of PaLM2-Otter, we see a 10% improvement over GPTQ. Similarly, for INT2 quantization of Gemma2-27B and PaLM2–Bison, we see 3-5% gains.

Furthermore, as we showed in our paper, CDQuant acts as an excellent replacement for GPTQ in algorithms that use GPTQ as a subroutine (see Table 3). For instance, replacing GPTQ with CDQuant in QuIP, FrameQuant and AWQ showed 3-5% performance gains even on larger models such as Gemma2-27B. We would like to note that these results are highly non-trivial as (a) we are working with SOTA algorithms, and (b) providing a simple general purpose, plug-and-play approach to boost any SOTA technique that relies on GPTQ.

Experiments on Llama: Unfortunately, due to certain organizational restrictions, we cannot use Llama models for research as that would violate the terms of use. To compensate for that, we run experiments with the PaLM2, Gemma-1 and Gemma-2 family of models. In Riviere et al., 2024 [1], Gemma-2 27B has been shown to have comparable performance to Llama-3 70B. Furthermore, PaLM2 family models are production quality models which are usually very hard to compress. We thus believe PaLM2, Gemma-1 and Gemma-2 to be reasonable replacements for Llama-2 and Llama-2

OWC initialization: We would like to highlight that, for a fair comparison, we used OWC initialization for both GPTQ and CDQuant. Thus, the performance gains observed in our experiments are due to the proposed coordinate descent algorithms.

Using 1280 data samples instead of 128: We would like to note that our method works perfectly fine with 128 samples. We used 1280 samples for both GPTQ and CDQuant as it increased the Hessian approximation and improved the quantization performance for both CDQuant and GPTQ (this is especially the case with larger models with large FFN dimensions). CDQuant and GPTQ both leverage second-order statistics to quantize the LLMs and thus benefit from more examples. Furthermore, using 1280 data samples doesn’t add any significant computational overhead over using 128 samples and we do not see a strong reason for restricting ourselves to 128 samples. Ultimately, we care more about quality and thus went with 1280 samples in our work.

[1] Riviere et al., 2024, Gemma 2: Improving Open Language Models at a Practical Size.

Thanks for the authors' response.

Given the novelty of the method, which extends GPTQ by incorporating greedy search, and the limited improvement in accuracy, I will keep the original score.

"But we are unsure why this should make our technique less novel compared to GPTQ."

GPTQ, introduced two years ago, demonstrated the ability to quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth of weights to as low as 3 or 4 bits with negligible accuracy degradation compared to the uncompressed baseline. This breakthrough significantly advanced the quantization accuracy and efficiency of LLMs, enabling—for the first time—the execution of a 175 billion-parameter model on a single GPU for generative inference.

In comparison, I do not observe a comparable level of improvement and contribution from CDQuant.

As shown in Table 7 and Table 8, under the configurations of w3a16g128 and w4a16g128, the accuracy of GPTQ, CD, and BCD is comparable.

"As shown in Table 7 and Table 8, under the configurations of w3a16g128 and w4a16g128, the accuracy of GPTQ, CD, and BCD is comparable."

In Tables 7 and 8, the headroom for improvement was very little, because bf16 has similar performance as w4a16g128. For instance, for Gemma-2 27B, the avg bf16 accuracy is 67.55 and the avg accuracy with w4a16g128 is 67.45. So it's impossible to expect any technique to show improvements in such settings.

That being said, for w3a16g128, we do show some improvements. For instance, for Gemma-2 27B, GPTQ has 66.4 avg accuracy, our technique has 66.87 and topline bf16 is 67.55. So, our technique bridges the gap between GPTQ and baseline. We believe, the results should be interpreted with topline in consideration.

Finally, we would like to note that for settings where topline is significantly better than GPTQ, we do see significant gains. For instance, for Gemma-1 7B quantization using w3a16, GPTQ has avg 40.87 accuracy, CD has 50.51, and topline bf16 has 58.35. Similarly, for PaLM2-Gecko quantization using w3a16, GPTQ has avg 39.39 accuracy, CD has 41.12, and topline bf16 has 43.84 accuracy.

Please also take a look at our INT2 quantization results where we have significant gains over GPTQ.

We also request the reviewer to take a look at some of the recent papers on PTQ techniques. For instance, Table 1 in AffineQuant paper (https://openreview.net/pdf?id=of2rhALq8l). The improvements over baselines diminishes as we increase the weight precision. This is expected because the gap between bf16 and baseline INT4 performance diminishes. The real value of these techniques shows up at extreme quantization (INT2, iNT3) and/or smaller models.

Similarly, please look at Table 1 in the SqueezeLLM paper: https://arxiv.org/pdf/2306.07629, and Table 3 in MagR paper: https://arxiv.org/pdf/2406.00800.

The authors highlight that CDQuant is intended as a replacement for GPTQ. While CDQuant demonstrates certain performance improvements over GPTQ in the w2 and w3 setting, the results have a considerable gap from floating-point accuracy and practical requirements.

On the other hand, as they pointed out, under the w4a16g128 configuration (where CDQuant offers no advantage) , GPTQ achieves results very close to floating-point accuracy and satisfies the requirements of practical applications.

I have always believed that there is room for improvement in GPTQ, and further advancements are necessary—such as addressing the model quantization generalization issues caused by the Hessian matrix. For this reason, I am always supportive of related research efforts.

However, CDQuant provides limited theoretical innovation, particularly in terms of its integration of GPTQ and Greedy Search, and it falls short of fully addressing the core challenges associated with GPTQ. Moreover, the greedy search approach undermines its quantization efficiency. Overall, CDQuant fails to demonstrate sufficient potential to replace GPTQ.

So the score (5) is appropriate, and I will maintain it.

We thank the reviewer for their feedback. Below, we address some of the key concerns raised.

Computational cost: We would like to first note that on a multiple GPU setting, GPTQ and the CD variant of CDQuant have comparable runtimes. For FFN1 (FFN2) quantization, CD is 5x (2x) slower than GPTQ on 8 H100 GPUs. For the single GPU setting, the gap between their runtimes is more pronounced. However, as shown in Table 22 in Appendix F.4, CDQuant, even with 1/8th of the iterations, achieves better perplexity than GPTQ. Also, in Figure 1 in Appendix F.4 (please look at the updated paper), CD’s L2 activation reconstruction error converges in roughly 1/8th of the iterations across several settings. This reduction in iterations makes the runtime of CDQuant comparable to GPTQ. Moreover, in Table 21, in Appendix F.4, we show that replacing BCD with CD for FFN2 quantization does not lead to a significant performance drop. Since quantizing FFN2 is a bottleneck for BCD, this substitution significantly speeds it up. With the above two modifications, BCD can be considerably sped up, and CD could even run faster than GPTQ. As future work, we will work on writing kernels to implement the gathers and scatters in our algorithm efficiently.

Experiments on Llama: Unfortunately, due to certain organizational restrictions, we cannot use Llama models for research as that would violate the terms of use. To compensate for that, we run experiments with the PaLM2, Gemma-1 and Gemma-2 family of models. In Riviere et al., 2024 [1], Gemma-2 27B has been shown to have comparable performance to Llama-3 70B. Furthermore, PaLM2 family models are production quality models which are usually very hard to compress. We thus believe PaLM2, Gemma-1 and Gemma-2 to be reasonable replacements for Llama-2 and Llama-2

Convergence guarantees: We would like to note that the least squares minimization problem that we are solving at each layer is NP hard (see line 156 in our paper). This is a combinatorial optimization problem, and takes exponential (in dimension) time to converge to global optimum. However, one can easily show that our algorithm converges to a special form of local optimum where modifying a single coordinate doesn’t reduce the loss value. The same can not be said of GPTQ which cycles through each coordinate only once. This is also the reason why we see an improvement in performance over GPTQ.

Performance on models with trillions of parameters: Unfortunately, we do not have access to open source models with trillions of parameters or resources to run evals on them (getting perplexity evals and few shot evals requires a lot of resources). That being said, we believe similar performance trends observed in our paper will hold even for models with trillions of parameters. We base this claim on two main reasons: (1) SOTA trillion parameter models are MoE models, with each expert roughly the size of the largest model experimented in our paper, (2) the core problem we are solving (quantized least squares problem) remains the same across models. Consequently, we believe the gains we see here translate to bigger models.

Comparison between AWQ and OWC: Thanks for the suggestion. Below we compare OWC and OWC-CD (the two initialization strategies introduced in our paper) and find them to outperform AWQ. The following table shows the perplexity numbers for Gemma-2 9B quantization using AWQ, OWC, OWC-CD. As can be seen, OWC, OWC-CD provide better initializations than AWQ (OWC-CD has blanks because it is designed only for sub-channel quantization).

[1] Riviere et al., 2024, Gemma 2: Improving Open Language Models at a Practical Size.

We thank the reviewer for their feedback. Below, we address some of the key concerns raised.

Computational cost: We would like to first note that on a multiple GPU setting, GPTQ and the CD variant of CDQuant have comparable runtimes. For FFN1 (FFN2) quantization, CD is 5x (2x) slower than GPTQ on 8 H100 GPUs. For the single GPU setting, the gap between their runtimes is more pronounced. However, as shown in Table 22 in Appendix F.4, CDQuant, even with 1/8th of the iterations, achieves better perplexity than GPTQ. Also, in Figure 1 in Appendix F.4 (please look at the updated paper), CD’s L2 activation reconstruction error converges in roughly 1/8th of the iterations across several settings. This reduction in iterations makes the runtime of CDQuant comparable to GPTQ. Moreover, in Table 21, in Appendix F.4, we show that replacing BCD with CD for FFN2 quantization does not lead to a significant performance drop. Since quantizing FFN2 is a bottleneck for BCD, this substitution significantly speeds it up. With the above two modifications, BCD can be considerably sped up, and CD could even run faster than GPTQ. As future work, we will work on writing kernels to implement the gathers and scatters in our algorithm efficiently.

Improvements from CDQuant: While we agree with the reviewer that improvements are less pronounced compared to GPTQ for W4A16, the gains are significant for extreme quantization (2 or 3 bit quantization; 5-10% gains in a number of settings). These extreme quantization settings have received significant attention of late and are crucial for widespread deployment of LLMs. We believe our technique can provide value in such settings. Additionally, as demonstrated in the paper, CDQuant can be used as a plug-and-play replacement for GPTQ in numerous state-of-the-art PTQ methods like QuIP, FrameQuant, QuaRot, and AWQ, and provides an easy way to boost their performance. We believe this last property makes our technique especially valuable in practice.

Using CDQuant on layers with extreme outliers: In the presence of extreme outliers, we observed Hessian eigenvalue clipping helped us effectively handle the outliers (it helped boost GPTQ performance as well). The Hessian eigenvalue clipping is reminiscent of rescaling done in AWQ. This is also similar in spirit to methods like QuIP and FrameQuant that use rotation to handle extreme outliers. SqueezeLLM uses non-uniform quantization and a sparse-and-dense quantization scheme, both orthogonal to CDQuant. We expect CDQuant to benefit from these methods as well.

Applying CDQuant to QAT: This is an interesting suggestion that we are planning to explore in the future. We believe coordinate descent based techniques could be a very good alternative to Straight Through Estimation (STE). There are various ways in which one could use CD for QAT. For instance, after each step of SGD (in full precision), one may take the full precision weights and use our technique to obtain quantized weights. On a related note, we would like to highlight that a recent NeurIPS 2024 oral paper by Malinovskii et al., [1] explores the use of greedy coordinate descent as an alternative to STE for vector quantization.

We hope we addressed the reviewer’s concerns, and are happy to answer any further questions. We would really appreciate it if the reviewer reevaluates our work.

[1] Malinovskii et al., 2024, PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression.