GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration

Thanks for your comments and positive feedback. Please check our response below.

1. Bitwidth settings are limited to W4A4/W2A4. Considering the need for near-lossless quantization of LLMs/ViTs in some application scenarios, W6A6/W8A8 results would be a plus.

Thank you for this suggestion. We have conducted additional experiments with W6A6 bitwidth on LLaMA2-7B and LLaMA3-8B to evaluate GPTQv2 in near-lossless quantization scenarios. The results are presented below (reporting WikiText-2 perplexity):

As expected, the improvements in the W6A6 setting are more modest compared to those in lower bitwidth scenarios, since higher precision quantization already preserves most of the model's capabilities, leaving less room for enhancement. Nevertheless, GPTQv2 still consistently outperforms GPTQ across both models.

2. Missing baselines for ViT quantization. Rotation-based methods, such as QuIP, can also be applied in ViT quantization, and FrameQuant[1] is omitted.

Thanks for letting us know about the FrameQuant paper. As far as we can tell, QuIP and FrameQuant are performing weight-only quantization on ViTs, which would be less effective than LLM decoding, as the ViT inference is compute-bounded instead of I/O-bounded.

Nevertheless, we test DeiT-S with 2-bit per-channel quantization with or without QuIP incoherence processing. The results are shown below, with * indicating our implementation.

3. Figure 4 (a) seems to be missing bars for latency comparison. The latency overhead of GPTQv2 compared to GPTQ is non-negligible, especially on higher dimensions.

Sorry for the missing histograms. This appears to be a browser-specific rendering problem. If you download our paper, the histograms should be visible when viewing the PDF in Chrome or in standard PDF readers. We will ensure the figure is displayed correctly in all formats in the revised version of the paper.

For the GPTQv2 vs GPTQ latency comparison, we can give a theoretical upper bound here. According to Algorithm 1, the operations of v2 are at most 2x of v1. With sufficiently high dimensions, we can expect the latency may approach 2x. However, in practice, we observe 30%-50% more time in typical LLMs (7B to 405B).

4. For SpinQuant, I think it should be categorized as FT-free since it only involves optimizing the rotation but not updating the weights. In addition, the author of SpinQuant mentioned that optimizing rotation is highly efficient, 0.5h for L3-8B in their paper. It could be the difference in hardware system, please double check it.

We maintain that SpinQuant should be categorized as finetuning-based rather than finetuning-free for the following reasons:

• From a computational perspective, optimizing rotation matrices or weights involves the same core operations - both require backpropagation through the network, use of the Straight-Through Estimator (STE), and gradient-based optimization. The fact that the optimization target is a rotation matrix rather than weight values does not reduce the computational requirements.

• Conceptually, SpinQuant modifies the effective weight representation through rotation optimization. Whether directly updating weights or optimizing transformations applied to weights, both approaches adjust the model's parameters that will be quantized.

Regarding the finetuning time, it’s due to the SpinQuant authors using 8 A100 GPUs to finetune the rotation matrix. Therefore, in Section 5.3, we explained that “we additionally report the GPU Hours (on one A100) required to run the algorithm. Although in practice, SpinQuants runs on 8 A100 GPUs ”

5. It would be interesting to see if the proposed GPTQv2 could be applied to Diffusion Transformers such as Q-DiT [2].

Thanks for the reference on transformer-based diffusion models. We noticed that Q-DiT pointed out an important observation that activation distribution undergoes continuous changes across timesteps. In this case, the activation asymmetry may accumulate not just through layers but through time steps as well. We expect GPTQv2 will have better performance if $\Delta \mathbf{X}$ can capture more information as we did in the experiments of quantization order (Appendix B.1). We will discuss this problem of diffusion models in our paper related work.

Thank you very much for your comments and thorough review. Please check our response to your questions.

1. What does the lambda in eq.10 represent for? Is it a hyper-parameter? If so, then why there is a gradient on it?

$\lambda$ is the Langrange multiplier, which is not a hyperparameter. By taking derivatives of the Lagrangian with respect to both $\Delta\mathbf{w}$ and $\lambda$ and setting them to zero, we simultaneously: (1) ensure the quantization constraint is satisfied exactly ($\partial L/\partial \lambda=0$ enforces $\Delta\mathbf{we}_q^\top+\mathbf{w}_q-\hat{\mathbf{w}}_q=0$), and (2) find the local minimum of the objective function. This approach follows standard constrained optimization techniques used in the original OBS and OBQ frameworks.

2. In table. 2, why the performance improvement is more significant on fine-tuning quantization methods than fine-tuning free ones, when compared with GPTQ? It would be better to deeply discuss about the phenomenon.

We think the reason why finetuning quantization performs worse is the need to handle the massive outliers in activations. This issue was addressed only with QuaRot (FT-free) and SpinQuant (FT-based) until recently.

3. In table. 4, why the ppl result of GPTQv2' is worse than GPTQ, while the zero-shot avg result is better than GPTQ, which is counter-intuitive.

This is an excellent observation. Perplexity and zero-shot accuracy measure different aspects of LLM capabilities. Perplexity primarily evaluates next-token prediction on the pre-training distribution (a form of memorization), while zero-shot accuracy tests the model's ability to generalize knowledge to new tasks. When applying only the second term of our method, we're optimizing for the residual output error from previous layers, which better preserves the model's generalization capabilities at the expense of exact next-token prediction. The full GPTQv2 balances both aspects by combining both terms. This suggests that different quantization objectives might be optimal depending on the downstream task priorities.

4. I curious about the performance improvement when activation quantization is added before weight quantizations. It seems like the improvement compared to GPTQ is more significant under this condition. In my perspective, the quantized $\tilde{\mathbf{X}}$ will introduce noise into $\Delta \mathbf{X}$, therefore, it should be worse.

Thanks for the question. We kindly refer you to our Algorithm 2 in Appendix C. If the activation quantization is enabled during calibration, we will disable it when computing and caching $\tilde{\mathbf{X}}$ to ensure we always use the FP model activation. This is why the improvement will be more significant.

5. Seems like the histograms in Figure 4(a) are missing.

Sorry for the missing histograms. This appears to be a browser-specific rendering problem. If you download our paper, the histograms should be visible when viewing the PDF in Chrome or in standard PDF readers. We will ensure the figure is displayed correctly in all formats in the revised version of the paper.

Thank you for your positive assessment of our theoretical foundation and experimental results. Regarding your concern about hardware-level deployment and overhead analyses, we would like to clarify that GPTQv2 maintains full compatibility with GPTQ's quantization format since we did not modify the quant() function in Algorithm 1. This means that our method can leverage all existing hardware-optimized kernels and infrastructure developed for GPTQ without additional overhead during inference. Take an example of the format of AutoGPTQ library, for every quantized layer, the variables are defined as

class QuantLinear(nn.Module): 
    def __init__(self, bits, group_size, in_features, out_features):
        self.bits = bits
        self.group_size = group_size
        m, n = in_features, out_features
        self.qweight = torch.zeros((n, m//32 * self.bits), dtype=torch.int32)
        self.qzeros = torch.zeros((n//group_size, m//32 * self.bits), dtype=torch.int32)
        self.scales = torch.zeros((n//group_size, m), dtype=torch.float16)
        self.g_idx = torch.zeros(n, dtype=torch.int32)

The same quantization format in GPTQv2 can immediately benefit from specialized kernels like Marlin and ExLLaMA without requiring new hardware optimizations. Currently, we are integrating GPTQv2 into popular quantization libraries, and we will expand our hardware-specific analyses in the next version of the paper.

Thank you for your positive feedback on our manuscript. We appreciate your interpretation of GPTQ as a "sequential PTQ style" method versus the "distillation style" of our GPTQv2. Please see our responses to your specific concerns below:

1. The terms "asymmetric calibration" and "symmetric calibration" doesn't seem to be very intuitive, and maybe a bit confusing with the symmetric/asymmetric quantization. In fact, this set of terms is not used a lot in the manuscript. Maybe author can consider adding a few sentences around the definition of these terms to enhance the connection between the main proposed concept, so that the readers could grasp the main idea easier.

We agree that our terminology could be clearer. The terms "symmetric" and "asymmetric" specifically refer to the calibration objective: symmetric calibration uses the same input activation $\mathbf{X}$ for both the quantized and full-precision weights, while asymmetric calibration accounts for the input activation discrepancy between quantized weights and full-precision weights. We will add clearer definitions of these terms in the introduction and highlight their conceptual differences.

2. Since this work is meant to be an improvement of the original GPTQ, it would be beneficial to start the discussion with a comparison to vanilla GPTQ, i.e. weight only, per-group quantization. Maybe author could consider moving Appendix B.3/Table 7 into main manuscript, with a few more examples from different Llama models.

Thank you for your suggestion. We agree that demonstrating improvements over vanilla GPTQ (weight-only, per-group quantization) is important since many existing libraries target this setup. In the revised version, we will move Table 7 to the main text and expand it with comprehensive comparisons across multiple LLaMA models. Here are additional results for LLaMA3-8B's perplexity:

3. Even though (possibly) the majority of researchers still calls it GPTQ, the original author officially published their work at ICLR 2023 in the name of "OPTQ". I would not suggest the author of this work to change all the names/acronyms in the manuscript, but in respect of the choice of the original "GPTQ" authors, maybe include both names during first mention/citation and state that for only GPTQ will be used afterward for simplicity reason.

This is an essential problem that we were not aware of. We'll add a footnote in the introduction to clarify the naming issue. Thanks again for your suggestion.