FPTQ: FINE-GRAINED POST-TRAINING QUANTIZATION FOR LARGE LANGUAGE MODELS

$**Q1:**$ Degradation of the MMLU score and comparison with LLM-QAT.

$**A1:**$ Indeed, there is an observable degradation of the MMLU score. Considering we apply a more aggressive quantization (W4A8), a certain degree of performance degradation is reasonable and expected. A more severe degradation is seen in LLM-QAT. We quote their reported MMLU scores (in their Appendix) here for a fair comparison. Note under the same bit width setting W4A8, our PTQ method FPTQ outperforms their costly QAT method by a clear margin.

Table v. Comparison with LLM-QAT W4A8 on MMLU

It's important to highlight that our MMLU scores are very close to those achieved with W8A8 or W4A16 quantization. We believe this is a testament to the effectiveness of our approach. Furthermore, it's worth noting that existing methods can experience significant performance drops even with W8A8 or W4A16 quantization on some sensitive models. The trade-off here is the significantly improved computational efficiency, which is a key goal of our work.

$**Q2:**$ Significance of fine-grained quantization.

$**A2:**$ We concur with your viewpoint that the observed inference characteristics could be applicable to a broad range of quantization papers utilizing W4A8. Nonetheless, our emphasis on fine-grained quantization stems from its unique implementation in our work and its contribution to the overall effectiveness of our approach. We would like to underscore that while fine-grained quantization can help reduce accuracy loss (see also $**A1**$ to Reviewer yeXC), it can potentially lead to a significant decrease in inference speed as well (see also Analysis on computation efficiency in Section 6).

$**Q3:**$ The selection v0 and v1 and their sensitivy.

$**A3:**$ They are empirical values, see also $**A3**$ to Reviewer WDjT. Note that v0 and v1 serve as a ruler to select which layers require advanced quantization techniques. According to our statistical analysis over a large body of LLMs, the two numbers can be mostly kept the same.

$**Q4:**$ Cost of context-decoding stage.

$**A4:**$ Type conversion does add extra cost but it also depends on the implementation. We have revised Figure 2 (b) for a more accurate depiction.

$**Q5:**$ How to determine clipping value when employing activation static quantization?

$**A5:**$ In our experiments, we don't apply clipping for activation. As for LAE in Eq.1, we use min/max for smoothing.

$**Q6:**$ Has KV cache quantization be implemented?

$**A6:**$ Yes, we apply the KV cache as a reusable buffer to save the per-tensor static quantization result of KV’s activations.

$**Minors**$

Typos are fixed, see also our common response.

Thank you for the detailed answers and results. I have read the authors' rebuttal as well as other reviews. I would like to keep my rating.

$**Q1:**$ Comparison with SmoothQuant at W4A8 and further component analysis.

$**A1:**$ We give the comparison and the step-by-step ablation in Table iv below. A vanilla SmoothQuant W4A8 is much inferior. In contrast, our proposed FPTQ substantially resolves the degradation in the W4A8 setting. We emphasize that FPTQ takes full advantage of fine-grained weight quantization, a layer-wise activation quantization strategy, and logarithmic activation equalization, altoghther to have on-par or better performance compared with SmoothQuant W8A8 in Table 3.

Table iv: SmoothQuant vs. FPTQ strategies on the LAMBADA dataset.

These results are now updated in the revised manuscript.

$**Q2:**$ Concerned on the time cost plot of W4A8 vs. W8A8 and W4A16.

$**A2:**$ Indeed, during the context decoding stage when the computation bound is met, the W4A8 and W8A8 would have close computation cost theoretically. However, it also depends on practical input lengths where W4A8 enjoys I/O benefit, especially at shorter input lengths. Similarly, W4A8 should share a similar time cost with W4A16 in the self-decoding stage. This figure is redrawn to be more accurate.

$**Q3:**$ How was v0 and v1 chosen?

$**A3:**$ See $**A3**$ to Reviewer WDjT.

$**Q4:**$ Real hardware results.

$**A4:**$ We would be very glad to share when it gets ready. However, we focus on a theoretical upper bound here, see also $**A1**$ to Reviewer WDjT.

$**Q1:**$ Inference speed and expensive operations.

$**A1:**$ In this paper, we are primarily concerned with the theoretical upper-bound of inference time in W4A8 bit widths. The real inference speed, however, largely depends on the specific hardware support (GPU, CPU, FPGA, etc.) and the amount of engineering effort put into implementations.

Costly operations. Below we provide an end-to-end performance comparison of per-token dynamic quantization and per-tensor quantization under the W8A8 setting. Notice that per-token dynamic quantization adds extra overhead that ranges from 5% to 10%.

Table i. Latency of LLaMA2-13b tested on a A100 80G GPU

$**Q2:**$ The degradation is shown on MMLU.

$**A2:**$ It is expected that the degradation increases as we chase narrower bit widths. Noticeably, we've updated LLM-QAT W4A8 results in Table 4 while FPTQ generally prevails over it. This attests that FPTQ is a strong PTQ method that even outperforms the QAT method. Nonetheless, the majority of our MMLU results are comparable to or better than W8A8 and W4A16, There are orthogonal ways such as weight-clipping, GPTQ, distillation, or applying better calibration data to mitigate this quantization gap, however, it is not the key focus of this paper. We maintain the current settings for a fair comparison.

$**Q3:**$ On how v0 and v1 is selected.

$**A3:**$ The values of v0 and v1 (recommended as 15 and 150) were determined empirically. We discover that per-tensor activation quantization already generates low MSE for the range (0, v0). While for (v0, v1) it calls for LAE to smooth the range first before per-tensor quantization to have an acceptable MSE. For intractable layers like FC2 that go far beyond v1, we resort to per-token quantization. According to activation distributions over a large body of LLMs in Table ii (also see Fig.5 - Fig 10 in the appendix), this selects QKV, and FC1 for LAE before per-tensor quantization, while keeping the Dense layer untouched.

Table ii. Activation ranges of different types of layers in various LLMs

After the application of the proposed LAE on QKV and FC1, we have a smoothed activation range that is much easier to quantize, as shown in Table iii below.

Table iii. Smoothed activation ranges (min/max) of different types of layers in various LLMs

$**Minor:**$

• Table 4 bold numbers are mismarked and it is removed in the revision.
• Page 5, two strategies refer to per-channel strategy and fine-grained strategy. Modern GPUs support per-block computation for tensors which can be exploited for the fine-grained strategy.
• For the KV cache, we use per-tensor static quantization throughout all the experiments.
• In Table 6, the variations due to different calibration datasets are actually small. Our intention was to demonstrate that our method isn't limited to specific calibration sets and that our performance remains robust even on randomly selected data.