any4: Learned 4-bit Numeric Representation for LLMs

We thank the reviewer for their constructive feedback, particularly recognizing that the paper “presents a compelling case”, and “provides strong evidence to support its claims and makes a valuable contribution to the field of LLM quantization”, as well as the important suggestions to improve the quality of the paper.

We address the reviewer’s comments as follows:

• Calibration Sample Construction: The calibration text sample (that we have provided in Section A.2 in the Appendix) was manually crafted with sentences spanning diverse domains: Fiction, News, Code, Math, Facts, and concatenating them together. Our intuition was that having a single sample with many diverse topics would be sufficient, rather than extracting many samples from a Common Crawl dataset and hoping that they will cover diverse domains.
• Combining any4 with AWQ or GPTQ: We have attempted to combine any4 with AWQ during the rebuttal period and provided results in Table B (in response to Reviewer pDZ9) in this rebuttal. We show that indeed combining AWQ with ANY4 improves the results of AWQ.
• Native FP4 Support in New Nvidia B200 GPUs: We acknowledge that new hardware is supporting FP4 as a native format for computation but we emphasize that our approach is particularly effective for PTQ and can enable arbitrary number of bits:
  • In our experiments, we observed that FP4 leads to lower Post-Training Quantization (PTQ) accuracy compared to NF4 and our proposed ANY4 format. i.e., FP4 does not perform well for zero-shot quantizing without training.
  • Nevertheless, FP4 works well for Quantized Aware Training (QAT). Native support for computing in FP4 enables training models faster with FP4. However, our paper explicitly focuses on post-training quantization (PTQ), a practical setting where model weights are quantized without retraining.
  • Furthermore, our method also achieves strong results with ANY3, a 3-bit format for which there is currently no native hardware support—highlighting the broader applicability of our lookup-table-based quantization approach beyond current hardware capabilities. Moreover, it could be used for 6-bit quantization [H] which other papers are exploring.
• Clarification on ANY2: We agree ANY2 is not competitive at this stage; we included it to illustrate the extensibility of our method across bit widths, even at the extreme low end.
  • Moreover, QuIP which performs well on 2-bit quantization is orthogonal to our approach and hence may be integrated in future work to enhance 2-bit quantization performance.
• Novelty: Our method is novel in that it is the first to apply scalar LUT-based quantization minimizing output error in LLMs without any finetuning, unlike prior work [F, G] that provided a formulation in reducing output error rather than weight error but required finetuning.

References

[E] “Ultra-Low Precision 4-bit Training of Deep Neural Networks”, Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi (Viji) Srinivasan, Kailash Gopalakrishnan, https://papers.nips.cc/paper_files/paper/2020/hash/13b919438259814cd5be8cb45877d577-Abstract.html, NeurIPS 2020

[F] “And the Bit Goes Down: Revisiting the Quantization of Neural Networks”, Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, Hervé Jégou, https://arxiv.org/abs/1907.05686, ICLR 2020

[G] “Extreme Compression of Large Language Models via Additive Quantization”, Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, Dan Alistarh, https://arxiv.org/abs/2401.06118, ICML 2024

[H] "FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design", Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, Shuaiwen Leon Song, https://arxiv.org/abs/2401.14112

We would like to thank the reviewer for their constructive review and comments, including that the approach was “rigorously benchmark[ed]”, the claims are “supported by experimental results and methodological explanations”, and that it’s implementation “enhances the paper’s real-world applicability”, as well as the important suggestions to improve the quality of the paper.

Please find below our remarks regarding the reviewer’s requests and comments:

• Limited Theoretical Foundation / Formal Proof: We would like to highlight that our approach is based on a theoretical derivation that we laid out in Section 4.1 of the paper. That mathematical derivation to minimize the reconstruction error of the output of each of the model’s linear layers, eventually lead to a modified k-Means clustering that guides us to apply clustering on the product of weights, mean activations, and group scaling factors (as described in Equation 14).
  • To prove the claims of our formal proof, we have provided in this rebuttal Table A (in the response to Reviewer pDZ9), empirical results that prove that each term in Equation 14 was necessary to minimize the loss of the model.
• Calibration Dataset Concerns:
  • In Table A5 in the Appendix, we show that our single sample calibration set outperforms other datasets such as WikiText-2, C4, and The Pile.
    • The results show that our approach of a single curated sample with sentences covering diverse topics (fiction, non-fiction, code, and math), as shown in Section A.2 in the Appendix could be sufficient to calibrate a quantization algorithm, in contrast to large datasets where each sample typically covers a single domain.
  • Nevertheless, our approach can work by calibrating with arbitrary datasets with arbitrary number of samples.
• Calibration Cost:
  • Applying our single-sample calibration takes only 1–3 seconds, while applying a larger number of samples (e.g., 128) that is required by other algorithms could take 10 or 20 seconds.
  • We have found that the total time to run our quantization algorithm on Llama2 7B is 10 minutes, which is similar to the time reported by other quantization algorithms: AWQ and GPTQ.
• Interaction with Other Acceleration Methods: Overall, our quantization algorithm is orthogonal to other optimizations.
  • Torch.compile: We have already integrated our INT4 implementation of our tinygemm library with torch.compile. We still haven’t yet integrated the ANY4 implementation within the library with torch.compile, but we believe it is straightforward.
  • FlashAttention: FlashAttention in general (and FlashAttentionv3 in particular) only optimizes activation-activation matrix multiplication in the attention operation (i.e., Y = softmax(QK.T / sqrt(d) )V) while our work focuses on quantizing weight-activation matrix multiplication (for attention that would be Q=XWq, K=XWk, V=XWv, O=XWo, and we also quantize all the matrix multiplications in feed forward layers). Hence, from the perspective of end-to-end speedup of a model, our work is orthogonal to FlashAttention.
• Other
  • Moving Results from Appendix to Main Body: We agree with this proposal and plan to do that in the Camera Ready version if the paper is accepted.
  • Discussions on Other Optimization Approaches: We can add discussions to the paper but we can mention here:
    • Quantization in general is orthogonal to other optimization approaches like pruning and neural architecture search.
    • We have focused in this paper on Post-Training Quantization (PTQ) that does not require retraining or fine-tuning of the model, while pruning requires extensive fine-tuning or continual pretraining, and NAS requires training from scratch or continual pretraining.
• Strengthening Evaluation:
  • Ensuring Statistical Validation: For the camera ready paper, we plan to re-run some of the text generation experiments with different seeds and report the average, and measure perplexity on different random subsets of the respective datasets and report their averages. However, we have noticed that in most papers on quantization, and on LLMs in general, measurements on perplexity as well as text generation are only provided for a default seed.
  • Consistent Experimental Conditions: For perplexity, we have re-used the same evaluation script that is used in GPTQ, that is also used in AWQ, and used the exact same seed to select the subsets of data. For downstream tasks, we have used LM Evaluation Harness and BigCode Evaluation Harness with the default settings provided in each library. We believe that our experimental conditions are consistent but we welcome further suggestions to improve consistency.

We would like to thank the reviewer for their constructive review, including “higher confidence in the presented method” due to presenting results on generation and coding tasks, and thank the reviewer for checking the “soundness of [the] math” of the “full derivation of [the] method”, as well as for the important suggestions to improve the quality of the paper.

Please find below our remarks:

• Benefit of Group Quantization with Row-wise LUT: the reason we chose row-wise (look up table) LUT is to minimize the storage overhead.
  • Each LUT consists of 2^n_bit FP16 values. So for 4-bits, each LUT consists of 16 FP16 values. Having such a LUT for each group of 128 values is a high overhead, while having it for each row (in Llama2 7B each row has 4096 values) the overhead is negligible.
  • On the other hand, grouping requires 2 FP16 values (one for scale and one for offset) for each group. So we can afford having relatively small group sizes for grouping.
• Comparison with SqueezeLLM: In the paper we haven’t compared with SqueezeLLM because SqueezeLLM keeps a portion (albeit small) of it’s weights in high precision (in their code they keep 10 rows of each weight matrix in high precision as well as 0.45% of outlier and sensitive values in the matrix). Nevertheless, we compare it here as requested by the reviewer in the Table below.
  • Despite that any4 does not rely on storing rows or portions of its weights in high precision, its perplexity is competitive with SqueezeLLM.
  • Moreover, we would like to highlight that storing rows or outlier/sensitive values in high precision is orthogonal to any4 and could be combined with it.
  • The numbers reported below were copied from the SqueezeLLM paper. We tried to run the SqueezeLLM code to quantize Llama3 models, to compare with any4, but we hit into runtime errors in their code.

Table D: Comparison of any4 with SqueezeLLM

• Any4 Under RTN Umbrella: We categorized ANY4 under round-to-nearest (RTN) because in RTN given a list of possible values (whether those values are a LUT as in ANY4, or predefined as in INT4, FP4, or NF4), we round each weight to the nearest one.
  • Any computation done in ANY4 algorithm is used to obtain the optimal values of the LUT, rather than modifying the values of the weights or activations. While quantization algorithms do modify the values of weights and/or activations:
    • AWQ scales down the activations and scales up the weights.
    • GPTQ quantizes each column in a weight matrix sequentially, and every time a column is rounded to the nearest number, the remaining unquantized weight values are modified to mitigate the reconstruction error of rounding previous columns.
• Accuracy on All Tasks:
  • Perplexity is a Less Noisy Indicator then Downstream Tasks: Accuracies on downstream tasks, especially generation tasks, could be noisy. EvalArena [D] studies the noisiness of such downstream tasks. E.g., HumanEval and MBPP appear in EvalArena with a significantly low signal to noise ratio, which explains why results in Table A1 may show ANY4 not always outperforming others on those specific tasks.. On the other hand, perplexity is a less noisy metric as it measures average loss on all tokens in each sample (rather than evaluating a final token or evaluating a pass or fail for a whole sample), and any4 does consistently well on perplexity in Table A1.
  • 4-bit vs. 3-bit vs. 2-bit: Table 1 compares perplexity on various bitwidths with quantization algorithms that do process weights and/or activations. While the results show QuIP tends to be better than our approach on lower bits (always better on 2-bits and sometimes better on 3-bits), our approach is always in the top for 4-bits.
  • Overall: instances where any4 is not the top-performing method are a natural outcome of our extensive evaluation across a diverse set of models, generation types, model sizes, and tasks.
• Other:
  • Typos and Formatting: We thank the reviewer for pointing out the typos and issues in formatting. We have applied all the fixes.
  • Theoretical Claims: Please see "Limited Theoretical Foundation / Formal Proof" in our response to Reviewer aGNw.

References [D] EvalArena, https://crux-eval.github.io/eval-arena/

We thank the reviewer for their detailed and constructive feedback, which will help improve the paper.

Please find below our remarks:

• Optimizing Weights Directly: Please find the results in Table A below. First row shows the results of optimizing weights directly. The other 2 rows show the results of using the 2 additional terms of Equation 14 in our paper, i.e., multiplying with activations and scales. These results confirm that our derivation (Eq. 14) is essential for optimal performance.

Table A: PPL Results on Any4 Quantization on Llama3.2 1B (Lower is Better)

• Lack of Novelty: Please see our response to Reviewer U2Ld
• Significant Improvement for Llama3: Quantizing Llama 3 is known to be more challenging than Llama 1 and 2, often resulting in larger accuracy drops [A]. This is likely due to its larger pretraining dataset (8T tokens vs. 2T for Llama 2 [B, C]), which makes it better trained—and therefore harder to compress without loss. In contrast, older models are relatively undertrained and easier to quantize, so the impact of advanced quantization methods appears smaller. Thus, the stronger results on Llama 3 highlight the effectiveness of our approach, especially on newer, more robust models where quantization is harder and matters more.
• PTQ Results using NF4: We provide in Table B the results of combining AWQ with NF4 as well as results of combining AWQ with ANY4. These results show that combining preprocessing-based methods like AWQ with our numeric format leads to further improvements.

Table B: Combining AWQ with Different Numeric Formats

• Algorithm Time Comparison: We have measured the time to quantize a Llama2 7B and found it to be approximately 10 minutes, which is similar to the time we have measured to run both AWQ and GPTQ. Therefore, we conclude that quantization times of different approaches are similar.
• Different Group Sizes:
  • We used group size 128 as it is the default in many quantization papers (e.g., AWQ, GPTQ).
  • In the Appendix we have provided results for group sizes 64, 128, 256, 512, 1024 for Llama3.2 1B.
  • As requested by the reviewer, we also provide here results for group size 32, as well as the other group sizes we already had in the Appendix
  • Please note the bitsandbytes library that we use for FP4 and NF4 quantization doesn't support group size 32. | | 32 | 64 | 128 | 256 | 512 | 1024 | |--|-|-|-|-|-|-| | FP4 | N/A | 16.19 | 17.11 | 18.12 | 20.43 | 2.3E6 | | NF4 | N/A | 14.27 | 14.63 | 14.98 | 15.38 | 7.8E5 | | ANY4 | 13.54 | 13.75 | 13.95 | 14.09 | 14.24 | 14.34 |

Table C: Llama3.2 1B Perplexity on C4 (Lower is Better)

• Other:
  • Formatting and Typos: We thank the reviewer for the detailed comments and we will apply them to the camera ready paper.
  • Equation 17 following Equation 14: We thank the reviewer for highlighting this important point. While Eq. (14) defines a global row-wise objective, Eq. (17) corresponds to a local minimization step within a K-Means-style alternating optimization procedure.
    • Specifically, Eq. (17) performs the E-step, assigning each weight $w_{s_{i,j}}$ to its nearest codebook value in $Q_i$, treating activations $x_j$ as constants. This simplifies to a local nearest-neighbor assignment. The M-step (Eqs. (19)–(20)) then updates $Q_i$ by minimizing the total reconstruction error across the row.
    • Thus, while Eq. (17) does not solve the global objective in Eq. (14) directly, it is part of an iterative process that does. We will revise the text to clarify this decomposition and explicitly note that Eq. (17) performs local minimization within the broader alternating optimization scheme.
  • Downstream Task Performance: Please check "Perplexity is a Less Noisy Indicator then Downstream Tasks" in our response to Reviewer yZMB.

References [A] “How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study”, https://arxiv.org/abs/2404.14047v1, April 2024

[B] “Scaling Laws for Floating Point Quantization Training”, https://arxiv.org/abs/2501.02423, January 2025

[C] “The Llama 3 Herd of Models”, https://arxiv.org/abs/2407.21783, July 2024