FlatQuant: Flatness Matters for LLM Quantization

Q1. For an exhaustive comparison I would suggest adding a few more state-of-the-art methods [1, 2]. The former provides other strategy for learning rotation and the latter accounts for outliers tokens in KV cache.

[1] Lin H. et al. Duquant: Distributing outliers via dual transformation makes stronger quantized llms //Advances in Neural Information Processing Systems. – 2025. – Т. 37. – С. 87766-87800.

[2] Chen, Mengzhao, et al. "Prefixquant: Static quantization beats dynamic through prefixed outliers in llms." arXiv preprint arXiv:2410.05265 (2024).

A1. Thanks for the recommendation regarding the recent works. We will add them in the revised version. PrefixQuant is mainly experimented with static quantization and is a technology orthogonal to our method, we leave the integration of PrefixQuant with FlatQuant as the future work. Below we conduct a detailed comparison between FlatQuant and DuQuant.

• Comparison with DuQuant in Accuracy. FlatQuant significantly outperforms DuQuant across multiple benchmarks.

• Comparison with DuQuant in Latency. DuQuant has the same number of online transformations (each consisting of two matrix multiplications and one channel permutation) as FlatQuant (each consisting of a Kronecker-decomposed matrix multiplication). It can be seen that FlatQuant achieves comparable speedup without kernel fusion and even faster speedup with kernel fusion.

---

Q2. What are the speed-ups on newer generation of GPUs (say GPU 4090) or high-end GPU on prefill and decode? Would you expect them to be lower or higher?

A2. Thanks for the valuable suggestion. Currently, we haven't implemented INT4 kernels on other GPUs, but we can approximately estimate the speedup. In the following, we estimate the speedup for a single linear layer with hidden dimension $d$ on A100 GPU. The speedup for attention layers can be estimated likewise.

• A100 Specs. For A100, we have $\text{Perf} _{\text{BF16}}=312\text{ TFLOPS},\text{Perf} _{\text{INT4}}=1248\text{ TFLOPS},\text{Bandwidth}=1555\text{ GB/s}$

• Prefill Speedup. Assume $s=2048$ and $d=4096$ where $s$ is the number of tokens in the prefill. Since in the prefill stage, the linear layer is mostly compute-bound, while the online quantization and transformations are memory-bound, we can estimate the prefill speedup accordingly:

$\text{FLOPs} _{\text{Linear}}=2sd^2, \text{Mem} _{\text{Quant+Kron}}=2.5sd+4d$

$\text{Prefill Speedup}\approx \frac{\text{FLOPs} _{\text{Linear}}/\text{Perf} _{\text{BF16}}}{\text{FLOPs} _{\text{Linear}}/\text{Perf} _{\text{INT4}}+\text{Mem} _{\text{Quant+Kron}}/\text{Bandwidth}}\approx 3.21$

• Decoding Speedup. Similarly, the decoding speedup can be estimated with

$\text{Mem} _{\text{BF16}}=2d^2+4d,\text{Mem} _{\text{INT4}}=d^2/2+2.5d,\text{Mem} _{\text{Quant+Kron}}=6.5d$

$\text{Decoding Speedup}\approx \frac{\text{Mem}_ {\text{BF16}}}{\text{Mem}_ {\text{INT4}}+\text{Mem}_ {\text{Quant+Kron}}} \approx 3.98$

---

Q3. In addition, it could be insightful to measure speed-ups with other sequence lengths.

A3. Thanks for the helpful advice. In the following, we provide more speedup results with other sequence lengths as a complement to Figure 4:

• Prefill speedup with batch size 1.

• Decoding speedup with batch size 64.

Q1. ...a clearer quantitative comparison of what constitutes the previous SOTA.

A1. We consider QuaRot and SpinQuant as previous SOTA in Table 1-2, and compare with the latest DuQuant in A1 to Reviewer x5Xi.

---

Q2. The notion of "flatness" is somewhat intuitive but would benefit from a more rigorous definition or metric.

A2. Thanks for the suggestion. We have provided a quantitative definition of flatness in Appendix D.1 Line 1106-1113, and visualized the associated flatness score in Figure 7 and Figure 11.2. We will move this part to the main text.

---

Q3. The claim that "FLATQUANT with RTN is sufficient to be competitive" is intriguing but lacks thorough theoretical explanation.

A3. Thanks for pointing this out. We suspect the affine transformations alone make the distributions friendly to RTN, where the flatness is quantitatively measured in Appendix D.1 and Figure 7. We decide to leave theoretical explanations in future works.

---

Q4. The paper would benefit from a more detailed explanation of why learnable clipping after transformation is more effective than before transformation (as shown in Table 17).

A4. Thanks for the question. This can be attributed to the preservation of important outlier channels.

• "LCT before Transformation" consistently performs worse, as it directly clips important outliers in LLMs.

• "LCT after Transformation" performs significantly better because the affine transformations redistribute outliers across channels. Even after clipping, the inverse transformation can effectively recover the original outliers.

We will further clarify this issue in our revision.

---

Q5. Some minor typos and grammatical issues exist.

A5. Thanks for the attentive feedback. We will fix these issues in the revised version.

---

Q6. How would FLATQUANT perform when incorporated into more advanced quantization schemes like mixed-precision quantization, where different layers use different bit-widths? This could affect my evaluation as it might demonstrate broader applicability.

A6. Thanks for raising this interesting question. Below we explore some orthogonal mixed-precision strategies to enhance FlatQuant.

• Weight Importance Score Analysis. We follow SqueezeLLM[1] to use Fisher information for the estimation of weight importance, and simply sum over the scores to estimate the overall importance of different parts in LLMs, including different linear layer types and Transformer layers.

• Mixed-precision Results. According to the overall importance, we use W8A8 for the top 5 Transformer layers as well as all the down projection layers, which proves to be an effective way to improve FlatQuant.

[1] Kim, Sehoon, et al. "Squeezellm: Dense-and-sparse quantization." arXiv preprint arXiv:2306.07629 (2023).

---

Q7. Have you analyzed how different initialization strategies for the affine transformation matrices affect convergence speed or final performance?

A7. Thanks for the thoughtful feedback. We initialize the transformations with random orthogonal matrices by default and find FlatQuant is generally robust to initialization strategies.

---

Q8. How would FLATQUANT perform in even more extreme scenarios, such as W3A3 quantization? Some results are shown in Appendix Table 14, but a more detailed analysis of limitations would be valuable.

A8. Thanks for the advice. As shown in Table 14, although FlatQuant achieves notable improvements over QuaRot, it still suffers significant accuracy degradation. W3A3 needs both improved algorithms and hardware support, which still remains an open question.

Q1. Regarding techniques for suppressing outliers..., the authors overlooked the use of infinity-norm regularization for reducing the weight range.

A1. Thanks for the recommendation regarding related works. We will add the discussions in the revised manuscript.

---

Q2. The results are primarily empirical, lacking sufficient theoretical justification for the effectiveness of the Kronecker decomposition-based linear transformation. Specifically, it remains unclear why this approach outperforms rotation-based transformations. A deeper theoretical analysis would strengthen the claims.

A2. Thanks for the advice. Compared with rotational transformations, FlatQuant relaxes the optimization space from Stiefel manifold to the general linear space, which has more potential to suppress outliers. We leave more detailed theoretical analyses in future work.

---

Q3. It is counterintuitive that FlatQuant outperforms AffineQuant significantly...

A3. In the following, we discuss the superiority of FlatQuant over AffineQuant in more detail as a complement to Line 631-637:

• Improved Expressivity. As discussed in Figure 5, we find that the Kronecker decomposed transformations deliver comparable accuracy with the full-size transformation, which demonstrates its expressivity. In contrast, AffineQuant optimizes strictly diagonally dominant transformation to ensure invertibility, which may restrict its expressivity and make the transformation more like a per-channel scaling (see Figure 7 in the AffineQuant paper).

• Applicability to All Linear Layers. FlatQuant applies Kronecker-decomposed linear transformations to all linear layers with minimal overhead. In contrast, AffineQuant directly learns a full-size transformation, and can only apply it to the output projection linear layer for WA quantization so that the transformation can be merged into preceding linear layer to avoid the formidable overhead. The other linear layers can only use per-channel scaling.

---

Q4. The contributions are somewhat incremental, extending existing work on block-wise quantization and linear transformations for smoothing weights and activation values.

A4. We elaborate on the key differences between FlatQuant and previous works below:

• Linear Transformations v.s. Orthogonal Transformations. While previous works (e.g. QuaRot) explore orthogonal transformations to smooth outliers, we find linear transformations can achieve better flatness, delivering superior quantization accuracy as shown in Table 1-2.

• Kronecker Decomposed Matrices with Kernel Fusion. While linear transformations suffer from high computation and memory consumption, we mitigate it with Kronecker decomposition along with kernel fusion, ensuring practical speedup with minimal overhead.

• Learnable Clipping After Transformations. As detailed in Appendix C.4, a key difference of learnable clipping in FlatQuant is that it is applied after the pre-quantization transformations. This can help avoid damaging critical outliers during activation clipping.

---

Q5. What is the role of Kronecker decomposition-based transformation in mitigating outliers? Can we achieve similar effects by imposing alternative low-complexity structures, such as low-rank decomposition?

A5. The learnable transformation to remove outliers needs to satisfy 1) invertible, so that the transformation is computational invariance (similar to QuaRot); 2) lightweight so that little inference overhead is introduced. Kronecker decomposition is a valid approach to satisfy both conditions, where the invertibility is ensured from (Line 661-663). However, low-rank decomposition does not satisfy 1) invertibility. We leave the exploration of other alternatives in the future.

---

Q6. I wonder how FlatQuant performs on 2-bit weight quantization.

A6. Thanks for the question. In the following, we experiment with 2-bit asymmetric per-channel weight-only quantization and compare FlatQuant against two strong uniform weight-only quantization methods GPTQ and QuIP[1]. For GPTQ, we use activation reorder and grid search the best weight clipping thresholds.

[1] Chee, Jerry, et al. "Quip: 2-bit quantization of large language models with guarantees." Advances in Neural Information Processing Systems 36 (2023): 4396-4429.

Q1. The authors do not provide a comparative assessment of GPU memory consumption...

A1. Thanks for the helpful advice. Below, we provide more results on the memory consumption. The experiment results are consistent with our theoretical analysis in Appendix B.2 Line 736-739, demonstrating the minimal memory overhead brought by the online transformations in FlatQuant.

• Peak memory usage for decoding a single token on one Transformation layer of LLaMA-2-7B model with KV caches of different lengths and batch size 1.

---

Q2. The paper lacks exploration of potential biases introduced by Kronecker decomposition...

A2. Thanks for the thoughtful feedback. As shown in Figure 5, experiments demonstrate that Kronecker decomposition preserves expressivity, achieving accuracy comparable to full-size transformations. We leave theoretical justifications as future work.

---

Q3. ...A more targeted ablation would clarify the relative importance of affine transformations versus clipping thresholds.

A3. Thanks for the constructive advice. Below we provide more results and discussions to demonstrate the effectiveness of LT:

• Full Ablation Table. We provide full ablation results as a complement to Table 3, which clearly showcases the effectiveness of each part in FlatQuant, especially LT.

• Comparison between FlatQuant's affine transformation and SpinQuant's orthogonal transformation. For both methods, we use asymmetric activation quantization following SpinQuant and do not activate weight or activation clipping. For FlatQuant, we also do not use PS. It can be seen that the affine transformations in FlatQuant are more effective in easing outliers, leading to higher accuracy especially when RTN is used.

---

Q4. The authors treat each layer's quantization independently, … , potentially overlooking cascading effects.

A4. Thanks for this valuable question. We tried to minimize the MSE loss between ground truth and outputs given quantized inputs instead of given full-precision inputs (as in Equation 4) to mitigate cascading effects. However, results show that full-precision inputs lead to better performance. To learn the mechanism behind, we think this is similar to the "teacher forcing" in training RNNs (i.e., use the ground truth instead of auto-regressive output for training at each step). Similar effects were also verified to be helpful in model quantization [1].

[1] Bai, Haoli, et al. "Towards efficient post-training quantization of pre-trained language models." Advances in neural information processing systems 35 (2022): 1405-1418.

---

Q5. The paper would benefit from a clearer explanation of how the online transformations are initialized,....

A5. Please see A7 to Reviewer pwvD.

---

Q6. The hyperlinks throughout the paper appear to be non-functional,...

A6. We have double-checked the manuscript and the hyperlinks function well on our side. We are sorry for the inconvenience, and maybe a different PDF browser could resolve your issue.