FlatQuant: Flatness Matters for LLM Quantization

We thank the reviewer for the positive review and constructive comments. In the following, we elaborate on the method part of FlatQuant (including the invertibility of $\mathbf P$ and the use of Kronecker decomposition) and provide more experiment results to showcase the effectiveness of FlatQuant.

---

Q1. The manuscript does not clearly explain how the invertibility of $\mathbf P$ is maintained during training. Detailing the methods used to ensure this could improve understanding of the method.

A1. We thank the reviewer for raising this important question. As discussed in Appendix B.1, FlatQuant ensures invertibility by employing a singular value decomposition (SVD) re-parameterization trick during training. In the following, we provide more details.

• The SVD Re-parameterization. For a learnable affine transformation matrix $\mathbf P$, we decompose it as $\mathbf P=\mathbf U\Sigma\mathbf V^\top$, where $\mathbf U,\mathbf V$ are orthogonal matrices, and $\Sigma$ is a diagonal matrix. We ensure the invertibility of $\mathbf P$ by forcing the elements of $\Sigma$ larger than zero. The optimization of orthogonal matrices can be readily accomplished with Cayley parameterization. Therefore, $\mathbf P^{-1}$ can be easily computed with $\mathbf V\Sigma^{-1}\mathbf U^\top$.

• Comparison with Direct Inversion. In Appendix B.1, we also compare this SVD training strategy with the naive strategy of directly inverting the matrix $\mathbf P$, which cannot ensure the invertibility of $\mathbf P$ theoretically. Although we empirically find that this naive strategy can still produce invertible matrices in the training process, it needs FP32 training to stabilize the optimization process. After thorough comparisons between these two strategies, we finally adopt the SVD training which enjoys both training stability and efficiency.

---

Q2. I notice the use of Kronecker decomposition during inference and SVD during calibration, but the specific methodologies for Kronecker decomposition are not detailed. Elaborating on these could clarify the approach.

A2. We would like to clarify that FlatQuant directly learns the Kronecker-decomposed transformations during calibration and uses them during inference. The detailed process is as follows:

1. We prepare two individual learnable decomposed matrices of size $n_1$ and $n_2$ for each affine transformation, where $n_1$ and $n_2$ are determined based on the strategy introduced in Section 3.1 (Line 230~234).

2. Each decomposed matrix is then constructed using SVD re-parameterization trick (detailed in Appendix B.1) to ensure training stability and efficiency, where two orthogonal matrices and one diagonal matrix are used to construct one decomposed matrix.

3. These transformation matrices are inserted into the network and trained using the calibration data.

4. After training, the SVD matrices can be combined to obtain the learned decomposed transformation matrices. After that, as shown in Figure 3, some of the transformation matrices, along with other learnable parameters, are merged into the weights. The remaining transformations are retained online and efficiently executed using our fused kernel.

---

Q3. The potential approximation errors from Kronecker decomposition and their impact on model performance are not mentioned. Experimental results in this area would be beneficial.

A3. We thank the reviewer for the advice. Our experiments demonstrate that Kronecker decomposition achieves comparable accuracy to full-size transformations while significantly improving inference efficiency.

• As shown in Figure 5, different decomposition sizes have minimal impact on perplexity. However, the choice of decomposition size significantly affects inference efficiency.

• To further validate the impact of decomposition on accuracy, we conducted experiments on LLaMA-3-8B with and without decomposition. The table below shows that decomposition maintains high accuracy:

Q4. The overhead of dynamic activation clipping during inference should be explored.

A4. We thank the reviewer for the suggestion. We measure the latency of quantizing activations of different shapes both with and without activation clipping. The results are shown in the following table.

• As can be seen, activation clipping incurs marginal latency overhead since it only amounts to adding a scaler scaling factor to the calculated max and min tensor with shape [bsz, seq_len].

---

Q5. The individual contributions of $\alpha_a$ and $\alpha_w$ should be explored.

A5. We appreciate the reviewer's advice. We conduct additional ablation studies for learnable weight clipping and learnable activation clipping as a complement to Table 4. As shown in the table below, both learnable weight clipping and activation clipping are effective components to increase quantization accuracy.

Q6. The ablation study should include separate evaluations for "PS", "LCT", and "PS+LCT" to comprehensively demonstrate the improvements from each component.

A6. We thank the reviewer for pointing this out. We include ablation results for "PS", "LCT", and "PS+LCT" in the following table as a complement to Table 4. It is shown that without the online transformations to smooth outliers, the quantized models completely collapse.

We further build "PS" and "LCT" on top of online Random Orthogonal Transformations (ROT) to better showcase the strength of "PS" and "LCT". It can be seen that each component contributes positively to the final model accuracy.

---

Q7. The distribution of $\mathbf W_d$, $\mathbf W_q$, $\mathbf W_k$, and $\mathbf W_v$ and their corresponding activations is not shown. Visualizations like Fig. 2 for these weights and activations would be beneficial.

A7. We thank the reviewer for the constructive advice. We have added visualizations for the weights and activations of these linear layers in Figure 16-17 of the revised manuscript. As can be seen, FlatQuant can achieve flat distribution for weights and activations across different linear layers.

---

Q8. I am curious about the speedup ratio of the method on Hopper architecture GPUs (e.g., H100, H800).

A8. We thank the reviewer for the question. Unfortunately, we currently lack access to Hopper architecture GPUs (e.g., H100, H800) and are unable to provide speedup results for these devices at this time. However, we are committed to extending our evaluations to more GPU architectures, including Hopper, as soon as these resources become available.

---

Q9. Could the authors explain why the per-channel scaling is employed? A proper $\mathbf P'$ can contain the ability to balance outliers between the weights and activations, for example, $\mathbf P'=\mathbf P\text{dig}(\mathbf c)$.

A9. As stated in Section 3.1, FlatQuant directly learns the decomposed transformations. However, we observe that such decomposition can result in an incomplete diagonal structure in the transformation matrix, which may affect its ability to balance outliers between weights and activations. To address this issue, we explicitly introduce a learnable scaling vector before the decomposed transformation to encourage the balance of quantization difficulty between weights and activations.

Dear Reviewer,

Thank you for the time and attention you've dedicated to reviewing our paper. As the discussion stage is coming to an end, please let us know whether we have addressed your concerns or if any further discussions are needed.

We look forward to your feedback and thank you once again for your valuable contribution to our work.

Best, Authors

Dear Reviewer tWUe,

We are still waiting for your response to our feedback and open to any of your concerns.

Thanks! Authors of FlatQuant.

We thank the reviewer for the thorough review of our paper and for offering valuable feedback. In response, we have provided additional explanations and experimental results to address each of the points raised.

---

Q1. Even with Kronecker decomposition, FlatQuant still requires a pre-quantization step before each linear input, leading to O(n) memory accesses and O(nlogn) computational overhead, where n is the model's feature dimension. For a single block of LLaMA, this transformation needs to be applied more than six times, increasing the model's complexity. In contrast, SpinQuant requires only one pre-quantization transformation, which may offer a more significant advantage. I would be very grateful if more comparisons on the additional computational complexity and parameter quantity improvement brought by Spinquant and FlatQuant could be provided.

A1. We would like to highlight that FlatQuant can achieve notable accuracy improvement compared with SpinQuant as shown in Table 1 and Table 2, while binging minimal inference overhead (e.g., <0.01x speedup loss). In the following, we show the speed-up comparison between FlatQuant and SpinQuant.

• Speedup Comparison with SpinQuant. We summarize the speedup of FlatQuant and SpinQuant after incorporating different online transformations in the following table. It can be seen that with the designed efficient kernel, FlatQuant can achieve better speedup than SpinQuant.

• Memory Consumption Comparison. Both FlatQuant and SpinQuant incur negligible memory overhead. We compute the parameter count of each online transformation below: (1) $\mathbf{P}_a: 2 (\sqrt{h_d})^2$; (2) $\mathbf{P}_o: a^2$; (3) $\mathbf{P}_h: (h_d / a)^2$; (4) $\mathbf{P}_u\ _g: 2 (\sqrt{h_d})^2$; (5) $\mathbf{P}_d: 2 (\sqrt{h_i})^2$, where $h_d$ denotes the hidden dimension, $h_i$ denotes the intermediate dimension, $a$ denotes the number of attention heads. The additional memory consumption brought by FlatQuant in one Transformer layer during inference is $2(4h_d+2h_i+a^2+(h_d/a)^2)$ bytes, which only consumes about 0.11MB extra memory space for LLaMA-2-7B. The memory overhead for SpinQuant is $2((h_d / a)^2+2h_i)$ bytes, which only amounts to 0.07MB for LLaMA-2-7B.

---

Q2. AffineQuant ensures that affine transformations remain invertible by employing diagonal dominance, but it's unclear how FlatQuant maintains invertibility. This information seems to be missing from the current paper and would be important for understanding the method's robustness.

A2. We thank the reviewer for raising this important question. This is already discussed in Appendix B.1, where FlatQuant ensures invertibility by employing a singular value decomposition (SVD) re-parameterization trick during training. Specifically,

• The SVD Re-parameterization. For a learnable affine transformation matrix $\mathbf P$, we decompose it as $\mathbf P=\mathbf U\Sigma\mathbf V^\top$, where $\mathbf U,\mathbf V$ are orthogonal matrices, and $\Sigma$ is a diagonal matrix. We ensure the invertibility of $\mathbf P$ by forcing the elements of $\Sigma$ larger than zero. The optimization of orthogonal matrices can be readily accomplished with Cayley parameterization. Therefore, $\mathbf P^{-1}$ can be easily computed with $\mathbf V\Sigma^{-1}\mathbf U^\top$.

• Comparison with Direct Inversion. In Appendix B.1, we also compare this SVD training strategy with the naive strategy of directly inverting the matrix $\mathbf P$, which cannot ensure the invertibility of $\mathbf P$ theoretically. Although we empirically find that this naive strategy can still produce invertible matrices in the training process, it requires FP32 training to stabilize the optimization process. After thorough comparisons between these two strategies, we finally adopt the SVD training which enjoys both training stability and efficiency.

Q3. It is also puzzling that the paper does not report zero-shot performance for weight-only quantization. As shown in other works, LWC tends to cause overfitting, and the perplexity metric may not reflect true performance. I would be very grateful if more weight-only results could be provided.

A3. We thank the reviewer for the constructive suggestion. FlatQuant does not introduce significant overfitting on the calibration dataset and achieves consistent results across various downstream tasks.

• Below, we provide the detailed zero-shot performance of FlatQuant for weight-only per-channel quantization on LLaMA3-8B as a complement to Table 5.

---

Q4. OmniQuant only trains smooth parameters, while FlatQuant adds the inverse affine matrix, pre-quantization, SVD decomposition, and learnable activation clipping operations. Strangely, the paper reports that FlatQuant is 10 times faster than OmniQuant, which raises concerns. A more equitable and transparent comparison will be employed if a more detailed breakdown of the timing comparisons is provided, including information on the implementation details and the hardware used for both methods.

A4. We have reviewed our paper and confirmed that we did not claim FlatQuant is 10 times faster than OmniQuant. In fact, we believe the calibration time required by both methods is comparable, typically one hour on a single GPU for a 7B model (Line 339). Below, we explain in detail why FlatQuant does not introduce more calibration overhead compared to OmniQuant.

• Calibration Time Analysis. FlatQuant leverages the observation that Kronecker decomposed transformation matrices are similarly effective in flattening activations and weights. As shown in Figure 5, this approach achieves comparable perplexity results. Consequently, FlatQuant directly learns the decomposed transformations, making it computationally efficient, with minimal extra overhead during the calibration stage. Additionally, our implementation of SVD uses two orthogonal matrices and a vector to construct a transformation matrix. This implementation incurs an additional 9% calibration time compared to directly optimizing one matrix.

Dear Reviewer,

Thank you for the time and attention you've dedicated to reviewing our paper. As the discussion stage is coming to an end, please let us know whether we have addressed your concerns or if any further discussions are needed.

We look forward to your feedback and thank you once again for your valuable contribution to our work.

Best, Authors

Dear Reviewer X7ak,

We are still awaiting your response to our feedback. Specifically, we have demonstrated above that for W4-only quantization, both FlatQuant-GPTQ and FlatQuant-RTN-g128 can successfully retain 99% of the fp16 performance; FlatQuant also achieves considerable speed-up compared with SpinQuant even without the kernel fusion. We are eager to address any concerns you may have.

Best regards, The Authors of FlatQuant.

We note that the reviewer's concerns mainly focus on two aspects: (1) the relationship between flatness and quantization error, and (2) the computational overhead comparison with QuaRot and SpinQuant.

To address these concerns, we would like to clarify some key observations to resolve potential misunderstandings, and then respond to each question in detail.

1. Although FlatQuant does not explicitly optimize for flatness, our results in Figure 1 and 2 demonstrate that the learned transformations naturally flatten weights and activations.

2. We further quantify and plot the flatness changes during the learning process, as shown in the Appendix C.1 of the revised manuscript (Line 906~936). The flatness curves consistently and smoothly decrease as optimization progresses, indicating that FlatQuant promotes flatness, which is the underlying reason for FlatQuant's flat quantization error surface.

3. FlatQuant is designed lightweight, with computational overhead comparable to QuaRot and SpinQuant, both with and without kernel fusion.

4. We appreciate the reviewer highlighting the Kronecker decomposition and operator fusion in our method and their potential generalizability to methods like QuaRot and SpinQuant. This is one of our key contributions, and we are grateful for the recognition.

5. FlatQuant achieves a significant accuracy boost with only marginal inference overhead.

---

Q1. The flatness and quantization error expressed in the authors' paper do not strictly correspond. A simple example [3,4,5,6] has significantly higher quantization error at 2-bit than [0,0,0,sqrt(86)];

A1. We appreciate the reviewer's observation. While the example provided is mathematically valid, it represents an edge case that does not generalize to practical LLM quantization scenarios. For LLMs, millions of parameters in each layer rarely align perfectly, and a lower step size resulting from flatness offers significant benefits.

General Relationship Between Flatness and Quantization Error. A flatter distribution reduces the quantization range, which in turn minimizes the step size and directly decreases quantization error. In LLMs, the majority of weights and activations do not align perfectly with quantization grid points. Consequently, flatter distributions are essential for reducing expected rounding errors and improving overall quantization efficiency.

---

Q2. ... in fact, the authors only optimize the loss in Eq. (4) in their paper, and there is no direct correlation with flatness;

A2. We thank the reviewer for raising this point. While we do not explicitly optimize flatness, our results indicate the optimization of the simple layer-wise quantization loss defined in Equation 4 is enough to encourage flatness in weights and activations, which is the underlying reason for FlatQuant's flat quantization error surface.

• Empirical Evidence. Figure 1 and 2 demonstrate that the learned transformations naturally flatten weights and activations. This suggests that minimizing the layer-wise MSE loss promotes flatness.

• FlatQuant Leads to Flatness. To further address this concern, we have quantified flatness and plotted its evolution during optimization in Figure 8 with details in Appendix C.1 (Line 906~936) of the revised manuscript. The results show that the flatness of weights and activations (measured as their deviation from a perfectly flat distribution) improves smoothly as optimization progresses. This provides further evidence that the affine transformations of FlatQuant actually learn to flatten the distribution of weights and activations to lower the quantization error, even though we do not include flatness in the optimization target.

Q3.In the paper, the authors mainly compare the method with QuaRot and SpinQuant, and from my point of view, the way that authors compare is unfair, because the computational overhead of FLATQUANT is totally different from QuaRot and SpinQuant. Taking QuaRot as an example, only the rotation matrix in Query and Key and the rotation matrix after SiLU need to participate in the online computation in QuaRot, while all other matrices can be merged into the weights, while FLATQUANT has a large number of matrix multiplications that need to participate in the matrix computation of the floating-point, which introduces quite a lot of computational burdens compared to QuaRot, so I think the author's comparison is totally unfair. Meanwhile, the authors did not clarify this part of the paper;

A3. There can be some misunderstanding of our approach. A key idea in FlatQuant is to ensure the online transformations are lightweight. Despite more transformations are introduced in FlatQuant, its computational overhead is still comparable to that of QuaRot or SpinQuant. While QuaRot merges most matrices into weights, FlatQuant's optimizations, such as Kronecker decomposition (see Section 3.1), ensure that the runtime burden is effectively mitigated.

• Computational FLOPs. FlatQuant's online transformations contribute only 2.61% of the total FLOPs of FP16 LLaMA-2-7B, ensuring a low computational overhead, as detailed in Appendix C.5. While FlatQuant includes two more transformations than QuaRot (i.e., $\mathbf{P_a}, \mathbf{P_{ug}}$), they are lightweight, contributing only 23.10% of the total extra FLOPs for online transformations. This results in a negligible latency increase (0.08× without kernel fusion, 0.01× with kernel fusion), please refer to the table in A4 below for details.

• End-to-End Latency. In practice, the actual end-to-end speedup difference between FlatQuant and naive INT4 quantization is minimal (<0.10×) during both prefill and decoding stages, indicating that the introduced computational overhead is negligible.

• Fairness of Comparison. We compare the speed-up between FlatQuant (both with and without kernel fusion) and QuaRot in the tables of A4 below. The empirical results further validate that FlatQuant's computational overhead is on par with QuaRot without kernel fusion, and surpass QuaRot after kernel fusion.

Q4. While I strongly agree that the author's use of Kronecker decomposition and operator fusion reduces the runtime burden, which is also valid for QuaRot, the author's Speed Up comparison of Online Transformations in Figure.6 leads readers to mistakenly believe that FLATQUANT's accuracy-performance of the In fact, on the one hand, QuaRot's additional floating-point computation is less than FLATQUANT's, and on the other hand, the Kronecker decomposition and operator fusion reduces Hadamard's runtime burden under GEMM computation (instead of Walsh-Hadamard Transformation), so I think the comparison in Figure.6 is biased and misleading.

A4. We thank the reviewer for the positive recognition of our use of Kronecker decomposition and operator fusion. Below, we provide additional analysis to clarify potential misunderstandings about Figure 6:

• Speedup Without Kernel Fusion. We conduct a comparison of speedup without kernel fusion against QuaRot. The table below (also revised in Figure 6 of the revised manuscript) demonstrates that even without kernel fusion, FlatQuant achieves comparable speedup to QuaRot. This is primarily because the Kronecker decomposition and highly optimized GEMM kernel on GPUs.

• Contribution of Kernel Fusion. Incorporating kernel fusion (detailed in Section 3.3) further reduces FlatQuant's inference overhead, achieving a speedup gap of only 0.07× compared to naive INT4 quantization. We summarize the results in Figure 6 in the following table.

• WHT Has Similar Latency with GEMM. Despite WHT having fewer FLOPs than the vanilla matrix multiplication with GEMM, it can only be applied to cases where the size is an integer power of 2. In practice, we find that the fast_hadamard_transformation library used in QuaRot offers a similar speed-up compared to GEMM (e.g., both about 0.02 ms when the input shape is $2048\times 4096$). This is because the GEMM kernel is highly optimized on the low-level with parallelism on Tensor Cores.

• Other Limitations of Hadamard Matrices. Hadamard transformations are not well-defined for all sizes. Manually defining Hadamard matrices for special cases, such as the intermediate size of Qwen-2.5-7B, can be challenging and may hinder their applicability. Additionally, Hadamard matrices can limit the expressiveness of the transformation and thus affect the accuracy of quantization.

I also agree with official comment by Reviewer X7ak. FlatQuant should be compared with AffineQuant, DuQuant[1] (NeurIPS2024, so it is not necessary). Although I strongly agree with the effectiveness of the methodology, I still think the comparison of the author's experiment problematic.

I understand rotational invariance very well, and the enhancement of the author's method compared to rotational invariance is mainly brought about by the fact that the author did not follow rotational invariance.

Therefore, I want to give 4 score, however there is no choice. So I will retain my original score.

By the way, the author is provided triton kernel, not cuda. It seems both reviewer X7ak and tWUe make mismake.

[1] Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., Song, L., Sun, Z. and Wei, Y., 2024. Rotation and Permutation for Advanced Outlier Management and Efficient Quantization of LLMs. arXiv preprint arXiv:2406.01721.

Dear Reviewer,

Thank you for the time and attention you've dedicated to reviewing our paper. As the discussion stage is coming to an end, please let us know whether we have addressed your concerns or if any further discussions are needed.

We look forward to your feedback and thank you once again for your valuable contribution to our work.

Best, Authors

Dear Reviewer RQEg,

We are still waiting for your response to our feedback and open to any of your concerns.

Thanks! Authors of FlatQuant.

Q5. What's the perplexity improvement of pure learnable clipping threshold?

A5. We thank the reviewer for the insightful question. We have added ablation results for "LCT," "LT" (learnable transformation), and their combination ("LT+LCT") in the revised manuscript. The results demonstrate that while "LT" outperforms pure "LCT" by a large margin, their combination, combined with our clipping-after-transformation strategy, achieves the best performance.

---

Q6. In table3, in perplexity, whether the difference of SpinQuant and FLATQUANT is the learnable clipping threshold? Is the perplexity improvement mainly from pure learnable clipping threshold?

A6. Fair Comparison with SpinQuant (without Weight/Activation Clipping). We thank the reviewer for the valuable question. In the following, we make a fair comparison between FlatQuant and SpinQuant without weight or activation clipping. We reproduce experiments with FlatQuant and SpinQuant, without weight or activation clipping, under the same quantization settings (asymmetric activation and symmetric weight quantization). The results are summarized in the table below. Notably, SpinQuant relies heavily on GPTQ for maintaining accuracy, while FlatQuant achieves better accuracy with the RTN weight quantizer. This further highlights the superiority of FlatQuant.

Effectiveness of LCT. Besides, while SpinQuant does not use activation clipping due to marginal improvement, we find that learnable activation clipping, when combined with our method, leads to noticeable improvements as already discussed above.

---

Q7. In Figure5, why the speedup curve is not almost-symmetricly distributed? The speedup of 2048 is substantially slower than 2's.

A7. The asymmetry in the speedup curve arises from irregular memory access patterns, particularly when $n_1$ exceeds 64. Below is a detailed explanation:

• As discussed in Section 4.3, left-multiplying the activation tensor by $\mathbf P_1\in\mathbb R^{n_1\times n_1}$ results in irregular memory access, making it more computationally expensive than right-multiplying by $\mathbf P_2\in\mathbb R^{n_2\times n_2}$, even though both have identical FLOPs.

• In Figure 5, $n_1$ ranges from 1 to 2048. When $n_1 = 2048$, the activation tensor is left-multiplied by a $2048 \times 2048$ matrix and right-multiplied by a $2 \times 2$ matrix. This is significantly slower than the reverse case ($n_1 = 2$), where left-multiplication involves a $2 \times 2$ matrix and right-multiplication involves a $2048 \times 2048$ matrix.

We thank the reviewer for the valuable feedback. Below, we address the raised concerns in detail.

---

Q1. In perplexity, learnable transformation and learnable scaling of FLATQUANT are borrowed from existing works.

A1. We elaborate on the key differences between FlatQuant and previous works below:

• Kronecker Decomposed Matrices vs. Diagonally Dominant Matrices. While AffineQuant optimizes strictly diagonally dominant matrices, FlatQuant uses unconstrained affine transformations to enhance flatness, minimizing quantization loss. With Kronecker decomposition, FlatQuant reduces computational overhead, enabling transformations in all linear layers to minimize the quantization loss, unlike AffineQuant's selective application due to higher costs.

• Learnable Scaling After Transformations. A key difference of learnable scaling in FlatQuant is that it is applied after the pre-quantization transformations. This can help avoid damaging critical outliers during activation clipping (e.g., LLM.int8()) and further improves accuracy. The detailed discussions on the order of learnable scaling and pre-quantization transformation are provided in Table 16 in Appendix C.3, and re-discussed in the response below to Q3.

---

Q2. Learnable clipping threshold does not compare with existing CNN works like NWQ.

A2. We thank the reviewer for the suggestion regarding related works. While our focus is primarily on LLM quantization, we would still clarify the differences from NWQ below:

• Different Research Focuses. NWQ primarily addresses challenges related to gradient optimization in QAT, while FlatQuant focuses on PTQ by introducing fast and learnable affine transformations and learnable clipping thresholds to handle outliers.

• Orthogonality of These Two Methods. While NWQ could potentially complement FlatQuant in a training setting, it does not address the unique challenges of PTQ for LLMs, such as handling activation and weight flatness. We believe the scope of NWQ is largely orthogonal to FlatQuant.

---

Q3. ... also does not show detailed analysis on learnable clipping threshold why not causing important outliers clipped, as Llm.int8().

A3. We thank the reviewer for the inspiring question. FlatQuant preserves important outliers by applying the learnable clipping threshold (LCT) after the pre-quantization transformation. A detailed comparison of different clipping strategies is provided in Table 16 and Appendix C.3:

• "LCT before Transformation" consistently performs worse, as it directly clips important outliers in LLMs.

• "LCT after Transformation" performs significantly better because our fast and learnable affine transformations redistribute outliers across channels, making them less sensitive to clipping. Even after clipping, the inverse transformation can effectively recover the original outliers from the clipped activations.

---

Q4. In speedup, Kronecker Decomposition still involves online transformation and can not achieve pure int computation.

A4. We thank the reviewer for the insightful comment. Although Kronecker decomposition involves online transformations, the computational overhead is minimal:

• Minimal Overhead. Online transformations account for only 2.61% of the FP16 LLaMA-2-7B model's FLOPs, leading to a speedup gap of less than 0.10x compared to naive INT4 quantization (see Appendix C.5 Line 1075~1112 and Section 4.3).

• Accuracy Improvement. FlatQuant achieves less than 1% accuracy drop on LLaMA-3-70B with W4A4KV4 quantization, making the computational cost negligible given the significant accuracy gain.

Dear Reviewer,

Thank you for the time and attention you've dedicated to reviewing our paper. As the discussion stage is coming to an end, please let us know whether we have addressed your concerns or if any further discussions are needed.

We look forward to your feedback and thank you once again for your valuable contribution to our work.

Best, Authors

Dear Reviewer YdzG,

We are still wating for your response to our feedback, particularly the comparisons with NWQ.

Thanks! Authors of FlatQuant

We thank the reviewer for highlighting both the strengths and areas for improvement in our work. In the following, we answer each question with additional analysis or experiment results.

---

Q1. Figure 3 is confusing and needs better explanation. The caption should better explain the relationships between subfigures and clarify the flow of operations.

A1. We thank the reviewer for the thoughtful feedback. In the following, we provide a detailed explanation of Figure 3.

• Figure 3(a) illustrates different components of FlatQuant, including affine transformation on weights (marked in red), per-channel scaling (marked in green), online affine transformation on activations and (de-)quantization operators (both marked in blue). Only the blue-marked components introduce inference overhead; the others are merged into LLM weights during inference.

• Figure 3(b) demonstrates the inference dataflow and integration of FlatQuant in one LLaMA Transformer block, which is detailed in Section 3.2. For the self-attention module, $\mathbf P_a$ is applied before query, key, and value projections, while $\mathbf P_o$ is applied before the output projection. We use $\mathbf P_h$ and $\mathbf P_v$ for flattening key and value caches, respectively. Notably, $\mathbf P_v$ is merged into the value projection weights to reduce overhead. Similarly, in the feed-forward network, $\mathbf P_{ug}$ flattens the input for the up and gate projections, while $\mathbf P_d$ flattens the input for the down projection layer.

• Figure 3(c) provides the illustration of the fast and learnable affine transformation in FlatQuant, which is introduced in Section 3.1.

---

Q2. While the efficient implementation and speedup is impressive, theoretical complexity analysis is crucial for algorithm development. The evaluation lacks important theoretical analysis:

• No detailed computation complexity analysis;

• Missing information about total parameter counts (including decomposed transform matrices);

• No comprehensive memory consumption analysis for inference

A2. We thank the reviewer for the constructive advice. The computation complexity is analyzed in Appendix C.5. For parameter counts and memory consumption, we have included it in the revision (Line 1113~1118). Specifically,

• Computation Complexity. As detailed in Appendix C.5 "Total FLOPs of Online Transformations" (Line 1075~1112), the total FLOPs of the online transformations in a Transformer block amount to $8bsh_d\sqrt{h_d} + 2bsh_da + 2bsh_d^2/a + 4bsh_i\sqrt{h_i}$, where $h_d$ denotes the hidden dimension, $h_i$ denotes the intermediate dimension, $a$ denotes the number of attention heads, $b$ denotes the batch size, and $s$ denotes the sequence length. For LLaMA-2-7B, the FLOPs of online transformations only account for approximately 2.61% of the total FLOPs of the FP16 model when $s=2048$.

• Total Parameter Counts and Memory Consumption. We compute the parameter count of each online transformation below: (1) $\mathbf{P}_a: 2 (\sqrt{h_d})^2$; (2) $\mathbf{P}_o: a^2$; (3) $\mathbf{P}_h: (h_d / a)^2$; (4) $\mathbf{P}_u\ _g: 2 (\sqrt{h_d})^2$; (5) $\mathbf{P}_d: 2 (\sqrt{h_i})^2$. The total parameter count is $4h_d+2h_i+a^2+(h_d/a)^2$. The additional memory consumption during inference is $2(4h_d+2h_i+a^2+(h_d/a)^2)$ bytes, which only consumes about 0.11MB extra memory space for LLaMA-2-7B.

---

Q3. Citation gap for learnable clipping threshold work, such as "Duanmu H, Yuan Z, Li X, et al. SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models[J]. arXiv preprint arXiv:2405.06219, 2024."

A3. We thank the reviewer for the recommendation regarding related works. We have incorporated citations for SKVQ and other works on learnable clipping thresholds in Appendix A of the revised manuscript (Line 712~715).

Q4. Regarding the Kronecker decomposition: What is the perplexity performance without decomposition (direct training)?

A4. Compared with directly training full-size transformation matrix, Kronecker decomposition achieves similar accuracy on downstream tasks while significantly reducing inference latency.

• As shown in Figure 5, different decomposition sizes have minimal impact on perplexity. However, the choice of decomposition size significantly affects inference efficiency.

• To further validate the impact of decomposition on accuracy, we also conduct experiments on LLaMA-3-8B with and without decomposition. The results are summarized below.

---

Q5. Model generalization:

• How does the method perform on other model architectures (Gemini, Qwen, etc.)?

• Are there any architectural limitations or requirements?

• Would different model architectures require different decomposition strategies?

A5. FlatQuant is broadly applicable to different Transformer-based architectures without any architectural limitation or specific requirement.

• To further validate the generality of FlatQuant, we conduct experiments on the Qwen-2.5-Instruct models. FlatQuant achieves near-lossless quantization (e.g., only 0.21% accuracy loss on QA tasks for Qwen-2.5-Instruct-32B). The results on language modeling and QA benchmarks are summarized below and included in Table 11 of the revision.

Q6. How sensitive is the method to different initialization strategies?

A6. FlatQuant is generally robust to different initialization strategies.

• By default, the initialization of online transformations is done with random orthogonal matrices. Using five different random seeds for initialization, the standard deviations of WikiText-2 perplexity (PPL) and QA accuracy are 0.008 and 0.24%, respectively, demonstrating FlatQuant's robustness to random seeds.

• We further experiment with different initialization strategies, including initializing with identity matrices and random invertible matrices. The results below show that these initializations do not affect the results heavily.

---

Q7. What is the impact of calibration data selection?

A7. FlatQuant is robust to calibration data selection. Ablation studies presented in Appendix C.3 (Table 15) demonstrate consistent performance across various tasks with different calibration data. A summary of the results is provided below: