TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstruction

We sincerely thank you for providing such detailed feedback to improve our manuscript. The detailed response is attached below. Feel free to discuss it with us if you have any further questions that prevent you from accepting this work.

Q1: While the paper claims to establish a new state-of-the-art, SpinQuant, as discussed in the related work, appears to far outperform the proposed method. It would be beneficial if the authors could initialize their model from SpinQuant and then apply their method to demonstrate its effectiveness.

Ans: Thanks for the suggestion. First, we want to clarify that SpinQuant and QuaRot both use the rotation transformation to enhance the weight-activation quantization by removing activation outliers. The function of our work is more similar to GPTQ, which is used to optimize weights that can be combined with SpinQuant. To show our method can be combined with SpinQuant, we conduct experiments on LLaMA-3-8B W4A4KV16 using SpinQuant official learned matrices here and export it to the code framework that we experimented with, (which we found has slightly higher wikitext perplexity than their paper reported in our code framework, presumably due to SpinQuant uses one fewer online Hadarmard transformation than QuaRot). The results are shown below:

Q2: Line 141 mentions that GPTQ can not be used to improve other transformation-based methods. However, GPTQ can significantly enhance the performance of rotation-based transformation like SpinQuant and QuaRot.

Ans: We apologize for any confusion here. In our original text, we meant scale transformation only rather than rotation transformation. Admittedly, the text may cause some confusion to readers, therefore, we revise Section 3.1 (line 141) to properly describe the problem statement. We change it to “However, this Technique (GPTQ) makes it hard to improve scale-transformed models like AWQ” and add a footnote saying that “GPTQ can be combined with rotation-transformed models like QuaRot. We also compare it in Sec. 4.”

Q3: The rationale for using progressive freezing instead of a dynamic regularization term (i.e., AdaRound) for optimizing rounding values is not clearly articulated, nor is it convincingly demonstrated why this method surpasses AdaRound. The explanations in Lines 075 to 077 lack substantial evidence.

Q4: The paper does not compare its progressive rounding approach with existing rounding methods such as AdaRound, AdaQuant, and FlexRound, missing an opportunity to justify the need for progressive rounding.

Ans: Thank you for the question. We noticed that this concern was raised by other reviewers. Therefore we added a section in Appendix A to compare different rounding methods and illustrate why PAR is better than original rounding learning methods. We also refer you to our General Response to check our results for the rounding optimization ablation study. The results we demonstrate in General Response show that our rounding choice is better than the other two rounding methods both in layer-wise and block-wise reconstruction objectives. For instance, AdaRound with block-wise objective only obtains 9.05 perplexity on Wikitext2 while our method has 6.82 perplexity. In summary, our PAR consistently outperforms the other rounding methods regardless of objective. We think the reason is that we explicitly control the hardness of rounding variables through the progressive approach, which is better for handling the billion-level parameter space.

As for FlexRound, we have checked the paper but we found they focus on traditional CNN PTQ mainly. The LLMs results are done on OPT models with 8-bit quantization, which is different from the scenarios where we comparing. We also found out that FlexRound is not compatible with the code framework that we experimented with (LLMC). Given the limited time in rebuttal, we are not able to implement that method on our own. However, we do find that SignRound (Cheng et al., 2023) demonstrates that SignRound is able to achieve higher performance than FlexRound. In light of this observation, we believe that our method can outperform FlexRound on LLMs.

Q5: Lines 230 to 231 discuss the potential negative impacts of optimizing the scale parameter in quantization. However, experimental evidence is needed to substantiate that learning is detrimental, as the current argument does not conclusively link optimization to performance degradation.

Ans: Changing the scale parameter in the quantization step would fundamentally change the overall flooring results in Eq. (4), and thus will significantly affect the rounding variable optimization. As an intuitive example, assume the flooring result of $w_1/s$ is 2 and the rounding variable $\sigma(\nu_1)$ is optimized towards 1. In this case, by changing $s$, the flooring result could be 3 and thus fundamentally change the gradient direction of the rounding variable. As a result, the optimization process becomes unstable. This problem is stated in the original AdaRound (Nagel et al., 2020) work that scales should be fixed in the rounding optimization. To enable optimization, we adjust the dequantization scale factor, which is decoupled from the quantization step and does not require STE.

Q6: The paper optimizes significantly more parameters than previous methods using limited data, raising concerns about potential overfitting. A user study or chat test could provide essential validation.

Ans: Thank you for your advice. To demonstrate the capability of our model, we follow OmniQuant (Xiao et al., 2023) to compare the instruction-tuned models for chatbots. The models are evaluated on the Vicuna benchmark. We use LLaMA-2-7B-chat and compress it by W3A16g128 weight quantization. Using GPT-4 evaluation, our TesseraQ model has a 59% win rate against the OmniQuant checkpoint.

Q7: Line 473 states a significantly larger value than AWQ/OmniQuant, but it's unclear what this comparison signifies as these methods do not optimize rounding to flip variables.

Ans: Thanks for your question. This description validates our claim in the introduction section that to further improve existing scale-transformation and weight-clipping methods, adjustment to the weight tensor is necessary. The experiment here demonstrates that TesseraQ is able to optimize a huge amount of parameters in low-bit PTQ, which is why we improve over AWQ/OmniQuant. We meant to provide an intuition of what TesseraQ optimizes and also demonstrate the layer-wise difference in rounding optimization. Nevertheless, we agree that the original sentence may cause some confusion. We have revised the sentence to “This proves that weight tensor-wise adjustment can be used to significantly improve previous scale transformation adjustments like AWQ. ” to improve clarity.

Q8: The necessity of the sigmoid function in Equation (9) should be experimentally validated. Why not use $v\times s\times (\theta^q - z)$ with v initialized to 1.

Ans: Thank you for this question, we chose this option because we try to avoid more hyper-parameter tuning of the learning rate of this scale parameter. If we directly optimize the scale factor, it could go to zero or even negative values if the learning rate is not well-tuned, making the learning process extremely unstable, a similar reason why AdaQuant does not perform well on LLMs. Using sigmoid reparameterization, which is also applied to rounding variables, can apply the same learning rate for all variables. Besides, sigmoid reparameterization does not bring any cost in deployment, it can be safely merged into the original scale factor for the quantized checkpoint.

Q9: Further evaluation is needed on the effect of larger $t$ values in the expression $\frac{1}{exp(tx)}$ and other scheduling functions in the PAR Schedule to demonstrate the method's robustness to different schedules.

Ans: Thanks for the advice, we additionally experimented with $t=6$ and $t=7$ and updated the results in Figure 2. To summarize, the$t=6$ and $t=7$ slightly degrade the performance from the optimal $t$, with 0.2 higher perplexity results and 0.2% lower average accuracy. We emphasize that this performance variation is much smaller when compared to the gap between AWQ and TesseraQ performance. In practice, we apply the same PAR schedule to all experiments and observe consistent results. Considering the small performance variation between different t and the limited time given in the rebuttal, we can only conduct more experiments regarding different PAR schedules in the next revision of our paper.

Q11: The author does not provide results for the proposed method without using transformation-based initialization.

Ans: Thank you for the question. To clarify, transformation methods (like AWQ/OmniQuant) adjust weight distributions globally, while TesseraQ performs precise per-weight adjustments within a constrained range. These approaches are complementary - global adjustment first, then local refinement. Our goal is to demonstrate that a fine-grained adjustment after the global coarse adjustment can significantly improve performance. Without transformation-based initialization, our fine-grained weight adjustment approach may not optimize fully and yield sub-optimal performance. We would like to emphasize that TesseraQ is a booster on top of transformation-based initialization as mentioned in the contribution of the paper.

Q12: Why didn't the authors compare their method with OmniQuant for weight-activation quantization?

Ans: Thank you for your question. Originally we did not put it since we only have LLaMA-3 model results and it is time-consuming to implement OmniQuant on them. As suggested by some reviewers, we realized it is not comprehensive enough to only include LLaMA-3 models. Therefore, in the newly updated Table 3, we include the OmniQuant performance on LLaMA-1/2 as well. As can be seen, previous approaches, such as AWQ, OmniQuant, QLLM, and OS+ all obtain similar results (49~51% average accuracy). In comparison, our method (initialized on AWQ) significantly improves the W4A4 performance by a huge margin (55% average accuracy).

We want to thank reviewer wxS8 for his/her quick reply. While we are actively running the additional experiments, we'd like to clarify the problem with SpinQuant first. The SpinQuant rotation setup is not the same as that of QuaRot. Indeed, R1 and R2 are learned, and R3/R4 are inserted as online Hadamard rotation. However, note that R4 is only inserted for KV cache quantization, which is both implemented here in SpinQuant and in QuaRot. When applying W4A4KV16 quantization, which is the case in our rebuttal experiment, R4 is not inserted.

The difference between QuaRot and SpinQuant lies in R3 online transformation, where QuaRot applies partial online Hadamard Transform to input of V_proj layers, and applies full Hadamard rotation matrix to O_proj layers weights here, while SpinQuant does not insert the online transformation to V_proj and rather applies rotation to O_proj weights and V_proj weights both in head dimension here.

In our experiments, we directly save the FP16 checkpoint after the SpinQuant rotation matrix here to ensure we have the accurate FP16 checkpoint and then run experiments under our framework. We believe our results already use the full performance of SpinQuant.

I have checked the paper (v2 of SpinQuant, the latest version before iclr submission deadline). R4, inserted for down_proj is indeed employed for w4a4kv16 and R3 is only used for the kv cache quantization. The names of the rotation matrix in the clarification given by the authors are different from the SpinQuant paper (v1, v2, and v3), which makes me feel so confused. Therefore, I hope to see the results for SpinQuant with R1, R2, and R4 employed.

By the way, the authors claim that their results already used SpinQuant's full performance, which is not convincing. Specifically, R1+R2+R4 achieves 7.1 wikitext PPL for w4a4kv16, which I think is the ideal performance. Additionally, through only employing R1 and R2, the PPL achieved by the authors is about 6 points higher than the paper (v3). I ran SpinQuant's original code two months ago and can get a similar performance in its paper.

We appreciate reviewer wxS8’s detailed comment on TesseraQ. Before we answer your questions, we would like to strengthen several points.

1. The reviewer kept claiming we made an unfair comparison of related works like SpinQuant. However, any mismatch in SpinQuant+GPTQ also exists on our SpinQuant+TesseraQ. We did not take any advantage of setting mismatch in SpinQuant. The same logic applies to QuaRot with activation clipping. The experimental settings of QuaRot, QuaRot+GPTQ, and QuaRot+TesseraQ are the same. Thus the universality of experimental settings across different methods makes our comparison and results fair.

2. Based on the reviewer's question, it seems like the reviewer remains curious about the performance of TesseraQ under all prior transformation-based methods available online. However, we have tested in many cases (AWQ, OmniQuant, QuaRot, SpinQuant, No initialization). In all cases, we have demonstrated TesseraQ’s ability to consistently improve the existing method’s quantization. Compared to other weight quantization work, we have done a more comprehensive analysis. We believe our experiments in the original paper and post-rebuttal in the revised version are sufficient and demonstrate TesseraQ’s performance-boosting capabilities on prior methods.

3. It seems like the reviewer focuses on several cases but ignores our other major experiments. In Table 1/2, we have demonstrated a significant improvement in 2-bit weight-only quantization. Yet the reviewer maintains a clear rejection opinion because we lack the complete version of weight-activation transformation results in Table 3, which is orthogonal to our major contribution that focuses on weight-only quantization. More importantly, we have established TesseraQ’s superior performance compared to prior works like QuaRot, OmniQuant, and AWQ in the pre-rebuttal submission. In post-rebuttal, the reviewer requested to add SpinQuant results. We would like to note that SpinQuant is an arXiv paper, not yet published, and the reviewer’s criticism about our paper’s results being unfair based on an arXiv work is not appropriate. Having said that, we have recreated experiments from SpinQuant with our framework and have provided sufficient justification and qualitative/quantitative comparison. Across all experiments, we have shown the TesseraQ’s effectiveness.

Thanks again for the concise reply. I do not have any further questions. The authors have clearly made a substantial effort to conduct thorough analyses and experiments to refine their method. Considering the large volume of LLM quantization papers published this year, I find that the most notable contributions are those that provide clear, practical enhancements in terms of performance or speed. Many papers claim state-of-the-art results; therefore, it is crucial to compare this method against their full versions to ascertain its true value. Although a plug-and-play method might improve existing models, its effectiveness could be limited if the full versions already deliver strong results. Furthermore, the results reported in the paper show significant divergence from previous studies. For instance, 1) the similar Perplexity (PPL) performance of SpinQuant+GPTQ versus SpinQuant+TesseraQ does not demonstrate improvements, as discussed in numerous papers. Additional testing on more datasets and models (similar to the full version of QuaRot) is necessary to fully evaluate its impact. 2) The authors have chosen not to use activation clipping for QuaRot-RTN, and the results including activation clipping also diverge significantly from those reported in its paper. The PPL for LLaMa-2-7B is 13.01 reported by the authors in the rebuttal phase, which is approximately 5 points higher than the figure in Table 3 of QuaRot's latest version, where QuaRot-RTN even incorporates KV cache quantization. A similar discrepancy is observed with downstream tasks, e.g., about 6 points lower on PIQA.

Considering that there may be other results in the paper that exhibit the same issues mentioned above (I have no time to check one-by-one), and given that the rebuttal does not allow for major revisions, I do not recommend the paper for acceptance at ICLR at this time. I suggest addressing the aforementioned problems, particularly aligning the results with previous studies and replacing the results of baselines with those of their full version, which I believe are serious issues. Additionally, providing a detailed description of the experimental setup would help clarify the methodology and strengthen the paper's contribution.

Note: Apologies for requesting a comparison with SpinQuant (submitted to arXiv in May), but I could not find the specific policy regarding this matter on the ICLR website. It is worth mentioning that this helps me observe the above concerns.

We thank the reviewer for the constructive comments and suggestions. We hope our response will address your concerns, see detailed comments below.

Q1: 3.2 Some expressions are not very clear, and some variables have not been explained.

Ans: We apologize for any unclear expressions or unexplained variables. We would like to explain the core idea of Progressive Adaptive Rounding (PAR) here. The main idea of PAR is to optimize the binary rounding variables effectively. To tackle this problem, we relax them into continuous variables by applying the sigmoid function, so that these continuous variables can be updated using block reconstruction loss and gradient descent. However, we expect all variables to be binary after optimization, therefore, we propose a progressive approach, that hardens partial variables at a time and then optimizes the remaining soften variables for loss optimization. The learned rounding variables are 0/1, which can be merged into original weights, and then we can maintain the original quantization function for deployment.

We are happy to improve any expressions that the reviewer finds unclear.

Q2: The new rounding error optimization method proposed in the paper mentions that compared to previous methods (https://arxiv.org/pdf/2004.10568) does not rely on regularization loss, but HS and sorting were introduced. It doesn't have a significant advantage. For example, computational complexity and convergence speed.

Ans: Thank you for the question. In terms of computational complexity, our method is indeed similar to the original AdaRound. As for training convergence, we argue that our PAR is better when applied to LLMs. In Appendix A, we add a new set of ablation studies on different rounding methods, including the original AdaRound and the STE-based optimization, AdaQuant. Our results demonstrate that PAR is the best choice in LLM PTQ.

Q3: The paper combines the proposed rounding error optimization method with other methods for experimentation and comparison with other quantization methods, but does not use other rounding error optimization methods for the same experiment, and does not demonstrate the superiority of this method. For example, the Adaround mentioned in the article.

Ans: Thanks for the question. Again we refer you to our General Response and Appendix A for a fair comparison where we initialize all experiments with the same AWQ checkpoint. The results we demonstrate in General Response show that our rounding choice is better than the other two rounding methods both in layer-wise and block-wise reconstruction objectives. For instance, AdaRound with block-wise objective only obtains 9.05 perplexity on Wikitext2 while our method has 6.82 perplexity. In summary, our PAR consistently outperforms the other rounding methods regardless of objective. We think the reason is that we explicitly control the hardness of rounding variables through the progressive approach, which is better for handling the billion-level parameter space.

Q4: When comparing quantitative indicators such as W4A4/W3A3 that include activation, the QuaRot initial method was used, but it was not compared with using the QuaRot method alone. The advantage of the proposed new rounding error optimization method in improving accuracy cannot be clearly demonstrated. Suggest to use QuaRot as a baseline separately to include the results.

Ans: Thank you for your suggestions. In Table 3 we already provided the QuaRot baseline results. To summarize, we show that TesseraQ + QuaRot is better than GPTQ + QuaRot, which is again better than QuaRot itself. Moreover, we update Table 3 to include LLaMA-1/2-7B models as well to show a more comprehensive comparison. For example, the QuaRot results for LLaMA-3-8B are here:

thank for your reply， I keep my score

Thank you for your positive feedback on our work, we will try our best to address your questions, please check our detailed reply below.

Q1: The paper proposes PAR instead of the common STE implementation. There is a lack of justification why direct utilization of STE for the rounding policy estimation would result in worse performance so PAR is significant in the overall design.

Ans: Thank you for the suggestion. We noticed that this is a general question raised by multiple reviewers so we conduct experiments in Appendix A, please check our general response. Regarding the common STE implementation, we assume the reviewer is asking for AdaQuant (Hubara et al., 2021). We agree that STE, in theory, has higher potential since they do not limit the optimization space to binary choices. However, the problem with AdaQuant is its unstable tuning. AdaQuant directly finetunes the floating-point weights, which is trained via discrete quantization function in forward calculation and STE in backward calculation. Consequently, AdaQuant faces instant change in forward and estimated gradient in backward, which becomes extremely sensitive to the learning rate selection and may require different learning rates for different layers/blocks in the model.

In AdaRound original paper (Nagel et al., 2020) Table 5, the authors compare their method with STE and find out AdaRound outperforms the STE approach. Similarly, in our reproduced experiments in Appendix A, we found that AdaQuant obtains extremely unstable results.

Q2: The paper purposes an effective rounding optimization add-on for enhancing the block-wise quantization for LLM, but there is a lack of theoretical analysis of the different between: A) BLOCK-WISE & Normal Rounding (round to the nearest) and B) the proposed BLOCK-WISE & Rounding Optimization. If A) is well-trained and fully converged, would it theoretically have similar mathematical foundations as B)?

Ans: Thanks for your question. We’d like to clarify that option (A) Block-wise & Normal Rounding can be regarded as the same methodology in OmniQuant (Shao et al., 2023). They calibrate the transformation scales as well as the weight clipping range using the BLOCK-WISE objective while maintaining the normal rounding function. As shown in our experiments, especially in Table 1 first several rows, TesseraQ optimized on OmniQuant checkpoint (option B) clearly improves the original OmniQuant performance (option A) in W2A16 quantization. This indicates that B) has higher potential than A). For mathematical foundation explanations, we also refer you to our reply to your Q4, which uses an simple example to demonstrate effectiveness of rounding optimization.

Q3: Why does the paper adopt Progressive Adaptive Rounding (PAR) instead of the more common Straight-Through Estimator (STE) for rounding policy estimation? What specific limitations or performance issues with STE justify the need for PAR in the overall design? If STE is applied, whether such progressive-stage training is still required?

Ans: Please check our answer to Q1. In short, STE-based calibration is less stable and requires careful hyperparameter tuning. On the other hand, our progressive approach always optimizes continuous soft variables and explicitly controls the overall hardness. Therefore, our method is easier to optimize in practice.

Q4: Could the authors justify the theoretical basis for the improvement achieved by rounding optimization over the baseline?

Ans: Thank you for your question. Here we use an example from AdaRound (Nagel et al., 2020) to theoretically demonstrate why rounding-to-nearest (RTN) is not optimal. The RTN approach essentially minimizes the weight space distance while we need to consider the loss function space minimization. In PTQ, this can be realized by applying Taylor expansion to the loss function so that our actual objective is $\Delta W^\top H^W \Delta W$, where $\Delta W$ is the distance between quantized weight and FP weight and $H^W$ is the Hessian matrix with respect to weights. Now consider a two-variable example $W=[w_1, w_2]$ and the Hessian matrix $H^W = [[1, 0.5], [0.5, 1]]$. In such a case, the overall objective becomes $\Delta w_1^2 + \Delta w_2^2 + \Delta w_1\Delta w_2$. We can find that the term $\Delta w_1 \Delta w_2$ requires a different sign of $\Delta w_1$ and $\Delta w_2$ to minimize the overall objective. In this case, rounding to the nearest may not give optimal results if $\Delta w_1 \Delta w_2$ is positive. Therefore, finding the optimal rounding choice to minimize the objective is necessary. For more details, we refer the reviewer to the original AdaRound paper.

Thanks for the authors' careful and detailed responses.

I have tried the purposed PAR instead of direct STE in one of my previous project where also a series of binary 0 or 1 is expected from the model output. (If it was implemented correctly) I could see a little slowdown of the convergence time of my model (probably due to the progressive design) but the the output series is a more stable than my previous STE implementation where some batches won't not result in a relatively-huge changes in the output behavior.

I think the most of my concerns are taken care of by the authors. I also have gone through carefully all the comments and reviews, thus I decide to maintain my score.

Thank you for your effort again.

Q4: The technical novelty is relatively incremental. Both block reconstruction and scale tuning are well-established in PTQ literature. While the paper makes useful improvements, it builds primarily on existing techniques rather than introducing fundamentally new concepts.

Ans: We would like to clarify the contribution of our work. Although the block-reconstruction and rounding optimization framework is proposed in the literature, few works have successfully implemented them on LLMs due to the challenging nature of the billion-level parameter space. Our proposed framework demonstrates that rounding optimization, if properly implemented with our progressive approach, can be used to significantly improve the LLM quantization performance. In the revision, we also add experiments to show the effectiveness of PAR over traditional weight optimization methods like AdaRound and AdaQuant. We have also revised Section 3.1 to emphasize this point in our problem statement.

Q5: The choice to restrict rounding to ±1 steps seems limiting. Why not allow multiple step sizes as in AdaQuant? This decision could impact optimization flexibility, especially for ultra-low bit quantization where precision is crucial. Has this design choice been empirically validated against alternatives?

Ans: Thank you for your question. AdaQuant, in theory, has higher potential since they do not limit the optimization space to binary choices. However, the problem with AdaQuant is its unstable tuning. AdaQuant directly finetunes the floating-point weights, which is trained via discrete quantization function in forward calculation and STE in backward calculation. Consequently, AdaQuant faces instant change in forward and estimated gradient in backward, which becomes extremely sensitive to the learning rate selection and may require different learning rates for different layers/blocks in the model.

In AdaRound original paper (Nagel et al., 2020) Table 5, the authors compare their method with STE and find out AdaRound outperforms the STE approach. Similarly, in our reproduced experiments in Appendix A and General Response, we found that AdaQuant obtains extremely unstable results. Consequently, we use a lower learning rate but obtain less optimized results.

Q6: The effectiveness with weight transformation techniques is unclear. Specifically, would TesseraQ maintain its benefits when applied after Hadamard matrix transformations (as in QuaRot)?

Ans: Yes, TesseraQ follows the standard uniform weight quantization rules after merging the learned rounding variables. When combined with QuaRot, we learn the rounding choices for the Hadamard transformed weight $\mathbf{H}\mathbf{W}$ and maintain the original benefits. After TesseraQ+QuaRot, we can use the same original QuaRot inference pipeline, i.e., uniform weight-activation quantization plus online FP32 hadamard transformation.

[1] Maddison, Chris J., Andriy Mnih, and Yee Whye Teh. "The concrete distribution: A continuous relaxation of discrete random variables." ICLR 2017.

Thank you for acknowledging the clarity and significance of our work. We appreciate your comments and would like to address your concerns below.

Q1: The paper lacks detailed technical comparisons with similar methods. While mentioning prior approaches like AdaRound and AdaQuant, it doesn't experimentally demonstrate why the proposed methods are superior to these methods. The rationale behind several design choices (sigmoid reparameterization; harden score metric in Eq 6; avoiding scale optimization in the quantization step) isn't fully justified through comparative experiments.

A1: We understand your concern regarding some detailed design choices of our work. First, regarding the comparison to AdaRound/AdaQuant, we add an ablation study of rounding optimization choices in Appendix A. Please check our general response and the updated manuscript for more details.

Second, other design choices have mostly been addressed in the existing literature. We explain the insights here. Regarding the sigmoid reparameterization, this is the standard relaxation step in {0, 1} binary space which is essential to differentiate the rounding variables and provide a smooth loss landscape [1]. For the reason of avoiding scale optimization in the quantization step, we did this since changing the scale would fundamentally change the overall flooring results in Eq. (4), and thus will result in a significant change in rounding variables gradients. This problem is stated in the original AdaRound (Nagel et al., 2020) work that scales should be fixed in the rounding optimization. To enable optimization, we adjust the dequantization scale factor, which is not affected by quantization operation and does not require STE.

Last, we agree with the reviewer that the Harden score (HS) metric could be interesting to study. We conduct experiments to compare the random selection of Harden variables and the selection based on the HS score. The results are shown below

Q2: The inconsistency in model selection between Tables 1, 2 and 3 makes it difficult to draw comprehensive conclusions. Q3: The paper would be stronger with complete results for newer models like LLaMA-3 across all experiments.

Ans: Thank you for your advice. We have updated Table 1 to include LLaMA-3 models and more baseline results like SignRound, and GPTQ with QuaRot. Meanwhile, for weight-activation quantization comparison, we updated Table 3 to add LLaMA-1/2-7B so that all LLaMA series are evaluated. It is worth noting that our method still outperformed other methods on these newly tested models. For example, in Table 3, we reported that our method achieves 55% average accuracy of LLaMA-2-7B W4A4 quantization, while all existing scale-transformation methods like QLLMs, AWQ, OS+, OmniQuant obtain 49~51% average accuracy.

Q3: Runtime and GPU memory analysis is limited to the 7B model. As the runtime and memory consumption are important metrics, the authors should provide comprehensive efficiency metrics across different model sizes.

Ans: Thank you for this suggestion. Here we provide a detailed Runtime comparison and GPU memory on LLaMA-2-7B/13B/70B models when applying TesseraQ. Overall the GPU resources, and algorithm runtime scales with the calibration dataset size. For the 70B model, we can apply CPU offloading to block input/output data to save extra memory, at a little bit cost of extra runtime.

*denotes apply CPU offloading to block input/output data

Thank you for reviewing our work and providing useful suggestions. Please check our detailed reply to your questions/comments.

Q1: The writing in Section 3.2 of the paper is not sufficiently clear, making it somewhat difficult to read and understand.

Ans: We apologize for any confused expression or equation in Section 3.2. We would like to explain the core idea of Progressive Adaptive Rounding (PAR) here. The main idea of PAR is to optimize the binary rounding variables effectively. To tackle this problem, we relax them into continuous variables by applying the sigmoid function, so that these continuous variables can be updated using block reconstruction loss and gradient descent. However, we expect all variables to be binary after optimization, therefore, we propose a progressive approach, that hardens partial variables at a time and then optimizes the remaining soften variables for loss optimization. The learned rounding variables are 0/1, which can be merged into original weights, and then we can maintain the original quantization function for deployment. We also modified the methodology section to improve its clarity. Feel free to suggest detailed modifications that we can make.

Q2: Outdated Comparison: The comparison methods in Table 1 appear to be somewhat outdated. The authors should compare with more advanced weight-only methods to demonstrate its superiority.

Ans: Thanks for providing the feedback on Table 1. We did not report some baseline results since they missed some results of either LLaMA-1 or LLaMA-3. We are happy to provide a more detailed comparison with some most recent works in Table 1. We now add LLaMA-3-8B/70B quantization results and 2 additional baseline results, the SignRound and the GPTQ with QuaRot. Overall we find our method still outperformed additional baselines by a large margin, especially on smaller models and lower bitwidth. For example, on LLaMA-3-8B W2A16g128 quantization, our method obtains 10.03 perplexity while AWQ is 334 and the GPTQ with QuaRot is 17.4.

Q3: Besides, other tables comparison seems inadequate. The authos should also report the comparison results with other Rounding method, such as Adaround and signRound.

Ans: Thanks for your suggestion. We agree that a rounding method comparison would be beneficial. We provided a new section in Appendix A to compare other rounding methods. We also explain our revision, please check our General Response. SignRound is now compared in Table 1 as well. For example, on LLaMA-2-13B W2A16g128 quantization, SignRound has 7.68 perplexity while our TesseraQ has 5.92 perplexity.

Q4: The author should discuss the method's limitation.

Ans: One potential negative effect of TesseraQ would be its relatively higher calibration time compared to other layer-wise optimization methods. In Section 4.3, we report the calibration data size’s impact on accuracy and algorithm runtime. The LLaMA-2-7B model takes 3~6 hours depending on the data to finish the rounding optimization, which is larger than AWQ/GPTQ which is around 0.5 hours. However, we emphasize that TesseraQ is still faster than end-to-end quantization-aware training which requires significantly more GPU resources to compute and store gradients. We will leave accelerating the rounding optimization as our future development goal and discuss it in our conclusion section.

Q5: Why the author didn't compare with Adaround and signRound in Table1 and other experiments?

Ans: We apologize for missing SignRound results in Table 1, which has been added for comparison right now. Notably our method is significantly better than SignRound in perplexity metric. For example, in LLaMA-2-7B W2g128 quantization scheme, our method obtains 6.82 perplexity while the SignRound method crashed and got NAN value.

For AdaRound, it is designed originally for convolutional-based networks. As pointed out in GPTQ (Frantar et al., 2022), traditional rounding optimization methods are complex and challenging to scale to billions of parameters. In fact, there is no official AdaRound implementation for LLM, and SignRound (Cheng et al., 2023) provided their own implementation results in AdaRound, where they demonstrate that SignRound is able to achieve higher performance. In light of this observation, we believe that our method is able to outperform both rounding optimization methods. In addition, our new ablation study in Appendix A also confirms this observation. Please check our General response and the new Appendix A for more details.

Q6: Compared with the traditional Layer-wise Rounding method Adaround, what is the superiority of the proposed TesserQ, is the block-wise reconstruction granulity or the Harden and Soften design? Why? The author should add more related discussion and results to analysis it.

Ans: Thank you for the suggestion. To answer your question, both block-wise reconstruction granularity and PAR algorithm choice contribute to the final performance. The results we demonstrate in General Response show that our rounding choice is better than the other two rounding methods both in layer-wise and block-wise reconstruction objectives. Additionally, block-wise objective works better than layer-wise objective as it considers the attention and MLP inter-layer activity. Nevertheless, even with the block-wise objective, the rounding optimization method is essential to improve the performance. For instance, AdaRound with block-wise objective only obtains 9.05 perplexity on Wikitext2 while our method has 6.82 perplexity. In summary, our PAR consistently outperforms the other rounding methods regardless of objective. We think the reason is that we explicitly control the hardness of rounding variables through the progressive approach.

Thank you very much for the authors' response, which addressed most of my concerns. I have carefully read the comments from other reviewers, particularly the discussion from reviewer wxS8. I share similar concern regarding the supplementary compared experiments of the SpinQuant method, mainly focusing on the misaligned accuracy comparisons, which leaves me with some doubts. I acknowledge the contribution of this paper to LLM quantization from the perspective of rounding; however, due to the incompleteness of some experimental results, I maintain my original score.