OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models

Q7: In the experiments, the authors mention retaining the Softmax output at full-precision owing to its long-tail distribution. It would be insightful to know the implications of quantizing the Softmax output to 8-bit. How does this quantization impact the overall model performance and accuracy?

A7: We have conducted experiments on quantizing the Softmax output, reporting the impact on model performance. As shown in the following table, we can find that quantizing the softmax into 8-bit and 6-bit brings acceptable performance degeneration, which demonstrates that block-wise calibration can compensate for the loss of 8-bit and 6-bit Softmax quantization. However, 4-bit Softmax quantization brings significant performance loss, which requires further exploration and additional trick such as log2 quantization in RepQViT[1]. We have added this experiment in Table A7 of our revision.

Q8: Sensitivity to channel-wise scaling and shifting initialization.

A8: Good advice, sorry for the missing of this ablation. As shown in the following table, we can find that both careful initialization of scaling and shifting can improve the final performance. Specifically, scaling initialization is more important than shifting due to scaling plays the main role in alleviating outliers. We have added this experiment in Table A6 of our revision.

We sincerely appreciate the time and efforts you have dedicated to reviewing our paper. Should you have any further inquiries, please let us know and we are more than delighted to discuss with you and run more experiments for any pieces of your interests in our work.

[1] Repq-vit: Scale reparameterization for post-training quantization of vision transformers

Q4: Additionally, an investigation into whether an iterative application of LWC and LET would yield performance improvements could provide valuable insights.

A4: We want to emphasize that in our approach, LWC and LET are trained simultaneously, and we have also explored an iterative training approach by iterations or epochs. The results, as presented in the following table, clearly indicate that training LWC and LET simultaneously yields the best performance. As aforementioned in Q3, LWC leverages the extremums after LET as proxies, allowing it to capture the dynamic changes in the weight range introduced by LET. This synergy between LET and LWC creates a progressive process, where both techniques reinforce each other rather than interfere. To further support this statement, we conducted an additional experiment (last row in the table), training LWC and LET iteratively with double the epochs. The results show that simultaneous training with 20 epochs achieves comparable performance to iterative training with 40 epochs. This demonstrates the effectiveness and efficiency of training LWC and LET simultaneously. We have added this experiment in Table A8 of our revision.

Q5: Visualization of $\gamma$ and $\beta$ distribution.

A5: Indeed, we have plotted the distribution of learned \gamma and \beta in Figure A1 and presented the analysis in Section A5 in the Appendix. Through the visualization in Figure A1, we can find that LWC can learn more pronounced clipping when meeting with lower quantization precision (lower bits or greater group size). Such a phenomenon makes sense due to that lower quantization precision has smaller representation space, and consequently is expected to clip more weights to obtain better trade-offs.

Q6: An investigation into the individual and cumulative impact of LET on each linear layer could provide deeper insights into the efficacy and necessity of LET across different layers of the model.

A6: Good question. In our paper, we exclude the LET of the second linear layer due to the high sparsity of features after the non-linear layer leads to unstable gradients. Therefore, we have four LET pairs, represented as [ln1, (q_proj, k_proj, v_proj)], [v_proj, out_proj], [Q, K], and [ln2, fc1]. To validate the effect of each part, we respectively remove the LET from each layer and report the average perplexity and the average accuracy on 6 zero-shot tasks like Table 2. As shown in the following table, we can find that all four LETs can improve the performance, specially for the [ln1, (q_proj, k_proj, v_proj)] pair. Such results also demonstrate that the activation outliers are more serious after layer normalization layers. We have added this experiment in Table A5 of our revision.

We sincerely appreciate the time and efforts you have dedicated to reviewing our paper. Should you have any further inquiries, please let us know and we are more than delighted to discuss with you and run more experiments for any pieces of your interests in our work.Reviewer PTbL

We appreciate your valuable insights and suggestions. We have summarized the updating of our revision in Summary of Paper Updating and provided an in-depth discussion about the novelty of our method in Genera Response. Additionally, we have made the following improvements to address your concerns.

Q1: The major weakness of the paper is the noticeable absence of experimental comparisons with Outlier Suppression+ (OS+) (Wei et al., 2023).

A1: We have added the comparisons with OS+ in Table 2 of our manuscript. A comparison overview is as follows.

Q2: Distinctions with Outlier Supression+.

A2: OmniQuant differs from Outlier Suppression+ in several aspects, including parameter optimization through gradient updates, block-wise optimization, expansion of equivalent transformation to attention operations, and the introduction of learnable weight clipping (LWC). Details are as follows.

• Both OmniQuant and OS+ try to find the optimal equivalent transformation parameters. OS+ solves this problem by pre-defined shifting strength and grid-searched scaling parameters. However, OmniQuant obtains both shifting and scaling parameters through gradient optimization.

• OmniQuant leverages block-wise optimization while OS+ executes grid searching through a mix of single linear layer and multiple linear layers objective(Eq.6 and Eq.7 in their paper). Block-wise optimization considers interaction within a block and can produce better performance as demonstrated in BRECQ. However, it is a challenge for OS+ to expand to block-wise quantization, which would significantly enlarge the solution space and increase the time of grid searching.

• LET expends the equivalent transformation to attention operation (Eq. (5) in our paper), but OS+ only does the equivalent transformation on linear layers.

• OmniQuant also introduces learnable weight clipping (LWC), which further facilitates the low-bits (2,3) weight-only quantization, while Outlier Suppression only supports weight-activation quantization.

We have added this discussion in Section A2 in our revision. You can also refer Genera Response for more discussion about the novelty of our paper.

Q3: The proposed LWC learns a clipping strength instead of clipping threshold in PACT (Choi et al., 2018) and LSQ (Esser et al., 2019). However, the paper lacks a clear articulation of the advantages of LWC over PACT and LSQ, particularly in scenarios where it is combined with LET and weights are frequently changed. A more thorough explanation of the benefits and underlying mechanics of LWC in such contexts would be beneficial.

A3: To clarify the advantages of LWC, we have included Figure A5 in the appendix, illustrating the dynamic change in weight ranges during training with different clipping-based methods. The figure emphasizes that LET, by dramatically altering weight ranges, poses a challenge for traditional absolute clipping thresholds or step size learning. In contrast, LWC calculates relative clipping strength, utilizing extremums after LET as proxies. This approach enables LWC to adapt to changing weight ranges more naturally, addressing the optimization of difficulty introduced by LET.

Dear Reviewer PTbL,

We hope that our response can address your concerns. As the deadline for discussion period is approaching, we really appreciate if you can let us know whether there still exists any further question about the paper or the response. We are looking forward to further discussion.

Best regards,

Authors of Paper 1585

Q4: Authors should present Figure 1 (a) in a quantitative way. It lacks actual numbers for cost of time and performance.

A4: Thank you for the suggestion. We have incorporated quantitative results into Figure 1(a) to provide a more comprehensive understanding of the cost of time and performance.

Q5: Learnable Equivalent Transformation (LET) isn't novel and can be seen in common quantization kernels such as LLM.int8. I suggest authors to elaborate more on comparison with PACK and LSQ instead of LET.

A5: As mentioned in Q2, LET is the first one to introduce gradient optimization into the equivalent transformation of large language models, which significantly pushes the limitation of low-bits quantization. To demonstrate the superiority of the proposed method, we have compared it with state-of-the-art weight-only (GPTQ, AWQ) and weight-activation (LLM-QAT, Outlier Suppression +) quantization methods in our paper. Additionally, we have transferred PACT and LSQ to the weight quantization in Table A14, showcasing that LWC outperforms previous clipping-based methods. As for the activation quantization, we leverage MinMax quantization due to that directly leverages LSQ in activation quantization leading to poor performance than MinMAx quantization as shown in Table 4 of the LLM-QAT paper. You can also refer Genera Response for more discussion about the novelty of our paper.

Q6: The latency benchmark only contain fp16 and OmniQuant. Strong baseline such as GPTQ is lacking. How does OmniQuant runtime efficiency compare to GPTQ and AWQ?

A6: OmniQuant, GPTQ, and AWQ are all uniform quantization methods, allowing their quantized models to be deployed with the same latency. In our paper, the weight-only quantization kernel is not our primary focus, and we leverage existing MLC-LLM to validate the practical deployment of our proposed methods. As shown in the following table, we present the latency compared between MLC-LLM, GPTQ kernel, and AWQ kernel. Note that all of these three manners can be used to deploy OmniQuant’s quantized models.

We sincerely appreciate the time and efforts you have dedicated to reviewing our paper. Should you have any further inquiries, please let us know and we are more than delighted to discuss with you and run more experiments for any pieces of your interests in our work.

Thanks for your valuable and supportive reviews. We have summarized the updating of our revision in Summary of Paper Updating and provided an in-depth discussion about the novelty of our method in Genera Response. Below, we explain our method and contribution more clearly in Q1 to Q2 and address the weakness from Q3 to Q6 as follows.

Q1: The LET in OmniQuant decides the computation operands ordering for max hardware efficiency. The implementation is based on MLC.

A1: With full respect, Learnable Equivalent Transformation (LET) in OmniQuant focuses on channel-wise scaling and shifting to facilitate low-bits quantization by transferring quantization difficulty between weights and activation. Importantly, LET is independent of hardware efficiency considerations. After training, all LET and LWC parameters are absorbed into the original module without introducing additional memory or computational burden. Therefore, OmniQuant's quantized models can seamlessly deploy through existing frameworks such as MLC-LLM.

Q2: The biggest contribution of this work is the learnable weight clipping (LWC).

A2: With all due respect, the novelty and primary contribution of OmniQuant lies in being the first to introduce differentiable gradient optimization into learnable equivalent transformation (LET) and learnable weight clipping (LWC) for large language models. Previous PTQ methods for LLMs such as Outlier Suppression+ or AWQ solve such problems by grid searching, which limits the performance in low-bit scenarios. Furthermore, while the concepts of equivalent transformation and clipping are not new, we have made some innovative improvements to make the optimization process more stable. For equivalent transformation, we expend the equivalent transformation from linear layer to attention operation (Eq. (5) in our paper). For weight clipping, we propose taking the extremums after LET as proxies, which helps LWC to catch the dramatic changs led by LET as described in Section A5. These advancements enable OmniQuant to excel in both weight-only and weight-activation quantization scenarios.

Q3: While I highlight the advantage of LWC over AWQ, PACT, and LSQ, this evaluation for PACT and LSQ is lacking. What is the performance when compared to PACK and LSQ?

A3: As described in Algorithm 1 in Appendix, OmniQuant firstly leverages LET to transfer magnitude from activation to weight, and then quantizes the weight after LET with LWC. To demonstrate the advantage of LWC, we replace LWC in our pipeline with PACT (train the clipping threshold directly) and LSQ (train the quantization step size directly) in Table A14, showcasing that LWC consistently outperforms PACT and LSQ. Additionally, Figure A5 in the Appendix further clarifies the advantages of LWC by illustrating its adaptability to changing weight ranges introduced by LET, addressing optimization difficulties not addressed by traditional methods. Please also see Q2/A2 in responses to WDuc.

Q4: The optimization based on little data and backward propagation makes the learning easy to be overfitted. More validation should be conducted to prove the generalization ability of this learning.

A4: Acknowledging the potential for overfitting, we highlight that OmniQuant mitigates this issue through a constrained solution space. Specifically, optimization focuses on LET and LWC rather than all weights. Our results in Table A11 of the Appendix demonstrate OmniQuant's superior performance even with only 16 calibration samples. Additionally, we present performance on MMLU to further showcase the generalization ability of our approach.

Q5: There are some new kinds of ways to decompose the outliers, e.g., https://arxiv.org/abs/2310.08041. Comprehensive experiments are suggested to further enrich the validation.

A5: Thanks for your suggestion. Your referred paper QLLM (https://arxiv.org/abs/2310.08041) alleviates outlier channels by disassembling instead of scaling. In my view, Outier disassembling only considers the outlier channels. However, LET in our paper, as shown in Figure A2, can also adjust the range of normal channels to achieve a more quantization-friendly distribution. Therefore, the combined disassembling operation of QLLM and LET of our paper may further boost the performance of quantization models. We will undertake this exploration after QLLM releases its code and engage in discussions with the QLLM team for further insights.

We sincerely appreciate the time and efforts you have dedicated to reviewing our paper. Should you have any further inquiries, please let us know and we are more than delighted to discuss with you.

[1] Brecq: Pushing the limit of post-training quantization by block reconstruction

We thank the Review ZyAK for the time and effort to review our paper. We are glad that the reviewer found our contribution to constructing the gradient optimization framework. We have summarized the updating of our revision in Summary of Paper Updating and provided an in-depth discussion about the novelty of our method in Genera Response. Below, we address each identified weakness.

Q1: The novelty is limited. The overall framework is based on two existing methods.

A1: Thanks for the comment. We argue that although optimally determining clipping threshold and equivalent parameters have been explored in existing literature, our proposed OmniQuant is still novel enough by the optimal design. We clarify our novelty from $4$ aspects. 1) the first differentiable quantization framework for LLMs; 2) the novelty of LET; 3) the novelty of LWC; 4) the seamless combination of LWC and LET. Please refer General Response for more details.

Q2: The learnable idea is not new. In Outlier Suppression+, the scaling has been designed to be learned via a scheme that does not depend on gradient.

A2: Respecting your observation, we highlight that the novelty of our learnable approach lies in differential gradient updates for various quantization parameters. As illustrated in Figure A3, OmniQuant's differential gradient-based learning significantly reduces the quantization error compared with the grid-searching method employed in Outlier Suppression+. In experiments, Table 2 in the paper demonstrates the superiority of OmniQuant to OS+.

Q3: An in-depth comparison needs to be provided, including the experimental perspective and the theoretical perspective.

A3: OmniQuant differs from Outlier Suppression+ in several aspects, including parameter optimization through gradient updates, block-wise optimization, expansion of equivalent transformation to attention operations, and the introduction of learnable weight clipping (LWC). Details are as follows.

• Both OmniQuant and OS+ try to find the optimal equivalent transformation parameters. OS+ solves this problem by pre-defined shifting strength and grid-searched scaling parameters. However, OmniQuant obtains both shifting and scaling parameters through gradient optimization.

• OmniQuant leverages block-wise optimization while OS+ executes grid searching through a mix of single linear layer and multiple linear layers objective(Eq.6 and Eq.7 in their paper). Block-wise optimization considers interaction within a block and can produce better performance as demonstrated in BRECQ. However, it is a challenge for OS+ to expand to block-wise quantization, which would significantly enlarge the solution space and increase the time of grid searching.

• LET expends the equivalent transformation to attention operation (Eq. (5) in our paper), but OS+ only does the equivalent transformation on linear layers.

• OmniQuant also introduces learnable weight clipping (LWC), which further facilitates the low-bits (2,3) weight-only quantization, while Outlier Suppression only supports for weight-activation quantization.

We also incorporated a comparison with Outlier Suppression+ in Table 2 . An overview is as follows.

Dear reviewer ZyAK,

Thanks again for your valuable time and insightful comments. As the deadline for the Author/Reviewer discussion is approaching, it would be nice of you to let us know whether our answers have solved your concerns so that we can better improve our work. We are happy to provide any additional clarifications that you may need.

Best regards!

Q5: I noticed that the paper requires careful equivalent parameter initialization via the compared baseline SmoothQuant. I might wonder how it behaves without good initialization. For example, under asymmetric cases W4A8, W4A6, and W8A4, where the LLM-QAT shows SmoothQuant can behave terribly.

A5: Good advice, sorry for missing this ablation. To validate the impact of initialization, we first try to initial scaling as 1 and initial shifting as 0. We do experiments on LLaMa-1-7B with W4A4 quantization, and report the average perplexity and average accuracy. As shown in the following table, we can find that both careful initialization of scaling and shifting can improve the final performance. Specifically, scaling initialization is more important than shifting due to that scaling plays the main role in alleviating outliers.

Moreover, as shown in the following table, OmniQuant also demonstrates robust performance in asymmetric cases, ensuring effectiveness across various scenarios.

We have added these experiments in Table A6 and Table A17 of our revision.

Q6: I see the challenge of the two techniques is how to combine the two kinds of learning together as they influence each other. However, current techniques seem can not solve this problem well. Thus, I think it would be better if the paper gives more description, and design consideration to the combination part, which might increase the novelty. For example, maybe alternately train these two techniques. For example, maybe alternately train these two techniques.

A6: Thanks for the suggestion. We want to emphasize that in our approach, LWC and LET are trained simultaneously, and we have also explored an iterative training approach by iterations or epochs. The results, as presented in the following table, clearly indicate that training LWC and LET simultaneously yields the best performance. As aforementioned in Q2, LWC leverages the extremums after LET as the reference, allowing it to capture the dynamic changes in the weight range introduced by LET. This synergy between LET and LWC creates a progressive process, where both techniques reinforce each other rather than interfere. To further support this statement, we conducted an additional experiment (last row in the table), training LWC and LET iteratively with double the epochs. The results show that simultaneous training with 20 epochs achieves comparable performance to iterative training with 40 epochs. This demonstrates the effectiveness and efficiency of training LWC and LET simultaneously. We have added this experiment in Table A8 of our revision.

We sincerely appreciate the time and efforts you have dedicated to reviewing our paper. Should you have any further inquiries, please let us know and we are more than delighted to discuss with you and run more experiments for any pieces of your interests in our work.

Q3: What is the core novelty of the LET? I find it looks similar to Outlier Suppression+. While the paper says that Outlier Suppression+ takes a pre-defined migration strength, but this method does not and proposes to optimize the output. I'd like to point out that Outlier Suppression+ did not take a pre-defined strength and proposed to optimize the output for channel-wise scaling parameters earlier than this paper. Meanwhile, the paper also states that AWQ adopts a grid-searched channel-wise scaling, which also seems relevant to the technique in this paper. Therefore, can the paper compare these different designs and explain why the proposed way is the best? I did not find these and this could help us better understand the effectiveness.

A3: We have provided a detailed comparison of LET with Outlier Suppression+ (OS+) and AWQ in the table presented below.

• For the equivalent transformation, AWQ only considers scaling operation, while OS+ and our LET consider both scaling and shifting operation.

• For the execution position, AWQ and OS+ only carry equivalent transformation on linear layers, while LET also considers the matrix multiplication within attention (Eq. (5) in our paper). This point enlarges the solution space of equivalent transformation and brings further improvements as demonstrated in Table A5 of Appendix.

• For the optimization, OS+ leverages pre-defined shifting, both AWQ and OS+ find the scaling factors through grid searching based on some heuristic strategy. However, LET optimized all equivalent transformation parameters through end-to-end gradient descent, which significantly improve the performance. As acknowledged by reviewer ZyAK, making the learning of equivalent transformation stable and effective is a good contribution. Additionally, OmniQuant leverages block-wise optimization objectives, while both AWQ and OS+ take layer-wise optimization objectives. Block-wise optimization considers interaction within a block and can produce better performance as demonstrated in BRECQ. However, it is a challenge for OS+ and AWQ to expand to block-wise quantization, which would significantly enlarge the solution space and increase the time of grid searching.

We have added this discussion in Section A2 of our revision. In Figure A3 of Appendix, we also compare the quantization error of each block. We can find that OmniQuant significantly reduce the quantization loss compared with the grid-searching based method such as AWQ and Outlier Suppression +. You can also refer Genera Response for more discussion about the novelty of our paper.

Q4: Experiments shall be compared with the paper Outlier Suppression+ because you and they work on the same quantization problem, take the same equivalent transformation, and have similar optimization designs.

A4: We have incorporated comparisons with OS+ in Table 2 of our manuscript, and offered an overview of the W4A4 quantization with six zero-shot tasks as follows. We can find that OmniQuant outperforms OS+ in a range from 4.01% to 6.68% accuracy.

We would like to express our sincere gratitude to Reviewer WDuc for their valuable time and effort in reviewing our work. We have summarized the updating of our revision in Summary of Paper Updating and provided an in-depth discussion about the novelty of our method in Genera Response. Below, we address each identified weakness.

Q1: In 3.2, the paper claims that directly employing prior LSQ and PACT would produce unsatisfactory performance, as demonstrated in LLM-QAT. However, LLM-QAT says that the outliers for activation have a notable impact, bringing difficulty for clipping while this method works on weights here.

A1: We appreciate your clarification regarding LLM-QAT. Indeed, the confusion arises from our initial claim about the unsatisfactory performance of LSQ and PACT in weight quantization in Sec. 3.2 and Table A14. We have modified the expression to address any potential confusion. Thanks for the correction.

Q2: How can the proposed Eq. (2) solve the problem of learning clipping thresholds for outliers? In other words, what is the optimization difficulty (concept given in 3.2) of previous techniques, and how can Eq.(2) solve it? More explanation about the motivation is preferred. In the appendix, the paper says that LET would decline the convergence of LSQ and PACT because LET alters the weight distribution. However, weight distribution altering is a common case for LSQ and PACT in QAT. Also, combined with LET, the \gamma and \beta in the proposed Eq. (2) can also go up and down during learning as it optimizes the ratio of the changeable weight.

A2: Thanks for your question. We claim that LWC reduces the optimization difficulty in learning clipping thresholds because our LWC reparameterizes the learnable clipping threshold into the product of a learnable 0~1 clipping strength and the maximum or minimum of weights. The benefit is that no matter how the maximum and minimum of weights change caused by LET with the training going, our LWC would always learn a clipping strength under the reference of the current maximum and minimum of weights . Such a reparameterization technique has been utilized in various areas for better training properties. For example, batch normalization decomposes the neural activation into a variable approximately following normal distribution and learnable scale and shift parameters, which improves the training stability.

Here we provide more empirical evidence to clarify the motivation and advantages of LWC. As shown in Figure A5 in the appendix, it presents the dynamic change in weight ranges (maximun minus mininum) during training with different clipping-based methods. We can see that the weight range varies a lot as the training goes on (see the curve of Min-Max) when training our OmniQuant, which is caused by our proposed another component LET. It poses a challenge for traditional absolute clipping thresholds (PACT) or step size learning (LSQ). As we can see from Figure A5 (a & b), no matter how the weight range of Min-Max increases, PACT and LSR would always retain a relatively small weight range, which would clip many regular weights values to the threshold. In contrast, LWC calculates relative clipping strength and enables it to adapt to varying weight ranges, as shown in Figure A5 (c).

We hope that our explanations and visualizations will address your concerns.

Dear Reviewer WDuc,

We hope that our response can address your concerns. As the deadline for discussion period is approaching, we really appreciate if you can let us know whether there still exists any further question about the paper or the response. We are looking forward to further discussion.

Best regards,

Authors of Paper 1585

Thanks for your detailed reply. Some of my concerns have been addressed. I'd like to increase my score to 5. However, for the distinction compared to AWQ and Outlier Suppression+, I knew the difference but as stated in my first review, what I wonder is why the optimization method proposed in this paper is more beneficial. For example, the shifting technique is first proposed by Outlier Suppression+, though they use a pre-defined way, the paper seems not to show the results that learning the shifting vector can help a lot. Also, the table above shows the results of OmniQuant which takes both the LET and weight clipping. However, it did not show how much benefit can the gradient descent of LET bring compared to that of Outlier Suppression+ as that paper demonstrated lower accuracy obtained by learning. I think detailed analyses are very important as the difference is not big and novelty is limited in this part.

Thanks for your thoughtful review that will help us strengthen the manuscript. We have summarized the updating of our revision in Summary of Paper Updating. Below, we address each identified weakness.

Q1: Results on MMLU.

A1: As shown in the following table, we report the average accuracy on MMLU, showcasing that OmniQuant consistently achieves comparable or superior performance in reasoning ability. We have added this experiment in Table A16 of our revision.

Q2: Evaluating instruction-tuned models on MT-Bench or AlpacaFarm and evaluating stronger Chatbot.

A2: We have quantized Vicuna-v1.5 using W3A16g128 quantization and compared the performance of different methods through MT-Bench. The results reveal that OmniQuant consistently outperforms RTN and AWQ.

Q3: Discussion and comparison with related work- SpQR and SqueezeLLM

A3: Thanks for your suggestion, we have included the discussion and comparison with SqQR and SqueezeLLM in Section A7. However, we would like to emphasize two crucial points. Firstly, SpQR and SqueezeLLM are concurrent works, both available on arXiv in June 2023, which falls within four months of the ICLR submission deadline (Please refer to the policy in the last Q&A at https://iclr.cc/Conferences/2024/ReviewerGuide ). Secondly, Although OmniQuant may perform slightly worse than SqueezeLLM, our OmniQuant focuses on uniform (INT) quantization which offers simplicity and flexibility and has strong hard-ware support. Moreover, OmniQuant achieves competitive performance in both weight-only quantization and weight-activation quantization. In contrast, SpQR and SqueezeLLM only support weight-only quantization. We believe this distinction adds valuable context to the comparison.

We sincerely appreciate the time and efforts you have dedicated to reviewing our paper. Should you have any further inquiries, please let us know and we are more than delighted to discuss with you and run more experiments for any pieces of your interests in our work.

[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Dear reviewer FuSa,

Thanks again for your valuable time and insightful comments. As the deadline for the Author/Reviewer discussion is approaching, it would be nice of you to let us know whether our answers have solved your concerns so that we can better improve our work. We are happy to provide any additional clarifications that you may need.

Best regards!