Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models

Thanks for your positive reviews! We sincerely appreciate the time and effort you have dedicated to evaluating our work.

Q1. No novel ideas are proposed. It is more about empirical study paper. The main contribution of this paper is doing experiments for the current quantization algorithms, and conclusion might be change as new approaches evolve. This paper would be benefit from this by proposing a leaderboard that support evaluations of any quantization algorithms.

A1. Thanks for the suggestion. We fully understand your concerns as new approaches are evolving rapidly nowadays. On the one hand, we do our best to provide findings generalizable to different models. On the other hand, to continuously help the community, we will release our code (https://anonymous.4open.science/r/Quantized-Reasoning-Models-Anonymous-419C/README.md), which is built upon vLLM and LightEval that allows easy extension to more recent reasoning models. The idea to maintain a live leaderboard aligns with our plans, and we have started to prepare this already.

For example, we integrate it with a recently released reasoning model Qwen3-8B, and the results align with the general findings in Table 1 of our paper. For instance, 8-bit weight and activation is still lossless on most benchmarks for Qwen3-8B.

Q2. It does not really explain why certain quantization algorithms fail or do well. For example, it states "AWQ is preferred over GPTQ", "QuaRot is generally preferred over KVQuant". This paper could be benefit if author can connect those empirical conclusions (accuracy drop) with some theoretically justifications and provide some insights in designing new algorithms.*

A2. We have analyzed why certain algorithms are better than others in Appendix C and will move this part to the main text in the revision.

1. Calibration Data Matters. 1) GPTQ. We find that using reasoning data for calibration is critical for quantization methods that rely heavily on the calibration data for quantization error compensation (e.g. GPTQ). 2) AWQ. On the other hand, methods that only rely on calibration data for quantization parameter computation or outlier channel identification are robust to calibration data selection (e.g. AWQ). Detailed explanations are provided in Appendix C.1.

2. Be Careful of Model-specific Features. As shown in Table 1, we find that QuaRot generally outperforms KVQuant* thanks to the Hadamard transformation. However, as detailed in Appendix C.2, for models with extreme outlier channels on K cache, KVQuant* can significantly outperform QuaRot through per-channel K cache quantization, and the performance can be further boosted with the pre-bias quantization introduced in L628-645. These results show that quantization accuracy can be significantly improved by targeting model-specific features.

Q1. An in-depth analysis into the mechanism of quantization which leads to performance degradation will improve the paper further.

A1. Thanks for your valuable suggestion. More in-depth analysis can be found in appendix due to limited space. For example:

1. Calibration Data Matters. 1) GPTQ. We find that using reasoning data for calibration is critical for quantization methods that rely heavily on the calibration data for quantization error compensation (e.g. GPTQ). 2) AWQ. On the other hand, methods that only rely on calibration data for quantization parameter computation or outlier channel identification are robust to calibration data selection (e.g. AWQ). Detailed explanations are provided in Appendix C.1.

2. Be Careful of Model-specific Features. As shown in Table 1, we find that QuaRot generally outperforms KVQuant* thanks to the Hadamard transformation. However, as detailed in Appendix C.2, for models with extreme outlier channels on K cache, KVQuant* can significantly outperform QuaRot through per-channel K cache quantization, and the performance can be further boosted with the pre-bias quantization introduced in L628-645. These results show that quantization accuracy can be significantly improved by targeting model-specific features.

Q2. Is there any hypothesis why more difficult reasoning tasks and more reasoning steps show greater performance decline with quantization? Does precision of computation relate to the performance of reasoning?

A2. Thanks for the question.

As shown in Figure 5, the test-time scaling of low-bit quantized reasoning models is inferior to their full-precision counterparts. In L304-305, we speculate that the quantization error accumulated along the sequence may hinder effective reasoning, which makes the accuracy gap grow larger as the reasoning steps increase.

At a higher level, we hypothesize that quantization may impair the knowledge encoded in model weights as well as the reasoning ability, which combine to hurt the performance of quantized models on hard problems. We plan to systematically evaluate the knowledge loss and reasoning ability loss brought by quantization through the lens of Pass@K and Pass@1 metrics in future work.

Q1. The empirical contributions of this paper, though useful, do not yield any novel insights or clearly generalizable advice / rules of thumb that can be applied more generally in the field. While the paper does discuss preferred algorithms for quantization, it is not clear (and plausibly not correct) that these optimal algorithms would remain optimal, for, say, the next generation of reasoning models. Moreover, there is no conceptual analysis explaining or interpreting the findings and leading to higher-level findings. As a result, the study feels like a snapshot of what works now, as opposed to what we can learn about quantization in this space of models more generally.

A1. We conduct extensive experiments with the best effort to obtain generalizable conclusions, as outlined L48 - L62. To verify this, we further conduct more experiments on the recently released reasoning model Qwen3-8B, with results listed in A2. Our findings still apply to this new model, e.g., the lossless quantization can be achieved with 8-bit weight and activation quantization, etc.

As the first study in this field, we seek to balance the width and depth of this research. Aside from the general guidelines, we have also provided more conceptual analysis to strengthen the understanding of quantized reasoning models in our paper:

• Insights into the Performance of Quantization Algorithms. In Appendix C, we provide discussions on the underlying reason why certain quantization algorithms are preferred for reasoning models. For example, we find that the choice of calibration data matters for quantization methods that rely heavily on the calibration data for quantization error compensation (e.g. GPTQ) with reasons explained in Appendix C.1.

• Test-time Scaling of Quantized Reasoning Models. We first explore the test-time scaling of quantized reasoning models in Section 4.3, and find that the test-time scaling of low-bit quantized models is inferior to that of BF16 models, which may partially explain why harder problems with longer answers suffer more from quantization.

• Effectiveness on Real-world Deployment. In Sections 4.1 & 4.2, we confirm that quantized reasoning models can effectively benefit inference latency by decreasing step-wise latency and maintaining the output length.

Q2. Building on the previous point, I'm concerned that this paper will quickly lose relevance as the landscape continues to evolve rapidly: In an ideal world, this paper would be better shielded from this by proposing a framework / codebase that automates this quantization evaluation and maintains a live leaderboard. However, I don't see any such work proposed here.

A2. Thanks for your insightful advice. We completely understand your concerns, especially given how quickly LLMs are evolving these days. To continuously help the community, we will release our code (https://anonymous.4open.science/r/Quantized-Reasoning-Models-Anonymous-419C/README.md), which is built upon vLLM and LightEval that allows easy extension to more recent reasoning models.

For example, we integrate it with a recently released reasoning model Qwen3-8B, and the results align with the general findings in Table 1 of our paper. For instance, 8-bit weight and activation is still lossless on most benchmarks for Qwen3-8B.

BTW, the idea to maintain a live leaderboard aligns with our plans, and we have started to prepare this already. Thanks for the wonderful suggestion :)