Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning

We sincerely thank you for your great efforts on reviewing our paper. We are glad that you think our experiments are thorough and our method is effective. We make the following response to address your questions.

Q1: Regarding the performance on weaker base models and the generalization to non-Qwen architectures.

A1: Thank you and your question is what we have considered. We have performed our method on LLaMA3.1-8B-Instruct, which is an non-Qwen weaker base model. The results are already put in Table 6 in Appendix I. And we also display in the following Table 1 for your reference. Furthermore, we perform additional experiments on Qwen2.5-7B-Instruct in general reasoning domain (the training data is from WebInstruct-Verified [1], and the evaluation data includes MMLUPro and GPQA-Diamond), and show the self-improvements in the following Table 2. The results show that thinking-optimal scaling can also consistently improve the performance of weaker base models compared with random scaling in diverse domains, demonstrating the generalizability of our method.

Table 1: The self-improvement results of LLaMA3.1-8B-Instruct.

Table 2: The self-improvement results of Qwen2.5-7B-Instruct on MMLU-Pro and GPQA-Diamond.

Q2: Regarding future work on explorations in the RL setting.

A2: The primary scope and contribution of our work is to uncover the potential negative effects of scaling with excessively long chains of thought (CoTs). Through comprehensive empirical analysis, we propose an efficient and effective self-improvement strategy, and demonstrate its effectiveness primarily in the SFT/DPO setting. We agree that investigating the impact of CoT length in the RL setting is a valuable direction for future research, but it is out of the scope of this work. As discussed in the Limitations section, our findings are also applicable to RL-based scaling. In particular, since RL typically assigns positive rewards (e.g., 1.0) to all solutions with correct final answers, it is preferable to favor shorter correct solutions with fewer erroneous reasoning steps over longer correct ones that contain more errors. Over-rewarding the latter may inadvertently encourage the model to generate incorrect intermediate steps and rely on its imperfect self-correction ability, rather than producing the correct answer directly.

We hope the above clarifications address your concerns. We are glad to have further discussions if you have any further questions.

[1] Ma, Xueguang, et al. "General-reasoner: Advancing llm reasoning across all domains." arxiv 2025

We sincerely thank you for your positive review and your thoughtful suggestions. We are encouraged that you believe our paper is well-written and that our findings contribute to the understanding of test-time scaling. We make the following response to address your remaining questions.

Q1: Regarding the usage of system prompts to control the CoT length.

A1: Your concern is exactly what we have considered. To solve this issue, and as explained in Lines 141-143, we do not use the responses from QwQ-32B-Preview under different prompts directly. Instead, we filter and reorder the correct responses from QwQ-32B-Preview under the three reasoning efforts to ensure that, for the same problem, the lengths of the three corresponding responses increase sequentially and significantly from low to medium to high. This makes the length distributions of the responses under different reasoning efforts clearly distinguishable. Then, we set different system prompts for different subsets of responses under the three reasoning efforts, and merge all subsets to train tag models. By doing so, the fine-tuned tag models can follow the system prompts strictly and generate different lengths of CoTs when given different system prompts in the inference stage.

Q2: Regarding the performance of QwQ-32B-Preview on the evaluation benchmarks when prompting the model with different reasoning efforts.

A2: We also follow your suggestion to evaluate the performance of QwQ-32B-Preview on 3 evaluation benchmarks when prompting the model with different reasoning efforts. The evaluation parameters are the same as those used in our main experiments. The following results are consistent with the results on the tag models and re-validate our main claim: longer CoTs may not necessarily lead to better performance.

Table 1: The performance of QwQ-32B-Preview on 3 benchmarks when prompting the model with different reasoning efforts.

Q3: Regarding the comment "the ratios of erroneous reasoning rounds make more sense than the absolute number of erroneous reasoning rounds."

A3: We follow your suggestion to display the ratios of erroneous reasoning rounds in responses under different reasoning efforts. As the following table shows, the ratio of erroneous reasoning rounds also consistently increases as the reasoning effort increases.

Table 2: The statistics of responses under different reasoning efforts for training the tag models.

Q4: Regarding additional experiments on supporting the claim that including more erroneous rounds in the training stage can lead to greater adverse effects.

A4: We follow your suggestion to filter out those solutions that contain erroneous steps (identified by GPT-4.1) under the high reasoning effort, and finetune Qwen2.5-32B-Instruct on the filtered data, which yields Qwen2.5-32B-Tag-High-Filtered.

Table 3: The performance of the model trained on data with high reasoning effort after filtering out all solutions that are identified to contain erroneous steps.

We have some interesting findings: (1) After removing samples containing erroneous rounds, the trained model produces much shorter CoTs. The reason is that the solutions removed are usually very lengthy as they contain more self-reflections and self-corrections, thus the average length of the dataset after removing these samples is much shorter. (2) Removing samples containing erroneous rounds brings significant performance improvement on GSM8K, while causing certain performance degradation on harder benchmarks MATH500 and AIME2024. The reason is that after removing those samples, the model cannot learn to effectively perform deeper thinking such as correcting the errors it could make in previous rounds. GSM8K is relatively simple, so the model does not need to engage in excessive reflection and correction. Therefore, removing these behaviors actually improves model performance. On more challenging datasets, the advantages of longer reasoning chains that include reflections and corrections become apparent. These findings are consistent with the main claim of our paper.

Moreover, by comparing with the results in Figure 5 in the main text, we find that instead of directly removing correct solutions that contain erroneous steps, a more effective approach is to apply loss masking specifically to the wrong steps. The latter strategy allows the model to retain the ability to learn how to correct previous mistakes, without explicitly learning from the erroneous steps themselves.

Q5: Regarding the question that "Why not just using the same data presented in Fig. 4 and masking out the loss with erroneous steps to demonstrate the point?"

A5: The reason we do not use the responses for training tag models in the loss masking experiment is that it is very difficult to accurately locate which specific steps are erroneous in those responses, due to the informal nature of o1-like outputs. We have tried prompting LLMs to locate the exact erroneous steps, but found the results to be highly unreliable. In contrast, when we construct our own long CoT data, we can precisely segment the reasoning traces step by step during the generation process, and then apply loss masking to the specific erroneous steps as needed. We will include this clarification into the revision.

Q6: Regarding future work on explorations in the RL setting.

A6: The primary scope and contribution of our work is to uncover the potential negative effects of scaling with excessively long chains of thought (CoTs). Through comprehensive empirical analysis, we propose an efficient and effective self-improvement strategy, and demonstrate its effectiveness primarily in the SFT/DPO setting. We agree that investigating the impact of CoT length in the RL setting is a valuable direction for future research, but it is out of the scope of this work. As discussed in the Limitations section, our findings are also applicable to RL-based scaling. In particular, since RL typically assigns positive rewards (e.g., 1.0) to all solutions with correct final answers, it is preferable to favor shorter correct solutions with fewer erroneous reasoning steps over longer correct ones that contain more errors. Over-rewarding the latter may inadvertently encourage the model to generate incorrect intermediate steps and rely on its imperfect self-correction ability, rather than producing the correct answer directly.

We hope the above clarifications address your concerns. We are glad to have further discussions if you have any further questions.

We sincerely thank you for your positive feedback and strong support for our work!

To further investigate whether prompting models to generate longer responses also leads to decreased performance in other models, we conducted additional experiments on QwQ-32B. Since QwQ-32B includes both reasoning and summary components in its responses and typically generates much longer CoTs, we extended its max_seq_len to 32K to allow for sufficient reasoning and summarization. The results (averaged on 5 samplings) are presented in the following table. As shown, same trends are observed in this more advanced model: higher reasoning effort does not always yield better performance. However, we agree with your point that one possible reason is that current reasoning models may have inadequate instruction-following abilities and tend to overfit to the training prompt template, as you mentioned. Our reasoning effort-based fine-tuning experiments can help mitigate this issue by directly aligning the model’s response length distribution with specific reasoning efforts.

Table 1: The performance of QwQ-32B when directly prompting the model with different reasoning efforts.

Regarding the results in [1], which show that using incorrect responses in SFT can yield similar outcomes to using correct ones, we have two main explanations: (1) The SFT dataset in [1] is very small (only 50 samples), which allows the model to quickly overfit to the o1-like long reasoning trace format and generalize this reasoning style to other problems. In this scenario, the correctness of the reasoning chain has a relatively smaller impact on generalization compared to the format itself. (2) Another possible reason is data contamination in the Qwen2.5 model. Recent work [2] finds that even using incorrect rewards for RL on Qwen2.5 can also effectively improve model performance, and [3] suggests that this may be due to data contamination in the Qwen2.5 training data. We believe this could also partially explain the experimental findings in [1].

Regarding your final question about using a stronger model to correct the erroneous steps in the reasoning traces of high reasoning effort, we would like to clarify that: If the corrected steps are simply appended after the wrong steps as error correction steps, but loss masking is not applied to the erroneous steps during SFT, then both the experimental setup and results are essentially the same as those shown in our Figure 5 in the main paper. On the other hand, if the wrong steps are discarded and replaced directly with the corrected steps, this is equivalent to the experimental setup and results on our filtered high reasoning effort-based set (see Table 3 in our rebuttal).

We sincerely thank you for your insightful questions again, your support means a lot to us! We are glad to have further discussions with you if you have any follow-up questions.

[1] Du, Wei, et al. "The Challenge of Teaching Reasoning to LLMs Without RL or Distillation." 2nd AI for Math Workshop @ ICML 2025. 2025.

[2] Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." arxiv 2025

[3] Wu, Mingqi, et al. "Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination." arxiv 2025

We sincerely thank you for your positive review and your constructive comments. We are glad that you think our paper is well-motivated and very insightful. We are encouraged that you find our insights very useful. We make the following response to address your remaining concerns.

Q1: Regarding the statistical significance results.

A1: We have displayed the statistical significance results of Table 2 (self-improvement results on Qwen2.5-32B-Instruct) in Table 7 in Appendix I. Here, we show the standard deviation results of Table 6 (self-improvement results on LLaMA3.1-8B-Instuct) in the following table for your reference. We will add it into the revision.

Table 1: Standard deviation results of self-improvement results on LLaMA3.1-8B-Instuct.

Q2: Regarding the observations on more datasets and other reasoning tasks.

A2: We follow your suggestion to perform the same empirical validation in the general reasoning domain as we did in Section 3.2. Specifically, we prompt QwQ-32B-Preview to generate responses under different reasoning effort-based prompts on a subset of the WebInstruct-verified dataset [1], and curate a seed general reasoning dataset that contains 3.6K correct responses with varying reasoning efforts. We then finetune Qwen2.5-7B-Instruct on the seed dataset to get Qwen2.5-7B-Tag-General, and evaluate its performance on MMLU-Pro and GPQA-Diamond using the same reasoning effort-conditioned prompting strategy as in Section 3.2 (only sample once for each prompt and set max_seq_length to 8K considering the huge evaluation cost). The results are displayed in the following table. The findings are consistent with the observations in the math reasoning domain that excessive scaling with longer CoTs can bring negative effects to the model’s performance in general reasoning tasks. Also, we can find that the model needs more reasoning effort to perform well on the more challenging task GPQA-Diamond.

Table 2: The performance of Qwen2.5-7B-Tag-General on MMLU-Pro and GPQA-Diamond.

Q3: Regarding the performance of TOPS on other reasoning tasks.

A3: Following the setup in A2, we then perform our thinking-optimal scaling strategy on a held-out set of the WebInstruct-verified dataset to get Qwen2.5-7B-TOPS-General and perform random scaling to get Qwen2.5-7B-Random-General. The evaluation results in the following table show that our method can also work well for general reasoning tasks.

Table 3: The self-improvement results on MMLU-Pro and GPQA-Diamond.

Q4: Regarding the intuition on how the method works.

A4: Our method is backed by the empirical analysis in Section 3.2 and Section 3.3. As shown in Figure 2 and Figure 3, even though all the trainings are performed on solutions that are all ultimately correct, scaling with longer CoTs can adversely affect the model’s reasoning performance in certain domains, and there exists an optimal reasoning effort that varies across different tasks of varying difficulty levels. Figure 4 and Figure 5 help to explain such phenomenon: longer responses are more likely to contain more erroneous steps, and excessive training on such responses can degrade the model’s reasoning performance. Based on these intuitions, we propose to create a thinking-optimal dataset that contains the shortest correct response of the model on each prompt across all reasoning efforts, because this is the most efficient and effective trace that needs to be learned.

We hope the above clarifications address your concerns. We are glad to have further discussions if you have any further questions.

[1] Ma, Xueguang, et al. "General-reasoner: Advancing llm reasoning across all domains." arxiv 2025

We sincerely thank you for your great efforts on reviewing our paper and your helpful questions. We are glad that you think our target is a sensible goal and our method is good. We make the following response to address your questions.

Q1: Regarding the question "what does the "shortest reasoning trace" mean" and the comment "the dataset that's collected by (2) might be quite noisy".

A1: The "shortest reasoning trace" refers to the correct solution with the minimal response length that can be generated by the target model across all levels of reasoning effort. This trace represents the most efficient path the model can take to reach the correct final answer. As analyzed in Section 3.3, other longer responses are more likely to contain additional erroneous steps, even if their final answers are correct, and training on such responses can degrade model performance.

Based on these findings, we propose constructing thinking-optimal datasets using Eq. (2) to enable more effective test-time scaling. We agree with you that datasets collected via Eq. (2) may be noisy, because it is nontrivial to precisely define and find the truly shortest correct response. However, we believe that increasing the number of samples for each prompt under each reasoning effort can help mitigate this noise and improve the reliability of the thinking-optimal dataset.

Q2: Regarding the comment "the actual accuracies are not particularly noteworthy".

A2: The main scope of our work is to reveal the potential side effects of scaling with excessively long CoTs and to propose a more efficient and effective self-improvement strategy. Our self-improved model achieves comparable performance to QwQ-32B-Preview with more efficient and adaptive thinking modes using only 1.3K seed data, and consistently outperforms other direct distillation-based models which rely on more high-quality distillation data in both effectiveness and efficiency. Also, our thinking-optimal model significantly outperforms random scaling (Qwen2.5-32B-Random) under the same constraints, validating our initial motivation. Finally, we believe our method is also applicable when using other models such as QwQ-32B to curate seed data for self-improvement.

Q3: Regarding the question "What's the mapping from the number of erroneous reasoning steps to whether the final answer is correct or not?"

A3: For Figure 4, all candidate responses have correct final answers. The purpose of Figure 4 and Figure 5 is to elucidate the empirical findings presented in Figure 2 and Figure 3: although all training solutions ultimately yield correct answers, scaling with longer CoTs can adversely affect the model’s reasoning performance in certain domains. This is because longer CoTs are more likely to include additional erroneous reasoning steps, and excessive training on such responses may degrade the model’s overall reasoning ability.

Q4: Regarding the length distributions of the reasoning effort-conditioned model.

A4: Figure 3 displays the length distributions of the reasoning effort-conditioned models (i.e., tag models). As we can see, the reasoning effort-conditioned models achieve the best reasoning performance on GSM8K under the low reasoning effort with fewer than 500 average tokens, and achieve promising performance on AIME2024 under the high reasoning effort with more than 8000 average tokens. This helps to validate that the reasoning effort-conditioned models can actually generate reasoning chains of the appropriate lengths.

Q5: Regarding the qualitative discussion of what modulating the reasoning effort does to the reasoning chain.

A5: Thank you for your constructive question. We follow your suggestion to manually analyze the reasoning chains under different reasoning efforts. We find that in the vast majority of cases, responses under low reasoning efforts use fewer complete reasoning rounds compared to responses under higher reasoning efforts (consistent with Figure 4). This property manifests differently in simple and difficult problems:

(1) For simple problems (about 40% samples we checked), higher reasoning effort leads to overthinking, where the model continuously adopts alternative strategies to re-solve the problem. This causes the model to initially answer correctly but subsequently obtain incorrect answers through re-solving, and then become misled by these wrong answers, ultimately leading to incorrect responses.

(2) For some difficult problems, the property of engaging in continuous reflection, correction, and exploration under higher reasoning efforts enables the model to solve problems that cannot be addressed by lower reasoning efforts due to underthinking.

Q6: Regarding the question "As one changes the amount of reasoning effort, what happens to the distribution of answers?"

A6: We follow your suggestion to calculate the distributions of answers of reasoning effort-conditioned models under different reasoning efforts. Specifically, we calculate the average number of distinct answers in 5 samples per prompt under each reasoning effort. We also display the accuracy on each benchmark in the following table for reference. We find some very interesting phenomena: Generally, the reasoning effort that achieves the best performance on a certain benchmark also leads to the lowest average number of distinct answers per prompt. This indicates that, under the optimal thinking effort, the model can generate the most consistent answers across multiple samplings without either underthinking or overthinking. Your question is helpful for better understanding of empirical findings in Section 3.2 and we will add it into the revision.

Table 1: The distribution of answers under different reasoning efforts along with the average accuracy. We highlight the best accuracy and the lowest average number of distinct answers per prompt.

We hope the above clarifications address your concerns. We are glad to have further discussions if you have any further questions.

Dear Review CZmU,

We sincerely thank you again for reviewing our paper and providing helpful comments. We have addressed your questions about the shortest correct response, the self-improvement performance and the correctness of final answers in reasoning effort-conditioned training data in the rebuttal. Following your suggestion, we have included the discussion on the effects of reasoning efforts on reasoning traces, and provided an interesting empirical analysis on the answer distributions under different reasoning efforts. As the author-reviewer discussion deadline is approaching, we would like to know if you have any further questions, and we are glad to continue the discussion if there are any.

We are sincerely looking forward to your feedback, and your support means a lot to us! Thank you.

Best regards,

Authors

Dear Reviewer CZmU,

We sincerely thank you for your positive feedback. We have conducted additional experiments on other datasets (general reasoning tasks) and models (LLaMA). We put the new results in the following for your reference.

First, we put the self-improvement results of our method on LLaMA models in Table 1. As we can see, our method can also work well on other models.

Table 1: Self-improvement results on LLaMA3.1-8B-Instuct.

Additionally, we prompt QwQ-32B-Preview to generate responses under different reasoning effort-based prompts on a subset of the WebInstruct-verified dataset [1], and curate a seed general reasoning dataset that contains correct responses with varying reasoning efforts. We then finetune Qwen2.5-7B-Instruct on the seed dataset to get Qwen2.5-7B-Tag-General, and evaluate its performance on MMLU-Pro and GPQA-Diamond using the same reasoning effort-conditioned prompting strategy as in Section 3.2 (only sampling once for each prompt considering the evaluation cost). The results are displayed in the following Table 2. The findings are consistent with the results in the math reasoning domain that excessive scaling with longer CoTs can bring negative effects to the model’s performance in general reasoning tasks. Also, the model needs more reasoning effort to perform well on the more challenging task GPQA-Diamond.

Table 2: The performance of Qwen2.5-7B-Tag-General on MMLU-Pro and GPQA-Diamond.

Following the above setup, we then perform our thinking-optimal scaling on a held-out set of the WebInstruct-verified dataset to get Qwen2.5-7B-TOPS-General and perform random scaling to get Qwen2.5-7B-Random-General. The evaluation results in the following table show that our method can also work well for general reasoning tasks.

Table 3: The self-improvement results on MMLU-Pro and GPQA-Diamond.

We sincerely hope the above results address your last question well, and you can provide more support if you find these new results helpful! Your support means a lot to us! Thank you again.

Best regards,

Authors

[1] Ma, Xueguang, et al. "General-reasoner: Advancing llm reasoning across all domains." arxiv 2025