We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below.

Response to Weakness 1: Inefficiency During Inference

We acknowledge that one limitation of our method is that longer verification step is unavoidable during inference in full deployment mode (note: in fast deployment mode, the verification step is not needed though). This can indeed lead to longer response lengths for more complex tasks. To mitigate this issue, we conducted an additional experiment on R1.5B and R7B models by shortening the token budget in the verification step from 6k tokens to 2k tokens, and found that performance can be maintained while significantly reducing the number of tokens used. We present results for this "short verification" model in both its full (Thinker (short verification)) and fast (Thinker-Fast (short verification)) deployment modes:

Average Accuracy for Finetuned R1.5B Models

Response Length (in Token) for Finetuned R1.5B Models

Average Accuracy for Finetuned R7B Models

Response Length (in Token) for Finetuned R7B Models

We have also identified several concurrent works—ThinkPrune [1], ConciseRL [2], Structured Reasoning [3], and AdaptThink [4]—that also apply RL to improve mathematical reasoning on R1.5B with more advanced methods:

Comparison with Concurrent Works Finetuning R1.5B Models

Several observations can be made from the results above:

1. Shortening the verification tokens from 6k to 2k has little impact on final accuracy—showing a slight increase for the R1.5B model and a slight decrease for the R7B model. However, the total token length can be significantly reduced in full deployment mode.
2. Thinker (short verification) R1.5B models outperform other concurrent works in terms of accuracy while remaining competitive in terms of token length.
3. Thinker-Fast (short verification) R1.5B models use significantly fewer tokens than all concurrent works, while maintaining slightly worse performance on simpler benchmarks (MATH500 and AMC23), though with significantly worse performance on complex tasks (AIME24).

An additional advantage of Thinker—beyond the metrics shown above—is that users can choose the deployment mode based on task demands and computational cost constraints (e.g., fast mode for simpler tasks and full mode for complex ones) without training two separate models. In summary, Thinker offers a flexible trade-off while maintaining competitive or even superior performance compared to similar methods.

The above new results on short verification and the additional baselines will be included in the revised version of the main paper.

Response to Question 1: Thinker achieving higher accuracy despite generating longer responses than standard QA

In general, longer responses are usually associated with higher accuracy, as the model has more tokens to perform reasoning. The DeepSeek R1 paper [5] shows that both response length and accuracy increase during training. The increase in token length during training for both the baseline and Thinker models, as shown in Fig. 5b, is consistent with this trend. Manual inspection reveals that the models generally learn to perform more self-correction during training—e.g., "Wait, I may be wrong here, let me recheck"—for both the baseline and Thinker models. This behavior aligns with the final reward being based on the correctness of the final answer, giving the agent an incentive to perform more verification and try additional approaches when there is room to use more tokens.

We thank the reviewer again for their valuable feedback. We believe these new baseline comparisons, along with our new results, substantially strengthen the paper. We would be grateful if the reviewer would consider this new evidence and our responses in their final assessment. We welcome any further questions.

[1] Hou, B., Zhang, Y., Ji, J., Liu, Y., Qian, K., Andreas, J., & Chang, S. (2025). Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning. arXiv preprint arXiv:2504.01296.

[2] Dumitru, R. G., Peteleaza, D., Yadav, V., & Pan, L. (2025). ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models. arXiv preprint arXiv:2505.17250.

[3] Dong, Y., & Fan, H. (2025). Enhancing Large Language Models through Structured Reasoning. arXiv preprint arXiv:2506.20241.

[4] Zhang, J., Lin, N., Hou, L., Feng, L., & Li, J. (2025). Adaptthink: Reasoning models can learn when to think. arXiv preprint arXiv:2505.13417.

[5] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below.

Response to Weakness 1&4: Additional Baselines

We agree that comparing against additional baselines is valuable. We have identified several concurrent works—ThinkPrune [1], ConciseRL [2], Structured Reasoning [3], and AdaptThink [4]—that also apply RL to improve mathematical reasoning on R1.5B.

Comparison with Concurrent Works Finetuning R1.5B Models

The table above compares our method against these new baselines. To explore efficiency, we also trained a variant model using a shorter verification step (2k tokens instead of 6k). We present results for this "short verification" model in both its full (Thinker (short verification)) and fast (Thinker-Fast (short verification)) deployment modes.

Several observations can be made from the results above:

1. Shortening the verification tokens from 6k to 2k has little impact on final accuracy—showing a slight increase for the R1.5B model. However, the total token length can be significantly reduced in full deployment mode.
2. Thinker (short verification) R1.5B models outperform other concurrent works in terms of accuracy while remaining competitive in terms of token length.
3. Thinker-Fast (short verification) R1.5B models use significantly fewer tokens than all concurrent works, while maintaining slightly worse performance on simpler benchmarks (MATH500 and AMC23), though with significantly worse performance on complex tasks (AIME24).

An additional advantage of Thinker—beyond the metrics shown above—is that users can choose the deployment mode based on task demands and computational cost constraints (e.g., fast mode for simpler tasks and full mode for complex ones) without training two separate models. In summary, Thinker offers a flexible trade-off while maintaining competitive or even superior performance compared to similar methods.

The above new results on short verification and the additional baselines will be included in the revised version of the main paper.

Regarding simpler prompting-based methods like Self-Refine or Reflexion, we believe a direct comparison is challenging. First, while the effectiveness of these methods has been demonstrated on large, instruction-tuned models like GPT-4, they would likely perform poorly on our smaller base models, which are not instruction-tuned. Second, a direct comparison of the final outputs would be misleading due to the vast difference in computational cost: these prompting strategies are inference-time techniques requiring no finetuning, while our methods and baselines involve substantial RL training. One might suggest resolving these issues by applying RL to these prompting methods; however, this requires designing a good reward signal, which is a significant research topic in its own right. For these reasons, we consider this comparison outside the scope of our current work.

Response to Weakness 2: GRPO vs. PPO

We chose PPO as our primary RL algorithm because prior work [5, 6] showed that GRPO can be unstable for finetuning LLMs, sometimes leading to performance collapse. We also conducted preliminary experiments finetuning our models on the standard QA task using GRPO and confirmed the instability analyzed in [5, 6]. Given these findings, we focused our efforts on PPO, which provided more stable and reliable results.

Response to Weakness 3: Overfitting in the Baseline

We select the final model for evaluation based on its peak performance on a held-out validation set (i.e., early stopping). For example, the results for the Qwen 1.5B baseline reported in Table 1 are from the checkpoint at training step 900, before the performance degradation occurred.

Regarding statistical significance, we have conducted new multi-seed experiments to assess the variance from RL training. Please refer to our response to Reviewer hUT1 for a detailed presentation of these new results.

Response to Question 1: Thinker-Fast performing better on GPQA

We believe this interesting phenomenon is due to GPQA's nature as a knowledge-intensive science benchmark. Unlike our other benchmarks which focus on mathematical reasoning, GPQA is knowledge-intensive, covering biology, physics, and chemistry. Our primary hypothesis is that the fast thinking step helps to mitigate hallucinations on such tasks. This hypothesis is guided by prior work [7], which suggests that long reasoning chains trained via RL can lead to a higher degree of hallucination. While the long thinking process used by the full Thinker model and our baselines is beneficial for math, it may become a vulnerability on a knowledge-based benchmark like GPQA. By explicitly constraining the generation to a short format, Thinker-Fast limits the model's opportunity for flawed reasoning and forces it to rely more directly on its pre-trained knowledge.

This also explains why Thinker-Fast may outperform the full Thinker mode on this specific task. The deployment of the subsequent verification and slow thinking steps in the full mode, while allowing more room for reasoning, appears to partially negate the anti-hallucination benefit of the fast thinking step. The advantage is not entirely lost, however, as the full Thinker mode still outperforms the baseline on GPQA. We are actively exploring the implications of this finding for controlling hallucination in RL-finetuned LLMs.

We thank the reviewer again for their valuable feedback. We believe these new baseline comparisons, along with our clarifications on the training process and algorithm choices, substantially strengthen the paper. We would be grateful if the reviewer would consider this new evidence and our responses in their final assessment. We welcome any further questions.

[1] Hou, B., Zhang, Y., Ji, J., Liu, Y., Qian, K., Andreas, J., & Chang, S. (2025). Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning. arXiv preprint arXiv:2504.01296.

[2] Dumitru, R. G., Peteleaza, D., Yadav, V., & Pan, L. (2025). ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models. arXiv preprint arXiv:2505.17250.

[3] Dong, Y., & Fan, H. (2025). Enhancing Large Language Models through Structured Reasoning. arXiv preprint arXiv:2506.20241.

[4] Zhang, J., Lin, N., Hou, L., Feng, L., & Li, J. (2025). Adaptthink: Reasoning models can learn when to think. arXiv preprint arXiv:2505.13417.

[5] Hu, J., Zhang, Y., Han, Q., Jiang, D., Zhang, X., & Shum, H. Y. (2025). Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290.

[6] Rajani, N., Gema, A. P., Goldfarb-Tarrant, S., & Titov, I. (2025). Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them. arXiv preprint arXiv:2507.10616.

[7] Li, J., & Ng, H. T. (2025). The Hallucination Dilemma: Factuality-Aware Reinforcement Learning for Large Reasoning Models. arXiv preprint arXiv:2505.24630.

Thanks the authors for the responses. I do not have any further questions and keep my original score.

We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below.

Response to Weakness 1: Limitation to Math Reasoning Tasks

The primary focus of this paper is on improving the math reasoning capabilities of LLMs via RL, which is a highly active and focused research area. As evidence of this focus within the community, a recent survey [1] lists multiple concurrent papers dedicated specifically to math reasoning. We are hopeful that our work can contribute to this research frontier.

Nonetheless, we agree that demonstrating broader applicability is valuable. We have performed an evaluation of our models on a non-mathematical benchmark, GPQA Diamond, which consists of 448 challenging multiple-choice questions written by domain experts in biology, physics, and chemistry [2]. Despite being trained exclusively on math questions, our Thinker models also perform well on GPQA Diamond compared to both the baseline and the original pre-trained models (Table 1 in the main paper). This promising generalization performance leaves room for further exploration in future work, which we will acknowledge in the paper.

Response to Question 1: Two Tries Per Question

Our Thinker agent does not get two attempts at providing a final answer during inference. The process, as depicted in Figure 2 of the main paper, is as follows:

1. The model generates a fast answer in the fast thinking step.
2. The model verifies if the fast answer is correct in the verification step. This is a step internal to the model's own reasoning process and does not rely on any external verifier.
3. If the model verifies the fast answer as correct in the verification step, then the fast answer is defined as the final answer. Otherwise, the model goes into the slow thinking step, and the final answer will be the slow answer given in the slow thinking step.

Crucially, this process results in only one final answer being produced for each question. Therefore, the comparison with baselines is fair, as both our method and the baselines submit only a single, final answer for each question (i.e., the evaluation is Pass@1 for all methods). We will explicitly update the experimental setup section in our paper to make this single-attempt evaluation process clearer.

We thank the reviewer again for their valuable feedback. We hope this clarification addresses the main concern regarding evaluation fairness. We believe that with this resolved, the strengths of the paper as noted by the reviewer—its strong empirical performance and novel approach—solidify its contribution. We would be grateful if the reviewer would consider this clarification in their final assessment and are happy to answer any further questions.

[1] Li, Z. Z., Zhang, D., Zhang, M. L., Zhang, J., Liu, Z., Yao, Y., ... & Liu, C. L. (2025). From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419.

[2] Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., ... & Bowman, S. R. (2024, July). Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.

We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below.

Response to Weakness 1: Variance of Performance and Number of Experiments

New Multi-Seed Experiments

We agree with the reviewer that assessing the variance from RL training dynamics is important for validating the benefits of the proposed method. To address this concern, we have conducted new experiments for our main result (Q1.5B Thinker vs. Baseline), running each method with three different random seeds. The aggregated results are as follows:

We note that our baseline results are consistent with those reported by similar methods like ORZ [1], SimpleRL [2], and RL on DSR [3], which all performed RL training on the same Q1.5B model with a standard QA task, similar to our baseline. From the above table, we observe Thinker consistently outperforms the baselines. An independent samples t-test showed that Thinker (M = 27.41, SD = 0.40) performed statistically significantly better than the baseline (M = 25.12, SD = 0.22), t(4) = 8.76, p < .001. Additional comparisons between Thinker and four concurrent works with more advanced methods are provided in our response to Reviewer fcXk.

To address the reviewer's question about the "reflection pattern occurrence" (Figure 6), we analyzed this metric across our new multi-seed runs. The average scores, both over all training steps (steps 0-1200) and during the final training steps (steps 1100-1200), are presented below:

Across all three seeds, we observe that Thinker models consistently exhibit lower reflection pattern scores compared to the variant without the summarization subtask. This supports our hypothesis that the summarization step encourages the agent to develop a more concise chain-of-thought, thereby reducing reflective or repetitive language. However, as noted by the reviewer, it remains possible that the Thinker without Summarization may have converged to a different local optimum with inherently lower reflection patterns. Addressing this possibility would require additional seeds for Thinker without Summarization, which are currently beyond our computational budget. Nonetheless, the results presented in the table indicate that Thinker is less likely to settle into policies characterized by high reflection patterns, which we believe sufficiently demonstrates the benefits of incorporating the summarization step.

Additional Experiments

At the time of submission, the R7B runs were ongoing due to computational constraints, which is why their preliminary results were placed in the appendix. We have since completed these experiments and present the full results below:

As shown, Thinker demonstrates consistent performance improvements over the baseline on the R7B model. This result is also consistent with our findings on smaller models and suggests that our approach scales effectively.

Furthermore, to address potential concerns about method efficiency, we have also conducted two additional experiments (detailed in our response to Reviewer fcXk) showing that the token budget for the verification step can be significantly reduced while maintaining performance, thereby improving Thinker's efficiency.

In total, these new experiments bring the number of PPO runs presented to 13:

• Q1.5B: Thinker vs. Baseline (3 seeds per run)
• Q1.5B: Thinker without summarization
• R1.5B: Thinker vs. Baseline
• R1.5B: Thinker with short verification
• R7B: Thinker vs. Baseline
• R7B: Thinker with short verification

Each PPO run is computationally intensive, requiring approximately 960 A100-hours (8 GPUs for 5 days). The experiments presented here, including the new runs, represent over 10,000 GPU-hours. Given these substantial costs and our limited computational budget, we have focused our resources on the most important experiments. We hope this comprehensive evaluation is sufficient to address the concerns about statistical robustness.

We will incorporate all new results and analyses into the revised manuscript.

Response to Question 1: Calculation of Standard Errors

The reviewer is correct that the method for calculating standard errors was not explicitly defined. We apologize for this oversight.

The standard error reported in Appendix B is not calculated over the questions in the dataset, but rather over 16 independent samples for a given benchmark. The procedure is as follows:

1. For a single model, we generate 16 samples per question from the benchmark with a temperature of 1.0.
2. This process yields 16 aggregate accuracy scores for the benchmark.
3. The standard error is then calculated over these 16 accuracy scores.

The purpose of this metric is to quantify the sampling variance of a single trained model (particularly relevant for benchmarks with a limited number of questions, such as AIME), which is distinct from the training variance observed across different RL runs. Our original intent was for this to be conveyed in the caption of Table 1, by placing the reference to the standard error after the sentence: “All scores are Pass@1 accuracy (%) averaged over 16 samples.” However, we now recognize that this implicit connection was too subtle. In the revised paper, we will include a clear and explicit definition of the standard error.

We thank the reviewer again for their valuable feedback. We hope our new experiments sufficiently address the concerns about training variance. We would be grateful if the reviewer would consider this new evidence in their final assessment. We welcome any further questions.

[1] Hu, J., Zhang, Y., Han, Q., Jiang, D., Zhang, X., & Shum, H. Y. (2025). Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290.

[2] Zeng, W., Huang, Y., Liu, Q., Liu, W., He, K., Ma, Z., & He, J. (2025). Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892.

[3] Wang, Y., Yang, Q., Zeng, Z., Ren, L., Liu, L., Peng, B., ... & Shen, Y. (2025). Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571.