We thank Reviewer KxXf for their critical feedback, which has pushed us to further strengthen our paper with new experiments and analysis.

W1: The improvement in generation accuracy is not statistically significant, except on the AMC dataset.

While the gains with the Qwen2.5 series are consistent, we acknowledge that stronger improvement is helpful. To further demonstrate the effectiveness and significance of RISE, we ran new experiments on the more recent Qwen3 base models, Qwen3-4B-Base and Qwen3-8B-Base, using 10K training data from DeepMath.

On these newer models, RISE achieves substantial and significant improvements in reasoning accuracy over the Zero-RL (PPO) baseline, with average gains of +5.6% and +5.6% for the 4B and 8B models, respectively. Verification accuracy also sees a massive boost (+20.9% for the 4B model and +16.8% for the 8B model).

These results clearly demonstrate the significant impact of our method. We will add them to the paper.

W2: Test-time benefit and fair comparison of compute.

This concern is based on a critical factual misunderstanding about the compute cost of verification. A verification response can be very short (e.g., "The final rating for the response is \boxed{-0.5}."), whereas a solution response is a long chain-of-thought. Thus, the premise that their token costs are "roughly the same" is incorrect.

We calculated the actual token consumption. On average, a verification response uses only 2% to 14% of the tokens of a full solution generation.

Therefore, comparing k=4 + verif to k=8 is not a fair comparison. A much fairer comparison, which generously overestimates the verification cost, would be against k=4 + 4*0.14 ≈ k=5 for the RISE models.

Here is the adjusted comparison. RISE with self-verification at k=4 consistently outperforms the stronger PPO baseline at k=5. This demonstrates a clear and compute-efficient benefit at test time.

W3: Offline verification experiment weakens the claim.

We believe there is a misunderstanding of the purpose of this ablation. The result in Figure 6 does not weaken our claim; on the contrary, it strengthens our central thesis that online verification is crucial. The entire point of the offline experiment was to show that simply mixing in offline verification data is not effective, as the verification skill is not learned jointly with on-policy generation. This highlights the limitation of decoupled approaches and motivates the need for our integrated, online framework.

Additionally, as discussed in Section 5.5, verification is generally an easier task than reasoning, so the translation from verification accuracy to reasoning accuracy is not always linear. In Figure 7 (right), we show that the accuracy of self-verified responses improves substantially with RISE, providing a more direct evidence that strengthened self-verification contributes to better reasoning performance.

Thank you for the detailed response. I will focus my evaluation on the technical aspects of your rebuttal, but I feel it is necessary to comment on the tone. Phrases such as "This concern is based on a critical factual misunderstanding about the compute cost of verification" and "We believe there is a misunderstanding of the purpose of this ablation" are unnecessarily confrontational. Please see my response below.

Test-time benefit and fair comparison of compute

This issue is not due to a misunderstanding. It arises from a lack of information in the manuscript. Because a generative verifier necessarily produces a full chain-of-thought, so it is reasonable to expect a substantial token cost. The example in Figure 14 appears to be nearly 100 tokens. If the paper does not account for this compute, I believe pointing it out is a valid critique. Moreover, a recent paper has shown that generative verification is ineffective under the small compute budgets [1], so I believe this concern is well founded. Finally, because my concern here is solely about test-time compute, the appropriate baseline is RISE without verification, not PPO.

Offline verification experiment weakens the claim

This issue is not the result of a misunderstanding. Training the verifier using an offline dataset does not improve reasoning accuracy (Figure 6, left).

Summary

The paper claims that training a single model as both generator and verifier using RL and using online solutions to generate verifications, improves both generation and verification performance. However, I still have two concerns:

1. Whether the verifier is trained on offline or online solutions seems to have no meaningful impact on reasoning accuracy.
2. Even when the verifier achieves higher accuracy, it offers no meaningful gain in reasoning performance at test time once the additional compute cost is taken into account.

References

[1] Nishad Singhi et al., When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning, arXiv 2025.

Summary

In summary, we address your remaining concerns with new data and clarifications:

Concern 1: Whether the verifier is trained on offline or online solutions seems to have no meaningful impact on reasoning accuracy.

Our method, RISE, is built on online self-verification. Our experiments demonstrate that this online training of verification consistently and meaningfully improves reasoning accuracy. On the newer Qwen3 models, RISE delivers average gains of +5.6% over the PPO baseline, with improvements of over +10% on the Minerva Math. The limited gains from the offline ablation only strengthen our claim that learning from online, on-policy data is critical, validating the core design of the RISE framework.

Concern 2: Even when the verifier achieves higher accuracy, it offers no meaningful gain in reasoning performance at test time once the additional compute cost is taken into account.

Verification responses are computationally efficient (2–14% of a solution's token cost). Our new experiments, using the fair comparison you suggested, show that adding self-verification provides a lightweight yet effective performance boost, especially for larger models and in low-budget settings (e.g., +3.4% at k=2 for the Qwen3-8B model). Furthermore, RISE-trained models exhibit superior majority voting accuracy even without explicit verification, demonstrating an inherent test-time advantage.

We hope these clarifications can address your concerns. We thank you again for your constructive engagement.

Dear Reviewer KxXf,

Thank you for the detailed follow-up and for pushing us to further clarify our contributions. We sincerely apologize if our previous phrasing came across as confrontational; our intent was only to highlight the key technical distinctions that we believe are central to our work. We value this discussion and have incorporated your feedback to strengthen our analysis.

We will address your two remaining concerns below.

1. On the Test-Time Benefit and Compute Cost of Self-Verification

We agree that the token cost of verification was not detailed in the original manuscript, and we thank you for pointing out this omission. We will add a detailed breakdown to the appendix to prevent any confusion.

Your concern is that the compute cost of verification may outweigh its benefits. As you advise, the baseline now changes to the RISE model itself, using majority voting. Our analysis shows that even after accounting for the minor additional cost, self-verification provides the performance gain, especially for larger models and in low-budget scenarios.

Verification is computationally inexpensive. As shown in our new analysis (including the latest Qwen3 models), a verification response consumes only a small fraction of the tokens required for a full solution generation.

Self-verification improves performance at a matched compute budget. Following your suggestion, we compare RISE + self-verify against a RISE baseline with an equivalent solution budget. The results show the advantage for using the lightweight self-verification explicitly.

• Comparison at k=4: Comparing against the equivalent RISE baselines, RISE + self-verify shows the benefit on 4 out of 5 models, with the largest models seeing the biggest gains.

• Comparison at k=2: The benefit is even more pronounced in low-budget scenarios, where each sample is more critical. At k=2, self-verification provides a substantial boost across all model sizes.

Finally, we wish to highlight another key result: even without explicit test-time verification, RISE-trained models achieve higher majority voting accuracy than PPO models under an equivalent budget (see Figure 2). This suggests that our method inherently produces more self-consistent models, offering a built-in test-time advantage.

2. On the Impact of Online vs. Offline Verification on Reasoning

We appreciate you highlighting the result from the offline verification experiment. To be precise, offline verification does yield a small, consistent improvement over the PPO baseline across all model scales, as shown in the table below.

However, you are right that this impact is not large, and we believe this finding is not a weakness of our work, but rather a motivation for it. The core thesis of our work is that the verification skill must be learned online, using the model's own on-policy generations, to meaningfully improve reasoning. The relative ineffectiveness of the offline approach is precisely the evidence that supports our claim and highlights the necessity of our proposed online framework, RISE.

In contrast to the small gains from offline data, our online method, RISE, yields higher and more meaningful reasoning improvements. As shown in our first rebuttal, on the newer Qwen3 models, RISE achieves average gains of +5.6% on both the 4B and 8B models over the PPO baseline across 5 benchmarks. This difference between the offline and online results confirms that the tight, integrated loop in RISE is the key to its success.

Thank you for conducting new experiments, which make your claims much stronger. I recommend dropping the left plot of Figure 6. In that plot, Qwen2.5-1.5B-Instruct shows higher reasoning accuracy with offline verification than with online verification, which partially weakens your claim. I believe it is sufficient to claim that training with online verification boosts verification accuracy and allows the model to function reliably as a verifier at test time. My initial lower score reflected concerns about the robustness of your evaluation, but after our discussion, I now find the evaluation much more reliable. Please add the new test-time scaling experiments to the Appendix; including them will clearly make your paper even stronger.

Dear Reviewer KxXf,

As the discussion period is concluding, we want to thank you again for your thoughtful engagement. Your feedback encouraged us to demonstrate the effectiveness of our method on more recent models, run compute-matched test-time evaluations, and clarify the motivation behind our online training framework, all of which we believe have meaningfully strengthened the paper.

We hope our detailed responses and the new results have addressed your concerns. We remain available for any further questions you may have.

Best regards,

The Authors

We thank the reviewer PrjU for finding our idea “interesting and probably quite novel”.

Q1&Q2: Motivation for a single, self-verifying model.

These questions get to the heart of our motivation. Our goal is not just to build a separate verifier, but to improve the generator's own reasoning process.

The key issue we address is "superficial self-reflection", where the emergent self-reflections do not lead to correct final answers. By training a single, unified model to both solve and verify, RISE encourages the model to develop an internally consistent and correct reasoning process. A model that can reliably check its own work is more likely to produce robust and trustworthy reasoning. This aligns with trends in state-of-the-art models (e.g., OpenAI's o1, DeepSeek-R1) which heavily feature self-reflection as a core capability.

Q3: I am quite unfamiliar with the related literature and I have little to no expertise on this topic, but I suspect the claim in Sec 2 line 80-82 might require some further clarification. The authors seem to claim no previous work fine-tuned the verifier model using verifiable rewards, is that correct?

To clarify our claim in Lines 80-82: our novelty lies in explicitly and simultaneously training both the generator and the verifier within a single, online RL loop, using on-policy data and a unified objective. While others have trained verifiers separately or used them to generate rewards, to our knowledge, this tightly-coupled, joint-training framework for both skills is novel.

Q4: Sec 5.1: do you consider any baseline where generator and verifier are RL-tuned independently? That seems a much more relevant baseline than Zero-RL, which only finetunes the generator/problem-solver. Also, how is SFT trained?

Tuning a separate verifier is an interesting direction, but it addresses a different research question (building the best separate components). Our primary goal is to ablate the effect of our proposed joint-training objective. Therefore, the most direct and scientifically sound baseline is Zero-RL, which uses the identical RL setup but without our proposed verification objective. This allows for a clean comparison demonstrating the benefit of RISE's core contribution.

Regarding SFT: SFT models are instruction-tuned on the golden solutions from the MATH-Hard (Level 3-5) dataset, and detailed implementations can be found in Appendix C.

Q5: Sec 5.2: lines 208-211: this paragraph seems to suggest the authors expect the verification performance of a model to decrease with model size and that RISE limits this problem. However, I would expect the verification performance to increase with model size; instead, in Table 1, it seems that the verification performance of RISE decreases. Still, other models in Table 1 generally show improved verification performance when increasing parameters. Hence, I have some doubts about the strength of the authors' argument, can they maybe comment on this?

Thank you for the opportunity to clarify. We do not claim that verification performance should decrease with model size. We believe the self-verification results in Table 1 show variance across scales, not a decreasing trend. This is likely because self-verification on its own outputs is a relatively easy task that even smaller models can learn well, leading to comparable performances.

To further validate our claim, we evaluated the models on a fixed external solutions (GPT-4o responses on the 5 math benchmarks). Here, we see no decreasing trend; rather, all RISE models maintain strong verification accuracy across scales.

Q6 & Q7: Verification frequency and metrics.

We completely agree that higher frequency is not the goal; more effective verification is. As we argue in the paper, the increased frequency is meaningful precisely because it leads to better outcomes.

• First, we kindly note that the paper does not claim a "substantial increase" for all models; we explicitly state the increase for the 1.5B model is modest and the increase "becomes substantial as scale grows (+1.09% for 3B and +1.05% for 7B)" (Line 322)." We will rephrase this in the revised version to ensure clarity.
• More importantly, as shown in Figure 7 (right), the accuracy of the self-verified subset of responses is significantly higher for RISE models. This shows the verifications are not just more frequent, but more impactful, helping the model correct or confirm its reasoning path.
• Regarding new metrics (Q7), we agree that measuring when verification is "needed" would be insightful. However, this is non-trivial to define objectively, as we cannot know the model's internal state of uncertainty. We believe our current metric—showing improved accuracy when self-verification is invoked—is also a direct indicator of its effectiveness.

Societal Impact: The authors claim no societal impacts as their work focuses on improving reasoning capabilities on tasks such as mathematics. However, claiming that improving reasoning capabilities just has no societal impacts seems a bit an oversimplification to me.

Thank you for this important point. You are right that "no societal impact" is an oversimplification. We will revise this section. Our intended meaning was that our work, as a foundational training framework evaluated on public math datasets, does not introduce immediate, direct societal risks. However, we acknowledge that any advance in AI reasoning capabilities has potential downstream impacts, both positive and negative. We will add a sentence stating that the method we propose should be developed and applied responsibly.

Thank you for your response. We are glad that our initial rebuttal resolved some of your concerns and are willing to discuss the remaining questions you have.

On the potential baseline for a separately trained verifier

In this work, we focus on improving the intrinsic reasoning capability of an LLM by enhancing its self-verification behavior, rather than optimizing the performance of a generator-verifier system. As suggested in prior work [1][2][3], self-verification is considered an essential behavior of strong reasoning models, as it enables the model to assess its own reasoning process and correct any mistakes. Moreover, as noted by [4], a potential benefit of self-verification over using a separate verifier is that reasoning models internally encode a notion of correctness, which self-verification can leverage to further improve verification efficiency.

On the other hand, we agree that verification can be performed by a separate verifier model or even by external tools. However, such setups involve multiple entities and require systematic collaboration between the generator and verifier, which is somewhat beyond the scope of our main focus. In addition, involving a separate verifier model introduces challenges in ensuring a fair comparison to our current experiment setting, due to efficiency differences (e.g., the need to run inference over multiple models).

That said, we believe including this ablation would enrich the research depth of the paper, particularly in evaluating the effectiveness of different system designs. We will incorporate an experiment involving separately RL-tuned generator and verifier models and provide the corresponding analysis in the future version, given the limited time and resources during this rebuttal period.

[1] Gandhi, Kanishk, et al. "Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars." arXiv preprint arXiv:2503.01307 (2025).

[2] Zeng, Weihao, et al. "Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild." arXiv preprint arXiv:2503.18892 (2025).

[3] Yang, Shiming, et al. "Demystifying Long Chain-of-Thought Reasoning." Forty-second International Conference on Machine Learning.

[4] Zhang, Anqi, et al. "Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification." arXiv preprint arXiv:2504.05419 (2025).

On the limitations for verification frequency and verified-response accuracy metrics; suggestions for a new metric

To characterize the reflective behavior of LLMs, prior work has introduced metrics such as verification frequency [1][2], where an increase is often interpreted as a desirable signal due to its empirical correlation with improved reasoning accuracy. However, as previously discussed, this metric has some limitations and should be interpreted with caution. Similarly, we agree that verified-response accuracy (referred to as "verification accuracy" in your response) does not provide the full picture, as it reflects correlation rather than causality or necessity. (That said, we still believe defining the necessity of self-verification is challenging, as verification on correct reasoning paths may still yield benefits, such as reinforcing internal consistency and increasing confidence.) In our revision, we will provide clearer explanations of these metrics' limitations and interpretive boundaries.

To address this concern, we followed your suggestion to examine the intersection between verification requests and mistaken reasoning trajectories. Due to limited time and resources, we conducted this experiment on the AIME 2024 dataset (30 problems and 240 responses per group). Specifically, we constructed a judging prompt and leveraged an advanced model, GPT-4o, to assess whether each reasoning trajectory contained an error prior to the invocation of self-verification. We then reported the percentage of responses that invoke self-verification on erroneous trajectories (i.e., when verification is actually needed), alongside the overall benchmark accuracy.

The results show that RISE models request verification more often on trajectories that are actually going wrong, and they constantly achieve better overall accuracy. We will conduct further experiments to ensure the reliability of this conclusion.

We hope these clarifications, new results, and our commitment address your concerns. We are grateful for your insightful feedback, which has genuinely helped us strengthen the paper.

Thank you for the positive evaluation and for recognizing the thoroughness of our experiments and the utility of our method.

W1&W2: Clarity on baseline name and verification reward.

Thank you for these excellent suggestions for improving clarity.

1. Baseline Name: We agree. "PPO" is a more precise and standard name for the baseline. We will change "Zero-RL" to "PPO" in the revised paper.
2. Verification Reward: Absolutely, let us further clarify it. The verification reward is computed based on the exact match between the score extracted from the model’s verification response and the reward provided by the rule-based outcome verifier for that solution. Specifically:

• +1.0 reward: for correctly predicting the score in the specified format (e.g., \\boxed{1}).
• -0.5 reward: for predicting an incorrect score in the correct format.
• -1.0 reward: for generating an invalid format (e.g., missing the \\boxed{} wrapper).

We will add these explicit details to Section 4.1.

Q1: More out of curiosity, have the authors tried testing RISE on other verifiable domains such as code?

This is an excellent point. While our math-trained models were not exposed to other domains, we conducted a zero-shot transfer evaluation to test their generalizability. We evaluated the models on general knowledge (MMLU-Pro), science (GPQA), and code (HumanEval).

The results show that RISE, trained only on math, consistently improves both reasoning and verification accuracy across these diverse, out-of-distribution domains. This demonstrates that the underlying self-verification skill learned by RISE is robust and transferable.

Due to resource constraints, we have not yet conducted full RL training on non-math domains, but we believe these results strongly support our generality claims and we will add them to the paper.

Q2: How would the authors consider applying a similar strategy to learned reward models?

This is an insightful direction. In this work, we pair a more challenging task (reasoning) with a relatively easier auxiliary task (verification), and find that co-training improves both. Reversing this setup, i.e., using generative reasoning to facilitate training a learned reward model, may be less straightforward, as the auxiliary task is harder than the main on. However, it remains a promising direction worth exploring, particularly for the learned generative reward models.

Q3: Related to question 1, what would the authors think would be the effects of doing multi-task RLVR with this auxiliary loss?

We believe RISE is extendable to multi-task RLVR settings. The key would be to ensure that the tasks (and their verification counterparts) are balanced in difficulty and that their reward scales are properly aligned. With a well-curated multi-task dataset, we expect RISE to be highly effective.

Q4: Finally, how sensitive is the performance of this method to the prompt chosen for the verification task?

This is an important practical question. We ran an additional experiment on Qwen2.5-7B using a paraphrased verification prompt with minor structural changes.

The results show that RISE is robust to the paraphrasing of the verification prompt. While verification accuracy saw a minor drop, the primary reasoning performance remained stable and still significantly outperformed the PPO baseline.

Thank you for the response. It's great to see the followup results and addresses most of the additional questions that I had. I will keep my score to reflect this.

We thank Reviewer rqQc for the positive and constructive feedback, highlighting our elegant solution and robust results.

W1: All experiments are in maths with exact-match verifiers. It is unclear whether the method extends to domains where verification is noisy or differen domain (e.g., code-related QA, general reasoning, factual QA). A transfer study or cross-domain demonstration would strengthen generality claims.

This is an excellent point. While our math-trained models were not exposed to other domains, we conducted a zero-shot transfer evaluation to test their generalizability. We evaluated the models on general knowledge (MMLU-Pro), science (GPQA), and code (HumanEval).

The results show that RISE, trained only on math, consistently improves both reasoning and verification accuracy across these diverse, out-of-distribution domains. This demonstrates that the underlying self-verification skill learned by RISE is robust and transferable.

Due to resource constraints, we have not yet conducted full RL training on non-math domains, but we believe these results strongly support our generality claims and we will add them to the paper.

W2: While verification accuracy soars, reasoning improvements over Zero - RL are relatively small (≤ 2 pp on most benchmarks). The paper could discuss why better self - checking does not translate into larger solving gains. Also, is there any quantified correlation between the verification accuracy and the final RL performance?

Thank you for this thoughtful question. The difference in gains stems from the fact that verification is an easier task than multi-step reasoning. As noted in prior work, a "Generation-Verification Gap" exists where models learn to verify more easily than they learn to generate correct solutions. Our results are consistent with this phenomenon.

While verification gains don't translate 1-to-1 to reasoning gains, stronger verification directly contributes to better reasoning. As we show in Section 5.5 (Figure 7 right), the accuracy of solutions that invoke self-verification is significantly higher in RISE models than in the baseline. This shows that the learned skill is being used effectively.

Regarding a quantified correlation, we show a positive trend between the two reward types during training in Figure 4. However, the exact correlation is complex and depends on the base model and dataset. The key takeaway is that improved self-verification, enabled by RISE, provides a more reliable mechanism for the model to identify and correct error reasoning paths, leading to overall performance improvements.

W3: Because the verifier learns from on-policy data and shares parameters with the generator, it may overfit to its own solution style. Evaluating on external solutions or holding out a verifier dev set would alleviate this concern.

This is a valid concern. To test for overfitting, we evaluated our RISE-trained verifiers on their ability to judge the correctness of external solutions generated by GPT-4o.

Our results show that RISE models outperform strong, dedicated verifiers like GPT-4o and Math-Shepherd on these external solutions. This indicates that the verifier learned by RISE generalizes well beyond its own generation style and has acquired a robust, non-overfit notion of correctness.

W4: The paper does not address cases where the verifier is confidently wrong, or how such failures might compound during training.

This is handled directly by our reward mechanism. As described in Lines 153-163, the verifier receives a reward signal based on whether its predicted score (e.g., correct/incorrect) matches the ground-truth score from the outcome verifier.

Critically, if the verifier is confidently wrong (e.g., it labels a wrong solution as correct), it receives a negative reward and is penalized. This explicit feedback loop directly discourages overconfident errors and prevents them from compounding during training, forcing the verifier’s prediction to align with the ground-truth outcome.