Boosting LLM Reasoning via Spontaneous Self-Correction

We appreciate your constructive feedback and the time you have devoted. Please kindly find our detailed responses to your concerns as follows.

Q1 Self-correction turns. While the presentation and theory in the paper is general-purpose, the experiments were performed with a lesser number of self-correction turns. ... It is confusing how the model is generating more rounds in the RL stage than it has been taught in the SFT stage.

A1 We firstly would like to clarify that D_pair contains both one-turn (question, correct solution, verification) and two-turn (question, incorrect solution, verification, correct solution) trajectories, as described in Section 3.2 and Algorithm 2. We will include both cases in Figure 2 (left) accordingly.

Besides, we would like to confirm that PairSFT with one-turn (without correction) and two-turn (with correction) generations successfully initiates the dynamic multi-turn behavior. Our formulation and training data promotes spontaneous message and adaptive generation based on verification outcomes (see prompt in appendix for reference). Therefore, the looping until verified correctness behavior in Figure 2 (right) is an accurate illustration of the rollouts in practice.

Table 2 presents our per-turn performance analysis over turn1->2, where the majority of self-correction occurs. In practice, all finetuned models perform multiple rounds of self-reflection. We hereby present the complete results below, where the first table shows the turn2->3 performance of all models, and the second table shows the all-turn performance of the 8B model (as the other stopped reflection earlier). Results suggest that the 8B model reaches a maximum of 6 turns while the 70B models reach a maximum of 3 turns across all 500 evaluation questions. This observation aligns with our discussion in Section 4.2, where stronger models tend to achieve correct solutions sooner. We also observe that the amount of questions requiring additional solutions drops over turns, aligning with the looping until verified correctness behavior. Overall, SPOC achieves improvement over turns.

We will include both tables in the appendix of our revised manuscript.

Q2 Discussions on thinking models. It is unclear how the method is required in the light of thinking models that can perform self-verification/self-correction before generating the final answer. ... We observe that the authors show that SPOC works on top of Deepseek-Distill models too so it will be helpful to have more nuanced discussion on why self-correction is needed for thinking models too.

A2 SPOC triggers self-verification/correction based on its judged correctness of the previous message, which introduces binary correctness labels for each message. Hence, SPOC applies process supervision to reward correct and penalize incorrect solutions and verifications. Our ablations underscore the importance of such process-level supervision. Relying solely on final answer correctness, thinking model’s outcome reward does not supervise intermediate reflection steps within the long CoT, potentially resulting in undesirable excessively long reasoning [2,3] and hindering efficient compute usage.

SPOC is compatible with R1’s long thinking, demonstrated by experiments on DeepSeek-R1-Distill models. These experiments not only support our accessesment of SPOC’s effectiveness and generalizability across initial model capabilities, but also serve as an exploration towards reflective reasoning in long CoT. As noted in Section 5, to address the prohibitive length of long CoTs, an interesting future direction is to extend SPOC to partial solutions in long CoTs, using step-level process rewards to guide RL training and enable dynamic revisions when errors are detected until reaching the final answer.

Q3 Related work. In Line 38, ... prior work has studied test-time scaling in traditional self-reflection too.

A3 We appreciate the reviewer for bringing up [4]. This concurrent work is akin to SPOC in a sense that at inference time self-correction is triggered by results of self-verification, which differs from traditional reflection works [1,5]. Differently, [4] involves no training to initiate/improve spontaneous reflection, thereby requiring an explicit prompting to generate each single message, while SPOC performs real-time adaptive reflective reasoning in a single inference pass, allows for more flexible deployment. We will discuss [4] in related works and revise our presentation in our next version of the manuscript.

(continued next)

Hi,

I thank the authors for their diligent rebuttal.

1. Self-correction turns

• It is good to see that the model is not limited to 1 or 2 turns during inference even though the training (SFT) data subjects it to limited turns. I think this point is nuanced and not properly mentioned in the paper which led to the confusion.
• It is interesting to see that 8B and 70B models go uptill 6 and 3 turns, respectively, and the revised paper should include those numbers.

2. Thinking models

• I agree that the reward modeling on final answer correctness as well as verification steps can provide potentially useful signals; and quite unique to the design of this approach.

3. Related work and presentation

• Sounds good!

Having gone through other reviews, I do agree that the paper will benefit from inclusion of more models and more popular RL algorithms like GRPO in the future versions. I have increased my score to 7 to reflect my updated opinion.

Thanks for your feedback and the time you have taken. We provide detailed clarifications on your concerns as follows.

Q1 Self-improvement. SPOC relies on synthetic data generated by the base model for fine-tuning, which may introduce bias if the initial model has systematic errors.

A1 Self-improvement approaches finetune the base model with fully self-synthesized data, serving as an important complement of distillation when expert distillation data is often unavailable or expensive in practice. To mitigate potential bias embedded in initial models, we conduct extensive experiments covering a diverse range of initial model capacities. In addition to widely-used Llama models, we also experimented on DeepSeek-R1-Distill models (both 8B & 70B) which are distilled from DeepSeek-R1 with modified configs and tokenizers [3]. Such experimental extensiveness promotes sufficient coverage of initial model capabilities and reasoning levels, and our results demonstrate consistent effectiveness and generalizability across initial models and task difficulties.

Q2 Policy optimizer. SPOC with RLOO performs much better than RAFT, so it makes more sense to use RLOO as the default RL algorithm and compare it with RLOO as the baseline.

A2 We aim to showcase the improvement achieved by SPOC’s reflective reasoning, rather than merely reaching a better performance using a stronger baseline optimizer. Empirically, SPOC with RAFT reaches consistent performance boosts across base models and task difficulties, as shown in Table 1. RAFT’s robustness allows for wide applications over diverse learning scenarios [1,2]. Our extensive experiments promote proper and fair interpretations of SPOC’s enhancements and robustness.

We apply both RAFT and RLOO to provide a comprehensive understanding, as they follow different optimization schemes, where the former selects the best-of-N response while the latter uses all generated responses to optimize the policy. Consistent and significant performance improvements shown in our results demonstrate our method’s effectiveness.

Q3 Policy optimizer. SPOC only integrates with RAFT and RLOO. It is not clear whether it will be compatible with other RL algorithms.

A3 As mentioned above, RAFT and RLOO are representative multi-rollout algorithms following different optimization schemes. Considering single-rollout as a special case of multi-rollout (K=1), other single-rollout algorithms (eg. REINFORCE or PPO or extended variants) also naturally fit our learning formulation. Our learning formulation and empirical experiments demonstrate SPOC’s general compatibility with any RL algorithm.

---

References:

[1] Xiong, Wei, et al. "Self-rewarding correction for mathematical reasoning." arXiv preprint arXiv:2502.19613 (2025).

[2] Xu, Tengyu, et al. "The perfect blend: Redefining RLHF with mixture of judges." arXiv preprint arXiv:2409.20370 (2024).

[3] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

We appreciate your insightful feedback and the time you have dedicated. Please kindly find our responses to each point as follows.

Q1 Model family. The experiments are done within the same model line (Llama).

... at least discuss why the method should transfer beyond Llama ...

A1 Despite the Llama family, our experiments cover different model capacities and parameter scales (especially 70B full-finetuning often lacking in literature). Our experiments also include DeepSeek-R1-Distill models (both 8B & 70B) which are distilled from DeepSeek-R1 with modified configs and tokenizers [6]. Due to different parameter scales, training data, and training approaches, base models covered in our work exhibit diverse reasoning capabilities and behaviors. Since SPOC demonstrates consistent effectiveness and generalizability across a broad spectrum of initial models and task difficulties, our empirical results strengthen SPOC's applicability to other model families.

Q2 Related work and main advantage. The concept of first curating data with reflective reasoning behaviors and distilling it via SFT or RL is not quite new. I'd be curious to learn the unique finding of this paper.

In general, the manuscript would benefit from clarifying what is the main advantage of SPOC.

A2 Compared to recent literature, SPOC showcases the following unique advantages:

1. Simple mechanism design and dynamic reflective reasoning. Our training pipeline effectively bootstraps the model’s spontaneous self-verification and correction, enabling explicit signaling of continuation/termination, thereby adaptive generation.
2. Joint optimization of both solutions and verifications. The solution outcome reward does not supervise intermediate messages within a multi-turn generation before the final answer. Besides outcome reward, SPOC applies process supervision to reward correct and penalize incorrect solutions and verifications. Ablations underscore the importance of such process-level supervision.
3. Consistent enhancement across initial models and task difficulties. Extensive experiments demonstrate SPOC’s effectiveness and generalizability as highlighted earlier.

We appreciate the reviewer for bringing up related works. We will incorporate the following discussions in our revised manuscript:

1. [1-2] utilize multi-agent conversations. Similar to MCTS-DPO [7], [1] focuses on constructing preference pairs with shared prefix for preference optimization, rather than enabling reflective reasoning. [2] synthesize training data via social scenario simulations which involves complicated mechanism design and numerous instruction prompting, whereas SPOC relies on much simpler pipeline and minimal prompting. Besides, [2] distill from DeepSeek-R1-Distill-Qwen-32B for reasoning tasks, while SPOC is a self-improvement approach that fine-tunes initial models with fully self-synthesized data.
2. Recent works on long COTs have emerged [3,6] as an effective approach. Beyond outcome rewards, SPOC provides message-wise supervision to guide learning. Moreover, SPOC’s multi-turn formalism enables compatibility with R1’s long thinking, demonstrated by experiments on DeepSeek-R1-Distill models.
3. [4-5] leverages distillation on small datasets, with training data curated using strong models and multiple selection criteria. [4] proposes budget forcing to enforce sequential test-time scaling, while [5] investigates the impact of prerequisite knowledge. Both works complement the line of self-improvement approaches.

(continued next)

We highly appreciate the reviewer’s follow-up feedback. Please kindly find our further clarifications as follows.

Q5 Application scenarios. While I appreciate the simplicity and effectiveness of SPOC, the arguments presented do not sufficiently clarify the specific scenarios where SPOC would be preferable to existing methods.

A5 Self-correction is a desirable capability that enables the model to improve over initial responses. SPOC applies to scenarios where (1) self-correction is beneficial for reasoning tasks, (2) agent use cases that require multi-turn execution, and (3) adaptive response generation efficiently utilizes computational cost at inference time. Specifically for point (3), long-thinking models have excessive thought length [8], resulting in high computational costs at deployment and performance degradation. SPOC seeks to improve reasoning performance in the meantime achieving efficient test-time scaling. Per-turn performance analysis (table 2 and tables in responses to Reviewer VSGv) suggest that stronger models tend to achieve correct solutions sooner, without trying many additional self-corrections. This observation aligns with our discussion in Section 4.2.

Moreover, we’d like to clarify that SPOC is not intended to replace existing methods. In fact, SPOC is well compatible with R1’s long thinking, demonstrated by experiments on DeepSeek-R1-Distill models. These experiments support our accessesment of SPOC’s effectiveness and generalizability across initial model capabilities. Furthermore, the multi-turn formalism serves as an exploration towards reflective reasoning within long CoT. As noted in Section 5, to address the undesiarable prohibitive length, an interesting future direction is to extend SPOC to partial solutions within long CoTs, using step-level process rewards to guide RL training and enable dynamic revisions when errors are detected until reaching the final answer.

Q6 Process feedback. I'm not sure about the characterization of the reward as "process-based"...

A6 We appreciate the reviewer for bringing up [1], which characterizes a step-level process reward. To clarify, our notion of process supervision is at the message level (illustrated in Figure 3a), as compared to the outcome reward solely on the correctness of the entire trajectory’s final answer (illustrated in Figure 3c). Figure 3c characterizes a setting where the entire trajectory is considered correct if the final answer is correct, regardless of potentially incorrect intermediate verifications/solutions. Such reward does not penalize unnecessarily long trajectories, and is not as effective as message-level supervision demonstrated by our ablations.

Concurrent work on self-correction [2] adopts a similar description. We will include the clarification above in the revised version of our manuscript.

Q7 Self-improvement and distillation. It's unclear why one would choose SPOC over distillation, particularly when options like the R1 API are readily and affordably available. If we consider the scenario of training the strongest models (e.g. no better model for distillation), then cold starting with RL is sufficient to self-improve as demonstrated by R1.

A7 We’d like first to clarify that this work does not advocate choosing SPOC over distillation. Distillation and self-improvement are well compatible with each other. Our experiments on DeepSeek-R1-Distill models fit the scenario of applying both methods. Obtained by distilling from R1, these models already achieve highly strong results over reasoning tasks, and SPOC further boosts their performance consistently.

Also, this paper is not intended to replicate R1, as it was concurrently developed. We mainly aim to investigate the benefit of dynamic spontaneous self-corrections, and introduce a learning framework to initialize such behavior and improve reasoning capabilities.

---

References:

[2] Ma, Ruotian, et al. "S $^ 2$ R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning." arXiv preprint arXiv:2502.12853 (2025).

We appreciate your constructive comments and the time you have taken. Your valuable suggestions have significantly elevated our work. Please kindly find our detailed response below.

Q1 Verifier Reliability. Beyond the reported Verif.Acc.@t1, were the verifier's false positive and false negative rates measured? Additionally, was the verifier's behavior on slightly incorrect explanations examined, and could you share any related statistics such as a confusion matrix or an error-type breakdown?

A1 We provide more detailed diagnostics in the table down below. Each confusion matrix corresponds to a base model and task pair, with the rows and columns indicating the actual and predicted solution correctness, respectively — i.e., diagonal cells represent the true positive (TP) and true negative (TN) rates while the off-diagonal cells represent the false positive (FP) and false negative (FN) rates. We observe the following phenomena:

1. On easier tasks, the proposer has higher solution accuracy, and the verifier tends to show higher TP&FP and lower TN&FN.
2. Stronger models that reach higher solution accuracy also have higher TP&FP.
3. The small model’s high verification accuracy attributes largely to its higher TN.

We will include the table in the appendix of our revised manuscript.

Q2 Per-Turn Performance Analysis. Could you provide a per-turn performance analysis, similar to Table 2 for MATH500, for the AMC23 and AIME24 benchmarks to better understand SPOC's generalizability?

A2 We provide the per-turn performance statistics for AMC23 and AIME24 as follows. The results are consistent with MATH500 analysis in Table 2. SPOC generally improves or maintains performance on the second solution turns. The smaller model has lower final accuracy yet larger turn-wise improvements, while larger models tend to achieve correct solutions sooner at turn1. Moreover, turn-wise corrections occurs less in these two challenging competition benchmarks, as they contain significantly fewer questions than MATH500. We will include both tables in the appendix of our revised manuscript.

Q3 Computational Cost. The online RL phase requires 32 × H100 GPUs (Appendix B), which is prohibitively expensive for most research groups and small teams.

A3 We utilized stated compute to accelerate experiments across various initial models and parameter scales (especially 70B full-finetuning often lacking in the literature), so as to comprehensively evaluate our method’s effectiveness and generalizability. Our results not only demonstrate SPOC’s efficacy across model scales, but also present interesting different behaviors by different model sizes (such as observations on FP&FN above), promoting a better understanding of SPOC’s strengths.

Each 8B run takes approximately 8~12 hours with the stated compute. If the runtime and experimentation on large models are not a concern, 4 x H100 GPUs are sufficient to conduct the RL training.

---

Verifier reliability performance

Per-turn performance analysis

AIME 2024

AMC 2023

Thank you for your insightful feedback and the time you have taken. Please kindly find our detailed responses below.

Q1 Policy optimizer. The paper lacks a GRPO baseline which would test the model could learn to interleave solving and verifying with online RL on the same data -- without which it is hard to gauge their method's effectiveness.

A1 We would like to highlight that our message-level RL framework is compatible with any policy optimization method. We aim to showcase the improvement achieved by SPOC’s reflective reasoning, rather than merely reaching a better performance using a stronger baseline optimizer. To provide a comprehensive understanding we apply RAFT and RLOO, two representative algorithms following different optimization schemes, where the former optimizes over the best-of-N response while the latter optimizes over all generated responses. Our experiments have shown that SPOC with either optimizer achieves consistent and significant performance improvements across model capacities and benchmark difficulties, demonstrating our method’s effectiveness.

GRPO is conceptually very similar to RLOO, as they rely on similar advantage update to provide process supervision for every message relative to the others [1]. Since our results already show that RLOO yields stronger results than RAFT, our experiments present sufficient evidence to demonstrate the benefits of the advantage function.

Q2 Related work. it is unclear how SPOC compares to other long CoT methods, which have also demonstrated interleaved solving, backtracking, and verification in longer CoT generations

A2 We appreciate the reviewer for bringing up related works. We will include discussions below to updated manuscript.

Gandhi et al. [2] investigate how priming with controlled behavioral datasets affects subsequent RL training, where such datasets are either curated by Claude-3.5-Sonnet on Countdown or by Qwen-2.5-32B from OpenWebMath. In contrast to distillation from stronger models, SPOC is a self-improvement approach that fine-tunes initial models with self-generated synthetic data throughout the training pipeline.

Marjanovic et al. [3] conduct a systematic investigation of DeepSeek-R1’s reasoning processes. Compared to R1’s long CoTs, SPOC employs a spontaneous reasoning strategy that adapts generations based on verification outcomes. Beyond R1’s solution outcome reward, SPOC applies process supervision to reward correct and penalize incorrect solutions and verifications. Moreover, SPOC’s multi-turn formalism is compatible with R1’s long CoT, demonstrated by experiments on DeepSeek-R1-Distill models (both 8B & 70B) in Table 1. SPOC shows consistent effectiveness and generalizability across initial model capacities and task difficulties. As noted in Section 5, to address the prohibitive length of long CoTs [3], an interesting future direction is to extend SPOC to partial solutions in long CoTs, using step-level process rewards to guide RL training and enable dynamic revisions when errors are detected until reaching the final answer.

---

References:

[1] Lambert, Nathan. "Reinforcement Learning from Human Feedback." arXiv preprint arXiv:2504.12501 (2025).

[2] Gandhi, Kanishk, et al. "Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars." arXiv preprint arXiv:2503.01307 (2025).

[3] Marjanović, Sara Vera, et al. "DeepSeek-R1 Thoughtology: Let's think about LLM Reasoning." arXiv preprint arXiv:2504.07128 (2025).