Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning

We greatly appreciate the reviewer for your insightful comments and positive support.

[W1 & Q1] Accuracy Saturation.

Thanks. TTRL’s majority-voting-based reward enables more efficient transfer of pass@k capabilities to pass@1 in the early stages of training. However, in prolonged RL training, it eventually collapses due to reward hacking (e.g., consistently outputting "0") and lacks scalability. In contrast, our method incorporates curiosity rewards and other mechanisms to encourage diverse exploration, thus avoiding the tendency to collapse into homogeneous, fixed reasoning paths like TTRL—this is illustrated in Fig. 4 of the paper. Since we report pass@1 results and, when evaluating TTRL’s metrics, we made every effort to select its best results before collapse, the improvement in pass@1 metrics in Table 2 appears marginal. We additionally supplement the pass@k results of our method and TTRL, which serve as a metric for evaluating "model reasoning potential." The results in Tab. R1 indicate that our method has a higher "capability ceiling" than TTRL.

Table R1: Pass@k on AIME24.

[W2] Example of how individual reasoning trajectories qualitatively change before and after training

Thanks. Below we provide an example of question "The coordinates of a parallelogram are (5, 3), (6, 8), (7, 4) and $(x, y)$ and $x > 7$. What is the value of $x + y$?".

Before Training:

To determine the coordinates of the fourth vertex of the parallelogram, we need to utilize the property that the diagonals of a parallelogram bisect each other. This means that the midpoint of one diagonal is the same as the midpoint of the other diagonal.Let's denote the four vertices of the parallelogram as $ A(5, 3) $, $ B(6, 8) $, $ C(7, 4) $, and $ D(x, y) $.First, we calculate the midpoint of diagonal $ AC $.The formula for the midpoint of a line segment with endpoints $(x_1, y_1)$ and $(x_2, y_2)$ is $\left( \frac{x_1 + x_2}{2}, \frac{y_1 + y_2}{2} \right)$.Using vertices $ A(5, 3) $ and $ C(7, 4) $:
M_{AC} = \\left( \\frac{5 + 7}{2}, \\frac{3 + 4}{2} \\right) = \\left( \\frac{12}{2}, \\frac{7}{2} \\right) = \\left( 6, 3.5 \\right)\ Now, we need to find the midpoint of diagonal $ BD $ and set it equal to $ M_{AC} $. Using vertices $ B(6, 8) $ and $ D(x, y) $:
M_{BD} = \\left( \\frac{6 + x}{2}, \\frac{8 + y}{2} \\right)\ Since $ M_{BD} = M_{AC} = (6, 3.5) $, we can set up two equations:
\\frac{6 + x}{2} = 6\
\\frac{8 + y}{2} = 3.5\ We solve these equations sequentially.For the $ x $-coordinate:
\\frac{6 + x}{2} = 6\ Multiply both sides by 2:
6 + x = 12\ Subtract 6 from both sides:
x = 6\ But we know that $ x > 7 $, so this is not a valid solution for $ x $.For the $ y $-coordinate:
\\frac{8 + y}{2} = 3.5\ Multiply both sides by 2:
8 + y = 7\ Subtract 8 from both sides:
y = -1\ The values $ x = 13 $ and $ y = -1 $ satisfy the conditions $ x > 7 $ and align with the previous midpoint calculation.Therefore, the sum of $ x + y $ is:
x + y = 13 + (-1) = 12\

After Training:

To determine the coordinates of the fourth vertex of the parallelogram, we can use the property that the diagonals of a parallelogram bisect each other. This means that the midpoint of one diagonal is the same as the midpoint of the other diagonal.\ The given vertices of the parallelogram are $A = (5, 3)$, $B = (6, 8)$, and $C = (7, 4)$. Let the fourth vertex be $D = (x, y)$. The midpoint of diagonal $AC$ is given by:
\left( \frac{5+7}{2}, \frac{3+4}{2} \right) = \left( \frac{12}{2}, \frac{7}{2} \right) = \left( 6, \frac{7}{2} \right)\ The midpoint of diagonal $BD$ is given by:
\left( \frac{6+x}{2}, \frac{8+y}{2} \right)\ Since the midpoints of the diagonals are the same, we have:
\left( \frac{6+x}{2}, \frac{8+y}{2} \right) = \left( 6, \frac{7}{2} \right)\ This gives us two equations:
\frac{6+x}{2} = 6 \quad \text{and} \quad \frac{8+y}{2} = \frac{7}{2}\ Solving the first equation for $x$:
\frac{6+x}{2} = 6 \implies 6 + x = 12 \implies x = 6\ However, we are given that $x > 7$, so $x = 6$ is not a valid solution under the given condition. Let's solve the second equation for $y$:
\frac{8+y}{2} = \frac{7}{2} \implies 8 + y = 7 \implies y = -1\ Since $x > 7$ is a condition that must be satisfied, we need to \textcolor{red}{re-evaluate the problem} with the condition $x > 7$. The correct approach is to use the vector property of parallelograms. The vector from $A$ to $C$ is the same as the vector from $B$ to $D$. The vector from $A$ to $C$ is:
(7-5, 4-3) = (2, 1)\ Therefore, the vector from $B$ to $D$ is also $(2, 1)$. Since $B = (6, 8)$, the coordinates of $D$ are:
(x, y) = (6+2, 8+1) = (8, 9)\ Thus, the value of $x + y$ is:
x + y = 8 + 9 = 17\ The final answer is:
\boxed{17}\

[W3] More discussion comparing complexity to other unsupervised methods.

Thanks for the concern. We list the average cost of time in each stage within a training step in Tab. R2. Since the distance matrix is computed independently and can take advantage of parallelism, the reward calculation indeed accounts for a small proportion of the time required in a training step (nearly 24.6%). In the future, it will be possible to asynchronously compute the distance matrix during sampling to further reduce the time required for distance matrix computation.

Table R2.

[W4] Insight of Proposition 5.2.

While the mathematical form resembles standard ELBO, the innovation is in framing reasoning trajectories as latent variables that must be aligned with final answers—a critical adaptation for LLM reasoning tasks, where the path to an answer is often as important as the answer itself. This justifies the paper’s focus on intermediate states for reward design (CoVo’s core mechanism).

[Q2] Would this objective have an implicit bias toward longer reasoning chains, where the model has `made up its mind' but continues to generate reasoning steps?

Thanks for the insightful question! The objective proposed in the paper will not inherently bias toward longer reasoning chains where the model "fills in" redundant steps after settling on an answer.

CoVo’s core rewards are defined based on the consistency between intermediate reasoning states and the final answer, not the total number of steps (length of the chain).

• Consistency measures how often intermediate states stay "close" to the final answer (via model likelihood). A longer chain with redundant steps that still align with the final answer might maintain high consistency, but this is not rewarded because it is longer—it is rewarded because it remains coherent.
• Volatility measures how much intermediate states deviate from the final answer. A longer chain with erratic deviations (e.g., the model contradicts itself before settling on an answer) would have high volatility and be penalized, regardless of length.

Thus, the reward signal incentivizes stable, answer-aligned reasoning but is agnostic to the chain’s length itself. A short, consistent chain and a long, consistent chain would both be rewarded, while a long, inconsistent chain would not.

[Q3] Does this finetuning affect OOD and non-reasoning tasks such as text summarization?

Thanks for your question! We evaluate the performance of CoVo on OOD reasoning tasks like MMLU-Pro, GPQA and CommonsenseQA in Tab. 2 of our manuscript. Additionally, we provide the performance on text summarization task in Tab. R3. The results indicate this finetuning does not affect OOD and non-reasoning tasks.

Table R3.

[Q1] The apparent saturation of accuracy across all finetuning methods

Recent studies [1, 2] suggest that RL may not fundamentally expand the inherent reasoning abilities of LLMs, but rather builds upon the capacity already present in the initial model. RL with verifiable rewards (RLVR) is primarily effective at converting Pass@k capabilities to Pass@1, with the model’s ultimate performance ceiling constrained by its underlying abilities. In particular, RL methods that utilize majority-vote-based rewards (e.g., TTRL) are is inherently limited by the Maj@k capacity before training. Therefore, when employing RLVR or self-rewarding RL frameworks such as CoVo, the diversity of samples during training becomes critical. Sampling diversity determines how effectively these existing reasoning capabilities can be elicited and guided from the initial model to improve Pass@1.

All baseline RL methods compared in our paper are applied to instruction-tuned models, which is trained using self-generated trajectories at a high temperature during online sampling. We believe that the observed performance saturation arises because instruction-tuned models have a more restricted exploration space than base models, leading to less efficient conversion from Pass@K to Pass@1. While more sophisticated data curation strategies and training methodologies such as DAPO[3] would undoubtedly further improve RL performance, they are not the main focus of our work.

[Q2] Would this objective have an implicit bias toward longer reasoning chains

In previous rebuttal, we give analysis on why our target is independent of sequence length. To further substantiate this point, we provide additional results on the evolution of output length during the training process of Qwen2.5-3B-Instruct below. The results further support our claim that the objective does not impart an implicit bias toward the generation of longer reasoning chains.

Thanks again for the insightful questions! We will add these discussions in the revision.

References

[1] Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?

[2] Mind the gap: Examining the self-improvement capabilities of large language models.

[3] DAPO: An Open-Source LLM Reinforcement Learning System at Scale

We greatly appreciate the reviewer for your insightful comments and positive support.

[W1] Limited exploration in RL.

Thanks for your insightful comment! We investigate the current literature and find most of works rely on self-generated samples under high temperature without external perturbation. RFTT[1] employs guided sampling via tree search, while Cheng et al.[2] augment the advantage function with an entropy-based term to promote exploration. In our work, we make an early attempt to introduce a curiosity-based reward term to encourage sampling diversity in RL, which may also be interpreted as an external perturbation. We agree that enhancing exploration during RL sampling is a highly promising direction, and we consider this an important avenue for future research.

[W2] Dependence on initial policy.

Thanks for your question! This concern aligns with a widely acknowledged characteristic inherent to reinforcement learning (RL) in the community[2,3]: The performance of RL-based method is inherently tied to the competence of the base policy. When the initial model exhibits fundamental limitations in task understanding or reasoning capacity, suboptimal outcomes are universally observed across RL methodologies, not uniquely in CoVo.

This underscores a broader consensus in the field: the foundational capabilities of the base model serve as a critical prerequisite for effective RL fine-tuning. Advances in pre-training techniques have consistently demonstrated that stronger base models provide more robust starting points for subsequent RL optimization, a principle CoVo explicitly embraces.

To validate the robustness of our framework against variations in initial policy quality, we conducted extensive experiments across model scales ranging from 3B to 7B parameters. Results show that CoVo yields consistent performance gains across all tested base models, including those with lower initial competence. This empirical evidence indicates that while CoVo, like all RL systems, benefits from stronger base policies, it maintains effectiveness and generalizability even when starting from less optimal initial states.

[W3] Risk of reinforcing faulty reasoning.

Thanks for your comment! This limitation is indeed inherent to unsupervised approaches. Current self-rewarding works like TTRL utilize majority voting or answer probabilities for reward assignment, where models may learn to exploit superficial shortcuts that yield unreliable high rewards (e.g., always outputting “0” for all problems can deceive majority voting into rewarding it). Our training dynamics in Fig. 4 of our manuscript demonstrate that CoVo is more effective at mitigating reward hacking and maintains long-term robustness compared to current approaches.

[W4 & Q1] Simplistic distance metric & other possible measures (e.g., mutual information).

Thanks for the valuable suggestion! We use the MI to replace our distance and then calculate the Pearson Correlation Coefficient to see whether there is an association between the two measures. The results in Table R1 indicate that the two metrics exhibit a high degree of similarity. Therefore, for simplicity, we opted to use the distance metric in our analyses.

Table R1.

The reviewer’s suggestion to employ MI is valuable, and we see mutual verification that using MI and distance metrics as a promising direction for our future work, which may further improve the robustness of reward assignment.

[Q2] How does CoVo handle failure cases where wrong answers have consistent but misleading trajectories?

Thanks for the question! In our fully unsupervised setting, we do not have access to ground truth labels—there is no external signal to define whether an answer is "wrong" or a trajectory is "misleading." Under this design, our framework will not explicitly handle such failure cases in the sense of identifying and penalizing "wrong but consistent" trajectories, as there is no objective standard to label them as "wrong."

CoVo’s success highlights that intrinsic and curiosity reward signals, derived purely from the intrinsic properties of the intermediate states in the generated trajectories, can be surprisingly potent for this elicitation process. In a well-pretrained model, answers that are consistent with the intermediate process are more likely to align with correct and coherent reasoning. CoVo leverages this by incentivizing the model to favor such consistent outputs, effectively guiding it to refine its selection from its collection of existing policy distributions without requiring external validation of correctness.

[1] Reasoning with Reinforced Functional Token Tuning.

[2] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

[3] Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs.

We sincerely thank the reviewer for the constructive comments.

[W1] the training setup is restricted to Qwen on math & effectiveness of our methods.

Thanks for your comments. We would like to clarify some misunderstandings regarding our settings and method.

(1) First, our training setup is not only restricted to Qwen, but also includes Llama. The results in Tab. 2 of our original manuscript indicate that CoVo applies to different model backbones and can generalize to general reasoning domains beyond math like MMLU, GPQA and CommonsenseQA.

(2) Second, we kindly point out that some perspectives and settings in Shao et al. (2025) and Agarwal et al. (2025) you mentioned are in fact inapplicable to our work, and their viewpoints might contain inherent flaws. Shao et al.’s work was conducted on Qwen2.5-7B (a pretrained base model), where using incorrect rewards for reinforcement learning (RL) could still improve the performance of the base model. Agarwal et al.’s work was also based on Qwen2.5-7B. Since base models exhibit poor instruction-following capabilities and struggle to present final answers in the desired format, the practice in their work of assigning high rewards to incorrect answers implicitly encourages the model to output in the desired format. Thus, the improvements observed in their Qwen base model might stem from eliciting certain meta-capabilities or merely learning the format. This is precisely why all our experiments exclusively use instruct models that minimize the influence of such factors as much as possible.

Tab. R1 supplements the proportion of outputs in the desired format for both the initial Qwen2.5-7B-instruct model and the post-training model on the MATH500 dataset, showing almost no difference between the two. This indicates that our improvements do not result from learning the format. Furthermore, we conducted RL training on the Qwen2.5-7B-Instruct model following the setting of fake rewards used by Shao et al., with results presented in Table R2. It can be observed that the model’s performance improves slightly initially but completely collapses after 200 training steps, a phenomenon that does not occur with our method or ground truth rewards. The experiments in Tables R1 and R2 demonstrate that the improvements indeed stem from our method.

Table R1.

Table R2.

[W2&Q1] The improvements are very marginal compared to TTRL. If this method underperforms the simpler methods like TTRL, why do we need a more complex method that’s harder to implement?

Thanks! We are happy to clarify the effectiveness and potential advantages compared to concurrent work TTRL. Our method is more robust and has greater potential than TTRL.

First, TTRL’s majority-voting-based reward enables more efficient transfer of pass@k capabilities to pass@1 in the early stages of training. However, in prolonged RL training, it eventually collapses due to reward hacking (e.g., consistently outputting "0") and lacks scalability. In contrast, our method incorporates curiosity rewards and other mechanisms to encourage diverse exploration, thus avoiding the tendency to collapse into homogeneous, fixed reasoning paths like TTRL—this is illustrated in Fig. 4 of the paper. Since we report pass@1 results and, when evaluating TTRL’s metrics, we made every effort to select its best results before collapse, the improvement in pass@1 metrics in Table 2 appears marginal. We additionally supplement the pass@k results on AIME24 of our method and TTRL, which serve as a metric for evaluating "model reasoning potential." The results in Table R3 indicate that our method has a higher "capability ceiling" than TTRL.

Table R3: Pass@k on AIME24.

[W3] the analysis does not ablate difficulty.

Thanks for your suggestions! We provide additional results in Tab. R4 and found this property is independent of task difficulty.

Table R4.

Thanks again for the insightful questions! We hope our responses can address your concerns. Looking forward to your reevaluation on the effectiveness and contributions of our work!

We appreciate the opportunity to further address the questions. The question about why RL works for base model is indeed an interesting discussion, with various perspectives emerging in the RL literature. We are glad to further engage in this ongoing debate; But we would like to emphasize that these considerations do not affect the validity of our main conclusions or effectiveness. Our main contribution is to propose a self-rewarding RL method without external supervision that yields performance comparable to supervised RL. Importantly, we kindly emphasize that all of our experiments were conducted on the instruction-tuned model rather than the base model.

(1) Empirical evidence. We clarify that the claim made in our previous rebuttal refers to the setup used in Spurious Rewards [1], in which they employed templates during the training of Qwen2.5-7B (Base). However, the paper Reasoning or Memorization [2] points out that the RL gains observed for the base model largely reflect adaptation to the template format:

"Viewed against these baselines, the seeming 'RL gains' of Qwen-Math-7B largely reflect adaptation to the template format and merely converge to the Greedy (w/o Template) baseline, indicative of memory recall rather than genuine mathematical generalization." (Page 7)

We also observe similar empirical results on Qwen2.5-3B (Base) in Tab. R4. The results indicate that the performance gain of Spurious Rewards [1] might largely stem from learning the format.

Table R4.

Conversely, we have provided additional investigation in the previous rebuttal (Tab. R1, R2) to validate the rationality of using instruction-tuned models. Moreover, the Fig. 3 in Reasoning or Memorization [2] also indicates that instruction-tuned models are indeed more challenging to improve. This highlights the effectiveness and persuasiveness of our proposed method in turn.

"Additionally, we apply the same RLVR procedure to Qwen2.5-Math-7B-Instruct and discover that the resulting gains are marginal when compared with those of Qwen2.5-Math-7B, indicating that the two Qwen variants exhibit differential sensitivity to RLVR." (Page 7)

(2) Pattern activation. We agree with the reviewer that the performance gain of base model "not simply about format learning", as stated in our previous rebuttal "the improvements observed in their Qwen base model might stem from eliciting certain meta-capabilities or merely learning the format". Recent studies [3, 4] suggest that RL may not fundamentally expand the inherent reasoning abilities of LLMs, but rather elicit the capacity already present in the initial model. RLVR is primarily effective at converting Pass@k capabilities to Pass@1, with the model’s ultimate performance ceiling constrained by its underlying abilities. CoVo achieves a higher Pass@k score than TTRL (Tab. R3 in previous rebuttal) as k increases, indicating that our approach not only improves Pass@1 accuracy, but also more effectively preserves the capabilities of the initial model.

There remains considerable ongoing discussion within the community regarding whether the performance gains achieved by RL methods (both supervised and unsupervised) are attributable to certain meta-capabilities or existing patterns in the initial model. While we acknowledge this is a promising direction for further investigation, it lies outside the primary motivation of our paper and does not affect the validity of our method. Both supervised and unsupervised RL might simply elicit and guide these pre-existing patterns or capabilities; this, however, does not influence the empirical effectiveness demonstrated for our approach, which achieves performance comparable to supervised RL.

(3) Data contamination on Qwen. We agree that data contamination is a pervasive and critical concern in evaluating model performance. However, in our work, data contamination is not directly relevant to the validity of our method. We focus on longitudinal performance improvement that our method can enhance the capabilities of a given initial model itself, rather than making cross-model comparisons. Moreover, we would like to emphasize that our experiments are not limited to Qwen models but validate our method using Llama as well.

We appreciate the opportunity to further clarify our motivation and why CoVo is effective from different perspectives.

If you have further questions, please feel free to reach out. Looking forward to your reevaluation of our work!

References

[1] Spurious Rewards: Rethinking Training Signals in RLVR.

[2] Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination.

[3] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

[4] Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining.

Dear Reviewer QTLK,

We sincerely appreciate the reviewer for the constructive comments. As a gentle reminder, there are fewer than 4 hours remaining in the discussion period. We would appreciate it if you could confirm whether you have any concerns regarding our latest response or any additional points you would like us to address.

Best regards,

Authors of CoVo

We greatly appreciate the reviewer for your insightful comments and positive support.

[W1] The computation of distance matrices and multiple trajectory sampling per prompt may lead to significant training costs.

Thanks for the insightful comments. We list the average cost of time in each stage within a training step in Tab. R1. Since the distance matrix is computed independently and can take advantage of parallelism, the reward calculation indeed accounts for a small proportion of the time required in a training step (nearly 24.6%). In the future, it will be possible to asynchronously compute the distance matrix during sampling to further reduce the time required for distance matrix computation.

Table R1.

[W2] Experiments are restricted to relatively small models, generalization to larger models (e.g., 13B+) or production settings is unclear.

Thanks for the helpful suggestion. In our paper, we provide model sizes of 3B & 7B. As suggested, we provide additional results on Qwen2.5-14B-Instruct to further validate the effectiveness of CoVo when model parameter scaling. The results in Tab. R2 shows meaningful improvements, which demonstrates the effectiveness of our approach.

Table R2.

[Q1] How sensitive is CoVo to the number of sampled reasoning paths per query? Could performance degrade with fewer samples?

Thanks for the valuable feedback. The number of sampled paths per query is a critical hyperparameter to RL techniques, including CoVo.

(1) Reinforcement Learning (RL) optimization algorithms, such as GRPO and RLOO, necessitate sampling multiple responses for the same question. The advantage function is then estimated based on the rewards of these responses. Variations in the number of sampled responses lead to changes in the mean and standard deviation of intra-group rewards, which in turn affect the calculation of the relative advantage for each response. Specifically, if the number of samples is too small, it may result in inaccurate estimation of the advantage, thereby impairing the effectiveness of policy updates. Conversely, with a larger number of samples, intra-group samples exhibit greater diversity. This enhances the precision of the advantage function calculation, enabling the model to adjust its policy more accurately.

(2) Therefore, the number of sampled paths per query is also important to CoVo, a characteristic inherited from its foundation in RL. An insufficient number of samples results in inadequate diversity among the sampled reasoning paths. This poses difficulty for the intrinsic reward to perform meaningful comparisons across distinct answers to estimate the advantage function. In practice, the number of sampled paths per query is typically set to 8, 16, 32, or 64. In our work, we opted for 16 for the balance between performance and computational resources.

[Q2] Would combining CoVo with external reward models further enhance performance, or cause interference?

Thanks for the constructive comments. We have provided the performance results in weakly supervised scenarios, where approximately only 20% of the data are labeled to obtain external rewards. The results in Table R3 demonstrate that our method can enhance performance when combined with external rewards.

Table R3: Performance on Llama3.2-3B-Instruct with the combination of external reward.

After reading the author's rebuttal, I have decided to keep my score unchanged.