Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning

Thank you for your constructive comment. The following are our responses to each individual comment.

Response to Weakness about the missing comparison:

There is no comparison with Group Relative Policy Optimization (GRPO) in Deepseek.

We appreciate the comment and would like to clarify that we have indeed conducted both theoretical and experimental comparisons with GRPO.

In the final paragraph of Section 2.3, we discuss the similarities between GRPO and our proposed method. Specifically, GRPO and our method both employ a similar reward shaping mechanism, but the key distinction lies in our approach, which applies this operation exclusively to negative samples, whereas GRPO applies it to all samples.

Furthermore, Table 2 provides a comparative evaluation of these two strategies. The second row of Table 2 ("+reward shaping") represents the GRPO approach, while the last row showcases our proposed solution, demonstrating the superior performance of our method.

Thank you for your constructive comment. The following are our responses to each individual comment.

Response to Weakness3 about the dependency on the token-level reward model:

Dependency on the token-level reward model: A potential limitation of the approach lies in its reliance on a token-level reward model trained solely from binary outcome labels. While this design avoids the need for step-level supervision and is computationally efficient, the paper does not provide a thorough analysis of the model’s stability or accuracy, especially under distribution shifts or varying reasoning patterns. Since the token-level weights directly influence the policy optimization, any misestimation could affect learning dynamics, and further investigation into the robustness and generalizability of this component would strengthen the overall claim.

We acknowledge that our method relies on the token-level reward model training on binary outcome labels, but we believe it is a low-cost yet reliable solution.

Regarding the potential issue of distribution shift, it needs to be clarified that in our scheme, the token-level reward model is trained synchronously with the policy model using the same training data. This ensures that the data it uses for training and the data it needs to estimate within a single iteration are co-distributed, thereby mitigating the risk of misestimation due to distribution shift.

About the reliability of this reward model, we provide a visualization of its estimated values for both positive and negative samples in Appendix D2. For correct responses, the overall reward scores are high, especially at the end, although there are a few lower sections in the middle. For incorrect responses, the distribution of rewards is reversed, and the closer to the end, the lower the rewards. We believe the distinct distribution of scores for positive and negative samples offers evidence for the model's stability and accuracy.

Response to Question1 about the unclear presentation:

While the paper consistently refers to sampling from a dataset D, in Section 2.1, the distribution is denoted by ρ₀, which may cause confusion for readers. Clarifying this notation would improve readability.

Thanks for your suggestion. We will modify these confusing variable names in the next version to improve readability.

Response to Question2 about the insufficient disscussion:

It would be useful to discuss computational cost of BoN sampling during training, especially for weaker base models or larger datasets.

Thanks for your suggestion. The computational cost of BoN sampling primarily stems from the varying number of sampling attempts required to obtain correct samples, which is influenced by differences in problem difficulty and model capability. To analyze the computational cost during training, we sampled the training set and divided the problems into two categories—easy and hard—based on the initial policy model's pass rate on the training set. We then evaluated the Pass@N metric of the model at different training iterations on these two subsets. Due to resource constraints, we sampled 8 responses for each query.

As shown in the table below, the overall scores for the easy problem sets are significantly higher than those for the hard sets, indicating that greater effort is required to sample correct answers for difficult problems. As training progressed, the model's capability steadily improves, with the pass@1 scores increasing across both easy and hard sets. This suggests a higher probability of sampling correct answers, implying that using a stronger model can theoretically reduce computational overhead. Interestingly, the pass@8 score does not show a noticeable improvement during the training process, which is consistent with observations of Yue et al.[2].

We will add a corresponding discussion and evaluation results in the next version.

[2] Yue Y, Chen Z, Lu R, et al. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?[J]. arXiv preprint arXiv:2504.13837, 2025.

Response to Question3 about the typos:

There is a minor typographical issue in Section 3.1, where the sentence refers to “Section ??”, likely due to an unresolved reference placeholder.

Thanks for your reminder. We will fix the typos in the next version.

Thank you for your constructive comment. The following are our responses to each individual comment.

Response to Weakness1 about the alternative framing:

Alternative framing via DPO: A particularly interesting aspect of OREAL is the asymmetric learning strategy: behavior cloning on positive trajectories and reward shaping–based policy optimization on negative ones. This separation is both novel and intuitive. However, it also prompts a natural question: could a more unified preference-based approach, such as Direct Preference Optimization (DPO), be applied here by treating correct vs. incorrect outputs as preference pairs? If so, would it yield similar performance, possibly with simpler implementation or fewer hyperparameters? A comparative experiment with a DPO-style setup using the same BoN-derived data would provide valuable insights into the relative strengths of each formulation.

Yes, after distinguishing between positive and negative samples, DPO can be used for training. However, DPO has a potential flaw: it theoretically leads to a decrease in the generation probability of positive examples, even while increasing their relative generation probability compared to negative examples [1]. In contrast, our approach performs Behavior Cloning for positive samples, avoids this particular issue, which is precisely why we chose not to employ DPO.

[1] Pal A, Karkhanis D, Dooley S, et al. Smaug: Fixing failure modes of preference optimisation with dpo-positive[J]. arXiv preprint arXiv:2402.13228, 2024.

Response to Weakness2 about the limitation on initial policy strength:

Limitation on initial policy strength: The method seems to rely on having a reasonably good initial policy that can already generate some correct completions under BoN sampling. This assumption is implicitly reflected in the experimental setup, where initial policies are derived from either distilled or rejection-finetuned models. However, in real-world scenarios or low-resource domains, one may not have access to such strong initial models. It remains unclear how OREAL would perform when initialized from a significantly weaker model that produces mostly incorrect outputs. An empirical analysis of this case would help validate the framework's robustness and clarify its applicability in broader contexts beyond well-pretrained policies.

In Appendix C, we discuss the performance of our method across different initial policy models. As shown in Table A1, the performance of the model after RL is strongly correlated with the capabilities of the initial policy model itself. The stronger the initial policy model, the higher the performance that RL can deliver, indicating the importance of policy initialization.

Nevertheless, it is noteworthy that our method consistently makes substantial performance improvements, regardless of the initial policy model's strength, which demonstrates the robustness of our approach.

We acknowledge that for particularly challenging data (e.g., AIME-level problems), if the model fails to sample correct answers, the training effectiveness may indeed be compromised. However, this is an inevitable problem that all on-policy reinforcement learning methods have to face.

Thank you for your constructive comment. The following are our responses to each individual comment.

Response to Weakness about the insufficient explanation:

Insufficient explanation of algorithm generality and cross-domain applications

We acknowledge that this paper has limitations in discussing the generalization and cross-domain applicability of our algorithm. As indicated by the title, this paper specifically focuses on mathematical reasoning tasks. However, we believe that our method is broadly applicable to any task that provides a clear outcome reward signal (e.g., instruction following, programming problem solving, puzzle solving, etc.), because we only assume the reward is a binary signal, without making any assumptions about the specific task type.

To verify the effectiveness of our method in other domains, we conducted training and evaluation on the instruction-following task using the IFEval dataset [1]. As shown in the table below, the model's ability to follow instructions gradually improved during training, indicating that our method possesses generalization capability and cross-domain application potential.

We will add a corresponding discussion and evaluation results in the next version.

[1] Zhou J, Lu T, Mishra S, et al. Instruction-following evaluation for large language models[J]. arXiv preprint arXiv:2311.07911, 2023.

I appreciate the follow-up experiment. My concern has been resolved.

Thank you for your constructive comment. The following are our responses to each individual comment.

Response to Weakness1 about the algorithmic novelty:

The method is a combination of several components with various tricks being applied, making it hard to distinguish the algorithmic novelty.

Our method is not merely a combination of several components with various tricks being applied, it offers a theoretically grounded reinforcement learning algorithm for mathematical reasoning tasks. Specifically, our main contribution lies in demonstrating, through rigorous theoretical analysis and extensive experimentation, the following:

• For Positive Samples: In binary feedback environments, optimal policies can be effectively learned through behavior cloning on best-of-N sampled trajectories.

• For Negative Samples: Reward reshaping is essential to maintain consistency in the policy optimization objective.

• For Long Sequences: Different segments of a long sequence contribute disparately to the final outcome, necessitating a more fine-grained credit assignment function. This function can be effectively learned from the outcome reward itself.

Response to Weakness2 about the unclear presentation:

Some descriptions of the method are confusing. For instance, what does importance sampling refer to in the method, e.g., which expectation is IS estimating? What is reward shaping referring to? Is it different from traditional reward shaping that uses value potentials?

For the unclear presentation about "Importance Sampling". We agree that our use of the term importance sampling may be misleading. In our method, we do not perform importance sampling in the classical sense that policy updates are constrained by the difference between two distributions, such as using log-likelihood ratios between the new and old policies. Instead, we use a token-level reward model which trained on historical policy data to assign scores reflecting each token’s estimated contribution to the final reward. These scores perform fine-grained credit assignments under long reasoning sequences and are used as heuristic importance weights to prioritize the learning of high-contributing tokens and downweight or ignore low-contributing tokens. This is akin to importance-based selection or re-weighting, rather than the traditional meaning of importance sampling.

To improve clarity, we will revise the terminology to 'importance-based token selection' or 'fine-grained token re-weighting' rather than importance sampling in the final version.

For the unclear presentation about "Reward Shaping". As shown in Equation 2, since our positive and negative samples are derived from the results of BoN sampling, their reward distribution is inconsistent with that of uniformly sampling from the original distribution. Generally, this will overestimate the expected reward of the current policy on the corresponding input (high-reward responses are assigned a higher probability), which may affect downstream reasoning performance (as the optimization objective is not consistent with the pass@1 performance). Therefore, we introduce a coefficient independent of the model parameters to shape the negative sample rewards, theoretically ensuring that our optimization objective function exactly remains the original expected reward.

Regarding "traditional reward shaping that uses value potentials", which is typically associated with advantage estimation in traditional RL algorithms, we further discuss it in Section 2.4 and explain why we use the estimated advantage function for loss weighting instead of directly using it to calculate the optimization objective.

Response to Weakness3 about the misleading title:

The name Outcome Reward RL is somewhat misleading since the algorithm uses reward models to obtain dense process rewards, even though the reward model is trained using the outcome reward.

We name our work "Outcome Reward RL" because our training objective is to maximize a binary outcome reward. In this paper, we observe that for long-cot mathematical reasoning tasks, different parts of the sequence contribute differently to the final outcome, while relying solely on binary outcome rewards leads to the problem of reward sparsity. To address this, we introduce a token-level reward model to perform token-wise credit assignments during training. It is important to clarify that this token-level reward model is only used in the computation of importance sampling and does not serve as our final optimization objective, whose role is to assist in more effectively maximizing the overall outcome reward. Therefore, we believe that our method represents a reinforcement learning approach grounded in outcome-based rewards.

Response to Question1 about the details of Table2:

For the REINFORCE baseline in Table 2, is it based on outcome-reward or process-reward?

It is based on outcome-reward.