Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs

We sincerely thank the reviewer for their detailed feedback and for recognizing the novelty of our idea and the value of our analysis. We are grateful for your constructive criticism and the opportunity to clarify our work. We will address all weaknesses and questions, and we have a clear plan to revise our paper accordingly.

Q1: On the significance of Self-PRM's improvement

We acknowledge that the improvements in Table 4 might seem modest at first glance. However, we would like to highlight three key points:

1. Consistency: Self-PRM consistently matches or outperforms both majority voting and the external PRM across nearly all benchmarks and sampling sizes. In contrast, the external BoN w/ PRM is sometimes worse than simple majority voting, indicating that its guidance can be misaligned.

2. Meaningful Gains on Hard Problems: On challenging benchmarks like AIME, even a small percentage increase represents solving additional difficult problems. For example, with QwQ-32B on AIME24 (@k=32), Self-PRM achieves 90.0% accuracy, a notable improvement over both baselines at 86.7%.

3. Computational Efficiency: Crucially, Self-PRM achieves these gains with virtually zero additional computational overhead, as it uses internal reward signals calculated during the initial generation pass . An external PRM, however, can nearly double the inference cost. When factoring in efficiency, the benefit of Self-PRM becomes highly significant. We will add a discussion on this cost-benefit analysis to the paper.

Q2: On the generality of findings (mathematical reasoning)

This is a valid point. We chose this domain because mathematical problems offer clear, objective correctness criteria and structured, step-by-step solutions, making it an ideal "laboratory" for studying the co-evolution of problem-solving and process supervision capabilities.

Currently, there are no process reward evaluation benchmarks similar to ProcessBench in other domains, and constructing such benchmarks is challenging. Therefore, we have not extended the analyses in Table 2 and Table 3 to other domains. Regarding the results in Table 4, we have extended our investigation to the GPQA-diamond dataset, which contains a total of 198 questions specifically designed for the most challenging, PhD-level science questions. The specific results are presented below and we found that the conclusions remain consistent.

Regarding generalization, we hypothesize that the core finding will indeed transfer to other complex reasoning domains like coding, planning, or legal analysis. The underlying principle—that an agent learning to solve a complex task (the policy) simultaneously develops an internal model for evaluating the quality of its actions (the value function)—is a fundamental concept in reinforcement learning. As long as a domain requires multi-step, logical reasoning, we expect a similar synergistic development. We will explicitly add this discussion to our limitations section and highlight cross-domain validation as a critical direction for future work.

Q3: On the lack of a limitations section

We sincerely apologize for this oversight. We will add a dedicated "Limitations" section in the revised manuscript. This section will address the domain generality issue mentioned above and other points to provide a more balanced perspective on our work.

Q4: Self-REF on weaker models

In the Self-REF section, we have supplemented the results of qwen2.5-7B-instruct and qwen2.5-1.5B-instruct. Based on the results, we can observe that qwen2.5-7B-instruct with Self-REF shows improvements on GSM8K and MATH, while its F1 scores decrease on OlympiadBench and OmniMATH. In contrast, qwen2.5-1.5B-instruct with Self-REF exhibits a decline in performance across all test sets. These results are consistent with our expectations: the effectiveness of Self-REF does depend on the base model's reasoning ability. A weaker instruction-tuned model would generate lower-quality reasoning traces, which would act as noisy or incorrect supervisory signals. We will add this analysis to the final paper.

Q5: ProcessBench evaluation in Figure 1

For the evaluation shown in Figure 1, we prompt the model to act as a "Generative PRM". For each instance in ProcessBench (which contains a problem and a candidate solution), the model is asked to analyze the solution and output a judgment on its correctness, identifying the first erroneous step if one exists. We then compare the model's textual judgment to the ground-truth label to calculate the ProcessBench F1 score.

Q6: Self-PRM reranking method

We adopt the same evaluation method as used in ProcessBench to judge the correctness of each step. The self-PRM evaluates the correctness of all steps in each response, and then among the responses where all steps are correct, we select the final answer based on the consistency of the answers.

Q7: Reasoning step division in BoN w/ PRM

Following the protocol outlined in the ProcessBench paper, we utilize Qwen2.5-72B-Instruct for step segmentation. We will provide the specific prompt in the revised version.

Q8: Problem indices in Table 5: Problem indices in Table 5

The indices in Table 5 were specifically chosen from problems that the respective models (QwQ-32B and DeepSeek-R1) initially failed to solve correctly (i.e., failed at pass@1). This selection allows us to analyze the behavior of Self-PRM on what are considered 'hard problems' for each model. For these hard problems, we sampled 64 solutions to see how many the Self-PRM would judge as correct ($S_{PRM}$ ), and out of those, how many were truly correct ($S_{PRM}$ ). The "-" symbol indicates that the problem was not a hard problem for that particular model (i.e., the model solved it correctly on its first try). For instance, in AIME24 index 62, QwQ-32B succeeded initially, so it was excluded from this failure analysis and marked with "-". In contrast, DeepSeek-R1 failed the same problem, so its Self-PRM performance is shown. We will clarify this selection methodology in the paper to make the purpose of this fine-grained analysis clear. Thank you again for your thorough and helpful review. We are confident that by addressing these points, we can significantly improve the paper and hope you will consider re-evaluating its rating.

Thank you for your response. However, PRMs are known to be difficult to generalize across policy models (i.e., models that generate solutions) and tasks [1, 2]. Using Qwen2.5-Math-PRM-72B as the discriminative PRM for comparison is not a fair choice, since this PRM was not trained on the long-CoT responses produced by reasoning models such as QwQ-32B and DeepSeek-R1. As a result, it is out-of-distribution (OOD) for evaluating the responses of these reasoning models. Although this PRM has 72B parameters, its generalization ability remains limited.

Given this OOD issue, Self-PRM still shows only marginal improvement over the discriminative PRM, while incurring a significantly higher inference cost. Regarding inference cost, I suggest the authors compute it using the total inference FLOPs equation provided in Section 2.3 of [3], since model size alone is not an adequate measure. The paragraph-by-paragraph critique process of Self-PRM introduces substantially higher computational cost than that of a discriminative PRM.

For these reasons, I maintain my current score.

References

[1] ProcessBench: Identifying Process Errors in Mathematical Reasoning.

[2] The Lessons of Developing Process Reward Models in Mathematical Reasoning.

[3] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling.

We sincerely thank the reviewer for their careful reading and insightful feedback. We are very grateful that you appreciated the comprehensive experimental setup and our transparent approach to analyzing the results and limitations of our proposed method. Your main criticism regarding the potential misalignment between our framing and findings is very well-taken, and we would like to clarify our position.

Q1: Regarding the Framing: "PRMs are Unnecessary'' vs. "Current PRMs are Insufficient''

This is an excellent and sharp observation. We agree that our results do not dismiss the value of process-level guidance itself. In fact, our findings strongly support its value. Our central argument, perhaps not framed as clearly as it could be, is aimed at the conventional approach of building separate, external, and explicitly annotated PRMs. Our title, "Is PRM Necessary?", is meant to question this specific paradigm, which faces major challenges like prohibitive annotation costs and vulnerability to reward hacking.

Our work demonstrates that:

1. Pure RL can implicitly develop this process-level judgment capability without explicit supervision.

2. The resulting internal capability can be harnessed via Self-PRM to achieve performance gains, even outperforming existing external PRMs on strong models.

Therefore, we are not arguing that process supervision is without value. Rather, we are proposing that it is possible to achieve effective process supervision without a separate, externally trained PRM. Self-PRM is presented as a more direct, scalable, and model-consistent way to leverage this valuable guidance. We will revise the paper's introduction and conclusion to refine our framing, shifting the emphasis from "PRMs are unnecessary" to "A separate, externally-annotated PRM may not be necessary," making this more nuanced argument clearer.

Q2: Regarding Process Judgment Gains Preceding Accuracy Gains

Thank you for highlighting this key result from Figure 1. You are absolutely correct in your interpretation: the fact that process judgment improves early in training does suggest that process-level supervision could be a very effective and dense signal to guide RL. We see this not as a contradiction, but as a crucial piece of the puzzle that our work helps illuminate. Our contribution is showing that this powerful early signal does not necessarily need to come from an external PRM. Instead, it is a capability that emerges organically. This opens up exciting future possibilities, such as using Self-PRM during RL training to create a self-improvement loop, a point we will emphasize more in our future work section.

Q3: Does Self-PRM suffer from low precision on difficult problems more than no-PRM?

In our experiments, "no-PRM" refers to baselines like majority voting. Majority voting does not produce a confidence score or a precision metric in the same way; it simply selects the most frequent answer. The key finding is that while Self-PRM's precision can be low on difficult problems (i.e., it is overconfident and labels many flawed solutions as valid ), its ranking is still highly effective. As shown in Table 4, using Self-PRM to select the single best candidate (BoN w/ Self-PRM) consistently leads to higher final accuracy than both majority voting and direct sampling. This indicates that although Self-PRM's absolute judgment is imperfect, its relative judgment is superior to the baselines. We will add a paragraph to Section 4.4 to make this important distinction clear.

Thank you once again for your constructive criticism. We believe your feedback will help us significantly sharpen the paper's message and better contextualize our contributions.

We are extremely grateful to the reviewer for their thoughtful and encouraging feedback, as well as the high scores across the board. We are delighted that you found our work to offer interesting insights, appreciated the comprehensive empirical evaluation and the statistical validation via Chi-square tests, and found the experiments to be well-designed. Your positive assessment is a strong validation of our research direction. We will address the weakness you've pointed out and answer your excellent questions.

Q1: On the Limitation to the Math Reasoning Domain

This is a very fair point, and we agree that the current work's empirical validation is focused on the mathematical domain. We chose this domain because mathematical problems offer clear, objective correctness criteria and structured, step-by-step solutions, making it an ideal "laboratory" for studying the co-evolution of problem-solving and process supervision capabilities.

Currently, there are no process reward evaluation benchmarks similar to ProcessBench in other domains, and constructing such benchmarks is challenging. Therefore, we have not extended the analyses in Table 2 and Table 3 to other domains. Regarding the results in Table 4, we have extended our investigation to the GPQA-diamond dataset (The specific results are presented below.) and found that the conclusions remain consistent.

Regarding generalization, we hypothesize that the core finding will indeed transfer to other complex reasoning domains like coding, planning, or legal analysis. The underlying principle—that an agent learning to solve a complex task (the policy) simultaneously develops an internal model for evaluating the quality of its actions (the value function)—is a fundamental concept in reinforcement learning. As long as a domain requires multi-step, logical reasoning, we expect a similar synergistic development. We will explicitly add this discussion to our limitations section and highlight cross-domain validation as a critical direction for future work.

Q2: On Using Self-PRM to Improve the Policy during RL

This is not a weakness but a fantastic suggestion for future work, and we thank you for it. You have accurately identified a natural and exciting extension of our findings. The current work uses Self-PRM in a post-hoc reranking setting. Using the model’s own internal reward signal to provide dense, step-level feedback during RL training could create a powerful self-improvement loop, potentially accelerating learning and pushing performance even further. This is an excellent idea that directly builds on our work, and we will add it to our future work section as a promising research avenue.

Q3: Sensitivity to RL algorithms

Due to time constraints, we directly tested the performance of the open-source model Open-Reasoner-Zero-7B, which is based on the qwen2.5-7B-base model and trained using the PPO algorithm. On the Math, GSM8K, OlympiadBench, and OmniMath datasets, the problem-solving accuracy (%) of Open-Reasoner-Zero-7B are 77.9, 89.8, 42.0, and 37.4 respectively. On ProcessBench, its F1 scores (%) are 41.3, 58.8, 28.0, and 21.1 respectively. Based on these results, we can draw the same conclusion: after RL training, both the problem-solving ability and PRM capability of the model have improved. In the revised version, we will retrain a model based on the PPO algorithm, plot the corresponding curves in Figure 1, and provide a more detailed analysis.

Q4: Computational overhead of Self-PRM

Majority voting outperforms both Self-PRM and external PRMs in terms of efficiency. Majority voting merely derives the final answer based on the consistency of results, whereas both Self-PRM and external PRMs require invoking the model to judge the process scores before selecting the answer. In this paper, Self-PRM and external PRMs are utilized in the same manner; the only distinction lies in that external PRMs are derived from independently trained models, while self-PRM does not require additional training. Therefore, the efficiency of Self-PRM and external PRMs depends solely on the model parameters. The external PRMs employed in this paper is a 72B model, while QwQ-Self-PRM, being a 32B model, exhibits superior efficiency. In contrast, R1-Self-PRM, with a size of 671B, demonstrates significantly slower inference speed.

Q5: Self-PRM sensitivity to thresholds

We adopt the LLM-as-judge approach, where the model is directly prompted to judge the correctness of each step. Since this is a generative model, there is no involvement of a confidence threshold in this process.

We sincerely thank the reviewer for their valuable time and constructive feedback. We are pleased that you found the paper well-written, easy to follow, and the analysis in the "Problem-Solving V.S. Judgmen'' section insightful. We would like to address the identified weaknesses and questions to further clarify and strengthen our work.

Q1: On the conclusion regarding Self-REF (Table 3)

We appreciate the reviewer's close reading of this section. There seems to be a slight misunderstanding we wish to clarify. The reviewer states that "Qwen2.5-32B-Instruct has undergone an RL training process'', but according to the source technical reports, this model is a purely instruction-tuned model without reinforcement learning. Qwen2.5-32B-Instruct is a non-thinking model. Our paper explicitly categorizes it this way to test the effect of Self-REF in a setting where process reasoning has not been reinforced. The substantial F1 score improvement of +18.8 for this model provides strong evidence for our claim.

Conversely, R1-Distill-Qwen-32B, while not directly trained with RL, was distilled from DeepSeek-R1 and thus inherits its process judgment behaviors. In fact, R1-Distill-Qwen-32B is a thinking model. Therefore, it represents an intermediate case. The fact that its performance gain from Self-REF is minimal (+2.6) further supports our conclusion that as a model's internal reasoning becomes more aligned through RL (or distillation from an RL model), the benefit of weakly supervised signals from its own generated traces diminishes.

Q2: On the generalizability of the AIME benchmark results (Sec 4.3)

This is a fair point. We acknowledge that the AIME benchmarks contain a limited number of problems. Our intention was to use these highly challenging competition-level problems to test the limits of current PRMs in a difficult, out-of-distribution setting.

While the AIME results alone might not be broadly generalizable, they are presented as complementary evidence to our larger-scale findings on Processbench. Across Processbench, our core finding holds: strong RL-trained models like DeepSeek-R1 and QwQ-32B consistently outperform models explicitly trained with PRM labels. The fact that an external PRM also fails to provide a significant boost over majority voting on the difficult AIME problems further reinforces our main thesis. In our revision, we will soften the language in this section to more accurately reflect the limited sample size of the AIME benchmark while maintaining the core insight.

We have supplemented the analysis on the GPQA-diamond dataset, which contains a total of 198 questions specifically designed for the most challenging, PhD-level science questions. The specific results are as follows:

Based on the analysis of results on GPQA-diamond dataset, we find that the conclusion remains consistent with our previous findings: External PRMs provide little to no benefit for RL-based reasoning models.

Q3: Testing a smaller RL-trained model (Table 3)

Following your suggestion, we directly used an open-source, smaller RL-trained model: Qwen-2.5-7B-SimpleRL-Zoo. The specific results are as follows:

Based on the results, we can observe that the conclusions drawn from the 7B model are consistent with our previous findings: RL-trained models show minimal or negative gains, suggesting limited benefit when process reasoning is already reinforced.

Q4: More statistical results for Self-PRM (Sec 4.4)

We agree completely. A visualization would be much more powerful. We thank the reviewer for this constructive idea. In the revised version, we will add a plot showing the curve of the solution pass ratio versus Self-PRM precision. This will provide a clearer and more intuitive illustration to support our claim that Self-PRM's precision is low on difficult problems.

Q5: On the Limitation of the Mathematical Domain

We acknowledge this limitation. We chose the mathematical domain because it offers objective and verifiable ground truth for both final answers and intermediate reasoning steps, making it an ideal environment to systematically study the relationship between problem-solving and process supervision.

Currently, there are no process reward evaluation benchmarks similar to ProcessBench in other domains, and constructing such benchmarks is challenging. Therefore, we have not extended the analyses in Table 2 and Table 3 to other domains. Regarding the results in Table 4, we have extended our investigation to the GPQA-diamond dataset (The results are shown in Q2.) and found that the conclusions remain consistent.

We will explicitly state in the conclusion that our findings are currently confined to mathematical reasoning and that future work should investigate whether these principles generalize to other complex reasoning domains.

Thank you again for your insightful feedback. We believe that incorporating these changes will significantly improve the quality and impact of our paper.

We sincerely thank the reviewer for their valuable time and constructive feedback. We would like to address the identified weaknesses and questions to further clarify and strengthen our work.

Q1: Regarding the lack of depth on why RL scaling enhances PRM capability

• Our primary goal was to first systematically establish and validate the existence of the strong positive correlation between a model's problem-solving accuracy and its process judgment F1 score, a phenomenon we demonstrate persists across multiple datasets as RL training progresses. However, we agree that a deeper investigation into the underlying mechanism is a valuable addition.
• In our revision, we will add a new section with targeted analyses to explore this why ; these experiments will be conducted after the rebuttal period due to time constraints. Our hypothesis is that as RL training improves the policy, the model's internal value function becomes a more reliable estimator of the correctness of reasoning steps. To test this, we will analyze the evolution of the model's internal reward signals and their correlation with ground-truth step-wise correctness at different checkpoints during training. This will provide a more direct explanation for how RL implicitly fosters robust PRM capabilities without explicit supervision.

Q2: Regarding the overlooked scenario of using an external PRM for RL training

• We thank the reviewer for highlighting this important scenario. We acknowledge that our study did not test the efficacy of an external PRM as a source of dense reward signals during the RL training phase. Our experimental scope was focused on a different, but equally relevant, question: whether existing state-of-the-art PRMs can further enhance already strong reasoning models through post-hoc reranking. Our results show that for models like QwQ-32B and DeepSeek-R1, external PRMs offer no significant benefit over simpler methods like majority voting, whereas an internally derived Self-PRM does.

• While using a PRM for dense supervision is a valid approach, our work challenges the prevailing assumption that is considered necessary. The paper highlights that conventional PRMs face significant hurdles, including prohibitive annotation costs and vulnerability to reward hacking. Our core finding is that pure RL scaling presents a more direct and efficient path, as it not only boosts problem-solving but also inherently develops the desired process supervision abilities. We believe this finding, which shows that a complex capability can be an emergent property of a simpler training objective, is a significant contribution. We will clarify this distinction in the paper and position the dense supervision angle as a valuable direction for future research. In our subsequent research, we are attempting to use a self-PRM to guide the RL training process, where both the policy model and the self-PRM are improved simultaneously. More related results will be disclosed in future studies.