Free Process Rewards without Process Labels

1. The core concepts and theoretical framework have already been established in [1]. Even with the proposed extensions, the theoretical contribution remains marginal.

A: We note that the derivation of our work is different from [1] and provides a more general conclusion. [1] is tailored to DPO only and adopts a different reward representation (with a Z(X) baseline), with the implicit reward derived from entropy-regularized RL framework to ensure the optimality of the trained policy model. We aim for a reward representation that enables tractable Q value; we do not target and never claim an optimal policy in the entropy-regularized RL framework. Rather, our reward representation is defined and constructed from scratch, namely any representation is acceptable as long as it gives a tractable way to estimate Q value and makes Eq. (2) in line 137 hold. Compared to [1], this paper provides a fresh and more general perspective of implicit rewards, which we believe holds significant value.

In this way, our paper provides a novel and more general theoretical framework which then leads to empirical benefits, e.g. the legibility of a CE loss, which offers an alternative when pairwise data is harder to collect than response-level labels, and scenarios that are more data-scarce. One may also explore other effective objectives beyond DPO and CE within our theoretical framework. Therefore, we believe our generalization to unpaired losses holds great theoretical and practical merits compared to previous works.

[1] From r to Q∗: Your Language Model is Secretly a Q-Function. Rafailov et al. 2024.

2. While the authors suggest using cross-entropy (CE) loss to address scenarios with unpaired data, obtaining response-level labels may still pose challenges. Furthermore, the performance of CE loss is generally inferior to that of DPO.

A: We agree that in some cases response-level labels are still difficult to collect; our CE objective provides an alternative when such labels are available, and it can utilize the pairwise labels when they are easier to obtain.

Also, CE loss has its own advantages over DPO. As shown in Figure 5 and Figure 6, Implicit PRM with CE loss is more data efficient, while showing better performance when integrated with majority vote, presenting as an appealing alternative in practice as in many scenarios pair-wise data is hard to collect. Also, CE loss only requires one example for forwarding and backwarding which reduces memory overhead in RL training. Therefore, the generalization to unpaired losses remains valuable compared to pair-wise DPO in more data-constrained settings.

[2] Process Reinforcement through Implicit Rewards. Cui et al. 2025.

3. Evaluated only on mathematical reasoning; generalizability to code generation or open-ended generation tasks is untested.

A: Though we do not include other tasks in this paper due to our limited capacity and the limited space, after ICML submission deadline, there are recent works showing that Implicit PRM is helpful in best-of-N sampling on agent tasks [3], and adopting Implicit PRM for online RL, with brings substantial gains on coding [2] and function calling tasks [4].

[3] AgentRM: Enhancing Agent Generalization with Reward Modeling. Xia et al. 2025.

[4] Learning to Generate Structured Output with Schema Reinforcement Learning. Lu et al. 2025.

[W1] Limited Datasets

A: Though we do not include other tasks in this paper due to our limited capacity and the limited space, after ICML submission deadline, there are recent works showing that Implicit PRM is helpful in best-of-N sampling on agent tasks [1], and adopting Implicit PRM for online RL, with brings substantial gains on coding [2] and function calling tasks [3].

[1] AgentRM: Enhancing Agent Generalization with Reward Modeling. Xia et al. 2025.

[2] Process Reinforcement through Implicit Rewards. Cui et al. 2025.

[3] Learning to Generate Structured Output with Schema Reinforcement Learning. Lu et al. 2025.

Following reviewer’s suggestion, we also test on other math datasets and results are as follows. The policy model is Llama-3-70B-Instruct, and the baseline is Math-Shepherd in our implementations.

[W2] Is PRM better than ORM?

A: All ORMs in Table 1 are off-the-shelf from HuggingFace and are trained with different data and therefore not comparable to ours. They indeed consume much more data. For example, Eurus-RM-7B uses 287k pairs of data, SkyworkRM uses 465K, ArmoRM uses 1560K, while we only use 263K. Also, ORMs in Table 1 perform well on weak-to-strong settings which increases the average performance, but on Mistral-7B (20.4% vs 28.8% for best-of-64) and Llama-3.1-8B-Instruct (51.8% vs 57.0%), there are significant performance gaps to Implicit PRM.

Moreover, Implicit PRM only needs to forward the response once, with the only difference being log probs are divided into steps and summed up respectively to find the minimum step reward. We acknowledge that Implicit PRM brings extra inference overhead due to the reference model and we have explored this issue in Appendix. As the cost mainly comes from the additional reference model, in Appendix B.3.1, we ablated it in both training and inference and presented results in Table 5. We found that in practice they can be removed at least in this paper’s setup, avoiding the extra compute while maintaining the performance. We also provided explanations in Appendix B.3.1 on why it may be removed. In such cases, Implicit PRM brings negligible overhead during inference compared to an ORM.

[W3] Fairness of Baseline Comparisons

A: Please refer to the above response on how we can reduce the inference overhead of the reference model. Besides, following your suggestion, we present the compute-equivalent comparison between our best-of-4 and their best-of-8/12 as below:

Given the PRMs provide step-level supervision, did the authors conduct any experiments to show their implicit PRM is more effective than other PRMs or ORMs at refinement [1] or RL training [2]?

A: We supplemented comparison to ORMs with the same data as our Implicit PRM and still observed the superiority of Implicit PRM.

For RL training, a recent work [3] released after ICML submission deadline explored comprehensively how Implicit PRM can improve RL especially on sample efficiency compared to using verifiable outcome rewards only, namely the golden ORM. Results are shown as follows:

[3] Process Reinforcement through Implicit Rewards. Cui et al. 2025.

From the table, the gain delta from the proposed PRM is not that large, and on Mistra-7B is not helping, Besides, the pass@1 is not clear if it helps, would be great to show a curve here.

A: This might be a misinterpretation of Table 1. In fact, our approach achieves very strong performance.

We should compare the best-of-N performance to pass@1(greedy decoding w/o using reward models) to evaluate the effectiveness of the reward models. The pass@1, i.e. best-of-1, of each generation model is highlighted in the header and the caption. The first column for each model denotes the best-of-4 accuracy and is already much higher than pass@1 across the board, confirming the strong performance of our approach. Rows in the “open-source reward models” block are trained with different and much larger and more engineered data, and therefore are not comparable to ours. Those under “our implementations” are reimplementations of previous works controlling for confounding factors such as the base model and data, for fair comparisons.

Contrary to the reviewer’s observation that “on Mistral-7B it is not helping,” the gains on Mistral-7B are actually the largest: the pass@1 is 9.6% while best-of-64 increased to 28.8% for our DPO model, a 19.2 absolute gain. Improvements are still observed even when we use the 8B-sized PRM to assist 70B-sized generation models, namely in the weak-to-strong setup, evidenced by best-of-64 of 72.2% compared to pass@1 of 63.2%. Importantly, our reported performance improvement on MATH is challenging to achieve: contextualizing it within recent impactful literature [1,2], a <3% improvement on best-of-64 and 2% improvement compared to baselines is usually considered substantial as a significant improvement. We think our Implicit PRMs achieve strong performance in our experiments.

We also note that why we did not surpass the off-the-shelf baseline RLHFlow-8B-Mistral-Data can be explained by the unfair comparison, due to their advantages from training models using on-policy rollouts. Our evidences include: (1) RLHFlow-8B-Mistral-Data uses on-policy rollouts from Mistral-7B while ours use off-policy rollouts from Llama-3.1-8B-Instruct; (2) The same approach with rollouts from DeepSeek models underperforms ours; (3) Our approach outperforms RLHFlow-8B-Mistral-Data on Llama-3.1-8B-Instruct by a large margin.

Following the reviewer's suggestion, we plot the curve of pass@N and bes-of-N in this figures anonymously. from the figure we can see that our method consistently outperforms baselines when increasing the number of candidate responses (N).

[1] Math-shepherd: Verify and reinforce llms step-by-step without human annotations. Wang et al. 2023.

[2] Improve mathematical reasoning in language models by automated process supervision. Luo et al. 2024.

Whether this generalize to many other domain.

A: Though we do not include other tasks in this paper due to our limited capacity and the limited space, after ICML submission deadline, there are recent works showing that Implicit PRM is helpfull in best-of-N sampling on agent tasks [3], and adopting Implicit PRM for online RL, with brings substantial gains on coding [4] and function calling tasks [5].

[3] AgentRM: Enhancing Agent Generalization with Reward Modeling. Xia et al. 2025.

[4] Process Reinforcement through Implicit Rewards. Cui et al. 2025.

[5] Learning to Generate Structured Output with Schema Reinforcement Learning. Lu et al. 2025.

Would like to know more about the pass@k from 1 to 16, how the curve looks like, particularly for pass@1, how much it helps. Besides how many runs for this metric, would be great to show mean/std here.

A: Thanks for the suggestion. The curve of pass@N and bes-of-N can be found here. For each experiment we only run for one time since our early experiments show that there is little variance across runs and thus this has been a standard practice mainly due to the cost of the experiments [1,2,6].

[6] Autopsv: Automated process-supervised verifier. Lu et al. 2024.

The main concern is the empirical gain, if it's really showing the consistent gain. Would be great if this part can be explained more (particularly, why Mistral-7B no gain, and pass@4 for Llama/pass@64 for Llama-70B

A: Please see above responses on empirical gains. Particularly, our method has achieved substantial gains (pass@1: 9.6% -> best-of-64: 28.8%) on Mistral-7B rather than no gains, and we only underperform the baseline trained on on-policy Mistral-7B rollouts. Regarding pass@4 for Llama/pass@64 for Llama-70B (actually they are best-of-4 and best-of-64), though we did not achieve the highest accuracy, our performance is very close to the best-performing baseline, only with a difference of 0.4% and 0.8% respectively. We kindly ask the reviewer reevaluate our submission after these clarifications on the empirical gains.

We thank the reviewer for the constructive comments. Here are our responses.

The true test of PRM would be to run beam search or to use it as dense rewards for online RL.

A: We choose best-of-N as our setup because it is a standard practice in recent literature and presents as a valuable approach for inference-time scaling [1,2,3]. Testing PRMs for online RL is of course valuable, but can introduce additional confounding factors and much more overhead. This paper aims to isolate the quality of the reward model, so best-of-N sampling provides relevant and convincing evidence for our claim.

That being said, we notice that a recent work [4] after the ICML ddl closely resembles the reviewer's suggestion, which applies our implicit PRM in online RL and achieved strong performance and sample efficiency across various benchmarks. We think this proves our approach's potential in online RL.

We are now running the beam search experiments that the reviewer suggested. They are slow and we will provide update as soon as possible.

[1] Let’s Verify Step by Step. Lightman et al. 2023.

[2] Math-shepherd: Verify and reinforce llms step-by-step without human annotations. Wang et al. 2023.

[3] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. Snell et al. 2024.

[4] Process Reinforcement through Implicit Rewards. Cui et al. 2025.

The gains are small on poorer models like Mistral-7B... It seems that the PRM is not very useful on hard questions.

A: This might be a misinterpretation of Table 1. In fact, our approach achieves very strong performance on these questions.

We note that the pass@1 performance of each generation model is highlighted in the header and the caption, and the first column for each model denotes the best-of-4 accuracy and is already much higher than pass@1. We should compare the best-of-N performance to pass@1 to see the absolute gain of each model.

Contrary to the reviewer’s observation, the gains on Mistral-7B are actually the largest, with the pass@1 being 9.6% while best-of-64 increased to 28.8% for our DPO model. Compared to our implemented Math-Shepherd and AutoPSV, we achieved significant average improvements of nearly 3% and 5%.

As for the reviewer’s second point: a 5% average improvement on the MATH benchmark is challenging to achieve: contextualizing it within recent impactful literature [2, 5], a <3% improvement on best-of-64 and 2% improvement compared to baselines is usually considered substantial. Therefore, the average performance gains of our implicit PRM demonstrate its effectiveness on these challenging questions.

We also add comparisons to ORMs below; both are trained with the same data. The trend is consistent with that in the paper and we observe strong performance of our Implicit PRM.

[5] Improve mathematical reasoning in language models by automated process supervision. Luo et al. 2024.

How to optimally scale instructions and responses

A: According to our experiments in section 5.1, scaling up instructions from the same tasks to downstream tests and scaling up responses are both is helpful. We did not test data balance for the DPO objective, but we did observe benefits of that for CE loss.

Does the trained PRM extrapolate

A: Yes, Implicit PRM is able to transfer across model families and tests of the same task. For model transfer, all Implicit PRMs in Table 1 are trained from Llama-3.1-8B-Inst with its on-policy rollouts, and existing results have shown their effectiveness on improving Mistral-7B and Llama-3.1-70B-Inst models. For task transfer, we add an experiment to train our model solely on GSM8K and test on MATH as follows, and surprisingly, results show that the overall performance remains comparable to that trained on all instructions, indicating the superior task transferability of Implicit PRM.

If Implicit PRM works on other domains

A: Though we do not include other tasks in this paper due to our limited capacity and the limited space, after ICML ddl, there are recent works showing that Implicit PRM is helpful in best-of-N sampling on agent tasks [6] and online RL, with substantial gains on coding [4] and function calling tasks [7].

[6] AgentRM: Enhancing Agent Generalization with Reward Modeling. Xia et al. 2025.

[7] Learning to Generate Structured Output with Schema Reinforcement Learning. Lu et al. 2025.

We thank the reviewer for the valuable feedback and are glad that you find the method is suitable for the problem and the experiments and analysis are sound. Here are our responses.

the proof appears to have some issues. Furthermore, the derivation of cross-entropy loss is missing the normalization constant Z(x).

A: Thanks for the careful reading, but we believe there are misunderstandings.

As the reviewer uses different notations and does not mention the line of proof, we would like to kindly ask the reviewer for clarification on the specific steps in the proof, so that we can better address the concern.

Regarding the baseline $Z(X)$ in the cross entropy objective: Please note that our reward representation does not need a baseline $Z(X)$, which is different from the implicit reward defined in [2,3]. The implicit reward in [2,3] is derived from the entropy-regularized RL framework, and a $Z(X)$ must be included to ensure the optimality of the trained policy model. However, this is not the case in our paper. Instead, we aim for a reward representation that enables tractable Q value; we do not target and never claim an optimal policy in the entropy-regularized RL framework. Rather, our reward representation is defined and constructed from scratch, namely any representation is acceptable as long as it gives a tractable way to estimate Q value and makes Eq. (2) in line 137 hold. Therefore, we do not need to follow the restrictions as [2,3] and our reward representation does not necessarily relate to theirs.

Moreover, if a baseline term $Z(X)$ were added, one can prove that the following equation would not hold anymore, with the proof done by simply substituting the new reward representation (with a $Z(X)$) to the right-hand of the equation:

$$\sum_{i=1}^{t} \beta \log \frac{\pi_\phi(y_i \mid \mathbf{y}_{<i})}{\pi_\text{ref}(y_i \mid \mathbf{y}_{<i})} = \beta \log \mathbb{E}_{\pi_\text{ref}(\mathbf{y} \mid \mathbf{y}_{\leq t})} \left[ e^{\frac{1}{\beta} r_\phi(\mathbf{y})} \right]$$

I.e., proposition 3.1 will not consistently hold for all ORM objectives if we follow the reward representation as [2,3], and we won’t be able to find a tractable Q value.

We plan to add this discussion in the next version.

How do you manage the normalization constant in the cross-entropy (CE) update?

A: Please refer to the response on $Z(X)$ above. As stated in Proposition 3.1, we do not need a baseline in our reward representation.

The paper concludes that the implicit DPO reward outperforms the other objectives studied. Implicit DPO was originally proposed in [1]. Given this context, it is unclear what novelty this paper offers, as the best algorithm was introduced in previous work.

A: First, [1] directly adopts the theory from [3] which only applies to DPO. However, as discussed above, our work also provides a more general proposition with solid theoretical contribution, applying to any ORM objectives including CE loss as long as they use our reward representation, compared to both [1] and [3] that are tailored to DPO. Compared to these previous works, this paper provides a fresh and more general perspective of implicit rewards, which we believe holds significant value. Second, as shown in Figure 5 and Figure 6, Implicit PRM with CE loss is more data efficient than its DPO counterpart while showing better performance when integrated with majority vote, presenting as an appealing alternative in practice as in many scenarios pair-wise data is hard to collect. Also, the CE variant requires only one example for forwarding and backwarding while the DPO variant has to consider a pair of examples at the same time. As a result, the CE variant reduces memory overhead in RL training, as observed by a recent work that directly adopts our method (published after the ICML submission deadline) [3]. Therefore, we believe our generalization to unpaired losses holds great theoretical and practical merits compared to previous works.

In Figure 4, why does increasing the number of scaling instructions hurt performance?

A: We’d like to clarify: for most cases, increasing the number of instructions improves model performance. The only exception is using all instructions to train 8B-sized PRMs and test on Llama-3.1-70B-Inst. As the increased number of instructions come with Llama-3.1-8B-Instruct generated responses, we conjecture that the performance drop can be attributed to PRMs overfitting to responses from the small model and being unable to generalize to larger models. This discussion will be added in the revision.

[1] Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling Qui et al. 2024

[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Rafailov et al. 2023.

[3] From r to Q∗: Your Language Model is Secretly a Q-Function. Rafailov et al. 2024.