We thank the reviewer for her/his constructive comments. We provide inline responses below and answer to questions at the end.

The method’s performance is sensitive to PRM calibration

We agree that we introduce and additional parameter $\beta$, but in general $\beta=0.1$ shows a robust performance in most scenarios. In fact, Phi-4 results are very similar for $\beta=0.1$ which leaves only the Qwen-1.5b + Qwen-PRM as the only case where beta needed to be far away from the nominal $0.1$. This is also an understandable case since the model is rather small and the Qwen-PRM exceptionally good. It should also be pointed out that the dev-adjusted value generalized well to test.

All experiments fix the reference model to be the generator,

We do agree with this is a valid criticism. We would like to raise following points

• the reference policy does play a lesser role in the use of the KL-controlled objective for inference scaling / decoding compared to the standard use for RL, since there is no weight updates and thus less risk of policy drifting towards reward hacking.

• Even if we do not provide empirical results, mathematically, its behavior is well defined. For a reward of zero, it will converge to conventional MBRD from the reference policy (approximated by self-normalized importance sampling, see section 4.4). For a non-zero reward, OP-MBRD reweighs this samples to account for the effect of the reward and this relation is mediated by the parameter $\beta$.

• We did perform initial experiments using Granite as generator and Phi-4 and Phi-4-reasoning as a reference policies. We observed performance above BoN but below MBRD. We felt further exploring this line would be beyond the scope of the paper but we can complete and include this analysis in the final manuscript.

not compare against recently competitive inference-time techniques such as speculative decoding, stepwise verifier-guided reranking, or tree-of-thought search

We believe the main strength of the proposed approach is simplicity. It adds little complexity to BoN/MBRD while being more performant across scenarios, providing a well founded derivation/guarantees and an efficient version. The methods mentioned above can be seen as complementary and can be combined with OP-MBRD. For example speculative decoding allows generating from an expensive generator while using a cheaper one to propose completions. This could be used in MBRD/BoN or OP-MBRD to speed up generation, but it serves a different purpose. Regarding step-wise and tree-of-thoughts, see the answer to questions in the end.

The estimator derivation is difficult to parse without a strong RL background.

Indeed this can be improved. We had added a complete derivation as additional materials in the original submission (we should have indicated this in the manuscript). We can move this to the appendix, add an algorithm block description and more details / appendix references in the main paper body. Feedback of possible improvements to the additional material is also welcome.

Questions

Have you tried using pR ≠ p,

Yes, see comment about about Phi-4/ Phi-4-reasoning.

In cases where PRMs are misaligned (e.g., math PRM applied to code), could the model detect or correct for this dynamically? Would adaptive β tuning help?

We did look at the range of PRM values for the coding tasks, and the PRM assigned clearly lower values on average than for math tasks. This is probably an effect of the PRM working out-of domain. Result do still notably improve, so the PRM is not being totally ineffective. Tuning $\beta$ specifically for code may help but probably tuning the PRM for coding tasks is likely to help the calibration more. A PRM that is well tuned to a generator and a task, like in the Qwen experiments seems to have large positive effects on performance and particularly in the savings attained by using the efficient version of OP-MBRD.

Can you provide any timing comparisons or latency overhead metrics for OP-MBRD vs BoN and MBRD(EM*R)? How does wall-clock cost scale?

The main difference regarding latency is between MBRD (including OP-MBRD, MBRD(EM*R), etc) and BoN methods.

• MBRD requires computing pair-wise similarity between outputs which scales quadratically with $N$. This is not a problem for e.g. math, where the similarity function is a exact match and is very fast. It can get slower for other metrics like Rouge for large $N$. This can be however sped up with caching and pre-computation of operations such as tokenization.

• Also BoN methods and MBRD(EM*R) OP-MBRD versions require evaluation using an RM or a PRM. This is linear on $N$ but can be expensive if the Reward Model is big.

Could OP-MBRD be integrated with multi-step inference strategies (e.g., beam search or tree-of-thought reasoning)? Would the optimal policy still be applicable at intermediate steps?

This is a great point, there are two main possibilities:

• it is indeed possible to use the same principle for intermediate steps. The only difference is that the normal use of MBRD implies an argmax over the $Q$ score whereas in this case its is more useful to do a top-k and discard the lowest scoring components, as it would be done on a beam search algorithm. Regarding ToT, the same applies to state evaluation during decoding.

• further, if any of these methods produce $N>1$ hypotheses at the end of decoding, normal OP-MBRD can be used to for final post-processing. Importance weights from ancestral sampling can be accumulated in the same way as it is done for the PRM in normal OP-MBRD (See section 4.2).

Have you explored the method’s robustness under noisy PRMs? Some form of regularization may be necessary for stable deployment in open-ended domains.

It is true that the dependency on a RM’s ability to generalize across domains is a limitation of both the baselines and the presented method. The experiments cover 3 different PRMs with different performances and, given the results, one could argue that at least the variance in performance of the PRM seems to affect more the baselines like MBRD(EM*R) than OP-MBRD itself.

Thanks for the rebuttal. I will maintain my positive score.

We thank the reviewer for her/his constructive comments. We provide inline responses below and answer to questions at the end.

not particularly familiar with the string of works mentioned in L171 (DPG, GDC++, RSO [Liu et al., 2024], BRAIn). So I am not sure if there is any similar works worthwhile noticing or comparing.

This line of methods are indeed less well known that other RL alternatives such as PPO and DPO. They are mentioned here because their provide the inspiration for the method proposed. It should be however noted that all these approaches concern Reinforcement Learning while the approach presented here is a inference scaling / decoding technique. The overlap is merely leveraging the fact that one can sample from the optimal policy via importance sampling / rejection sampling. The RL approaches use it to derive an approximate gradient update rule for the the KL-controlled reward maximization objective (same objective from which PPO and DPO are derived). The approach presented here uses it to perform Minimum Bayesian Risk Decoding from the optimal policy, thus yielding a well defined way to integrate a generator policy, a similarity metric and reward model for inference scaling.

We thank the reviewer for her/his constructive comments. We provide inline responses below and answer to questions at the end.

Statistical robustness. The result in Fig 1/2 lacks variance quantification (e.g., error bars/confidence intervals), limiting reproducibility assessment.

We agree this can be improved. The main reason to avoid error bars was visibility. To compensate for this we had indicated in the figure footer the maximum standard deviation across samples observed. A second reason is that our main claim is that the method matches either BoN or MBRD(EM*R) across all scenarios, while those methods are only good at a subset of those scenarios. The differences for the relevant comparisons for this claim are rather large. We will explore however some better visualization of data dispersion. We would also like to note that all experiments are done with multiple repetitions (e.g. 256 for N=1 or 4 for N=64).

The proposed OP-MBRD primarily combines BoN and MBRD paradigms

Indeed the idea is rather simple. We would like to make the point that this is a strength, rather than a weakness, since improving upon these well established methods is not trivial and the idea is well founded theoretically.

Questions

Will you add error bars/standard deviations to results (e.g., Fig 1 pass@1 curves) to clarify performance stability?

We will do this. We may have to rethink how to represent these results to guarantee visibility. Maybe a separate graph for the efficient version.

Can OP-MBRD handle larger LLMs (e.g., 90B llama 3.2)? Would sample diversity effects differ at scale?

There is no limitation on the algorithm regarding model size, other than the cost of sampling being larger. As pointed out in the limitations section, a possible hypothesis is that bigger models will benefit more from self-consistency i.e. yield stronger MBRD results compared to BoN. Since OP-MBRD seems to be able to match the best results of MBRD/BoN across different scenarios we can assume it will not be affected or even will benefit in terms of the gap.

We thank the reviewer for her/his constructive comments. We provide inline responses below and answer to questions at the end.

submission would be much stronger if it provided some brief primers / algorithm

We can add a full derivation of the algorithm, including results already present in other papers, to the appendix. We had added such derivation as additional materials in the original submission (we should have indicated this in the manuscript). Feedback of possible improvements to this is welcome. In addition to this, we will add an algorithm block description and more details / appendix references in the main paper body.

simplifying assumption by setting the reference policy [...] as the generator

This is a valid criticism. We would like to raise following points

• the reference policy does play a lesser role in the use of the KL-controlled objective for inference scaling / decoding compared to the standard use for RL, since there is no weight updates and thus less risk of policy drifting towards reward hacking.

• Even if we do not provide empirical results, mathematically, its behavior is well defined. For a reward of zero, it will converge to conventional MBRD from the reference policy (approximated by self-normalized importance sampling, see section 4.4). For a non-zero reward, OP-MBRD reweighs this samples to account for the effect of the reward and this relation is mediated by the parameter $\beta$.

• We did perform initial experiments using Granite as generator and Phi-4 and Phi-4-reasoning as a reference policies. We observed performance above BoN but below MBRD. We felt further exploring this line would be beyond the scope of the paper but we can complete and include this analysis in the final manuscript.

not [...] comparing against policies trained using RLHF which makes the empirical section incomplete

See answer to questions below.

Questions

how is the intractable term calculated in the general case

The usual way in rejection sampling is using a sample estimate i.e. selecting the highest reward from the sample set. This has negligible computational cost and does not change OP-MBRD performance (just shifts the logits). We did test this but left the results out due to space. The only effect is that the savings of the efficient version are worse for low $N$ (compared to the fixed $1.0$ limit that can be used with PRM and any upper bounded reward). We also did additional experiments where setting a scaling factor ($\alpha>1$) as in $\alpha \cdot \max_{y'} R(y', x)$ on dev for the sample estimate increases the rejection rate and allows to recover the savings both for dev and test. This could be useful for non-upper-bounded rewards or pushing for more aggressive saving in efficient mode. We can include these results in the appendix.

have [baseline models] undergone some training with RL for reasoning?

We use models directly download from Huggingface with no further modifications. Following their technical reports all models have undergone some level of RL training. GRPO for Qwen, DPO and PPO for Granite, DPO for Phi-4.