Dear Reviewer YPfy,

Thank you for your positive review and for your support of our work! We are happy to provide detailed answers to your questions below, and we hope they will further increase your confidence in your assessment.

1. On Credit Assignment

Thanks for this great question. You are correct that there are no explicit steps to score since we train a token-level value model, which operates at the same granularity as the autoregressive LM. To see the benefit of selecting intermediate blocks with our value model, we performed the random search ablation presented in Figure 8. The results show that selecting blocks with a high value score leads to significantly improved final performance compared to random selection. This provides strong empirical evidence for a positive correlation between higher intermediate value scores and more promising reasoning blocks.

Moreover, we remark that despite having sparse rewards, our value model estimates the expected return at any partial response, and thus can still provide dense signals about the likelihood of this state leading to high reward. In other words, even when rewards are sparse, there are partial responses and blocks associated with non-zero rewards. We note that our approach is philosophically aligned with Sutton's "bitter lesson," favoring general methods that scale with data and compute over those relying on human-defined structures such as a "step." Indeed, our VGS method using a 1.5B value model, empirically outperforms a 7B PRM baseline trained with step-level feedback (Table 1, VGS w/ DeepSeek-VM-1.5B has 45.7% accuracy vs. VGS w/ Qwen2.5-Math-PRM-7B has 40.2% accuracy). Ultimately, we hypothesize it may be useful to learn an appropriate notion of "step" beyond the existing heuristics (i.e. separating by newline), but we leave that as promising future work.

2. On Controlling for Data Contamination

Thanks for this question. To ensure there is no data contamination, we will focus our evaluations on only AIME-25 and HMMT-25, which are math competitions that happened after the release dates of DeepSeek and OpenR1 (our dataset is a subset of OpenR1). This certainly controls for data contamination because we only evaluate on problems after the release dates of both the underlying model and the dataset.

For example, we will update the main comparison in Table 1 to the following:

Test-time scaling DeepSeek-1.5B ($N=256$)

Test-time scaling larger models with our DeepSeek-VM-1.5B:

Pass@1 Baselines

For our camera ready, we will also update our main plots to evaluate only on AIME-25 and HMMT-25, and we will add a discussion regarding how this ensures there is no data contamination.

3. On the Use of Other Base Models

We chose to focus on the DeepSeek series because they were the only open-weights reasoning models -- LLMs trained with RL to perform CoT reasoning -- at the time of submission. This focus also provided the benefit of a controlled study across different model scales (1.5B, 7B, and 14B) from the same architectural family, allowing us to investigate properties like scaling up generator sizes (Section 3.3) in a controlled manner.

4. On Performance in Cross-Domain Tasks

We believe that VGS would be broadly helpful for tasks that require planning (in the RL sense), where a sequence of correct decisions is needed to reach a final goal. This includes domains like math reasoning, as explored in our paper, but could also extend to applications like creative writing, where one might guide a story's progression toward a specific ending. The primary challenge across these domains is defining a reasonable outcome-based verifier to generate the reward signal for training the value model. This is precisely why we focused our study on competition math, where the verifier is well-defined and can be reliably implemented with tools like sympy.

Thank you again for your valuable questions and positive assessment. Please let us know if you have any additional questions.

Dear Reviewer TuTX,

Thank you for the positive review and your thoughtful feedback! We are happy to clarify the points you raised and appreciate your suggestions for improving the paper.

1. On Beam Width: This is a great question. Our ablations in Figure 7 show that a beam width of 2 is consistently optimal for our method across many inference budgets. We also note that prior works with non-reasoning models [6, 32, 22] also found a low beam width of four to be optimal. We hypothesize the success of lower beam width may be due to two factors. First, it is likely easier for the value model to perform pairwise comparisons than to rank a larger set of candidate blocks. Second, given the high pass@k of DeepSeek models, it is possible that generating more diverse responses is particularly helpful for arriving the right response.

2. On Greedy Search (Width 1): We did not explicitly run a width=1 experiment given the trend that we saw in our block size ablation (Figure 6). We see a clear trend that a 4k block size was optimal across many inference budgets for our value model.

3. On the Generalizability of VGS: We believe that VGS would be broadly helpful for tasks that require planning (in the RL sense), where a sequence of correct decisions is needed to reach a final goal. This includes domains like math reasoning, as explored in our paper, but could also extend to applications such as creative writing, where one might guide a story's progression toward a specific ending. The primary challenge across these domains is defining a reasonable outcome-based verifier to generate the reward signal for training the value model. This is precisely why we focused our study on competition math, where the verifier is well-defined and can be reliably implemented with tools like sympy.

Thank you again for the insightful questions and your positive evaluation. We hope our answers are helpful, and we will incorporate these points into the final version of the paper.

Dear Reviewer uxSh,

Thank you for your exceptionally thorough and insightful review. We are grateful for your detailed positive feedback on our work's significance, rigor, and clarity. We really appreciate the opportunity to respond to your questions and hopefully solidify your view of VGS as a top-tier contribution.

Responses to Questions:

1. On the Limits of Weak-to-Strong Generalization: Your hypothesis is correct. The shrinking performance gap is likely an out-of-distribution (OOD) issue, as the 14B generator produces reasoning paths that are qualitatively different from the 1.5B rollouts our value model was trained on.

   While retraining the value model on rollouts from a stronger generator is the ideal experiment, it is unfortunately too computationally expensive for us to perform during the rebuttal period. We also want to clarify that demonstrating weak-to-strong generalization, while an interesting result, is not the main intended use case of VGS. When applying VGS to guide frontier models, we envision training a value function of a similar size to the generator policy itself. Since the value model is only queried once per block, the computational overhead of this approach is negligible, as we show in Appendix H.

2. On the Critical Role of Block Size (4096 tokens): We agree this is a fascinating result, and we believe the optimality of a large block size is driven by a combination of two of your hypotheses: (a) Signal-to-Noise: The value model requires sufficient context to make an accurate prediction. A short block may not contain enough information to distinguish a promising path from a dead end, leading to noisy and unreliable guidance. (b) Task-Specific Coherence: Competition math problems often require long, uninterrupted lines of reasoning. We found that interrupting the generator too frequently with smaller blocks is disruptive to this coherent thought process. We don't think the amortized cost hypothesis (c) is a reason because the performance also degrades for blocks larger than 4096, which would not be explained by efficiency alone. We will add a qualitative example to the appendix, as you suggested, to illustrate this trade-off and expand this discussion in Section 4.1.

3. On Adapting VGS to Nuanced Domains: This is a critical question. The notion of a value function is very general and can be applied to any type of reward signal. For domains with more nuanced rewards, we would adapt the value model's training as follows:

   • For continuous rewards (e.g., the fraction of unit tests passed), we would suggest moving beyond simple regression and using a distributional RL (Bellemare et al. 2017) approach to model the entire distribution of possible rewards. A distributional approach to value-based RL has robustly shown improved downstream decision making and is supported by a large body of prior works (Imani et al. 2024, Farebrother et al. 2024, Wang et al. 2025).
   • For multi-faceted discrete rewards, we would use a more complex set of outcome classes, as you suggest. In general, we believe that VGS would be broadly helpful for tasks that require planning (in the RL sense), where a sequence of correct decisions is needed to reach a final goal. This includes domains like math reasoning, as explored in our paper, but could also extend to applications like creative writing, where one might guide a story's progression toward a specific ending. The primary challenge across these domains is defining a reasonable outcome-based verifier to generate the reward signal for training the value model. This is precisely why we focused our study on competition math, where the verifier is well-defined and can be reliably implemented with tools like sympy.

4. On the Power of Final Aggregation via WMV: This is a key insight. Our explanation for why Weighted Majority Vote (WMV) is more effective than Best-of-N (BoN) is twofold:

   • First, WMV is more scalable against the value model being "hacked" at large inference budgets. With many samples, BoN is more likely to select an outlier—a flawed path that receives an erroneously high score. WMV, by relying on consensus, is more robust to such outliers.
   • Second, WMV provides an implicit filter for unfinished responses. Incomplete generations will not produce the same final answer and thus will not be part of a majority cluster. BoN does not have this filter, and our value model sometimes assigns an unduly high score to an incomplete trace. As a concrete action, we will add results to the final version showing the performance of BoN after manually filtering for unfinished responses to further support this analysis.

Thank you again for your detailed and expert feedback. We believe that incorporating these analyses and discussions will significantly strengthen our paper. We are grateful for your review.

Citations

[1] Bellemare et al. 2017: "A distributional perspective on reinforcement learning", ICML 2017.

[2] Imani et al. 2024: "Investigating the histogram loss in regression", arXiv 2024

[3] Farebrother et al. 2024: "Stop regressing: Training value functions via classification for scalable deep rl", ICML 2024

[4] Wang et al. 2025: "The central role of the loss function in reinforcement learning". Statistical Science, 2025

Dear Reviewer 4pBh,

Thank you for your positive and insightful review! We are happy to clarify the points you raised and appreciate your suggestions for improving the paper.

On the Scope of Evaluation:

We chose to focus on the DeepSeek series because they were the only open-weights reasoning models -- LLMs trained with RL to perform CoT reasoning -- at the time of submission. This focus also provided the benefit of a controlled study across different model scales (1.5B, 7B, and 14B) from the same architectural family, allowing us to investigate properties like scaling up generator sizes (Section 3.3) in a controlled manner.

On the Minor Comment:

Thanks for this suggestion. We will update the table to highlight the best and second-best performing methods on each individual benchmark in the camera-ready version.

Responses to Questions:

1. Regarding the Random Search Ablation (Section 4.2): Your understanding is correct. Our intent is to cleanly isolate the effect of using our value model for selecting intermediate blocks versus having no selection rule at all. By removing the value signal entirely, we can directly measure its contribution to the search process. We agree that a likelihood-based selection is a strong baseline for a verifier-free search, and we will run this comparison and add a discussion to the final version.

2. Comparison to (Rashid et al., 2025): Thank you for this reference; the work is certainly related and merits a citation and discussion. There are two primary issues / challenges with applying reward guidance at the token level that make it less efficient and scalable:

   • Computational Cost: Token-wise guidance requires a classifier forward pass for every generated token [1,2,3], which is substantially more expensive than our block-wise method that only queries the value model once per block (e.g., every 4096 tokens).
   • Compounding Errors: An imperfect reward or value model, when queried too frequently, can introduce cascading errors that derail the generation process. Indeed, prior work by Mudgal et al. 2024 compared block-wise versus token-wise guidance and found that block-wise approaches performed better, which informed our decision to focus on these methods.

   We will expand on this discussion and add a citation to Rashid et al. to our related work section.

3. Comparison to Tree-of-Thought (ToT) Approaches: Thanks for the great question. Our Value-Guided Search can be viewed as an instantiation of the broader ToT framework, where our value model serves as the verifier to score different reasoning paths. Specifically, in the ToT framework, our policy model $\pi$ corresponds to the "thought generator", and our value model corresponds to the "state evaluator", which scores the generated blocks to determine which reasoning paths are most promising to pursue. Thus, VGS can be viewed as a specific, highly effective instantiation of this general framework. We show this particular approach is an efficient and powerful method for improving long-context reasoning, but we agree that exploring more complex ToT search strategies with a value model evaluator can lead to even more efficient reasoning, which we believe is a promising direction for future work.

Thank you again for your supportive review and constructive feedback. We will incorporate these clarifications and additions into our camera-ready version and believe your suggestions will significantly strengthen the paper.

Citations:

[1] Rashid, Ahmad, et al. "A critical look at tokenwise reward-guided text generation." arXiv preprint arXiv:2406.07780 (2024).

[2] Mudgal, Sidharth, et al. "Controlled decoding from language models." arXiv preprint arXiv:2310.17022 (2023).

[3] Zhou, Jin Peng, et al. "$Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training." arXiv preprint arXiv:2502.20548 (2025).

Dear Reviewer oEdM,

Thank you for your constructive review. We really appreciate the opportunity to respond to your questions and concerns.

Regarding Value Model Training Cost and Generalization:

(1) On Training Cost & Scalability: We apologize for the lack of clarity -- by "scalable," we refer specifically to our data collection pipeline. Our method avoids the need for expensive, step-by-step human or LLM-as-a-Judge annotations required by prior PRM approaches. This simplification is what enabled us to collect a large-scale dataset of 2.5 million reasoning traces (each having up to 16k tokens). Training on a dataset of this magnitude indeed requires significant compute. To ensure this effort benefits the entire community, we are open-sourcing the full dataset, model checkpoints, and codebase, allowing other researchers to bypass this one-time cost.

(2) On Generalization: Thank you for this point. While it is true that our 1.5B value model shows signs of OOD degradation when guiding DeepSeek-14B, we want to highlight that our value model still strongly improves MV when guiding DeepSeek-7B. We view this as a successful demonstration of weak-to-strong generalization: our value model can guide a ~5x larger model to strongly improve the MV baseline (Table 1, VGS guiding DeepSeek-7B has 56.4% accuracy vs. MV with DeepSeek-7B has 50.5% accuracy, so VGS shows a 6% improvement).

Moreover, in practice, value guidance is only a small fraction of the total inference compute as we showed in Appendix H, since the value model is only queried at every block of tokens. Thus, we believe training a large value model (of the same size as the generator or even larger) is a reasonable investment and we believe VGS is a promising way to for scaling test-time compute of reasoning models.

Regarding the Advantage over Step-Level Supervision:

We agree it is difficult to provide a direct "apples-to-apples" cost comparison. The key advantage of our approach is that it mitigates the need to define a "step," a notion that is difficult to formalize for complex reasoning traces (Guo et al. 2025). For example, if one were to use a simple heuristic like newlines, a single reasoning trace from DeepSeek can easily exceed 100 "steps." Using an LLM-as-a-judge for our 2.5M trace dataset would have required over 250 million such judgments, a staggering and impractical cost. In contrast, our outcome-based method requires only one cheap verification per reasoning trace.

We note that our approach is philosophically aligned with Sutton's "bitter lesson," favoring general methods that scale with data and compute over those relying on human-defined structures such as a "step." Indeed, our VGS method using a 1.5B value model, empirically outperforms a 7B PRM baseline trained with step-level feedback (Table 1, VGS w/ DeepSeek-VM-1.5B has 45.7% accuracy vs. VGS w/ Qwen2.5-Math-PRM-7B has 40.2% accuracy). Ultimately, we hypothesize it may be useful to learn an appropriate notion of "step" beyond the newline heuristic, and we leave that as promising future work.

We hope these clarifications address your concerns and highlight the significance of our contributions. Thank you again for your time and valuable feedback.

I have read the rebuttal and will update the score accordingly.