Dear Reviewer fQ6h,

We really appreciate your constructive and insightful comments.

---

[Weakness 1. Reason why Tables 4 and 5 are included. An analysis of training effort would have been better.]

Thank you for bringing this to our attention. First of all, as elucidated in Sections 5.1 and 5.2, our original intention behind the inclusion of Table 4 in the original manuscript is to support the idea that tree-search-based inference methods can be more effective, or as effective as, the Best-of-N search while being much more efficient than Best-of-N search in terms of FLOPs and the execution time, especially when tree search strategies can be paired with an effective verifier like TVM. Along this line, to investigate whether the performance of step-level beam search with TVM improves with an increase in the number of $K$ and $b$, we have included Table 5 in the original manuscript.

Reflecting on the reviewer’s insightful suggestion, we also agree that including an analysis of training efforts along with the analysis of inference efforts would improve the manuscript.

Following the reviewer’s suggestion, using 8×NVIDIA A100-80GB GPUs, we estimated FLOPs and measured the execution time of sampling $N_{tr}$ reasoning paths to train TVMs on GSM8K and MATH for Mistral 7B MetaMath.

<Table A. FLOPs and execution time of sampling $N_{tr}$ reasoning paths to train TVMs on the GSM8K ($N_{tr} = 100$) and MATH ($N_{tr} = 25$) benchmarks for Mistral 7B MetaMath>

We appreciate your valuable feedback. We have included Table A in Section 5.3 and have moved Table 5 in the original manuscript to Appendix E in the revised version.

---

[Weakness 2-1. Providing standard deviations in empirical evaluations would allow for a better assessment.]

Thank you for the helpful suggestion. We also agree that providing standard deviations is important for a better empirical evaluation. As the reviewer suggested, we first calculated the mean accuracy and standard deviation of TVM using step-level beam search and REBASE on GSM8K. These results were obtained from three random trials.

<Table B. Mean accuracy and standard deviation of TVM on the GSM8K benchmark. Three random trials are conducted.>

We appreciate your valuable comment and have integrated the experimental results from Table B into Table 1 of the revised manuscript. In the final version, we will also include the mean accuracy and standard deviation of TVM on the MATH benchmark.

---

[Weakness 2-2. It seems that the TVM approach can be applied to more general tasks.]

Theoretically, as the reviewer noted, the TVM approach is applicable to more general tasks. However, we first chose to focus on mathematical problem-solving tasks in this paper because, for math word problems, no automated tools are available to verify the exact correctness of candidate solutions when the ground truth answer is unavailable. In contrast, as highlighted in [1], tasks like code generation and neural theorem proving can benefit from automated tools, such as unit tests and proof assistants (e.g., Lean4), that can automatically determine the correctness of candidate solutions. Given this, we initially applied the TVM approach to math tasks and plan to extend it to more general tasks in future work.

[1] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, arXiv:2407

---

Once again, we sincerely appreciate your time and efforts in reviewing our paper. If you have any remaining issues or concerns, please do not hesitate to bring them to our attention.

Dear Reviewer E5Q5,

We really appreciate your constructive and helpful comments.

---

[Weakness 1. Reason why the token-level supervision in TVM is direct.]

The reviewer mentioned that TVM uses token-level direct supervision (line 113). However, in line 113, we stated that we propose TVM to equip a verifier with a more direct ability to evaluate partial solutions. We did not claim that TVM uses token-level direct supervision or that the token-level supervision in TVM is direct (e.g. human-annotated).

As the reviewer noted, ORMs, PRMs without human annotations, and TVMs all rely on outcome signals (i.e., 1 for a correct final answer and 0 otherwise) to construct supervision labels. Despite this shared dependency on supervision signals for outcomes, PRMs without human annotations outperform ORMs in Best-of-N search [1, 2] as they utilize step-wise distinct supervision compared to ORMs. Likewise, since TVMs leverage explicit token-level supervision with distinct correctness probability scores while outcome supervision in ORMs uses homogeneous labels determined by the correctness of a whole reasoning path, TVMs can more effectively assess whether a partial solution is progressing toward the correct answer during tree search at inference time. In this sense, TVMs do not suffer from the disadvantage of ORMs described in line 267, and we stated in line 113 that we propose TVMs to equip a verifier with a more direct ability to evaluate partial solutions.

[1] Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations, ACL 2024

[2] Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision, EMNLP 2024 Findings

---

[Weakness 2. The supervision of TVMs (Eq. 5) seems similar to the reward for ORMs (Eq. 3). Consequently, the reward for a token $t_{n,k}$ in ORMs will be close to the success rate of an intermediate reasoning path $\{t_{n,1}, \cdots, t_{n,k}\}$.]

While Eq. 3 in ORMs represents only the cumulative reward, Proposition 4.1 asserts that Eq. 5 in TVMs is equivalent to the expected cumulative reward. This key distinction highlights that our new token-level supervision scheme is practically different from the reward for ORMs. As seen in Table 1 of the paper, the TVM’s score for a token $t_{n,k}$ is closer to the actual success rate of an intermediate reasoning path ${t_{n,1}, \cdots, t_{n,k}}$, while the ORM’s reward for a token $t_{n,k}$ is less close to the actual success rate.

---

[Weakness 3. The explanation in line 306 assumes that PRMs do not use human annotations, but the false-negative problem would occur for PRMs without human annotations.]

Thank you for bringing this to our attention. Since recent PRM studies do not rely on human annotations [1, 2, 3, 4] due to the high cost of human labeling, our paper also focuses primarily on PRMs without human annotations, as noted in line 292. Consequently, the term 'PRMs' in line 306 actually refers to PRMs without human annotations. We appreciate your valuable comment and have updated the text to clarify this by replacing 'PRMs' with 'PRMs without human annotations' in lines 306-309. Moreover, to prevent further confusion, we have added the following clarification in line 310 of the revised manuscript: ``Hereafter, we refer to PRMs without human annotations simply as PRMs to keep the expression concise.’’

[1] Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations, ACL 2024

[2] Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision, EMNLP 2024 Findings

[3] AlphaMath Almost Zero: Process Supervision without Process, NeurIPS 2024

[4] Improve Mathematical Reasoning in Language Models by Automated Process Supervision, arXiv:2406

---

[Question 1. Clarification on Figure 1.]

Thank you for bringing this to our attention. First, since outcome supervision in ORMs labels all tokens in a reasoning path uniformly as either 0 or 1, ORMs can mispredict on challenging test problems, as shown in Figure 1. ORMs may reversely assign high scores close to 1 (e.g., 0.914 or 0.906) to incorrect intermediate steps and low scores close to 0 (e.g., 0.175 or 0.160) to correct ones. In contrast, PRMs without human annotations, which are trained with per-step correctness labels, avoid such reversed predictions. However, due to the fact that PRMs without human annotations are trained to determine an entire intermediate step as incorrect even if only the final few tokens are erroneous, they tend to under-value promising intermediate steps, assigning scores (e.g., 0.657 or 0.648) that are comparable to those of incorrect steps (e.g., 0.710 or 0.661). Consequently, both ORMs and PRMs perform suboptimally as verifiers for tree search strategies at inference time, as illustrated in Figure 1 and Sections 1 and 3.

---

Dear Reviewer 6Dfx,

We greatly appreciate your constructive and helpful comments.

---

[Weakness 1. Added complexity of token-level supervision to the training and fine-tuning process.]

To the best of our understanding, the reviewer’s concern is focused on the added computational complexity of token-level supervision during the training and fine-tuning of TVMs. Following [1], TVM is composed of an LLM backbone and an additional prediction head to predict token-wise probability scores. During the training phase, when the input sequence is fed forward, the prediction head generates predictions for each token, and the token-level supervision loss is computed. To evaluate the computational overhead introduced by this prediction head, we present a comparison of the parameter counts between the model backbone and the prediction head in Table A.

<Table A. Comparison of the parameter counts between the model backbone and the prediction head for Mistral 7B and Llama 3 8B>

We observe that the parameter count of the added linear prediction head is negligible compared to the verifier backbone. Thus, the added complexity is kept relatively small.

[1] Training Verifiers to Solve Math Word Problems, arXiv:2110

---

[Weakness 2. For difficult problems, the effectiveness of TVMs would be reduced to that of PRMs.]

If an LLM is too small to solve difficult problems correctly even once among a large number of samples, nearly all samples would be incorrect, rendering the effectiveness of TVMs comparable to that of PRMs. However, even moderately sized LLMs (such as 7B-scale models) with some capability to solve problems that are difficult for humans make the sampling process less demanding, thus ensuring that most samples would not be incorrect. Notably, in practice, for Mistral 7B MetaMath and Llama 3 8B MetaMath, TVMs significantly outperform PRMs on the MATH benchmark—one of the most challenging benchmarks.

---

[Question 1. Choice of $N_{tr}$.]

As mentioned in Appendix C, we set $N_{tr}=100$ for GSM8K, following [1]. For MATH, we set $N_{tr}=25$ based on [2], where the number of solutions per training problem for MATH is set to one-fourth of that for GSM8K. Experimental results in Section 5 demonstrate that these $N_{tr}$ values are effective for training TVMs.

[1] Training Verifiers to Solve Math Word Problems, arXiv:2110

[2] Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision, EMNLP 2024 Findings

---

[Question 2. Do $N_{tr}$ samples often share some prefix tokens? If they do not share any tokens, it seems that the effectiveness of TVMs would reduce to that of PRMs.]

As the reviewer astutely pointed out, if reasoning paths were sampled with a tiny $N_{tr}$, they might not share any tokens, leading the effectiveness of TVMs to reduce to that of PRMs. However, with a large enough $N_{tr}$ (e.g., $N_{tr} = 100$ for GSM8K as noted in the response to Question 1), we can obtain several reasoning paths that share prefix tokens. For instance, as exhibited in Figure 4, three reasoning paths share Terry took 8 bushels, Jerry took 3,, which consists of 14 tokens in terms of token count. In addition, there are 17 reasoning paths that share Terry took 8, which is composed of 5 tokens.

---

[Question 3. Interpretation of the right side of Figure 3 (b).]

The histogram on the right side of Figure 3 (b) illustrates the verifier score distribution for the intermediate steps of correct sampled solutions. Similar to the histogram on the left side of Figure 3 (b), it can be observed that PRM tends to under-value the scores of steps in correct solutions compared to TVM. Consequently, when employing PRMs with tree search strategies such as beam search, promising steps may be under-valued and subsequently pruned, resulting in lower performance.

---

Once again, we sincerely appreciate your time and efforts in reviewing our paper. If you have any remaining issues or concerns, please do not hesitate to bring them to our attention.

Dear Reviewer S2DY,

We greatly appreciate your constructive and insightful comments.

---

[Weakness 1-1. Discussion about the token-supervision signal for the latter part of reasoning paths.]

As the reviewer astutely pointed out, if reasoning paths were sampled with a small $N_{tr}$, the likelihood of reasoning paths overlapping after the first couple of dozen (e.g., 15–20) tokens could be low. In such a case, the token-supervision signal for the latter part of each reasoning path would be labeled uniformly as either 1 or 0, depending on whether the final answer is correct.

However, with a large enough $N_{tr}$ (e.g., $N_{tr} = 100$ for GSM8K as in [1]), we can obtain reasoning paths that overlap significantly beyond the first 15-20 tokens, up to the later steps of reasoning. As demonstrated in Figure 2 of the paper, reasoning paths can overlap up to the middle of the fourth step - which consists of the first 93 tokens in terms of token count. Below, we provide the complete version of the two overlapping reasoning paths that were illustrated in Figure 2. Bold text indicates the overlapped part between two reasoning paths in Figure 2, which is 93 tokens in total. Additionally, two complete reasoning paths for MATH are also provided below. The overlapping section between these two reasoning paths, totaling 129 tokens, is highlighted in bold.

[1] Training Verifiers to Solve Math Word Problems, arXiv:2110

Reasoning path 1 on GSM8K in Figure 2 (Correct)

Terry took 8 bushels * 14 ears of corn per bushel = 112 ears of corn.

Jerry took 3 bushels * 14 ears of corn per bushel = 42 ears of corn.

Linda took 12 bushels * 14 ears of corn per bushel = 168 ears of corn.

Stacy took 21 ears of corn.

Altogether, Bob's friends took 112 + 42 + 168 + 21 = 343 ears of corn.

Thus, Bob is left with 50 bushels * 14 ears of corn per bushel - 343 ears of corn = 357 ears of corn.

Reasoning path 2 on GSM8K in Figure 2 (Wrong)

Terry took 8 bushels * 14 ears of corn per bushel = 112 ears of corn.

Jerry took 3 bushels * 14 ears of corn per bushel = 42 ears of corn.

Linda took 12 bushels * 14 ears of corn per bushel = 168 ears of corn.

Stacy took 21 ears of corn + 168 ears of corn = 189 ears of corn.

Bob had 50 bushels * 14 ears of corn per bushel = 700 ears of corn to start with.

So, Bob has 700 ears of corn – 112 ears of corn – 42 ears of corn – 168 ears of corn – 189 ears of corn = 189 ears of corn remaining.

Reasoning path 1 on MATH (Correct)

$\mathbf{(x+1)\^2+2(x+1)(3-x)+(3-x)^2}$

$\mathbf{= (x^2+2x+1)+2(3-x)(x+1)+(9-6x+x^2)}$

$\mathbf{= x^2+2x+1+2(3x+3-x^2-x)+(9-6x+x^2)}$

$\mathbf{= x^2+2x+1+6x+6-2x^2-2x+9-6x+x^2}$

$\mathbf{=}$ $(x^2-2x^2+x^2)+(2x-2x+6x-6x)+(1+6+9)$

$= 0+0+16$

$= \boxed{16}$.

Reasoning path 2 on MATH (Wrong)

$\mathbf{(x+1)^2+2(x+1)(3-x)+(3-x)^2}$

$\mathbf{= (x^2+2x+1)+2(3-x)(x+1)+(9-6x+x^2)}$

$\mathbf{= x^2+2x+1+2(3x+3-x^2-x)+(9-6x+x^2)}$

$\mathbf{= x^2+2x+1+6x+6-2x^2-2x+9-6x+x^2}$

$\mathbf{=}$ $x^2-2x^2+x^2+2x-2x-6x+6+9-6$

$= \boxed{14}$

We appreciate your valuable feedback and have revised our manuscript to include this point and the above full reasoning paths in Appendix D.

---

[Weakness 1-2. The same $N_{tr}$ sampled reasoning paths per training problem are expected to provide more supervision in TVM, which seems a disadvantage.]

The reviewer mentioned that the same $N_{tr}$ sampled reasoning paths per training problem are expected to provide more supervision in TVM, but this only pertains to the case when $N_{tr}$ is small, as discussed in Weakness 1-1. Rather, it is important to emphasize that when using a large enough $N_{tr}$, token-level supervision signals can be effectively obtained (as also highlighted in Weakness 1-1). In this regime, TVM benefits from more detailed token-level supervision, even for the latter parts of reasoning paths. In our experiments, setting a commonly utilized $N_{tr}$ as in [1] was capable of providing this detailed supervision, which has been empirically shown to enable TVMs to outperform existing verifiers (ORMs and PRMs) when using tree search methods.

[1] Training Verifiers to Solve Math Word Problems, arXiv:2110

---

Dear Reviewer S2DY,

We hope this message finds you well.

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. With less than 48 hours remaining before the deadline for reviewers to post messages to the authors, we kindly remind you of our responses to your constructive and insightful comments. If you have any remaining questions or concerns, please feel free to reach out.

Best regards,

Authors