AlphaMath Almost Zero: Process Supervision without Process

Dear Reviewer 5iMV,

We are grateful for the time and effort you have invested in providing detailed and insightful feedback. We appreciate your recognition of the key contributions of our method, including eliminating the need for human annotation, the commendability of trajectories generated via MCTS, and proposing step-level beam search to reduce inference cost.

Distinctions from other works can refer to G1 in general rebuttal. Then, please find our detailed responses to each of your points below.

W1: The paper's results are all program-aided, a fact not clearly identified in the text. Consequently, the comparison with non-program-aided LLMs may not be fair enough.

In Line 208-211 of Section Experiments, we mentioned that To ensure a fair comparison, we primarily contrast our approach with the highest-performing SFT models that utilize an external tool - a Python code interpreter. These include MAmmoTH [43], MathCoder [35], ToRA [12], MARIO [44], MathGenie [23], and DeepSeek-Math-Instruct [29]. More implementation details can be found in Appendix C.. In addition, in the Tool column in Table 2, we also marked whether each method uses program.

Most of the 7B SOTA methods, e.g., ToRA, DeepseekMATH-instruct, use python programs to perform mathematical reasoning. As the program can mostly address the numerical computation in math reasoning. For larger LLMs, a recent example shows that most LLMs may fail for "9.9 and 9.11, which is bigger?". However, with the assistance of program, LLMs can easily solve such problems [r1].

[r1] Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification

The comparison with non-program-aided LLMs is for reference only, which is a convention in previous works, e.g., ToRA, MAmmoTH, etc.

Furthermore, previous works relied on human or GPT-4 annotation process to supervise the data, as shown in the Seed Data Annotation column of Table 2. Our almost unsupervised/weakly supervised setting v.s. supervised setting in previous works is actually an unfair comparison to our method, but we still achieve very competitive results.

We hope this clarification addresses your concern, and we are open to further discussions should you have any additional questions.

W2: The benchmark only includes GPT models in the 'Proprietary Models' section. It would be beneficial to include other LLMs such as Gemini and Claude for a more robust evaluation.

First, we include the GPT-4 results quoted from previous works for reference only, by following the convention of related works. The "Proprietary Models" are not fully accessible all over the world and require global credit card for the API usage. It is difficult to evaluate on other proprietary models, especially on the OOD data, but we would like to include the GSM8K/MATH results of Gemini and Claude from relevant papers in revised version.

Additionally, all "Proprietary Models" should be larger in size than the 7B model, which is the main focus of our related works, making it difficult to justify fairness.

W3: An image illustrating the Step-Level Beam Search (SBS) would greatly aid in explaining the algorithm.

In Line 168-178 and Algorithm 2, we introduced the SBS by modifying MCTS. However, we can view the SBS from the perspective of traditional (token-level) beam search (BS).

For simplicity, we use SBS(1, 5) and max step=3 for illustration and we will add an image in our revised paper.

1. The policy model samples 5 candidates for step 1, denoted as $a_1^1, a_1^2, \ldots, a_1^5$. (similar to 5 candidate tokens in BS)
2. The value model reranks 5 candidates, and selected the best, e.g., the top-1 is $a_1^1$. (BS may use some variant of logprobs for reranking, such as length penalty version)
3. The policy model samples 5 candidates for step 2 based on prefix $a_1^1$, denoted as $(a_1^1,a_2^1), (a_1^1,a_2^2), \ldots,( a_1^1,a_2^5)$.
4. The value model reranks 5 candidates of first two steps, and selected the best. For example, we assume the top-1 is $(a_1^1,a_2^1)$.
5. Repeating above process in 3 and 4 will get the final best path $(a_1^1, a_2^1, a_3^1)$.

L1: The framework does not generate a reward model that can facilitate other LLMs, which potentially limits its applicability.

Thank you for your helpful comments. We would like to address that our framework can also facilitate other LLMs.

• The step-level evaluation signals (Q-values) generated by our framework can be used to train a separate value model. Reviewer 2HCk concerned about the performance of training the value model and policy model separately. We share the experimental results here.

From the table, our AlphaMath has similar overall performance to the two separate models, but two separate models requires more inference time and computing resources, which is what we want to avoid.

• Our joint model (when using value model head only) can also score other LLMs with Python Code as long as the output format of other LLMs is reorganized by following the our defined REACT format.
• If the LLMs insist to use CoT solution, it needs to retrain the value model in the MCTS framework with step-level CoT solutions.

L2: The framework cannot be extended to tasks with open-ended outputs, limiting its potential use cases.

In Line 518-520, we have stated that our approach only works on tasks that are evaluated based on the actual answers in future work section. To further clarify, we will add the scope of our method in Limitation.

We sincerely appreciate your invaluable feedback and would be delighted to engage in further discussion to address any questions you may have, with the aim of improving and enhancing the quality of our work.

Sincerely,

Authors of 6802

Dear Reviewer 5iMV,

Thank you for your continued engagement.

In Lines 199-202 and 682-684 of our paper, we provide a detailed introduction to this dataset. Specifically, OCWCourses (OCW) focuses on STEM (Science, Technology, Engineering, and Mathematics) problems, particularly emphasizing calculations in physics, chemistry, and biology, which present a high-quality out-of-domain dataset and greater challenges compared to MATH. This dataset is sourced from [1]. The dataset link on huggingface hub. As the DeepSeek uses this testset and one of our main models is built upon DeepSeek pretrained base LLM, we also use this testset for a fair comparison.

We hope this clarification addresses your concerns.

Additionally, do you still have questions about your previous concerns? We would appreciate it if you could share any further questions you may have. We look forward to the opportunity to enhance your understanding of our work through continued dialogue.

[1] Lewkowycz, Aitor, et al. Minerva: Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems 35 (2022): 3843-3857.

Sincerely,

Authors of 6802

Dear reviewer,

Can you let the author and us know if you are satisfied or not with their rebuttal, and if you have any further comments?

Thanks,

AC

Dear Reviewer otxC,

We greatly appreciate your valuable comments and are pleased that you recognize the novelty and efficiency of our proposed method. Below, we address your comments in detail.

W1: Although the method does not rely on human-annotated process data, it still requires actual answers as reward signals. Future work could explore fully unsupervised approaches.

Thanks for the insightful suggestion. In Line 499-505 of Section Future Work, we also mentioned the possibility of "Really from Zero", quoted as A challenging yet profoundly meaningful future direction is to identify an appropriate reward definition that allows AlphaMath to eliminate dependence on actual answers, thereby achieving really from zero. .

A promising trial can be exploring in proof problems by leveraging the Lean programming language. Unlike traditional problems that require a final answer, proof problems focus on verifying both numerical computations and proof correctness. The Lean language is well-suited for this purpose, especially for proof derivation. For example, recent AlphaProof [r1] achieved 2024 IMO silver medal by applying our method-alike approach with Lean.

[r1] DeepMind. AI achieves silver-medal standard solving International Mathematical Olympiad problems

W2: Although the step-level beam search improves efficiency, the MCTS method still has some limitations in terms of computational complexity, which makes its practicality indistinguishable from majority voting.

Thanks for pointing out this issue. We totally agree with your claim that compared with majoring voting, inference with MCTS requires larger computational complexity. This is also exactly the reason why we mainly use MCTS in training rather than inference. Value estimation with MCTS can provide more accurate learning signals than simple Monte Carlo estimate, so training with MCTS framework is most beneficial to value model.

To address the inference issue of MCTS, we further proposed to modify it and achieve a very efficient inference method--step-level beam search (SBS, note that SBS cannot be used for training). In Line 168-172, we mentioned that However, MCTS is computationally intensive for simulations, making it less viable for use in production environments. To address this, we modify the MCTS inference process by eliminating the backup operation, introducing a simplified method, which we refer to as Step-level Beam Search (SBS), detailed in Algorithm 2. This approach does not construct the entire tree; instead, it dynamically selects the best child node during node expansion.

In Tables 2 and 3 of our paper, we also demonstrate the significant advantages of SBS in efficiency and effectiveness compared to MCTS inference, greedy decoding or majority voting. For example, SBS ($B_1=3$) is 5 times faster than MCTS and has 1.7% better performance. Meanwhile, the efficiency of SBS is similar to that of greedy decoding, with only a difference of 0.8s per question, but the efficiency is improved by 12.1% on the MATH dataset. We believe that these aspects demonstrate the superiority of our proposed SBS.

Compared to majoring voting, Table 3 of our paper cannot fully show the efficiency difference between SBS and maj@n. However, if we scale $n$ to large numbers, from the following table, we can see SBS shows significant advantage on accuracy.

We hope the above clarifications have enhanced your understanding of our work. We appreciate your valuable feedback and are eager to further discuss any additional questions or comments you may have to improve our manuscript.

Your insights are invaluable in our endeavor to refine and strengthen our research. Thank you once again for your thoughtful review.

Sincerely,

Authors of 6802

Dear reviewer,

Can you let the author and us know if you've read the rebuttal, and if you have any further comments?

Thanks,

AC

Dear Reviewer 15WC,

We sincerely appreciate your efforts in evaluating our manuscript. We have carefully considered your insights and critiques, then address each of your comments.

W1: Distinctions to [1,4]

Thanks for raising this issue. We will avoid over-claiming in future version. Highlighted contributions and distinctions to [1,4] can refer to the G1 in General Rebuttal.

W2/W3: Step segmentation; How to generate/stop a step

W2 and W3 are relevant, so we will address together.

In Lines 190-192, we already mentioned that the step is defined in the form of LLM REACT format and provided the examples in Appendix F. The REACT format step includes Thought(text analysis) / Action(python interpreter) / Action input(code snippet) / Observation(code output). In Line 620-627, detailed definitions and examples of the $C$-steps and $A$-steps are also provided. $C$-step is REACT step including code, and $A$-step is the last REACT step including final summary only.

We follow the REACT format to generate and stop a single step. In REACT, one can define the possible stop tokens for Observation. As shown in Appendix F, we can define the stop tokens, as "Observation" for round 0 and ["</code>", "</step>"] after round 0. When encountering any stop token, the LLM generation will stop. In open-source toolkit vllm or Openai API, stop token is an argument, thus it's not necessary to generate the entire solution and truncate it. In summary, the step-level sampling can still be efficient.

More details can refer to REACT paper or relevant API document. This is not our focus, but we will add a relevant description in future version.

W4: Explain trade-off rollout and max depth=8

We apologize for misuse of follow in line 125, which should be design. In AlphaGo, every rollout requires a light-weighted fast policy to sample hundreds steps in depth until terminal node. In AlphaGo Zero, there is no rollout at all, and the value estimation only depends on prediction of value model in previous round. Our defined value estimation $(1-\lambda)V_{\phi}(s_t) + \lambda r(s_{t+1})$ has the same form as AlphaGo. Differently, we set $\lambda$ as indicator function $I_{terminal}(s)$. Lines 126-128 also explain it means such setting rollouts for only one-step. If the rollout reaches terminal, value adopts the only reward, otherwise, the value model prediction.

As we explained for REACT, depth=8 means 7 code steps + 1 summary step. Even for MATH, 7 code snippets are enough to solve most problems. E.g., in ToRA (ref [12] in paper) found GPT-4 can solve 83.1% MATH problems within 3 code snippets. Since our base model is weaker than GPT-4, we relax it to 7. Table 4 in PDF file shows the detailed step statistics.

W5: Explain assumption

Our assumption is $Q(s_t, a_t) = r(s_t, a_t) + V(s_{t+1}) = V(s_{t+1})$ for non-terminal nodes.

• The 1st "=" is not assumption, but the definition of state-action and state values [r1,r2]. It is due to deterministic transition $s_{t+1}=cat(s_t, a_t)$ in Eq. (1) for LLM output.
• We mentioned why the 2nd "=" holds in footnote 1. The reward of non-terminal nodes is 0. This is a common definition in RL if the reward can only be determined until the end of the episode.

[r1] Sutton & Barto's Reinforcement Learning: An Introduction

[r2] The UC Berkeley course: Deep Reinforcement Learning (CS285)

W6: Explain labels are V, not z

In standard RL, most value fitting algorithms aim to learn intermediate or step-level values [r1,r2] with various value estimation techniques. Therefore, we didn't make the modification; instead, AlphaGo series did.

The reason why learning from final reward works is that AlphaGo series applied thousands of simulations or sampled trees in MCTS. In the MSE value fitting, which is theoretically a regression problem, the regression signals 1 and −1 are proportional to the number of positive and negative rewards in simulations. The ratio asymptotically approaches the true value due to the large sampling size. Thus, the overall gradient direction for MSE should be asymptotically accurate. In LLMs, we cannot afford thousands of simulations.

W7: Explain value selection in MCTS inference

Selection with maximum Q-value is not a modification, but one of popular methods, as introduced by textbook [r1] Sec. 8.11 MCTS, quoted as some mechanism that depends on the accumulated statistics in the tree, e.g., the largest action value, or action with the largest visit count).

Similarly, due to the large amount of simulations in AlphaGo series, using visit count is a robust estimation. In our case with much fewer simulations, we found maximum Q-value is more robust. In the submitted code offline_inference.py, we already implemented visit count. For the response purpose, we include the relevant result in Table 5 of PDF file.

Actually, using the variant of visit count $\propto N^\alpha$ in AlphaGo series is a modification. Previous works such as [4] follow the variant without considering sampling size, which may lead to suboptimal.

W8: Discuss large B1 with fewer steps, and majority voting

As we explained in previous response, each REACT code step includes text analysis, a code snippet and its execution output. If the output indicates an error message, the LLM will generate a new REACT step by rewriting the code snippet until no error occurs or max depth reaches. When B1 increases, there are more candidates for the value model to select from. Our MCTS training framework encourages the value model to choose reasoning steps that don't output error message. Thus, the average number of overall reasoning steps decreases as B1 increases.

Please kindly refer to G2 in General Rebuttal for the discussion of majority voting.

W9 and Limitations

Please kindly refer to G3 in General Rebuttal.

We sincerely appreciate your valuable feedback and look forward to address any further concerns you may have.

Sincerely,

Authors of 6802

Dear author,

Can you let the author and us know if you've read the rebuttal, and if you have any further comments?

Thanks,

AC

Dear Reviewer EVyZ,

We greatly appreciate your detailed review and the recognition of the strengths of our paper. Due to word limit, we apologize for shortening your questions. We will address each of your comments in detail, hoping to alleviate any concerns you may have.

W1: Duplicate citations

We apologize for the error and will correct all reference errors in the revised version.

W2: Explain acc of some levels in round 3 declined

Thanks for highlighting this issue. In Line 194-195, we have discussed the selection of $K$ using the Elbow method on overall accuracy. The details including next round (round 4) can refer to Table 1 in PDF file. As we discussed in Line 564-568, the mixed increase and decrease performance on finer-grained levels is normal after training (overall accuracy) converges.

W3: What about CoT

Due to limited time, we cannot do full experiments on CoT. For a fair comparison, we follow most SOTA methods in the setup of CoT+Code, because it has been proven (eg, [r1] or ToRA) that the performance of COT is less effective to CoT+Code, and LLMs struggle with numerical calculation. A recent famous example is that LLMs fail to compare 9.9 and 9.11.

W4: Other OOD data

Thanks for the suggestion. We did not use the GSM8k_hard dataset because it does not represent a truly OOD scenario. It uses the same questions of GSM8k by varying digits, and many factual errors exists, such as the negative number of people.

As introduced in Appendix C.5, the two OOD datasets we used are significantly more challenging than GSM8K and MATH, with different question sources and domains (e.g., physics). So they can better represent the true OOD scenario.

For the response purpose, we evaluate on GSM8K-robust and its corresponding method, as shown in Table 2 in PDF file.

W5: Training data size; Try Llama2 or Mistral

In Line 186-187 , we noted that our raw training data only contain 15K QA pairs from GSM8K/MATH. Using these 15k pairs, we generate 59k solution intermediate processes by MCTS, significantly smaller than existing SOTA (776K in DeepseekMath-instruct). In addition, process annotation does not rely on GPT-4 or human.

In Table 4 of Section 4.6 and Table 5 in Appendix, we already explored the effect of other base LLMs, including general LLM Llama3, and SFT models (MARIO). We think Llama3 can represent the general LLMs, such as Llama2 and mistral.

W6: Hyperparameter beta in Eq(8)

We assume it refers to Eq(6). We conducted hyperparameter selection in our preliminary experiments. In this response, we would like to share our findings that $\beta$ does not significantly impact model performance. We list the results in Table 3 in PDF file for your reference.

W7: Log of training loss

We plot the loss in Figure 1 in PDF file for your reference.

W8: Accuracy of value model

Thanks for your valuable comments. It is theorectially difficult to evaluate accuracy of intermediate step values without human annotations. Even in the realm of reinforcement learning(RL), it is also a difficult task if no step reward is returned. Thus, most RL algorithms care more about the values ranking (e.g., via argmax), given current state or action.

In Figure 5 of Section 4.5, we have followed the convention of RL to illustrate the relationship between the value predictions of intermediate and final steps by plotting the distribution of $Q$-values against final reward/correctness. It offers a more intuitive understanding. On the testset, our value model predictions align well with expectations, particularly for correct steps.

In Line 277-298, we also discussed the minor flaws of the value model. As shown in Fig. 5, the value model will score $\approx1$ for some final wrong solutions, because the final incorrectness doesn't mean all steps are wrong, and correct steps may exist for early timestamp.

W9: More related works

Due to space limitation, we focused on data annotation and value model, as our title indicated. We totally agree that other fields, such as MCTS, are also important and we will revise accordingly.

W10: Performance of different rounds

Thanks for your insightful comments. Actually, in Figure 3/4/6/8/9, Table 5, Appendix D, we have performed extensive experiments to investigate the performance of different rounds, that covers your concern.

• policy model in Figure 3/4/6/8/9, greedy decoding w.o. value
• value model in Figure 4/6, SBS inference with value
• case studies in Appendix D, intuitively show how Q-values changes
• problem solving rate in Figure 3/9

W11: Labels of value model

We aim to clarify it by discussing the offline Q-learning method. Generally, the initial $Q$ values of each node in the MCTS tree are set to 0 and are subsequently updated as indicated in Eq. (4) and $Q$ update in Line 132.

Lines 143-147 explain how the values are calculated for the 1st round. Since the value model is randomly initialized, the value head's prediction (1st rightside term in Eq. (4)) should be $\approx0$. Thus, $Q$ update is primarily influenced by the final reward/correctness. Theoretically, with more MCTS simulations/rollouts, the $Q$ values will gradually converge to their true underlying ground truth labels.

For other rounds, Lines 120-132 describe the value calculation process. In 1st rightside term of Eq. (4), the value model is trained on the previously collected values, providing a more accurate prediction than $\approx0$ in 1st round. Thus, the $Q$ update in Line 132 may rely not only on the final reward but also on the value model's prediction.

[r1] Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification

We believe our explanations have clarified your concerns and offer a clearer understanding of our work. Thank you for your constructive feedback. We are eager to engage in further discussions to enhance and refine our work.

Sincerely,

Authors of 6802

Dear reviewer,

Can you let the author and us know if you've read the rebuttal, and if you have any further comments?

Thanks,

AC

Dear Reviewer 2HCk

We sincerely thank you for your thorough evaluation and for acknowledging the clarity of our paper, as well as the effectiveness and significant performance improvements we've demonstrated. Due to word limit, we apologize for omitting your question and replacing them with numbers and key words. Below, we address each of your comments in detail.

W1: Unfair comparison. Relationship between reward model and SBS, try best-of-n.

We appreciate the reviewers' feedback and the opportunity to clarify experimental comparisons.

1. In Line 223-224, we mentioned the greedy decoding results are included in Table 2. To clarify, it is marked as AlphaMath (K=3), where $K$ means the training iterations. Actually, due to our novel experimental setups, the experimental comparison puts our results at a disadvantage relative to previous works.

• Our training method is categorized as almost unsupervised / weakly-supervised learning, whereas previous works primarily employ supervised learning. If we were to exclude the human or GPT-4 annotated solution processes from the training data in previous works, and train their models using only question-answer pairs, we hypothesize that their performance would drop drastically.
• The dataset for supervised training in previous SOTA DeepSeek-Math-Instuct includes 776K question-process-answer tuples. In contrast, our method only relies on the 15K question-answer pairs.
• As shown in Table 2, in greedy decoding setup, AlphaMath achieves a performance score of 53.6, which is comparable or better than all previous supervised methods except DeepSeek-Math-Instuct's 57.2. The relatively small gap between an almost unsupervised method and a supervised method is noteworthy and underscores the potential of our approach.

2. As illustrated by the SBS (step-level beam search) inference in Algorithm 2, our results are already presented as best-of-n, but on a step-level basis. The value loss (the 2nd and 3rd rightside terms in Eq. (6)) indicates that the value model is trained at the step-level. During inference, for example, in a setup of $B_1/B_2=1/5$ the SBS needs to choose the best out of 5 generated outputs at each step, using the step-level score prediction provided by the value model. In summary, the value model and SBS inference algorithm are highly integrated to improve the performance, by selecting step-level best-of-n.

W2: More n for maj@n.

Thanks for the suggestions to make our experiments in a more comprehensive view.

First, We want to clarify that majority voting is not the focus of our work. Our training algorithm is fundamentally different from previous methods, which naturally leads to a distinct SBS inference algorithm compared to the majority voting used in earlier approaches. We propose an autonomous step-level process annotation method by leveraging the cooperation between policy and value models within the framework of MCTS. This policy-value reinforcement learning framework naturally favors the SBS inference algorithm.

Second, Table 2 is primarily used to compare our almost unsupervised / weakly-supervised results with previous methods in traditional supervised learning. Table 3 presents an analysis experiment of our own model to explore the trade-off between performance and efficiency using different inference algorithms.

Third, as suggested, we would like to include the performance of maj@n with varying n values. From the table, our SBS is significantly better than major@n. More ressults can refer to Table 6 in PDF file.

We hope the above explanations address your concerns and help you better understand our work.

Q1

We have addressed the relevant issue in the response for Weakness 1 and 2.

Q2: Performance of separating policy and value models.

In the realm of reinforcement learning (RL), whether or how to share networks between policy and value models is a long-debated design issue, [r1,r2] have noted that using two separate networks is usually easier to implement and more stable to train. However, this design is inefficient optimization as it doesn't allow for the sharing of internal representations. On the other hand, shared networks are challenging to train because the different losses generate gradients in various directions and scales, requiring careful tuning of the loss weights.

Making the training of shared networks stable is actually one of the contributions in our work as a RL method. Beyond this consideration, our insistence on tuning the shared architecture is driven by computational cost. Using two LLMs during inference nearly doubles the computational expense. Sharing networks had also been adopted in some famous RL works, such as AlphaZero.

For the response purpose, we'd like to include the new experimental results with two separate models. We set the temperature to 0.6, $B_1/B_2=1/5$, and compare our AlphaMath (joint training) with two separate models, as shown in the following table:

From the table, our AlphaMath has similar overall performance to the two separate models, but two separate models requires more inference time and computing resources, which is what we want to avoid.

[r1] Sutton & Barto's Reinforcement Learning: An Introduction

[r2] The UC Berkeley course: Deep Reinforcement Learning (CS285)

We hope the above clarifications will help you better understand our work. We are happy to communicate with you further if you have any questions to enhance our manuscript.

Sincerely,

Authors of 6802

Thanks for your valuable response.

Our training method is categorized as almost unsupervised / weakly-supervised learning. The relatively small gap between an almost unsupervised method and a supervised method is noteworthy and underscores the potential of our approach.

Thanks for your suggestion about [1]. We will cite this paper in our related works, and categorize our work as bootstrapping reasoning.

More n for maj@n. Could the authors provide more details on the experimental settings?

We directly use the open-sourced vllm inference framework by setting the number of generations n=20. The parallelism is implemented by the vllm. We think the reason why the inference time of n=20 is $5\times$ longer than n=5 is that the Python code execution is on CPU rather than GPU. As we mentioned, we use the LLM REACT agent, including both text+code generation (GPU parallel) and code execution (CPU parallel).

What are the results of weighted voting and best-of-n? I believe a fair baseline should also incorporate the value/reward model during inference.

In response of W1, we mentioned that as illustrated by the SBS (step-level beam search) inference in Algorithm 2, our results are already presented as best-of-n, but on a step-level.

The value loss (the 2nd and 3rd rightside terms in Eq. (6)) indicates that the value model is trained at the step-level. During inference, for example, in a setup of the SBS needs to choose the best out of 5 generated outputs at each step, using the step-level score prediction provided by the value model. In summary, the value model and SBS inference algorithm are highly integrated to improve the performance, by selecting step-level best-of-n.

MCTS is a baseline that has already incorporate the value model.

Since our value model is step-level in the setup of LLM REACT agent, for a fair comparison with incorporating the value model, we just implemented the step-level weighted voting based on our submitted code. For each step, we first sample 5 candidates, and select the step with largest weight summation. Then, we proceed to the next step.

Given our step-level value model, there is not significant difference of above two setups, demonstrating the robustness of our step-level value model and SBS inference algorithm.

Dear reviewer,

Can you let the author and us know if you've read the rebuttal, and if you have any further comments?

AC