Response to Reviewer Comments

Dear Reviewer NpJW:

We would like to express our sincere gratitude for your time, your thorough review and valuable feedback on our manuscript. Your comments have provided us with important insights that will help improve the quality of our work.

W1: Ablation Study on Threshold Decision

R1: We appreciate your suggestion regarding the threshold decision process. Though our approach is grounded in cognitive theory, an ablation study is needed in the revised manuscript to demonstrate the impact of different thresholds. We add the BoN results of ASPRM models trained with threshold of 0.5%, 1%, and 1.5%. However, larger thresholds (3%, 5%, 10%) mean perform more rollouts. Due to computational resource constraints during rebuttal, we are unable to conduct extensive experiments with various models and higher threshold combinations, we will conduct further ablation studies when we have more computational resources available. Table 1 and Table 2 shows the BoN results of ASPRM trained with threshold at 0.5%, 1.0%, and 1.5%, compared with the baselines, and a 0.5% change means that each solution has one less step on average. We find that although there is some fluctuation, a greater number of segments within the range of 0.5% to 2% generally means better judging capability. We used more models (MetaMATH-Llama-7b / -13b - 70b) for generation to supplement the results. We hope for your understanding and approval due to the lack of additional results.

Table 1: Bo64 results on Math500.

Table 2: Bo64 results on GSM8k.

Besides, we also investigated the influence of various thresholds on the performance of TVD. The results have been collated and presented in the following two figures: Figure 1 and Figure 2.

W2 : Model Influence of Confidence Threshold

R2: We use the pecentage of whole distribution of model confidence, instead of a fixed threshold, this means that different models will indeed have different threshold values, and different tasks will also have different threshold values. We use a table to illustrate this, the following Table 3 below shows the values at a 2% confidence threshold for different models and tasks. We find that for the same split ratio, models with greater capabilities and simpler tasks tend to have higher threshold values.

Table 3: 2% confidence threshold for different models and tasks.

W3 and T1: Figure, Table and Typographical Errors

R3: Thank you for highlighting the issues with the table, the up and down arrows' color and the typographical errors in our manuscript. We will correct these inaccuracies in the revised version to improve clarity and readability.

---

Thank you again for your time, thorough review and constructive feedback. Your insights have significantly helped us identify areas for improvement in our work. We welcome any additional suggestions that could further enhance the quality and rigor of our research.

Response to Reviewer Comments

Dear Reviewer LC8A:

We greatly appreciate your insightful comments and suggestions and thank you for your time. Due to the character limit, we have summarized your questions. If we have missed or misunderstood any points, please let us know.

Related Work

Thank you for your valuable suggestions regarding related work. We will incorporate literature on reward models for LLMs in our revised manuscript, including BT model, GenRM from Zhang et al. (2024), Pairwise RM from Jiang et al. (2023), Cobbe et al. (2021), Li et al. (2022) and so on.

Q1 and W1: Suboptimal Concerns

R1: Thank you for expressing your concerns. We provide a statistical results to show the relation between segmentation numbers and task difficulty. As shown in the figure, there are indeed some segmentation issues in the lower-left corner and on the right side, where few difficult problems are segmented into fewer steps, or certain easy problems are segmented into more steps. Although these cases are rare (fewer than 2%), the issue does exist. We believe that combining rule-based methods with AdaptiveStep would result in better divisions.

Q2 and W2: Threshold Settings

R2: Thank you for your insightful suggestion. We will include more percentage settings in our experimental evaluation to provide a more comprehensive analysis of our method's performance across various conditions. We used more models for generation to supplement our results. Table 1 and Table 2 shows the BoN results of ASPRM with threshold at 0.5%, 1.0%, and 1.5% compared with the baselines, and a 0.5% change means that each solution has one less step on average. Due to computational limitations, we are unable to scale up more rollouts in a short time, but we will conduct further analysis in subsequent versions, we hope for your understand. We find that although there is some fluctuation, a greater number of segments within the range of 0.5% to 2% generally means better evaluation capability.

Q3, Q4 and W6: Implementation Details

R3: Thank you for highlighting the missing methodological details. Regarding data preprocessing, we have provided training data specifications in Sec.4 (Parameter Setting), which documents 30 solutions per problem and 8 rollouts per step for labeling. In Sec.4.4 (Construction Efficiency), we present a comparison of computational costs with others.

For ASPRM training, we employed a batch size of 256 and learning rate of 1e-6 (Mistral), 2e-6 (Llama), 5e-6 (Deepseek), consistency with the Math-Shepherd parameters in the OpenRLHF script.

For baselines, we utilized the released models of other methods and rigorously adhered to their prescribed usage for evaluation on our dataset. At the 7B model scale, our reported Math-Shepherd BoN results exceed those in the original paper, demonstrating the fairness of our comparisons.

Q5 and W4: OmegaPRM and RLHF

R4: Thank you for raising these questions. OmegaPRM is a method that integrates binary search into MCTS to construct data, while ours is a fine-grained step segmentation method. The methods are orthogonal, and since this and data have not been released, we did not use it as a baseline for comparison. RLHF is indeed a great comparison approach, but the computational resources required for RLHF are currently beyond our capacity. Therefore, we opted for the more lightweight BoN and TVD methods for evaluation, which are commonly used in previous reward model works.

Regarding the combination with these methods, we believe that fine-grained segmentation helps OmegaPRM more accurately identify errors, thereby improving the error detection efficiency. As for RLHF, positions with low confidence indicate that the model likely generate other branches. Compared to rule-based step-level RLHF, we believe that using AdaptiveStep allows the model to better learn from these branches, leading to good results. We will also validate these claims when sufficient computational resources are available.

Q6 and W3: Commensense Domain

R5: Utilize to commensense domains remains challenging for process reward models. Some researchers employ powerful models like GPT-4o or human judgment for evaluation [1]. We believe AdaptiveStep offers advantages in this context: our segmentation method identifies low-confidence model outputs, suggesting more choices and higher error probability, thereby improving labeling efficiency.

[1] O1 Replication Journey: A Strategic Progress Report: Part I

---

We sincerely thank you for your constructive feedback and insightful recommendations. These comments have significantly helped us identify areas for improvement in our manuscript. We look forward to incorporating more suggestions from you to enhance the quality and clarity of our research.

Response to Reviewer Comments

Dear reviewer ZPdL:

Thank you for your insightful comments and valuable feedback. Below, we address the concerns raised and provide clarifications and improvements.

W1: Concerns about experiments results

R1: Thanks for your comment and we will revise our paper to clarify the setting/scope of the SOTA statement accordingly. We argue that the overall performance of ASPRM-M exceed that of the baselines. Specifically, ASPRM-M significantly outperforms baselines across all TVD experiments. In the BoN-Mistral generated data, ASPRM-M performs better than Math-Shepherd when N is small. In positions with few instances, such as Figure 4(b) for Bo64 and Figure 4(d) for Bo16 and above, ASPRM-M is slightly worse than Math-Shepherd. In subsequent experiments, such as those involving position generalization, ASPRM also performs well.

W2: Llama compared with Mistral

R2: Thank you for bringing this up. The experiments involving Llama and Mistral were simply presented in the same figure for space efficiency. There was no intention to directly compare the performance of these base models. We likely would not have conducted the experiments with the Mistral-based ASPRM-M model if we want to make a direct comparison.

W3: Reward Model Related Work

R3: We appreciate the suggestion to add related work on reward models. We will revise the manuscript to include relevant references and discussions on prior work in this area (e.g., the Bradley-Terry model, GenRM, Pairwise reward model, etc.).

W4: RLHF and More Models

R4:
RLHF Methods:
While RLHF methods are often used to evaluate reward models, they are not strictly necessary for this area. Several influential works on (process) reward models have not included RLHF experiments but use BoN or test-time scaling to evaluate the model’s capability, yet they remain impactful in the field [1, 2]. Our focus was on proposing and testing a novel reasoning step segmentation method. Given the substantial computational resources required for RLHF experiments, we are unable to include them in this study. We kindly hope that the reviewer takes this into consideration.

More Models:
Regarding the inclusion of more model scales, we have conducted experiments on a wider range of model sizes and thresholds in Table 1 and Table 2, which we will incorporate into the updated manuscript, we observe that as the model size increases, the PRM’s ability to make judgments remains effective.

[1] Let's Verify Step by Step
[2] Training Verifiers to Solve Math Word Problems

---

Q1: Our Contributions

We sincerely request that the reviewer reconsider the contributions of our work. Our primary contribution is the introduction of a novel reasoning step segmentation method AdaptiveStep, which we validate in the PRM senario. The PRM trained with AdaptiveStep can guide the task model in generation and reasoning at the token level, which previous PRMs have not been able to do in our experiments. Moreover, we have successfully extended PRM to domains where step segmentation is difficult using rules (such as code generation). And we use a single model for data construction and reduces construction cost by 30% in the math domain and 60% in the code generation domain compared to rule-based methods.

In the experimental setup, we have made every effort to maintain consistency with previous work, including model selection, PRM training methods, and evaluation techniques. This was done to establish a baseline for a fair comparison, which consequently limited the potential for introducing significant innovations in the training of the PRM or the testing of BoN/TVD.

---

Q2: Credit and License

Thank you for pointing this out. We use this dataset only for research purposes and will strictly regulate its usage context and license. We will do our best to align our open-source work with LiveCodeBench, which collects data from LeetCode weekly contests.

---

General Rebuttal

We propose a novel reasoning step segmentation method and have evaluated it within the context of process reward models, supported by statistical analysis and extensive experiments. While RLHF methods are a reasonable evaluation approach, the heavy computational resources required for such experiments made it unfeasible for us to incorporate them, so we use the lightweight BoN and TVD method. As highlighted earlier, many impactful reward model studies have not employed RLHF experiments, yet they have been influential in the field.

---

We greatly appreciate the your thoughtful feedback and your time for upgrading our work once again, and we hope that these clarifications will be satisfactory. We welcome any further constructive comments that can help improve the quality of our work.

Response to Reviewer Comments

Dear reviewer T2AZ:

We sincerely appreciate your constructive feedback and the time you have devoted to reviewing our manuscript. We try to address your concerns point by point.

W1: Discussion or empirical analysis for failure cases of AdaptiveStep

Response: Thank you for highlighting this important aspect. Though explicitly classifying "failure cases" in the model's segmentation is challenging as what is regarded as "failure" for humans indeed aligns with the model's confusion points, we still identified several instances of suboptimal segmentation results.

We understand that failure cases can be categorized from two perspectives. The first perspective involves words that should remain intact but are erroneously split into separate parts. The second perspective concerns instances where the model fails to segment at positions prone to errors. We illustrate the first type of failure cases through the following examples, while the second type is demonstrated through a figure.

Example 1: "We can rewrite the quad / ratic as $y = 3(x^2 + 2x) + 9$"

Example 2: "Sub / stituting these values" — where "quadratic" and "substituting" are incorrectly segmented despite being complete words

Example 3: "Natal / ie would need to trade" — where "Natalie," a proper noun referenced in the question, is inappropriately segmented

We conduct a statistical analysis of the aforementioned type where words are split, and find that about 2% of the segmentation belongs to this category. However, since this split is determined by the model itself, we retained these splits during the training process.

Additionally, in the very begining of our work, we observed that approximately 3% of split points occur at the beginning of solutions, which we classify as erroneous segmentations, we removed these split positions because they indicate that the solutions have not yet started to be generated.

For the second perspective, due to limitations of the base model, certain complex questions exhibit little segmentation in their solutions, as illustrated in the figure, we believe that the problems in the lower-left corner (the 1.62% with low accuracy after 64 generations and segmentations numbers lower than 5) represent problems that exceed the model's capabilities, and require additional supervisory information to assist.

In summary, aside from the examples that are beyond the model's capabilities to judge (which account for about 5% of the total questions), there are 5% of the splits that, from a human perspective, seem unreasonable. We have removed some of these (those at the beginning of the solutions) and retained others. We will explain this in more detail in the appendix. Thank you again for your highlighting.

W2: Algorithm definition

Response: Thank you for your suggestion. We will include the following algorithmic in out next version:

Algorithm: PRM Training from Confidence-Guided Rollouts

Inputs:

• Q: A sequence of questions
• N: Number of responses to generate per question
• J: Number of rollouts per split point

Output:

• A trained process reward model (PRM)

for each question q in Q do
    // Step 1: Generate N responses for the question
    responses ← GENERATE_RESPONSES(q, N)

    // Step 2: Compute confidence distribution for the responses by eqution (1)
    confidence_distribution ← COMPUTE_CONFIDENCE(responses)

    // Step 3: Determine threshold and find split points
    threshold ← COMPUTE_THRESHOLD(confidence_distribution)
    split_points ← FIND_SPLIT_POINTS(confidence_distribution, threshold)

    for each split in split_points do
        labels ← empty list

        // Step 4: Perform J rollouts for each split point
        for i from 1 to J do
            label ← PERFORM_ROLLOUT(split)
            APPEND(labels, label)
        end for

        // Step 5: Aggregate rollout labels into a hard estimate by eqution (2)
        prm_label ← HARD_ESTIMATE(labels)

        // Step 6: Train the PRM with the obtained labels by eqution (3)
        TRAIN_PRM(prm_label)
    end for
end for

---

We welcome any additional feedback you may have regarding our manuscript or this response. We are committed to incorporating your suggestions to enhance the quality of our work. Thank you again for your valuable insights.