AutoPSV: Automated Process-Supervised Verifier

Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:

R W1 Presentation of Section 4.2.1:

Thanks for the suggestion. To avoid potential misunderstandings, we will change the term to "calculation error detection."

We would like to provide the following clarification for Section 4.2.1. We introduce a method to automate the acquisition of ground truth by detecting errors in mathematical calculations. This approach provides an effective and robust automated framework for detecting math calculation errors without requiring manual annotations. Based on this framework, we further establish a PCH benchmark as described in Appendix E. This addresses the primary challenge of evaluating whether a model detects all faulty steps due to the lack of ground truth process annotations.

R W2 No baseline in the main experiment (Table 6 & 7)

We will move the baselines from Section 6.1 to the main experiments (Tables 6 & 7) and separate the math and common reasoning tasks into different subsections for better presentation.

R W3 OOD evaluations are required to examine the generalizability

To address your concern regarding the need for out-of-domain (OOD) evaluations, we utilized the trained OSV+PSV model from the original manuscript. We conducted additional OOD tests on MMLU, BBH, and ARC-C. As shown in the table below, our method consistently outperforms baselines on three benchmarks.

R W4/5 Comparisons with MCTS-based Methods:

1. GPU Hours Comparison: we provide a comparison of GPU usage for both training and annotation costs:

As the number of steps required to solve a question increases (e.g., MATH compared to GSM8K), the time required for MCTS annotation increases quadratically. Considering training and annotation costs, the total GPU hours for AutoCV are approximately 1/8 of those for MCTS on GSM8K and approximately 1/40 for MATH. We will update Table 9 to reflect this and provide a more balanced view of the efficiency of AutoCV compared to MCTS-based methods.

2. Flexibility and Further Performance Enhancement: MCTS-based methods require ground truth on the correctness of final answers. In contrast, AutoCV generates process annotations by detecting relative confidence variations without needing ground truth annotations. This flexibility allows AutoCV to utilize redundant unlabeled questions, improving the model's performance. The results are provided below:

When MCTS and OSV-only training are limited to GSM8K labeled data, our AutoCV extends to apply 7K unlabeled math problems following the Evol-Instruct method from WizardLM [1]. Including unlabeled data further enhances the model's capabilities, demonstrating the value of the AutoCV approach.

R Q1 Verifier Performance with Different Generators

We applied the Llama2-13B [2] model and followed the settings in our paper, testing the MATH test set. The results are presented below:

From the table, it is evident that increasing the model size to 13B improves the performance of AutoCV across all settings. This demonstrates that our method works effectively even with strong response generators like Qwen, not just weaker ones. It shows that AutoCV performs better when applied to stronger and larger LLMs.

[1] WizardLM: Empowering Large Language Models to Follow Complex Instructions. ICLR 2024

[2] Llama 2: Open Foundation and Fine-Tuned Chat Models. Arxiv 2023

Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:

Response to W1 Presentation:

1.1 No mention of the #samples being reranked

Our task involves selecting the correct candidate from five responses. Therefore, we compare our method to the Pass@5 performance in each table, including Table 3 in Section 4.1. Specifically, we noted on line 256 that "Pass@5 represents the upper limit of performance."

We will add this detail to the table captions and the main context to enhance clarity.

1.2 Analysis of the aggregation function

We utilize the product of step-level scores as the aggregation function. Additionally, we have analyzed different aggregation functions, specifically the minimum and the product of step-level scores, following the setting of [1]. Below, we present the results on GSM8K:

The product aggregation function multiplies the confidence scores of each step, leading to a compounded probability that represents the overall likelihood of a correct response.

1.3 Why the training data was different in Table 4

The OSV model is continuously trained from the GSM8K fine-tuned model with the addition of a value head.

Therefore, in Table 4, the training data for OSV (Mistral) is generated from the fine-tuned Mistral model, while the training data for OSV (Phi) is generated from the fine-tuned Phi model. This distinction in training data sources explains the performance disparity observed between OSV models for different response generators.

To avoid any confusion, we will add further clarification in the manuscript.

Response to W2 Significance of results:

2.1 Quality of process labels

Table 10 shows that retraining the model with process labels only from AutoCV yields better performance than self-consistency, indicating that these labels are of good quality and successfully inherit information from the outcome-supervised model without requiring ground truth annotations.

Additionally, Table 5 in Section 4 (Preliminary Findings) further demonstrates the effectiveness and reliability of AutoCV's relative confidence variation in detecting errors during math reasoning.

2.2 Why train a PSV if using PS data drops performance (Table 10) and OSV is equally effective and provides process-annotation scores?

The training data for the PSV model does not utilize ground truth labels; it uses process supervision data generated by detecting confidence changes between steps (as in Eq. 3). Therefore, in the OSV+PSV training phase, we can include both the GSM8K/MATH training set and unlabeled math problems.

In Table 10, the PSV model maintains performance close to the OSV model. Notice that OSV relies on ground truth labels to determine the correctness of each solution, but PSV does not. To further illustrate the no-ground-truth benefits, we generated 7K unlabeled math problems following the Evol-Instruct method from WizardLM [1], which are generated by LLMs without gold solutions. We conducted an additional experiment on GSM8K by including these unlabeled questions. Methods like MCTS and OSV-only training cannot effectively utilize these unlabeled questions.

MCTS and OSV-only training is limited to GSM8K training data, our AutoCV extends to apply unlabeled data, contributing to a performance gain (e,g, 61.41 to 63.11 for Mistral-Instruct). Including unlabeled data further enhances the model's capabilities, demonstrating the value of the AutoCV approach.

2.3 Advantage of AutoCV compared with MCTS-based methods beyond efficiency.

The advantage of AutoCV is not just efficiency over MCTS-based approaches but also the ability to leverage unlabeled data for enhanced performance. MCTS-based methods require ground truth on the correctness of final answers. In contrast, the process annotations in AutoCV are generated by detecting relative confidence variations without needing ground truth annotations. This flexibility allows AutoCV to utilize unlabeled data, improving the model's performance. The experiment result is provided in the response to 2.2.

R Q1 How AutoCV work with Agent-based reasoning ?

Integrating AutoCV into agent-based reasoning traces is an interesting direction to explore. We can dynamically update labels based on feedback at each step instead of marking all subsequent steps as incorrect after detecting an error:

1. Detect the confidence variation $\Delta_{conf}^{t}$ as defined in Eq (3). If $\Delta_{conf}^{t}$ is below an error threshold, mark the step as incorrect.
2. Continue labeling subsequent steps as incorrect until $\Delta_{conf}^{t}$ exceeds a correct threshold.

The correct and error threshold settings may depend on experimental findings, similar to how we choose the threshold in AutoCV, as shown in Table 5.

[1] WizardLM: Empowering Large Language Models to Follow Complex Instructions. ICLR 2024

[2] Let's Verify Step by Step. ICLR 2024

Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:

R W1 Novelty and Substance of Our Paper

1. Theoretical Contribution: While reference [22] demonstrates that $f_\theta$ can indicate the probability of reaching the correct final outcome, our contribution extends this by exploring the change in $f_\theta$, denoted as $\Delta_{conf}^{t}$, between time steps $t$ and $t+1$. Unlike [22], which applies $f_\theta$ for reranking during decoding, our AutoCV demonstrates the effectiveness and robustness of $\Delta_{conf}^{t}$ for error detection, as detailed in Section 4.
2. Experimental Contribution: We demonstrate the performance gains of applying $\Delta_{conf}^{t}$ for process-supervision training in Sections 5 and 6. AutoCV combines the strengths of output supervision and process supervision to automatically annotate reasoning steps, thereby enhancing model performance.
3. Scalability: Unlike [22] and other MCTS-based methods, which limit their experiments to math tasks, our method is generally applicable across both math and commonsense reasoning. Furthermore, unlike methods limited to settings where ground truth is available, AutoCV can leverage both labeled and unlabeled data, significantly expanding the dataset for training. This flexibility enhances the robustness and applicability of our approach. More details on this can be found in R W4.

R W2 Naming Issue

Thank you for the suggestion. To better reflect the concept of changes in confidence from step t to step t+1, we propose using the term "Step-level Confidence Change. This name more accurately conveys the idea of confidence change between consecutive steps.

Similarly, to reflect these aspects more clearly, we propose renaming the system to "AutoPSR: Automated Process-Supervised Reasoner". This name highlights the focus on automated process labeling within the context of multi-step reasoning and chain-of-thought prompting. We use AutoCV during the rebuttal for consistency, but we will revise the term in the revised manuscript.

R W3 Discussion in Section 4.2:

To address the confusion regarding Section 4.2, we provide further clarification.

Section 4.2 aims to automate the evaluation of the effectiveness and robustness of detecting relative variations defined in AutoCV for generating process labels.

In Section 4.2.1, we automate the acquisition of ground truth by detecting errors specifically in mathematical calculations. This approach eliminates the need for manual process data ground truth. We successfully labeled mathematical calculation errors in the outputs of Llama2 for 7500 GSM8K training data samples. This process, as stated in line 195, "establishes a benchmark for PCH detection." To avoid potential misunderstandings, we will revise the term "hallucination in math reasoning" to "math calculation error" in the revised manuscript.

In Section 4.2.2, we show that our method performs well in detecting calculation errors, as evidenced by precision, accuracy, and F1 scores. As noted in line 209, "Table 5 demonstrates that our method using confidence variation effectively detects calculation errors." This validates our method as a reliable source of process supervision information, establishing an experimental basis for automating process annotations in Section 5.

We will include the clarification above in the revised manuscript for better presentation.

R W4 Small Gains with OSV + PSV compaed with OSV only:

It is important to highlight that these gains, while seemingly small, are statistically significant. We conducted statistical significance tests over five repetitions. Experimental results consistently produced p-values well below the significance level of 1e-6. For example, with Mixtral-Instruct, the t-statistic was -15.06 with a p-value of 1.09e-07 This statistical significance underscores that the improvements observed are not due to chance and validate the effectiveness of the proposed OSV + PSV approach.

Moreover, it is essential to consider the broader applicability and additional benefits that OSV + PSV provides. Unlike OSV-only, which is limited to settings with available ground truth, OSV + PSV can leverage both labeled and unlabeled data. This capability allows for the utilization of a more extensive and diverse dataset, which is particularly advantageous in real-world scenarios where labeled data may be scarce.

To illustrate the benefits, we generated 7K math problems following the Evol-Instruct method from WizardLM[1], which are generated by LLMs without gold solutions. Both MCTS and OSV-only training cannot leverage these unlabeled data. The results are as follows:

In this context, OSV+PSV (GSM8K) refers to the original AutoCV setting, while OSV+PSV (GSM8K+WizardLM) includes process annotations sourced from both GSM8K and WizardLM unlabeled questions. The addition of unlabeled data leads to noticeable improvements across all response generators. For instance, the performance of Mistral-Instruct improves from 61.18 (OSV) to 63.11 (OSV+PSV with GSM8K+WizardLM). These results further demonstrate the value of the AutoCV approach.

[1] WizardLM: Empowering Large Language Models to Follow Complex Instructions. ICLR 2024