Part 1 of Our Rebuttal

Dear Reviewer AjnZ,

Thank you for your thoughtful and detailed feedback. We have addressed each point carefully and believe our responses clarify the novel contributions and effectiveness of our proposed methods. We hope these clarifications and planned updates meet your concerns.

Comment:
The technical contributions of this paper are somewhat limited. For example, the introduction of a verifier has already been highlighted; Comparing the CoT reasoning with its formal version have also been explored.

Response:
Thank you for your valuable feedback. We acknowledge the prior work in verifiers within math reasoning. We respectfully clarify that the paper offers two primary contributions with unique insights:

1. Systematic Comparison of Verifier Training Methods: We agree that scaling inference-time compute and employing verifiers are existing approaches, yet the field lacks a comprehensive overview of verifier training techniques. Through our dataset and verifier comparison, we conclude that the SimPO method is the most effective approach among current methods, offering valuable insights and benchmarks for future verifier-building efforts.

   • Revision: We have revised the abstract to accurately position this contribution.

2. Novel CoTnPoT Verification Approach: We identify a key weakness of LLM-based verifiers: they often overlook subtle calculation errors and inconsistencies in math reasoning and struggle to verify highly abstract and structured code. To address these limitations, we propose a novel method called CoTnPoT, designed to make verification more comprehensive and robust. Our approach fundamentally differs from previous work [1] and [2], as we focus on translating between math and code rather than explicitly verifying language solutions in code format. Translation, in this context, is a more feasible task than direct verification, especially for complex math problems where verification can be challenging. For example, while the major experiments in [1] and [2] used simpler datasets like GSM8k, MultiArith, and MMLU, achieving a performance improvement of 63.69 -> 73.54 on MATH, our method achieves a notable improvement from 59.7 -> 76.9. Additionally, unlike prior studies focused mainly on math reasoning, our approach is effective in both math and code reasoning, demonstrating a broader application potential.

   • Revision: Lines 99-100 and 509-515 clarify the motivation and distinction of our approach, with the contribution statement revised accordingly.

[1] Don’t trust: Verify–grounding llm quantitative reasoning with autoformalization. ICLR 2024.

[2] Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. ICLR 2024.

---

Comment:
The improvement of collaborative verification is relatively marginal compared with SimPO.

Response:
Thank you for highlighting this point. We would like to emphasize that while SimPO is indeed effective, CoTnPoT contributes unique strengths that complement the overall verification process, especially in complex tasks.

For math reasoning, model-based verifiers are proficient at catching surface-level logical errors, such as operator misuse or numerical inaccuracies (see Figure 6). However, they often miss subtler mistakes, like calculation errors or deeper logical inconsistencies. For instance, the verifier frequently assigns a high score to expressions such as $3.5+2.5+4.5+1.5=13$, where the correct solution should yield 12.

Although the improvement from SimPO verifier is significant, CoTnPoT addresses a different category of error, providing a more thorough and nuanced approach to detecting LLM errors. By focusing on distinct error types, CoTnPoT enhances the robustness of the verification process and bolsters overall accuracy.

---

Comment:
Simply merging these models' knowledge, even without verification, could also enhance the reasoning performance of the LLMs.

Response:
We appreciate this insight and agree that knowledge distillation from multiple strong models could improve performance. However, our study finds two limitations with this approach:

Scaling training resources may lead to performance saturation, which we reference in [3], making it less effective than scaling inference resources.

Verifier training allows weak-to-strong generalization, enabling improved reasoning in weaker models. For instance, our experiments demonstrate that using data from models like Mistral, Phi3, and InternLM2-Math to train SimPO yields improvements in larger models, such as LLaMA-70B and Qwen2-72B, which traditional knowledge distillation cannot achieve.

Most importantly, we see verifier training and knowledge merging as orthogonal approaches that, when combined, could further enhance reasoning performance.

[3] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

---

(Connect to Part 2)

Part 2 of Our Rebuttal

Comment:
There is a notable performance imbalance between CoT reasoning and PoT reasoning.

Response:
Thank you for pointing out this discrepancy. Our method actually benefits from cases where PoT outperforms CoT. In these instances, the PoT solution acts as a strong indicator of correctness, supporting the collaborative verification process.

Our approach primarily focuses on verifying CoT solutions, regardless of whether CoT or PoT performs better on a given task. However, we recognize that PoT can be preferable in certain tasks, and for such cases, we suggest directly utilizing PoT and applying our verification method to code reasoning tasks, such as MBPP, as shown in Table 2.

---

Comment:
What is the performance improvement achieved by fine-tuning the base model on the generated dataset?

Response:
Thank you for this question. We are currently implementing this experiment and will present the results in the rebuttal as soon as they are available.

---

Comment:
How does the performance curve change with varying sizes of sampled solutions?

Response:
We appreciate this suggestion. We are currently working on adding this performance curve and will include the results in the rebuttal as soon as possible.

---

Comment:
It seems that CoTnPoT sacrifices some recall, which may degrade the upper bound of LLM reasoning performance. Are there any potential solutions to address this issue?

Response:
We acknowledge that CoTnPoT, by filtering solutions, can lead to a recall reduction. As a potential solution, we propose using CoTnPoT as a scoring mechanism rather than a strict filter. Instead of outright removal, each solution would be assigned a score based on CoTnPoT results (1 for passing, 0 for failing). By integrating these scores with SimPO verifier scores, we can retain more solutions and alleviate the recall degradation.

---

Respectfully,

Authors of ICLR 5271

Dear Reviewer R5WH,

Thank you for your insightful feedback, which has been valuable in refining our work. We look forward to further clarifying and enhancing our study based on our discussion!

---

Comment 1:

"Have the authors tried to compare their approach with verification based on say 50% of the test cases?"

Our Response:

We appreciate this thoughtful question and understand there may be a slight misunderstanding regarding our approach. Our verification process for coding tasks relies solely on the coding problem, the solution, and the CoTnPoT commentary, with no test cases utilized in the verification phase. Instead, test cases are employed to compute the final performance.

In our study, we evaluate the pass@1 rate in Table 3 to measure our verifier's ability at Best-of-N, highlighting the verifier’s effectiveness, in alignment with prior studies on MBPP and MBPP+.

---

Comment 2:

"What is the point of doing CoTnPoT for code? How to do?"

Our Response:

We apologize for any lack of clarity in describing CoTnPoT within the context of code reasoning. Here, we aim to clarify both the rationale and implementation for CoTnPoT in coding tasks:

1. Rationale for CoTnPoT in Code Reasoning:
   As noted in lines 226-229 of our paper, the abstract and structured nature of code often renders it challenging for verifiers to fully understand, leading to similar scores for different solutions. This similarity indicates difficulty in accurately identifying nuanced errors within the code. CoTnPoT aids in mitigating this issue by providing descriptive commentary to enhance interpretability and reduce scoring ambiguities for the verifier.

2. Implementation of CoTnPoT in Code Reasoning:
   We have revised Figure 2 to better illustrate how CoTnPoT functions across math and coding tasks. Additionally, Section 2.3 will be expanded to distinguish the application of CoTnPoT in these different domains.

   • CoTnPoT for Math: We sample multiple CoT solutions ($S_{CoT}$), translate each into PoT format ($S_{PoT}$), filter out any $S_{CoT}$ solutions if their answers don’t match $S_{PoT}$, and subsequently apply an LLM-based verifier on the remaining $S_{CoT}$.

   • CoTnPoT for Code: For coding tasks, we first sample multiple PoT solutions ($S_{PoT}$), then write a descriptive commentary ($S_{Des}$) based on $S_{PoT}$. This commentary is concatenated with $S_{PoT}$, providing a richer input for the LLM-based verifier to analyze.

---

Comment 3:

"Even so, it seems that SimPO is already very strong and CoTnPoT may not be necessary."

Our Response:

Thank you for highlighting this point. We would like to emphasize that while SimPO is indeed effective, CoTnPoT contributes unique strengths that complement the overall verification process, especially in complex tasks.

For math reasoning, model-based verifiers are proficient at catching surface-level logical errors, such as operator misuse or numerical inaccuracies (see Figure 6). However, they often miss subtler mistakes, like calculation errors or deeper logical inconsistencies. For instance, the verifier frequently assigns a high score to expressions such as $3.5+2.5+4.5+1.5=13$, where the correct solution should yield 12.

Although the improvement from SimPO verifier is significant, CoTnPoT addresses a different category of error, providing a more thorough and nuanced approach to detecting LLM errors. By focusing on distinct error types, CoTnPoT enhances the robustness of the verification process and bolsters overall accuracy.

---

Comment 4:

"CoTnPoT should be compared with Best-of-2N, from the perspective of computational resource."

Our Response:

We appreciate the reviewer’s suggestion regarding computational resources, and we acknowledge the importance of fair comparison. In response, we have initiated additional experiments to compare CoTnPoT with Best-of-2N and will share these results in our final rebuttal submission.

Best,

Authors of ICLR 5271

Dear Reviewer FHq1,

Thank you for your constructive feedback on our work. We appreciate the opportunity to clarify and enhance our paper based on your suggestions.

---

Comment:

The approach seems to be applicable generally for reasoning domains, but the authors only test on math reasoning. To what extent can the technique be applied to other domains?

Response: We agree that testing the technique across multiple reasoning domains could strengthen the generalizability claims of our approach. In addition to math reasoning, we have also evaluated our model on code reasoning tasks, which is another critical area within reasoning domains. As shown in Table 2, our approach demonstrates robust performance across both math and code reasoning, suggesting its applicability to diverse reasoning tasks.

---

Comment:

While there are improvements above the baselines, they vary dramatically between models.

Response: Our method is designed to develop stronger verifiers on top of existing reasoners, so the performance can indeed vary depending on the specific reasoner employed. However, when we use a fixed backbone reasoner, our method consistently achieves significant improvements over various baselines. This consistency is demonstrated in Table 3, where using the same reasoner shows reliable performance enhancements, highlighting the robustness of our verifier design.

---

Comment:

To what extent can the technique be applied to other domains?

Response: Thank you for this question. The verification approach outlined in our paper has significant potential for broader application in two key domains:

1. Output Selection and Verification:

LLMs face inherent limitations as auto-regressive generators, where they only have limited context during the early stages of generation. When an error or suboptimal strategy is introduced early in the output sequence, it becomes challenging for the model to produce a correct final response. Additionally, due to their stochastic nature, LLMs often generate multiple varied responses to the same input. Techniques like sample-then-rank or best-of-N strategies are therefore commonly used to improve the consistency and accuracy of outputs, as seen in models like GPT-o1. By using a verifier, which has access to the entire generated response upfront, it is possible to provide more precise feedback on sampled candidates, leading to more reliable output selection.

2. Synthetic Data Generation:

As sourcing high-quality training data from real-world resources becomes increasingly challenging, state-of-the-art models (e.g., LLaMA 3.2) are increasingly relying on synthetic data generated by LLMs. A critical aspect of this approach is maintaining the quality of generated data. The verifier method serves as an effective filtering tool to eliminate low-quality data, thus facilitating a more robust self-learning cycle and enhancing knowledge distillation and transfer for LLMs. By incorporating a verification mechanism, synthetic data quality can be better controlled, making it a valuable component in training pipelines for future LLMs.

---

Comment:

The paper could benefit from more analysis on what is learned by the verifiers, for example if there are certain types of solutions where the verifiers perform better?

Are there certain types of programs/solutions where the verifier is better at judging?

Response: Thank you for this insightful suggestion. Figure 6 in our paper illustrates examples where our verifier effectively detects reasoning errors in solutions, showing that the verifier is sensitive to errors involving incorrect use of numbers, operators, or knowledge points.

We have also included additional descriptions to explain that while model-based verifiers can detect surface-level logical errors, they are less sensitive to subtle calculation mistakes or deeper logical inconsistencies. For example, the verifier may score incorrect answers highly if they involve only minor calculation errors. As shown in Figure 6, our verifier sometimes assigns a high score to equations like $3.5 + 2.5 + 4.5 + 1.5 = 13$, even though the correct result should be 12. This limitation motivated the development of our CoTnPoT method, which aims to improve detection of these finer-grained mistakes.

Revision Details: The above points have been clarified in the updated manuscript on Lines 221–227.

---

We hope that these revisions and clarifications address your concerns. Thank you once again for your valuable feedback.

Best regards,

Authors of ICLR 5271

Thanks to the authors for their response. I maintain my current score.

Response: We agree that testing the technique across multiple reasoning domains could strengthen the generalizability claims of our approach. In addition to math reasoning, we have also evaluated our model on code reasoning tasks, which is another critical area within reasoning domains. As shown in Table 2, our approach demonstrates robust performance across both math and code reasoning, suggesting its applicability to diverse reasoning tasks.

While the authors point out their evaluation on code reasoning, as Reviewer R5WH points out, considering code reasoning without any test cases is somewhat unrealistic (especially as sample tests are readily available). Also, invoking the verifier requires additional queries, and the relevant baseline should be sampling more times. Here, evaluation with test cases is crucial.

Response: Thank you for this insightful suggestion. Figure 6 in our paper illustrates examples where our verifier effectively detects reasoning errors in solutions, showing that the verifier is sensitive to errors involving incorrect use of numbers, operators, or knowledge points.

We have also included additional descriptions to explain that while model-based verifiers can detect surface-level logical errors, they are less sensitive to subtle calculation mistakes or deeper logical inconsistencies. For example, the verifier may score incorrect answers highly if they involve only minor calculation errors. As shown in Figure 6, our verifier sometimes assigns a high score to equations like (3.5 + 2.5 + 4.5 + 1.5 = 13), even though the correct result should be 12. This limitation motivated the development of our CoTnPoT method, which aims to improve detection of these finer-grained mistakes.

Actually, I was looking for a larger-scale analysis of the verifier. This is only one problem, so it may not be representative of the rest of the dataset. Knowing when the verifier is and is not effective seems to be a crucial step to the success of methods like these.

Part 1 of our Rebuttal

Dear Reviewer dNMD,

Thank you very much for the detailed and thoughtful review, which has been invaluable in improving our work. We are fully committed to addressing each of your concerns and clarifying any areas that may have lacked precision in our initial submission. Should you have any further questions or additional concerns, please feel free to reach out to us. We would be more than happy to engage in further discussion to ensure our contributions are presented as clearly and accurately as possible. Our responses:

---

Comment:

"The paper lacks significant novelty; the primary differences lie in the volume of data, variety of sources, and minor variations in training."

Response: We acknowledge that certain statements may have overclaimed novelty in our draft. However, we respectfully clarify that the paper offers two primary contributions with unique insights:

1. Systematic Comparison of Verifier Training Methods: We agree that scaling inference-time compute and employing verifiers are existing approaches, yet the field lacks a comprehensive overview of verifier training techniques. Through our dataset and verifier comparison, we conclude that the SimPO method is the most effective approach among current methods, offering valuable insights and benchmarks for future verifier-building efforts.

   • Revision: We have revised the abstract to accurately position this contribution.

2. Novel CoTnPoT Verification Approach: We identify a key weakness of LLM-based verifiers: they often overlook subtle calculation errors and inconsistencies in math reasoning and struggle to verify highly abstract and structured code. To address these limitations, we propose a novel method called CoTnPoT, designed to make verification more comprehensive and robust. Our approach fundamentally differs from previous work [1] and [2], as we focus on translating between math and code rather than explicitly verifying language solutions in code format. Translation, in this context, is a more feasible task than direct verification, especially for complex math problems where verification can be challenging. For example, while the major experiments in [1] and [2] used simpler datasets like GSM8k, MultiArith, and MMLU, achieving a performance improvement of 63.69 -> 73.54 on MATH, our method achieves a notable improvement from 59.7 -> 76.9. Additionally, unlike prior studies focused mainly on math reasoning, our approach is effective in both math and code reasoning, demonstrating a broader application potential.

   • Revision: Lines 99-100 and 509-515 clarify the motivation and distinction of our approach, with the contribution statement revised accordingly.

[1] Don’t trust: Verify–grounding llm quantitative reasoning with autoformalization. ICLR 2024.

[2] Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. ICLR 2024.

---

Comment:

"Overclaimed contributions, such as 'To address this, we scale up inference-time computation…' and 'We develop inference-time verifiers to enhance the reasoning capabilities of LLMs.'"

Response: We acknowledge these inaccuracies and have revised the statements to reflect our approach more accurately:

1. Revised to: “To address this, we adopt a widely used method to scale up inference-time computation by generating multiple reasoning paths and employing verifiers to assess and rank the generated outputs by correctness.” (Lines 17-18)
2. This item has been removed from the contributions list.

---

Comment:

"CoTnPoT should be compared with Best-of-2N to ensure fair use of computational resources."

Response: We appreciate this suggestion. We are currently conducting additional experiments to compare CoTnPoT with the Best-of-2N approach. Results will be included in the rebuttal once the experiments are complete.

---

Comment:

"Clarify the distinction between CoTnPoT in math vs. coding tasks."

Response: We have clarified this distinction by updating Figure 2 and providing a detailed description in Section 2.3.

• CoTnPoT for Math: Generate multiple CoT solutions $S_{CoT}$, translate each to PoT solutions $S_{PoT}$, and filter out $S_{CoT}$ if its answer does not match $S_{PoT}$, followed by LLM-based verification.

• CoTnPoT for Code: Generate multiple PoT solutions $S_{PoT}$, generate comments $S_{Des}$ based on $S_{PoT}$, concatenate $S_{PoT}$ and $S_{Des}$, and apply LLM-based verification.

  • Revision: Figure 2 and Lines 256-274 have been updated to clarify this distinction.

---

(connect to Part 2)

Part 2 of our Rebuttal

Comment:

"Grammar issues on line 219 (preference-tuning) and line 128 ('surpassing the performance of existing verifiers')."

Response: We have corrected these grammatical issues:

• Line 219: Updated “reference-tuning” to “preference-tuning.”
• Line 128: Revised wording to improve clarity.

---

Comment:

"Inaccurate terminology: 'recall rate' in Figure 1 should be 'Pass@64'; 'sample-then-rank' should be 'best-of-N.'"

Response: We appreciate this feedback and have revised terminology for clarity:

• Revisions: Lines 60-61 and 75-76 now use the terms “Pass@64” and “best-of-N,” respectively.

---

Comment:

"Why was MAmmoTH-7B-plus used for math reasoning verification but not in previous sections?"

Response: We chose MAmmoTH-7B-plus for two reasons:

1. Enhanced performance with an upgraded model.
2. Demonstrates the generalizability of the training method across different models. We acknowledge that this could be confusing but believe it does not affect the paper’s conclusions.
   • Revision: We expanded this explanation in Lines 370-374.

---

Comment:

"Figure 2 is misleading in its comparison."

Response: We acknowledge that Figure 2 could cause misunderstanding. To improve clarity, we have removed Figure 2, and we emphasize the clearer comparisons in Table 3.

• Revision: Removed Figure 2, referencing clearer comparisons in Table 3.

---

Comment:

"Contradiction in reducing reliance on external LLMs but using them for math reasoning."

Response: Apologies for the confusion. Our aim is to minimize reliance on external models in coding verification by using the same LLMs to write comments. However, many math models lack the capability to translate CoT to PoT, necessitating the use of external LLMs.

• Revision: Further clarified in Lines 246-247.

---

Comment:

"Inconsistent number of candidate solutions in Figure 4."

Response: We accept this suggestion and have standardized the number of candidate solutions in Figure 4.

• Revision: Updated Figure 4 to show consistent testing with 100 candidate solutions.

---

Comment:

"Generalizability of the verifier, line 82."

Response: Previous verifiers were often trained on outputs from a single model, making them specific to particular solution formats. In contrast, our verifiers are trained on diverse backbones, enhancing generalizability. We realize that our original statement may have caused confusion, so we have removed it for clarity.

• Revision: Adjustments made on Lines 81-83.

---

Best regards,

Authors of ICLR 5271