Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-Thought Critic

Thank you for reviewing our paper and offering constructive feedback. We have prepared detailed responses to address each point of your concern.

The experimental results seem not very satisfactory.

• On the one hand, Iterative Refinement could be more challenging for LLMs. To work properly, it requires the model to

  1. Detect errors in its output,
  2. Refine the mistakes,
  3. Exit if no further errors are detected.

  This pipeline could be more sensitive to error accumulation. Moreover, it only edits on a single example and has a limited retry, which is sample-efficient, but may not explore the solution space more actively, as self-consistency does.

• On the other hand, self-consistency is a strong baseline, as it requires massive sampling. It leverages diverse reasoning paths and tries to mitigate the stochastic of a single sample. But under the method of Critic As Filter, we actively filter out problematic attempts and perform the majority vote on the more reasonable candidates, rather than equally account for all the predicted answers as the vanilla majority vote does, which further increases performance. This in turn demonstrates our model's strong ability to critique.

About the ability of self-reflection

• Besides the main results, through out-of-domain evaluation in Section 4.4 and Table 4, we find our model demonstrates generalized ability to critique and refine.
• In terms of LLM's self-reflection ability, currently it's still an open question, and we hope our work as a valuable exploration could shed light on future studies in this area.
• On the other hand, as long as we adequately improve the models' ability to critique, we could achieve test-time performance increase in the form of "self-reflection". As mentioned in our paper, after Critic-CoT training, the ability to critique and generate both improves, but they are not exactly identical. Notably, the critique ability can surpass the task-solving ability, allowing the model to detect errors even when it has a low probability of generating a valid solution, as the papers [1][2] also suggest. This indicates that by strengthening the model’s CoT critique ability beyond its generation capability, we can leverage this discriminative power to reject imperfect responses and achieve positive performance gains.

[1] Self-critiquing models for assisting human evaluators

[2] CriticBench: Benchmarking LLMs for Critique-Correct Reasoning

Data open-source

Due to the large scale of our training data, we didn't upload the dataset yet. We plan to first conduct further verification of the training data quality, and then open source it after the paper is accepted.

Missing reference

Thank you for pointing out the missing references. We have cited the paper "Large language models cannot self-correct reasoning yet" in Section 2.2, and we would cite the paper "Self-contrast" and several other papers in the revision.

Terms of "Distant Supervision"

In information extraction, "distant supervision" means automatically generating large-scale weakly-labeled data, with the help of an existing database. And here in our paper, we try to leverage this term, since our Critic-CoT training data is also automatically constructed without human annotation. Several other works also use the term "distant supervision" in other domains beyond information extraction:

• Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systems, EMNLP 2024 Findings
• Learning To Recognize Procedural Activities with Distant Supervision, CVPR 2022
• SciNLI: A Corpus for Natural Language Inference on Scientific Text, ACL 2022
• TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios
• LiteSearch: Efficacious Tree Search for LLMS

Sorry for the confusion it brings. For better precision, we would consider using the term "weak supervision" to better describe our data construction process.

In conclusion, We hope that our responses provided address your primary concerns. If you have any additional questions or concerns, please do not hesitate to reach out.

Thank you for the feedback, and sorry for the confusion. We would like to make further explanation as follows.

1. Regarding distant supervision

The dataset used by the author to generate the critique corpus already has accurate answer annotations (Line187-200)

• The dataset we use, GSM8K and MATH, has only the ground truth of the final answer, typically a number or an expression. By comparing the prediction answer and ground truth answer, we can only determine whether the entire multi-step attempt is correct or not at the outcome level, while at the process level, the final answer do not provide granular information about the correctness of intermediate reasoning steps, thus the exact correctness of each step remains unknown. Therefore, we need human annotation or an automatic process to label these steps.
• Previous work [1] manually annotated the intermediate steps, which is resource-intensive and costly. In our work, we automate the data construction process, which weakly supervises the critique (identifying which steps are wrong and why) and refinement, and replace the human with models like GPT-4-Turbo, which is more cost-friendly and scalable. Since our approach does not rely on precise human annotations at the intermediate level, we categorize it as weak supervision.

[1] Let's Verify Step by Step, ICLR 2024

2. Regarding the experiment

I don't quite understand what the author claimed in their reply that it is more efficient than self-consistency (i.e., sample-efficient).

There might be some misunderstandings regarding our experimental setups. Sorry for the confusion, and we would like to clarify our setting of "Iterative Refinement", "Self Consistency", "Critic As Filter" as follows.

• In Self Consistency, we generate N=96 samples, and pick the most frequent answers.
• In Critic As Filter, based on the N=96 samples, we additionally generate N=96 critiques, and in total 192 invocations. Then, we filter out erroneous attempts predicted by critique and keep more reasonable candidates, and select the most frequent answer among these candidates.
• In Iterative Refinement, we follow previous works of "self-correct", and only sample one attempt first, then generate one corresponding critique: if it's predicted to be correct, then exit and output the answer, if not, generate one refinement, and critic it again. Compared with Self Consistency which generates a fixed number of N=96 samples, we believe Iterative Refinement is more sample-efficient than Self Consistency.

To further address your concern, we further conduct statistical analysis regarding Iterative Refinement on the dataset of GSM8K, and verify its sample-efficiency:

• Invocations of Self-Consistency on GSM8K: 1319 * 96 = 126,624
• Invocations of Iterative Refinement on GSM8K: among 1319 test cases, our Critic-CoT model predicts 274 problematic instances and iterates 1627 times (on average 5.94 rounds for each wrong case), which makes in total 1319 * 2 + 1627 * 2 = 5892 invocations, which is 21.5 times less than the number of calls of self-consistency.

We hope our response would clarify the details and make it more comprehensive. If you have further questions, please do not hesitate to inform us.

Thank you for reviewing our paper and offering constructive feedback. We have prepared detailed responses to address each point of your concern.

W1&W3: Based on the description, it seems that CoT for the critique (e.g., those in Figure 1) is not well used.

Whether CoT is beneficial for critique is indeed worth investigating. In Section 4.5 and Table 5, we have conducted the ablation study with the two settings, and demonstrate the necessity of CoT:

1. Process Label: remove the CoT critique and only let the model predict if the step is correct
2. Outcome Label: let the model directly predict the whole attempt is correct

We find that the accuracy of critique drops. On GSM8K, our Critic-CoT model is 92.3%, while the Process Label without CoT is 89.5%, and the Outcome Label is 88.0%. Similarly, on MATH, the critique accuracy drops from 72.2% to 63.8% with Process Label, and further 63.0% with Outcome Label.

This indicates the effectiveness of CoT for critique. Also, we hope these two baselines could serve as a NL version of approximal PRM / ORM. We would refine our presentation and more explicitly justify the necessity of CoT for critique.

W2: it is somehow unclear the gains from in-domain data augmentation or from the critic-cot.

As discussed in Sections 4.3 and 4.5 of our paper, the improvements can be attributed to two key factors: training with critique and refinement, and application of critique and refinement at test time.

• Strengthening the ability to critique and refinement would not compromise the task-solving performance, but improve it. Therefore, our critic model already exhibits noticeable performance improvements in task-solving, even in the absence of additional critique steps during the inference phase.
• At the inference phase, we can actively leverage the model’s ability to reflect on its reasoning and correct mistakes via Critic As Filter and Iterative Refinement, which leads to additional improvements.

Q1: How to define one step?

For task-solving, we instruct the model to present the attempt in a step-by-step manner, and the model would decide how long a step is, usually a sentence or a short paragraph. For example, in Figure 3, "Let the three consecutive integers be x-1, x, and x+1. The sum of these three integers is 3x." is a step.

Q2: How to identify pred from the attempt?

In the original MATH dataset, they wrap the final answer with "\boxed{answer}". As the prompts in Appendix A.6 demonstrate, we let the model print the answer in the format "\boxed{answer}", then the model would write down the answer in this latex expression. We extract the regular expression \boxed{.*} from the model output, and obtain the valid answer expression with matched parenthesis. We would add Python code for answer extraction in the Appendix in revision.

import re

def extract_boxed_expressions_custom(text):
    stack = []
    current_expr = ""
    i = 0
    while i < len(text):
        if text[i:i+7] == r"\boxed{":
            if stack:
                current_expr += text[i]
            stack.append("{")
            i += 7
        elif text[i] == "{" and stack:
            stack.append("{")
            current_expr += text[i]
            i += 1
        elif text[i] == "}" and stack:
            stack.pop()
            if stack:
                current_expr += text[i]
            else:
                current_expr = re.split('=', current_expr)[-1]
                return current_expr
            i += 1
        elif stack:
            current_expr += text[i]
            i += 1
        else:
            i += 1
    return ''

def extract_answer(text):
    predict_answer_span = re.findall(r'(\\boxed\{.*\})', text, re.DOTALL)[-1]
    predict_answer = extract_boxed_expressions_custom(predict_answer_span)
    return predict_answer

Q3: Could the authors manually check tens of examples and provide the percentage of instances with perfect CoT reasoning?

• To address your concern, we each sample 100 correct answer solutions, on GSM8K by GPT-3.5-Turbo and GPT-4-Turbo, and MATH by GPT-4-Turbo, then manually check if all intermediate steps are correct.
  • The results are demonstrated in the following table. We find that in general, the assumption holds: Despite the presence of a small amount of noise, the data generated in this manner can simultaneously ensure both quality and scalability. Also, from GSM8K to MATH, as the reasoning traces become longer and more complicated, the percentage of correct answer but with wrong intermediate steps increases.

    Data	Model	Intermediate Accuracy of Correct Answer Attempt
    GSM8K	GPT-3.5-Turbo	97%
    GSM8K	GPT-4-Turbo	99%
    MATH	GPT-4-Turbo	93%

In conclusion, we hope that our responses address your concerns, and we will revise our manuscript accordingly. Please do not hesitate to inform us of any additional concerns or questions you may have.

Dear Reviewer,

As the rebuttal phase nears its conclusion, we have diligently provided detailed responses to address your previous concerns. We hope that you would review our replies and identify any remaining concerns or additional questions you might have. We recognize your busy schedule, but if possible, we kindly request that you share any further inquiries or feedback you may have. We are eager to further discuss and clarify any points as needed.

Thank you very much for your valuable time and assistance.

Best Regards,

The Authors

Dear Reviewer B7Pw:

Thank you again for the insightful feedback. We would like to take this opportunity to briefly revisit and clarify the key concerns you raised. Specifically,

• The role of CoT in critique and the comparison between PRM and ORM: In Section 4.5 (Line 479-518) and Table 5, we have conducted an ablation study comparing the NL versions of "PRM" and "ORM." and the results confirm the effectiveness of stepwise CoT critique.
• Source of improvement: We now add a discussion in Appendix A.2.2 (Lines 895-905) regarding the factors contributing to performance improvement. The two factors include:
  • Critic-CoT training
  • The inference methods, namely Critic As Filter and Iterative Refinement during test-time.
• Process Correctness of Correct Answer Attempt: We now add the manual evaluation of the intermediate steps correctness of attempts with the correct answer in Appendix A.3 (Line 945-950) and Table 7. We find that in general, the correct final answer is a strong indicator of accurate CoT reasoning.
• Identifying the prediction answer: We extract the predicted answer using the pattern \boxed{.*} from the model’s output. We now add the details and the corresponding Python code in Appendix A.4 and Table 9.
• Definition of a step: As mentioned in our previous response,

For task-solving, we instruct the model to present the attempt in a step-by-step manner, and the model would decide how long a step is, usually a sentence or a short paragraph. For example, in Figure 3, "Let the three consecutive integers be x-1, x, and x+1. The sum of these three integers is 3x." is a step.

As the rebuttal period is drawing to a close, we would appreciate your feedback on whether our responses sufficiently address your concerns. If there are any remaining issues, we would be happy to provide further clarifications before the end of the rebuttal period. Thank you once again for your time and consideration.

Thank you for the thoughtful and constructive feedback. We appreciate the opportunity to clarify and address your concerns. Below, we provide detailed responses to each of your points.

W2&Q1: The quality of critique data is crucial to train a valid NL critique at step-level.

• A: The assumption that the final answer is correct, then all intermediate steps are correct
  • To address your concern, we each sample 100 correct answer solutions, on GSM8K by GPT-3.5-Turbo and GPT-4-Turbo, and MATH by GPT-4-Turbo, then manually check if all intermediate steps are correct.
  • The results are demonstrated in the following table. We find that in general, the assumption holds: Despite the presence of a small amount of noise, the data generated in this manner can simultaneously ensure both quality and scalability. Also, from GSM8K to MATH, as the reasoning traces become longer and more complicated, the percentage of correct answer but with wrong intermediate steps increases.

    Data	Model	Intermediate Accuracy of Correct Answer Attempt
    GSM8K	GPT-3.5-Turbo	97%
    GSM8K	GPT-4-Turbo	99%
    MATH	GPT-4-Turbo	93%

• B: All Att and Cri are from sampling, that randomness will also introduce some errors, those generated data should be also evaluated and reported in paper.
  • To further address your concern, we each sampled 100 entries (50 with the correct answer and 50 with the wrong answer) from the critic-cot data on GSM8K and MATH, and conducted a manual verification to verify the accuracy of the step-wise critiques.
  • For the critique of the correct answer attempt, it is valid if there is indeed no error in all the intermediate steps; for the critique of the wrong attempt, it is valid if the first error step and the reason for the error are both identified. A refinement is correct, if the continuation steps are flawless.
  • The results of manual verification are demonstrated in the table, and we can see that the data we automatically constructed maintains a high level of accuracy at the step level, which can well support the critique training process. | Data | Critique of Wrong Attempt | Refinement of Wrong Attempt | Critique of Correct Answer Attempt | | --- | --- | --- | --- | | GSM8K | 86% | 96% | 100% | | MATH | 84% | 96% | 94% |
• Overall, our constructed training data, despite containing a small amount of noise, could effectively enhance the model’s critique capabilities, as evidenced by our experimental results.
• Thanks for your valuable advice, we would conduct a more rigorous and comprehensive human validation experiment, and the corresponding results will be included in the revision.

Q2: The Iterative Refine improvements on MATH task without re-ranking are not significant, thus, I’m not convinced self-critique refinement with single pass is a valid approach.

Could you clarify further on the meaning of "self-critique refinement with single pass"? Does it refer to our data construction process?

If so, during the training data construction, single-pass critique refinement with retries is applicable for generating training data. By leveraging the ground truth answers as an effective filter, we can sample multiple single-pass critique-refinement pairs for each problem until success. Given the large size of problem in our dataset and there are multiple attempts at each problem, we can afford to bypass instances where valid critique-refinement pairs are not produced, and still obtain a sufficient amount of data overall.

W1: “Self-critique” ability used in this paper is distilled from a stronger model (e.g. GPT4-Turbo).

Our training data of step-wise CoT critique is indeed constructed by GPT-4-Turbo via weak supervision, which is to teach the model to critique and refine its reasoning traces. In fact, additional supervision signal is necessary due to the inadequate ability of vanilla LLMs to "self-critique". Previous studies like [1] have demonstrated the performance of vanilla LLMs even degrades after self-correlation. Moreover, compared to constructing data via human annotation, we automate the data construction process and replace the human with models like GPT-4-Turbo, which is more cost-friendly and scalable.

[1] LARGE LANGUAGE MODELS CANNOT SELF-CORRECT REASONING YET. ICLR 2024.

W3: It’s not clear to me where the main improvements in Table 1-3 come from, it probability comes from GPT4-Turbo annotations (supervision).

As discussed in Sections 4.3 and 4.5 of our paper, the improvements can be attributed to two key factors: training with critique and refinement, and application of critique and refinement at test time.

• Strengthening the ability to critique and refinement would not compromise the task-solving performance, but improve it. Therefore, our critic model already exhibits noticeable performance improvements in task-solving, even in the absence of additional critique steps during the inference phase.
• At the inference phase, we can actively leverage the model’s ability to reflect on its reasoning and correct mistakes via Critic As Filter and Iterative Refinement, which leads to additional improvements.

Q3: It looks like self-critique iterative refine works in the re-ranking setting, this suggests that the self-critique accuracy is not high, we need some analysis in detail.

• We would like to argue that, our model under the Critic As Filter Setting demonstrates performance increase with a large margin, which indicates the high accuracy of critique, since the model can effectively detect erroneous attempts and filter out more reasonable candidates with a higher possibility of correctness. Additionally, as demonstrated in Table 5, on GSM8K our model has the critique accuracy of 92.3%, and 72.2% on the more challenging MATH dataset.
• Meanwhile, in MATH dataset, even for GPT-4-Turbo, it's still difficult to refine successfully during training data construction. As mentioned in Section 4.2, we provide the reference solution to reduce intensive retry. In this setting, GPT-4-Turbo has only 78.3% accuracy on first-try refinement, 89.8% success rate under 16 retry, and on average 2.0 retry. If we remove the reference solution, GPT-4-Turbo performs much worse, with only 18.6% accuracy on first try refinement, 51.2% success rate under 16 retry, and on average 10.2 retry, which is unacceptable. Therefore, the inherent limitation on model's refine ability constraints the iterative performance on MATH dataset.
• Also, as the paper [2] suggests, detecting errors could be easier than making refinements on the already sub-optimal attempts. Therefore, while refinement on hard problems may have a lower success rate, as the condition of critique is easier than generation holds, it could be more feasible to generate a batch of attempts and filter by critique. Thus, in MATH, we see Critic As Filter outperforms Iterative Refinement. We also have provided a closer inspection of Critic As Filter and Iterative Refinement in Appendix A.1.1, A.1.2, and A.1.3.

[2] Self-critiquing models for assisting human evaluators

In conclusion, we hope that our responses address your concerns, and we will revise our manuscript accordingly. Please do not hesitate to inform us of any additional concerns or questions you may have.

Dear Reviewer,

As the rebuttal phase nears its conclusion, we have diligently provided detailed responses to address your previous concerns. We hope that you would review our replies and identify any remaining concerns or additional questions you might have. We recognize your busy schedule, but if possible, we kindly request that you share any further inquiries or feedback you may have. We are eager to further discuss and clarify any points as needed.

Thank you very much for your valuable time and assistance.

Best Regards,

The Authors

Besides, I expect more experiments on broad reasoning tasks beside math

In fact, as shown in table 4, we have evaluated two out-of-domain tasks, StrategyQA and AGIEval, and the corresponding experiments demonstrate the reasoning capability of our model is generalizable. StrategyQA is an implicit multi-hop reasoning task constructed from Wikipedia, and AGIEval contains standardized exam questions from law, english, logic, and math. Our model, especially the one trained on MATH, obtained positive performance gain on the two datasets: on StrategyQA, the accuracy increased from 76.2% (vanilla Llama-3-70B) to 80.1% through iterative refinement, and on AGIEval, the accuracy increased from 56.6% to 63.7%. This indicates the generalization beyond the training domain of math.

improvements are limited for both GSM8K and MATH.

Through the training of critic-cot, the performance of GSM8K rises from 89.6% (vanilla Llama-3-70B) to 93.3% through iterative refinement, and similarly from 51.1% to 56.6% on MATH. We believe this performance improvement to be significant. As discussed in Sections 4.3 and 4.5 of our paper, the improvements can be attributed to two key factors:

• Strengthening the ability to critique and refinement would not compromise the task-solving performance, but improve it. Therefore, our critic model already exhibits noticeable performance improvements in task-solving, even in the absence of additional critique steps during the inference phase.
• At the inference phase, we can actively leverage the model’s ability to reflect on its reasoning and correct mistakes via Critic As Filter and Iterative Refinement, which leads to additional improvements.

Furthermore, we will revise the result tables to clearly emphasize these performance gains.

In conclusion, we hope our responses address your concerns regarding contribution, critique quality, generalization, and observed improvements. We are committed to revising our manuscript to incorporate these clarifications and enhancements. Please do not hesitate to share additional feedback or questions.

Thank you for reviewing our paper and offering constructive feedback. We have prepared detailed responses to address each point of your concern.

At least for me, the novelty of the proposed method is limited, as it is pretty similar to existing self-reflection-based approaches.

We would like to restate the distinction between our study and existing research, as well as highlight our contributions:

• As we mentioned in the introduction (L42-L52), existing self-reflection works exhibit significant limitations.
  • They are often overly simplistic, relying on basic prompts to directly point out the error, which do not incorporate deliberate step-by-step examination in completely Chain-of-Thought reasoning.
  • They lack a targeted training process, and the ability of the vanilla LLMs to critique and refine often falls short of the demands required by complex reasoning tasks.
  • Therefore, these limitations significantly diminish the effectiveness of self-critic and, further, self-correct. In fact, studies like [1] have demonstrated the performance of vanilla LLMs even degrades after self-correlation.
• To address the above issues, we introduce the stepwise CoT critique format and corresponding automatic data construction framework, which is able to strengthen the model's System-2-like critic and refine abilities. Experiments also demonstrate that our approach can consistently outperform baselines without critic-CoT. Moreover, we investigate the intrinsic correlation between critique and task-solving abilities within LLMs, discovering that these abilities can mutually reinforce each other rather than conflict.
• In summary, we believe our framework and findings not only differentiate our work from existing methods but also shed light on future research in this area.

[1] LARGE LANGUAGE MODELS CANNOT SELF-CORRECT REASONING YET. ICLR 2024.

In particular, the proposed method does not guarantee that the generated critic is reasonable, as long as the refined output (taking the generated critic as additional inputs) contains the correct answer.

We acknowledge the importance of ensuring the quality of critiques in our framework. Given the large scale of the dataset and the high cost of intensive human annotation required for examining stepwise critiques, we currently employ weak supervision to filter and obtain more reasonable critique data while ensuring scalability. Although the process may introduce a small amount of noise, the quality of our constructed data effectively injects the model with the ability to critique and refine on a step-wise basis, thereby enhancing reasoning performance.

To further address your concern, we each sampled 100 entries (50 with the correct answer and 50 with the wrong answer) from the critic-cot data based on GSM8K and MATH, and conducted a manual verification to verify the accuracy of the step-wise critiques.

• For the critique of the correct answer attempt, it is valid if there is indeed no error in all the intermediate steps; for the critique of the wrong attempt, it is valid if the first error step and the reason for the error are both identified. A refinement is correct, if the continuation steps are flawless.
• The results of manual verification are demonstrated in the table, and we can clearly see that the data we automatically constructed maintains a high level of accuracy at the step level, which can well support the critique training process. Thanks for your valuable advice, we would conduct a more rigorous and comprehensive human validation experiment, and the corresponding results will be included in the revision. | Data | Critique of Wrong Attempt | Refinement of Wrong Attempt | Critique of Correct Answer Attempt | | --- | --- | --- | --- | | GSM8K | 86% | 96% | 100% | | MATH | 84% | 96% | 94% |

Thanks to the author for his efforts to clarify. Generally speaking, I like this idea of critic-cot. Hoping the author can include more reasoning-related benchmarks later, such as code.