Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoning

W3: I'm not sure why it is stressed that CoSC can work with zero shot as opposed to proprietary models that require few shots.

Thanks for your careful reading and bringing this to our attention. Our emphasis on CoSC’s zero-shot capability was intended to highlight the practical usability of the model in real-world applications, where providing task-specific examples might not always be feasible.

We agree that proprietary models, being more general-purpose and not fine-tuned for these specific tasks, require few-shot examples to adapt to this reasoning paradigm. This distinction underscores the trade-off between model generality and task-specific optimization.

---

W4: It can be misleading to say it outperforms "all the advanced proprietrary LLMs" as that covers ALL proprietary LLMs including the multimodal ones. It should be "all non-multi-modal proprietrary LLMs". (BTW, I am not sure why multi-modality distinction actually matters here?)

Thank you for your thoughtful feedback. We have revised the wording in our revised paper from "proprietary LLMs" to "non-multi-modal proprietary LLMs".

Regarding the distinction, while multi-modal LLMs process questions through language, their visual capabilities can enhance understanding of spatial or geometric concepts, which may improve mathematical reasoning in tasks like Geometry. Despite this, our CoSC models, trained solely on language, outperform some multi-modal models in specific mathematical benchmarks, demonstrating the effectiveness of our chain of self-correction approach in improving mathematical reasoning.

---

W5: Can you highlight in bold the best performing proprietary models in Table 2 (which I think is GPT4o and Claude 3.5 Sonnet).

Thank you for your valuable suggestion. We have updated Table 2 to highlight the best-performing proprietary models in bold in our revised paper. For the MATH dataset, GPT-4o is the best-performing proprietary model, while for the GSM8K dataset, the top-performing proprietary model is Claude-3.5 Sonnet.

---

W6: Typos.

Thank you for your careful review. We have corrected these typos in the revised version of our paper.

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1 & Q1: Novelty is not clear. Please explain in detail what are the important differences in technique of your approach and ToRA.

Thank you for your valuable comment. We would like to clarify that ToRA is also fine-tuned on data generated by GPT-4, rather than utilizing a prompting method, as mentioned in Line 86-87 and Line 437-438 of the original paper.

Although both our method and ToRA are fine-tuning approaches, there are fundamental differences in their underlying principles and functionalities. ToRA integrates Chain-of-Thought (CoT) and Program-of-Thought (PoT) strategies to enhance reasoning capabilities in mathematical problem-solving. However, it is primarily limited to single-round reasoning and lacks self-correction capabilities. Notably, while the design of ToRA may imply multi-round reasoning, it does not explicitly support multi-round inference. The reasoning process in ToRA typically involves only a single round, with additional iterations occurring only in extremely rare cases, as demonstrated in Table 4, where the number of such iterations is nearly zero.

In contrast, our method incorporates self-correction as an intrinsic feature of LLMs and is explicitly designed to support multi-round reasoning. This enables the model to iteratively refine its responses, correcting errors and improving accuracy over several round of reasoning. These advancements contribute to a more robust, iterative approach to problem-solving. Experimental results clearly show that our method outperforms ToRA by a clear margin across all three model sizes (7B, 13B, and 34B) on both the MATH and GSM8K mathematical benchmark datasets. This underscores the superior effectiveness of our approach in enhancing mathematical reasoning capabilities.

---

W2 & Q2: Evaluation results seem not to show very drastic gains. Can you compare a prompt-only version of your approach on the best proprietary models and show improvements over them? Will it improve upon their results further if you use GPT-4o to generate training data?

Thank you for your insightful comments. Regarding the performance gap between our method and proprietary models such as GPT-4o and Claude 3.5 Sonnet, we acknowledge that there is notable room for improvement. However, it is important to note that proprietary models operate on an entirely different scale in terms of training data volume, computational resources, and architectural design. This makes direct comparisons inherently challenging. Our primary goal of this paper has been to advance the capabilities of open-source models using feasible computational resources, rather than directly competing with state-of-the-art proprietary systems.

As suggested by the reviewer, we conducted an additional experiment comparing a prompt-only version of our CoSC approach on GPT-4o (one of the top proprietary models) with the original GPT-4o on the MATH and GSM8K datasets. The table below shows our results alongside the official GPT-4o benchmarks on these datasets. Our results surpass the official GPT-4o performance on both datasets, suggesting that self-correction-style reasoning is a broadly effective strategy for enhancing the reasoning capabilities of LLMs.

[1] https://openai.com/index/hello-gpt-4o/.
[2] https://ai.meta.com/blog/meta-llama-3-1/.

Finally, while CoSC prompts can enhance proprietary models, this approach relies on closed systems that are inaccessible to the broader research community. In contrast, our CoSC method is an open-source framework, providing self-correction capabilities that are publicly available. We believe this represents a critical step toward democratizing advanced reasoning capabilities and fostering further innovation within the research community.

Q2: I think to make this clasim it should be shown that prompting vs fine tuning on the same model would produce such a huge difference (e.g. the prompting approach on your open source models performs so much worse than your fine-tuned versions). However, the marginal improvements with respect to SoTA will still remain.

Thank you for your valuable feedback. We conducted experiments comparing the performance of the prompt-based version and the fine-tuning version of our CoSC approach on the same base LLM, CodeLLaMA. The results are shown in the table below. We observed that the fine-tuning version of CoSC significantly outperforms the prompting version by 28.9% and 37.9% in accuracy on the MATH and GSM8K datasets, respectively. These results clearly demonstrate that fine-tuning is far more effective than prompting when embedding self-correction capabilities into LLMs.

For context, Self-Refine [2] achieves only a 0%-0.2% accuracy improvement on the GSM8K dataset over the baseline, as shown in Table 1 of their original paper. CSV [1], which uses natural language-based self-verification without relying on code, even results in a 0.4% accuracy decrease on the MATH dataset compared to the baseline, as shown in Table 4 of the original CSV paper [1]. It is worth noting that the MATH dataset is a more challenging benchmark compared to GSM8K. However, our CoSC approach can achieve inherent self-correction capabilities without relying on any external tools and improve accuracy by 3%, 2.2%, and 2.7% over ToRA for the 7B, 13B, and 34B model sizes on the MATH dataset, respectively. Compared to the minimal or negligible improvements seen in CSV [1] and Self-Refine [2], we argue that the performance gains achieved by our CoSC approach are particularly significant, especially considering the challenging nature of the MATH dataset.

Thank you for your feedback! Below we respond to the follow-up questions.

---

Q1: The notion of self-correction with respect to code and mathematical reasoning is becoming very prevalent in current research, so it is difficult to clearly separate the novelty of your technique.

Thank you for your insightful comment! We would like to highlight that the key differences between our CoSC and CSV [1] as well as Self-Refine [2] are outlined as follows.  

• Difference from CSV [1]. The main difference between CSV [1] and our CoSC lies in the approach to verification. CSV relies on external tools, specifically code, for verification, whereas our CoSC approach performs verification entirely in natural language without using tools, which relies on the model's inherent capabilities for this process. CSV [1] argues in Table 4 of their original paper that relying solely on natural language verification of CSV can compromise accuracy and negatively impact performance. In contrast, our CoSC approach challenges this view and successfully achieves self-correction using only natural language verification, facilitated by a fine-tuning-based method. Specifically, our CoSC verifies the consistency between the question, the Python program, and the program's outputs using only natural language, which has proven effective.  
• Difference from Self-Refine [2]. The primary difference between Self-Refine [2] and our CoSC is that Self-Refine is unable to effectively identify mathematical errors, whereas our CoSC can do so. As stated in Paragraph 4 of Section 3.3 in Page 5 of their original paper of Self-Refine [2], the modest performance improvements in mathematical reasoning stem from the inability of Self-Refine to accurately identify errors. Self-Refine cannot be applied effectively for weaker models, as stated in Page 7 in their paper. Despite leveraging powerful LLMs such as GPT-3, ChatGPT, and GPT-4, Self-Refine only achieves a minimal accuracy improvement of 0%-0.2% on mathematical reasoning tasks, as shown in Table 1 of their paper.  In contrast, our CoSC approach excels in error identification during mathematical reasoning. As demonstrated in Table 6 of our paper, the verification module in our CoSC achieves a classification accuracy of approximately 70%, showcasing its ability to effectively identify erroneous answers. Moreover, our CoSC is capable of significantly improving the performance of weaker LLMs compared to GPT-3, ChatGPT, or GPT-4 used in Self-Refine. Specifically, our fine-tuning-based CoSC yields significant improvements in mathematical reasoning, with average accuracy boosts of 35.9%, 33.9%, and 29.3% over CodeLLaMA on the MATH and GSM8K datasets for the 7B, 13B, and 34B model sizes, respectively.

We acknowledge the valuable contributions of CSV [1] and Self-Refine [2]. Both approaches offer important insights, and we will ensure they are properly referenced and discussed in the next version of our paper. While our CoSC approach presents key distinctions, we believe that ongoing dialogue with these works is crucial for advancing the understanding and capabilities of self-correction methods in LLMs.

[1] Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification. In ICLR, 2024.
[2] Self-Refine: Iterative Refinement with Self-Feedback. In NeurIPS, 2023.

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1: The authors train a small model on the output of a large model and find that it improves performance a lot (compare "model distillation" [1]) and then train the resulting model on its own output and find that it improves performance further (compare born again networks [2] or more generally "expert iteration").

Thank you for your insightful comment. We have now included references to these papers [1,2] on model distillation and expert iteration in Line 163 of our revised paper. Additionally, we have revised the paper to better contextualize our work and to explicitly acknowledge the contributions of these related research fields. Thank you for bringing this to our attention.

[1] Cheng-Yu Hsieh, Chun-Liang Li, et al. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. In ACL, 2023.
[2] Tommaso Furlanello, Zachary C. Lipton, et al. Born Again Neural Networks. In ICML, 2018.

---

W2: The comparison landscape is incomplete and potentially misleading: Notable omission of open-source models with strong mathematical performance (QWEN2, Orca-Math, ...) that match/surpass the achieved performance.

Thank you for your valuable feedback. Qwen2 serves as a pretrained base LLM, whereas Orca-Math is fine-tuned on Mistral. In contrast, our CoSC model is fine-tuned on CodeLLaMA, highlighting the foundational differences in the base models. These differences result in inherent disparities in model capabilities, which limit the direct comparison of the results in their current form.

Our primary objective in this work is not to achieve the best performance on mathematical dataset leaderboards but to evaluate the novel capabilities introduced by our method, such as self-correction and iterative reasoning. These features represent significant advancements in enhancing reasoning robustness and accuracy, which are independent of the choice of base model.

In this revised version, we have included a broader discussion of Qwen2 and Orca-Math in Section 2.1 to provide context for their performance. However, given the limited time during the rebuttal phase, we plan to leverage our CoSC training dataset to fine-tune Qwen2 and Mistral in future work. This will enable a fairer, more direct, and comprehensive comparison of their reasoning capabilities under a unified framework.

---

W3: The authors should indicate exactly which GPT-4 version was used for training data generation.

Thank you for your valuable suggestion. We have clarified that gpt-4-0613 was used to generate our training data and added this information in Line 368 of the revised paper.

---

W4: The phrase "for some complex solutions we can only get < 1" is slightly awkward.

Thanks for pointing that out. We have revised it accordingly in the updated version of the paper.

---

Q1: Do the experiments for Table 4 use a fewshot prompt for the non-CoSC models to tell them how to utilize multiple rounds of reasoning? How do the authors explain that the fraction of "more than one round of reasoning" very close to 0% for those models?

Thank you for your insightful question. We would like to clarify that no few-shot prompts were used for any of the models presented in Table 4. The observation that the fraction of instances involving "more than one round of reasoning" is nearly 0% for the non-CoSC models, i.e., CodeLlama and ToRA-Code, highlights their limited capability to perform multi-round reasoning effectively. It is worth noting that both CodeLlama and ToRA-Code used in this work are the official open-source inference models provided by their respective developers. This result underscores the distinct advantage of CoSC in enabling multi-round reasoning processes.

Q2: Unlike these proprietary models, our CoSC performs the inference in a zero-shot manner without demonstration - I don’t understand this. How do the proprietary models considered in the evaluation use demonstrations?

Thank you for your feedback. In the evaluation of proprietary models such as GPT-4 and Gemini, the results we report are based on their respective official technical reports [2,3]. These reports indicate that the proprietary models often utilize few-shot prompting, where a small set of examples is provided within the prompt to guide their reasoning processes.   In contrast, our method, CoSC, does not rely on such few-shot demonstrations during inference. Instead, CoSC leverages its intrinsic capabilities, enabled by the fine-tuning process, to perform zero-shot reasoning directly. This distinction highlights a fundamental difference in the inference paradigms: proprietary models benefit from carefully constructed few-shot examples to achieve optimal performance, whereas CoSC is designed to generalize robustly without such external demonstrations.

[2] OpenAI. GPT-4 Technical Report. In 2023.
[3] Google Deepmind. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. In 2024.

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1 & Q1: How is ToRA prompted in this evaluation? Is it possible to simply modify the prompt to allow it to self-correct? Is fine-tuning in CoSC necessary?

Thank you for your insightful comment. To clarify, ToRA is a fine-tuning-based method trained on data generated by GPT-4, as described in Line 86-87 and Line 437-438 of our original paper. Unlike prompting methods, ToRA does not rely on instruction-based input during inference. After fine-tuning, it operates in a zero-shot manner without requiring external prompts or examples for reasoning tasks.

In this evaluation, ToRA is directly applied to test tasks, without additional prompt modifications or few-shot examples. While it is theoretically possible to adapt ToRA by incorporating some prompts that simulates self-correction, this would fundamentally change its methodology and deviate from the original design.

Our method, CoSC, is designed with self-correction as an intrinsic capability. Achieving this requires specific data collection and fine-tuning to enable robust iterative reasoning. This necessity arises from the fact that embedding self-correction as an inherent ability in LLMs demands altering the model's underlying parameters, a process that extends beyond the scope of simple prompt adjustments.

---

W2: Consider moving the algorithm a little bit earlier in section 3.2.1.

Thank you for your helpful suggestion! Based on your feedback, we have moved the reference to Algorithm 1 to Line 260 in Section 3.2.1 in our revised paper to make it more accessible and to improve the flow of the presentation.

---

W3: From Figure 1, it seems that ToRA does not perform any self-correction, so what does the 0.1% cases for ToRA in Round=2 mean?

Thank you for your valuable comment. According to the ToRA paper [1] and Figure 1 of our paper, ToRA generates a sequence consisting of a natural language guidance (r), a program (p), and an output (o) for a given question. This process is repeated until the model places its final answer within the “\boxed{}” symbol. The resulting trajectory is denoted as r₁p₁o₁...rₙ₋₁pₙ₋₁oₙ₋₁rₙ, where rₙ contains the answer.

For the case of Round=2 in ToRA, it implies that after the first round of reasoning (r₁p₁o₁), the answer generated by the model is insufficient. As a result, the answer is not placed within the “\boxed{}” symbol. Consequently, ToRA continues the reasoning in the second round (r₂p₂o₂), and the final answer is placed within the “\boxed{}” symbol in r₃. For a detailed description of this generation process, please refer to Section 2.1 of the ToRA paper [1].

One thing to note is that although the design of ToRA may suggest the use of multi-round reasoning, it does not explicitly possess the capability for multi-round inference, as clearly demonstrated in Table 4. Instead, the reasoning process of ToRA typically involves a single round of reasoning, with additional iterations occurring only in rare cases when the initial response is insufficient.

[1] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. In ICLR, 2024.

---

W4: Line 142: “revolutionize the accuracy” - please remove revolutionize here.

Thank you for your comment. We have revised the phrase in our updated version of the paper, removing the word "revolutionize" as suggested.

---

W5: Line 215: what if the conclusion is never reached?

Thank you for your feedback. As stated in Line 367 of our paper, we set a maximum limit of three self-correction stages if a conclusion is not reached. We have further clarified this point in Line 241 of our revised paper.

---

W6: Grammar issues.

Thank you for your careful reading. We have made the necessary revisions to address the grammar issues in the updated version of our paper.

---

W7: Can you add more description to the appendix? Some parts of the appendix like Appendix B are quite unclear.

Thank you for your valuable feedback. In Appendix B, we provide an example of how our CoSC model generates corresponding answers in response to a query. Specifically, Line 1147-1148 contain the question posed to the CoSC model, while Line 1149-1210 present the answer generated by the model. We have clarified these details in the revised version of our paper for better understanding.

Thank you for your response. I am updating my overall score to 6.

I would encourage the authors to explore incorporating evaluation methods that incentivize "self-correction" through the use of a large number of in-context examples, rather than relying solely on fine-tuning. My understanding is that fine-tuning a model for self-correction might reduce its performance on other tasks. If this understanding is incorrect, I would appreciate clarification. Additionally, I would like to see a more detailed discussion on the future directions of this research. For instance, How does this research fit in the context of more recent models trained for reasoning such as GPT4-o1, DeepSeek R1, or Qwen QwQ?

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1 & Q1: The comparison is not apples to apples. Did you try any normalizing experiment between CoSC and ToRA where both are trained on the same number of samples?

Thank you for your insightful comment. Our method is not a continuation or follow-up to ToRA. Instead, it fundamentally differs from ToRA in both its underlying principles and functionality. ToRA integrates Chain-of-Thought (CoT) and Program-of-Thought (PoT) approaches to enhance the reasoning capabilities of LLMs in mathematical problem-solving. However, it is primarily restricted to single-round reasoning and lacks self-correction capabilities. In Table 4, we can see that the reasoning process in ToRA typically involves only a single round, with additional iterations occurring only in extremely rare cases. In contrast, our method incorporates self-correction as an intrinsic feature of LLMs and is explicitly designed to support multi-round reasoning. These advancements enable more robust and iterative problem-solving.

To address your concern about the fairness of the comparison, we conducted an experiment to directly compare our CoSC model with ToRA [1], the top-performing open-source method. For fairness, we used the same number of training samples (69K) as reported in the official ToRA paper, representing its best official result. The results, presented in the table below, clearly demonstrate that CoSC outperforms ToRA on both datasets.

This result underscores that the superior performance of CoSC is not merely due to a larger training dataset but is instead attributed to the effectiveness of its self-correction mechanism. This mechanism enables CoSC to iteratively refine its outputs, providing a significant advantage over ToRA's approach. We believe this focused evaluation highlights the robustness and efficiency of our method under comparable settings.

[1] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. In ICLR, 2024.

W2: There is no ablation to clarify what capabilities are gained at a more granular level. CoSC has a generation and a verification step, unlike previous methods. The paper does not analyze these separately.

Thank you for your valuable comment. An ablative analysis addressing this point can be found in Appendix D.1 of the original paper. As indicated in Table 6, the verification module achieves a classification accuracy of approximately 70%, demonstrating the model's ability to identify erroneous answers. On the other hand, the correction module effectively reduces the errors identified during the verification stage by about 25%, confirming its effectiveness. These results are based on experiments conducted with the CosC-Code model at different sizes, specifically 7B and 13B, to ensure the reliability of our findings.

---

W3: The main novelty of this paper is the verification/self-correction step. Without an extensive evaluation showing these capabilities have improved, it is hard to assess the effectiveness of the proposed method.

Thank you for your valuable comment. In fact, we have conducted several experiments to assess the effectiveness of the verification/self-correction step of our method, and we reported the results in Tables 5 and Table 6 of the original paper. In Table 5, we observe that a single round of reasoning without self-correction leads to an approximately 7% decrease in accuracy. Furthermore, in Table 6, the verification module successfully identifies around 70% of erroneous answers, and the correction module reduces these errors by approximately 25%.

We appreciate any specific suggestions you may have for further evaluating the self-correction capabilities and are open to exploring additional validation steps.

Q2: Compared to ToRA and looking at Table 5, it seems like the accuracy when using just one round of reasoning is lower than ToRA. Does that mean the data generated is worse than ToRA's when it comes to reasoning without any self-correction steps?

Thank you for your insightful observation regarding Table 5. The lower accuracy of our method compared to ToRA in a single round of reasoning does not necessarily indicate that the data generated by our method is inferior. Instead, it reflects the design focus of our approach, which emphasizes multi-round reasoning and self-correction.

Our method is explicitly optimized to leverage iterative reasoning, where the self-correction mechanism refines intermediate outputs over multiple rounds. As a result, while the single-round performance may appear lower, the overall performance in multi-round scenarios demonstrates significant improvements. This trade-off highlights the unique strengths of our method in handling complex reasoning tasks, which rely on iterative refinement rather than single-pass outputs.

We appreciate your point and believe it underscores the complementary nature of our approach to ToRA. As clearly shown in Table 4, the reasoning process of ToRA typically involves a single round of reasoning, with additional iterations occurring only in very rare cases.

---

Q3: Because the models were trained on code, do they have an improved sense of code understanding?

Thank you for your thoughtful feedback. To address your question, we evaluate ToRA-Code and our CoSC-Code (both 7B models) on the MBPP dataset [2], which measures code understanding and generation. Both models are trained on mathematical datasets without code instructions.

The results show that CoSC-Code achieves a pass@1 score of 37.6%, outperforming ToRA-Code, which achieves 30.8%. This indicates that our CoSC approach enhances out-of-distribution generalization on code-related tasks compared to ToRA. We attribute this improvement to the iterative reasoning and self-correction mechanisms embedded in our CoSC framework, which likely contribute to a stronger capacity for structured problem-solving and logical reasoning, even in domains like coding.

[2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, Charles Sutton. Program Synthesis with Large Language Models. In 2021.