Large Language Models have Intrinsic Self-Correction Ability

Thank you for taking the time to review our paper and for providing valuable feedback. Your comments have helped us identify areas where further clarification and improvements were needed. We have addressed your concerns in detail below.

Eq3 is just wrong, p(A|\tau) should be p(A, R | \tau). Basically, p(X, Y | Z) = p(X | Y, Z) x p(Y|Z) (minor typo: \tau should be \tau_1). For the same reason, Eq 4 is also incorrect.

We thank the reviewer for this careful observation, and we apologize for the incorrect equation. Indeed, it should be $p(A, R | \tau_1)$ and $p_{SC}(A′,D,R2,A,R1|τ3,τ2,τ1)$.

Sec 4.1 opens by “the randomness in decision-making diminishes as the temperature decreases”; isn’t this just a trivial statement?

We agree with the reviewer that this sentence in itself may appear trivial, particularly to experts like the reviewer. However, it serves as a foundational basis for clarifying subsequent claims about the relationship between temperature increases, the likelihood of random flipping, and the degradation of self-correction ability. If we simply state, "As temperature increases, the likelihood of random flipping in the answer rises, leading to a degradation of self-correction ability," many readers may find this unclear and question the validity of the claim. The same goes for Section 4.1, which discusses the theoretical impact of temperature on decision-making.

In Equation 8, the authors take temperature to extremes to show that answers have more likelihood to flip (which is obvious even without all these equations).

We’d like to point out that $[0,\infty)$ is the value range for T, and Equation 8 does not say that T must go to $\infty$. Rather, Equation 8 shows that on the value range of $[0,\infty)$, the derivative of variance is greater than or equal to 0, with a higher value for larger T.

But more importantly, this finding has no meaningful connection to self-improvement.

We respectively disagree with this comment. The impact of T on self-improvement is indeed shown in Equation 8. As we note: “the variance of the decision increases monotonically with the model’s temperature, and the model with 0 temperature will be less likely to give the wrong decision due to less randomness,” and that “With an increasing variance, D has an increasing possibility of being flipped to the other side.” When temperature-induced flips happen, they will change more answers from correct to incorrect compared to vice versa (which is the argument of Proposition 2.1).

Even if it does, it does connect to self-improvement, it is somewhat unrealistic since practical applications rarely use such extreme temperatures.

As we have suggested above, Equation 8 does not mean extreme temperatures. Rather, it means that self-correction gets worse with increasing temperature.

In line 194, they cite Table 1 as evidence of "improvement in accuracy after SC", but the difference is merely 0.08% (75.92% to 76.00%).

We are very glad that you have this observation! Indeed, SC's performance looks very marginal on the Commonsense QA dataset, but this is due to the dataset's poor quality. As you may have also noticed, the improvement of chain-of-thought (CoT) on this dataset is also very marginal (75.35->75.92). CoT is a well-established and highly effective prompt engineering method, yet it exhibits only a limited effect on this particular dataset.

A brief look at some of the questions and model responses in this dataset reveals its poor quality. Other than the unfiltered questions, which models typically refuse to answer due to their nature, some questions have wrong or multiple answers. Nonetheless, we present the results of this dataset to align with other works, such as Huang et al., especially since our study arrives at different conclusions. Maintaining a degree of consistency in the use of the data is essential.

While Table 2 shows statistical significance across multiple datasets, the actual improvements are typically within 1%. This hardly constitutes meaningful improvement.

First and foremost, we would like to remind the reviewer that statistical significance indicates the presence of improvement. The meaningfulness of this improvement depends on the context and is not the primary focus of this work. As highlighted throughout the paper, our objective is to investigate whether intrinsic self-correction exists in LLMs. Through comprehensive analysis and experiments, we demonstrate that it does. While an improvement of 1%-2% may seem modest, it has the potential to serve as a foundation for future research aimed at amplifying this effect.

Our main point is that increasing temperature leads to a deterioration in self-correction

I see. I misunderstood what you're doing here. But again, isn't this somewhat self-evident? Pretty much anything gets worse with higher temperature. Basically, you can prove a statement like "increasing temperature leads to a deterioration in X" for your favorite choice of X, just because the next token distribution becomes noisy/uninformative for higher temperature.

Taking one step back, what you're proving here using temperature analysis (which takes up a good chunk of your draft; S2.2 and S4) does not seem to support with your main message, "LLMs have intrinsic self-correction ability". If I am misunderstanding anything here, please let me know.

Thank you for taking the time to review our paper and for providing valuable feedback. Your comments have helped us identify areas where further clarification and improvements were needed. It seems the listed Weakness is explained more in detail in Questions, so we will address the questions raised by you. If you have further confusion, please let us know in subsequent discussions. We have addressed your concerns in detail below.

What are the novel contributions of this paper over previous works on intrinsic self-correction? Given that the paper investigates the topic related to prior work on self-correction (e.g., Pan et al., 2023), it would be helpful to clarify how this research extends beyond or differs from earlier findings.

We provide a review of related works in Appendix A. The survey by Pan et al. is a very good summary of the big field of LLM answer correction, and under their characterization, our work falls into post-hoc correction-> self-correction (section 5.1 of their paper). We have included the reference of this survey paper in our revised related works to direct future readers to this excellent piece of work.

Regarding the novelty of our work, as we mentioned in the introduction, there have been recent debates on whether the intrinsic self-correction ability of LLM exists. Huang et. al’s work basically claimed that all the previous intrinsic self-correction papers were not under fair experiments. According to our observation, this debate on the existence of intrinsic self-correction in LLMs is primarily because the field of intrinsic self-correction lacks theoretical analysis, and people can always debate in experimental settings. Pan et al. also noted this in their future work section, urging for more theoretical foundations of this topic (their paper Section 7.1). As we highlighted in our list of contributions (first three points), our novelty is to provide theoretical analyses on this topic and give guidelines on the experiment setting. In addition to theoretical analyses, we also provide empirical experiments to show that what we say is true.

Our work argues this problem through the prompt-and-answer decomposition, as well as the connection with CoT and self-verification. There are several concurrent works that came out in the same period as our work. They tackle this problem from different angles, and we have discussed them in the related works section as well. “.Li et al.(2024) argue on the existence through the LLM’s confidence towards different questions,” and “Liu et al.(2024a) and Liu et al.(2024b) focus on a different theoretical perspective of the intrinsic SC ability….Their work focuses more on the convergence analysis.”

We additionally provide an analysis of the effects of temperature and prompt design on intrinsic self-correction. To the best of our knowledge, no previous works systematically studied temperature’s effect on self-correction. While there have been a lot of different prompt designs from previous works (and those are the root of the debate on the existence of intrinsic self-correction), we analyze the effects of semantic meaning and structure of prompts on intrinsic self-correction, which was not done previously.

Why is some certain experimental results not obviously show the advantage of self-correction? Is it from the method itself, or there are other failure modes?

This is expected. As we try to argue in the paper, self-correction is just an implicit self-verification. When the model is already good at answering, it can only occasionally correct its mistakes in a dataset, and hence, there are no expectations of having big differences before and after intrinsic self-correction. We have said in Lines 454-456 that “Importantly, our results align with the self-verification results obtained by Weng et al. (2023) where they also obtained a slight improvement using a different model.”

So to directly answer the question. This is from two parts, the method itself and today’s models. However, this does not mean the intrinsic self-correction method is not useful. The vanilla intrinsic self-correction does not have a big difference, but as long as it exists, there have been and will be more future works to amplify its effects.

Limited Model Sizes and Architectures:

We are happy to provide more results supporting our claim. We additionally conduct experiments on three benchmarks related to math and logic on Qwen 3b,7b, and 14b using the same settings as other experiments. We provide a snapshot of the new results below:

Thank you for taking the time to review our paper and for providing valuable feedback. Your comments have helped us identify areas where further clarification and improvements were needed. We have addressed your concerns in detail below.

The paper is very difficult to read. In particular, the equation part. Proof 2.1 is very confusing and many explanations are not given

We apologize for any confusion caused by our theoretical analysis. We will try to help you understand them by answering your confusion.

in Line 869, "which we denote as correct(A ∈ Q) = λ > k1". What is λ? The portion of correctly answered questions/all questions?

Yes, your understanding is correct. We updated the definition of λ in Line 868 to make it clear as “...some LLM has the true ability to answer (and not purely guess) a subset A∈Q correctly, which we denote its accuracy as correct(A ∈ Q)=$\frac{|A|}{|Q|}=λ>\frac{1}{k}$.”

And I do not think the probability of that hallucination randomly changing the answer is equal: tokens are of different importance to a sentence (answer), thus, some hallucinated tokens may not change the final answer (this is also related to how the authors define hallucination, but I do not see any justification here).

Your claim on the difference in token importance in the sentence is correct, but in the proof, we are analyzing the answer deterministic token, which is just one token in the QA setting, for example, “Yes/No”, “True/False” in a claim judgment problem or “A/B/C/D” for a multiple-choice problem. We have made this clear in the revised paper.

Also, what is h? The portion of hallucinated responses?

Yes, your understanding is correct. We define h as the percentage of LLM’s key answer deterministic token that is changed. We add a sub-clause after the definition of h as “where h% of answers are changed.”

I also do not understand what the authors mean by h∗(1−λ)/(k−1) answers will be changed from incorrect to correct.

First, in Lines 872-875, we provide our assumption that the LLM’s responses’ selected logits (confidence) are in a Uniform/Gaussian distribution, following the analysis made by Becker & Roberto (2024). Then, since we have h% of answers changed, and the accuracy of LLM is λ, the expected value of incorrect answers that are changed is h∗(1−λ). Since there are k answer choices and incorrect answers must originally be chosen from an incorrect answer choice, for each question, the answer change has k-1 choices, of which one is the correct answer. Thus, in total, the expected number of answers that are changed from incorrect to correct is (the expected value of incorrect answers that are changed)*(the chance that it gets changed to correct) = h∗(1−λ)/(k-1).

Also, why h∗λ> h∗(1−λ)/(k-1)?

First, we will reiterate what is written in Line 888-890: It is not hard to see that $h*\lambda > \frac{h*(1-\lambda)}{k-1}$, as this is the same equation of $\lambda>\frac{(1-\lambda)}{k-1} \Rightarrow \lambda*(k-1)>1-\lambda \Rightarrow \lambda*k>1$ which is the assumption we make above.

Now, we show a detailed step-by-step analysis of how we get the result:

$h*\lambda > \frac{h*(1-\lambda)}{k-1}$

$\lambda > \frac{(1-\lambda)}{k-1}$ $\quad\quad$ divide each side by h

$\lambda ∗ (k − 1) > 1 − \lambda$ $\quad\quad$multiple each side by k-1, since k>=2, the sign is not flipped

$\lambda ∗ k > 1$ $\quad\quad$ expand the equation on the left and each side + λ

In Line 869, we stated that λ > 1/k, which is the same as the derived result above.

Line 283, "(a1/T −(1−α)1/T)" should be "α" not "a".

Thank you for pointing this out. We have corrected it in the revision.

The selection of models is limited. More open sourced model should be included for a more comprehensive evaluation.

Before we provide more results, we want to point out that Huang et al’s paper was accepted by ICLR with 3 models and 3 benchmarks. We are not saying that’s the “golden” standard (as there was no standard), but we just also hope the reviewers can be more generous in the critiques on the number of models as they can never be enough..

We have already provided 4 models and 6 benchmarks. Nonetheless, we are happy to provide more results supporting our claim. We additionally conduct experiments on three benchmarks related to math and logic on Qwen 3b,7b, and 14b using the same settings as other experiments. We provide a snapshot of the new results below:

To make $h*\lambda > \frac{h*(1-\lambda)}{k-1}$ hold true, $\lambda > 1/k$. But I do not see why λ > 1/k is always true.

According to the definition in Line 948-949: some LLM has the true ability to answer (and not purely guess) a subset of A ∈ Q correctly, A does not contain purely guess answers so |A| is simply $n$. And according to their calculation, $correct(A \in Q) = \frac{|A|}{|Q|} = n/m$. With the step-by-step calculation, I hope that it is clear the authors realise their errors in their presentation and be more careful with the claims and proofs in their submission.

We are extremely grateful for the reviewer’s unambivalent rating of 10 for our work. We hope there will be more readers like you who can appreciate the potential significance of our work in providing “best practices for prompt sets” that “can both be directly applied in engineering work and used to inspire future analysis.” Thank you.

Below, we will address your concerns.

They also make some claims that feel more dubious to me, like “LLMs are naturally underperforming their intrinsic ability” which feels a little underdetermined to me.

We agree with you that the statement (“LLMs are naturally underperforming their intrinsic ability”) was not very precise, and indeed it was almost impossible to quantify LLM’s true intrinsic ability. What we meant was that the LLM’s true intrinsic ability (whatever it was) can be considered as the theoretical upper bound of its knowledge. We argue in our paper that, such a true intrinsic ability cannot be reflected through its answers because of potential hallucinations during decoding.

Our results are in line with your intuition as well. We argue that, through intrinsic self-correction, we bring LLM’s output closer to its theoretical upper bound. This is essentially what other prompt engineering methods try to do. That’s the beauty of our paper’s insights and our unique contribution. Thank you for seeing this so clearly.

I wasn’t the most well-positioned to assess the novelty of this work. I took a look through Pan et al. 2024’s survey “Automatically Correcting Large Language Models” paper to help check this, might be useful to reference for other readers?

Thank you. We have included this very much relevant review paper in our revision.