Faithful and Unfaithful Error Recovery in Chain of Thought

I recommend targeting more realistic errors made by recent LLMs. (1) First, this paper only targets artificial mistakes in the numbers of arithmetic reasoning. (2) In addition, the proposed three types of errors (page 5) are not representative of mistakes made by recent LLMs. I believe that recent strong LLMs do not often make simple copying errors or calculation errors. Analysis of too simple errors that recent models will not make is not very useful. This paper should first analyze the actual mistakes made by LLMs.

Thank you for your insightful feedback. We agree that analyzing more realistic errors made by recent LLMs is an important direction for future work. Our current study focused on artificial errors so that we can carefully control the experimental conditions, and eliminate any possible confounds. However, these may not fully capture the mistakes of state-of-the-art models in the wild.

In the revised manuscript, we will expand our discussion of this limitation and highlight the need for future research to:

1. Characterize the types of errors current high-performing models actually make on various reasoning tasks.
2. Design controlled experiments to study recovery from these naturalistic error types.

We believe our work provides a useful foundation for this future research.

We also see value in studying artificial errors in more complex reasoning domains, as a bridge between our current work and future naturalistic error studies.

Open-source models:

Thank you for raising this important point. We agree that using open-source models would make it easier to investigate how different training methodologies affect the error recovery behaviors we have found.

We did perform some initial experiments on Llama-2 70B. However, the baseline recovery rates for this model was too low to allow for meaningful statistical analysis of the effects of our experimental manipulations. For example, in the GSM8K calculation error condition, Llama-2's recovery rate was only 11%, compared to 80% for GPT-4.

We hypothesize that the lower recovery rate in Llama-2 (and GPT-3.5) may be due to lower overall reasoning abilities in these models.

In the revised version of the paper, we will present results for Llama-3, which may have higher recovery rates due to improved reasoning ability.

Natural vs. artificial errors:

We appreciate this feedback and agree that studying LLM errors in the wild is an important direction for future research. As you note, investigating naturalistic errors would likely require a different study design than the controlled error paradigm we employed here to probe specific questions about recovery behaviors.

We believe both approaches have value. Our current study sacrifices some external validity for experimental control. Studying naturalistic LLM errors has more direct implications for applications, but limits the types of questions that can be asked about LLMs and their error recovery behavior.

Regarding your suggestion to report the likelihood ratios of our edited CoTs compared to the original ones – this is a great idea for assessing the plausibility of our artificial errors. We will include this analysis in the revised version. If the likelihood does not substantially decrease, that would suggest our errors are reasonably plausible and could arise through the model's own sampling process.

Correctness of original CoT transcripts:

Thank you for pointing this out. This was incorrectly stated in the paper. We manually inspected the region surrounding the target error that we introduced, and filtered any CoT transcripts with incorrect reasoning in this region. Incorrect reasoning occurred in 1-13% of the CoT transcripts, depending on the dataset.

(1) I was wondering whether the findings in this paper could be generalized to open-source / smaller models? For example, could the authors add some experiments with llama 3 series in different model sizes?

We performed experiments on Llama 2 70B, but the recovery rates were too low to allow for meaningful analysis. We will perform experiments with Llama 3 for the revised version.

(2) Previous work has investigated how to make LLMs generate more effective chains of thought, incorporating aspects like selection [1], diversity [2], uncertainty [3], and difficulty [4]. I am very interested in how the author's findings in this paper can be applied to enhance the model's reasoning capabilities. Could the author possibly discuss these works to provide some outlook?

Thank you for providing these relevant citations on improving LLM reasoning. We see several connections between these papers and our findings on error recovery.

The uncertainty-based methods in [1] and [3] could potentially identify examples where error recovery is most important. The complexity-based prompting from [4] is related to our observation that models recover more effectively when expecting errors in a problem. The diversity approaches in [2] are related to our findings on the effects of varied error types and magnitudes on the rate of recovery.

We appreciate you pointing us to these works and will incorporate a discussion of these connections in the revised manuscript.

Clarification question 1:

This was incorrectly stated in Table 1. We manually inspected the region surrounding the target error that we introduced, and filtered any CoT transcripts with incorrect reasoning in this region. Incorrect reasoning occurred in 1-13% of the CoT transcripts, depending on the dataset.

Clarification question 2:

Faithful recovery and unfaithful recovery are reported as absolute quantities, not as a proportion of all recoveries. 1 - (faithful + unfaithful) is the rate of non-recovery responses.

Comprehensiveness of analysis:

We agree that extending this analysis to other reasoning domains beyond math word problems is an important direction for future work. While introducing controlled errors is most straightforward for numerical problems, we believe similar methodologies could be adapted for other tasks, such as using semantic similarity to define larger vs smaller errors. We will note this as an important area for follow-up research.

Regarding the sample size, 300 instances are being evaluated per dataset. This number was chosen based on a statistical power analysis. Given the large effect sizes observed, our analyses were significantly overpowered, and fewer samples likely would have sufficed. We will clarify this rationale in the revision.

For model diversity, we focused on GPT-4 because it was the strongest model available at the time of submission. Preliminary experiments with Llama-2 70B yielded recovery rates too low for meaningful analysis. However, we agree this is a limitation and will include results for Claude Opus in the revision.

Terminology and framing:

We appreciate the suggestion to reconsider the "faithful" terminology, as we agree it implies stronger claims about the model's internal reasoning than we can confidently make. We primarily used it to connect to prior work, but given your feedback to reframe the positioning of the paper, we will adopt alternative terms like "(un)acknowledged" or "(un)supported" recovery.

Thank you also for suggesting that our work can stand on its own in studying error recovery behavior. Those specific quotations are related to previous work, for example https://arxiv.org/abs/2307.13702, which identifies all error recoveries as unfaithful. We will revise the framing, clarifying the remarks on prior work while focusing more on the recovery phenomenon itself.