Chain of Thoughtlessness? An Analysis of CoT in Planning

Thank you for the thorough review.

Responses to questions:

1. The lexicographic stacking problem is a special case. For a given number of blocks, there is only one problem which requires stacking them all in lexicographic order, as the syntactic stringency of the order fully determines the problem. All of our instances are between three and 20 blocks total, thus giving us exactly 17 possible lexicographic problems to test. You are exactly correct that the model may only perform well on these examples–in fact, they are constructed to explicitly require only syntactic matching. We will make this clearer in the text and description, and we will highlight this in the final table by shading the relevant squares a different color.

2. Section 6 extends our results to previously studied natural language problems. We took domains that had been examined in the original CoT paper–coinflip and last letter concatenation–and created versions where more reasoning steps are necessary. Neither of these is “action-related”. In both cases, previous work has claimed CoT learns the necessary procedure by example. In the same section, we also examine a very simplified natural language arithmetic domain where the model only needs to repeatedly simplify one digit by one digit arithmetic expressions. We discuss the results and point out that we see the same trends as before in letter concatenation and arithmetic simplification.

As for the varieties of CoT listed: In-Context Learning (ICL) CoT and Zero-shot are both explicitly covered.

If by Complex CoT, the reviewer is referring Fu et al’s 2023 paper Complexity-Based Prompting for Multi-step Reasoning, then, to quote from it: “We observe a clear trend on both GSM8K and MathQA: complex prompts perform on par with simple prompts on hard cases, while achieving more clear gains on cases with fewer number of reasoning steps.“ This directly matches our results. Low reasoning step number problems are very amenable to CoT improvements, but these gains disappear when generalization is attempted.

3. We assume the reviewer is referring to Figure 2. We’re not saying that CoT doesn’t improve raw performance. We’re saying that the mechanism underlying it does not seem to involve learning the procedure or algorithm demonstrated. If it did, then we’d expect to see generalizable performance improvements. Table-to-stack problems are not particularly complex: every block is on the table, and the model is tasked with stacking them in a predetermined order.

The stacking prompt proceeds very similarly to an explicit n-shot plan-and-solve prompt: first the model figures out which is the bottom block, and then it figures out, step-by-step, which block goes on top of the previously placed block. Every one of these steps requires only a simple syntactic transformation that LLMs are known to excel at, but when the model is tasked with doing this for instances larger than the ones demonstrated, it fails to extend the procedure correctly.

This is even clearer in the case of last letter concatenation. There, the model need only list every word and its last letter and then concatenate them together. It is able to do this perfectly for up to a few words, but afterwards, performance quickly plummets. Our main point, however, is that CoT does not work the way that it has been described in previous work. The model may do better, but it doesn’t learn and apply a new algorithm.

About experimental details: All code and data in this paper will be made public, together with instructions for setting up and running the same tests with any API-accessible LLM. We will add the details about the exact models and temperatures we used to the paper. They are reproduced below:

Temperature was set to 0 for all experiments except those already explicitly mentioned in the current text (self-consistency and some single-digit arithmetic). We used the static models when possible: GPT-4-Turbo is gpt-4-turbo-2024-04-09 and GPT-4 is gpt-4-1106 in the OpenAI API. Claude-3-Opus was accessed mid-April 2024.

The instances within the Blocksworld domain were generated using the PDDL generators provided by the International Planning Competitions. Within each problem class an equal number of instances were generated across the number of blocks. The intended test set for zero-shot, progression proof and universal algorithm had a total of 270 instances (15 instances per # of blocks). For the stacking prompt, the test set had a total of 261 instances as there are only 6 stack combinations for 3 block problems. Finally, there was only one instance per # of blocks for the lexicographic case.

The coinflip, lastletter, and arithmetic domains are detailed in appendices A.3, A.4, and A.5. CF and LLC are both domains extended directly from Wei’s 2022 Chain of Thought paper, modified to allow for arbitrary length instances. Exact distributions are in the appendix.

The multi-step arithmetic domain is a synthetic domain. We will add the random generation procedure for problems that we used to create the dataset: generate the innermost number m; uniformly rejection sample an operation # and number n, such that n#m=k is an integer from 1 and 9; if the number of reasoning steps is enough, stop, otherwise set m<-k and jump to the rejection sampling step.

The core point of our paper is not that CoT is sensitive to prompts and demonstrations, but that CoT, contrary to previous claims, does not in-context teach LLMs general and robust algorithmic procedures. We show that CoT depends on specific prompts being narrowly constructed and customized to the generality of the problem class and the length complexity of the instance itself. Our results provide critical evidence that counters the current claims and consensus that CoT unlocks human-like procedural reasoning abilities within LLMs from a few demonstrations, and suggest that pattern matching rather than procedure following may be a better explanation for its success.

Thank you for your reply.

1. Perhaps we are misunderstanding, but this response seems contradictory: robustly learning and applying a correct algorithmic procedure and learning a “step-by-step thought” are the same thing. Our claim is that, while the improvements seen do exist, they are brittle in ways that robustly learning the requisite step-by-step process wouldn’t be–raw improvement does not contradict this. The evidence necessary for our claim is the rapid deterioration of this improvement on instances that require more reasoning steps, a phenomenon we demonstrate across multiple diverse multi-step reasoning domains, from variants of Blocksworld to last letter concatenation and arithmetic expression simplification. In the camera ready version, we will make much clearer both this distinction and how we are using the evidence we gather to show the claim we make.

2. Outside of zero-shot methods (“let’s think step by step”), manual CoT construction is always domain-specific, and requires demonstrating an in-context procedure for each exemplar. We do not claim anything about CoT’s sensitivity to particular prompts. Consider the Last Letter Concatenation domain: performance is perfect for smaller instances, yet falls quickly on larger ones, despite the prompt and procedure being the same in all cases. Our contribution is to provide critical evidence that the intuitions underlying previous work–that CoT demonstrations of procedures allow LLMs to learn those procedures and apply them in-context, an anthropomorphization arising from the response format engendered by the technique and the popular name for it–miss the mark. We constrain our claims and evaluations to CoTs that are designed for and applicable to the problems they’re presented with.

3. In its current form, our paper does explicitly discuss GSM8k, CommonsenseQA, MAWPS, AsDiv, and others. We go further in depth on arithmetical reasoning benchmarks and construct a simplified natural language synthetic benchmark in section 6, under the heading “Multi-step Arithmetic on Single Digit Numbers”. In particular, we discuss a fundamental limitation of these benchmarks: they generally require only a couple of reasoning steps. Only ten percent of all GSM8k problems (a benchmark which explicitly attempted to increase the number of reasoning steps necessary) require more than five steps, and none are over eight. This sort of narrow problem distribution is insufficient to properly evaluate whether the model has learned and applied the correct procedure or is merely pattern matching to syntactically similar templates.

Note also the quote we brought up in our rebuttal from the Complex CoT paper, which we will include together with a deeper discussion in our camera ready version: “We observe a clear trend on both GSM8K and MathQA: complex prompts perform on par with simple prompts on hard cases, while achieving more clear gains on cases with fewer number of reasoning steps.” Robust generalization is still not seen even with additional prompt engineering or manipulation.

Furthermore, as mentioned in our paper, CommonsenseQA and similar domains do not explicitly require multi-step reasoning. In fact, the CoT exemplars given in Wei’s 2022 Chain of Thought paper for CSQA are all exactly two steps. Because CSQA questions require broad knowledge, and are very amenable to memorization, there is no way to scale the dataset to instances that necessarily require more steps–these are problems that mainly test what the model knows rather than whether the model can reason in a generalizable manner.

Respectfully, we found this review highly inconsistent and in places incoherent. We wonder if there was some transmission/saving error when the reviewer posted it. Nevertheless, let us respond to the review as it was posted. We hope our clarifications persuade you to rethink your overall evaluation of the paper.

Lines 84-88 are a summary of our results: “Overall, this case study calls into question assumptions about the generalizable effectiveness of chain of thought, and suggests that LLMs do not learn new, general algorithms in context, but instead rely on some form of pattern matching to achieve prompt-design-specific performance increases.”

The reviewer’s quote is from Wei’s 2022 paper introducing Chain of Thought. The full context from the paper:

We explore how generating a chain of thought—a series of intermediate reasoning steps—significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain-of-thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting.

Our paper directly challenges the generality of this claim. While CoT does improve performance in terms of raw accuracy in many domains, our results show that it does not do so in a way that generalizes across problem complexities, despite including explicit demonstrations of algorithms that do generalize along these dimensions. Thus, we position our paper as a counterpoint to Wei’s “lower bound” approach–our results give evidence about the upper bound of CoT efficacy as part of a critical examination of the assumption that CoT unlocks human-like procedural reasoning from a few examples.

There may be some confusion about the meaning of the title. The full title is “Chain of Thoughtlessness? An Analysis of CoT in Planning”. The word “thoughtlessness” does not refer to the opposite of thought, but to the opposite of thoughtfulness. What we are questioning with this title is whether “chains of thought” generated by LLMs are produced in a careless way–one which does not seem to involve generalizable procedure learning but instead looks to consist of slapdash pattern matching. We would also argue that the word “thought” in the phrase “chain of thought” is already very loaded and anthropomorphized, and so is fair to criticize.

Furthermore, we examine three domains other than blocksworld: coinflip, last letter concatenation, and an arithmetic benchmark. The first two are direct extensions of previous work that made explicit claims about the breakthrough efficacy of CoT on these domains. We confirm that CoT prompting does generalize fairly far in the coinflip domain, but find that on last letter concatenation and arithmetic expression simplification, performance drops substantially when the domain is extended, showcasing that the model did not correctly replicate the procedure demonstrated in the prompt.

We wish to reiterate that the central claim of our paper is not that CoT does not lead to any performance improvement, but, to quote the reviewer’s strengths section, to offer evidence against “the widely held belief that CoT enables LLMs to learn and apply general reasoning strategies.” We will clarify this point further in the final copy, and we will change the phrase “does not meaningfully enhance performance” to “does not robustly generalize.”

We’d also like to point out that, in table 1, Zero-shot CoT improves or worsens (depending on the model) performance by around one percentage point, making no meaningful difference, and that the Blocksworld Universal algorithm is already a fairly narrow prompt which can only perform well on a loose relaxation of the domain. Note that the improvements become more significant the more granular and problem-specific the prompts become–which is a point we make in the paper: to effectively utilize CoT, the user needs to greatly restrict the problem or write different CoTs for many subproblems. We will add this analysis to the appendix.

Blocksworld instances were generated using PDDL generators provided by the International Planning Competitions. The intended test set for zero-shot, progression proof and universal algorithm had a total of 270 instances (15 instances per # of blocks between 3 and 20). For the stacking prompt, the test set had a total of 261 instances as there are only 6 stack combinations for 3 block problems. Finally, there was only one instance per # of blocks for the lexicographic case. We will add these details to the appendix.

On formatting:

• Almost every paragraph begins with a linking sentence introducing the new topic. We will also improve the prose: lines 40-47 will be rearranged to improve flow, lines 48-53 will be mostly removed. We can add headings to the paragraphs in the related work as follows: What is CoT?, Enhancements to CoT, Problems with CoT, Generalizability of CoT, Current Opinions. We will also rework the final paragraph of the related work section to increase clarity and readability.
• Section 3 will be a subsection of related work.
• Due to the amount of detail provided in each prompt, it is infeasible to provide examples in the main body. We compromise by putting the prompts in the appendix and creating figure 1, which illustrates each of the three levels of generality in the blocksworld domain. We will make it clearer in the text which sections corresponds to which problem.
• We do provide a definition of PDDL in the background section on line 146, but we will add a parenthetical to the introduction to clarify.
• We have corrected the figure references.

Thank you for your reply.

We reiterate that our global response as well as the response to you clearly points out that we are only claiming that CoT doesn’t robustly learn the algorithmic procedure demonstrated in prompt, as shown by the performance deteriorating as the number of reasoning steps increases (despite the fact that the given procedure does solve the problem). This is not contradicted by raw improvements on problems similar to the given CoT exemplars. As we pointed out in our individual response to you, your original review seems to have misread this.

We would also like to draw your attention again to the fact that in addition to planning tasks, we have done experiments on extensions of standard CoT benchmarks–including last letter concatenation (Section 6)--and those results too are in agreement with our claim about Cot not leading to general procedure learning.

Finally, we note–as you yourself can readily see–that all the other reviewers have had significantly more positive assessment of the paper.

We look forward to your reconsideration of your assessment.

Thank you for the thoughtful review.

The progression proof CoT prompt does include a meta-prompt, but we failed to include it in the original draft. Here it is:

The plan correctness is defined in terms of states resulting from executing the actions in the plan. An action is executable in a state when all its preconditions hold in that state. The state resulting from the action execution consists of everything in the previous state with the addition and deletion of add and delete effects of the action. Plan correctness is defined as follows: if the first action in the plan is applicable in the initial state, i.e., its preconditions are all present there; and the second action is applicable in the state resulting from applying the first action to the initial state, this process continues until the state resulting from the application of the last action in the last but one state gives rise to the final state where all the goals are satisfied. We will add this to the appendix.

We’ve also fixed all the missing and incorrect references, including the ones you mentioned. Thank you for catching these!

As to your question, we did also test multiple prompts across our experiments, to ensure that minor details of prompt selection did not impact our results, and chose the best performing ones for fairness. In the Blocksworld domains, we tried four varieties of the universal prompt, ranging from more to less explicit algorithm demonstrations. We compared a CoT that required finding the base block first and ones that instead serialized the goal conjuncts, eventually settling on the prompt that gave the best performance. We also compared varieties of lexicographic prompt while looking for a clearcut example where CoT guarantees length generalization over n-shot prompting, and we checked reverse lexicographic problems, as well as problems where the given query had the opposite order of the prompts given. We found some variation across all of these, but nothing that contradicted the general trends we observed in our reported findings. All data we gathered, whether featured in the main body of the paper or not, will be publicly released on github, and we will add a discussion of these prompt variations to the appendix.

We also considered prompt variations in the non-planning domains:

In the coin flip and last letter concatenation domains, we generated the same kinds of prompts that Wei’s 2022 Chain of Thought paper used. However, as we needed more varied names, we drew ours from a different source (the US Social Security administration, as discussed in appendices A.3 and A.4), and partitioned these into 1, 2, and 3-length names. We measure the length as the number of tokens the GPT-4-Turbo tokenizer (CL100K_Base) requires to encode them. We found no significant differences in performance between them, and included equal mixtures of all three kinds of prompts in our reported experiments.

We also varied the number of examples–from 1 to 3–and the correctness of the examples: either all correct examples or all incorrect. While incorrect examples do sometimes show a small decrease in performance, it is almost the same, and the overall trend is unaffected (see also Invalid Logic, Equivalent Gains, Schaeffer et. al. 2023 for a more thorough analysis of wrong CoTs).

In the letter concatenation task, we also tried a prompt where, instead of saying “let’s think step by step” we merely added a “[Thought]” tag to the end of the direct zero-shot prompt. This retained some of the improvement of normal zero-shot CoT, but didn’t do quite as well. In the arithmetic task, we varied the requirements of the CoT from free-form thoughts to requiring intermediate answers to be tagged to requiring intermediate answers and computations to be tagged. We also tried explicitly including an instruction that every intermediate computation will be a single digit integer. Performance was roughly the same across all, and the overall downward trend was unaffected by these modifications.

We will add these additional experiments and associated charts together with this discussion to the appendix.

Thank you for the thorough review.

Analyzing the data using only the tables is somewhat misleading. Figures 2 and 3, as well as the appendix figure A.1.1 show more clearly that the bulk of the improvement for all prompts is on the few-block problems, whereas if the procedure shown in these CoTs were followed, we would expect this improvement to be much more robust on examples of larger length than those shown in the examples.

To the point about the progression proof prompt: “Chain of Thought” is a loosely defined term in the literature. Intermediate “thoughts” vary from piece-by-piece rationales to compositional reasoning steps. The progression proof CoT we use is taken from previous literature that studied LLM performance on this domain ([1] call it a “state-tracking CoT prompt”). At each step, this prompt provides not just reasons for the given action, but also the current state from which to determine which action to take next. In their 2023 survey of CoT techniques, Zhang et. al. specify that “the intermediate processes of CoT reasoning [...] can encompass solutions, intermediate reasoning steps, or any relevant external knowledge pertaining to a question.” The progression proof provides partial solutions and their effects, thus making it a valid CoT instantiation.

In fact, the procedure provided, if followed, should ensure that the output plan is valid. Yes, this prompt doesn’t provide an algorithm that precisely specifies every part of the reasoning process, but neither does any CoT–all CoTs assume that the model can handle some parts automatically. For example, in grade school arithmetic tasks, the selection of relevant numbers and the exact sequencing of the steps needed to solve the problem (which can be the hardest part of the problem, something which is exploited by teachers who introduce unnecessary additional information) is not specified in the chain of thought prompts provided in the original Chain of Thought paper (Wei et. al. 2022). Note also that, while the progression proof guarantees the executability of the output plan, we have data showing that this guarantee is not respected by the LLMs we test, thus showing they did not execute the given procedure:

If they had, they would have generated a high or perfect percentage of plans that were executable. GPT-4-Turbo generated 9.36% executable plans, Claude-3-Opus was 59.63%, and GPT-4 was 38.15%. We will include these figures in the appendix as additional analysis of the LLMs’ inability to execute the procedures demonstrated in the CoT.

The central claim of our paper is that, because CoT prompts do not lead to the model robustly learning the correct algorithmic procedure, effective CoT prompts require both strong specificity to the problem class and specificity to the length complexity of the problem itself. Our data generally supports the problem class specificity hypothesis, as zero shot CoT does worse than the progression proof, the progression proof CoT generally does worse than the universal algorithm, the three-part universal algorithm does worse on two out of three models than the two-part stacking prompt (which is itself a version of the universal algorithm that assumes all blocks are already on the table, but is otherwise identical to it–hence how close their performance is), and the stacking prompt does worse than the lexicographic prompt, which does the best. The other part of our central claim–that CoT demonstrations do not induce length generalization–is supported not just across all but the most trivial (the lexicographic case) of our blocksworld domains, but is also shown very clearly by our analysis of the last letter concatenation and arithmetic expression simplification tasks (see Figure 3).

On lexicographic stacking performance: In all of our problems, both the n-shot and the CoT prompts use the same examples. The lexicographic n-shot prompts tend to fail because the model neglects exactly this part of the rules: “I can only pick up a block if the block is on the table and the block is clear. A block is clear if the block has no other blocks on top of it and if the block is not picked up.” Instead the model will stack A on B, then pick up B (which is illegal based on the explicit rule cited because B is not clear) and stack it on C, and so forth. GPT-4 fails all lexicographic problems which are larger than the examples given in this exact same manner. The CoT prompt specifies a two-part procedure: first determine which block is the base of the tower, then stack all the blocks in the correct order on that base. This is sufficient to correct the issue in most, but not all, cases, most likely by aligning the most probable completion with the correct semantics in this particular case.

Typos and formatting issues: Thank you for pointing these out! Missing references have been added and incorrect ones have been fixed.

We will add the n-shot prompts to the appendix, as well as releasing the entire codebase and data publicly on github.

[1] Valmeekam K, Marquez M, Sreedharan S, Kambhampati S. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems. 2023 Dec 15;36:75993-6005.