Thank you for your review of our paper, which you correctly describe as “demonstrating that current models lack the ability to use their resources and capabilities fully” and “well motivated in terms of potential applications for the evaluation and understanding of LLMs.”:

Let us address the concerns that you raised:

Weakness 1: Do models have the necessary capabilities? If not, does the measurement mean much?

It’s true that LLMs are far from universally capable AIs, and sometimes have surprising weaknesses and failure modes. (The paper you cite is already a few years old, so some of these failure modes have already disappeared.)

For this reason, a significant part of our paper is devoted to assessing whether models do have the necessary capabilities to do the tasks that we test them on. In general they do. It’s also worth emphasizing that our tasks allow a range of outcomes beyond just fail/pass, and allow us to measure how close a model came to a “perfect” answer. The tasks are hard enough that no model solves them perfectly all the time, and easy enough for even GPT-3.5 to often make at least some progress on them.

Furthermore, our goal-directedness metric is computed by comparing the performance of models on a combined task, with performance on different subtasks.

We have also tailored the instructions, to minimize the frequency of models misunderstanding the task or interface.

Weakness 2. Seeds and results

Apologies, the paper should of course mention this (we’ll make sure to bring it back). For the main results and ablations, models are always run on 5 seeds. The error bars show the Standard Error of the Mean (SEM). The standard deviation is about twice the width of the SEM bars (sqrt(5) times wider, to be precise).

Weakness 2: Small number of datapoints (evaluation with only 3,4 and 5 blocks) and a single environment

What is a sufficient number of experiments is of course always somewhat subjective. But let’s us emphasize that within our blocksworld environment, we have tested models on a range of different tasks and objectives:

• the Build Highest Tower task (which mostly requires measuring),
• the Equal Towers task (which requires significant cognitive effort and execution skills), and
• the Falling Tower task, which requires perseverance in spite of adversity. So even though it is a single environment, we still test a range of different kinds of tasks. And for each of these tasks, we have tested for a range of different numbers of blocks.

Across essentially all these tasks and settings, the finding that most models lack some goal-directedness has been consistent.

In reviewer yUcf’s opinion, “The experiments conducted are very close to "minimal but complete" in that they include all and only the aspects that are necessary to study goal-directedness, which makes for very clear results.”

Since the submission of the paper, we have also developed an anagram environment. Please see the global comment.

Weakness 3, clarity

Thanks for pointing this out. We will make sure to reiterate the importance of understanding goal-directedness also in the conclusion, and perhaps find a way to highlight it more in the introduction.

As pointed out by reviewer yUcf, a nuanced understanding of LLMs is essential as LLMs become ever more widely deployed. In particular, many expect them to soon be deployed as “agents” acting autonomously on behalf of users, where goal-directedness is a key property for both safety concerns and capabilities.

Thank you for your review of our paper which measures the goal-directedness of LLMs, for which you found the “conclusions drawn regarding the goal-directedness of several state-of-the-art LLMs … persuasive.”

Let us address each of the concerns you raised:

The discussion of related work is insufficient:

Our paper provides an extensive literature review over related work both in machine learning and neuroscience and psychology, spanning more than a page and resulting in more than 3 pages of references.

We will be happy to also include the paper you mention. Briefly, they are primarily interested in measuring the capabilities of LLMs to reason about goal-oriented tasks, while we are interested in the motivation of LLMs to use capabilities to solve such tasks. A main finding of their paper is that Tree of Thoughts (ToT) reasoning is less effective for reasoning about sequential reasoning tasks, while Chain of Thought (CoT) prompting only works in certain sequential reasoning scenarios, and is detrimental in others. They also offer a useful characterisation of key properties of goal-directed tasks, which we’ll consider incorporating.

The contributions of the paper are relatively limited:

As summarized at the end of our introduction, our contributions are five-fold:

• a conceptual definition of goal-directedness suitable for LLMs, and
• a principle for evaluating it empirically (Section 3),
• an open-source implementation of 5 tests in a Blocksworld environment, and
• an assessment of the goal-directedness of 4 LLMs (Section 4), as well as
• a review of related work in psychology/neuroscience and AI (Section 2). As pointed out by reviewer yUcf, a nuanced understanding of LLMs is essential as LLMs become ever more widely deployed. In particular, many expect them to soon be deployed as “agents” acting autonomously on behalf of users, where goal-directedness is a key property for both safety concerns and capabilities.

The framework is too simplistic and would not capture goal-directedness in other tasks and domains?

As reviewer yUcf points out, “The proposed definition of goal-directedness is clear, straightforward (this is a positive thing), and well-motivated. The experiments conducted are very close to "minimal but complete" in that they include all and only the aspects that are necessary to study goal-directedness, which makes for very clear results.”

Furthermore, our framework can be easily used to evaluate LLM goal-directedness both during and after training, and the principle is easily transferable to other tasks and environments. The experimental design is carefully crafted to facilitate this. In our response to reviewer yUcf, we briefly describe how we have applied the same principle to an anagram environment.

Further investigation is needed into the internal mechanisms of the models.

Mechanistic interpretability is beyond the scope of our work. Despite ongoing efforts to understand the internal workings of deep neural networks, and LLMs in particular, spanning 100s if not 1000s of papers from top universities and AI labs, the field is still far from answering much more basic questions such as how facts are stored or simple questions answered (the situation is not so different from Neuroscience, in fact). The fact that the behavioral approach pursued in our paper doesn’t rely on a detailed understanding of the internals of the models is one of its key strengths. We will emphasize this in the updated version of our manuscript.

It is essential to test the influence of auxiliary methods (e.g. COT, TOT):

We effectively already test Chain-of-thought, as the system prompt encourages the models to “reason-step-by-step” before outputting their next action (line 183 in the paper). That said, we agree that it would be interesting to test the influence of various scaffolding techniques. We discuss this in the limitations section (line 482). We hope others will use our framework (which will be open sourced upon paper publication) to systematically explore how other scaffolding techniques impact the goal-directedness of models.

Please let us know if our answer addresses your concerns or if you have additional questions.

Thank you for your review, which recognises we found limited goal-directedness in state-of-the-art language models, using a method reviewer yUcf described as “novel, while also … elegantly simple and powerful in its generality.”

Rigorous definitions

Does Line 79 mean the difference between difference for the optimal policy?

No, and apologies the paper was insufficiently clear on this point. The comparison is not intended to involve the optimal policy at all (only optimal use of capabilities). We will change the misleading sentence on line 79 to "The gap between the agent's actual performance and the agent's expected performance if it made full use of its capabilities."

For example, the Build Highest Tower task requires the agent to build a maximally high tower out of any two blocks. The optimal policy takes measurements until all block heights are known to two decimals with high certainty, then stack the two highest blocks. But agents typically only have the capability (or patience) to measure block heights to one decimal point. Conditioning on this (lack of) capability, an agent’s expected performance will be worse than optimal (as it won’t be able to reliably select the highest block).

However, it often turns out that agents’ suboptimal performance is not fully explained by their lack of capabilities. Instead, they perform even worse than one would expect from just knowing their capability to measure block heights accurately (and ability to select the blocks with the highest estimates). The gap between this actual performance, and the expected performance given full use of capabilities, is what we interpret as a goal-directedness deficit. Figure 5 in the paper shows this gap for the build highest tower task (and Section 4.3 discusses it).

Formal def of E[regret | optimal use of capabilities]? (Line 174)

Formally in RL terminology, the agent is inhabiting a POMDP, where its capabilities correspond to options. For example, an agent capable of measuring blocks to 1 decimal can be viewed to have a measuring option which brings it to an (information) state where the block height is known to 1 decimal. The capability-options define an MDP over states and belief states, with an action-set consisting of these capability-options.

Now we can more formally define:

E[regret | optimal use of capabilities] = optimal return in the original task - the return of the optimal policy in the capability-options belief-MDP.

Algorithm 1 motivation

The idea behind Algorithm 1 is simply to simulate an agent with a particular set of capabilities.

To this end, we first sample a set of “actual” block heights (Step 2).

We then account for the agent’s measurement capability, by sampling a measurement error for the height of each block, using the distribution of errors we’ve seen the agent make in a separate capability test (Step 3).

Next, we assess the agent’s planning ability, by

• Sampling a set of configurations that the agent might have conceived of, based on the number of configurations the agent generated in a separate capability test (Step 4),
• Computing the height the agent would have assigned to each configurations, according to its flawed block heights estimates from Step 3 above (Step 5), and then
• Picking a preferred configuration based on these height estimates, here taking into account the agent’s ability at picking out the best configuration given some height estimates (Step 5), as usual based on an independent assessment of the agent’s ability at this

Finally, we take into the agent’s ability at actually building the tower it intends to (Step 8). Again, this is based on an independent assessment of the agent’s ability to execute an intended plan.

Line 250: On 205 you mentioned sigma to be .1 * true height, yet here you also added a constant factor.

Thanks for pointing this out. We added a constant term to our model of agent measurement errors, to account for the possibility that agents adapt the number of questions to the noise observed in the measurements. An explanation would have been warranted.

On reflection, we have decided that it is cleaner to instead just sample one of the agent’s actual measurement errors on a block of similar height from the capability check. This sidesteps the need to fit a model, and the extra assumptions this entails. The results are only marginally affected. We will update the paper to reflect this change.

Result analysis

Why does the regret for GPT 3.5 fluctuate (i.e., drop and then increase) in the observed experiments?

The slight dip at 4 blocks may well be a statistical fluke, as the mean at 5 blocks is well within the SEM of the 4-block result. More surprising perhaps is the dip at 4 blocks for both GPT-3.5 and GPT-4 on line 420. One possible explanation for this is that putting an equal number of blocks in each tower is a reasonable heuristic for making equally high towers. This heuristic works better for an even number of blocks.

Thank you for your feedback and your positive review!

Weakness 1: "turtles all the way down"?:

This is a good point. There is in general no single, objective way to break down a task. Nevertheless, any particular breakdown can still give evidence of a lack of goal-directedness, if the performance on the combined task is worse than would be expected from the performance on the subtasks. This relies on a tacit assumption that doing smaller, more well-scoped tasks requires less goal-directedness than doing larger tasks.

A breakdown needs to satisfy two key properties:

• It should be clear to the agent how to compose the subtasks to solve the full tasks. (This means we cannot break the task down into too miniscule components, such as steps of a Turing machine, since then it would not be obvious how to compose the tasks.)
• There should be no alternative way of solving a task other than the one suggested by the breakdown.

We will endeavor to highlight these important points more in the updated manuscript.

Weakness 2. How can we be confident that we have now reached a pure source for capabilities measurements?

We’re not confident that there usually is such a thing as a complete breakdown of a task into “primary” capabilities. As discussed in the response to Weakness 1 above, there will typically be many ways to break down a task into smaller capabilities. Any breakdown that satisfies the principles above can give evidence about lack of goal-directedness.

Weakness 3. The only results are shown in the BlocksWorld environment.

The reason we choose the blocksworld environment is that it lets us formulate a range of different kinds of tasks and objectives in a unified environment. Thus we were able to define:

• the Build Highest Tower task (which mostly requires measuring),
• the Equal Towers task (which requires significant cognitive effort and execution skills), and
• the Falling Tower task, which requires perseverance in spite of adversity. So even though it is a single environment, we still test a range of different kinds of tasks.

Nevertheless, we do recognise that testing the principle in other environments would also be valuable. Please see the global comment describing another setting in which we’ve tested our goal-directedness metric.

Questions 1 and 2

See responses to weakness.

Question 3

Goal-directedness is a property of an agent and may well vary across tasks -- different tasks have different levels of complexity and require the use of different capabilities. Nevertheless, the fact that both Build Equal Towers and Falling Towers tasks, as well as the just implemented anagram task, have the same “winner” (Gemini-1.5-Pro), indicates there may be some level of generalization across tasks.