Measuring Goal-Directedness

Thanks for the review.

Lower bound. It’s true that our algorithm can only guarantee a lower bound in the unknown-utility case, but this is true whenever one performs SGD on a nonconvex loss function to estimate some quantity. With neural networks, we often get good estimates of the quantity regardless. We’ve added a note to clarify.

Misleading name. It’s true that MEG only measures the evidence for goal-directedness exhibited in the behaviour (with respect to the variables we measure), not necessarily the full goal-directedness of the agent. But this is generally true of tests: a high-school maths test cannot measure the full ability of a star mathematician, since there is limited opportunity to provide evidence for it while completing the test. Similarly, depending on the environment in which we measure MEG, we may see relatively little evidence for an agent’s goal-directedness. But we think it’s still fair to say that MEG is measuring goal-directedness, just as the test is measuring mathematical ability.

We should of course bear this point in behind, and think of MEG as measuring the goal-directedness with respect to a given environment distribution. We have updated the paper to clarify this.

Equation 1. The fact that the maximising policy obtains the same expected utility as pi is non-obvious but follows from Ziebart’s result. I’m not sure why we didn’t include a proof of this – we have included one now. Thanks for drawing our attention to it.

Just to further clarify Proposition 3.2, as it may still not be clear. The upper bound holds whether or not there exists a unique optimal (or anti-optimal) policy, but iff there exists such a policy, and pi is that policy, then there is equality in the upper bound. Do you think we should spell this out more explicitly in the statement?

We appreciate your extensive and careful review.

Motivation. Agreed emergent goal-directedness is a core AI safety problem. Updated to be more explicit.

Terminology. We agree our approach is not a unique interpretation of Dennett, and his view not the only one in the literature. We have changed "the literature" to "Dennett". Would “philosophically motivated” be preferable to “philosophically grounded”?

Mechanistic vs behavioural. Agreed our approach is behavioural rather than mechanistic. We think both approaches should be pursued. Behavioural approaches seem more tractable, but mechanistic approaches will generalise better off-distribution. We have added this point to the discussion.

Prop 3.1. The proposition is correct – MEG considers an agent goal-directed wrt a utility function both if it maximises it and if it minimises it. We have made the proof more explicit.

We admit this may be unintuitive. Arguably trying to pessimise a utility function is just as goal-directed as trying to optimise it. But on the other hand, perhaps MEG should be signed. We did consider this. Would the reviewer argue in favour?

Prop 3.2. The proposition is correct, but was stated ambiguously. We have updated it to say, “there is equality in the upper bound if and only if pi is a unique optimal (resp. anti-optimal) policy”. Is the meaning now clear?

Is unknown utility meg always high? There does alway exists a reward function to rationalise a (deterministic) policy. But this only applies to state-action or state-action-state rewards, not state rewards (ignoring the constant reward function). It’s the same with MEG: all (deterministic) policies are maximally goal-directed with respect to utility functions over <observation,action> pairs. But if the set of functions if over some downstream variable (e.g. states in an MDP), most policies will not be maximally goal-directed. We think MEG gets this right -- it's hard to say an agent is not optimising its own behaviour, but its easy to say whether it's optimising some variable out in the world.

Complexity of utility functions.

Any way of measuring goal-directedness needs a way to avoid the trivial result that all policies are maximally goal-directed. One option is indeed to penalise more complex utility functions. But MEG does it in a different way: by considering utility functions over specific sets of variables. If the variables considered don’t include the agent’s actions, then most policies will not be very goal-directed, for any complexity of utility function. So we think it’s best to present MEG without an additional complexity penalty, although we acknowledge that adding one may be natural. We have now mentioned this in the paper.

Scalability. MEG does require repeatedly solving an RL problem. MEG is about as scalable as MCE IRL, so we expect it to scale better than UTM-based approaches (Agents and Devices). But we have not demonstrated the scalability here, so we have now weakened the language. We agree that much usefulness depends on scaling. However, there is also value in conceptual clarification, and taking a step towards scalable methods.

Generalisation. Agreed distributional shifts are a key motivation. Rather than trusting the imitation policy to generalise correctly, a better approach could be to think of MEG as measuring goal-directedness on a given distribution, and to proactively seek out distributions to test on, e.g. where unknown-utility MEG is much higher than intended-reward MEG. This may let us detect goal misgeneralisation.

Prior work. We have added new sections in the appendix comparing to A&D and IRL. One key difference with IRL: IRL does not try to tell you how goal-directed a policy is, just what its goal is. Consider an agent that performs well on the intended reward by default, but badly under a distributional shift. IRL would just tell you that the utility function of the agent changes, but MEG can tell us whether the behaviour is strongly goal-directed on the new distribution, i.e. whether we have capability misgeneralisation (low MEG) or goal misgeneralisation (high MEG).

Does pursuing goals require a causal effect? This is a common view in philosophy (see section 3.1 of the SEP article on agency). But it’s not universal, e.g. evidential decision theory is concerned with pursuing goals even if one cannot causally affect them. We think that MEG can be generalised to an “evidential” version, but it’s beyond the scope. We have added a reference and a footnote.

KL div, cross entropy, Haarnoja.

• MEG is similar to the minimum KL with a maxent policy. You would remove the second term (the entropy of the uniform policy) and add a term for the negative entropy of the policy itself. The first change is just a translation. The second would mean that a less entropic way of achieving a certain utility would be considered more goal-directed than a more entropic way.

• The direction of the cross entropy gets us a maxent policy that well-predicts the agent’s policy, rather than vice versa. Reversing it would mean maximising the maxent policy's probability of the agent's highest probability action, i.e. ignoring other aspects of the policy.

• Haarnoja’s results are essentially a useful alternative formulation of Ziebart’s. We could probably reformulate our results along the lines of Haarnoja’s, but chose to focus on Ziebart.

Harm. We agree the relationship is not indirect. Probably we meant to include the word “measuring”, but this too now seems like an overstatement. We have removed the sentence.

Limitations. We have added the following to the limitations section: scalability; mechanistic vs behavioural approaches; more on distributional shifts. Has the rest been satisfactorily clarified?

Let us know if our response is enough to increase your score, or whether further changes would be needed. Thanks again.

Thanks for your review. We agree that the problem of measuring goal-directedness is of great significance and we’re glad you feel we have made a novel contribution here.

Societal implications. We agree this should be discussed in more detail. Thanks for the suggestion of an extended discussion in the appendix about the concern of bad actors using our ideas to create more dangerous systems. We have added a section in which we weigh various considerations, and then explain why we ultimately think it’s not a major concern.

Empirical demonstration. The contribution of this paper is primarily theoretical and conceptual, and so we decided to include experiments mostly for illustrative purposes. Investigating scalability, robustness, and practical applications (like detecting goal misgeneralisation) are promising directions for future work.

Question about the unknown utility setting. Good question. Every system you model will be goal-directed with respect to a utility function over its own actions, but beyond that, it’s not the case that by including enough variables in your causal model, you can always find a utility function towards which the system is goal-directed. In fact, we can apply Theorem 4.1 to this question, and say that if the agent is not very goal-directed with respect to some Markov blanket of variables surrounding its action, then it’s no more goal-directed with respect to any variable further away in the causal model.

Your river example is an interesting one. Depending on how you model it, it may indeed be the case that the river appears quite goal-directed. If we think of the river as being able to change to flow in any direction at various points along its path, but choosing to follow the path it does, then yes, it will have high goal-directedness with respect to its travel distance to the sea, as you suggest. This is in line with famous examples of very simple agents, such as a thermostat.

On the other hand, it’s debatable that you should model the river as “being able to” flow in any direction. In general, the smaller the set of actions the system has available, the lower its maximum goal-directedness, as we show in Proposition 3.2. Therefore a more reasonable way of modelling the river might lead to much lower goal-directedness. And if we put the river in a setting where some other system sometimes disrupted its path, then since the river would not adapt to overcome these disruptions, this would also lead to low goal-directedness.

The necessity of assuming a causal model. Something we perhaps under-emphasised in the paper is that if you have black box access to the environment or a simulator, then you don’t need an explicit causal model, you just need the ability to measure the variables you’re interested in under various different policies. So that’s one way in which you can avoid worrying about hidden confounders. We have updated the paper to make this clearer.

Of course, we still need to be measuring the important variables. Interpretability-based approaches to measuring goal-directedness may have an advantage over our behavioural approach here. We have added this point to a discussion of the pros and cons of taking a behavourial vs mechanistic approach to measuring goal-directedness in the paper.

The second part of your question is about what happens if an agent is goal-directed with respect to variables we can’t measure because they are internal to the agent. There are two cases here: either the agent is also (perhaps instrumentally) goal-directed with respect to external variables, or it isn’t (as in the case of “wire-heading”). In the first case, we may be able to measure this by considering the external variables. The second case is of less concern to us, since our motivation is about the risks arising from agents pursuing goals in their environment.

Thanks again for the thoughtful review.

Thanks for the review.

Requiring a causal model. A full causal model is not strictly necessary for computing MEG. What is needed is a simulator where you can measure the variables of interest under different policies. We have updated the paper to make this clearer. That said, investigating how MEG can be applied in practice is indeed an important direction for future work.

Robustness to distributional shifts. One simple idea is to proactively seek out distributions where MEG is especially high, either by hand or by using MEG as an optimisation target. It would be especially interesting to find distributions such that goal-directedness with respect to the intended reward function is low, but goal-directedness with respect to some other reward function is high. This could be a way to detect goal misgeneralisation (https://arxiv.org/abs/2105.141110).

How MEG can fail. One failure mode is that if you measure goal-directedness with respect to the agent’s own actions (or more correctly, state-action pairs in an MDP, or observation-action pairs in a CID), then the agent will appear maximally goal-directed regardless of its policy. This means that if you measure the goal-directedness of an LLM with respect to the text it generates, for example, it won’t tell you much. Instead you need to measure the goal-directedness with respect to some downstream variable (such as a user’s response, or the outcome of a task). We have updated the appendix to mention this example.

Can similar policies get very different MEG?. Yes, at least under some reasonable notions of policy-similarity. Consider an example where two policies in an MDP agree in every state except the start state, where one policy takes action a1 and the other takes a2. If a1 leads to a part of the state space where the policies are both very goal-directed, and a2 leads to one where they are not, then there can be an arbitrarily large difference in goal-directedness, despite the policies agreeing in almost every state. This underlines that MEG measures the goal-directedness on a particular distribution. We have added this example to the appendix.

The case where the causal model is only partially known. As mentioned above, we can tackle this case as long as we can simulate the environment under different policies. The simulation may only partially specify a causal model, as it may not allow interventions on non-policy variables. It may be interesting to explore other forms of partial knowledge.

How MEG relates to agency. Agency is an overloaded term with a range of meanings and connotations. Goal-directedness is often considered a key component of agency. For example, Dennett's intentional stance only applies to systems pursuing goals in some sense, and Dung (https://philarchive.org/rec/DUNUAA) argues that it underlies other aspects of agency such as autonomy, empowerment, and intentionality. Some of these notions are still vague, but it's an interesting line of future research to try to make these other aspects precise, and formally and/or theoretically relate them to the notion of goal-directedness we present here.