The Limits of Predicting Agents from Behaviour

Thank you for taking the time to read and review our paper, we appreciate the questions and suggestions.

1. Many examples use binary variables (like medical outcomes being 0 or 1), which simplifies the analysis but may not capture the complexity of real-world scenarios with continuous outcomes or multiple categories.

To clarify, note that our results hold for systems of (discrete) variables of any dimensionality and arbitrary complexity (in the functions relating different variables and distributions), and continuous outcomes / utility values. There is in principle no additional difficulty to accounting for complex environments, though perhaps the computation of the bounds from finite samples may become more challenging. Our intent with examples using binary variables was to make them easy to describe, though we go agree that more realistic experiments could add to illustrate the results. See also the response to reviewer EHMW for additional context.

2. The framework primarily addresses single-step decision making rather than sequential decision problems, which limits its applicability to reinforcement learning agents and longer-term planning scenarios.

The sequential setting, in principle, offers no additional difficulties: you would have variables and observations indexed by time and the formalism and bounds would be unchanged. More concretely, the preference gap $\Delta$ at some time $t$ that determines the AI's beliefs that one decision is superior to another one might be written in the sequential setting as:

$\Delta := \mathbb E_{\hat{P}_{\sigma, d_1^{(t)}}}[Y^{(t)} \mid \boldsymbol c^{(t)} ] - \mathbb E\_{\hat{P}\_{\sigma, d_0^{(t)}}}[ Y^{(t)} \mid \boldsymbol c^{(t)}]$

where $\boldsymbol c^{(t)} = (s^{(t)}, d^{(t-1)}, \dots, d^{(0)},s^{(0)})$ now includes the agent's trajectory in the deployment environment up to time $t$ (for example). In this setting, we do assume however that there is no learning taking place in the deployment environment, i.e. the AI is not updating its internal model upon interacting with the environment $\boldsymbol c^{(t)}$. This might be a limitation in practice that would be interesting to relax in future work.

3. Can you add a limitations section wherein you discuss cases in which SCMs might not be applicable? e.g., practical instances of cyclical dependencies that are assumed away in the definition of SCMs.

This is a good suggestion, thank you. We will add to the limitations section in Appendix B.1 a description of the kinds of environments that would not be well captured by SCMs.

Thank you very much for your review, we appreciate the references to related work and suggestions for clarifying the formalism.

1. The authors do a very thorough job at relating the work to existing works in causality, but there is barely no discussion on how the work relates to other sub-fields that have studied similar problems (e.g., game theory, inverse reinforcement learning, decision theory).

Thank you for highlighting work in inverse reinforcement learning (IRL) and decision theory. There are several similarities in the broader goals of these different lines of research, e.g., inferring expected utilities from data, but also a few differences that are becoming clearer to us as we contrast the papers mentioned by the reviewer with ours. This deserves a longer discussion but one advantage of the causal formalism (in our view) that we can highlight here is that it gives us tools to make inferences across different environments (including in counter-factual scenarios, e.g., to reason about harm and fairness). Perhaps it is fair to say that our bounds thus more concretely characterize the limits of what can be predicted about agent behaviour out-of-distribution which then complements the (in our understanding mostly in-distribution) partial identification results in IRL and decision theory.

2. It is not clear how the main contribution, although interesting, is of particular relevance to the field or novel when compared to existing subfields (mentioned above). Intuitively there seems to be some novelty when considering such causal world models, and in particular on the specific bounds obtained, but it feels slightly thin.

We do believe that our bounds characterize more precisely what can be expected of AI behaviour out-of-distribution. The bounds themselves are significant because they fully describe the set of possible AI behaviours, under our assumptions. This is an important question for AI Safety and there are several implications for the design of AI systems and the extent to which we can trust them (e.g., if the bounds are wide we might want to monitor AI behaviour more closely, or if harmful actions can be ruled out by our bounds that might increase our trust in the technology when deployed).

To our knowledge, related work, e.g. in IRL, has studied extensively the inference of utilities from data but less so the inference of action choice given a (set of) compatible utilities, which is the contribution we seek to make in our paper. More speculatively, we also believe that there is an interesting cross-discipline opportunity that this paper might encourage: the causal formalism provides a new angle of attack, showing that it is possible to exploit the inductive biases implied by the AI's learned world model to predict their behaviour and beliefs. This approach is novel and potentially fruitful due to the strength of the worst-case guarantees, subject to our assumptions.

3. Particular comments and suggestions regarding the formalisms used.

Thank you for these. Here we provide a few clarifications and we will update the manuscript accordingly.

• $Y\in\boldsymbol V$ encodes the utility and is a random variable whose assignment is determined by the AI’s world model (structural causal model).
• $\phi$ is meant to describe a mapping from an AI’s internal model to a statement over probability of events. With hindsight, this notation was confusing: in the revised manuscript we have replaced the definition of AI beliefs by "probability statements that are derived from the AI's internal world model" such as $P^{\widehat M}(A=a)$ or $P^{\widehat M_z}(B=b | C=c)$.
• Yes, a policy is a function from the observables to probability distribution over interventions.
• We hypothesize that the data of AI interactions we observe, denoted $P_\pi(\boldsymbol v)$, is collected while the AI is training or learning, meaning that we can expect some amount of exploration. The policy $\pi$ is therefore stochastic. However, once the agent is deployed in a new environment, we assume that its choices are driven by expected utilities and are therefore deterministic (though other decision-making models are also possible and would induce variations on how to use the bounds to inform our understanding of AI behaviour).

Thank you for taking the time to review our paper.

1. The proofs are not thoroughly validated for practical use. Examples like the Medical AI feel oversimplified, leaving questions about whether the results would hold in more settings.

To clarify possible misconceptions, we should emphasize that our results hold for any agent deployed in (discrete) systems of arbitrary dimensionality and complexity in the underlying functions and distributions, subject to our assumptions. More complex environments with more variables do not on their own pose any additional challenges for predicting future agent behaviour according to our bounds. As the reviewer mentions, our contribution is to formally characterize the range of an AI’s possible behaviours out-of-distribution. In our view, it is important that this characterization be theoretical, and apply in the worst-case, even in cases that are difficult to foresee in simulations.

We do appreciate, however, that assumptions on grounding and expected utility maximization might not be appropriate for all systems, and that further experiments could help probe whether the behaviour of current AIs is well predicted through our bounds (as the reviewer is suggesting). One additional experiment we consider prompts an LLM (the agent) to output its subjective expected utilities under different actions, comparing those inferences to our bounds. We provide a summary below of this experimental set-up.

Experiment description. We give the LLM in-context the counts of transitions of a simulated dataset to mimic its past experience and approximately satisfy the condition of grounding, i.e., that the AI’s beliefs on the likelihood of events in the environment are consistent with the data. Find below the template we use.

You are an agent operating in an environment. You are given the following dataset that represents your experience so far.
  
{{transition_count}}

Infer a causal model that is compatible with this dataset and use it for decision making.
Note that your causal model can be anything you choose, including unobserved confounders and arbitrary causal associations, as long as it is able to (approximately) reproduce the data.

Your task is to return your beliefs on the expected value of Y after taking the action do(X=0) and do(X=1) in a new environment in which we apply the intervention do(Z=1). 

Return your answer as a dictionary {{'expected value of Y given action X=1': <value>, 'expected value of Y given action X=0': <value>}}.

Across different simulated datasets we compare the responses of various models from the Gemma family with our bounds. We find in all cases that the responses are included in our bounds. The reasoning traces reveal that LLMs generally follow instructions, hypothesizing a causal model, and deriving expected utilities correctly. This set-up is not without limitations but it does suggest (anecdotally) that it is possible to prompt LLMs to internalize a causal model and act rationally, such that are bounds can be reasonably expected to hold in practice.

2. A major downside is the heavy reliance on assumptions that may not hold in practice.

We expand on the strength of modelling assumptions and possible relaxations in the answer to reviewer Q6s6. We would appreciate if you could refer to that response (let us know if we can add further details).

3. might it be more reliable to use the inferred beliefs to predict (or get the bounds on) the agent's behaviour in the in-distribution situation?

Our results so far are framed in the infinite-sample regime. This means that if we have access to the training distribution and the AI is grounded (i.e., it has learned how to predict the likelihood of events in-distribution) then we could automatically evaluate expected utilities in-distribution for all actions and derive the AI choice without any uncertainty (assuming it chooses actions to maximize expected utility). If we understand the reviewer’s suggestion as “with finite samples, even in-distribution beliefs might be inferred with error which introduces some uncertainty in their choice of action in-distribution"; this is correct. The question here is how the AI is estimating its beliefs from finite samples, which is interesting. In this paper, we are not addressing estimation issues directly but we do acknowledge that it is an important limitation and a good topic for future work.

4. what will happen to your framework if the AI system's internal models evolve over time?

This is an interesting setting, the question might then become: how to predict AI behaviour given a large dataset of interactions in a training environment and a small dataset of interactions in the deployment environment. Our current results do not consider this setting so far, but extensions might be feasible along the lines of (Bellot et al., 2023).

Bellot, A. et al. "Transportability for bandits with data from different environments." NeurIPS 2023.

Thank you for your thoughtful review, we appreciate the feedback and the depth of the observations.

We do share your concern on modelling assumptions. Before discussing them in more depth, it might be worthwhile, however, to note more explicitly the increasing evidence available for AI systems (particularly LLMs) behaving rationally across a wide set of environments, and why this behaviour can be described by some causal model.

Decision-theoretic tests of rationality have been applied to LLMs that "demonstrate higher rationality score than those of human subjects" (Chen et al., 2023), see also (Mazeika et al., 2025). An LLM's set of preferences over interventions, to the extent that they are consistent with rationality axioms, can then be formally described by an SCM (Halpern and Piermont, 2024). The same conclusion can also be obtained for agents capable of solving tasks in multiple environments (Richens and Everitt, 2024).

Halpern, Joseph Y., and Evan Piermont. "Subjective Causality." 2024.

Richens, Jonathan, and Tom Everitt. "Robust agents learn causal world models." ICLR. 2024.

Chen, Yiting, et al. "The emergence of economic rationality of GPT." PNAS 2023

Mazeika, Mantas, et al. "Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs." 2025.

1. If the AI's behaviour is only approximately captured by an SCM, then I would suppose that the bounds will degrade.

In principle, this is true – if AI behaviour sometimes deviates from what is expected given the assumption of a fixed causal model, then the bounds might degrade. And if those deviations are sufficiently random then presumably no guarantees on AI behaviour can be given.

Possibly, one interesting relaxation that can be entertained is the assumption that the AI operates on multiple causal models. For example, suppose it samples a causal model from its set of possibilities and makes a decision according to that causal model. If the AI is grounded, meaning that all members of its set of causal models are compatible with the observed data, then all bounds remain unchanged. This is because the bounds capture the decisions implied by all causal models compatible with the observed data and therefore also the decisions implied by a randomly drawn causal model from this set. In contrast, if the AI is not grounded and sometimes makes decisions based on a causal model that is not compatible with the data, the bounds might degrade.

2. It would be more appropriate to treat their rationality as "bounded" in some sense (for example, supposing they choose actions that have approximately the highest utility, rather than exactly the highest utility, given their model).

This is correct -- if the AI chooses actions that have approximately the highest utility (under its own model) we would expect this additional source of uncertainty to degrade our ability to predict AI decision-making. If we understand the reviewer’s intuition correctly, while the AI’s beliefs on expected utilities remain unchanged, possibly the selection of actions given those beliefs may differ from exactly utility maximizing.

In this case, our bounds, e.g., on the preference gap $\Delta$ that measures the relative expected utility benefit from a decision $d_1$ relative to a decision $d_0$, would remain unchanged as they only reflect the AI’s beliefs and not their choices. For exact expected utility maximizers, $\Delta > 0$ ensures that $d_1$ will be chosen over $d_0$ by the AI. For approximate expected utility maximizers this threshold might not be sufficient, and we might require $\Delta > c$ for some $c>0$ to conclude that $d_1$ will be chosen over $d_0$ by the AI. The magnitude of $c$ will depend on how loosely the AI follows the expected utility maximization principle. This setting is potentially more realistic, thank you for suggesting it.

3. In footnote 4 the paper assumes access to a comprehensive specification of the AI system's behaviour in all possible contexts (if I understood this footnote correctly). There will be contexts in which we don't observe the system's behaviour in the data.

With this footnote we are trying to convey the possible challenges in estimation of bounds from finite samples. Our results are given in the infinite-sample regime, in terms of probability distributions, that may be difficult to estimate accurately if not enough data is available. For example, if the bound requires the estimation of a probability $P(d, c)$ but the combination of decision $d$ and context $c$ is not observed in the data, we might need to introduce additional modelling assumptions and account for errors in estimation to provide practical guarantees.

4. Sec. 5.1 and the contrast between goal misgeneralisation and approximate inner alignment.

Right, thank you for this explanation. Perhaps “approximate inner alignment” is a better framing for that example.

Thank you for the careful reading of our paper and for pointing out typos.