Thanks for your feedback! Below we respond to your comments, and point you to the general response in which we address common feedback from all reviewers.

We will update the explanation surrounding the definition of II-MAIDs to make it clearer how we formalise higher-order beliefs.

Regarding your concerns about the realism of infinite-depth belief hierarchies: We think this is not a problem for two reasons. First, our framework can be used for finite-depth hierarchies. Recursive beliefs “bottom out” when an agent ceases to model other agents as having beliefs, and instead treats them as probabilistic parts of the environment. We will clarify this in the paper. Second, we should allow infinite-depth hierarchies to avoid a stark departure from the game theory literature: concepts such as Nash equilibria and common knowledge are closely tied to infinite-depth belief hierarchies.

Unfortunately we do not understand your comments in the “Questions” section. Can you please clarify what you were asking?

“The limitation section begins with a number of upbeat statements that would better be placed in the conclusion or parts of the abstract.” This section is “Conclusion and Limitations”, but we will reformat this part of the paper to increase its clarity.

We agree that the lack of a better solution concept is the primary limitation. However, we focus on laying the theoretical foundation for our setting, and believe developing a novel solution concept is out-of-scope.

LM Demonstration

Our PDF and the text below describe how our framework applies to the classic Sally-Anne false belief task from ToM literature. (We use the name Bob instead of Sally for notational reasons.)

• Fig. 1 (in the new pdf) shows the II-MAID, $\mathcal{S}$, of the Bob-Anne false belief task, Fig. 1 (a) represents Anne and Bob's beliefs and Fig. 1 (b) is the LM task. $\mathcal{S} = (\mathbf{N}, S^*, \mathbf{S})$ where $\mathbf{N} = \{A,B,G\}$ are the agents Anne ($A$), Bob ($B$), and GPT-4o ($G$).

• First, consider Anne's beliefs about the game. Let $S^A = (\mathcal{M}^{S^A}, (P_i^{S^A})_{i\in \mathbf{N}})$ be the subjective MAID that Anne believes in with certainty, $P_A^{S^*}(S^A) = 1$. (Recall that the notation $P_i^S(S')$ refers to the probability that agent $i$ in subjective MAID $S$ assigns to subjective MAID $S'$.) Anne's beliefs about the world are represented by the MAID $\mathcal{M}^{S^A}=(\mathcal{G}_A, \bm{\theta}_A)$, shown in Fig. 1 (a). The causal graph $\mathcal{G}_A$ includes the decisions of both Anne and Bob, their utilities, and the ball position $L$. We suppose that Anne gets utility for correctly locating the ball and, according to Anne's beliefs, Bob also wants Anne to find the ball. Additionally, Anne has beliefs about Bob's beliefs, and is certain that Bob has the same beliefs as Anne herself, i.e., $P^{S^A}_B(S^A)=1$.

• Now consider Bob's beliefs. Suppose Bob and Anne share beliefs about the causal structure in Fig. 1 (a). However, whereas Anne believes the game is cooperative and Bob gets utility if she finds the ball, Bob actually gets utility if Anne looks in the wrong location (so the agent's MAIDs differ only in the CPD parameter for Bob's utility, $\theta_{U^B}$). Bob has correct beliefs about Anne's beliefs, i.e., $P^{S^B}_A(S^A)=P^{S^*}_A(S^A)=1$.

• We can represent the LM prediction task, which is the objective $\mathcal{M}^{S^*}$ shown in Fig. 1 (b) (excluding the red arrows). This simply extends the original Bob-Anne MAID with decision and utility variables for the LM. The LM observes the information in the prompt, i.e., where Anne puts the ball ($D^A$), where Bob moves it ($D^B$), and where the ball ends up ($L$). We suppose that the LM gets utility for correctly predicting where Anne looks for the ball, e.g., because it is fine-tuned to correctly answer questions.

• What does GPT-4o ``believe" Anne believes? We argue that it is reasonable to represent GPT-4o's subjective model of this situation as the MAID in Fig. 1 (b) because it adapts its behaviour to correctly answer the questions. In fact, Richens ([I] in the general comment) shows that robust adaptation requires a causal model of the data generation process, which, in this case, includes the other agents. GPT-4o correctly predicts Anne's action (Table 1. (b)), indicating that, GPT-4o has the correct model $P^{S^*}_G(S^*)=1$, and in particular correctly models Anne's beliefs $P^{S^*}_G(P^{S^*}_A(S^A)=1)=1$. Additionally, GPT-4o is able to adapt its answer when we posit different beliefs for Anne (Table 1. (c)) -- suggesting that it is able to reason about how other agent's beliefs influence their decisions.

• So in full, $\mathcal{S} = (\mathbf{N}=\{A,B,G\}, S^*=(\mathcal{M}^{S^*},(P_A^{S^*},P_B^{S^*},P_G^{S^*})), \mathbf{S} = \{S^A,S^B,S^*\})$.

• The behaviour exhibited by the three agents is not a Nash equilibrium -- Anne does not play a best response as she falsely believes Bob will not move the ball. However, supposing GPT-4o gets utility for making correct predictions, then GPT-4o's correct prediction of Anne (Table 1. (b)) is a best response to the behaviour of the other agents. Furthermore, GPT-4o's behaviour is subjectively optimal with respect to the II-MAID representation in Fig. 1 (b) and how Anne intuitively acts given her beliefs.

• Human children often incorrectly predict that Anne will look in the box because they are not yet capable of sophisticated ToM. One way to capture this in our theory is to model the children as believing that the red arrows in Fig. 1 (b) are part of the task. That is, they believe that other agents have access to all the same observations as they do.

Thanks for your helpful feedback on the paper – we are glad you appreciated our solid mathematical contribution. Please see our general response which addresses a number of shared concerns.

“the proposed II-MAID framework appears overly complicated for modeling Theory of Mind“

Whilst we agree that II-MAIDs are pretty heavy formal machinery, we think they are substantially simpler than previous frameworks! For instance, MAIDs are a much more compact and intuitive representation than EFGs (compare Fig 1 vs Fig 2 in the paper). Additionally, we think II-MAIDs are simpler and more elegant than previous representations of ToM in influence diagrams, such as NIDs [11], whilst also being more expressive / general.

“The paper introduces numerous assumptions and definitions without clear explanations which hinders the readability. As a non-expert in the field, some details in the paper are difficult to read.”

We appreciate that the paper is technically demanding for a non-expert. We will edit the paper to include clearer explanation and connection to the examples to improve the readability. We note that both other reviewers commented positively on our presentation and accessibility.

“Could the authors elaborate on how this framework might be scaled up?”

II-MAIDs are an extension of the literature on causal and probabilistic graphical models, such as Bayesian networks and influence diagrams. These models have been adopted widely in domains such as diagnostics [A, B], robotics [C], risk analysis [D], and many more areas [E]. MAIDs in particular have been shown to have computational benefits over other game representations [15]. Additionally, if the task of specifying the model is too complex for humans, the causal structure and the parameters of the distributions can be learned from data [F].

Whilst our examples are simple for pedagogical reasons, we appreciate that it is not obvious our framework can be applied to more complex real-world scenarios to someone unfamiliar with this literature, and we will expand our discussion to reflect this.

“Given that Theory of Mind is fundamentally about real-life interactions, would it be possible for the authors to provide some experiments or toy examples to illustrate the key concepts?”

Please see our new one-page pdf and our response to reviewer VEtx, where we include a new demonstration of how our framework applies to the standard Sally-Anne false-belief test from the literature evaluating ToM in LMs. To the best of our knowledge, no previous work has presented a formal model of this task.

[A] Richens, J., Lee, C., & Johri, S. (2020). Improving the accuracy of medical diagnosis with causal machine learning. Nature Communications, 11. https://doi.org/10.1038/s41467-020-17419-7.

[B] Pingault, J., O’Reilly, P., Schoeler, T., Ploubidis, G., Rijsdijk, F., & Dudbridge, F. (2018). Using genetic data to strengthen causal inference in observational research. Nature Reviews Genetics, 19, 566 - 580. https://doi.org/10.1038/s41576-018-0020-3.

[C] Hellström, T. (2021). The relevance of causation in robotics: A review, categorization, and analysis. Paladyn, Journal of Behavioral Robotics, 12(1), 238-255.

[D] Cox Jr, L. A., Popken, D. A., & Sun, R. X. (2018). Causal analytics for applied risk analysis. Cham: Springer International Publishing.

[E] Pourret, O., Na, P., & Marcot, B. (Eds.). (2008). Bayesian networks: a practical guide to applications. John Wiley & Sons.

[F] Glymour, C., Zhang, K., & Spirtes, P. (2019). Review of Causal Discovery Methods Based on Graphical Models. Frontiers in Genetics, 10. https://doi.org/10.3389/fgene.2019.00524.

Thanks for your feedback! We hope that the global response addresses your concerns regarding how this work relates to the literature on safe ML.

ToM literature

Thank you for these references to the ToM literature – this is extremely useful and we will update our related work section to reflect this literature. See below a draft of additions we would make to our discussion.

Many previous works [A, C, E, G] have designed AI systems that can model the mental states of the humans or other agents with whom they are interacting. These models achieve more user-tailed dialogues [A, C], efficient plan acquisition [E], and better strategies in multi-agent tasks [G]. Other works [H, I, J] train systems that engage in higher-order reasoning. They make better decisions by reasoning about a human’s model of its own future decisions [H], generate more user-tailored explanations of decisions [I], and better model an opponent’s values [J]. An early Q-learning-based method [AN] allows for recursive reasoning of arbitrary fixed depth.

Much effort has been spent evaluating the ToM capabilities of LLMs. Early works [K, L, M] found strong performance from frontier models on false belief tests. More recent works found that these models struggle with inferring second-order beliefs [Q], detecting faux-pas [R, Z], adversarial versions of classic tests [S], and complex versions of false belief tests [T]. This indicates that early works were too optimistic, perhaps relying on spurious correlations and shortcuts [S, V]. II-MAIDs provide a rigorous formalism for evaluating LM ToM.

As you highlighted, our lit review missed some important existing game theoretical models for higher-order beliefs. Recursive Modeling Method (RMM) [AA, AB] models agents that may be uncertain about aspects of other agents’ models, including their payoff function. Beliefs about other agents are represented in a hierarchical structure. Unlike with II-MAIDs, full observability of the state is assumed, and the depth of recursive reasoning is always finite.

Interactive POMDPs (I-POMDPs) [AE] generalise RMM by allowing for partial observability of the state, and generalise POMDPs by allowing for an agent to have beliefs about models of other agents. Bayes-Adaptive I-POMDPs (BA-IPOMDPs) [AM] allow for agents to update beliefs about transition and observation probabilities throughout an episode. II-MAIDs generalise I-POMDPs by dropping the assumption of Markov transition dynamics and allowing for uncertainty about all aspects of the game, including the number of other agents playing, action spaces, the existence of certain decision nodes, etc.

Questions:

We agree that our framework does not capture broader aspects of ToM and will update our paper to reflect this (as discussed in the general response). We hope that this question is sufficiently addressed by our discussion of ML safety in the global response, and by the new LM demonstration we provide.

[A] Rabinowitz, N., et al. (2018, July). Machine theory of mind. In International conference on machine learning (pp. 4218-4227). PMLR.

[C] Qiu, L., et al. (2022, September). Towards socially intelligent agents with mental state transition and human value. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue (pp. 146-158).

[E] Bara, C. P., et al. (2023). Towards collaborative plan acquisition through theory of mind modeling in situated dialogue. arXiv preprint arXiv:2305.11271.

[G] Cross, L., et al. (2024). Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models. arXiv preprint arXiv:2407.07086.

[H] Çelikok, M. M., et al. (2019). Interactive AI with a theory of mind. arXiv preprint arXiv:1912.05284.

[I] Akula, A. R. et al. (2022). CX-ToM: Counterfactual explanations with theory-of-mind for enhancing human trust in image recognition models. Iscience, 25(1).

[J] Yuan, L., et al. (2021). Emergence of theory of mind collaboration in multiagent systems. arXiv preprint arXiv:2110.00121.

[K] Bubeck, S., et al. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.

[L] Holterman, B., & van Deemter, K. (2023). Does ChatGPT have theory of mind?. arXiv preprint arXiv:2305.14020.

[M] Brunet-Gouet, E., Vidal, N., & Roux, P. (2023, September). Can a Conversational Agent Pass Theory-of-Mind Tasks? A Case Study of ChatGPT with the Hinting, False Beliefs, and Strange Stories Paradigms. In International Conference on Human and Artificial Rationalities (pp. 107-126). Cham: Springer Nature Switzerland.

[Q] Ma, Z., et al. (2023). Towards a holistic landscape of situated theory of mind in large language models. arXiv preprint arXiv:2310.19619.

[R] Strachan, J. W., et al. (2024). Testing theory of mind in large language models and humans. Nature Human Behaviour, 1-11.

[S] Shapira, N., et al. (2023). Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763.

[T] Borji, A. A categorical archive of chatgpt failures (2023). arXiv preprint arXiv:2302.03494.

[V] Sap, M., et al. (2022). Neural theory-of-mind? on the limits of social intelligence in large lms. arXiv preprint arXiv:2210.13312.

[AA] Gmytrasiewicz, P. J., et al. (1991, August). A Decision-Theoretic Approach to Coordinating Multi-agent Interactions. In IJCAI (Vol. 91, pp. 63-68).

[AB] Gmytrasiewicz, P. J., & Durfee, E. H. (1995, June). A Rigorous, Operational Formalization of Recursive Modeling. In ICMAS (pp. 125-132).

[AE] Doshi, P., & Gmytrasiewicz, P. J. (2011). A framework for sequential planning in multi-agent settings. arXiv e-prints, arXiv-1109.

[AM] Ng, B., et al. (2012). Bayes-adaptive interactive POMDPs. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 26, No. 1, pp. 1408-1414).

[AN] Yoshida, W., et al. (2008). Game theory of mind. PLoS computational biology, 4(12), e1000254.