How FaR Are Large Language Models From Agents with Theory-of-Mind?

Thank you so much for your helpful feedback and suggestions. Our responses to questions are as follows:

[Main concern with this paper is contribution in light of Gandhi et al. 2023. doing the same]

Gandhi et al. 2023. proposes BigToM, which is an excellent benchmark for measuring ToM in LMs. While there is a surface-level similarity in the task setups (both contain “action” probing), there is an important distinction between action probing in BigToM and T4D: probing beliefs of others’ actions such as “What will A do?” (BigToM) vs. probing model’s own action decisions based on beliefs of others such as “Who should you provide help with?” (T4D). Specifically, in T4D, models have to treat themselves as situated agents and decide their own actions, based on potential reasoning about others’ mental states as well as predictions of others' actions. We argue that these are complementary directions, and both are important to understand LLMs’ capabilities related to ToM. In summary, T4D builds on previous benchmarks by focusing on models’ capabilities to determine actions for themselves based on mental state reasoning. We have added this in the extended related work section in the updated draft in Appendix A.

(Gentle note that according to the last question in FAQ from ICLR Reviewer Guide (https://iclr.cc/Conferences/2024/ReviewerGuide), “if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2023, authors are not required to compare their own work to that paper.” and “they may be excused for not knowing about papers not published in peer-reviewed conference proceedings or journals, which includes papers exclusively available on arXiv.” and Gandhi et al. 2023. Was released on arXiv on June 21, 2023).

[Controlled experiment with true beliefs]

Thanks for the great point! We did controlled experiments with true beliefs (i.e., the item stays in the same location) paired with all our experiments with false beliefs. We have added these results in the appendix F. Notably, since we are probing the model agent's action choices given observations, when all characters hold true beliefs, it is unclear to whom the model agent should help provide information. So, instead of assigning gold labels for the controlled set, we calculate the delta value between the false belief set and the true belief set. The higher delta indicates that the model is more inclined to help when there is a false belief, and the lower indicates that the model might not be using ToM to decide actions space (since it is similarly likely to choose the character with or without false beliefs). We found that the delta is very low (<5%) for all LLMs when tested 0-shot, and applying FaR helps increase the delta values to around 30%. This indicates that models do not tend to leverage mental state thinking in T4D, but FaR can help them locate relevant ToM inferences to determine the right action options.

[Human studies include 3 examples while models do not]

We tried prompting models zero-shot with 3 examples and observed no clear difference from Table 1 (+/-2%). As for humans, we reran human studies with a subset that we do not show any task examples and also found similar results (90% of instances reached 95% agreement on the correct answer).

[Could there be spurious correlations in the examples?]

ToMi was extended from an earlier work [1] that uses QA to probe ToM using sally-anne tests by removing biases from the character name mentions (discussed in Section 3 of the ToMi paper [2]). Furthermore, we included an observation-answer-only baseline without the questions and found close-to-random performance, confirming that there is no strong spurious correlation between the observation and the answers.

[Suggestion on figures and formatting]

Thank you very much for the constructive suggestions! We will be sure to address them in the next version of the paper.

[1] Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. Evaluating theory of mind in question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2392–2400, 2018.

[2] Le, Matthew, Y-Lan Boureau, and Maximilian Nickel. "Revisiting the evaluation of theory of mind through question answering." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Thank you so much for your helpful feedback and suggestions. We believe all of your concerns can be addressed within the response period. Our responses to weaknesses are as follows:

T4D Weakness:

[Clearer formulation of the multi-step inference process]

Thanks for the suggestion. Indeed, for models to succeed at T4D, they need to reason about utilities, ToM inferences, observations, etc. In Section 4.2 (especially Table 2), we break down the multi-step inference process from collected human-written rationales and defined three reasoning challenges: question decomposition that captures reasoning about utilities (who benefits from information the most is who needs it most to fulfill a goal), ToM inferences that capture reasoning about mental states (A has a false belief of a location of an item), and common sense assumptions that capture reasoning about observations (percepts). Moreover, we provide experimental results of the model's performance change when we give hints regarding each challenge. We will articulate this part more clearly.

[T4D which is based on ToMi suffers from the same limitations as ToMi]

ToMi is a large-scale and well-established benchmark for ToM (also echoed by [2]). We chose to apply T4D on ToMi to better compare with well-grounded related work and for the community to build on. Since we were starting from a new task, building on an existing dataset ensures we only change one variable at a time. Moreover, we acknowledged the limitations of ToMi in our paper by including generalization tests of modified story structure as well as Faux Pas in Section 6.4.

[Unsurprising that adding “hops“ leads to lower zero-shot performance.]

Instead of merely adding more hops of reasoning, we devised T4D in a principled way by changing the probing questions from others’ beliefs or actions to the model’s own action decisions based on inferences. Further, Section 4.2 discusses why this shift makes T4D more challenging: the models have to self-discover what to reason about instead of being given hand-holding inference questions as the original ToMi.

[Alternate probing methods]

Good suggestion! We tried generative LM probing in the early stage of designing T4D, i.e., asking models to complete a sentence and observing a similar trend. However, we decided on the multiple-choice format because providing options makes the task more well-defined because evaluating free-form generation without probability logit access is error-prone ([1]). Further, adding multiple choices also circumscribes the scope and the goals of the task -- open-ended generation leaves room for fantastical outputs, whereas we are interested in specifically probing the model's capabilities for T4D grounded in the situation.

[Related work of using theory of mind for planning]

Thanks for the pointers, we have added them in a new extended related work section in Appendix A due to space constraints. We would like to note that T4D differs from this line of work by exposing a key limitation of LLMs even with short-term social reasoning tasks: connecting inference to action is not trivial; we see this insight as orthogonal to ToM for planning and in a strategic environment.

[Other theory of mind benchmarks]

Newer benchmarks such as BigToM are very good studies measuring ToM in LMs. While there is a surface-level similarity in the task setups (both contain “action” probing), there is an important distinction between action probing in BigToM and T4D: probing beliefs of others’ actions such as “What will A do?” (BigToM) vs. probing model’s own action decisions based on beliefs of others such as “Who should you provide help with?” (T4D). Specifically, in T4D, models have to treat themselves as situated agents and decide their own actions, based on potential reasoning about others’ mental states and predictions of others' actions. We argue that these are complementary directions, and both are important to understand LLMs’ capabilities related to ToM. In summary, T4D builds on previous benchmarks by focusing on models’ capabilities to determine actions for themselves based on mental state reasoning.

(Gentle note that according to the last question in FAQ from ICLR Reviewer Guide (https://iclr.cc/Conferences/2024/ReviewerGuide), “if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2023, authors are not required to compare their own work to that paper.” and “they may be excused for not knowing about papers not published in peer-reviewed conference proceedings or journals, which includes papers exclusively available on arXiv.” and Gandhi et al. 2023. was released on arXiv on June 21, 2023, some other papers in the suggested list are also only on arXiv or released later than May 28, 2023).

[Probing method restricted to questions about helping others]

In 6.4, we also include results from T4D-Faux Pas tests, where we convert the original tests from [5] that ask questions of “Who committed a faux pas?” to probing action decisions about providing emotional support “Who should you provide emotional support?”. In future work, we plan to extend to more diverse domains, including scenarios from BigToM and others.

[Closed-sourced models]

We also probed Llama 2-70B on T4D and have included results in the Appendix F in the updated paper. We find a similar trend of models struggling on T4D (22%) significantly more than ToMi (63%).

FaR Method Weakness:

[Prompt for FaR is much longer and is much more task specific]

RE length: Prior work has shown that length can both hurt and help LLMs, ([3], [4]) and it is unclear if more complex prompts are always better. Further, we experiment with baselines such as ToT, which have similar length prompts.

RE task-specific: We show that FaR can generalize to other domains different from the original ToMi set in Section 6.4. Furthermore, through analysis in Section 4.2, we find that the key bottleneck of LLMs finding T4D hard is that without hand-holding inference questions such as “what are A’s mental states about X?”, models could not locate the relevant inference to decide an action. Thus, we designed FaR specifically to not leak the mental states and instead of proposing a generalizable reasoning structure about predicting and then reflecting (discussed in Section 5, first 2 paragraphs).

[Why not alternative such as two-step: first ask mental states, then determine action]

A key reasoning challenge in T4D is that models must self-discover that they should leverage mental state reasoning to decide actions. If asked in a 2-step way, the task will leak about "what to infer about" to LLMs. In contrast, FaR did not leak that models should reason about the mental states and provide a reasoning structure.

[Other prompting methods like tree-of-thought should be compared]

Zero-shot tree of thought is one of our baselines, as included in Figure 5, which has a similar number of words as FaR (for both input and output). As for self-critique, our method (FaR) already includes a flavor of self-critique in the Reflect step.

[Test on other domains]

We include story structure changes and T4D-Faux Pas generalization tests in 6.4, both of which show consistent improvements from FaR. In our pilot study, we also tested another widely-used ToM test: the smarties test, where the label of a container does not match the content, and found a similar trend in experimental results.​​

[1] Shapira, Natalie, et al. "Clever hans or neural theory of mind? stress testing social reasoning in large language models." arXiv preprint arXiv:2305.14763 (2023).

[2] Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13960–13980, Toronto, Canada. Association for Computational Linguistics.

[3] Fu Y, Peng H, Sabharwal A, Clark P, Khot T. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720. 2022 Oct 3.

[4] Madaan A, Yazdanbakhsh A. Text and patterns: For effective chain of thought, it takes two to tango. Findings of EMNLP, 2023

[5] Shapira, Natalie, Guy Zwirn, and Yoav Goldberg. "How Well Do Large Language Models Perform on Faux Pas Tests?." Findings of the Association for Computational Linguistics: ACL 2023. 2023.

Thank you so much for your helpful feedback and suggestions. Our responses to weaknesses and questions are as follows:

[Curious about what it will look like with different prompting methods, such as Chain-of-Thought and Tree-of-Though]

We did include comparisons with chain-of-thought and tree-of-thought prompting methods as shown in Figure 5 and explained in detail in Section 6.1. We also included the full prompts in Appendix D.

[FaR breaks the T4D process into three steps. However, the motivations for the breakdown have not been introduced in this work.]

The three steps are our attempts to analyze what makes T4D challenging. We arrived at the three reasoning challenges from collecting human-written rationales as explained in detail in Section 4.2 and Appendix C. Additionally, through experiments, we found that providing hints regarding each challenge results in very different LLM performance, which helped us speculate that the key reason why T4D is hard for LLMs is that models struggle to locate relevant inference evidence to determine their actions. FaR actually did not incorporate the 3 steps as it would leak the reasoning process. Instead, FaR proposes a generalizable reasoning structure with two steps: Foresee and Reflect that helps models to arrive at relevant inferences.