SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs

Thank you for your review! Here are some responses to your helpful comments:

> Importance of evaluating the ToM capabilities of LLMs?

See our general overall response above for examples of applications where failing on applied ToM would be critically problematic. For instance, having an AI personal assistant that cannot distinguish its own state of mind from that of its user and other independent individuals, would then fail to predict other’s behavior and not be able to highlight critical information in time. Furthermore, an AI judge that fails to appropriately consider whether one has a “guilty mind” could make poor judgments like blaming innocent people for another's harm, generally incorrectly judging someone’s actions as wrong or stupid (like purchasing moldy chips when they couldn’t see through the Pringles can to know it was moldy). We will highlight such motivating examples in the paper.

More generally, a fundamental challenge of modern LLMs is their opacity, i.e., our lack of understanding about how they behave and when they fail. This has been a major stumbling block and risk for more widespread LLM adoption. Our work develops a richer understanding of LLM behavior in the critical area of reasoning about other people’s beliefs, something that is fundamental to any conversational system. So as well as specific practical applications, addressing this fundamental problem with current LLMs in a general-purpose way is an important motivation for this work, and one where we contribute both new resources and new insights that we believe are both novel and impactful - we would be grateful if you could reassess your judgment on this. Again we will clarify this in the paper.

> Why does o1-preview's built-in inference-time reasoning tokens help?

The built-in inference-time reasoning tokens are akin to the chain-of-thought responses, although lengthier, indicating the model ruminating more. As noted in Appendix H, empirically we notice that the o1-preview model uses a lot more tokens than other models with CoT*. This further highlights the novelty of the gap our paper exposes - even this recently released model, using so many reasoning tokens to reason about simple 2 sentence stories, still shows a significant gap in explicit and applied ToM performance (see Table 2).

We do not have access to the specific o1 reasoning tokens due to the closed-access nature of the o1-preview model, but we analyze CoT examples across our interventions on models for which we can access the CoT output, discussing them in Lines 478-484, with detailed CoT outputs in Appendix I. Our analysis shows that more fine-grained human interventions/hints generally help the performance more by increasingly reminding models of key factors to consider. One hypothesis regarding o1-preview's built-in inference-time reasoning tokens being helpful in ToM reasoning is that they go through a longer reasoning process, which could potentially involve backtracking or self-questioning along the way (mimicking human intervention), leading to better performance.

> Human baseline?

The strict crowdsourcing requirements for SimpleToM ensured that each instance was labeled with unanimous agreement from three qualified workers. These workers independently verified and all agreed on the following:

• the key information in each story is something that Person X is unaware of
• the next likely “unaware action'' is appropriate if and only if Person X is unaware of the key information
• the “aware action'' is appropriate if and only if in the counterfactual situation where Person X is somehow aware of the key information

Requiring agreement among all 3 qualified workers on such questions minimizes ambiguity and ensures the high-quality nature of the labels in our dataset. Given the simplicity of the scenarios, we can expect a very high human baseline. It is unlikely that other annotators would significantly deviate from the unanimously agreed-upon answers obtained during the verification process.

We also note in the paper that with enough hints, the strongest model (Claude-3.5-Sonnet with CoT* + MS in Table 5) gets 97% overall performance, with consistently >95% on the 3 different types of questions. The close to 100% performance attained across the mental state, behavior, and judgment questions illustrates the high-quality answerability of the questions and is an indicator that the human baseline ceiling would be close to 100%.

Thank you so much for your response and for engaging in the discussion with us!

You are correct these are speculative examples. Note, though, this is not an application paper comparing AI products for deployment, but rather, a basic research contribution paper with both an innovative dataset and surprising findings about frontier LLMs. We believe there is significant value in fundamental research on LLMs within the community, particularly when addressing broad and fundamental questions such as “how well can LLMs reason about other people’s beliefs?”. This is not an esoteric question, given we live in a world where different people have different beliefs, and we hope our examples help convey this.

We follow the definition of explicit vs applied ToM as used literature (see lines 196-199), and cover both explicit theory of mind (conceptual knowledge about others’ mental states; i.e., via (a) mental state question) and applied theory of mind (the ability to apply that conceptual knowledge in different contexts; i.e., via (b) behavior and (c) judgment questions). While there has been much work testing LLMs on explicit ToM, their applied ToM capabilities are much less studied. In particular, our finding that LLMs can superficially report other people’s beliefs correctly (giving the impression of understanding their mental state), but then act as if otherwise (when predicting/judging behavior), has not been observed before (despite many earlier ToM studies), and we hope is a particularly interesting and impactful finding.

Thank you for your review of our paper and for your positive feedback on our work!

> Bolded words

We will remove some bolding in Section 5. Thank you so much for the suggestion!

> Interesting Llama-3.1-8B outlier in Table 2

Llama-3.1-8B’s score of 54.6% on behavior judgment is basically near random chance (50%), but this judgment score is indeed much higher compared to many other models where this is below random, including its bigger counterpart Llama-3.1-405B. This reveals the following insights about the Llama-3.1-8B model:

(1) It has less bias to being consistently wrong in the judgment task than the other models.

(2) Analyzing its performance on behavior prediction (BP) and behavior judgment (JU) reveals that despite this higher judgment score, there is much inconsistency within itself in the behavior-to-judgment reasoning chain. For instance, in 33% of the cases, the model predicts the behavior wrongly but inconsistently gets the judgment right; in 17% of the cases, it predicts the behavior correctly but still gets the judgment wrong. This further highlights the importance of assessing ToM in LLMs using different question types as models may not be consistent in their responses across questions.

> Human personality

Injecting a persona into the prompt would be an interesting follow-up experiment to see the impact of social factors on ToM performance, thanks for the suggestion!

PERSONAS = {
    "lawyer": "You are a partner at the law firm, recognized for their extensive knowledge of healthcare laws.",
    "worker": "You are a factory worker who doesn't trust the COVID-19 vaccine.",
    "atheist": "You are an atheist, philosophy lecturer who encourages open dialog about faith and belief systems.",
    "psychology_student": "You are a university psychology student who is currently studying creativity and personality.",
    "psychologist": "You are a clinical psychologist collaborating with the music therapist to provide holistic patient care."
}

Thank you for your review! Here are some responses to your helpful comments:

> Does ToM reasoning fundamentally differ from other types of knowledge?

ToM reasoning, being a form of social reasoning, is indeed inherently different from factual or logical reasoning because it requires making implicit commonsense inferences, making sense of social norms, context-dependent actions, and social dynamics, which are often not explicitly presented in models’ training data and not governed by fixed rules like factual or logical reasoning. E.g., it would be difficult to imagine someone writing something blatantly obvious like “In the supermarket, you can’t see through a Pringles can.” As the nature of the knowledge and reasoning is different, it is worthwhile to conduct experiments targeted at ToM capabilities and not use the performance of LLMs on factual, ontological, or arithmetic reasoning to infer their ToM capabilities.

While our work also involves the challenge of applying knowledge, it has broader implications as well. ToM is fundamentally about deeply understanding others' behavior and enabling effective interaction. In this sense, many existing studies have overlooked the bigger picture of what ToM truly entails and aims to achieve. Our work seeks to go beyond the question of whether LLMs possess conceptual knowledge of ToM, to the application of ToM-related reasoning when appropriate, thus expanding and deepening the scope of the discussion.

> Unique challenges that ToM reasoning poses for LLMs beyond needing CoT reasoning?

To close the gap between explicit and applied ToM, we show that more needs to be done than just using CoT reasoning. Table 5 shows that with a generic CoT prompt like “think step by step”, the behavior prediction accuracies are consistently more than 20% lower than mental state prediction, while judgment prediction accuracies are below random chance. Even with the task-specific CoT* (explicitly guiding models to think about the awareness of each person) gives behavior and judgment accuracies significantly below the mental state prediction (e.g., judgment scores of 77.8%-86.7% vs. mental state prediction scores all above 95%). To further close the gap, we found that we also need to combine CoT* with appending the relevant mental state question that is instance-specific (dependent on the story and character of interest in the story, not just generally the ToM task). Our work thus presents the surprising finding that very detailed interventions are needed to close the gap between explicit and applied ToM in the simple 2-sentence stories.

More generally, we think it is important to highlight these ToM-specific challenges, as real-world usage of LLMs rarely involves active customized "CoT prompting" by the user, rather the (late-stage) training of LLMs now involves learning to automatically generate appropriate reasoning chains without special prompting. The existing instruction training sets generally use examples from traditional reasoning tasks like math, not examples geared towards the kind of ToM reasoning. This lack of training partially explains the poor performance of even the fully trained frontier models, and hopefully, our work will inspire more ToM-focused training data in the LLM development pipeline.

> Why the focus on false belief setups?

To test LLMs’ ToM capabilities, the understanding that others’ mental states may be different from their own, like the classical Sally-Anne and Smarties tests, we focus our analysis on a series of false belief setups that distinguish between reasoning on the basis of the state of the world and reasoning on the basis of someone’s beliefs about the state of the world. We also have the instances of stories that were annotated as PersonX is “likely aware” of the key information from our filtering annotation process, and we will also release this subset as the control setup.

> Importance of evaluating the ToM capabilities of LLMs?

See our overall general review response above for some concrete examples of applications where failing on applied ToM would be problematic.

> Do behavior prediction and judgment require the same level of reasoning?

This is a great question! While these tasks both involve applied ToM, behavior judgment requires the LLM to first implicitly perform behavior prediction to infer what the likely action is, and then additionally compare the predicted behavior to stated behavior to judge whether the stated behavior is reasonable. Therefore, as illustrated in the inference graphs in Figure 1, behavior judgment requires an additional step of reasoning. We will update the paper to further clarify this difference in the reasoning required.