Towards Machine Theory of Mind with Large Language Model-Augmented Inverse Planning

Dear Reviewer HwkP,

Thank you for your thorough and helpful evaluation of this work. We agree with you that a more thorough evaluation of LAIP is merited, and we respond to you point by point below.

The authors overlook the discussion and comparison of a closely related method presented in the paper "MMToM-QA: Multimodal Theory of Mind Question Answering." This work also employs language models for scalable Bayesian inverse planning, generating likelihoods of different actions under various hypotheses in a similar way. It also concludes that Bayesian inverse planning with smaller language models can outperform larger models, albeit on different tasks.

Thank you. We agree that the method and the technique employed in Jin et al. (2024) is very relevant to our discussion, and will discuss this work in greater detail as a relevant piece of related work.

The authors may consider including additional baselines, such as Sim-ToM [2], SymbolicToM [3], along with similar methods like BIP-ALM [1] and LIMP [4]. These approaches have shown significant improvements in LLM reasoning on Theory of Mind benchmarks.

We agree that including additional baseline methods to compare our performance against would fortify the evidence we provide. Given our current experiments focus on tracking agents’ goals across time in text-based contexts, we feel that Sim-ToM and Symbolic-ToM, which primarily track agents’ true, false, or second-order false beliefs, are less well-suited as baselines for our current experiments. Likewise, MuMA-ToM only accepts multimodal input. However, in our response to Reviewer Xehq, we show preliminary results comparing our results to ReAct, but we also plan to include BIP-ALM as a baseline comparison in addition to Reflexion.

Although the experimental setup follows a classic design, it appears somewhat simplistic and limited. The proposed method aims to serve as a general approach to machine Theory of Mind, so it should be tested in more complex, real-life-like scenarios that involve inferring different human mental states (e.g., beliefs, goals, plans, desires). It would be beneficial if the authors tested their method on at least one additional ToM benchmark to further demonstrate its effectiveness.

We agree with the reviewer that a more complex ToM benchmark would serve as a more rigorous test of the model. To that end, we have adapted LAIP to the MMToM-QA benchmark (Jin et al., 2024). Our preliminary results indicate that LAIP is well-suited to this task, placing 94.8% of the probability mass on the correct answer on a task in the “goal given true belief” condition. However, we plan to run LAIP on the full suite of results in order to allow us to more directly compare it to other baselines, including BIP-ALM.

[cont...]

Dear Reviewer HwkP,

Thank you again for your review. We have written our general response here where we outline the changes to our updated manuscript. We also summarize below the relevant changes we have made to address your concerns.

We expanded our discussion of the related work to include several additional works, including BIP-ALM. Additionally, in order to address your concerns about limited baselines and benchmarks, we added four additional baselines to Study 1, and we tested LAIP against the goal inference questions of the MMToM-QA benchmark, which we found outperformed other models, including BIP-ALM, Sim-ToM, and Symbolic-ToM.

Do you think that these changes are sufficient to improve your evaluation of the paper?

Dear Reviewer Gb3Q,

Thank you for your supportive evaluation of our submission. We will address your comments point by point below.

The paper could use some diagrams to better capture the intuition of inverse planning, perhaps by including an example of agent paths through the restaurant environment in Figure 1.

We agree that a clearer version of Figure 1 would help readers understand the task more clearly, and plan to overhaul the figure to include a sample task trajectory and example visibility for an agent based on its position.

The paper could use some diagrams to better capture the pipeline of different LAIP variations, or at least the main LAIP approach, including the main components of the pipeline (the LLM, the hypotheses, the generated actions, the true grounded actions, the true hypothesis) and how they are elicited.

Thank you, we agree that a larger figure that identifies the pipeline we use for different variations of LAIP could also clarify how these approaches differ from or resemble one another, and to clarify the elicitation of these hypotheses. We will create a graphic that presents the main variants we consider in our work.

Some discussion about previous work in using likelihood estimates of strings of text generated by LLMs would be welcomed so that the reader can better contextualize how LLMs have been/can be used for such tasks.

While this is not exactly a weakness, I think there are some interesting philosophical questions about how LLMs are able to capture these kinds of causal relationships in strings of text that are worthy of discussion. Also questions about how a visually grounded VLM might behave differently than a text-only LLM.

We agree that these are both important questions. One commonality that we identify across both of these points is that many of these models, including LIMP (Shi et al., 2024), BIP-ALP (Jin et al. (2024), CLIPS (Zhi-Xuan et al., 2024) as well as ours, involve some degree of using LLMs to decompose visual scenes, text-based descriptions of scenes, or other open-ended information into structured, parsed text.

We also agree that some discussion of LLM-generated probabilistic estimates would strengthen the work. Recently, Bradshaw et al. (2024) arXiv:2411.03486v1 [cs.AI] have explored how GPT-4o generates plausible distributional properties when simulating vote shares in both past and future United States elections. Although this is a domain in which there is a high volume of both historical and speculative data that may inform the plausibility of its outputs, it is plausible that various higher-order properties of probability distributions may be represented within LLMs.

Dear Reviewer aqP8,

Thank you for your helpful comments and constructive feedback on our work. We believe your comments raise a number of important points, which we address below.

The benchmarks provided seem insufficient for a comprehensive evaluation. Currently the comparison has been made with LAIP(LLM computes posterior), LAIP (single LLM call), pure LLM, and a steadily updated model. They are the variation or downgraded versions of the proposed LAIP model. They are good baselines though.

How is LAIP compared to sota approaches, or pure-math BIRL models? (Ramachandran 2007)

Thank you for raising this point. As we note in our responses to Reviewer Xehq and Reviewer HwkP, we agree that our work would be strengthened through the inclusion of a broader number of baselines, including other recent and SOTA approaches. We have run preliminary tests of our model against ReAct, and find LAIP substantially outperforms it on our task. Additionally, we plan to test Reflexion (Shinn et al., 2024) and BIP-ALM (Jin et al., 2024) on our task.

The paper claims that the scalability issue is a big problem, and I fully agree. This is exponentially more difficult to calculate the bayesian probability when the state space gets a little larger. That hinders the development of inverse reinforcement learning from this certain direction. However, the paper’s chosen scenarios are simple enough that priors could almost be calculated manually, raising questions about scalability to more complex settings. It can hardly convince that LLM can help solve it as the task might not be too challenging for gpt4o.

What are the computational and scalability limits of this approach in terms of state space and trajectory length?

We agree that scaling is a potential concern with the LAIP model in more complex environments. We describe in greater detail in our response to Reviewer Xehq our perspective on the degree to which we anticipate LAIP can scale in its current form. The primary challenge of the current approach is that if the hypothesis space grows sufficiently large, generating action probabilities across many candidate hypotheses poses a steep computational overhead, especially as the reviewer notes with longer trajectories.

One potential solution to this approach is an approach in which hypotheses are not held static across the entire trajectory length but can be iteratively revised, pruned, and introduced in a manner similar to particle filtering/sequential Monte Carlo approaches. This would allow persistently low-probability hypotheses to be filtered out, such that they would not require simulation when their prior probability is sufficiently low.

Nevertheless, our preliminary experiment with LAIP in the MMToM-QA environment, which has a more open-ended action space than our task and longer trajectories, suggests that LAIP is able to effectively scale to reason about more complex tasks in which the number of objects to track and candidate actions is much larger. As we describe in our response to Reviewer Xehq, although preliminary, our result outperforms the average from BIP-ALM. We plan to test across the other conditions of the benchmark to show its effectiveness across a broader set of scenarios and assumptions.

The paper lacks an explanation for how smaller, less reliable LLMs consistently perform well under LAIP. The results shown in Figure 3 indicate that several small LLM models perform much better with LAIP, but there is no mechanism that ensures the models can still contribute.

Can you give analysis on how LLM can boost the small LLM models?

We thank the reviewer for raising this point. We believe this finding reflects two strengths of the task decomposition that we use to assess action probabilities. When an LLM directly evaluates the probability of a hypothesis across a full trajectory, or computes and updates probabilities within the language model, it is liable to make errors in computation, mathematical reasoning, or application of mathematical functions; these errors are especially likely to be pronounced in smaller language models that may be less expressive. However, by prompting the LLM to solve smaller tasks, such as possible actions in a given state, assuming one of several candidate hypotheses is true, the LLM may be more likely to provide accurate responses, especially if the posterior is mathematically computed rather than computed using an LLM prompt.

[cont...]

Dear Reviewer aqP8,

Thank you for your additional remarks. We have written our general response here where we outline the changes to our updated manuscript; however, we also briefly address your subsequent questions here.

Additional Benchmarks: I appreciate the authors’ agreement to run more benchmarks, as mentioned in the response. However, it is unclear if the manuscript has been updated to include these results. Highlighting new additions in the text (e.g., with a different color) would make it easier to track changes and assess the updates.

On Lines 412–441, we detail the results of LAIP on the MMToM-QA benchmark. We find that it outperforms the BIP-ALM model proposed in Jin et al. (2024), as well as other models such as Sim-ToM and Symbolic-ToM, in an average across all goal conditions, and remarkably so in the goal-updated condition. The large action and state space and long trajectories of this benchmark also provide strong evidence of LAIP's ability to handle more complex, challenging problems that may remain difficult for a pure-math IRL approach.

Scalability and Complexity: The proposed solution to scalability challenges, such as filtering hypotheses with low probabilities, looks promising. Including experiments to validate this approach would greatly strengthen the work. While filtering might help address the computational burden, determining appropriate thresholds and effectively ranking hypotheses in larger state spaces remain challenges. Additionally, generating 20 hypotheses is relatively modest; more evidence on handling larger-scale problems would add significant value.

We agree with the reviewer that solving scalability is an important concern. Effectively determining which hypotheses to maintain and which to discard, or when to revise a hypothesis, and how much effort to expend on this process of considering multiple hypotheses is one which we believe to be a challenge, but one that is within the abilities of LLMs. We intend to work on this issue, particularly as it concerns modelling the social or motivational aspects that may underlie when human beings engage in effortful cognition to infer someone else's beliefs, desires, or goals using inverse planning; however, the goal of this work is to first demonstrate the utility of this approach in open-ended environments. While these represent promising avenue for further iteration and improvement, they may be out of scope of this initial paper in which we demonstrate the basic utility of the algorithm.

Nevertheless, we have added several paragraphs in the discussion of the paper (Lines 509–524) to address this important issue and highlighted techniques that we intend to use to address further environmental complexity in future work, including particle filter-inspired model designs.

Small LLMs in LAIP: The explanation of how smaller LLMs can perform well under LAIP is reasonable and provides good intuition. Addressing this quantitatively, such as with analyses or experiments demonstrating the performance boost for smaller models, would make this argument much more convincing.

Thank you. We have added an analysis comparing the relative improvement between the baseline model between the nested improvements of the model on Study 2, focusing on the difference between the full model and the model where the LLM computes the posterior (Lines 397–406). We find that large models such as GPT-4o and LLaMA 3-70B show the least improvement, while Gemma and Mixtral, the smallest models, show significant improvement and the largest effect size difference between these two conditions.

Thank you again for your efforts in responding to the feedback. I appreciate the thoughtfulness of your responses and the potential directions outlined for future work. I hope these updates and clarifications will help enhance the impact and understanding of your contribution.

Thank you for your constructive reviews. We hope that the changes to the manuscript and the comments above outlining these changes address your concerns. Do you feel satisfied that we have done so sufficiently to increase your evaluation?

Dear Reviewer Xehq,

Thank you for your thoughtful and constructive feedback. We will address your comments and questions point by point.

Limited baselines: the baselines in the paper do not reflect the current SOTA of these approaches. For Study 1 you should add few shot prompting, CoT, ReAct, Reflexion type baselines as well. Zero shot baseline is not a strong baseline We agree with you that exhibiting more robust evidence of LAIP’s effectiveness by comparing to existing baselines and other recent approaches is important.

To address this point, we have run several new experiments which we describe below, and we plan to include additional baseline comparisons for Studies 1 and 2, including zero-shot CoT, few-shot CoT using reasoning about a different trajectory, ReAct, and Reflexion, to compare against the predictions of the LAIP model.

For Study 1, we have run additional preliminary results using a generic zero-shot CoT prompt, LAIP with zero-shot CoT prompt, and ReAct, using a generated prior across hypotheses (all using GPT-4). We find that, across an average of 6 runs, LAIP with CoT correctly identifies Hypothesis 2 as the most likely (51.9% probability), albeit less than the full model (77.2% probability). Both of these new experiments substantially outperform the zero-shot CoT baseline (11.6% probability) and ReAct (3.6% probability).

Experiments seem narrow in scope—focused on the food truck type toy tasks from cognitive science. The true promise of the method is that it should be more scalable than bayesian methods yet a majority of the experiments were with environments that Bayesian methods could handle. The number of potential hypotheses are often small as well, simplifying the process of latent state inference to a multiple choice problem. It would be interesting to see the methods on open ended ToM inference or multiagent decision-making tasks or benchmarks. Needs testing in diverse, unpredictable environments

We also agree that showing that LAIP can succeed in more open-ended environments where the hypothesis or action space are less constrained is a valuable addition to this work; indeed, showing that LLM-generated hypothesis and action spaces can be leveraged to successfully reason using inverse planning in open-ended domains is the largest conceptual contribution that we hope to be able to more fully demonstrate.

Following the suggestions of another reviewer, we have adapted the LAIP model to the MMToM-QA benchmark (Jin et al., 2024). This benchmark uses a more complex environment with a larger number of free parameters, in which the observed actor traverses through an apartment with various rooms, each of which contains various containers or surfaces on which various foods, utensils, and household objects are located. On this task, the LLM must reason about the likely beliefs or goals of an actor given their trajectories.

In a preliminary run from the “goal given true belief” condition in MMToM-QA using the LAIP model, we have found LAIP identified the actor’s true goal with 94.8% of the probability mass on the correct answer, compared to 77.3% across all “goal given true belief” tasks for the text-only BIP-ALM model with fine-tuned GPT-J. We will run across additional conditions in order to provide a more comprehensive benchmarking of our model.

Scalability and computational efficiency aren't thoroughly addressed—how does LAIP perform with more complex tasks?

What's the computational overhead of combining LLMs with Bayesian inverse planning?

We thank the reviewer for raising these important concerns. We anticipate that by including more complex tasks in which the number of actions or hypotheses the model needs to consider is higher, we will provide good evidence that LAIP is capable of scaling in terms of its effectiveness in solving more complex environments.

We also acknowledge that the computational overhead of LAIP can increase considerably, particularly as the number of hypotheses to consider increases to larger amounts. Given that LAIP seems to be particularly useful at improving the performance of smaller models (e.g., LLaMA 3:7B), this might mean that LAIP might enable or facilitate smaller LLMs to perform more effectively on belief, desire, or goal inference tasks, trading off against increased computational overhead.

We will further elaborate on the overhead resulting when the number of plausible candidate hypotheses is very large as a potential limitation of LAIP’s applicability. We will also evaluate potential solutions to this issue; for example, we anticipate combining LAIP with particle filter/sequential Monte Carlo-based approaches to approximate Bayesian inference could allow the model to entertain a small number of hypotheses at a time, eliminating “particles” with low-probability hypotheses and allowing for new hypotheses to be proposed from a prior across e.g., other relevant features of the current state of the environment.

[cont...]