A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment

We would like to thank Reviewer 2hbs for their review. Please find below our responses.

Our intention in this paper is to present a new methodology for evaluating Language Models, and the results of applying such a methodology to current state-of-the-art models. We thank the reviewer for noting that the approach is “valuable for assessing LLMs’ spatial reasoning”, and that it ”could set a new standard for LLM evaluation, addressing key questions about their ability to reason about and act in physical environments”, as well as their comments about the novelty and clarity of our work.

‘The benchmark is also set up to test out-of-the-box LLMs, yet the comparison to children can feel unbalanced since LLM agents might benefit from a more complex setup than just these three actions.’

We selected the actions and commands given to LLMs to be analogous to those given to children. Children use keys to move forward, backwards, left, or right, and they arguably think and plan their next steps, even if only implicitly. Likewise, including additional actions for the LLMs to use would have provided them with an advantage not available to the children. We have adopted this simple setup to allow for the fairest possible comparison between children and LLMs.

‘There’s not much clarity on how the framework could directly support model training or help develop improved physical reasoning abilities in LLMs.’

The goal of this paper was not to directly support model training but to create a new framework to evaluate LLM agents on commonsense spatial reasoning. However, one of the observations from the experiments was that LLMs struggled with navigation, and a further follow-up would be to fine-tune these models on simple navigation tasks. We have discussed this potential future direction in the “Limitations and Future Work” subsection. The reviewer may also be interested in our new section Appendix A, in which we discuss selected case studies from our experiments, which might provide greater insight into the underlying mechanisms governing LLM performance. These observations could also help to inspire new research directions for model development.

'Overall I think the paper is proposing a benchmark for an important problem. However, I would like to hear the authors' opinions on the design principles of how LLMs interact with the environment.'

Three principles we would like to highlight that shaped our design of the environment interface are:

• The interface should be accessible to most researchers: This precluded us from choosing options such as having the models choose an action for each frame in the environment, which would currently be prohibitively expensive for many researchers, and could present challenges to all but the models with the longest context windows.

• There should be a minimal chance of confounds from the interface: As the reviewer highlights in their second point in the “Weaknesses” section, complex setups for agents may introduce confounds that make the comparison to children unfair. For this reason we chose to prioritise simplicity in the environment, avoiding the complex setups found in other papers, which may confound the comparisons we would like to make. We hope this helps to illustrate why we made the decision not to include actions such as “use tools” or “save to memory”, as the Reviewer suggests including such commands would make it harder to understand which capabilities are possessed robustly by the model, and would make comparisons with other 'kinds of intelligence' such as children more difficult.

• The interface should be aligned with the current literature: We adopted the ReAct framework to improve the LLM's decision-making process (Yao et al. 2022). The ReAct approach combines reasoning and acting by allowing the model to generate reasoning traces alongside actions (i.e. commands Go and Turn), which has shown improved performance on agentic tasks. By integrating ReAct, we encourage the LLM to first reason about the environment, identifying visible objects and their spatial relationships relative to the agent, before producing action scripts, such as Turn(15) and Go(10).

We thank the reviewer for raising their rating from a 1 to a 3. We would be pleased to discuss any remaining concerns about our paper: Please could the reviewer describe which of their concerns are still unaddressed?

'What real-world physical phenomena are represented in this environment?'

As the environment is built using the Unity Physics engine, it provides fast and realistic physical dynamics. In particular, all objects in Animal-AI have masses, volumes, and friction coefficients, are subject to the pull of earth-like gravity, and display realistic inertia and momentum. Note that the environment only represents physical dynamics to the extent that it allows researchers to test physical common sense reasoning. Animal-AI has already been used extensively to study physical common-sense reasoning in reinforcement learning agents and humans, and there is a growing body of experiments using the environment that replicate experimental designs used in real-world laboratories with animals and children.

'How does the work demonstrate that the VLMs can reason about these physical phenomena?'

Our work does not demonstrate that the VLMs can reason about these physical phenomena, and provides evidence through their failure that they lack the ability to perform such reasoning. This highlights a significant shortcoming in current systems.

Were a model to achieve good performance on these tasks, it would provide stronger evidence of their ability to perform such reasoning than current alternative evaluations because (1) The tasks in AAI draw on the history of experimentation in comparative cognition to ensure they are robust and focused on different facets of reasoning, and (2) The AAI environment requires LLMs to respond dynamically to the demands of a simulated 3D environment.

'What are the primary challenges in building such a benchmark, and how does this paper address them?'

Generally, we would summarise the primary challenges of building benchmarks as (1) ensuring they precisely and comprehensively measure the capability(s) of interest, and (2) ensuring the results are informative and actionable, identifying when and where models are successful and what their key failure modes are. LLM-AAI addresses the first point by providing a robust and well-validated framework for studying physical common-sense reasoning in multi-modal LLMs. It is an improvement on the current state-of-the-art which is to use Q&A-style benchmarks, because LLM-AAI uses a rich and veridical 3D environment and is less easily gamed or contaminated (models cannot be trained on all the state-action pairs present in AAI, whereas they can be for static text-and-image benchmarks). LLM-AAI addresses the second point by covering a variety of physical common-sense reasoning tasks using validated experiments from cognitive science. This allows us to precisely identify the kinds of tasks that contemporary LLMs can solve and those in which they fail, thus informing future work on developing better training curricula and model architectures. Moreover, AAI facilitates the iterative development of new experimental designs, leading to deeper insights about system capabilities.

The Psychology literature provides a rich source of inspiration for the kinds of capabilities we should be evaluating in machine learning systems and how we should do so, but it is currently a challenge for machine learning researchers to benefit from this as it is a distinct body of work which few of them will be familiar with. LLM-AAI addresses this challenge by providing a common space in which researchers from Psychology can collaborate with researchers developing Language Models for the benefit of both fields.

'Are there any comparable previous works? What specific advantages does this work offer over them?'

We summarise the state-of-the-art in physical common-sense evaluation in LLMs in Section 2. Most closely related to LLM-AAI is work employing VirtualHome to study ‘LLM Agents’ (e.g., Huang et al., 2022; Xiang et al., 2024; Li et al., 2024). However, the focus of these environments is to evaluate whether LLMs can follow instructions when interacting with an environment. They do not seek to directly measure the physical common-sense reasoning of these systems. This is the niche that LLM-AAI fills.

We would like to thank Reviewer 772C for their review. Please find below our responses.

The reviewer’s summary of our paper suggests we compare the results of LLMs with the performance of animals. While the Animal-AI environment could be used to make such comparisons (either by conducting the same experiment in Animal-AI and in the physical laboratory, or by using VR) we do not make such comparisons in the paper.

We thank the reviewer for their suggestion of an improvement to the terminology of our paper. There is some variation in terminology in the literature. We have now stated in the introduction that the subject of the paper is specifically multi-modal LLMs. A reason we do not want to limit ourselves to the term ‘VLMs’ is that the LLM-AAI framework could be extended to generate purely textual descriptions of observations of the environment, meaning that text-only LLMs would be able to interact with it. We performed some initial experimentation along these lines, but due to the limited performance of these models, we do not present results from them in the paper. Nevertheless, this is an option available for future research if and when text-only LLM capabilities improve in this domain.

We would challenge the reviewer’s suggestion that the environment lacks reflections of real-world physics. The Animal-AI environment is built using the Unity physics engine. While the shapes and textures of objects are minimalist in cosmetic details, they behave like objects in the real world in terms of their physics. All objects in Animal-AI have masses, volumes, and friction coefficients, are subject to the pull of earth-like gravity, and display realistic inertia and momentum. We have included a discussion of these points in Section 3.

We were unsure what the reviewer meant by “focusing mainly on semantic” aspects. Would the reviewer be able to clarify what they meant here, and how we might broaden the focus of LLM-AAI to address their concerns?

We view our central contribution of the paper to be the presentation of a framework for studying physical common-sense reasoning in LLMs, using well-described and validated experiments from comparative cognition. Furthermore, we present evidence that state-of-the-art multimodal LLMs lack physical common-sense reasoning, although they are able to interact effectively with an environment on simple tasks. As reviewer rqxH notes, this “methodology’s simplicity and focus make it easy to interpret and isolate the abilities being measured”. This is echoed by reviewer 2hbs, who notes that LLM-AAI is “valuable for assessing LLMs’ spatial reasoning”.

'What can the community learn from this work, apart from performance difference?'

In addition to the performance difference between child and LLM data, the community can also take two methodological lessons from this paper. Firstly, we have demonstrated how, instead of using static benchmarking to learn broad lessons about the task performance of LLMs, we can draw upon the rich history of experimentation in comparative cognition (the study of behaviour in non-human animals) to collect fine-grained data about LLM capabilities, and to make direct comparisons with other ‘kinds of intelligence’, such as humans and reinforcement learning agents. Secondly, we have demonstrated how simulated environments can be used as an alternative to traditional benchmarks in LLM evaluation.

Both of these lessons have been articulated to some extent in other works, but we believe that our work is original in combining them. We have suggested that the Animal-AI environment is well suited to putting these lessons into practice.

We would like to thank Reviewer rqxH for their review. Please find below our responses.

Firstly, we were pleased to see the reviewer’s acknowledgement of our approach’s benefits over the limitations of traditional, static benchmarks; we believe that our contributions will help to progress the science of evaluation of LLMs beyond its current static benchmark-centric state.

‘A significant limitation of this paper lies in the simplicity, or “toy” nature, of the Animal-AI environment. More robust environments, such as Minecraft or Omniverse, may better reveal practical competencies in physical reasoning due to their greater fidelity to real-world dynamics.’

We would push back against the claims that the Animal-AI environment is a toy environment that is less robust and has less fidelity than for example Omniverse. The Animal-AI environment is built using the Unity physics engine. While the shapes and textures of objects are minimalist in cosmetic details, they behave like objects in the real world in terms of their physics. All objects in Animal-AI have masses, volumes, and friction coefficients, are subject to the pull of earth-like gravity, and display realistic inertia and momentum. The engine is powerful and expressive, like that used in Omniverse, and is more physically realistic than the physics of Minecraft. We have added a discussion of these points in Section 3.

We agree with the reviewer that results in overly simplistic environments can sometimes be misleading. Our approach is centred around a ‘virtual laboratory’ — the Animal-AI environment — where we strive to find the balance between having too simple an environment (which could present too easy a challenge and lead to false confidence in a model’s capabilities), and having too complex an environment (which could introduce additional demands that are irrelevant to the capability being investigated). Indeed, following best practices in experimental design, we seek to maximise the complexity of the environment in ways that allow us to precisely test the capability of interest (physical common sense) and minimise its complexity in ways that would obfuscate that capability (photorealistic detailing of objects, a large action space, etc.). We believe that our results indicate that Animal-AI strikes the right balance for the current wave of systems; nontrivial challenges in the environment are very difficult and unlikely to be solvable by LLMs in the near future, and yet, in the spirit of laboratory research, our approach isolates the capabilities of interest (e.g., object permanence, causal reasoning, numerosity), without adding unnecessary complexity. While Omniverse and Minecraft could offer an alternative for building these experiments, we disagree that they offer “greater fidelity to real-world dynamics”. Moreover, their complexity and, in the case of Minecraft, highly stylised design, would introduce further confounds into experimental design. Finally, the Animal-AI environment is specifically designed for conducting behavioural experiments drawn from cognitive science — Omniverse and Minecraft are not.

We would argue that the performance differences are themselves part of our novel insights; we provide the first results for physical common sense in LLMs that can take actions in an environment, alongside comparisons with humans and reinforcement learning agents.

In addition to these results, we believe that our key novel contributions are the benchmark and the LLM-AAI framework. We hope that our benchmark and baseline results will be used to evaluate future generations of LLMs, fuelling the development of LLMs with greater physical common sense; a capability which many embodied agents require. Moreover, we hope that our framework, which will be open-source, will serve the research community as both a theoretical template and a usable tool for testing the embodied cognitive capabilities of future LLMs and other multimodal systems.

We agree with the reviewer that further interpretations of the LLMs’ performance and behaviour would improve our paper. We have taken these comments on board and have added a new section (Appendix A) where we highlight several key insights from the LLMs’ verbal and action responses that may shed some light on potential underlying mechanisms, and could provide a springboard for future work.