We thank the reviewer for the positive comments and the valuable feedback.

Do frontier models fail due to the lack of familiarity with prompts or because they have weak spatial cognition? Our experimental evidence indicates that the failures are most likely due to weak spatial cognition. Please see the common response.

Unclear use of Chain-of-Thought (CoT) techniques in prompting As illustrated in the Appendix and as noted by the reviewer, we add “Think step by step” uniformly for all task prompts. This is a form of zero-shot chain-of-thought as proposed in Kojima et al., NeurIPS 2022. Since we benchmark on 15 tasks, we have not found a way to provide more detailed chain-of-thought prompts in a task-agnostic way. It is possible to guide the model to solve specific tasks through step-by-step instructions, but this requires providing human prior knowledge about the task solving techniques and may not evaluate the cognitive competency of the models that we’re interested in.

Kojima, Takeshi, et al. "Large language models are zero-shot reasoners." Advances in neural information processing systems 35 (2022): 22199-22213.

Is there any study in psychology or cognitive science show that basic spatial cognition is related to solving geometry problems? Yes, there are several studies looking into the importance of spatial abilities for success in geometry and more broadly, in STEM). We have cited a couple of representative studies in the paper (L041) and have provided a larger list of studies below:

Shea, Daniel L., David Lubinski, and Camilla P. Benbow. "Importance of assessing spatial ability in intellectually talented young adolescents: A 20-year longitudinal study." Journal of Educational Psychology 93.3 (2001): 604. Kozhevnikov, Maria, Michael A. Motes, and Mary Hegarty. "Spatial visualization in physics problem solving." Cognitive science 31.4 (2007): 549-579. Wai, Jonathan, David Lubinski, and Camilla P. Benbow. "Spatial ability for STEM domains: Aligning over 50 years of cumulative psychological knowledge solidifies its importance." Journal of educational Psychology 101.4 (2009): 817. Newcombe, Nora S. "Picture this: Increasing math and science learning by improving spatial thinking." American educator 34.2 (2010): 29. Young, Christopher J., Susan C. Levine, and Kelly S. Mix. "The connection between spatial and mathematical ability across development." Frontiers in psychology 9 (2018): 755.

Does AlphaGeometry, a model trained for geometric theorem proving, perform better on spatial cognition tasks? This would be an interesting study for future work. If a more general-purpose LLM/VLM had similar capabilities as AlphaGeometry, we would be very curious to see if it performs better on spatial cognition tasks.

Thanks for addressing my concerns. I increased the soundness and maintained the rating which inclined to acceptance of this paper.

We thank the reviewer for the positive comments and valuable feedback.

How do we ensure the tasks are presented to the model in a way that is ecological? Please see the common response.

Can you structure the paper to organize tasks based on common spatial cognition features? All large-scale experiments (except route retracing) are currently evaluating cognitive mapping and planning (Tolman, 1948), i.e., the ability to build spatial representations of the environment and use them to reason about and navigate in environments (Section 3.1). Route retracing evaluates the ability to build route maps, i.e., the ability to memorize a route and follow the route in future trials (narrow strip maps from Tolman, 1948).

The small-scale spatial cognition experiments evaluate the following cognitive abilities: spatial perception, spatial visualization, spatial orientation, selective spatial attention and visuospatial working memory (Section 3.2). Spatial perception is the ability to perceive spatial information in visual inputs (e.g., horizontal nature of water surface in the water level test and angles between lines in the judgement of line orientation test). Spatial visualization is the ability to perform multi-step mental manipulations of 2D or 3D stimuli (mental rotation test and minnesota paper form board test). Spatial orientation is the ability to imagine being in a different position in space and seeing the surroundings from the new perspective (perspective taking test and maze completion task). Selective spatial attention is the ability to selectively attend to a particular region of space while ignoring others (selective attention task and corsi block-tapping test). Visuospatial working memory is the ability to store visual information in memory and manipulate them to solve tasks (corsi block-tapping test, spatial addition test and cambridge spatial working memory test). We have updated Figure 3 to reflect this organization.

Tolman, Edward C. "Cognitive maps in rats and men." Psychological review 55.4 (1948): 189.

Have you tried egocentric text representations? Yes, we tried similar experiments during the early stages of our work. To allow LLMs to perform visual navigation tasks, we tried using GPT-4v to describe images and use the descriptions as inputs for an LLM to navigate. However, we abandoned this approach because there was significant perceptual aliasing due to the fine-grained nature of the action space in large-scale experiments. For example, turning left/right by 30 degrees or moving forward by 0.25m does not change the textual description in most cases.

We also tried a version of the text-only experiments where we described the elements of the array instead of providing a comma-separated 2D array. However, we did not notice significant performance differences between the two approaches. Moreover, the text-based descriptions significantly increased the context length of the LLM, the models to exceed their maximum context length. For larger arrays, this could also cause performance degradations associated with very long contexts for LLMs (Liu et al., 2024).

Liu, Nelson F., et al. "Lost in the middle: How language models use long contexts." Transactions of the Association for Computational Linguistics 12 (2024): 157-173.

Do LLMs perform better with BEV text than VLMs with BEV images because text arrays are smaller than BEV images? From Table 1, GPT4o and GPT4v achieve average scores of 28.8 and 25.9, respectively, on BEV image, and 32.6 and 27.6, respectively, on BEV text. While these differences are non-trivial, they’re still too small to arrive at the above conclusion. Furthermore, the image tokenization process is significantly different from the text tokenization process. So, we cannot compare these processes directly.

What is the number of images per task in Table 3? We have updated the table (now Table 6 in the updated paper) to include the number of images and videos generated per task. Large-scale spatial cognition: There are 30 videos for ego image and BEV image tasks in large-scale spatial cognition. These videos form the basis for the questions and navigation tasks. Additional images are rendered for the navigation tasks conditioned on the agent’s actions. Small-scale spatial cognition: There are 1557 images for the QA tasks (all tasks other than MCT and CSWM). The number of images per task varies from 20 for PTT to 300 for SAdd. Note that some tasks have multiple images for the same image (e.g., MFPB, WLT, SAtt and CBTT), while others have multiple questions for the same image (e.g., PTT and MRT).For interactive tasks like MCT and CSWM, we render images conditioned on the agent’s actions.

Why does the angle become 78 degrees in Figure 6? In Figure 6, the coordinate systems of the top-down visualization (shown on the left) and the rendered BEV images (shown on the top-right) are different. The top-down visualization and the rendered BEV images are rotated versions of each other, i.e., you rotate the top-down visualization by 90 degrees clockwise to get BEV image. For the first question, the observer is standing below F in the BEV image. Correspondingly, the observer would be standing to the right of F in the top-down visualization. From the BEV image perspective, N is to the right on the same row as F. Note, that the visualization on the left is provided to the reader of the manuscript only. Models or our human baseline see only the BEV images. We have clarified this in the captions for Figures 6 and 7.

We thank the reviewer for their valuable feedback. As we understand it, the reviewer has two main concerns:

Concern 1: The image and video inputs may not be provided to the model correctly, resulting in poor results.

Please see common response above.

Concern 2: We only evaluate two VLMs for image relevant tasks.

Please note that we evaluated five VLMs in our submission (see Table 2). We evaluate two closed-source models (GPT-4o and GPT-4v), and three state-of-the-art open-source models (Pixtral 12B, Phi 3.5 vision and Llava interleave 7B). Among these models, only GPT-4o and GPT-4v support evaluating on long videos (up to 240 frames for ego videos) as noted in L418 - 420. We now additionally include results from Claude 3.5 Sonnet on small-scale spatial cognition tasks in Table 2, bringing the total number of VLMs to six (see updated paper). It obtains results similar to GPT-4o on majority of the tasks. It performs considerably better at the multimodal versions of SAtt and CSWM, and on the text-only version of CBTT. We were unable to evaluate Claude 3.5 Sonnet on large-scale spatial cognition tasks due to API failures on requests with long videos.

We address other questions below.

What image resolutions / aspect ratios do we use? For majority of our experiments, we use square images. The image resolution and aspect ratios are task-dependent. They are listed in Table 5 in the updated paper. Note that our human baseline uses the exact same images and resolutions for each task.

How do we handle aspect ratios? We provide the images to models as is without pre-processing. For most models (especially closed-source ones), the processing of the image beyond the input stage is outside our control. We rely on the model creators to correctly process the images. As the reviewer points out, rescaling images can affect metric distance estimation in both 2D and 3D. Our tasks are designed to be independent of the image resolution, and require only relative judgements (including the distance estimation task as noted in L210 - 217). We do not change other properties, such as aspect ratio.

What are the durations of videos and number of sampled images? The video duration depends on the path taken and whether the videos are of BEV images or ego images.

For BEV videos, a video frame is generated after every step (i.e., a one-step movement in the grid). Landmarks are clearly visible in this case and cannot be missed. The number of frames in a video frames from 13 to 72. We do not subsample the video frames when providing them to the model.

For ego videos, a video frame is sampled after every navigation action (i.e., turn-left, turn-right, move-forward). Following prior navigation literature (Savva et al., ICCV 2019), we use a forward movement step of 0.25m and turn angle of 30 degrees. This sufficiently fine-grained action space keeps the number of frames manageable (ranges from 61 - 240), while ensuring that the landmarks are not missing from the video. We further ensure that landmarks are not missed by sampling the trajectories as follows. To visit a landmark, the agent navigates close to the landmark (within 1m) and turns around to face it. When looking at it, we stop the agent for N=10 steps at the landmark to ensure it is prominently visible in the video for a longer time. The agent then moves away to visit a new landmark. We include some examples of walkthrough videos in the updated supplementary materials zip. Since GPT-4o and GPT-4v were unable to process 240 frames at once, we subsampled the videos by a factor of two to evaluate them on our benchmark.

Savva, Manolis, et al. "Habitat: A platform for embodied ai research." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

We have clarified our image pre-processing details in Appendix A.3 in the updated paper.

How do we carefully select the encoding to ensure compatibility with text tokenizers? We use comma-separated arrays of characters or four-letter words. Please see the common response for more details.

How is dimension of text array determined? Does this affect the tokenization? The array size varies from 3x3 to 20x20 for all tasks other than Maze Completion Task (MCT). The largest array size for MCT was 31x31. No, the dimensions of the 2D text array do not affect the tokenization.

We thank the reviewer for their valuable feedback and for sharing these references.

How does the proposed benchmark go beyond already existing benchmarks like Valmeekam et al., 2024, Momennejad et al., 2024, Yamada et al., 2023 and Ivanova et al., 2024?

Thank you for the references! They are relevant and we have incorporated them into our paper in the “Spatial reasoning in large language models” section in related works. There are two high-level differences that distinguish SPACE from all of these papers.

• Prior work evaluates cognitive mapping and planning abilities, which are related to the large-scale spatial cognition tasks, maze completion task and perspective taking tasks from SPACE. SPACE evaluates spatial cognition, which entails a broader umbrella of cognitive abilities than prior work (L097 - L111). We additionally evaluate the following abilities (L291 - L298)
  • spatial perception (water level test, judgement of line orientation test)
  • spatial visualization (mental rotation test, minnesota paper form board)
  • selective spatial attention (selective attention task, corsi-board tapping task)
  • visuospatial working memory (corsi-board tapping task, spatial addition task, cambridge spatial working memory task).
• SPACE aims to mimic classical animal cognition experiments. These are primarily visual in nature. Accordingly, SPACE provides multimodal task presentations that are compatible with VLMs. However, we also provide text-only translations of the multimodal tasks to evaluate LLMs that do not support the vision modality. Additionally, the parallel multimodal and text-only presentations of tasks allows us to compare the text and vision capabilities of VLMs on the same task (L474 - L480).

These are value-adds in SPACE when compared to the papers mentioned above. Here is a more detailed discussion of the similarities and differences between the tasks in SPACE and prior work.

• PlanBench (Valmeekam et al.) and CogEval (Momennejad et al.) evaluate LLMs on text-based planning tasks such as navigation, delivery logistics planning and block stacking. The navigation tasks from CogEval are related to SPACE’s large-scale spatial cognition tasks like route retracing and novel shortcuts. CogEval samples graph structures with varying levels of connectivity, where each node is a room (potentially containing an object) and edges represent the connectivity between rooms. This simulates high-level planning and navigation and is unlike what animals / humans deal with when performing the same tasks in the animal cognition literature. We replicate fine-grained navigation with egocentric and birds-eye view observation spaces akin to classical cognitive science experiments.
• Yamada et al. presents spatial reasoning tasks based on map traversals. Specifically, they create maps/graphs of certain shapes (squares, triangles, circles, etc.) and simulate a traversal through the graph, where each node contains a unique object. After the traversal, the model is expected to identify the object at the current location (i.e., self-localization). This task is related, but complementary, to SPACE’s large-scale spatial cognition tasks. SPACE focuses on perspective-taking (i.e., if you are at X, where is Y?) and map sketching abilities. Yamada et al. design abstract graph domains intended to evaluate text models and present the inputs in a narrative story form. In contrast, we mimic observation spaces and environments that were designed for animal cognition, and evaluate both text and multimodal models.
• EWOK (Ivanova et al.) study spatial plausibility reasoning (e.g., given the context “The piano is in front of A. A turns left.”, is it plausible to say “The piano is right of A”?). On the other hand, we evaluate complementary spatial cognitive abilities such as cognitive mapping / spatial orientation, spatial perception, spatial visualization, selective spatial attention and visuospatial working memory (L196, L291 - 298).