Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference under Ambiguities

We sincerely thank reviewer EAzE for their insightful feedback and constructive comments, as well as for recognizing the novelty of our benchmark, the clarity of our writing, the significance of our work in vision-language research and AI alignment, and its implications for developing more globally applicable models. Given the overall positive sentiment of the review, we kindly remind the reviewer that a score of 5 is marginally below the acceptance threshold. In case there are any misunderstandings or mistakes, we appreciate it if the reviewer could reconsider the rating.

Below, we address the identified weaknesses, questions, and suggestions.

Response to Weakness #1 (novelty):

Thank you for acknowledging that “frame of reference is a rather novel take”!

As highlighted in the Introduction, frames of reference are crucial for studying spatial cognition across modalities, providing a foundational framework for understanding how spatial relationships are perceived, interpreted, and communicated. Selecting the appropriate FoR is key to resolving ambiguities in spatial language understanding, whether by following explicit instructions through perspective prompts or adhering to language conventions. To the best of our knowledge, existing benchmarks do not account for this aspect, as most assume an egocentric relative FoR with reflected coordinate transformation, aligning with English preferences. However, English speakers can adapt to alternative FoRs when instructed, and speakers of other languages may exhibit vastly different preferences.

Response to Weakness #2 (Multilingual study remains high level)

In this experiment, we hope to provide VLM’s preference and raise concerns that English may dominate the FoR preference conventions of other languages in multilingual VLMs.

The ground truth human preferences are based on established findings in the linguistics literature, mostly from Levinson, 2003 but also in Majid et al., 2004; O’Meara & Báez, 2011; and Bender et al., 2020. For example, Dutch, English, Japanese, Tamil, Hausa, Spanish, Norwegian, Chinese, Tzeltal, Tongan, and Farsi prefer Relative FoR; and Jaminjung, Mopan, Totonac prefer Intrinsic FoR. In the revision, we explicitly stated this in Section 4.5 and cited the corresponding studies.

Although there is a good number of languages with well-documented frame-of-reference preferences and established consensus in cognitive psychology. However, not all languages have been extensively studied or reached such a consensus. Our ongoing work aims to address this gap through systematic human studies. Unfortunately, this research falls beyond the scope of an AI conference paper, so we leave it for future exploration.

Response to Weakness #3 (When is egocentrism problematic)

“However, it does not deeply investigate whether this bias toward egocentrism is inherently problematic across all contexts or if there are scenarios where this behavior is acceptable or even preferable.”

While there is no global ground truth, the language we studied in the main paper demonstrates a known preference, e.g., as the reviewer noted, English prefers an egocentric relative FoR. To this end, better alignment with the language convention and intuition is preferred in communication.

Besides, the perspective-taking capability of VLMs (Section 4.3) can function as a benchmark, as the input and ground truth are unambiguous. We encourage future work in VLM training to address this specific challenge.

Therefore, a "bias toward egocentrism" is acceptable when communicating with English speakers by default or when explicitly instructed to do so. However, it is not acceptable when communicating with speakers of languages that prefer other conventions or when models are instructed to adopt an alternative perspective, such as the addressee's viewpoint. Alignment with human cognition and cultural conventions is preferred.

Response to Weakness #4 (Practical relevance)

Our findings provide valuable insights for VLM developers. As we mentioned in Section 5, “current training recipes for multilingual multimodal language models heavily rely on machine-translated captions (Chen et al., 2023c; Geigle et al., 2024); however, this practice can be problematic…To enable similar linguistic transmission in AI models, exposure to naturally generated multilingual image-text data is crucial (Romero et al., 2024).”

Also, VLMs have many applications in embodied AI and robotics and spatial reasoning is critical for making decisions in the 3D world. Moreover, embodied communication requires flexibility for accommodating different FoRs. For example, if you want a robot to pick up the cup on your left using VLMs, the robot has to reason from your perspective to select the cup for manipulation.

We sincerely thank the reviewer for your thoughtful and constructive feedback (rephrase using ChatGPT later), and for recognizing our paper to be systematic and rigorous, having reliable testing methods, and extensive multilingual analysis. Below, we respond to weaknesses.

Clarification

“In other words, the evaluation metrics proposed in the paper are not absolute criteria, like egocentric is not naturally better than addressee-centered, but rather serve as perspectives for observing how VLMs understand spatial information.”

This is true as we emphasize that this dataset and its associated evaluation metrics are designed to assess cognitive similarity and alignment with human spatial cognition.

However, we also emphasize that:

• While there is no global ground truth, the language we studied in the main paper demonstrates a known preference, e.g., English prefers an egocentric relative FoR. To this end, better alignment with the language convention and intuition is preferred in communication.
• The perspective-taking capability of VLMs (Section 4.3) can function as a benchmark, as the input and ground truth are unambiguous. We encourage future work in VLM training to address this specific challenge.

Response to Weakness (Insufficient psychological and cognitive science related work)

We kindly refer to Section 2.1 for how humans resolve ambiguities based on linguistic and cultural backgrounds, and we use human preference as the criteria for evaluation. For example, English prefers the egocentric relative FoR with reflected coordinate transformation, so when we are evaluating English, we use the reflected coordinate transformation as the ground truth.

These ground truth human preferences are based on established findings in the linguistics literature, mostly from Levinson, 2003 but also in Majid et al., 2004; O’Meara & Báez, 2011; and Bender et al., 2020. For example, Dutch, English, Japanese, Tamil, Hausa, Spanish, Norwegian, Chinese, Tzeltal, Tongan, and Farsi prefer Relative FoR; and Jaminjung, Mopan, Totonac prefer Intrinsic FoR. In the revision, we explicitly stated this in Section 4.5 and cited the corresponding studies.

Although there is a good number of languages with well-documented frame-of-reference preferences and established consensus in cognitive psychology. However, not all languages have been extensively studied or reached such a consensus. Our ongoing work aims to address this gap through systematic human studies. Unfortunately, this research falls beyond the scope of an AI conference paper, so we leave it for future exploration. In this work, we hope to provide VLM’s preference and raise concerns that English may dominate the FoR preference conventions of other languages in multilingual VLMs with this example.

We appreciate your thoughtful feedback and valuable insights, and for recognizing our novel approach, thorough and methodical studies, and well-organized and clear writing. Below, we respond to weaknesses, questions, and suggestions.

Response to Weakness #1 (Limited scope of spatial relations)

Our paper focuses on the ambiguous nature of spatial language and builds upon linguistic and cognitive science literature on spatial frames of reference. We chose lateral and sagittal directions as they are the most fundamental spatial indicators.

“There is a tight connection between the Relative FoR and the Intrinsic FoR: it seems that you cannot have a Relative FoR without an Intrinsic FoR. Like the Intrinsic FoR, the Relative FoR requires ‘parsing’ of objects – most importantly, a parsing of the self into front, back, left and right.” (Majid et al., 2004)

As we mentioned in the Limitations section, spatial relationships like near-far and above-below are left to future work as we need novel metrics to evaluate these terms.

Response to weakness #2 (Synthetic 3D images may not fully capture real-world complexities)

Our synthetically generated images have occlusions and varying camera angles. We refer to Section 3.2 and Figure 3 for examples of the rendered images. For reasons of using only synthetic images in our dataset: It’s hard to rotate the target object around the reference object in the real world with uniform steps for collecting natural images. We also want to carefully control other variables (e.g. camera pose) in the experiments for systematic analysis, which are hard to control in capturing natural images.

However, we added a case study to see if results in synthetic images generalize to real photos. We fixed the camera pose and manually rotated the target object around the reference object using (1) two equally sized red and blue balls; and (2) a laptop and the red ball. We keep everything else identical to the original COMFORT setup. We found that our findings still hold for alternative backgrounds. Please refer to Section Appendix C for more details.

Response to Weakness #3 (A more diverse linguistic and cultural context or human annotations)

While we have started this study and formulated the problem, more extensive investigation into more languages and cultures will be an interesting direction for future work. We agree that human annotations for the multilingual dataset will improve the quality, but due to the large number of images and languages we evaluated, it requires a collective effort from the community to make this happen: We need native speakers of different languages from different cultural backgrounds to label in total (720 + 57600) * 109 = 6356880 instances.

In fact, our ongoing work aims to address this gap through systematic human studies. Unfortunately, this research falls beyond the scope of an AI conference paper, so we leave it for future exploration. In this work, we hope to provide VLM’s preference and raise concerns that English may dominate the FoR preference conventions of other languages in multilingual VLMs with this example.

Response to Question #1:

Our findings suggest that current VLMs are not robust and consistent enough so we should be cautious when we apply VLM to autonomous driving. Additionally, spatial reasoning from another person’s perspective is important in embodied communications, but our experiments show that VLMs cannot flexibly accommodate multiple FoRs. Moreover, the failure to adhere to language-specific or culture-specific conventions in cross-lingual tests makes them unusable for people in a non-English speaking country.

Response to Question #2:

As discussed in Section 5, “current training recipes for multilingual multimodal language models heavily rely on machine-translated captions,” which can introduce significant challenges. Enhancing spatial reasoning is a critical step toward the broader goal of achieving better multilingual and multicultural alignment in vision-language reasoning, ultimately contributing to the development of fair AI.

Response to Question #3:

In Section 5 Discussions, we mainly discussed (1) future work is necessary to improve the consistency and robustness of spatial representations in these models, (2) future work should extend the current 2D VLMs to the 3D domain, by considering camera poses and multiview data (Yang et al., 2024) for training, (3) to enable similar linguistic transmission in AI models, exposure to naturally generated multilingual image-text data is crucial.

We are grateful for your thorough evaluation and positive feedback on our motivations from linguistics and cognitive science, well-supported conclusions and novel continuous definitions based on regions of acceptability, and clear writing structure. Below, we respond to weaknesses, questions, and suggestions. The main weakness identified by the reviewer is the refinement/elaboration of the cross-lingual experiments, which we detailed below:

Response to Weakness 1 (the results in Figure 8 are a little ambiguously presented):

We confirm that Figure 8 reflects GPT-4o's preferences based on the cosine parsing error, weighted by the speaking population of the top three languages in each region. To eliminate ambiguity, we revised the figure caption and the associated text in Section 4.5 to explicitly state this.

Response to Weakness 2 (ground truth preferences and conventions):

Ground truth human preferences are based on established findings in the linguistics literature, mostly from Levinson, 2003 but also in Majid et al., 2004; O’Meara & Báez, 2011; and Bender et al., 2020. For example, Dutch, English, Japanese, Tamil, Hausa, Spanish, Norwegian, Chinese, Tzeltal, Tongan, and Farsi prefer Relative FoR; and Jaminjung, Mopan, Totonac prefer Intrinsic FoR. In the revision, we explicitly stated this in Section 4.5 and cited the corresponding studies.

Although there is a good number of languages with well-documented frame-of-reference preferences and established consensus in cognitive psychology. However, not all languages have been extensively studied or reached such a consensus. Our ongoing work aims to address this gap through systematic human studies. Unfortunately, this research falls beyond the scope of an AI conference paper, so we leave it for future exploration. In this work, we hope to provide VLM’s preference and raise concerns that English may dominate the FoR preference conventions of other languages in multilingual VLMs with this example.

Response to Suggestions (Intended usage of the framework)

We intentionally use the term “framework” to emphasize that this dataset and its associated evaluation metrics are designed to assess cognitive similarity and alignment with human spatial cognition. While each studied language demonstrates a preference, we do not position this as a leaderboard-driven benchmark. However, the perspective-taking capability of VLMs (Section 4.3) can function as a benchmark, as the input and ground truth are unambiguous. We encourage future work in VLM training to address this specific challenge.

Thank you for your thoughtful and constructive feedback and for recognizing our novel approach to probing spatial perception, demonstration of anglewise judgments, and extensive experiments to support the conclusions and discussions. We address your questions below and have updated our paper according to your suggestions.

Response to Weakness #1 (synthetic data)

Thanks for your suggestion. We use synthetic data for the following reasons: It’s hard to rotate the target object around the reference object in the real world with uniform steps for collecting natural images. We also want to carefully control other variables (e.g., camera pose) in the experiments for systematic analysis, which are hard to maintain in capturing natural images. However, we added a case study to see if results in synthetic images generalize to real photos. We fixed the camera pose and manually rotated the target object around the reference object using (1) two equally sized red and blue balls; and (2) a laptop and the red ball. We keep everything else identical to the original COMFORT setup. We found that our findings still hold for alternative backgrounds. Please refer to Section Appendix C for more details.

Response to Weakness #2 (limited diversity of backgrounds)

Thanks for your suggestion. We added a new dataset for a case study with a brown background, while keeping everything else identical to the original COMFORT-BALL setup. We found that our findings still hold for alternative backgrounds. Please refer to Section Appendix C for more details.

Response to Weakness #3 Limited sections on multilingual experiments

Thanks for your suggestion. Our multilingual experiments focus on investigating the preferred FoR in different languages. Since we built on the experimental setup already laid out for the English language in prior sections, we focused only on the experiments for multiple languages, which may explain why this section appears more concise. Table 10 in the appendix mainly serves as a full record for readers who are interested in looking at specific languages, but we plotted the preferred FoR on the world map in the main paper.