SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Model

We sincerely appreciate the reviewer’s acknowledgment of our motivation and contribution, and we mainly address the reviewer's comments and questions below.

1. Quality Control.

a) What evidence supports the notion that these spatial relationships are meaningful to GPT-4o? Thank you for the questions. In fact, based on our experiment in Sec.3.4, GPT-4o's ability to interpret spatial relations is strongly related to different prompts. We experiment with two types of prompts for representing spatial information to prompt GPT-4o: Cord-prompt, which consists of object center coordinates, standing point, orientation, and instructions for calculating distances and rotation angles, and Spa-prompt, consisting of the calculated angles and distance based on the approaches we described in Sec.3.3. An example of each type of prompt can be found in Tab.11 in the Appendix. The following table indicates that Cord-prompt yields unsatisfactory results, revealing that LLMs lack strong 3D spatial reasoning capabilities when interpreting raw spatial coordinates. Our Spa-prompt significantly improves the quality of the generated dataset by providing qualitative spatial relations (e.g. distance, direction). The role of GPT-4o in our approach is primarily to organize this pre-processed spatial information and generate QA data. Therefore, there is minimal reliance on GPT-4o for interpreting the 3D scene environment, as nearly all spatial information is already systematically structured.

Besides, we would like to provide an example of how we prompt GPT-4 to help the reviewer gain a clearer understanding of our method. When prompting GPT-4o, we provide an explanation of how to interpret the spatial information. GPT-4o demonstrates a strong understanding of this pre-processed spatial information. Below is an example, which is also presented in Table 10 of the Appendix.

{"Left":{"table\_8": {"distance": 2.6, "passby": ["chair\_21"], "affordances": ["placing items on"], "attributes": {{"color":"red"}}, "angle": 257.48, "relations": ["close by chair\_36"]}}}.

From this situated scene graph, we know that on my left 257.48 degrees and 2.6 meters, there is a table\_8 that is close by chair\_36. You can place items on table\_8. If you go to table\_8, you could pass by chair\_21.

b) How this spatial information aligns with human perception. Thank you for the question. To evaluate whether GPT-4o is able to interpreate the spatial information and generate the correct dataset, we conduct a comprehensive human evaluation and introduce human scores based on two key criteria: language naturalness, which evaluates whether the text reads as if it were naturally written by a human, and spatial fidelity, which ensures that the data accurately reflects the 3D scene with correct spatial relationships. The detailed explanations are as follows.

• Language Naturalness: evaluates the flow, syntax, and semantics of the language, ensuring it reads naturally as if written by a human and adheres to common-sense knowledge in practical scenarios. For instance, a score of 1 could be assigned to a scenario like Standing beside a blanket while there is a toilet in the background, which is uncommon in a typical setting. Similarly, questions such as Which is cleaner, the trash bin or the bed? are unlikely to be asked by humans, reflecting low language naturalness.

• Spatial Fidelity: is critical to ensure that the generated data accurately represents the information in 3D scenes and adheres to factual details. This includes verifying the presence of objects within the scene and ensuring that spatial relationships between objects are correctly represented. Additionally, common sense knowledge must be considered, especially when unusual objects or scenarios are mentioned in the questions or answers. For example, in a 3D scene, a score of 1 is assigned to the instance where clothes are hung on a mirror. This error arises because the human-annotated data from 3RScan labeled the mirror's affordance as hanging, which misled the GPT model into generating an incorrect dataset.

Each criterion is rated on a scale from $1$ to $5$, and the average of these two scores is the overall human score. We randomly select $50$ examples from each task and compute human scores of situation, question, and answer, respectively. We mainly have the following findings:

1. The average scores align with the complexity of each task, with relatively lower scores for captioning and planning tasks.

2. To assess how our generated data compares to human-annotated data, we sampled $50$ examples from SQA3D and mix them with our dataset. We focus on the human score of different types of questions. As shown in the following table, our generated questions are comparable to the questions in SQA3D across various question types.

We sincerely appreciate the reviewer’s thorough reviews, as well as the recognition of our work's strengths and contributions from multiple perspectives. We have addressed the reviewer's concerns as follows.

1. Quality Control of Dataset We truly appreciate the reviewer raising this important issue. In the original version of the paper, although we provided human evaluation in Sec.3.4 on how different prompting strategies influence the quality of data, we agree with the reviewer's constructive suggestions to provide nuanced spatial and contextual fidelity analysis. Therefore, we conduct a comprehensive human evaluation and introduce human scores based on two key criteria: language naturalness, which evaluates whether the text reads as if it were naturally written by a human, and spatial fidelity, which ensures that the data accurately reflects the 3D scene with correct spatial relationships. The detailed explanations are as follows.

• Language Naturalness: evaluates the flow, syntax, and semantics of the language, ensuring it reads naturally as if written by a human and adheres to common-sense knowledge in practical scenarios. For instance, a score of 1 could be assigned to a scenario like Standing beside a blanket while there is a toilet in the background, which is uncommon in a typical setting. Similarly, questions such as Which is cleaner, the trash bin or the bed? are unlikely to be asked by humans, reflecting low language naturalness.

• Spatial Fidelity: is critical to ensure that the generated data accurately represents the information in 3D scenes and adheres to factual details. This includes verifying the presence of objects within the scene and ensuring that spatial relationships between objects are correctly represented. Additionally, common sense knowledge must be considered, especially when unusual objects or scenarios are mentioned in the questions or answers. For example, in a 3D scene, a score of 1 is assigned to the instance where clothes are hung on a mirror. This error arises because the human-annotated data from 3RScan labeled the mirror's affordance as hanging, which misled the GPT model into generating an incorrect dataset.

Each criterion is rated on a scale from $1$ to $5$, and the average of these two scores is the overall human score. We randomly select $50$ examples from each task and compute human scores of situation, question, and answer, respectively. We mainly have the following findings:

1. The average scores align with the complexity of each task, with relatively lower scores for captioning and planning tasks.

2. To assess how our generated data compares to human-annotated data, we sampled $50$ examples from SQA3D and mix them with our dataset. We focus on the human score of different types of questions. As shown in the following table, our generated questions are comparable to the questions in SQA3D across various question types.

3. We also evaluate how different prompting strategies influence the quality of the data. We experiment with two types of prompts for representing spatial information to prompt GPT-4o: Cord-prompt, which consists of object center coordinates, standing point, orientation, and instructions for calculating distances and rotation angles, and Spa-prompt, consisting of the calculated angles and distance based on the approaches we described in Sec.3.3. An example of each type of prompt can be found in Tab.11 in the Appendix. The following table shows the percentage of examples with high human scores (>=4) for each prompt across tasks. The results indicate that Cord-prompt yields unsatisfactory results, revealing that LLMs lack strong 3D spatial reasoning capabilities when interpreting raw spatial coordinates. Our Spa-prompt significantly improves the quality of the generated dataset by providing qualitative spatial relations (e.g. distance, direction).

We hope our extra analysis helps address the reviewer's concern.

Dear reviewer D9vh,

We sincerely appreciate the time and effort you’ve devoted to reviewing our work. We understand that your schedule may be quite busy, and we are truly grateful for your valuable feedback. As the discussion phase is nearing its end, we would greatly appreciate the opportunity to address any concerns or questions you may have. Thank you for your attention and consideration.

With regard,

Authors

We sincerely appreciate the reviewer's acknowledgment of our strengths and contributions. We address the reviewer's comments as follows:

1. How effective of automatically generated data compared to Human Data? Thank you for the insightful question. It is important to emphasize that an exact match score alone does not necessarily reflect a stronger situated understanding ability. After analyzing answers generated by Spartun3D-LLM, we observe that our method significantly influences LEO's behavior. Please see our experiments in Section 5.3 (Improved Situated Understanding). We extract questions in SQA3D starting with ''which direction", and answer includes ''left'', ''right'', ''forward'' and ''backward''. We observe that LEO is biased towards generating ''left'' 97% of the time. However, in human-annotated data(SQA3D), the distribution of ''left'' and ''right'' should be balanced. In contrast, our model produces a distribution that closely matches the ground truth, demonstrating the improved situated understanding ability and our generated automatic data is close to human data. Additionally, Table 8 in the Appendix presents a detailed breakdown of fine-tuned performance across various question types. The final EM score is the average performance across these question categories. Notably, Spartun3D demonstrates particular effectiveness in addressing ''what," ''is," and ''can" questions.

2. Performance Drops between Zero-shot and Fine-tuning. Thank you for the question. In the zero-shot setting, Spartun3D-LLM is trained using Spartun3D, which is constructed from 3RScan, whereas SQA3D is sourced from ScanNet. The performance drop is expected due to differences in the 3D scenes between the datasets. However, compared to LEO, Spartun3D-LLM achieves an approximate 20% improvement in the zero-shot setting, demonstrating enhanced situated understanding.

3. Scaling Effect. We sincerely appreciate the reviewer’s providing this constructive suggestion. In response, we conducted scaling experiments to demonstrate how model performance improves with the addition of Spartun3D datasets. Our evaluation of SQA3D shows consistent improvement as the dataset scales, underscoring the potential for further dataset expansion using our proposed method. The results are as follows, and we have included the corresponding effect curve in the new version of the paper. | % of Dataset | EM | |--------------|------| | 20 | 52.7 | | 40 | 53.3 | | 60 | 53.9 | | 80 | 54.7 | | 100 | 55.0 |

4. Clarification of L460-465. We apologize for the confusion caused by our writing. To clarify, our intended statement is: ``We re-trained LEO exclusively on their constructed dataset from 3RScan.''. LEO constructed its dataset using scenes sourced from 3RScan. However, its publicly available checkpoint was already fine-tuned on 3D tasks from ScanNet, including ScanQA, Scan2Cap, and SQA3D. To ensure a fair comparison and accurately evaluate zero-shot performance on SQA3D, we re-trained LEO exclusively on its constructed dataset from 3RScan. We have revised the corresponding part in our new version.

5. Navigation Performance. In Table 5, the differences between LEO and Spartun3D-LLM lie in both the training dataset and the alignment module. Specifically, The baseline here is LEO trained on its own constructed dataset and fine-tuned on other 3D tasks, excluding ObjNav. While Spartun3D-LLM is trained using the Spartun3D dataset and incorporates our specially designed spatial alignment module. The primary reason LEO performs poorly on navigation tasks in a zero-shot setting lies in the limited spatial information in its dataset. Unlike Spartun3D, their dataset lacks specially designed spatial information. As illustrated in the teaser (Fig.~1), when asked, What should you do to wash hands?, LEO generates the answer sink. While correct at an object level, this response lacks spatial context, highlighting that their model mainly emphasizes understanding objects and attributes. In contrast, every example in Spartun3D explicitly incorporates questions and answers involving spatial information. This design enhances the model's spatial reasoning abilities, which further helps improve the performance on downstream navigation tasks.

I want to thank the authors for their response. I think this clarified a lot of mis-understandings I had. I am inclined to raised my score to 6, I still have a few gaps in my understanding.

• I understand the experiment in Table-3 better now, thanks for adding a clear text on it in the paper. I think the main claim there is: LEO trained on 3RScan data they were already using does not generalize well to ScanNet based SQA3D dataset. When it is trained on the proposed Spartun3D dataset, it starts to generalize -- hence, this data is useful for training future models and would likely aid generalization. I agree with this. I do not agree, however, with the claim on Line 463-464 "generalization ability of our model". I do not see an evidence for Spartan3D-LLM generalizing better than LEO based on Table-3. Spartan3D-LLM is indeed better by a few percent than LEO in zero-shot setting, but it is also better than LEO on fine-tuned setting (in-domain). I think it is inconclusive whether the proposed model generalizes better or it is just stronger in general than LEO. (Maybe by "model" the intention was to refer to the methodology of generating the dataset?)

• I think the experiments with zero-shot generalization in Table-3 does not directly answer "whether the automated way of generating large amounts of data is better than human generated small-scale SQA3D data". 3RScan data is also generated automatically, and the experiment in Tabe-3 shows that their method of generating that data is worse than the proposed Spartan3D data generation. I do not expect the authors to do this experiment -- but perhaps one way to answer it is: automatically generate data on ScanNet, and compare performance on val set of SQA3D with varying amounts of generated data and human-collected data. Fine-tuning on automatic+real data and testing on val set of SQA3D can also answer this question. As the authors mention it: current fine-tuning performance does not tell us much since Spartan3D (based on 3RScan) has large domain gaps to ScanNet.

However, in human-annotated data(SQA3D), the distribution of ''left'' and ''right'' should be balanced. In contrast, our model produces a distribution that closely matches the ground truth, demonstrating the improved situated understanding ability and our generated automatic data is close to human data.

I still do not understand this and implications drawn in 5.3. Fact 1 is: SQA3D dataset is already balanced, yet LEO shows imbalance in its output space (I am assuming that LEO is not trained on Spartan3D data). Fact 2: Spartan3D-LLM which is LEO + spatial alignment loss + trained on Spartan3D dataset does not show the bias. It is unclear if this is because of Spartan3D dataset or spatial alignment loss. I think the authors are trying to say that spatial alignment loss is not disjoint from Spartan3D dataset, as the proposed dataset helps in using the loss. But couldn't this loss also be used with original training data of LEO? if that works, then the fix to bias is the loss and not necessarily the dataset.

I also don't understand the conclusion drawn by "generated automatic data is close to human data" -- SQA3D is human data and is already balanced and yet LEO has this bias. I am not sure how we are reaching at this conclusion based on the facts.

We sincerely thank the reviewer for their prompt feedback, which has been invaluable in improving the quality of our work.

1) Modification of Line 463-464. Thank you for your helpful feedback in clarifying our statement. Based on the reviewer's suggestion, we have updated the relevant sentence as follows and made the corresponding modifications in the paper.

LEO trained on Spartun3D (LEO+Spartun3D) shows significant improvement, demonstrating the effectiveness of our dataset. Further comparisons of Spartun3D-LLM with LEO+Spartun3D demonstrate a better zero-shot learning (i.e., generalization) capability of our model.

2) Generating dataset on ScanNet. We sincerely appreciate the reviewer’s suggestion regarding generating data from ScanNet to address concerns about SQA3D. We agree that leveraging ScanNet could provide additional insights to address the reviewer's concern. However, a key difference between ScanNet and 3RScan is that ScanNet does not include fine-grained annotations for the objects in each scene, e.g., size, shape, state, and affordance, while such meta information about objects is crucial for applying our automatic data construction pipeline to generate high-quality and accurate situated scene graphs or situated captioning and question answering data instances. It might be achievable to obtain such information by applying various external tools but given the limited time window, we decided to leave it for future work.

However, we would like to emphasize the value and broader applicability of our dataset beyond the specific scope of SQA3D. Our goal is to pre-train on a large-scale dataset, which is applicable to arbitrary tasks that need situated understanding. While SQA3D is a valuable resource, models trained on such human-annotated data are unable to generalize to other embodied tasks such as navigation and manipulation. In contrast, our Spartun3D dataset, including various situated tasks, enables models to excel not only in answering situated questions but also in tasks like robotic navigation in real-world environments.

3) Question related to Section5.3. Thank you for pointing this out. We conducted the following Table to help the reviewer understand where the fixed bias comes from. The Table shows the number of labels for the question starting with which direction in SQA3D test dataset. LEO exhibits a noticeable bias towards left, but when trained on our dataset (LEO+Spartun3D), this bias is significantly mitigated. While further adding our alignment loss~(Spartun3D-LLM) appears to further contribute to this improvement, the primary factor in addressing the bias is our dataset. We have modified Section 5.3 in the paper.

4) Closer to Human Data. We sincerely apologize for the inaccuracy in our previous rebuttal. We retract the statement and clarify that the distribution of the generated answers is more balanced and natural, better aligning with the distribution of human ground-truth labels.

We appreciate the reviewer's recognition of our contributions and the potential for future research in situated 3D scene understanding.

1) More error examples. Thank you for highlighting this important issue. We totally agree with the reviewer that detailed error analysis is necessary. We randomly sampled 50 examples from each task (200 in total) to validate the quality of our automatically generated data and manually assess the quality from the aspects of language naturalness and spatial fidelity. Language naturalness evaluates whether the generated texts are natural or written by a human, while spatial fidelity ensures that data accurately reflects the 3D scene. We observe 26 errors in total and summarize them into the following categories.

• Semantic Errors: The generated sentences may contain semantic mistakes. For instance, the answer You should go in front of you to water the potted plant.
• Common Sense Violations: The generated content may occasionally conflict with basic common sense knowledge, such as producing unusual questions and answers. For example, it might generate a question like If I want to store items, which object is most accessible? with the answer being trash bin. This issue arises because the human-annotated data includes an affordance for trash bins as objects for storing items. Such annotations inadvertently influence GPT-4o to generate QA pairs that conflict with common sense knowledge.
• Spatial Inconsistencies: Errors in capturing or reasoning about spatial relationships in the 3D environment primarily occur in Situated Planning tasks, which demand complex two-step spatial reasoning. These errors often arise because the second action depends on the outcome of the first action, and inaccuracies sometimes occur during the generation of the second action.
• Misalignment between visual context and textual descriptions: In some cases, the agent's view is obstructed due to the room layout or object size. For example, consider the situation: Standing beside the sofa, there is a closet on your left. However, the closet is actually located in another room and cannot be seen from the agent's current standpoint. To address this issue, we designed a scenario where the agent stands beside a pivot object and consistently faces the center of the pivot object rather than facing a random object that could potentially be obstructed. Additionally, we incorporated pass-by spatial information to enhance the agent's awareness of surrounding objects, providing a more comprehensive sense of the environment.

The following table presents detailed statistics of the errors for each category of generated text. Certain errors are task-specific; for instance, spatial inconsistencies predominantly occur in situated planning. Similarly, misalignment issues are more common in situated captioning, as captions often include descriptions of surrounding objects, which increases the likelihood of mentioning obstructed objects.

Finally, we also sample 80 answers~(20 for each type) predicted by our model and confirm that the model is not really affected by the little noise contained in the data. As shown in the following second table, Errors related to semantics, common sense, and misalignment errors are much less than the noise in training. However, spatial inconsistencies were observed. We attribute these spatial inconsistency errors to the inherent difficulty of spatial reasoning rather than to noise in the training data. This conclusion is supported by the observation in the two tables: during training, spatial errors occur primarily in planning. However, during prediction, spatial errors occur uniformly across all types of tasks.