Spatial 3D-LLM: Progressive Spatial Awareness for Advanced 3D Vision-Language Understanding

1. The motivation for introducing object distance measurement and layout editing data is unclear. These data are generated by filling a template with the ScanRefer dataset, and the key to solving these tasks largely relies on 3D visual grounding from ScanRefer (plus simple integer addition or subtraction). I question the necessity of further processing the existing ScanRefer dataset to create these data, as such processing renders the contributions of these two tasks incremental.
2. From Table-4, including the two proposed data types only marginally improves performance, raising concerns about their benefits.
3. For the ablation study, it would be more informative and convincing to show performance on existing benchmarks such as ScanRefer, ScanQA, and SQA3D, rather than only the new tasks. Specifically, I am interested in seeing the effect of training data (tasks) on ScanRefer in the ablations.
4. I would like the authors to provide more qualitative results. Specifically, please run the models for the ScanNet validation set on scenes scene0568_00, scene0169_00, and scene0300_00 with the following prompts:

• “Describe the scene in detail.”
• “List the objects and their quantities in the scene.” Additionally, for scene0169_00, please prompt the model with: “Find the red backpack” and visualize its location.

5. Missing discussions of object-level and point-based 3D LLM like PointLLM, ShapeLLM, MiniGPT-3D, and token-based 3D localization models like SceneScript.
6. I would like to note that Chat-3D-v2 has an updated version (Chat-Scene, NeurIPS 2024), which demonstrates much better performance. While a comparison with this new version may not be necessary, both the authors and other reviewers should be aware of the performance gap. I suggest that the authors highlight the fact that the proposed method does not require an off-the-shelf 3D detector to pre-detect objects in the scene. This is a significant difference from some of the baselines (including Chat-Scene), and direct comparisons may be unfair, as models with pre-detected objects typically show better results.
7. I am willing to raise my score if the authors can address most of my concerns.

My main concern is twofold. First, the method lacks an important comparison with LEO (An Embodied Generalist Agent in 3D World, ICML'24). The reported performance on public benchmarks seems to underperform LEO’s results, and this discrepancy is not sufficiently explained. Second, the newly introduced task does not include important baseline comparisons. For the measurement and movement task, incorporating several simple approaches as meaningful baselines would enhance the evaluation. For example, using off-the-shelf 3D segmentation and detection to extract object bounding boxes for direct measurement or constructing scene graphs from 3D scenes and using existing LLMs for both measurement and movement tasks.

• Lack of some details about the proposed METHOD dataset, like the annotation pipeline.

• Some experiment results are missing. Only ablation studies on the METHOD dataset are provided to show the effectiveness of different components. However, I'm more curious about the performance of the existing 3D VLMs on the task of object distance measurement and layout editing. I think this comparison experiment is quite important to verify your motivation.

• Some notations are confusing. In the method section, the visual referent representation after the Intra-Referent module is named $F_{vr}$, while the feature after the Inter-Referent module is also called $F_{vr}$. I suggest the author use a different notation for the feature after the Inter-Referent module.