SpatialLM: Training Large Language Models for Structured Indoor Modeling

We thank the reviewer for the comments. Below is our detailed response.

Weaknesses

1. Thanks for the question. Indeed, our work is inspired by SceneScript. However, a key difference is that SceneScript proposes a task-specific tokenizer, which is not compatible with the text tokenizers used in modern LLMs. And it trains the tokenizer with a task-specific decoder from scratch (please refer to Sections 4.2 and 4.3 of SceneScript paper for details). In other words, a new domain-specific language with the associated tokenizer and decoder is proposed in Scenescript. This is also noted in Section 4.1 (lines 302 - 303) of our paper.

   In contrast, SpatialLM directly exploits modern LLMs’ capacity for Python language generation, without designing any new tokenizer or decoder. Thus, much of our work is focused on how to align point cloud features to existing LLMs with a standard “Encoder-MLP-LLM” architecture (Section 2), which has not been carefully studied before.

   Furthermore, by building upon existing LLMs, our experiment in Section 3 also reveals other differences from SceneScript. For example, as shown in Tables 5 and 6, while training on existing datasets like Structured3D and ScanNet may be sufficient for SceneScript to achieve competitive performance, these datasets are too small for SpatialLM. Instead, by first training our new dataset, SpatialLM outperforms SceneScript on both tasks.

   To summarize, our work is the first to employ modern LLMs for structured scene reconstruction (i.e., layout estimation and 3D object detection) from point clouds. Our contributions include (i) a new large-scale dataset and (ii) thorough ablation studies on aligning point cloud features with LLMs. Furthermore, we show that SpatialLM can achieve competitive performances against existing task-specific models. We believe this is an important step towards equipping LLMs with general 3D scene understanding and generation capacities.

2. Thanks for the comment. In fact, after the paper submission, we found a bug in our implementation related to using Fourier encoding of voxel indices as positional embeddings to the output Sonata features. After fixing the bug, the performance of SpatialLM on both layout estimation and 3D object detection further improved. The new results are given below.

   Table 5. Experiment on layout estimation.

   Method	F1 (IoU$_\textrm{2D}$@0.25)	F1 (IoU$_\textrm{2D}$@0.5)
   RoomFormer	70.4	67.2
   SceneScript	83.1	80.8
   SpatialLM (ft. Structured3D)	32.8	17.9
   SpatialLM (ft. Ours)	51.2	38.3
   SpatialLM (ft. Ours -> Structured3D)	86.5	84.6

   Table 6. Experiment on 3D object detection.

   Method	F1 (IoU$_\textrm{3D}$@0.25)	F1 (IoU$_\textrm{3D}$@0.5)
   V-DETR	65.1	56.8
   SceneScript	49.1	36.8
   SpatialLM (ft. ScanNet)	2.9	0.7
   SpatialLM (ft. Ours)	33.8*	22.6*
   SpatialLM (ft. Ours -> ScanNet)	65.6	52.6

   As one can see, for layout estimation, SpatialLM consistently outperforms other baselines, with +3.4 IoU2D@0.25, +3.8 IoU2D@0.5 over SceneScript. For 3D object detection, we have +16.5 IoU3D@0.25, +15.8 IoU3D@0.25 over SceneScript, and +0.5% IoU3D@0.25, -4.2 IoU3D@0.5 compared to V-DETR.

   Finally, we note that our primary goal of the work is not merely competing for the highest scores on existing benchmarks. Instead, with the experiments, we aim to show a feasible path to tackle 3D scene understanding tasks with the MLLM architecture.

3. Thanks for the suggestion. We will revise the paper to present the related work section earlier.

Questions

4. Compared to domain-specific box texts, the Python script is more general and flexible. For example, one may specify various relationships, such as associating door/window with a wall, by passing the necessary parameters. It can also be easily extended to include an arbitrary number of object classes or other properties. Further, since LLMs have already seen a large amount of Python code data, it is easier to align the scene descriptions with other LLM capacities, such as tool calling.

5. Please refer to item #1 above.

6. To ensure the diversity of our dataset, we mainly resort to our access to a large repository of interior designs from a leading platform in the interior design industry. As described in Section 2.1, most 3D scene designs in our dataset are created by professional designers and used for real-world production. This way, we ensure that the distribution of 3D scenes in our dataset closely resembles that of real-world houses. Please refer to Section 1 of the supplementary material for statistics of the rooms and objects in our dataset.

   During dataset curation, we select designs (i.e., scenes) by considering the following factors: (i) ratings by the professional designers, (ii) the number of renderings generated by the design, (iii) total floor area > 20 m$^2$, and (iv) number of unique objects > 35. For objects, we pick the 59 common object categories based on occurrence while filtering objects in the long-tail distribution. And as mentioned in Section 2.1, we filter out small objects with a side length < 15cm.

I appreciate that the authors provide a detailed comparison with SceneScript. However, I think the difference is a little engineering, and my concerns regarding the incremental originality of this paper are not fully addressed. Furthermore, the experimental results are fixed during the discussion phase, making this paper potentially not ready for publishing. I would recommend that the author further consider the benefits of evolving the paradigm of SceneScript by building it upon LLM (maybe extend it to more general tasks), which can make the submission more convincing. Therefore, I would keep my original rating.

We thank the reviewer for the comments. Below is our detailed response.

Weakness

1. [Architectural contributions of our work] There seems to be a misunderstanding about the contributions of our work. In this paper, we use existing point cloud encoders such as Sonata and connect it to LLM with the standard “Encoder-MLP-LLM” architecture. We do not propose any new architectural modifications, such as “modifying the attention mechanism to reflect geometric proximity", as suggested by the reviewer.

   Instead, our primary goal is to train LLMs for structural indoor modeling tasks (i.e., layout estimation and 3D object detection). As stated in Section 1, our main contributions are

   • (1) We regard the structured descriptions as scripts of a general-purpose language (i.e., Python) and propose to predict the language in text form,
   • (2) We propose a new synthetic dataset for the task, and conduct the first empirical study on the best strategy of aligning the point cloud input with LLMs for structured scene modeling, and
   • (3) We empirically show that, by first training on our large-scale dataset and then on smaller downstream task data, our model gives competitive performance on public benchmarks.

2. [Model’s dependency on clean, discrete scene layouts] There seems to be a misunderstanding of the SpatialLM’s input. For example, the reviewer wrote “The model takes as input a structured scene layout consisting of object labels and their associated 3D attributes (position, size, orientation), and predicts the 3D bounding box corresponding to a natural language query.” But SpatialLM actually takes 3D point cloud as input and generate structured scene descriptions as output. In other words, the “structured scene layout” is the output of our model, not the input.

   Similarly, the reviewer wrote that “The method assumes access to structured scene layouts composed of object-centric tokens with precise 3D attributes including class, position, orientation, and size.” This statement is not accurate for the same reason.

   In terms of handling incomplete or noisy point cloud inputs, please note that in Section 3.2, we conduct experiments on ScanNet, which is a real dataset with point clouds reconstructed with imperfect RGB-D data with holes, occlusions, and noise. In Section 3.3, we also report 3D indoor modeling results with point clouds reconstructed from monocular videos. These experiments demonstrate the robustness of our model to real-world inputs.

3. [Investigation into the spatial bias mechanism] We point out that SpatialLM follows the standard “Encoder-MLP-LLM” architecture for MLLMs. We leverage existing networks such as Qwen2.5-0.5B as the base model and Sonata as the point cloud encoder. We respectfully refer the reviewer to the Sonata paper [1] for a detailed study about how the point cloud encoder captures spatial context and priors.

4. [Generalization under distributional shift, ambiguous language input, or sparse spatial cues] In Section 3.3 (Zero-shot Detection on Videos), we show that our model exhibits reasonable generalizability across datasets. However, as we acknowledged in Section 4, fine-tuning SpatialLM on a task-specific dataset is still needed to achieve the best performance. Further scaling up the data and model size could be a promising future direction.

5. [Limitation to task-specific improvements] We acknowledge that SpatialLM, in its current form, is a task-specific model for structured indoor modeling. Our plan is to focus on this well-established task first, and to show a feasible path to tackle 3D indoor modeling tasks (i.e., layout estimation and 3D object detection) with LLMs.

   In future work, we can improve its generalizability by generating open-vocabulary annotations with MLLMs and training them together with large-scale vision-language datasets. In fact, it is not uncommon for researchers to focus on one capacity of MLLMs at a time, such as video understanding [2] and function calling [3], with the understanding that the findings will later be integrated into a general-purpose foundation model.

Questions

6. [Model’s sensitivity to the accuracy and sparsity of 3D layout] Please refer to item #2 above.

7. [Analysis on spatial inductive bias] Please refer to item #3 above.

8. [Generalization to unseen spatial configurations] Please refer to items #4 above.

9. [Achieving spatial reasoning by finetuning with task-specific data alone, without architectural modifications] Please note that we do not propose any architectural modifications in this paper. Instead, we follow the standard “Encoder-MLP-LLM” architecture for MLLMs, and leverage existing networks such as Qwen2.5-0.5B as the base model and Sonata as the point cloud encoder.

Limitations

We respectfully refer the reviewer to our response to the paper's weaknesses above for discussions about the contributions of our work, the input of SpatialLM, investigation on the spatial inductive bias, generalization to unseen object configurations, and potential impact on 3D scene understanding.

References

• [1] Wu et al., Sonata: Self-Supervised Learning of Reliable Point Representations. In CVPR, 2025.
• [2] Zohar et al., Apollo: An Exploration of Video Understanding in Large Multimodal Models. In CVPR, 2025.
• [3] Zhang et al., xLAM: A Family of Large Action Models to Empower AI Agent Systems. In NAACL, 2025.

We thank the reviewer for the comments. Below is our detailed response.

Weaknesses

1. Thanks for the thoughtful comments. We acknowledge that SpatialLM, in its current form, is a task-specific model for structured indoor modeling. Our plan is to focus on this well-established task first, and to show a feasible path to tackle 3D indoor modeling tasks (i.e., layout estimation and 3D object detection) with LLMs.

   In future work, we can improve its generalizability by generating open-vocabulary annotations with MLLMs and training them together with general-purpose vision-language datasets. In fact, it is not uncommon for researchers to focus on one capacity of MLLMs at a time, such as video understanding [1] and function calling [2], with the understanding that the findings will later be integrated into a general-purpose foundation model.

2. Thanks for the suggestion. In the revised paper, we will add columns to the tables to make the model complexity and data factors clearer.

3. [Lines 24 - 26: difference from SceneScript] Thanks for the question. Indeed, our work is inspired by SceneScript. However, a key difference is that SceneScript proposes a task-specific tokenizer, which is not compatible with the text tokenizers used in modern LLMs. And it trains the tokenizer with a task-specific decoder from scratch (please refer to Sections 4.2 and 4.3 of SceneScript paper for details). In other words, a new domain-specific language with the associated tokenizer and decoder is proposed in Scenescript. This is also noted in Section 4.1 (lines 302 - 303) of our paper.

   In contrast, SpatialLM directly exploits modern LLMs' capacity for Python language generation, without designing any new tokenizer or decoder. Hence, the first contribution "regard the structured descriptions as scripts of a general-purpose language (i.e., Python) and propose to predict the language in text form".

   Furthermore, by building upon existing LLMs, our experiment in Section 3 also reveals other differences from SceneScript. For example, as shown in Tables 5 and 6, while training on existing datasets like Structured3D and ScanNet may be sufficient for SceneScript to achieve competitive performance, these datasets are too small for SpatialLM. Instead, by first training our new dataset, SpatialLM outperforms SceneScript on both tasks.

4. [Line 297: RoomFormer is a single-stage network using two-level queries] Thanks for pointing this out. We will fix it in the revised paper.

5. [Organization of the paper] Thanks for the suggestion. We agree that Section 2 primarily discusses the design choices made in SpatialLM through a series of ablation studies, whereas Section 3 compares SpatialLM to existing methods on various benchmarks. We will adjust the section titles to better reflect their content in the revised paper.

Questions

6. Please refer to item #3 above.

7. SpatialLM is trained on our new dataset with a mix of single-room layouts and multi-room layouts. Therefore, it can do inference in both single-room and multi-room manners.

8. For SPTv3, we follow common practice to leverage a pre-trained model as a starting point, rather than training it from scratch. Please note that our claim in lines 198 - 199 is more of a comparison of existing 3D point cloud encoders to powerful 2D image encoders, SigLIP and DINOv2, where one can freeze the image encoders during MLLM training and still obtain good results.

References

• [1] Zohar et al., Apollo: An Exploration of Video Understanding in Large Multimodal Models. In CVPR, 2025.
• [2] Zhang et al., xLAM: A Family of Large Action Models to Empower AI Agent Systems. In NAACL, 2025.

We thank the reviewer for the comments. Below is our detailed response.

• [Confirmation of the open-sourcing plan.] Yes, we will open-source the dataset as promised in the paper.

Weaknesses

1. [Waste of MLLMs' generalizability] We acknowledge that SpatialLM, in its current form, is a task-specific model for structured indoor modeling. Our plan is to focus on this well-established task first, and to show a feasible path to tackle 3D indoor modeling tasks (i.e., layout estimation and 3D object detection) with LLMs. By limiting to a predefined set of objects, we are able to conduct a direct comparison with SOTA methods, such as Roomformer and V-DETR, on existing public benchmarks.

   In future work, we can improve its generalizability by generating open-vocabulary annotations with MLLMs and training them together with large-scale vision-language datasets. In fact, it is not uncommon for researchers to focus on one capacity of MLLMs at a time, such as video understanding [1] and function calling [2], with the understanding that the findings will later be integrated into a general-purpose foundation model.

2. [Comparison to a strong 3D point cloud feature + a strong segmentation model] Indeed, several studies in the literature detect 3D objects via point cloud instance segmentation, as suggested by the reviewer. For example, in V-DETR, the authors also included some point cloud instance segmentation methods, such as 3D-MPA, as baselines. Here, we note that, as a bottom-up approach, the instance segmentation methods have several limitations. First, they can only output axis-aligned bounding boxes (AABB) without an orientation. Second, they may fail to predict the correct object sizes in the presence of holes, occlusions, and noise. In contrast, our method can infer full object sizes and orientations from sparse or occluded inputs (please see lines 284 - 291 of the paper for more discussion and some examples).

   Furthermore, as we stated earlier, our long-term goal is to equip MLLMs with the capacity to make structured predictions in text form for 3D indoor modeling. Therefore, a dense prediction method for instance segmentation is not suitable for our purpose.

3. [Use of Python language] Thanks for the suggestion. In Section 4.2 of the supplementary material, we provide some preliminary results on task adaptation via language-based prompts, including (i) detection with user-specified categories and (ii) semantic label completion. As for layout editing, we believe it is feasible with our current dataset by generating paired data with language instructions and editing actions in the future. But in this paper, our focus is on the reconstruction of the layout from point clouds.

4. [Robustness to imperfect point clouds] Please note that in Section 3.2, we conduct experiments on ScanNet, which is a real dataset with point clouds reconstructed with imperfect RGB-D data with holes, occlusions, and noise. In Section 3.3 of the supplementary material, we also report quantitative 3D indoor modeling results with point clouds reconstructed from monocular videos (Table 6). Please refer to the supplementary results for more discussions.

Questions

5. Line 167 (5-level hierarchical structure): The 5-level hierarchical structure refers to the 5 layers of neural blocks in the 3DCNN and Sonata/PTv3 encoders. The first layer doubles the feature dimensions but keeps the point cloud resolution the same, and the following 4 layers each downscales the point cloud resolution to half. We will add a figure in the revised paper to illustrate this structure.

6. Thanks for the interesting question. Potentially, we can use SpatialTrackerV2 [3] to reconstruct the 3D point cloud of both static scenes and dynamic moving objects with 3D tracking trajectories. Then, we may use a SpatialLM-like architecture to reconstruct the static layout and the moving object bounding boxes in different time steps auto-regressively. In addition, a large-scale point-cloud dataset with annotations of dynamic objects (e.g., humans) is needed.

7. In this paper, we only keep the 59 common categories while filtering objects in the long-tail distribution. To improve the model’s generalizability, one may augment the labels of all objects in our dataset with VLM captions. Specifically, one can get a diverse set of annotations by performing image captioning with rendered images of each object in our dataset (as shown in Figure 4 of the supplementary material). Alternatively, researchers may use a different dataset of their own for their specific applications. In either case, a one-stage training schedule as discussed in Section 2.3 is likely required.

   Besides, it is worth noting that the amount of training required also depends on the point cloud encoder. The Sonata/PTv3 point cloud encoder used in SpatialLM follows a DINO-like self-supervised training strategy. Such a strategy focuses on capturing local and global visual features, and does not consider alignment between visual features and text. A more powerful point cloud encoder with enhanced vision-language alignment will likely reduce the amount of training required, while better supporting language-based applications.

8. In this paper, we focus on training LLMs to generate a structured scene description from point clouds only. This is one of the limitations of the paper (as stated in lines 335 - 337). It cannot answer scene-related questions based on the point clouds in natural language, as it is not trained to do so. And after finetuning, its general language generation capabilities may also be severely impacted.

   However, as we discussed before, we believe our work is an important first step towards equipping LLMs with general 3D scene understanding and generation capacities. And by training together with other general-purpose vision-language datasets, we can restore and improve the model’s general language generation capabilities.

References

• [1] Zohar et al., Apollo: An Exploration of Video Understanding in Large Multimodal Models. In CVPR, 2025.
• [2] Zhang et al., xLAM: A Family of Large Action Models to Empower AI Agent Systems. In NAACL, 2025.
• [3] Xiao et al., SpatialTrackerV2: 3D Point Tracking Made Easy. In ICCV, 2025.

Thank authors for the response. My questions and concerns are largely addressed. I do notice the concern raised by reviewer ML3L, that the originality of this work is compromised by the prior work SceneScript. While I acknowledge the difference in technical approaches and the improved performance, I feel reviewer ML3L's concern is also valid.

Set aside this concern, I highly look forward to the opensourcing of the entire dataset. I believe it will be a great contribution of this work to the community. Thank the authors for confirming the open-sourcing decision. However, this is hard to assess in the current double-blind reviewing stage. In light of this, I am willing to stay at my rating, but wish to slightly tune it down to a hypothetical 4.5 for AC's reference.