Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers

W1: Difference with Set-of-Mark.

• Different ways to introduce object identifiers. Set-of-Mark attaches object Identifiers directly onto the image, relying entirely on the multimodal LLM’s OCR capability to perceive the identifiers from the image. This method is indirect and can introduce ambiguity, especially when there are many objects in the image. Our method explicitly assigns the identifiers to each object in the language prompt. Intuitively, it is easier for the LLM to understand the link between the object and its identifier from the text.

• Inefficiency of adapting Set-of-Mark to 3D. Adapting Set-of-Mark for 3D perception requires cross-view clustering of the segmented masks (from SAM) and labeling object identifiers on multi-view images. However, currently there is no MLLM that can handle a sequence of images with a good OCR capability.

• Prompt-based vs Training-based. Set-of-Mark is a prompt-based method, which could produce uncontrollable outputs. Our training-based method can stably handle a series of tasks including grounding, captioning, and QA.

At last, our SOTA performance across various 3D benchmarks demonstrates the effectiveness of our proposed method to use object identifiers. Thus, the similar concept of object identifiers in Set-of-Mark does not diminish the contribution or novelty of our work at all.

W2.1: Evidence of object referencing and grounding abilities.

Section 3.3, along with Figures 2 and 3, illustrates how object identifiers are utilized during interactions with LLMs. Users can reference objects with identifiers for tasks like 3D dense captioning (Scan2Cap), while the LLM responds with identifiers to ground objects for both single object grounding (ScanRefer) and multiple object grounding (Multi3DRefer). The superior performance of our model on benchmarks such as Scan2Cap, ScanRefer, and Multi3DRefer, demonstrates its efficient object referencing and grounding abilities.

W2.2: Results of 3D referring expression segmentation.

Table A. Evaluation results of 3D referring expression segmentation on ScanRefer.

Table A shows that our method surpasses previous baselines for 3D referring expression segmentation on ScanRefer. It is important to note that we did not provide referring expression segmentation results on ScanRefer simply because it is not a common evaluation metric for this dataset. Most previous baselines assess accuracy based on the IoU between the predicted boxes and the ground-truth boxes. The comparison based on box IoU can already demonstrate our model’s grounding ability.

W3: Why proposed object-centric representations alleviate the problem of 3D data scarcity.

As discussed in Section 1 (Lines 72-84), training robust scene-level representations typically requires a large amount of paired scene-language data, which is difficult to obtain. To overcome this challenge, we represent scenes using object-centric representations derived from well-trained 3D and 2D encoders. Benefiting from at least million-level pre-training, the 3D encoder excels at extracting spatial and shape attributes from point clouds, while the 2D encoder adeptly extracts rich features of object’s appearance from multi-view images. Intuitively, we then propose to combine the well-trained 3D and 2D representations explicitly at the object level, along with the object identifiers, to comprehensively represent a whole scene.

Unlike previous 3D LLMs such as LEO[20] and 3D-LLM[22], which require constructing additional data (near million-level) for pre-training or alignment, our method achieves state-of-the-art performance without the need for additional alignment data. Considering that we adopt similar instruction tuning techniques as theirs, the biggest difference between our model and their models is the design of representations. Thus, our superior performance with less data usage can indicate that our proposed object-centric representations alleviate the problem of 3D data scarcity to some extent.

W4: Lack of comparison with SOTA methods.

In Table 2, we've included previous SOTA methods for each dataset:

• ConcreteNet [45] with the highest Acc@0.5 on ScanRefer,
• M3DRef-CLIP [63] with the highest F1@0.5 on Multi3DRefer,
• Vote2Cap-DETR++ [10] with the highest CIDEr@0.5 on Scan2Cap,
• Scene-LLM [17] with the highest CIDEr on ScanQA and the highest EM on SQA3D.

In Tables 6–10 in the appendix, we provide comprehensive comparisons on each dataset by including additional SOTA models.

CORE-3DVG is a missed reference on ScanRefer. However, although it surpasses our method by 1.25% on Acc@0.25, our method still demonstrates superior grounding performance with a 6.39% higher Acc@0.5. We will include CORE-3DVG as a baseline for comparison in the future version.

W1.1: Concerns about the recent trend of one-stage replacing two-stage methods in 2D.

Thanks for pointing out a promising future direction. However, one-stage models require large-scale training, for example, MDETR used 1.3M image-text pairs for pre-training. Given current limited 3D data (1200 scenes and 150K language annotations), our SOTA performance across various benchmarks highlights the effectiveness of our two-stage architecture. The data scaling and the exploration of end-to-end architectures should be left as a future work.

Q1.1 & W1.2: Discussion on the scope of using object-bottlenecked 3D VL models.

Firstly, it's important to note that open-vocabulary is not claimed as a contribution in our paper, nor do the previous baselines (either one-stage or two-stage) include formal open-vocabulary evaluations.

As discussed in the Limitation section (Lines 594-599), we acknowledge the limitation of relying on object detectors. Based on object detectors, our object identifiers can refer to object-level instances or clustered objects but fail to represent part-level concepts. We will clearly state this in the revised version.

Although part-level detectors such as PointNeXt[a] could be adopted to extract part-level instances, we still lack well-trained part-level encoders and related benchmarks. It is important to highlight that current large-scale 3D datasets are object-level, such as Objaverse[b] and OmniObject3D[c], and a large-scale, high-quality concept-level dataset is lacking. Therefore, the exploration of open-vocabulary/concept-level evaluations should be left for future work.

[a] PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies. NeurIPS 2022.

[b] Objaverse: A Universe of Annotated 3D Objects. CVPR 2023.

[c] OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation. CVPR 2023.

Q1.2: Zero-shot evaluation on other datasets.

Table A. Evaluation results on scene-language tasks 3RQA, 3RDialog, and 3RPlan, based on 3RScan.

To assess the generalizability of our model, we follow the precedent set by the 3D LLM method LEO[20] and conduct an evaluation on their proposed tasks: 3RQA, 3RDialog, and 3RPlan. These tasks are built upon the 3RScan[d] dataset, which belongs to a different domain than ScanNet. We use the same detection results as LEO and leverage our pre-trained weights (including the projection layer and LLM pre-trained on ScanNet). The results shown in the table above demonstrate our model’s zero-shot capabilities on 3RScan.

[d] RIO: 3D Object Instance Re-Localization in Changing Indoor Environments. ICCV 2019.

Q2: Discussion on the previous object-centric tokenization approaches.

Compared to previous object-centric tokenization methods, our approach introduces a novel way to incorporate a sequence of {object identifier, object features} into the LLM, enabling it to solve different tasks in a unified question-answering format.

For instance, ViLBERT uses an additional task head to predict matching scores for object grounding, while VisualBERT ranks entities by comparing attention weights. These methods require task-specific designs and heads for different tasks, which is impractical for real-world human-assistant interactions.

In contrast, our method establishes a direct link between the object and its identifier, allowing the LLM to respond with an object identifier as the grounding result. This approach can naturally extend to more complex tasks, such as multiple object grounding (outputting several identifiers for multiple grounding results) and grounded captioning (producing complex captions interleaved with identifiers as the grounding result).

Q3 & W2.2: Results on Nr3D/Sr3D.

Table B. Evaluation results on Nr3D/Sr3D.

The results show that our model achieves state-of-the-art performance compared to previous expert models.

Q4 & W2.1: Importance of object identifiers.

Table C. Ablation study on object identifiers.

Object identifiers are crucial to our model's design. They allow users to reference objects for tasks such as 3D dense captioning (Scan2Cap), while the LLM uses these identifiers for grounding objects in both single object grounding (ScanRefer) and multiple object grounding (Multi3DRefer).

Without object identifiers, it is necessary to adopt an alternative method for grounding and dense captioning. Following 3D-LLM[22], we add special location tokens to represent bounding boxes in the LLM. Thus, the model can output bounding boxes for grounding tasks and input bounding boxes for dense captioning. As shown in the table above, training the model without object identifiers reveals a significant decline in performance, particularly in grounding and captioning tasks.

Thank you for your detailed comments and constructive suggestions. We appreciate your recommendations regarding additional discussions on model architecture and related object-centric methods, as well as the suggestion to include more results and ablation studies. We will incorporate these in the final version of the paper. Below, we want to further address your concerns.

Details of the ablation study on object identifiers:

Following 3D-LLM[1], we use 6 discrete tokens to represent a bounding box. Specifically, we add 1000 special tokens (<LOC000>, <LOC001>, ... , <LOC999>) into the language model's vocabulary to discretely represent numeric values in the [0, 1] range. (Coordinates were normalized into [0, 1]). For example, an original bounding box defined as (x=0.234, y=0.467, z=0.129, w=0.301, h=0.235, l=0.189) would be represented as: <LOC234> <LOC467> <LOC129> <LOC301> <LOC235> <LOC189>. This method has shown effectiveness in 2D models such as OFA[2] and Pix2Seq[3]. However, both our experiments and 3D-LLM's results indicate that the location tokens are not well-learned due to the lack of 3D data.

LEO[4] and LL3DA[5] chooses to feed the encoded feature of a bounding box (or a single point) to the LLM to represent the user-referred object. However, this design can not be directly applied for outputting a bounding box. Therefore, these models are not able to do grounding tasks.

It is worth noting that Scene-LLM[6] feeds scene tokens (after voxelization) directly to LLM as you suggested. However, this model is only evaluated on QA tasks, as it lacks the ability to reference specific objects for captioning or grounding.

Consequently, our design, which employs object identifiers, is pioneering in unifying these tasks among 3D MLLMs. Notably, our model even achieves superior performance compared to expert models, highlighting its potential as a promising direction for LLMs in 3D tasks.

Multi-tasking ablation:

Thanks for the advice. The comparison between multi-task and single-task training could provide more insights of our method.

We conduct an experiment of the single-task training on ScanRefer. The result in the table above shows that the joint training on multiple tasks enhances performance. However, it is important to note that the comparison might not be entirely fair, as reduced data for single-task training also leads to fewer training steps per epoch. Adjusting hyperparameters could slightly improve single-task training performance. Anyhow, we will include a more comprehensive comparison between single-task and multi-task training in the final version.

[1] 3D-LLM: Injecting the 3D World into Large Language Models. NeurIPS 2023.

[2] OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework. ICML 2022.

[3] Pix2seq: A Language Modeling Framework for Object Detection. ICLR 2022.

[4] An Embodied Generalist Agent in 3D World. ICML 2024.

[5] LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning. CVPR 2024.

[6] Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning. arXiv 2024.

W1: Case study of the reliance on pre-trained detectors.

Please refer to Figure 1 the attached PDF in the “Author Rebuttal”.

We provide several qualitative cases where the detected objects are imperfect (such as incomplete point clouds or an object being separated into two or more parts). Despite the direct influence of incomplete masks on grounding quality, there are successful cases in captioning and QA tasks. The model's ability to perceive the surroundings of the objects allows it to infer the correct captions or answers.

It is worth emphasizing that previous SOTA models such LEO[20], 3D-VisTA[68], and M3DRef-CLIP[63] are also two-stage models reliant on pre-trained detectors, which face the same challenge as us when the detector fails.

W2: Zero-shot evaluation on other datasets.

Table A. Evaluation results on scene-language tasks 3RQA, 3RDialog, and 3RPlan, based on 3RScan.

Considering the lack of 3D scene understanding benchmarks built upon other datasets, we follow the precedent set by the 3D LLM method LEO[20] and conduct an evaluation on their proposed tasks: 3RQA, 3RDialog, and 3RPlan. These tasks are built upon the 3RScan[a] dataset, which belongs to a different domain than ScanNet. We use the same detection results as LEO and leverage our pre-trained weights (including the projection layer and LLM pre-trained on ScanNet). The results shown in the table above demonstrate our model’s zero-shot capabilities on 3RScan.

[a] RIO: 3D Object Instance Re-Localization in Changing Indoor Environments. ICCV 2019.

W3: About zero-shot open-vocabulary generalization.

Firstly, it's important to note that open-vocabulary is not claimed as a contribution in our paper, nor do the previous baselines include formal open-vocabulary evaluations.

The Table A above reveals some zero-shot open-vocabulary abilities of our model on novel scenes in 3RScan. For achieving real open-vocabulary capabilities, we can adopt open-vocabulary detectors such as SAMPro3D[b] and Open3DIS[c], and scale up the language data to train a robust MLLM. This remains a valuable direction in this field, and we tend to leave it for future work.

[b] SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Scene Segmentation. arXiv 2023.

[c] Open3DIS: Open-vocabulary 3D Instance Segmentation with 2D Mask Guidance. CVPR 2024.

Q1: How do DINOv2 features work in the 3D scene understanding task?

As described in Appendix A (Ln 531-541), we extract 2D representations from the area of projected masks in multi-view images of each object. DINOv2, trained with self-supervision objectives at both image and patch levels, captures detailed local information such as shape and texture, providing fine-grained perception abilities. This aligns with our design goal of extracting rich object-centric representations. The ablation results in Table 4 demonstrate the importance of these 2D representations in our final model.

Table B. Ablation study on the 2D encoder.

We also add an ablation study replacing DINOv2 with CLIP, which was trained with image-level contrastive learning and tends to neglect rich pixel-level details. The results show that using the CLIP encoder leads to lower performance, particularly on grounding tasks.

W1: Details about multi-modal inputs, detectors, encoders, and LLMs used in other methods.

Please refer to Table 1 in the attached PDF in “Author Rebuttal”.

W2: Results on ScanQA test set.

W3 & Q2: Ablations about LLMs and training schema.

1. Size of LLM: We compared various sizes of LLMs: Vicuna 1B, 7B, and 13B. The 7B model achieves the best performance on grounding tasks, while the 13B model excels in captioning and QA tasks. The 13B model does not show significant performance gains over the 7B model, suggesting that the current task scope and data scale do not challenge larger LLMs.

2. Type of LLM: We tested the OPT-1.3B, which performs slightly worse than the Vicuna-1B of similar size. As for multimodal large models like LLaVA and LEO, basically they use the same LLM backbone (Vicuna-7B) and the projection layer (MLP) as ours. However, directly adapting their pre-trained weights to our model is unsuitable because we use different 2D or 3D representations.

3. Training schema: We compare one-stage training with two-stage training (fine-tuning on each dataset). Two-stage training improves performance on grounding tasks but significantly decreases performance on captioning and QA tasks. The performance decline in captioning and QA tasks may be due to these tasks being easier to converge, leading to overfitting during further fine-tuning.

W4: Computation and time cost.

Compared to 3D-LLM[22], our model's simpler design significantly reduces computation and time costs.

W5: Discussion about object-centric representation.

Firstly, it’s important to note that the object-centric representation is not claimed as a contribution in the paper. We state that our study is based on well-trained encoders, and our contribution is on how to incorporate a sequence of {object identifier, object features} into the LLM, which can solve various 3D scene-language tasks in unified formats. More comparisons among different object-centric representations can be left as a future work.

Q1: Importance of object identifiers.

Order of Object Identifiers: In our actual implementation, we randomize the order of object identifiers during training to minimize the influence of any inherent order distribution. Our results show that using a fixed order of identifiers yields slightly poorer performance compared to using a random order.

Removing Object Identifiers: Object identifiers are crucial to our model's design. They allow users to reference objects for tasks such as 3D dense captioning (Scan2Cap), while the LLM uses these identifiers for grounding objects in both single object grounding (ScanRefer) and multiple object grounding (Multi3DRefer). Without object identifiers, it is necessary to adopt an alternative method for grounding and dense captioning. Following 3D-LLM[22], we add special location tokens to represent bounding boxes in the LLM. Thus, the model can output bounding boxes for grounding tasks and input bounding boxes for dense captioning. As shown in the table above, training the model without object identifiers reveals a significant decline in performance, particularly in grounding and captioning tasks.

Q3: Details about training data.

As a comparison, LEO[20] used 1.2M language training data, and 3D-LLM[22] used 700K, which are several times more than our training data (155K).

Q4: Details about the inputs at inference.

All objects detected by the 3D detector are combined as input. For each object, the extracted 3D and 2D representations are projected into the LLM’s embedding space, becoming 3D and 2D tokens. Consequently, each object is represented by three tokens: an identifier token, a 3D token, and a 2D token. As illustrated in Figure 2, these objects form a token sequence of length 3$n$, where $n$ is the number of objects. For more details, refer to Section 3.3 for the prompt template and Section 3.2 for feature extraction.