3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Referred Object Grounding

Thank you for your valuable feedback. We have addressed the raised concerns by modifying the sequence generation method describing an object, adding a 2D object token alongside the object identifier token. This modification, inspired by the Chat-Scene method [1], improved the stability of the 3DGraphLLM's convergence without altering the core contribution of this work: developing a method for creating a trainable representation of the 3D semantic scene graph and an algorithm for generating a flat sequence from the graph based on the object's k-nearest neighbors. Additionally, we conducted further experiments with the NMS filter and found that it provides an additional improvement in the quality of duplicate object filtering.

LLMs are known for capturing a significant amount of common sense knowledge and show OOD generalization ability. ... Ideally, by using an LLM, I expect to see that in addition to taking the new modality (3D scene graph) as input, the adapted LLM also shows that its common sense knowledge is also being exploited in a non-trivial way, e.g. OOD generalization, achieving tasks other than grounding. ...

The revised version of 3DGraphLLM shows a significant boost in the performance of 3D Referred Object Grounding, Scene Captioning, and Question Answering (QA) tasks compared to baseline expert models that do not use pre-trained LLMs, including those that rely on multi-task learning (3D-Vista, PQ3D). As shown in Table 2 in the general response, methods based on LLMs significantly outperform expert models in the Situated Question Answering task (SQA3D). Finally, the use of LLMs enables answering various types of queries, including those requiring common sense knowledge (see Appendix Section C in the revised version of the paper).

The proposed method relies heavily on the performance of the underlying 3D segmentation and scene graph generation models used to encode 3D objects and relationships into features...

Indeed, the results of our experiments show that the greatest impact of using a 3D semantic scene graph is observed in when using GT instance segmentation of 3RScan scene dataset (up to an 8% improvement in object grounding). However, using pre-trained models for object proposals is a common approach for solving 3D Vision-Language tasks. For example, this approach is used in methods like Chat-3D-v2, Grounded 3D-LLM, and Chat-Scene. Despite the dependence on the quality of 3D segmentation, such methods, including 3DGraphLLM, show significantly better results than methods using voxel (Scene-LLM) or point cloud (3D-LLM) representations of the scene. Combined with nearest neighbor filtering and LoRa fine-tuning, 3DGraphLLM shows an improvement of about 3% for the 3D object grounding task, as well as an increase in the quality of Scene Captioning (+3-4% CIDEr@0.5), while maintaining comparable quality in the QA task (see Table 1 in the general response).

Why is the flat graph representation object-centric ..., instead of further flattened...? ... Therefore, there seems to be still a lot of waste in the proposed tokenization scheme.

When creating an object-centric approach to the flat representation of a scene graph, our motivation was based on two points:

• The object-centric scene representations work well when used as input to LLM(Grounded 3D-LLM, Chat3Dv2, Chat-Scene).
• We wanted to provide the LLM with not only the type of relationship between two objects but also information about which two objects the relationship connects.

In the naive approach, where we simply take the semantic relationship tokens and move them to the end of the object list, we lose information about which specific two objects are connected by the relation rel1. However, your comment is valid, and one direction of future work may be to reduce the number of tokens needed to encode the relationships between objects in our graph representation. We have added this direction for future work to the conclusion of the revised version of the article.

This paper proposes to train the model in a two-stage manner ..., but this is not being ablated.

We added an ablation for training in a two-stage manner, which shows that using pre-training with GT instance segmentation improves the performance of the 3D Referred Object Grounding task, while keeping the solution for the Scene Captioning task better than the baseline method, where no edges between objects are present. Please refer to Table 1 in the general response.

... I think these duplicates should be handled using filtering techniques like 3D NMS rather than the proposed naive minimum distance filter on the scene graph edges.

Thank you for the comment. We added the NMS filter to our pipeline, which resulted in an additional performance boost (see Table 3 in the general response).

[1] Huang, H., et al. (2024). Chat-scene: Bridging 3d scene and large language models with object identifiers. In The 38 Annual Conference on Neural Information Processing Systems.

Thank you for your response! Yes, we conducted this experiment during the rebuttal period. We removed repeated object tokens but kept the relation tokens next to the object tokens to ensure the approach also works in cases where the number of objects differs between scenes. That is, the sequence is the following:

$[<OBJ001> F_{1}^{2d}, F_{1}^{v}, F_{12}^{e}, F_{14}^{e}, ...<OBJN>F_{N}^{2d}, F_{N}^{v}, F_{Nk_1}^{e}, F_{Nk_2}^{e}$]

Below is a table comparing this approach (second row) with the approach described in the paper, where relations are represented as triplets (third row). As seen from the table, using relations in the form of triplets improves the quality of 3D Referred Object Grounding task performance.

Thank you for your valuable feedback. We would like to address the raised concerns:

The proposed modification introduces a parameter of minimum distance for filtering the nearest neighbors. It requires manual tuning, as is set to be 1 cm as in the paper. This requirement might potentially make it hard to apply this approach to new domains, such as a tabletop scene or a larger multi-floor multi-room indoor scene.

The primary contribution of our work is a method for creating a learnable representation of the scene graph for LLMs, which improves the performance of 3D Vision-Language tasks compared to methods that do not leverage semantic relationships between objects. When using ground-truth point cloud segmentation, 3DGraphLLM does not require a distance-based nearest neighbor filter. The motivation for selecting a distance threshold is solely to filter out duplicate objects. The minimum distance between two ground-truth objects in the ScanNet dataset is approximately 5 cm, so we expect that a 1 cm distance filter is sufficiently lenient. We also conducted experiments with the Non-Maximum Suppression (NMS) filter, which improved the proposed duplicate object filtering method (see Table 3 in the the general response) and can be independently applied in other domains. Furthermore, if it is possible to construct a higher-quality scene graph in a new domain, additional nearest neighbor filtering might not be necessary.

What is the difference between the “3DGraphLLM-0” (in table 3, 4) and “Chat-3D v2” baseline (in table 2)? It is said in Section 4.4 that they are equivalent (“The 3DGraphLLM version with zero nearest neighbors…, equivalent to the Chat3Dv2 approach..”). However, under the ScanRefer benchmark, the 3DGraphLLM-0 (in table 4) outperforms Chat3Dv2 (in table 2) by more than 10% margin in . Similarly, why is the performance of 3DGraphLLM-0 under GT segmentation not consistent in table 3 and 4?

The Chat3Dv2 baseline in Table 2 uses PointGroup as the instance segmentation method, as well as the Vicuna-7B LLM, and a slightly different training procedure. 3DGraphLLM-0 corresponds to the Chat3Dv2 version with the same training parameters as 3DGraphLLM: Mask3D as the instance segmentation method, LLAMA3-8B-Instruct LLM, and a two-stage training procedure. Table 4 differs from Table 3 in that it pertains to the Ablation study series in Section 4.4.1, where we use the frozen version of the LLM without LoRa fine-tuning, and do not introduce object identifier tokens. Note that in the revised version of the paper, Table 3 corresponds to the ablation study for different LLM models and GT pretraining. In the revised version of the paper, the closest baseline to 3DGraphLLM is Chat-Scene[1].

The proposed approach uses Uni3D for object encoding and VL-SAT for edge encoding. Since VL-SAT also contains an object encoder, which might be even more compatible with its own edge encoder, why is the Uni3D encoder used instead?

VL-SAT was developed to solve the task of generating a 3D semantic scene graph in the domain of the 3RScan scenes. It uses a fixed set of classes for vertex and edge classification along with the ground truth instance segmentation of the scene. Uni3D was trained on a large corpus of 1M point clouds, aligning point cloud representations with text descriptions and images, which is why we assume it has better generalization to new domains compared to VL-SAT object encoder. We expect that the compatibility of the vertices and edges representations is achieved through the training of projection layers and model LoRa fine-tuning. However, we consider as the future work the research on scene graph construction methods that do not rely on ground truth segmentation of the scene. These methods may produce consistent representations of point clouds and the relationships between them.

What are the common failure cases of the method?

A common failure case for our method occurs when answering questions that involve semantic relationships between objects that depend on the viewpoint (e.g., left/right, front/behind). Please refer to Appendix Section B in the revised version of the paper.

[1] Huang, H., Chen, Y., Wang, Z., Huang, R., Xu, R., Wang, T., ... & Zhao, Z. (2024). Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Thank you for your feedback. We would like to address your comments:

In the Edge Feature Encoder, the important issue is raised that the semantic relationships recognition methods are all trained on 3RScan scenes, while the visual grounding tasks are typically tested on ScanNet scenes. ... The authors claim that the solution is to use VL-SAT due to its good generalization in the new domain. I am very familiar with VL-SAT, and currently, there is no evidence showing that this method has good generalization in the new domain to address the mentioned challenge.

In our experiments, when using GT instance segmentation, incorporating VL-SAT edge features in our 3DGraphLLM method improves performance in the 3D Referred Object Grounding task by +5.6% on the ScanNet scene dataset (ScanRefer dataset, Table 4) and +7.5% on the 3RScan scene dataset (RioRefer dataset, Table 5). Although the use of VL-SAT semantic edges is slightly less effective for ScanNet scenes compared to 3RScan scenes, this experiment demonstrates that VL-SAT exhibits good generalization across the two scene datasets for our task. This may be due to the fact that we use VL-SAT edge features before their classification into 27 categories.

Similarly regarding VL-SAT, the authors seem to have used the VL-SAT pre-trained on 3RScan to test on the ScanNet dataset. This is equivalent to the results of RIORefer->ScanRefer in Cross3DVG. In Figure 5, when the number of nearest neighbors = 2, the Acc@0.5 reaches an astonishing 49.0, which not only significantly surpasses the result of RIORefer->ScanRefer in Cross3DVG (13.34) but also shows a remarkable difference compared to the RIORefer->RIORefer test result (20.21) given in Cross3DVG. .... I hope the authors can provide some explanations regarding the above points and present more experimental results on RIORefer during the rebuttal period....

As we understand Table 2 in the Cross3DVG paper, the values Acc@0.5 13.34 and 20.21 correspond to the ScanRefer->RIORefer and ScanRefer->ScanRefer settings, respectively. Given this, 3DGraphLLM indeed outperforms the expert model from Cross3DVG in the ScanRefer->ScanRefer setting. Moreover, in the revised version of the paper, 3DGraphLLM also surpasses other expert models in the ScanRefer->ScanRefer setting (see Table 2 in the general response). Unfortunately, conducting experiments with instance segmentation on the 3RScan scene dataset requires additional work to verify the generalization capability of Mask3D to 3RScan. Therefore, we leave these experiments for future work.

Table 4 needs to be completed (i.e., providing the results of 3DGraphLLM-2 with OneFormer3D segmentation when the minimal distance = 1cm), as this is crucial for validating the effectiveness of the minimal distance strategy.

Thank you for your comment. We conducted this experiment in the updated version of the paper. This experiment demonstrates that the combination of the NMS filter and the minimum distance filter is also effective for OneFormer3D:

The results of VL-SAT are based on training with 160 objects and 27 relationships. I would like to know how much these 27 relationships help in ScanRefer. ... I hope the authors can explore whether the annotations in the 3D Scene Graph Generation dataset are related to the descriptions in visual grounding.

We calculated the ratio of annotations in ScanRefer where object descriptions do not include any of the 27 VL-SAT relations. We use two methods to identify these descriptions. “Exact matching” refers to the precise detection of mentions of one of the 27 relations in the object descriptions in ScanRefer. As shown in the table below, in this case, more than half of the annotations in ScanRefer contain at least one of the 27 VL-SAT relations. Since we use a pre-trained LLM and VL-SAT hidden edge features, we also investigate how many ScanRefer annotations include synonyms of the 27 VL-SAT relations. We consider closely related synonyms such as "under," "on," "at," "is mounted," "is affixed," "near," and "next." Accounting for these spatial relations, the proportion of descriptions containing spatial relations different from 27 VL-SAT relations reduces to 12% in both the training and validation sets.

Thank you to the authors for their response. My concerns have been addressed, and I would like to raise my score to 7. However, since there is no option for that and only an 8 is available, I feel that an 8 is not appropriate for this paper. Therefore, I will maintain my current score and increase my confidence level to 5. Wishing you the best of luck!

Thank you for your valuable feedback. We have addressed the raised concerns by modifying the sequence generation method describing an object, adding a 2D object token alongside the object identifier token. This modification, inspired by the Chat-Scene method [1], improved the stability of the 3DGraphLLM's convergence without altering the core contribution of this work: developing a method for creating a trainable representation of the 3D semantic scene graph and an algorithm for generating a flat sequence from the graph based on the object's k-nearest neighbors. Additionally, we conducted further experiments with the Non-Maximum Suppression filter and found that it provides an additional improvement in the quality of duplicate object filtering.

The LLM used by this paper is the recent LLAMA3-8B-Instruct while other baselines use different and older LLMs. for eg. Chat3Dv2 and Grounded 3D LLM uses Vicuna-7B (based on LLAMA 2). This makes the comparison unfair with prior models.

We conducted an additional experiment with Vicuna1.5-7B (used by Chat-Scene[1] - the closest baseline for our method). As shown in Table 1 in the general response, 3DGraphLLM-Vicuna1.5-7B still provides an advantage compared to baseline methods. However, it is worth noting that semantic edges offer a greater quality boost for LLAMA3-8B-Instruct, which is why we use it in our method.

The paper does not report results on other benchmarks like ScanQA, SQA3D and Scan2Cap while still training on it, and showing ablations on it. The paper mentions that the focus is on visual grounding tasks, and that is why the evaluations are only shown in these benchmarks; however the key idea of using scene graph representation might also help for these other spatial reasoning tasks and if not, should atleast be reported for completeness in my opinion.

We have updated Table 2 in the revised version of the paper and in the general response with the results of the revised 3DGraphLLM method on the ScanQA, SQA3D, and Scan2Cap datasets. As shown in Table 2 (and Table 1 in the general response), the use of the scene graph also has a positive impact on solving the Scene Captioning task while maintaining comparable performance on the Question Answering tasks, ScanQA and SQA3D.

The boost in performance by using the proposed scene graph technique is only around 1-2% (Table-4) in the setup when models uses masks from a detector like Mask3D (which represents the realistic setup). This boost comes at a cost of additional complexities of making the scene graphs and querying the LLMs with additional tokens (additional ~800 tokens per query).

The primary contribution of our work is a method for creating a learnable representation of the scene graph for LLMs, which improves the performance of 3D Vision-Language tasks compared to methods that do not utilize semantic relationships between objects. When using ground-truth segmentation of the scene's point cloud, the performance gain from incorporating the scene graph reaches 5-8% (Table 4, Table 5 of the paper) for 3D Referred Object Grounding tasks. Note that in the revised version of the method, the performance boost due to the scene graph is 2-3% for LLAMA-8B-Instruct (Table 1 in the general response). Additionally, as shown in Table 1 in the general response, the use of the scene graph improves the performance of the Scene Captioning task (+3-4% CIDEr@0.5) while maintaining comparable performance for the Question Answering task. Finally, despite the use of additional tokens to describe the scene, adding the scene graph increases the inference time of the LLM in the 3D Referred Object Grounding task by only 7% (from 140 to 150 ms, see Figure 5 in the paper). For graph construction, we use the pre-trained VL-SAT model, which processes a scene with 9 objects in approximately 9 ms to construct a complete graph.

The paper claims to achieve SOTA results on ScanRefer and Multi3DRefer benchmarks in the introduction. These claims do not look correct -- as per Table-2, 3D Vista and M3DRef-CLIP outperforms the proposed model on ScanRefer. Besides several stronger baselines like BUTD-DETR [1] and PQ3D [2] outperform 3DGraphLLMs by large margins. Perhaps, the intention was to say that 3D GraphLLMs is SOTA LLM-based visual grounding model -- the claim in introduction may likely need to be revised appropriately. 3DGraphLLMs outperformM3DRef-CLIP baseline on Multi3DRefer benchmark. However, the baseline is trained on much lesser data compared to 3DGraphLLMs.

The revised version of 3DGraphLLM demonstrates superior performance compared to expert models, including stronger ones such as BUTD-DETR and PQ3D (see Table 2). Please, refer to Table 2 in the general response.

[1] Huang, H., Chen, Y., Wang, Z., Huang, R., Xu, R., Wang, T., ... & Zhao, Z. (2024). Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Thanks for the updated results!

I notice that in Table 1 of this response, some numbers are drastically different from those in Table 3 in the first version of the paper, across the board: ScanQA CIDEr metrics go up from ~67 to 83, ScanQA B-4 go from 7.8 to 12-15. Similar jumps also exist in all rows for Sqa3D and Scan2Cap.

The same thing happens between Table 2 in this response and Table 2 in the first version of the paper, where the baselines have similar numbers but 3DGraphLLM's numbers go up by 7-15 points.

Seems Table 2 and Table 3 in this response also have inconsistent results. Why 3DGraphLLM in Table 2 is better than 3DGraphLLM in Table 3 by >10 percent?

Can authors explain what causes this huge difference in the results?