Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention

Response 4.1: Addressing Weakness 1 -- On the Necessity of Intent3D Dataset and IntentNet model dedicated to 3D Intention Grounding

Thank you for this insightful question. While a modularized approach may seem flexible, it is not well-suited for addressing the fine-grained, multimodal reasoning required in 3D intention grounding. The modularized pipeline, which separates detection and intention reasoning, introduces two critical issues:

1. Error Accumulation: The pipeline accumulates errors from both the single-modal 3D detector and the LLM's reasoning.
2. Hallucination from LLMs: Large language models suffer from the hallucination problem, necessitating rigorous comparisons to evaluate their actual performance.

As demonstrated in Line475, we actually have already conducted experiments with a modularized approach using GPT-4 in Appendix (A.6 page 17). In this setup, a 3D detector identifies all objects in the scene, and GPT-4 determines which objects satisfy the intention. As shown in the table below, the performance of this modularized method is inferior to our IntentNet, underscoring its limitations.

This actually resonates with the trend in visual grounding field:

• In 2D, approaches have evolved from two-stage methods like MAttNet[1] to one-stage methods like SegVG[2], which emphasize necessity of multimodal fusion instead of modularization.
• Similarly, in 3D, methods have progressed from ScanRefer[3] to EDA[4], showing that advanced approaches increasingly prioritize integrated multimodal reasoning over modularized pipelines.

A dedicated dataset for 3D intention grounding is essential, as it provides a focused benchmark to rigorously drive progress in this multimodal reasoning problem. And our IntentNet provides a strong baseline whose novel contributions in verb reasoning and verb-object reasoning can bring inspirations to our community on tackling this intention grounding problem.

[1] MAttNet: Modular Attention Network for Referring Expression Comprehension, CVPR 2018

[2] SegVG: Transferring Object Bounding Box to Segmentation for Visual Grounding, ECCV 2024

[3] ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language, ECCV 2020

[4] EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding, CVPR 2023

Response 4.2: Addressing Weakness 2 -- On the Determination of Six Intentions per Object and the Completeness and Effectiveness of dataset

In our implementation, we find that six sentences on average are enough to cover diverse intentions toward each object, while preventing repetition. Also, we reference the scale of ScanRefer and Nr3D, two widely used datasets in the field of visual grounding. Based on this scale, an average of 6 intentions per object can result in a dataset of nearly 45K.

As detailed in Section 3.3, our dataset exhibits a rich diversity in sentence structure and verb-noun combinations to show its completeness:

• Verbs: A total of 1,568 distinct verbs are included.
• Nouns: A total of 2,894 distinct nouns are used.
• Object category: A total of 209 fine-grained classes.
• Diversity per category: On average, each object category is associated with 58 unique verbs and 77 unique nouns.

This extensive coverage ensures the dataset covers a wide range of human intentions across different scenarios.

Additionally, the performance in Tables 1 and 2 demonstrates that our model achieves strong results on both validation and test sets, confirming the effectiveness of the generated intentions which are sufficient for training.

Response 4.3: Addressing Weakness 3 & 4 -- Explain Eq(3) and provide a detailed figure.

Thank you for your comment regarding Eq. (3). We appreciate the opportunity to clarify it step by step for the meaning of $t$. In Eq. (3), $T$ indicates all the text tokens; $T_l$ is the subset of $T$, which indicates the verb tokens among the text tokens. Therefore, the $t$ of “$t \in T$” indicates a single text token which could be a verb or not, and t of “$t \in T_l$” is a single text token which is a verb. To provide further clarity, we have included a more detailed figure in the Appendix (A.11, page 22) of the revised paper. We highlight the title of the section in red color for your reference. This figure aims to illustrate the connections in the network and the key modules of our loss functions.

Response 4.4: Addressing Weakness 5 -- On the assumption of scaling effect of Verb Alignment.

Thank you for raising this insightful question. We think that the effectiveness of the Verb Alignment comes from its dense supervision signal instead of limited data. And we assume it can further bring improvement if we could scale the data in the future.

Firstly, in the 3D domain, acquiring large-scale real-world point cloud data is a significant challenge, as it is not as easily accessible as 2D data from the internet. In this context, exploring breakthroughs to make the best of available labeled data becomes crucial, such as our verb alignment.

Secondly, even if 3D real-world data were to become more abundant in the future, we believe that verb alignment will continue to be effective. This is because verb alignment enhances the utilization of supervision signals from individual sentences.

For traditional detection losses, detecting the target does not necessarily require a comprehensive understanding of the entire intention sentence. For example, in the intention “I want to sit comfortably while listening to music,” the target is “chair.” The model may only need to focus on the verb “sit” to infer the target, but losses from verb alignment encourage a more holistic understanding of the sentence, including the understanding of all verb-object links, e.g. “listening to music”. As a result, the supervision signal becomes denser, improving the model's overall reasoning capabilities.

Thanks for the response. The authors addressed my issues. I plan to keep my original rating.

Response 3.1: Addressing Weakness 1 -- Additional Comparisons and revision of typo.

To provide more comparisons with current LLM-based methods, we compare our IntentNet with LL3DA. Furthermore, we also compare with GPT-4 + Detector Proposal, as another two-stage baseline. Specifically, we use a 3D object detector to proposal objects within the scene. Given the object proposals and the intention, we prompt the GPT-4 to infer the target object. Lastly, we compare with EDA + Detector Proposal, where the object proposals are appended to the intention sentence and input to the baseline, EDA. These results, summarized in the table below, consistently demonstrate the superiority of our IntentNet.

Additionally, we have corrected the typo regarding the detector names (“GroupFree”) in Tables 1 and 2 (page 8) in the revised paper. We highlight the revision in red color for your reference. Thank you for pointing this out!

Response 3.2: Addressing Weakness 2 -- Additive Ablation Experiments

Thank you for the suggestion to include additive ablation experiments. Below, we provide the results of such experiments:

Notably, the inclusion of verb alone already provides a significant boost in performance, highlighting its importance in intention grounding.

Response 3.3: Addressing Weakness 3 -- Performance on Traditional 3D Visual Grounding

Thank you for raising this point. This work primarily focuses on 3D intention grounding, which is an orthogonal task to visual grounding. As such, IntentNet is specifically designed for intention grounding and is not evaluated on traditional visual grounding tasks. More concretely, IntentNet focuses on modeling the reasoning between verbs and objects inherent to human intentions, rather than the nouns or adjectives typically emphasized in visual grounding. Moreover, we hope that the benchmark contributions and modeling insights presented in this paper can inspire future research on joint training approaches for both intention grounding and visual grounding tasks.

Response 3.4: Addressing Question 1 -- Open Source

Yes, we will open source our dataset and code. Thank you for your interest!

Response 3.5: Addressing Question 2 -- Clarifying One-Stage Methods.

Thank you for pointing this out. However, this seems to be a misunderstanding.

[Our method] IntentNet is a one-stage method, as bounding box predictions are generated in an end-to-end manner rather than relying on a pre-trained detector for inference. As stated in Line 309 of the paper, the pre-trained detector is only used to provide additional visual references, and the box matching with text is implemented as an auxiliary loss, which does not participate in the actual inference process.

[Other one-stage methods] Moreover, we have already compared IntentNet with existing one-stage methods, such as BUTD and EDA. These models also directly fuse 3D data and text to predict bounding boxes, using a pre-trained detector in a similar way to just provide visual references.

Response 3.6: Addressing Question 3 -- Training Hardware and Time Cost

Thank you for your question. The training of IntentNet was conducted on 4 NVIDIA A6000 GPUs. The training process takes approximately 24 hours.

Response 2.1: Addressing Weakness 1 and Question 1 -- Advantages Compared to LLM-Based Multimodal Methods

[Advantages] LLM-based multimodal methods, such as Chat3D-v2 or the GPT-4 two-stage approach (Line475), primarily operate at the object level. This dependency on off-the-shelf detectors for object proposals introduces detector’s error and also limits their fine-grained multimodal fusion. Lack of multimodal fusion leads to their poor performance. Such kind of shortage have also been witnessed in 2D domian. For example, previous works including POPE[1] reveal that multimodal LLMs often hallucinate in the basic object existence problem due the lack of fusion. This challenge becomes more pronounced in fine-grained multimodal tasks like our 3D intention grounding, where comprehensive feature fusion and reasoning are critical. In contrast, IntentNet addresses this gap through its explicit intention modeling and fine-grained feature integration. Specifically, IntentNet employs point-word-level fusion and enables the model to reason over verb-object relationships explicitly.

[Experimental Evidence] Due to resource constraints during the rebuttal period, we choose to conduct a comparison with LL3DA. The results, summarized in the table below, show that IntentNet significantly outperforms LL3DA. Although LL3DA uses point-level features, its Interactor3D module relies on an off-the-shelf detector during inference for bounding box proposals. Errors from this detector propagate to the final predictions, limiting its effectiveness.

[Clarification on ReGround3D] Thank you for highlighting ReGround3D. As it is published within four months of our ICLR submission, it qualifies as a contemporaneous work under the ICLR 2025 policy, and we are not required to compare with it in our submission. Nevertheless, we will include its citation in the camera-ready version.

[1] Evaluating Object Hallucination in Large Vision-Language Models, EMNLP 2023

Response 2.2: Addressing Question 2 -- Advantages Compared to Large Language Models

While LLMs exhibit strong reasoning capabilities for pure language tasks, 3D intention grounding is a multimodal, fine-grained problem that requires joint reasoning across both vision and language. Relying solely on single-modality, language, for reasoning is insufficient. For example, consider the intention: "I want to sit down and rest." In a "living room", the target object might be a "sofa", while in a "bedroom", it could be a "chair" or "bed". Without fusing multimodal information, such as scene context, language-only LLMs cannot accurately ground intentions in 3D space. In contrast, our IntentNet explicitly integrates fine-grained point-level features with text features, enabling verb-object pair reasoning to analyze intentions. This joint feature fusion and reasoning framework allows IntentNet to outperform LLMs in this multimodal fine-grained scenarios.

Dear Reviewer,

Today is the final day of the ICLR rebuttal period. We noticed that your confidence score is relatively low, and we sincerely hope you might consider aligning with Reviewer qHNc, who has higher confidence and set their score as 6.

Please adjust your rate since the authors have completely responded your concerns and you have no more follow-up questions.

For your convenience, we have summarized our contributions and rebuttal in the comment: Summary of Our Contributions and Rebuttal Responses.

Best regards,

The Authors

Response 1.1: Addressing Major Concern -- Comparison with LLM + Detector

Thank you for raising this thoughtful concern regarding the potential baseline involving LLM + Detector and the fairness of our comparisons.

[Baseline] To address this, we want to highlight that in Line475, we mention that we have already conducted the LLM + Detector baseline, as detailed in Appendix (A.6 page 17). Specifically, we use a detector to propose objects in the scene. Given the proposed objects and the intention, we prompt one of the most powerful LLMs, GPT-4, to infer which types of objects the question targets. As shown in Table 5 (page 17) or the table below, this two-stage method performs worse than our approach due to detector errors and hallucinations from GPT-4.

[Fairness] Furthermore, we would like to clarify that 3D-VisTA and Chat3D-v2 in our comparisons are general-purpose models designed to handle general language, not specifically for noun-based language. As such, we believe that our comparisons with these models are fair.

Response 1.2: Addressing Major Concern -- Comparison with baseline + provided objects

To address the concern of whether the baseline might be improved given the objects expected to detect in a scene, we conducted an additional experiment, indicated as “EDA + Proposal”. Here, nouns/objects proposed by a detector are appended to the intention sentence as the input for EDA. The results are as below:

As shown in EDA + Proposal, directly providing nouns/objects expected to detect to EDA does not improve its performance, but rather introduces noise due to detector errors, which interferes model's reasoning. Actually, existing baselines (e.g., EDA, BUTD-DETR) have already use detectors to obtain "objects expected to detect" indirectly.

Response 1.3: Addressing Minor Concern -- Cases where IntentNet fails but others fed with objects succeed

Thank you for this suggestion. We actually have already provided some failed cases of our IntentNet in the Appendix (Section A.2, Fig. 6, page 16). In our revised paper, we further include visualizations of failed cases in Figure 8 (page 18). We highlight the title of the section in red color for your reference. Specifically, we observed that our model sometimes struggles with intentions involving multiple actions. For example, consider the intention: "I need to cool down on hot days while sitting in the living room." In this case, the primary intention is "cool down", which targets the object "fan". However, the secondary action "sitting" acts as a distractor, leading our model to incorrectly detect "couch" (blue bounding box). When we directly provide ground truth object noun to EDA, the task becomes simplified, allowing it to correctly detect the "fan" (green bounding box). We have included additional visualized examples in Figure 9 (page 18) for further reference.

Dear Reviewer,

ICLR has extended the rebuttal period to encourage your response. We sincerely appreciate the time and effort you have already invested, and we have actively addressed all your concerns in detail.

Since you have expressed the willingness to raise the score and we have already completely replied, is there any remaining problem you want to ask? Notably, Reviewer kRaB acknowledged that most of their issues have been addressed and decide to raise the score to an acceptance rate. Similarly, Reviewer qHNc has expressed that their concerns have been resolved and decide to maintain the acceptance rate.

Your response is critical for determining the outcome of our paper, and we value your input greatly!

Best,

The Authors

Dear Reviewer,

Today is the last day of ICLR rebuttal. Given that you previously indicated a willingness to raise your score, please adjust your rate since the authors have completely responded your concerns and you have no more follow-up questions.

Best regards,

The Authors