DenseGrounding: Improving Dense Language-Vision Semantics for Ego-centric 3D Visual Grounding

We would like to thank the reviewer for your insightful comments and valuable suggestions, which will help us improve our paper.

Q1 and W1 - Implementation Details for Prompt

Table 1: Statistics of the Augmented Prompt

Thank you for your valuable suggestion. We acknowledge that adding more details in Section 4.2 would improve the clarity of the paper. To address this, we have updated the manuscript to include the exact prompts used, examples of the enhanced descriptions generated by the LLM (see Appendix A.5.1), and statistics such as the total number of augmented annotations and their average length in Table 1.

Regarding the selection of the k most related descriptions in the LSE for augmenting an input text description, we select context descriptions from the SIDB that mention the same objects as the input description. If fewer than k such relevant descriptions are available, we randomly select additional descriptions from the same scene to introduce diversity. Further details can be found in Appendix A.5.2.

W2 – Novelty of the HSSE

Table. 2. Ablation Study on the proposed modules

Thank you for the comment. While attention mechanisms are widely utilized, our HSSE is uniquely tailored for the egocentric visual grounding task. Due to the sparse nature of egocentric views, each individual input offers only a partial glimpse and cannot fully capture the entire scene. The HSSE module is designed to effectively enhance interactions of multiview visual features across individual views, leading to a more comprehensive scene representation. It also facilitates improved cross-modal interactions between textual data and the refined, fine-grained visual features representing the scene. By specifically applying these methods to the challenges presented by sparse egocentric views, we address particular issues inherent in this domain. Moreover, as indicated in Table 2, integrating the HSSE module yields an overall performance improvement of 2.09% when used on top of LSE, and also achieving 2.09% in hard and 3.25% in overall performance when compared with the baseline method.

W3 – Missing Citations in Table 1

Thank you for bringing this to our attention. We have revised table in our paper to include proper citations for all referenced methods, ensuring appropriate acknowledgment of prior work.

Moreover, we will be releasing the improved annotations and code to benefit future research.

We sincerely appreciate your constructive feedback and will incorporate these improvements into our revised manuscript.

Thank you for acknowledging the clear motivations and effectiveness of our HSSE and LSE modules in addressing key challenges and achieving competitive results on the EmbodiedScan benchmark. Regarding your concerns, we have provided our detailed responses below.

W1 and Q1 Generalization and Robustness: "Could you evaluate the model on additional benchmarks to assess its generalization, and report on its zero-shot performance?"

Table 1: Cross-dataset zero-shot performance of Ego-centric 3D Visual Grounding Methods

Thank you for the comment. We would like to clarify that the training and validation scenes in EmbodiedScan are collected from different scenes in three distinct 3D datasets (3RScan, Matterport3D, Scannet), and ensure that the scenes in the training set are different from those in the validation set.

To further assess the generalization and robustness of egocentric 3D visual grounding methods, we have constructed a zero-shot setting within the EmbodiedScan dataset by reorganizing its samples. Specifically, we used the scenes from 3RScan and Matterport3D for training and then evaluated the model on the ScanNet scenes, which were not seen during training. The results, as shown in Table 1, indicate that our method significantly outperforms the baseline method, demonstrating the robustness of our approach in new and diverse scenes.

W2 and Q2 Scaling Analysis:

Table 2: Limited Data Scenario Comparison on Overall@0.25

Thank you for highlighting the importance of understanding how performance scales with varying data sizes, especially in data-scarce environments. In response, we have conducted a scaling study by training our model on varying amounts of data by subdividing the mini-training set into smaller subsets. As illustrated in Table 2, our analysis shows that DenseGrounding efficiently learns from limited data and consistently outperforms the baseline across different data sizes. Notably, when trained on just 40% of the training data, our method surpasses the baseline model that was trained on 80% of the data. This highlights the data efficiency and robustness of our approach, indicating its effectiveness even when training data is scarce. We have included detailed results and discussions of this scaling study in the revised manuscript's appendix to provide clearer insights into the model's scalability.

W3 Dependency on Complete Scene Graphs

We would like to clarify that our method does not rely on an explicit scene graph from the dataset. Instead, LSE is constructed using text descriptions and 3D locations, similar to the data used by other egocentric visual grounding baselines during training. We provide these text descriptions and bounding boxes to an LLM, allowing it to learn object relations from the provided information to augment the training data without relying on predefined scene graphs. We will clarify this aspect in the revised paper to address any potential confusion.

We sincerely thank you for your valuable feedback. We are committed to addressing these points to strengthen our work and provide a more comprehensive evaluation of our model, and we welcome any further discussions or clarifications.

Q1 and W1 - Dependence on Annotated Dataset

We would like to clarify that the SIDB is constructed exclusively from the training data, which is also used by baseline methods (revised in L320-L321). This ensures a fair comparison, as all methods have access to the same level of information. Moreover, we have conducted an ablation study to examine the impact of different context information in SIDB's construction on model performance. Specifically, as presented in Table 2 of our paper, we compare our proposed solutions LLM+DB(R) (using text descriptions only) and LLM+DB(R+L) (using text descriptions with 3D locations) against the baseline methods. Although our model's performance may degrade without location information, both versions of our method still significantly outperform all baselines.

Q2 and W3 - Use of Real or Rendered Images

Thank you for highlighting the need for clarity on whether we use real or rendered images in our experiments. We have updated our manuscript to include this detail. In our experiments, we follow the main experimental setting of EmbodiedScan for ego-centric visual grounding and use real images provided by the dataset.

For the ablation experiments, the EmbodiedScan dataset does not supply rendered images nor provide details on how to obtain them. Therefore, we are unable to replicate the ablation study comparing real versus rendered images as presented in the EmbodiedScan paper. Additionally, the ablation study in EmbodiedScan concerning rendered images focuses on 3D Object Detection, which differs from our task of Egocentric Visual Grounding.

Q3 - Augmentation Errors in LSE

Thank you for your question. Indeed, augmentation errors are possible because current LLM models inherently have hallucination issues, making some degree of noise inevitable. However, based on our experiments, the results we have obtained so far are quite satisfactory, showing observable improvements of 3.48%. Furthermore, with potential advancements in LLM technology and hallucination problems being addressed, performance may continue to improve.

Q4 - Object Category-wise Result

Thank you for your interest in a detailed, object category-wise analysis of our results. Given that the egocentric multiview grounding task of the EmbodiedScan dataset contains 163 object categories in the validation set, including a detailed table for each class, it is impractical due to space limitations in the main paper. However, we have provided a comprehensive graph illustrating the performance of our method across these classes in Appendix A.2 of our paper.

W4 - Clarification on Visual Inputs of Previous Models

We appreciate your feedback regarding the need to clarify the inputs used by previous models in Table 1. We have updated the caption and the main text to specify that, consistent with the experimental settings of EmbodiedScan, we and the baseline methods use RGB-D inputs only. This ensures that comparisons are fair and that all methods are evaluated under the same conditions.

W5 - Clarification on Camera Pose and Experimental Settings

The EmbodiedScan dataset provides the camera's intrinsic and extrinsic parameters, which include camera poses and calibration data. Similar to the baseline methods in EmbodiedScan, we utilize this readily available information in our method. We have updated Section 3.1 of our manuscript to explicitly mention the use of these camera parameters and to explain the experimental settings in greater detail.

W6 - Applicability to Other Datasets and Tasks

Table 1. Performance of LSE on Scanrefer

We appreciate your interest in the potential expansion of our method to other datasets and tasks. The LSE is designed to be a versatile component that can be applied to similar tasks beyond the EmbodiedScan dataset. Its ability to augment training data through language models makes it suitable for other related tasks where textual and visual information interplay is crucial. As demonstrated in Table 1, our proposed LSE module outperforms the baseline method EDA, demonstrating the possible extension of our method to more tasks.

W7 - Inclusion of Additional Related Works

Thank you for your recommendation to include a broader range of related studies. We agree that providing a comprehensive overview of related work is essential for contextualizing our contributions. In response, we have updated the Related Work section of our paper to include additional relevant studies, encompassing recent advancements in 3D visual grounding, visual question answering, and related domains.

Reference:

[1] Wu et al. EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding. CVPR 2023.

We sincerely appreciate your recognition of the novelty and effectiveness of our proposed method for multi-view ego-centric visual grounding.

W1: "proposed method lacks descriptions for certain model details...

Thank you for your comment. Regarding your concerns, we have added more details about these adopted modules in our paper in Sec. 3.2.

W2: "The paper does not provide a sufficiently clear explanation of how the LLM leverages the SIDB to enrich the semantics of descriptions..."

Thank you for your insightful comment. To enable the LLM to effectively enhance a single input data, it is essential for the model to have richer contextual information about the scene. To achieve this, we constructed the Scene Information Database (SIDB), which compiles text-based information about each scene using input descriptions available in the training set. By leveraging the SIDB, the LLM gains access to detailed contextual information, allowing it to provide more informed augmentations. To better illustrate our method, we have included the prompt in Figure 6, offering a clearer and more detailed explanation. We hope this addition satisfactorily addresses your concern.

Q1: "...whether it could also be applied to common 3D visual grounding tasks..."

Table 1. Performance of LSE on Scanrefer

Thank you for this suggestion. Regarding the two modules (HSSE and LSE) proposed by DenseGrounding, the HSSE module is specifically designed to address problems in ego-centric scenarios and does not apply to common 3D visual grounding tasks. However, the LSE module is applicable to common 3D visual grounding tasks. We used the LSE pipeline to augment the Scanrefer training set and presented the performance in Table 1. It can be observed that our method outperforms the previous SOTA in common 3D visual grounding.

Thank you once again for your valuable feedback, which will help us further improve the clarity and quality of our work.

References:

[1] Wu et al. EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding. CVPR 2023.

Thanks for the detailed responses. My concerns have been addressed, and the implementation details of the paper have also been clarified. As a result, I will maintain my previous score of 6.

We would like to express our gratitude for the detailed questions that could improve the quality of our paper.

W1 "Potential unfair comparisons and prior information leakage..."

1. Thank you for your comment. First of all, we understand your concern about potential data leakage and potential unfair comparison:

• • Preventing Potential Data Leakage: In our work, we ensure that the LLM does not have access to or utilize any information from the test set. We would like to highlight that the database is constructed based solely on the training set, as you stated. During training, the LSE module uses LLM to augment the training only based on the database (SIDB) constructed from training data which is also used in training other baseline methods. During inference, the LSE module is not used to augment any test input data. This practice aligns with standard protocols to prevent any unfair advantages or leakage of test set priors. Additionally, we would like to emphasize that in EmbodiedScan the training and testing scenes are collected from entirely separate environments. Therefore, the scenes used during the evaluation were neither seen by the model during training nor incorporated into the LSE module.
• • Ensuring Fair Comparison: Furthermore, to ensure a fair comparison, we provided a baseline that also utilizes additional contextual objects from the training data. It utilizes the additional context by concatenating multiple text samples describing the same target to form a new training set (denoted as "Concat Sample" in Table 2 of the paper). Under this experimental setting, our LSE module still outperforms this strong baseline.

2. Concerns on SIDB and Cost: Thank you for raising the point about the SIDB and the associated costs of using an LLM. We would like to clarify that the database (SIDB) is constructed offline solely to improve the training data as data augmentation. During inference, the LSE including SIDB is not used, so there is no additional cost or impact on flexibility at that stage. Regarding the cost of using the LLM, generating 1,000 samples incurs an approximate cost of $0.16, which we consider modest relative to the 3.48% improvement in overall accuracy achieved. This approach also reduces the need for extensive manual annotation, saving both time and labor. We will emphasize these points in the revised paper to address your concerns fully.

W2 and W3 Performance and Novelty of HSSE

Table. 1. Ablation Study on the proposed modules

We would like to highlight that the HSSE module is specially designed for the ego-centric visual grounding task. Given the sparse egocentric views, the input data of each individual view has a partial view and cannot fully represent the entire scene. The HSSE module is designed to effectively enhance multiview visual feature interactions between each view to obtain a better representation of the scene, as well as further facilitate cross-modal interactions between text and obtained fine-grained visual features.

Our approach leverages these methods to address the unique challenges presented by sparse egocentric views. Furthermore, as demonstrated in Table 1 of this comment, the inclusion of the HSSE module yields a 2.09% improvement in performance on hard cases compared to the baseline method. When the HSSE module is applied in conjunction with the LSE module, the overall performance improves by 2.09%.

W4 Details about SIDB is not sufficient

Thank you for pointing out the need for a more detailed description of the SIDB construction and the selection of relevant scene information. Specifically, the construction of SIDB involves collecting all the text descriptions (the text input of training samples, such as "Find the bin in front of the table") from each of the scenes in the training data, as well as the locations of the objects described in these descriptions. When augmenting a raw text description during training, we find 50 related text descriptions in the scene from the SIBD. We select these descriptions by finding the ones that describe the same instances as the raw text description. Then, we provide this information to the LLM and request an augmented output. We have included a more detailed explanation in Appendix A.5.

We sincerely thank you for your valuable feedback and welcome any further clarifications or questions they may have.

Thank you for your continued feedback and for highlighting concerns regarding the performance of our HSSE module on the hard split. We appreciate the opportunity to provide further clarification.

The hard split specifically focuses on samples containing multiple instances (more than three) of the same object class, which requires strong textual understanding to disambiguate and accurately localize the target object. In our framework, the LSE module is designed to enhance textual context, making it particularly effective for these cases and explaining its strong performance on the hard split.

In contrast, the HSSE module is aimed at improving fine-grained visual semantics from ego-centric views. Upon evaluating the metrics, we observed a performance difference of only 0.11% in the hard split, equating to 5 incorrectly classified samples out of all validating samples (more than 50k). Thus, this minor difference is considered comparable, especially when viewed alongside the overall improvement of 2.09% in Table 1. While HSSE may not directly improve performance on the hard split as significantly as LSE, its contribution to overall performance remains substantial.

Table 2. Ablation on Proposed Modules under IoU ≥ 0.5

We would like to sincerely thank you for your valuable feedback, which has prompted us to conduct a deeper exploration of this issue. In our further analysis, we hypothesize that the IoU threshold of 0.25 introduced by the EmbodiedScan benchmark may not necessarily require highly fine-grained predictions, which could explain why the performance of the HSSE module is not fully apparent under this metric. To better assess the impact of HSSE, we evaluate the model under a stricter IoU ≥ 0.5 threshold, where predictions and ground truth bounding boxes must overlap by more than 50%. This stricter threshold requires a more accurate prediction of bounding boxes, making it a more rigorous test of the model's ability to capture fine-grained visual details. Under this metric, HSSE demonstrates a significant improvement. As shown in Table 2, incorporating HSSE alongside LSE brings significant gains, with improvements of 3.47% on hard samples and 3.35% in overall performance under this stricter metric.

These results underscore the strengths of the HSSE module in providing fine-grained representations of ego-centric views, highlighting its complementary effect when combined with the LSE module. By enhancing the model's ability to capture fine-grained visual cues, HSSE significantly improves the precision of the predicted bounding boxes. This aligns with our motivation for designing the HSSE module, which aims to refine the model's visual understanding.

We would like to thank you once again for your insightful comments, which have helped us improve the clarity of our explanation and the overall presentation of our work.