Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention

To Reviewer rjTU

Q10. Time and memory complexity comparisons

Thanks for the suggestion. We show the FLOPs and inference time of each proposed module and a comparison with the baseline model M3DRef-CLIP in Tab. R5. All experiments are conducted on Multi3DRefer validation set on a single NVIDIA A100 GPU. The reported FLOPs and inference time are the averaged over the validation set.

We observe that the dynamic box proposal module and the dynamic multi-view renderer in the dynamic vision module contribute marginally to the computation. The additional computation in the language-informed spatial fusion module is also minimal. In other words, our model achieves better grounding performance without significantly increasing computations.

Table R5. Computational cost for proposed modules

Q11. Comparison of the additional metric F1@0.25 with M3DRef-CLIP

Please note, M3DRef-CLIP did not provide quantitative results over F1@0.25 in their tables. Here, we use their model to provide additional comparisons with M3DRef-CLIP over F1@0.25 in Tab. R6. We observe that our D-LISA achieves a better overall F1@0.25 score, especially for multiple targets and sub-categories where the distractors of the same semantic class exist. This aligns with our observation for F1@0.5 results in Tab 4.

Table R6. F1@0.25 (↑) result on Multi3DRefer validation set

Q12. What is the motivation for camera pose offset prediction?

In L30-32, we motivated that the viewpoints of the renderer may need to be different across different scenes and object sizes. To avoid poor viewpoint initializations, we start from the fixed viewpoints and predict an offset for each viewpoint. In Eq. 5, we use an MLP to learn the camera pose offset for each view based on the average box size, as the object size is a strong indicator when choosing the rendering camera pose.

Q13. Why is the proposed LISA better than the language-conditioned spatial self-attention in ViL3DRel?

ViL3DRef [C] incorporates hand-selected features to guide spatial relations. While these hand-selected features perform well when provided with perfect (ground-truth) object proposals, they are not robust to errors, a typical issue with hand-designed features. In particular, when these features are used with detected (noisy) object proposals, their performance decreases.

As is shown in Tab. A1 and Tab. A2, though ViL3DRel works well on the Nr3D benchmark which provides ground truth box proposals, the performance is much worse when validating on the ScanRefer benchmark where no ground truth proposals are provided. We compare our LISA and the language-conditioned spatial self-attention in ViL3DRel on the Multi3DRefer benchmark in Tab. A4. The results validate the effectiveness of our module.

• [C] S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev. "Language conditioned spatial relation reasoning for 3d object grounding". In NeurIPS, 2022

Q14. What would the performance be if camera pose offset were removed?

Please see the ablation study in rows 1 and 2 of Tab. 4 where using the camera pose offset (DMR) improves the model performance.

Q15. How is the predicted proposal probability supervised? 

The predicted probability is end-to-end supervised by weighting the detector features with the probabilities, as is shown in Eq. 4. To encourage fewer numbers of boxes, we also propose to regularize with the loss proposed in Eq. 3.

Q16. How does the threshold 0.5 in Eq.2 affect the performance?

Thanks for the suggestion. As we used a Sigmoid nonlinearity in Eq. 1, the maximum-likelihood estimate results in a threshold of 0.5. Here in Tab. R7, we further conduct experiments, following the setting in Sec. 4.3, but using different thresholds. We observe that using 0.5 results in the best performance.

Table R7. Additional ablation studies on the filtering threshold.

1. Complexity of the dynamic multi-view renderer.

2. Would L2 distance also be affected by noisy detected proposals?

3. What would the performance be if camera pose offset were removed? (Q14)

4. The results in Table R7 do not match the performance reported in Table 4.

5.  Novelty and Engineering-focused of the proposed modules

To Reviewer WVQA

Q8. The dynamic vision module only removes the low-probability boxes and uses NMS to filter overlapping boxes, which is not novel.

We believe the reviewer is referring to the dynamic box proposal module. For the dynamic box proposal module, we do not claim NMS to be the novel contribution. We clearly state "we employ a dynamic box proposal module with non-maximum suppression (NMS)" in L98-99.

Our contribution to this module is the learning of dynamic proposal probability, supervised by the new dynamic proposal loss introduced in L113-116. This enables the end-to-end training by weighting the detector features with the probabilities, as is shown in Eq. 4. Additionally, we also propose the dynamic multi-view renderer to have dynamic camera viewpoints based on different scenes.

We respectfully disagree with the reviewer that the proposed method is not novel. To the best of our knowledge, prior works in 3D grounding have not studied these aspects. We would be grateful if the reviewer could provide references to back up this novelty claim.

Q9. Novelty of the LISA module

Our baseline model M3DRef-CLIP follows 3DVG-Transformer, which directly uses spatial distances as additional attention weights without further reasoning. Prior works, like ViL3DRef [A] and CORE-3DVG [B] explored the spatial relations with hand-selected features. These hand-selected features work with ground truth object proposals but lead to worse performance when the object proposals are predicted, i.e. noisy. As is shown in Tab. A1 and Tab. A2, though ViL3DRel works well on the Nr3D benchmark which provides ground truth box proposals, the performance is much worse when validating on the ScanRefer benchmark where no ground truth proposals are provided.

To address these shortcomings, we propose to use language-guided spatial scores to balance the visual attention weights and spatial attention weights. We believe our LISA design greatly differs from the existing works.

• [A] S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev. "Language conditioned spatial relation reasoning for 3d object grounding". In NeurIPS, 2022
• [B] L. Yang, Z. Zhang, Z. Qi, Y. Xu, W. Liu, Y. Shan, B. Li, W. Yang, P. Li, Y. Wang, et al. "Exploiting contextual objects and relations for 3d visual grounding". In NeurIPS, 2023.

To Reviewer W21s

Q4. Comparisons with SOTA methods on Nr3D benchmark in Tab. A2.

As is mentioned in L415-416, the Nr3D benchmark assumes perfect object proposals, which is not the most realistic setting. Hence, we follow M3DRef-CLIP to consider the setting where object proposals need to be detected.

We note that M3DRef-CLIP has made a similar observation: models designed for non-perfect objective proposals generally perform less effectively in the perfect object proposal setting. In Tab. A2, we report the comparisons for completeness of the evaluation. However, we do not believe this is a fair comparison as the models are designed with different intentions.

Q5. Additional ablation study on the proposed modules.

Here we provide additional ablation by gradually adding proposed modules in Tab. R3. The experiments follow the setting in Tab. 4. We observe that each proposed module improves the model. The overall model with all proposed modules achieves the best result.

Table R3. Additional ablation studies on combined modules.

Q6. Why is the rendering with dynamic pose better?

Sorry for the confusion. It is difficult to judge what is "more informative" from a model perspective. The provided illustrations in Fig. 4 (a) and (b) meant to show the differences in the learned poses versus the fixed pose. Empirically, we observe that the dynamic camera pose leads to better grounding performance, see row 1 and row 2 in Tab. 4.

Q7. What is the effect of NMS on the proposed method?

As is mentioned in L214-216, adding an NMS module removes the duplicate predictions and leads to a higher recall, thus improving the F1 score. The state-of-the-art method M3DRef-CLIP did not include the NMS module in their pipeline. We show the effect of the NMS module by adding a baseline of M3DRef-CLIP with NMS in Tab. 1.

To provide a more comprehensive analysis of the NMS module, we show the additional comparison between our proposed D-LISA and D-LISA without NMS on Multi3DRefer. See Tab. R4 below. We observe that our designed D-LISA outperforms the baseline M3DRef-CLIP both with and without the NMS module. Using the NMS module would lead to a higher F1 score compared to not using it.

Table R4. The F1@0.5 (↑) w/ and w/o the NMS module.

To Reviewer su8L

Q1. Core issues targeted for multi-object 3D grounding

In L26-33, we summarized the targeted issues and the proposed solutions. Concretely, we identified that:

• object proposals are selected based on a fixed maximum number,
• the feature extractions from the proposals are based on a set of fixed camera poses for all scenes,
• the fusion module, of the existing method, lacks effective reasoning over spatial relations among objects.

We believe these aspects are crucial to building an effective multi-object 3D grounding system. We then propose a dynamic vision module and a language-informed spatial fusion module to address these issues. We will clarify the introduction.

Q2. Ablation studies on different types of scenes and queries

We evaluate the contribution of each proposed component in Tab. 4. The performance of individual modules is shown in rows 2, 3, and 4 respectively. The different categories of scenes and queries are introduced in L197-201, including

• zero target without distractors of the same semantic class (ZT w/o D);
• zero target with distractors (ZT w/D);
• single target without distractors (ST w/o D);
• Single target with distractors (ST w/D);
• multiple targets (MT).

These scene and query categories follow the prior work M3DRef-CLIP.

Additional ablations for different query types, including queries with spatial, color, texture, and shape information are reported in the table below. In Tab. R1, We report the F1@0.5 (↑) metric on the Multi3DRefer validation set. We observe that each proposed module effectively improves the performance for the queries that contain spatial, color, and shape information, and is competitive with the baseline for queries with texture information. The overall model achieves better grounding performance across all query types than the baseline.

Table R1. Additional ablation studies on question types

Q3. The computational cost for each proposed module

Thanks for the suggestion. We show the FLOPs and inference time of each proposed module and a comparison with the baseline model M3DRef-CLIP in Tab. R2. All experiments are conducted on Multi3DRefer validation set on a single NVIDIA A100 GPU. The reported FLOPs and inference time are averaged over the validation set.

We observe that the dynamic box proposal module and the dynamic multi-view renderer in the dynamic vision module contribute marginally to the computation. The additional computation in the language-informed spatial fusion module is also minimal. In other words, our model achieves better grounding performance without significantly increasing computations.

Table R2. Computational cost for proposed modules

Dear authors, thank you for your rebuttal and response.

I have looked over the paper and reviews and have a small suggestion. I found the paper title to be somewhat confusing as "Dynamic" is typically refers to static vs dynamic scenes. I would recommend you consider renaming the paper to

• "Multi-Object 3D Grounding with Dynamic Proposals and Language Informed Spatial Attention" or
• "Multi-Object 3D Grounding with Dynamic Modules and Language Informed Spatial Attention"