Robust Cross-modal Alignment Learning for Cross-Scene Spatial Reasoning and Grounding

Thanks for your valuable comments and insightful suggestions. We have carefully looked into all the comments and suggestions. Attached is our point-by-point response.

Q1: The visual results are not clear. I cannot directly observe the advantages of the proposed method without the aid of the word descriptions, such as Figure 6.

R1: Thank you for pointing this out. Figure 6 presents a visual comparison of the predictions from our method and two representative baselines under the matching-then-grounding pipeline, with many details noted in the caption. In the next version, we will enhance the figure by adding key annotations to the visual pipeline and using more prominent colors to highlight the advantages of the proposed method compared to existing baselines.

Q2: Eq. (1) can be moved to the end of Section 4 when all the modules have been introduced, making the organisation clearer.

R2: Thank you for your constructive suggestions regarding the organization and readability of our paper. In the next version, we will revisit the structure and adjust the placement of Eq. (1) in accordance with your recommendation.

Q3: More details should be provided, such as the parameter settings of the loss function.

R3: We have conducted a comprehensive sensitivity analysis of all parameters, as presented in Section C.2 of our Supplementary Material. Additionally, the parameter settings used in our implementation have been made publicly available and are detailed in Section B.3 of our Supplementary Material.

Q4: Some texts in the figures are not too small, such as Figure 5.

R4: Thank you for pointing that out. We will increase the figure size and text font in the next version.

Q5: A limitation part can be included in the paper to make this research more comprehensive.

R5: We have conducted discussions of the limitations and potential impact of our proposed method and data, which is included in Section D of our Supplementary Material.

Q6: The author's method (i.e., CORE) focuses on the accurate alignment of text and objects. Are there any more innovative designs in the perception and components of 3D environments?

R6: Yes. Our CORE includes innovative designs tailored for 3D environmental perception within both the Cross-Scene 3D Object Reasoning framework and the two proposed modules, namely the Robust Text-Scene Aligning (RTSA) module and the Tailored Word-Object Associating (TWOA) module. Specifically, our framework introduces a targeted masking strategy during the feature encoding stage, which masks objects, their class semantics, or their positional embeddings in the 3D environment. This targeted enhancement helps improve the model’s perception of object characteristics, semantics, and spatial positions within scenes. Secondly, the RTSA module aggregates global text and scene features by computing attention at both the feature and token levels for fine-grained word and object features. This enables the model to adaptively learn the interrelations among features (e.g., object color, shape, etc.) and the relative importance of objects within a scene, thereby enhancing scene perception. Finally, the TWOA module employs a novel Screening Attention mechanism to filter out features of objects and words that are irrelevant to the target object, thereby facilitating focused scene perception for precise grounding.

Q7: Why does the proposed method not work well in the Sr3D dataset?

R7: In fact, this issue stems from limitations of the Sr3D dataset itself. All existing methods, not just ours, generally exhibit poor performance on the Cross-Scene Spatial Reasoning and Grounding (CSSRG) task when evaluated on Sr3D. Specifically, CSSRG requires identifying a target object based on a text query across hundreds of candidate scenes. The scene-level discrimination of the queries has a direct impact on the performance, followed by grounding within the matched scene. However, in Sr3D, object descriptions across different scenes tend to be similar and lack sufficient scene-level discrimination, leading to significant performance degradation. This is also the motivation behind our introduction of the CrossScene-RETR dataset, which is designed to address the aforementioned limitation for a more reasonable and comprehensive evaluation of the CSSRG task.

Q8: The authors should give more explanations about the training process of the proposed method.

R8: Due to page limits, this part has been included in the Supplementary Material. Specifically, additional explanations on the two proposed modules and training details are provided in Section B of our Supplementary Material. Furthermore, the detailed training process of the method can be found in Algorithm 1 on page 7 of our Supplementary Material.

Q9: How about the running speed and model parameters compared with other methods?

R9: In Table 2 and Table 3 of our Experiment Section, we provide a detailed comparison of the running speed between our proposed method and existing 3D Visual Grounding (3DVG) methods. The results show that our CORE achieves over a 100-fold increase in average speed compared to the 3DVG methods, demonstrating its high efficiency in the CSSRG task. Additionally, our proposed CORE is a lightweight CSSRG baseline with only 201.92 M parameters. Its parameter count is comparable to, or even smaller than, the recent 3DVG methods (e.g., VisTA, GPS, etc.), but slightly higher than that of earlier methods such a 3D-BUTD. Although these 3DVG methods have a comparable number of parameters, they struggle to achieve satisfactory efficiency on the CSSRG task, which further validates the advantage of our CORE’s matching-then-grounding pipeline for this challenging task.

Q10: For Table 1, how to get these statistics？ Are they the statistics provided in the documentation of those datasets or the authors' own rough estimates?

R10: The data in Table 1 are accurately computed based on our proposed CrossScene-RETR dataset. Additional statistics, visualizations, case studies, and analyses for our CrossScene-RETR and its subsets are provided in Section A, Figures 1 to 4 and 8 to 10, and Table 1 of our Supplementary Material. These materials help readers better understand and apply our proposed task and dataset.

Q11: For the targeted masking strategy, how to decide the mask location and probability? Whether the choice of mask have a significant impact on the experimental results?

R11: 1) As mentioned in the paper, our masking strategy focuses solely on visual attributes such as class semantics, color, shape, and spatial details. Specifically, for text modality, we mask the original text by replacing words in predefined attributive and spatial vocabularies. For point-cloud modality, we selectively mask the object, class-semantic, and positional embeddings. It is worth noting that each targeted location has a 10% probability of being masked. This value is relatively low and follows a similar setting to BERT. 2) No. To thoroughly analyze its importance and sensitivity, we conducted a comparison on the CSSRG task using different masking probabilities $p$ ranging from 0 to 0.3 on the CrossScene-RETR dataset, as shown in the table below. The results indicate that a small degree of targeted masking consistently improves the performance, while excessive masking leads to weak performance degradation. This demonstrates that, without excessively masking information, the choice of mask does not significantly impact the experimental results. This experiment and analysis will be included in the next version of our paper.

Thanks for your valuable comments and insightful suggestions. Attached is our point-by-point response.

Q1: It appears that CORE also needs to iterate through all scenes to match them before grounding objects. Why is its efficiency significantly higher compared to 3D Visual Grounding (3DVG) methods?

R1: Unlike 3DVG methods that exhaustively and sequentially traverse scenes, our proposed CORE efficiently 'traverses' all scenes in a retrieval-based manner to match the scene containing the target object. More specifically, in CORE, both the text queries and scenes are first aggregated into coarse-grained common representations. Then, supported by the proposed Robust Text-Scene Aligning (RTSA) module, the text queries can directly measure similarity with the 3D scenes in the common space. This process can be efficiently implemented via matrix multiplication on the GPUs. In contrast, conventional 3DVG methods, which only support single-scene inference, can only perform inference over one scene at a time. Each inference requires fine-grained reasoning between dozens of objects and words, which significantly undermines their efficiency in cross-scene scenarios.

Q2: Section 4 could benefit from a more detailed explanation, as certain aspects remain unclear. For instance, 1) the process by which RTSA obtains common representations is not fully elaborated. 2) How does the variation help mitigate the impact of false-negatives?

R2: Due to page limits, the details of Section 4 have been fully elaborated in Section B of our Supplementary Material. More specifically, 1) Details on how RTSA derives common representations are included in Section B.1 and Figure 5 of our Supplementary Material. Specifically, we design attention at the token-feature and feature-dimension level for fine-grained word and object features, enabling the model to learn adaptive interrelation-focused and feature-focused weights. Subsequently, the two different weights are applied to the fine-grained features, aggregating them into common representations. 2) The variation $\mathcal{L}\_{m}$ introduces an adaptive parameter $q$, which adaptively increases for more reliable false negatives. As proven in Section B.2 of our Supplementary Material, as $q$ approaches 1 from 0, the robustness of $\mathcal{L}\_{m}$ gradually improves and converges to its lower bound, i.e., the MAE loss, which could prevent overfitting to false-negatives.

Q3: The use of data subsets is confusing. It seems there are two different strategies for grouping the data into subsets, yet Figure 7 analyzes them together.

R3: Sorry for the confusion. We independently divide our CrossScene-RETR into different subsets based on two factors: the challenge level of object retrieval and the linguistic styles of the query texts. The former follows the splitting criteria of ScanRefer [A], and is detailed in the Scene Analysis Phase in Section 5.1. The latter is described in the Text Generation Phase in Section 5.1 and further elaborated in Section A.2 of our Supplementary Material. Each subset has its own characteristics and analytical significance. Therefore, we present them collectively in Figure 7 using radar charts to provide a comprehensive comparison of baseline performances. In future versions, we will integrate and clarify the description of the subset division.

References:

[A] Chen, Dave Zhenyu, Angel X. Chang, and Matthias Nießner. "Scanrefer: 3d object localization in rgb-d scans using natural language." European conference on computer vision. Cham: Springer International Publishing, 2020.

Thanks for your valuable comments and insightful suggestions. Attached is our point-by-point response.

Q1: The figures (Figure 4, 5, 6) are hard to read. The texts in these figures are too small.

R1: Thank you for the suggestion! We will increase the figure size and text font in the next version.

Q2: 1) Could the authors provide some clarification about the “Spatial Reasoning” part in Cross-Scene Spatial Reasoning and Grounding (CSSRG)? CSSRG seems to only include the grounding stage, and the concept of spatial reasoning appears somewhat overly broad. 2) Maybe more tasks should be included instead of relying on a single grounding task.

R2: 1) “Spatial Reasoning” runs through the entire CSSRG solution pipeline, not just the grounding stage. More specifically, during the scene matching stage, 3D scene features are reasoned over a collection of scenes based on text features to identify the scene whose spatial structure best matches the description. In the object grounding stage, reasoning is performed within the matched scene to ground the target object described in the text description. 2) This paper is an initial attempt to perform spatial reasoning and object grounding across a scene collection. In the future, we will continue to enrich CSSRG downstream tasks based on your valuable suggestions.

Q3: The introduction of the proposed CrossScene-RETR dataset is too brief. This dataset is an important contribution to the community and should be highlighted in the manuscript.

R3: Thank you for recognizing the importance of our proposed dataset! Since we introduced a new task, CSSRG, it is essential to provide a clear and comprehensive description of it. Due to page limits, we had to condense the dataset and method sections, with much of the content provided in our Supplementary Material. In the future version, we will incorporate more details and insightful statistics from the Supplementary Materials into the main body.

Q4: Failure case analysis is missing.

R4: Thanks. In the next version, we will include some failure case analysis to provide a more comprehensive evaluation of our CORE method.

Q5: 1) From Figure 2, 141 scenes are used in the CSSRG task, whereas only 1 scene is used in the 3DVG task. 2) Does the number of scenes affect the performance of CSSRG?

R5: 1) You may have misunderstood the implication of Figure 2. It illustrates that existing 3D visual grounding (3DVG) methods require 0.16 seconds per scene to perform inference on the 3DVG task. When directly applied to the CSSRG task, which contains 141 scenes, its inference time dramatically increases to 21.2 seconds per scene, accompanied by a significant performance degradation. Therefore, directly applying 3DVG methods to the CSSRG task is impractical, and adopting an efficient matching-then-grounding approach (i.e., our proposed CORE method) becomes necessary. 2) As shown in Figure 2 and discussed above, the number of scenes significantly affects the performance of conventional 3DVG methods on the CSSRG task. In contrast, CORE attempts to first efficiently align global scene and text representations and then perform object grounding, which is virtually unaffected by the number of scenes.

Thanks for your valuable comments and insightful suggestions. We have carefully looked into all the comments and suggestions. Attached is our point-by-point response.

Q1: The accuracy of Cross-Scene Spatial Reasoning and Grounding (CSSRG) stays low (≈23 % Acc@0.25), so practical use is distant.

R1: CSSRG is a challenging task introduced for the first time in our paper, which requires reasoning across hundreds of scenes to locate only one single target object based solely on a text query. On the one hand, compared to existing baselines, which achieve less than 5% CSSRG performance in terms of Acc@0.25 on the CrossScene-RETR datasets, our proposed CORE attains 22.99% Acc@0.25, achieving more than 4.6× performance improvement. On the other hand, CORE achieves only 6.24% Acc@0.25 on the existing ScanRefer dataset, whereas it reaches 22.99% Acc@0.25 on our proposed CrossScene-RETR dataset. These results demonstrate that the proposed method and dataset bring significant performance improvements to the CSSRG task, broadening its practical applicability.

Q2: Experiments rely on a single 3D backbone; generality to newer encoders is untested.

R2: We understand your expectation for a comprehensive evaluation. However, we would like to emphasize that our proposed CORE is a general framework for addressing the CSSRG task, into which existing 3D backbones can be seamlessly plugged and utilized. Without loss of generality, we follow the backbone setting adopted in recent 3D Visual Grounding (3DVG) works [A, B, C] to ensure consistency when comparing with 3DVG methods in our comparative experiments. In this setting, we have already adopted newer and task-specific 3D backbones, i.e., Mask3D [D] (2023) for object proposal extraction and Spatial Transformer [E] (2023) for feature encoding. Due to time constraints, we are unable to conduct evaluations on multiple 3D backbones during the rebuttal stage. In the future, we will focus on exploring new 3D backbones to deliver up-to-date applications and generality analysis.

Q3: The four 3DVG text datasets use heterogeneous text formats, yet the paper gives no preprocessing strategy for syncing them.

R3: Although the text comes from different datasets and has heterogeneous text formats (e.g., .json, .csv, etc.), the processing pipeline described in our paper simply involves using the corresponding text file parsing approaches to load texts from the raw data files and obtaining general embeddings through the pre-trained BERT, following mainstream 3DVG methods [A]. This does not require any specialized text preprocessing or style synchronization, and can directly provide general text embeddings for subsequent use by all methods. In the future, we will provide more details to eliminate any ambiguity, and we will open-source all relevant code after the peer review stage.

Q4: Scope is limited to indoor static scenes.

R4: Our CORE is a general cross-scene object grounding framework for the CSSRG task, which is not limited to indoor environments and can also be applied to outdoor scenes. Due to the limited availability of suitable outdoor 3D-text multimodal data, most existing 3D object grounding works and our proposed CSSRG are currently focused on these indoor data. Our work is an initial attempt, and in the future, we aim to target outdoor scenarios and build dynamic outdoor CSSRG datasets to support real-world applications such as autonomous driving.

References:

[A] Zhu, Ziyu, et al. "3d-vista: Pre-trained transformer for 3d vision and text alignment." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[B] Jia, Baoxiong, et al. "Sceneverse: Scaling 3d vision-language learning for grounded scene understanding." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[C] Huang, Wencan, Daizong Liu, and Wei Hu. "Advancing 3d object grounding beyond a single 3d scene." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

[D] Schult J, Engelmann F, Hermans A, et al. Mask3D: Mask Transformer for 3D Semantic Instance Segmentation[C]//2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023: 8216-8223.

[E] Chen, Shizhe, et al. "Language conditioned spatial relation reasoning for 3d object grounding." Advances in neural information processing systems 35 (2023): 20522-20535.

Dear Authors,

Thank you for taking the time to address my comments. You're response addressed most of my concerns. I have no further questions at this time.