RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation

Thank you for your positive feedback and recognition of our contributions. We greatly appreciate your commendation of our clear articulation of motivation and quantitative analysis. We're pleased you acknowledged our method's effectiveness in spatial relation reasoning for 3D scenes, expressing interest in our rule-guided weak supervision strategy. Now, let's address the specific concerns you've raised and offer further clarification:

---

Q1: The paper does a good job comparing 3D-RES methods. But traditional 3D Visual Grounding methods using bounding boxes are more mature. I'd like to see how these older methods perform on this task. This would give a more complete quantitative comparison with the proposed method.

A1: Thank you for your insightful suggestion. Based on your advice, we adapted the high-performing methods 3DVG-Transformer and 3D-SPS from 3D-REC for 3D-RES and tested their performance, as shown in Table A. Our method still demonstrates a significant advantage, outperforming by over 10 points.

Table A: Comparison with 3D Visual Grouding methods. * we reproduce results by extracting points within the boxes as segmentation mask predictions using their official codes.

[a] 3DVG-Transformer: Relation modeling for visual grounding on point clouds. ICCV 2021

[b] 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. CVPR 2022

---

Q2: The appendix talks about LLMs for target word localization. But it doesn't compare them directly with 3D Visual Grounding methods based on LLMs. Could LLM-based approaches be better? I'd like to see a comparison between specialized lightweight models and general LLMs.

A2: Thank you for your valuable suggestion. We compared our model with LLM-based 3D RES models SegPoint [c] and Reason3D [d], as shown in Table B, and our model still demonstrates a significant advantage, leading by more than 2.6 points.

Table B: Comparison with LLM-based methods.

[c] SegPoint: Segment Any Point Cloud via Large Language Model. ECCV 2024

[d] Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model. Arxiv 2024

---

Q3: The paper should provide details on the superpoint feature extraction mentioned in line 119. I'm curious if superpoint features cover all targets. If not, what's the missing rate?

A3: We appreciate your insightful feedback. The superpoint generation mechanism in RG-SAN ensures that no instances are missed, as the superpoints comprehensively cover the entire scene. Superpoints are essentially fine-grained fragments that group semantically similar points together. They do not overlap with each other and collectively constitute the whole scene, guaranteeing that all objects are included within superpoints. This setup ensures that every object is included within the superpoints, as detailed in the upper left corner of Figure 2 in our paper. Therefore, the issue of missing objects does not arise in our framework.

---

Q4: The layout of Tables 2 and 3 is off, and the font in Figure 2 is too small. This makes it hard to read the details.

A4: Thank you for your suggestions. To improve readability and facilitate understanding, we will adjust the layout of Tables 2 and 3 and enhance Figure 2, including resizing the font, in the new version.

---

Q5: The text processing procedure isn't detailed enough. For example, the interaction process of the DDI module. I'd recommend including this description in the main text.

A5: Thank you for your suggestion. We will include the details of DDI interactions in the new version to enhance understanding for readers.

---

Q6: Will the code for RG-SAN be open-sourced in the future?

A6: Thanks for your interest. In the paper, we have already provided the code via an anonymous link and also uploaded a copy in the supplementary materials. We also commit to releasing the code once the paper is accepted.

Thank you for your positive feedback and recognition of our work. We appreciate your acknowledgment of our exploration of text-conditioned 3D spatial perception and the effectiveness of the TLM and RWS modules. We're also glad you found our video demo and Figure 3 visualizations clear and insightful. Now, let's address your specific concerns and provide further clarification:

---

Q1: More details can be added regarding the superpoint feature extraction and text feature processing sections.

A1: We appreciate your valuable suggestions. We will provide a detailed description of superpoint feature extraction and text feature processing in the new version.

---

Q2: The paper mentions that the Sparse 3D U-Net used as the visual backbone is pre-trained. On which datasets was it pre-trained? Would using different pre-trained backbones result in performance variations?

A2: Thank you for your insightful question. The 3D U-Net we used has been pre-trained on 3D instance segmentation tasks [45]. Additionally, following your suggestion, we explored alternative backbones, including PointNet++ [39], used by the classic work 3D-VisTA [a], and another superpoint-based backbone, SSTNet [28], as detailed in Table A. Our findings indicate that the performance with PointNet++ [45] and our employed SPFormer [45] are comparable, demonstrating the adaptability and effectiveness of our proposed modules across different backbone architectures. We will include this discussion in the final version.

Table A: Ablation study of the Visual Backbones.

[a] 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment. ICCV 2023.

---

Q3: Figure 2 has too many colors, making it somewhat cluttered and potentially confusing for readers. It is recommended to simplify and optimize the color scheme.

A3: Thank you for your detailed feedback. We will optimize Figure 2 in the next version to enhance its clarity and readability.

---

Q4: It is suggested to include some bad cases to enhance the completeness of the work.

A4: Thank you for your valuable suggestions. We will include bad cases and corresponding analyses in the new version.

---

Q5: Will the complete code for the paper be open-sourced for additional exploration?

A5: Thank you for your interest. In the paper, we have already provided the code via an anonymous link and also uploaded a copy in the supplementary materials. We also commit to releasing the complete code once the paper is accepted.

I apologize for the delayed response. I've been quite busy lately, but I wanted to take a moment to wrap things up. First, I’d like to thank the authors for their detailed response. It’s impressive to see that the proposed method performs effectively across different backbones. After carefully reviewing all the discussions, I find this paper to be very valuable. The exploration of text-conditioned 3D spatial perception from both 3D spatial relationships and natural language structure perspectives provides constructive guidance for 3D cross-modal understanding, which is indeed a challenging aspect of human-computer interaction. The authors have elegantly addressed this issue without introducing additional data or annotations, which is truly inspiring. The contributions of this paper have been widely recognized by everyone involved.

I also noted Reviewer 6dEW's comments regarding some minor issues with the formula descriptions. In my view, these do not affect the overall readability of the paper and can be easily addressed with minor revisions. Therefore, I believe this paper deserves a strong score, and I am willing to champion it.

Thank you for your positive feedback and acknowledgment of our paper's strengths. We're pleased you appreciate our approach of attributing 3D spatial properties to text for 3D multimodal spatial perception modeling, recognize the promising performance of our model on ScanRefer, and underscore the effectiveness of our proposed methods. Now, we'll address the specific concerns you've raised to provide further clarification:

---

Q1: RG-SAN adopts superpoints as fundamental visual units for feature extraction and segmentation. While previous works have employed similar approaches, I am interested in understanding the segmentation quality of superpoints themselves. For instance, how many superpoints exist solely within individual objects? This is crucial because if a superpoint spans across two objects, it inevitably affects the segmentation results.

A1: Thank you for your insightful feedback. In practice, due to the fine granularity of superpoints and their tendency to aggregate semantically similar points, most superpoints cover only a single object. To verify this, we conducted a statistical analysis of the superpoints in the ScanRefer [5] dataset. If a superpoint contains points from more than one object, it is classified as containing multiple objects; otherwise, it is categorized as containing a single object. Our analysis reveals that 99.55% of the points are within superpoints that cover a single object, with a missing probability of less than 0.5%. This indicates that the issue of multiple objects within a single superpoint has a negligible impact on the final results and does not warrant special attention.

---

Q2: RG-SAN trains spatial awareness networks using the centroid coordinates of target objects. Here, does "object center" refer to the geometric centroid or the center of mass (where the former denotes the center of the bounding box and the latter denotes the mean coordinate of all points belonging to the object)? Given the inherent sparsity of point clouds, these two centers may exhibit significant differences.

A2: Thank you for your valuable comments. We use the centroid of all superpoints belonging to an object, representing the average coordinates of these superpoints. We conducted comparative experiments between this centroid setting and another centroid setting. As shown in Table A, the results indicate no significant differences between the two approaches.

We will include this discussion in the revised version to enhance the clarity of the paper.

Table A: Comparison of the Center of Box and Center of Mass.

---

Q3: The statistical analysis of the importance of spatial information for 3D-RES should be included in the main text. This will help readers understand the motivation of the paper from both qualitative and quantitative perspectives.

A3: Thank you for your suggestion. We will include the statistical analysis on the importance of spatial information for 3D-RES in the paper to enhance the rigor of the paper.

---

Q4: Although this paper discusses its limitations, it does not provide failure cases or corresponding analyses. Specifically, for the segmentation of plural nouns, it remains unclear whether only one object is recognized or if segmentation fails altogether. It would be beneficial to include either qualitative or quantitative analysis in this regard. Including this part would make the paper more comprehensive and facilitate follow-up research and improvements.

A4: Thanks for your insightful suggestion. We will include visualizations of failure cases and provide a qualitative analysis in the new version.

---

Q5: Does "object center" in this paper refer to the geometric centroid (center of the bounding box) or the center of mass (mean coordinate of all points)?

A5: Thank you for your detailed question. We are referring to the centroid, where the center coordinate is defined as the mean of the coordinates of all superpoints belonging to the object.

Q1: Using spatial information is not new. Sec. 3.2.2 and 3.2.3 and Eq. (7) to (12) resemble those in [22]. Spatial information in 3D visual grounding has been explored, like [6].

A1: Thank you for your valuable feedback. Indeed we acknowledge that our use of positional encoding is based on previous methods[51][22]. However, our primary contribution lies in modeling the spatial positions and relationships of noun entities within sentences for the 3D-RES task, which has also been recognized by reviewers p7dx, 8gkL, and MnE2. This approach has two core differences compared to [22]:

(1) Unlike [22], which uses zero initialization for queries and random initialization for positional information, our queries and positions are text-driven from the start. This improves mIoU by 4.4 points, as shown in Tab. 2 and the newly added Tab. A.

(2) Unlike [22], which supervises all target instances' positions, in 3D-RES, only the core target word is supervised. Our novel RWS method constructs spatial relationships for all noun instances using only the target word's positional information, improving mIoU by 2.3 points, as shown in Tab. 4.

Table A: Comparison of MAFT [22] with our RG-SAN in ScanRefer Dataset.

In summary, our core innovation lies in constructing spatial positional information, rather than just using positional encoding, as done in [6]. Using positional encoding is a regular operation after generating spatial information. We also explored other positional encodings, such as 5D Euclidean RPE, achieving similar results in Tab. 2.

Following your suggestion, we will further compare and analyze our work with the highly relevant and interesting study [22] in the new version to clarify our contributions.

---

Q2: The writing needs improvement: K_i (line 120) and c_i (line 123) are not defined, P^t in Eq. (3) with two subscripts is inconsistent with P^t in Eq. (5), and Table t (line 166) and q (Eq. 7) are not introduced.

A2: Thank you for your detailed feedback. We adopted the representation from [28] to formulate our approach as concisely as possible. Following your suggestion, we will make improvements in new version.

---

Q3: The TLM module resembles [22]'s architecture. Have you compared their performance on ScanRefer in Tab. 1?

A3: Thank you for your constructive suggestions. We conducted a detailed comparison with [22] in Q1A1 and reported the suggested performance. Our proposed RG-SAN improves by 6.7 points, demonstrating its effectiveness. We will include this discussion in new version. Your suggestions will enhance the robustness of our contributions.

---

Q4: In Eq. (2), how do you initialize W_E and W_S to obtain initial representations?

A4: Thank you for your attention to the details of our paper. In Eq. (2), W_E and W_S are initialized randomly. We will include this information in new version to enhance clarity.

---

Q5: For Eq. (5), what is the intuition of adding positional encoding to position features? Does the order of visual features impact the attention output?

A5: Thank you for your constructive question. Adding absolute position encoding is common in computer vision [51]. Changing the input order of visual features does not affect the final attention output because (1) the attention mechanism is order-invariant, and (2) the position encoding is tied to the visual tokens' 3D positions (xyz), so altering input order does not impact these positions.

---

Q6: Why is positional encoding added in Eq. (6) but concatenated in Eqs. (9) and (10)?

A6: Thank you for your insightful inquiry. Our experiments show that addition and concatenation for positional encoding in Eq. (6), (9), and (10) yield similar results, as shown in Tab. B. The differences are negligible, so either approach can be used without significantly impacting the outcome.

Table B: Ablation of positional encoding usage, where "Cat" denotes concatenation, while "Add" denotes direct addition.

---

Q7: Evaluation of RTS for finding the target.

A7: Thank you for your question. We previously evaluated RTS's ability to find the target using LLAMA2 70B in Sec. F of the supplementary materials, achieving an 80% match rate. However, LLAMA2 is not entirely accurate, making it an unreliable benchmark.

To better validate RTS, we annotated the text of 9,508 Val set samples to mark the target word positions. RTS achieved an accuracy of 93.4%, compared to 63.7% for the Top1 method, confirming our algorithm's effectiveness.

We will include this evaluation in new version and open-source the annotations to ensure reproducibility.

---

Q8: The explanation for Tab. 4 points out Top1 tends to select different nodes variably (line 281), but RTS is also choosing different nodes?

A8: We apologize for the confusion. In line 281, "different" refers to predicted nodes that are not the target noun word. Top1 often selects nodes other than the target word, such as adjectives or verbs, leading to semantic confusion and an accuracy of only 63.7%.

In contrast, RTS accurately identifies the target word based on syntax, regardless of its position, achieving an accuracy of 93.4% as Q7A7 points out. This precise selection enhances semantic accuracy and significantly improves performance, as shown in Tab. 4.

We will revise the description in new version to improve clarity.

Thanks for the detailed rebuttal. The authors have addressed some of my concerns. Here are some feedback:

For Q2: You should not follow the notations from other work and assume the readers could follow. Please clearly define the notations in the revised version.

For Q5: If the order of visual features should not affect the output, then you should not add positional encoding to the visual features. Same to the positional features. If you add positional encoding, then the attention output would change if the inputs are permuted. This is a technical flawless.

For Q6: I hope you could be consistent about how you deal with the positional encoding (if you need to add it). Looks from the new ablation result using 'add' for both lead to the best performance.

I have also read the reviews from other reviewers. I would maintain my original rating as some of my concerns are not well-addressed.