We appreciate that you recognize the significance of our work. We will respond to your concerns in the following:

Q1: For the validation of TGNN and X-RefSeg with the RoBERTa backbone

Thanks for your detailed reading and pointing out this. We reimplement the TGNN and X-RefSeg with the same RoBERTa backbone as ours to conduct experiments on ScanRefer benchmark, since these exisitng works have not applied RoBERTa backbone as language module for their experiments. We then follow their settings to train and validate the performances. As shown in the table below, we can obtain their newly corresponding results, 28.83% mIoU for TGNN (RoBERTa) and 30.72% mIoU for X-RefSeg (RoBERTa), which still shows inferior performance to ours 33.74%, which further validates the superiority of our single-stage prardigm. Besides, we observe that when the loss coefficient is set to 10, our LESS outperforms the performance of TGNN and X-RefSeg on all the metrics. More discussion will be added in our paper.

Q2: For the more querie and layers do not bring performance gains

Thanks for pointing out this. With an excessive number of queries, too many queries become redundant, leading to overlapping or duplicate proposal masks. This redundancy does not contribute to better segmentation and can instead introduce noise, even degrading performance improvements. Besides, Increasing the number of layers can lead to overfitting, especially with the limited supervisory signals available in our label-efficient setting, binay mask supervision.

Q3: For reasons for good performance of proposed aligment modules

Thanks for your detailed reading and pointing out this. Point-Word Cross-Modal Alignment module aligns fined-grained point and word features. Also Query Mask Predictor module and Sentence Query Alignment modules make the coarse-grained alignment between masks and sentences. The coarse-grained sentences-masks and fine-grained words-points effectively coupled to enhance the capability in capturing multi-modal context, which is helpful for our aligment modules.

Q4: For the training time and inference time

Thanks for your detailed reading and pointing out this. As shown at General Respons and in the following table, we evaluate the training and inference time both of TGNN and X-RefSeg. All experiments are conducted on an NVIDIA 4090 GPU and the batch sizes of three methods are kept the same. For the two-stage training and inference of TGNN and X-RefSeg, we followed the settings in their open source codes. We can find that our LESS consumes less time than both of TGNN and X-RefSeg in both training and inference.

Thanks again for these constructive comments, we will take all these experimental results or further discussions into consideration into our revised version. If any questions, please kindly let us know.

Dear reviewer, thank you very much for your detailed and constructive comments.

Q1: For the validation on other dataset

Thanks for your detailed reading and pointing out this. We also conduct the experiments on the Nr3d and Sr3d datasets, as shown at General Response and in the following table. We can find that the performance of our LESS surpass the performance of TGNN by 3.6% and 1.8% on Nr3d and Sr3d respectively. Such results present promising potential for the future works and shed new light on label-efficient and single-stage exploration for R3DIS task.

Q2: For whether not select the mask with the highest score

Thanks for your detailed reading and pointing out this.

Firstly, using a weighted sum enables a soft decision-making process rather than a hard and binary choice. This can help in cases where multiple masks have similar scores, allowing the final prediction to be an aggregation of these maks, which can be more accurate than any single one.

Besides, we also conduct the ablation study which selects the mask with the highest score in the ScanRefer dataset, as shown in the following table. It can be shown that aggregating multiply masks leads to better performance than selecting only the highest one. This approach can help in capturing nuances and fine details that might be missed when relying on a solely single mask, which provide a more robust and accurate final prediction for the final model performance.

Q3: For the cite of Mask3D

Thanks for reminding us with this. We will add Mask3D[1] to our related literature and extend further discussions with Mask3D.

[1]. Mask3d: Mask transformer for 3d semantic instance segmentation, ICRA 2023.

Thanks again for these constructive comments, we will take all these experimental results or further discussions into consideration into our revised version. If any questions, please kindly let us know.

Thanks for your detailed and constructive comments. Next we address your questions.

Q1: For the metrics of Referring 3D Instance Segmentation

Thanks for pointing out this question. For the metrics of Referring 3D Instance Segmentation task, our experimental metrics are constantly followed by the previous works[1,2], called Acc@kIOU. Acc@kIOU measures the accuracy of test samples with an IoU higher than specific threshold k, where k ∈ {0.25, 0.5}. And we can find that the calculation method is much similar with the APx metric for common instance segmentation task. Hence, we choose the used Acc@kIOU in our main paper as the previous methods for the same Referring 3D Instance Segmentation task. We hope this explanation can help to clarify your confusion.

[1]. Text-guided graph neural networks for referring 3d instance segmentation.

[2]. X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks.

Q2: For the validation on other dataset

Thanks for your detailed reading and pointing out this. We also conduct the experiments on the Nr3d and Sr3d datasets, as shown at General Response and in the following table. We can find that the performance of our LESS surpass the performance of TGNN by 3.6% and 1.8% on Nr3d and Sr3d respectively. Such results present promising potential for the future works and shed new light on label-efficient and single-stage exploration for R3DIS task.

Thanks again for these constructive comments, we will take all these experimental results or further discussions into consideration into our revised version. If any questions, please kindly let us know.

Thanks for your detailed and constructive comments. Next we address to your concerns as follows:

Q1: For the novelty of the module design

The core motivation of our approach is to explore a novel single-stage R3DIS to fully embrace semantic concepts from text query and visual cues into a unified paradigm, while leveraging an efficient binary supervision. Most of existing works focus on two-stage pipeline to reach R3DIS: first instance proposal network and then a new network to classify these proposal into the corresponding text query, which separate the interaction between text query and visual cues. To investigate label-efficient and single-stage pipeline in R3DIS task, we delve into fusion mechanism for better aligning visual features with textual embeddings within a cross-attention manner. Specially, we design coarse-grained feature alignment and fine-grained feature alignment strategies, corresponding to our QMP and SQA, and PWCA, respectively. Besides, the utility of area regularization and point-to-point contrastive learning are effective to eliminate ambiguities among multiple objects and backgrounds, since single-stage approach possibly struggles with positive false and negative true samples without a well-trained proposal generator network. All of the proposed components in our paper lead to our competitive SoTA and pioneering single-stage R3DIS LESS. Moreover, our methods needs only binary mask annotations to train the model while the previous ones need heavy both instance labels and semantic labels, which strikes to label-efficient and single-stage approach for this task.

Q2: For the explanations of why TGNN and X-RefSeg perform worse and the cases they fail

Firstly, as mentioned in line 31-34 in the paper, owing to accuracy of mode and the lacking of linguistic guidance, TGNN and X-RefSeg may miss the target in the pre-segmentation stage. In this case, it is impossible to provide language-related high-quality instance proposals for the second stage. Besides, compared to our fine-grained and coarse-grained alignment, their visual-language alignment is weak, usually applying language guidance at the second stage only. We deduce these are the main reasons why both of them performed poorly. Moreover, if the targets are severely missed in the first stage or queries are complex, TGNN and X-RefSeg will perform worse.

Q3: For the loss coefficient of area regularization

We conduct further ablations in Q7 together.

Q4: For the running time compared to pervious two-stage approaches

We also conduct comparisons to address this concern, please kindly refer to the table of our General Response to all Reviewers.

Q5: For the results with Acc@0.5 are worse than those of TGNN and X-RefSeg while the results with Acc@0.25 are opposite

Since TGNN and X-RefSeg are all two-stage methods, that their 3D instance semantic segmentation networks in the first stage are first well-trained by both instance labels and semantic labels to support higher precision instance proposals for the second stage than ours. This benefits for the Acc@0.5 metric. However, as shown in Q7, we can still find that our LESS also surpasses both TGNN and X-RefSeg in Acc@50, which indicates our model is better on not only locating coarse location but also exact matching, thanks to our elaborate alignment strategies for training between language information and visual cues.

Q6: For the comment of detailed setting of TGNN and X-RefSeg

In our main paper for the R3DIS task, TGNN and X-RefSeg both leverage a two-stage paradigm, which consists of a 3D instance semantic network and a instance matching network. Obviously, TGNN and X-RefSeg rely on instance labels, semantic labels and binary masks to finalize referring 3DIS task rather than "only binary masks", the efficient way we used. Noting that, we trained 3D spares U-Net’s architecture from scratch without its pretrained weights, which is different from previous methods.

Q7: For how does the loss coefficient affect the result

We further conduct more ablation studies of λp2p, λseg and λarea. It can be seen that λp2p presents marginal influence in terms of final performance and we tend to set with default 0.05. And reducing λseg is advantageous for obtaining decent performance. We analyze that this is due to the reduction of λseg has the effect of amplifying the impact of the area regularization loss, thereby facilitating a stronger discrimination between backgrounds and foregrounds, leading to local mask predictions. We still observe improvements when λarea increases, especially for Acc@50 metric, which indicates our area regularization loss can effectively exclude backgrounds and get more precise masks. We have uploaded the corresponding results on the PDF, please kindly refer to that due to the Rebuttal Words Limitation.

Q8: For the ablation studies of the number of queries

We further conduct more ablation studies in terms of the number of queries on ScanRefer. As shown in the following table, we ablate the number of queries as {5, 15, 20}, it can be observed that keeping more queries brings higher accuracy while lower accuracy when using fewer queries. We suppose that fewer queries can not help to learn comprehensive feature patterns while an appropriate number of queries is enough to cover the needed feature patterns.

Q9: For the validation on other dataset

We also conduct the experiments on the Nr3d and Sr3d datasets to address this concern, please kindly refer to the table of our General Response to all Reviewers.

Thanks again for these constructive comments, we will take all these experimental results into consideration into our revised version.

Thank you for your questions and suggestions. A recent work, OpenScene[1] has established a straightforward and efficient baseline network in the field of open vocabulary 3D segmentation using a simple cross modal distillation method based on primary cosine smilarity driven, but also with a pioneering explroation. The objective of our work is to present a straightforward and effective single-stage baseline for the task of 3D referring segmentation. In addition to a single stage architecture design, our work makes a further contribution in the form of a single-stage coarse-to-fine cross-modal alignment network and a coarse-to-fine background point filtering loss function. These have been developed with the aim of achieving state-of-the-art performance under efficient binary mask supervision, which have been demonstrated with extensive ablations mentioned before.

Thank you again for your suggestions. We sincerely hope that you will be able to evaluate the contribution of our work based on its overall design and the differences that it presents in comparison to previous works [2][3], and also the hitting cornerstone of single-stage pipeline of this task.

[1]. Openscene: 3d scene understanding with open vocabularies, CVPR 2023.

[2]. Text-guided graph neural networks for referring 3d instance segmentation, AAAI 2021

[3]. X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks, AAAI 2024