A Unified Framework for 3D Scene Understanding

• To Question 2: “The design choice of Mask Decoder Layer is weird…”

Reply: Good question! Here's a detailed discussion: 1) For the order of cross/self-attention, please note that the $k, v$ from$f _ i$ represent features from the entire scene. Cross-attention can effectively evaluate the relationship between the sampled $q_u$ and the scene $k$, while self-attention evaluates the pairwise importance among each $q _ u$. By applying cross-attention first, a global perspective is incorporated into $q_u$, helping self-attention make further adjustments (see below experiments). We also would like to point out that such order is a common operation and is widely used in other representative works [1,2]. 2) For the mentioned duplicate information, we argue that since $q_u$ is randomly selected from $f_i$, incorporating global perspectives in cross-attention is advantageous. The presence of 'duplicate information' is not an issue, as such overlap is typical for query, key, and value in 2D or 3D transformer methods [3,4]. 3) As mentioned in 1) and 2), we reiterate that a global perspective is essential, as demonstrated by the results in the second table below, which show that simultaneously incorporating a global perspective into $q_u$, $q_p$, and $q_t$ significantly enhances performance, further supporting our argument.

[1] UniVS: Unified and Universal Video Segmentation with Prompts as Queries. CVPR 24.

[2] Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 22.

[3] DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. ICLR 23.

[4] CenterFormer: Center-based Transformer for 3D Object Detection. ECCV 22.

• To Question 3: “Table 6 shows that unifying the tasks are hurting…then what is the point of unifying all the tasks together…”

Reply: We first propose a simple baseline to unify six 3D segmentation tasks. However, we observe challenges in multi-task joint optimization. We would like to clarify that instead of presenting our final results, the goal of Table 6 is to prove multi-task unification as a challenging topic where introducing new tasks might hurt the performance of existing tasks. To relieve these impacts, we propose UniSeg3D, which builds inter-task associations to jointly optimize the associated tasks. Experiments demonstrate that our UniSeg3D not only supports six tasks but also surpasses currently specialized SOTA approaches on all tasks, verifying the motivation of our method. Considering that this is the first work successfully unifying six 3D segmentation tasks, we believe it would be valuable and interesting for people in this area.

Thanks for providing feedback and taking the time to review our work!

• To Weakness 1: “The writing needs improvement...”

Reply: Thank you for carefully reading this paper. We apologize for any confusion regarding notations. Specifically, $K_{v}$ refers to the length of the class name vocabulary; the Multiplication in equation (3) is indeed a matrix multiplication; Positive and negative samples denote samples successfully matched or miss-matched with labels, respectively; and $\mathbf{mask}_{pos}$ in equation (6) represents the mask predictions from the positive samples. We will carefully revise these notations in the revised version.

• To Weakness 2 & Question 1: “…models/design choice…different tasks are not consistent…model A is better…model B…” / “…compare models/design choice…table 4(a)…compare line 2 and line 4…”

Reply: UniSeg3D is the first model unifying six 3D segmentation tasks. OneFormer3D is the only existing unified model for 3D segmentation, covering just three tasks. To compare comprehensively and demonstrate our model's superiority, we first evaluate UniSeg3D on OneFormer3D's three tasks, selecting the best checkpoint. This checkpoint is used to test interactive, referring, and open-vocabulary tasks. Remarkably, without specific checkpoint selection, UniSeg3D outperforms current specialized SOTA methods. Additionally, different complete convergence checkpoints show minimal differences in performance, making both models viable. In practical applications, checkpoints can be chosen based on preferred tasks while maintaining good performance across other tasks.

For table 4(a), lines 2 and 4 both represent our methods. We empirically find that, with the fine-tuning trick, the setting of line 4 performs better compared to line 2 (see below table). Thus, we employ the setting of line 4 for our method. Note that the fine-tuning trick improves performance without extra inference costs.

• To Weakness 3: “Without the extra fine-tunning trick… Table I and IV in appendix…questionable.”

Reply:  Thank you! Without the trick, UniSeg3D already achieves highly competitive performance against OneFormer3D, supporting three additional tasks. It consistently outperforms specialized SOTA methods in interactive, referring, and open-vocabulary segmentation before fine-tuning. We argue that achieving full convergence while optimizing six tasks simultaneously is challenging. Hence, we introduce a simple trick requiring only a few epochs of fine-tuning, allowing UniSeg3D to surpass all current specialized SOTA methods across six tasks. Importantly, this fine-tuning trick does not add any additional inference cost, making it a practical solution.

Besides, considering Table I and IV in the appendix, we would like to clarify two points: 1) For the instance segmentation task, mAP is a more comprehensive metric than mAP$_ {50}$ and mAP$_ {25}$. UniSeg3D achieves the same mAP as OneFormer3D but shows notable improvement under stricter metrics (e.g., mAP$_ {85}$, mAP$_ {90}$), indicating more precise mask generation; 2) For the referring segmentation task, mIoU is more comprehensive, and UniSeg3D outperforms the SOTA method X-RefSeg3D by 4.1 mIoU, proving UniSeg3D's superior performance in this task.

• To Weakness 4 & Question 4: “…interactive segmentation is the best task…no analysis….” / “…table 1 could not support the claim that interactive segmentation is superior in mask prediction…”

Reply: Good suggestion! Visual prompts are crucial for interactive segmentation, providing precise object location priors that guide focus on foreground targets and reduce background noise. The interactive segmentation can be considered as an extension of instance segmentation with the addition of strong object location priors. Consequently, it excels in mask prediction, achieving 76.0 mIoU, 7.9 higher than the 68.1 mIoU of instance segmentation. We will add discussions and this comparison to Table 1 in the next version for clarity.

For questions 2 & 3, please see the reply presented in the “comment” part.

Thanks for providing feedback and taking the time to review our work!

• To Weakness 1 & 2: “Marginal performance gain...” / “Additionally, in Table 6, unifying…”

Reply: Good question! Unifying the tasks into a single model could save computation consumption and benefit real-world applications, which has become a trend in the community [1][2]. In this paper, we aim to achieve six 3D segmentation tasks within a single model in one inference. We first propose a simple baseline unifying such six tasks and find that it can not achieve ideal performance, and we argue that the reasons are from two aspects: challenges of creating unified representations and difficulties of multi-task joint optimization for these tasks. We would like to clarify that the goal of Table 6 is to prove multi-task unification a challenging topic instead of our final results, e.g., introducing new tasks might hurt the performance of existing tasks. To relieve these impacts, we propose UniSeg3D, which uses queries to unify the representations and employs knowledge distillation with ranking-based contrastive learning to jointly optimize associated tasks. Extensive experiments demonstrate that our method not only supports six tasks but also surpasses currently specialized SOTA approaches on all tasks. Besides, following your suggestion, we provide the standard deviation of UniSeg3D by training 3 times. Referring to the Table below, we report slight standard deviations on six tasks.

[1] UniGS: Unified Representation for Image Generation and Segmentation. CVPR 24.

[2] UniVS: Unified and Universal Video Segmentation with Prompts as Queries. CVPR 24.

• To Weakness 3: “Some minor issues…”

Reply: Thank you for pointing out these minor issues. We will carefully revise them in the next version.

• To Question 1: “Equation 3: The current notation…”

Reply: Sorry for this unclear statement. Actually, they are indeed cross products. We will add descriptions in the revised version.

• To Question 2: “Table 1: where does…”

Reply: Thanks! To make full use of knowledge contained in superior 3D scene understanding tasks, we propose knowledge distillation to transfer knowledge from superior tasks to other tasks, which improves multi-task performance without extra inference costs. The key to knowledge distillation is to utilize the task-predicting segmentation masks of the best quality to guide the other tasks, i.e., using a teacher to guide the students. Considering that visual prompts can provide location priors indicting precise object positions, naturally making the interactive task perform best in mask prediction. To prove this, we provide quantitative results (mIoU) in Table 1 of the manuscript. Essentially, the main difference between instance and interactive segmentation is w/o or w/ visual prompt. Here, we further compare the instance segmentation with interactive segmentation. Referring to the Table below, the interactive segmentation significantly outperforms instance segmentation by 7.9% mIoU, indicating the superior quality of predicted masks provided by the interactive segmentation task. We will add these discussions to make Table 1 more clear in the revised version.

• To Question 3: “Notations: $𝑒^{𝑃}$ and $𝑒^{𝑇}$ in Equation 4…”

Reply: Good suggestion! Following your suggestion, we now add the corresponding legends in Figure 3 for better clarity, which can be found in Figure-R 3 of the attached PDF.

• To Question 4: “Section 3.3: the motivation…”

Reply: Sorry for this unclear statement. In this paper, we propose the knowledge distillation loss to implement inter-task association, i.e., transfer the 3D scene understanding knowledge from the superior task (a good teacher) to the rest of the tasks. Therefore, considering that the visual prompts provide strong object location priors (see the reply to question 2), we employ interactive segmentation as the teacher. We use this teacher to provide auxiliary guidance, which improves multi-task performance without extra inference costs. At the same time, we would like to clarify that we mainly employ the interactive segmentation as the teacher, instead of the referring segmentation, to benefit other tasks (including the referring segmentation). The experimental results prove that under such inter-task association, we achieve considerable improvements compared with the baseline.

• To Limitation:

Reply: Titling 3D segmentation research as 3D scene understanding has been a common practice in this community[1][2]. We will explore the relationships between 3D segmentation and other 3D scene understanding tasks in the future. Besides, UniSeg3D can provide pixel-level semantic information, which might serve as potential reliable input for other 3D understanding tasks.

[1] OpenScene: 3D Scene Understanding With Open Vocabularies. CVPR 23.

[2] Exploring Data-Efficient 3D Scene Understanding With Contrastive Scene Contexts. CVPR 21.

Thank you for your valuable feedback!

• To Weakness 1: “Since the feature for visual prompt is sampled from the superpoints, the quality of the visual prompt significantly influence the overall performance of the model.”

Reply: Thanks. For the interactive segmentation task, it is intuitive that using different visual prompts will report different results. To investigate its influence, we study the three variants, including random, center, and prompts provided by the SOTA method (AGILE3D [1]), and the performance is shown in the table below. We can find that under the fair setting, i.e., using the prompts from AGILE3D, the proposed UniSeg3D achieves notable performance improvement compared with the SOTA. For the other tasks, due to attention masks in cross attention layer, the visual prompt will not influence their performance. We will add these discussions in the revised version.

[1] AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation. ICLR 2024.

• To Weakness 2: “The experiments are only conducted on ScanNet-based datasets, as a unified model, the authors should provide more experiments on different datasets to validate the method as done in previous works[1][2].”

Reply: Good suggestion! To the best of our knowledge, ScanNet is the only publicly available dataset supporting six tasks. Thus, to explore a unified framework supporting six tasks in a single model, we mainly conduct experiments on the ScanNet-based datasets in the manuscript.

Unfortunately, we note that the work[1] mainly focuses on visual grounding tasks, i.e., it is a 3D object detection task, while this paper focuses on the various segmentation tasks. The second work[2] explores the open-vocabulary segmentation task on the Replica dataset[3], one of our supported tasks. Hence, we follow work[2] to conduct zero-shot open-vocabulary segmentation on the Replica[3] dataset. The Table below demonstrates that the UniSeg3D surpasses the current SOTA approach Open3DIS.

As a supplement, we conduct experiments on the S3DIS dataset. Note that the S3DIS dataset does not provide text expressions, i.e., it can not support the referring segmentation task. Here, we evaluate the panoptic, semantic, instance, and interactive segmentation tasks on the S3DIS dataset. We can see that UniSeg3D outperforms current specialized SOTA approaches, especially for the challenging panoptic segmentation task, verifying the effectiveness of our method. We will include these new comparisons in the revised version.

[1] 3d-vista: Pre-trained transformer for 3d vision and text alignment. ICCV 23.

[2] Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. CVPR 24.

[3] The Replica dataset: A digital replica of indoor spaces. arXiv 19.

• To Weakness 3: “Unlike previous works[2][3], UniSeg3D directly learn the relation between text and 3D without any 2D supervision or guidance. It is a concern that how "open" is this framework in OVS task. Some visualized experiments on open-set text queries as in [2][3][4] could answer the question.”

Reply: Thanks for admiring the distinctiveness of our work! The proposed UniSeg3D directly learns relations between text and 3D, mitigating the dependencies on 2D supervision. Previous works[2][3][4] mainly employ two experiment settings to evaluate how “open” they are, i.e., using random open-set text prompts on the ScanNet200 dataset and the cross-dataset evaluation. Following your suggestion, we visualize the open-vocabulary segmentation results under these two experiment settings. Please see Figure-R 1 and Figure-R 2 in the attached PDF for details. As we can see, the UniSeg3D presents the applicability from the following aspects:

(1) support open-set classes which are not included in the training data (illustrated in Figure-R 1);

(2) support attribute descriptions such as affordance and color (illustrated in Figure-R 1);

(3) support cross-dataset point cloud data (illustrated in Figure-R 2).

Since the above three applications, we think the UniSeg3D presents a desirable performance in how “open.”

Thanks for providing feedback and taking the time to review our work! We promise that the training/inference codes, logs, and checkpoints will be released.

Weaknesses:

• To Weakness 1: “The main weakness of this work is that the proposed architecture is widely used for multi-modal perception tasks, limiting the model’s novelty. However, this is acceptable as long as the model’s performance is indeed great, as simple and effective models are usually similar.”

Reply: Thanks. This paper aims to establish a simple method to unify various 3D understanding tasks. Thus, we make the architecture as simple as possible, which can serve as a solid baseline for future unified research. We also appreciate that the reviewer agrees the simplicity of the proposed architecture is acceptable.

• To Weakness 2: “The paper claims knowledge distillation losses as one of its contributions. However, according to the ablations, adding these losses only makes marginal changes.”

Reply: Thanks. We first construct a unified baseline that achieves highly competitive performance on six different tasks. Based on this baseline, we further propose knowledge distillation losses, which not only maintain the consistent SOTA performance on referring segmentation and open-vocabulary segmentation tasks but also improve the panoptic segmentation, instance segmentation, and interactive segmentation tasks by notable 0.8 PQ, 1.3 mAP, and 2.1 AP, respectively. To fully explore its effectiveness, we choose the representative instance segmentation task for discussing the results on strict metrics (i.e., mAP$_{75}$), as shown below. We can see that the proposed knowledge distillation losses will bring more noticeable performance gains (+2.3%) under the strict metric compared with the baseline.

Questions:

Sorry for the unclear details. We give detailed explanations for each question in the following content.

• To Question 1: “The detailed process of producing super points.”

Reply: The superpoints are produced in an unsupervised manner, and they are formed by grouping points with locally similar geometric structures. Specifically, our procedures for producing superpoints can be divided into three steps:

Step 1: generate handcrafted local geometric features of the 3D points; Step 2: employ the farthest point sampling (FPS) algorithm in the coordinate space of point clouds to obtain the initial superpoint centers; Step 3: construct an association map between the points and superpoint centers.

We will add introductions of the detailed processes, producing superpoints in the revised version.

• To Question 2: “The criteria for sampling the corresponding points from the clicks.”

Reply: We sample the 3D points nearest to the clicks as the corresponding points. Specifically, in practice, clicks are clicked in 2D projections of 3D scenes. Therefore, the clicks are initially represented as 2D coordinates of the corresponding 2D projections. While reverse mapping the 2D projections into original 3D scenes, the clicks are concurrently reverse mapping to concrete 3D coordinates of the 3D scenes. Afterward, we sample points nearest to these concrete 3D coordinates as the corresponding points.

• To Question 3: “For the referring segmentation, how to decide the target segmentation mask from many text tokens?”

Reply: Each textual expression refers to a specific object segmentation mask, and the text tokens are individually encoded from respective textual expressions. We feed the text tokens into the mask decoder and generate respective mask predictions. Therefore, the target segmentation mask, i.e., the mask predictions, are naturally corresponded to the input text tokens in a one-to-one manner. With many text tokens, we decide the target segmentation masks to the individually respective input text tokens.

• To Question 4: “For L182-186, I am confused that isn't formula-5 already satisfied by using formula-4?”

Reply: Sorry for the confused statement. Actually, while $\mathbf{e}^{P} _ {i}\mathbf{e}^{T} _ {i}<\mathbf{e}^{P} _ {i}\mathbf{e}^{T} _ {j}$, the formula-5 is not satisfied by using formula-4. Specifically, the formula-5 is:

$\mathcal{L} _ {rank}=\frac{1}{B}\sum _ {i=1}^{B}\sum _ {j=1}^{B}\mathrm{max\left (0,\mathbf{s} _ {i,j}-\mathbf{s} _ {i,i} \right)}$,

where $\text{max}\left ( 0,\mathbf{s} _ {i,j}-\mathbf{s} _ {i,i} \right)= \mathbf{s} _ {i,j}-\mathbf{s} _ {i,i}$ if $\mathbf{e}^{P} _ {i}\mathbf{e}^{T} _ {i}<\mathbf{e}^{P} _ {i}\mathbf{e}^{T} _ {j}$, and $0$ otherwise. $\mathbf{e}^{P} _ {i}\mathbf{e}^{T} _ {i}$ and $\mathbf{e}^{P} _ {i}\mathbf{e}^{T} _ {j}$ are input parameters of formula-4.

Dear Reviewers,

We sincerely appreciate your time and effort in reviewing our paper. We hope our explanations have addressed your concerns. As we are in the discussion phase, we welcome any additional comments or questions regarding our response or the main paper. If further clarification is needed, please do not hesitate to mention it, and we will promptly address your inquiries. We look forward to receiving your feedback.

Best regard,

Paper 1782 Authors