XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation

Thanks for your careful review and constructive comments! Hopefully the following response will address your concerns.

1. Paper organization.

The organization should be improved ...

Thanks for your advice! We will reorganize the first two sections in our revised paper.

2. Clarifications.

(A) Mask-level alignment.

... clarification on how mask-level alignment ...

The most critical challenge in addressing the 3D open vocabulary problem is establishing 3D-text alignment. Previous approaches have tackled this by enforcing either global-wise or point-wise alignment between 3D features and 2D-text embeddings.

Global feature alignment typically involves calculating a global feature of a scene or view through pooling and applying contrastive loss with 2D-text embeddings. However, pooling often result in the loss of detailed information, leading to unclear and imprecise boundaries on novel categories.

Point-wise feature alignment matches 3D points with 2D pixels using camera parameters, and performs dense contrastive learning between them. While this method preserves boundary details, it incurs a significantly larger computational burden and less stable training, as the loss is highly susceptible to outliers.

Our proposed mask-level alignment presents an intermediate solution of these two approaches. Compared to global features, mask features retain more detailed information since there are dozens of masks within a single view. Additionally, mask features are more robust than point features, as the distraction from outliers is mitigated by the mask pooling operation.

... unified feature space ...?

The 2D features with high open-vocabulary capacity are embedded in the 2D-text feature space. By applying 2D-to-3D mask regularization to the 3D features, these 3D features are drawn towards the 2D-text feature space. Consequently, the fused features, which integrate both 2D and 3D features, are expected to align well with the 2D-text embedding space. This alignment is further conveyed by the superior open-vocabulary performance of XMask3D compared to PLA and OpenScene. Thus, we believe that concatenating text and fused features will effectively create a unified feature space that integrates 3D, 2D, and textual modalities.

(B) Main contributions compared with OpenScene and PLA.

... main contributions compared to PLA and OpenScene ...

Sorry for not directly highlighting our contributions relative to OpenScene and PLA.

OpenScene employs point-wise distillation and feature ensemble, while XMask3D utilizes mask-wise feature regularization. As discussed in Section 2(A), mask-wise contrastive learning is more robust and less computationally demanding compared to point-wise counterpart.

PLA proposes entity-level point-caption association through set difference and intersection operations, which is less precise and adaptable than our mask-level 3D-2D-text association. The mask for 3D-text alignment is adaptively predicted by the 2D mask generator, making XMask3D an end-to-end and integrated system.

Therefore, the core contribution of our paper is the introduction of a more adaptive and robust mask-level alignment technique.

(C) Implicit 3D captioner.

Does the Implicit 3D Captioner ...

According to the ablation studies in Table 3(a), the proposed implicit 3D captioner yields better novel class results compared to using vanilla text embeddings or implicit 2D caption embeddings, which convey the effectiveness of the implicit 3D captioner. However, unlike Cap3D, the implicit 3D captioner does not directly generate text captions. It only produces conditional features that work effectively with the XMask3D pipeline and we cannot guarantee its robustness in other 3D-to-text generation scenarios.

(D) Text-to-image diffusion model.

Can your text-to-image diffusion model ...

In our XMask3D pipeline, all weights of the denoising UNet are frozen. We do not generate images using the text-to-image diffusion model. Instead, we use the HxWxC features from the denoising UNet to generate 2D masks through the mask generator. Only the mask generator is trained on our datasets.

(E) View-level contrastive loss.

What is view-level contrastive ...

Sorry for the unclear statement. The ground truth for the view-level contrastive loss is the text embedding of the view image caption. The predictions are derived from the average pooling results of the 3D branch features, 2D branch features and fused features respectively. Therefore, we have three view-level contrastive losses and design three separate coefficients for them.

3. Additional visualizations.

... 2D Mask and 3D Mask in Figure 3 ...

Thanks for your insightful advice! In Figure 4 in the PDF attachment of the global response, we display the outputs from the 2D and 3D branches for the same samples shown in Figure 3 of our main paper.

4. ScanNet++ dataset.

... results on ScanNet++ ...

We apologize for not providing results on ScanNet++, as we followed the protocols of previous papers (OpenScene and PLA). Following your advice, we have submitted a request to download ScanNet++, but unfortunately it is still pending, so we are unable to conduct experiments within the rebuttal period. We appreciate your suggestion regarding this advanced dataset and plan to add it to our future work.

5. Resource comparisons.

... model parameters and FLOPs ...

Thanks for your suggestion! As the FLOPs of 3D models are conditioned on point cloud numbers, we only report the additional FLOPs of the 2D branch. In future work, we plan to replace the 2D branch with a more efficient and lightweight open vocabulary 2D mask generator.

Thanks for your careful review and constructive comments! Hopefully the following response will address your concerns.

1. Results comparison.

The authors don't compare with state-of-the-art 3D semantic segmentation OV3D.

Thanks for your suggestion! We will include this outstanding method in the ScanNet comparison in our revised paper. OV3D primarily focuses on EntityText extraction and Point-EntityText association, while XMask3D concentrates on enhancing mask-level interaction between 2D and 3D modalities. Therefore, the contributions of OV3D and XMask3D are orthogonal and could complement each other. Once the official code of OV3D is released, we plan to integrate our model with OV3D's advanced design in prompting LLM-powered LVLM models and Pixel-EntityText alignment to further enhance the performance of XMask3D.

2. Expanded limitations.

The authors highlighed fututre work in the limitation, it would be good if you can expand it with some limitation on the technical side or some failure cases.

(A) Technical limitations.

Currently, the denoising UNet from the pre-trained diffusion model requires significant computational resources and impacts inference efficiency. In contrast to PLA, which consists solely of a 3D model, XMask3D could be further improved by replacing the 2D branch with a more lightweight and efficient 2D open vocabulary mask generator.

(B) Failure case analysis.

Thanks for your constructive suggestion! We provide three failure cases of XMask3D in Figure 1 in the PDF attachment of the global response. We will include this figure and the following discussions in the supplemental material of our revised paper.

The first sample shows a bathtub with a shower curtain in the bathroom. However, the 2D/3D branch and the fused output of XMask3D all misclassify the shower curtain as curtain. This may be because these two categories are similar in object shape and texture, with the only difference being the surrounding environment. Since XMask3D only takes a corner of the room as input instead of the entire scene, the global environmental information is insufficient for making the correct category prediction.

The second sample shows a large area of picture on the wall. The 2D/3D branch and the fused output of XMask3D all misclassified it as wall, due to their similar geometry. In most cases, a picture is a small region on the wall, and this picture, as large as the wall, is a typical corner case. This failure case may reveal XMask3D's over-reliance on geometric knowledge and lesser consideration of texture information when encountering out-of-distribution samples.

The third sample shows a sink on a counter. Due to the occlusion problem, the sink point cloud is incomplete, negatively affecting the prediction of segmentation boundaries between the sink and the counter. This occurs because they are geometrically similar when the sinking-down part of the sink is missing.

Thanks for your careful review and constructive comments! Hopefully the following response will address your concerns.

1. Application of XMask3D on other scenarios.

The paper evaluates the proposed method on a limited set of benchmarks (ScanNet20, ScanNet200, S3DIS), all of which are indoor scene datasets. Authors could discuss how the method might perform on outdoor datasets.

We follow the experimental settings outlined in PLA for our experiments on indoor scene datasets. Based on your advice, we observed that other open vocabulary papers [1,2] also conduct experiments on the outdoor scene dataset nuScenes[3]. However, this dataset only provides point cloud data with annotations and raw image data without annotations. Without image-level segmentation ground truth, we cannot train the 2D mask generator. In our future work, we plan to replace the entire 2D branch with a pre-trained 2D open vocabulary segmentation model, enabling us to handle the nuScenes dataset without requiring 2D annotations.

However, we have made efforts to provide some qualitative results on the nuScenes dataset. First, we project the 3D annotations onto 2D images using the camera's intrinsic and extrinsic parameters. Due to the sparsity of the 3D point cloud, we further employ the k-nearest-neighbor algorithm to fill in the blank regions without pixel-point correspondence, as shown in Figure 3 in the PDF attachment of the global response. The projected 2D label maps and the outputs from XMask3D on several nuScenes samples are shown in Figure 2, demonstrating the potential of XMask3D in handling outdoor scenarios.

Additionally, the authors could provide a qualitative analysis of the model's potential limitations when applied to different environments.

As discussed above, the current architecture of XMask3D requires full annotations for both 2D and 3D data. Therefore, when dealing with outdoor datasets that lack fine-grained 2D annotations, the performance of XMask3D may decrease. Additionally, we provide a qualitative analysis of failure cases in 3. Failure case analysis, which reveals potential limitations of XMask3D under various circumstances.

2. Computational costs.

The reliance on the denoising UNet from a pre-trained diffusion model could be seen as a potential weakness or limitation, especially given the computational resources required for training and inference.

We acknowledge that the denoising UNet requires significant computational resources and slows down the inference speed. In our future work, we plan to address this limitation by replacing the 2D branch with a more lightweight 2D open vocabulary mask generator. Thank you for highlighting this potential weakness, which has inspired our future improvements.

3. Failure case analysis.

The paper could benefit from a more detailed error analysis to understand the failure modes of XMask3D, especially in novel category segmentation.

Thanks for your constructive suggestion! We provide three failure cases of XMask3D in Figure 1 in the PDF attachment of the global response. We will include this figure and the following discussions in the supplemental material of our revised paper.

The first sample shows a bathtub with a shower curtain in the bathroom. However, the 2D/3D branch and the fused output of XMask3D all misclassify the shower curtain as curtain. This may be because these two categories are similar in object shape and texture, with the only difference being the surrounding environment. Since XMask3D only takes a corner of the room as input instead of the entire scene, the global environmental information is insufficient for making the correct category prediction.

The second sample shows a large area of picture on the wall. The 2D/3D branch and the fused output of XMask3D all misclassified it as wall, due to their similar geometry. In most cases, a picture is a small region on the wall, and this picture, as large as the wall, is a typical corner case. This failure case may reveal XMask3D's over-reliance on geometric knowledge and lesser consideration of texture information when encountering out-of-distribution samples.

The third sample shows a sink on a counter. Due to the occlusion problem, the sink point cloud is incomplete, negatively affecting the prediction of segmentation boundaries between the sink and the counter. This occurs because they are geometrically similar when the sinking-down part of the sink is missing.

References

[1] Jihan Yang, et al. "RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding." CVPR. 2024.
[2] Qingdong, He, et al. "UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representation." IJCAI. 2022.
[3] Holger Caesar, et al. "nuscenes: A multimodal dataset for autonomous driving." CVPR. 2020.

Thanks for your careful review and constructive comments! Hopefully the following response will address your concerns.

1. About the less satisfactory results of classes that cover large areas.

While the method exhibits superior performance w.r.t. competing methods, it seems that the output fused embeddings yields to geometric inconsistent features for semantic classes that cover large areas of the point cloud such as wall, ceiling and floor. This is evident in both partitioning settings when the class is either base or novel (Table 5 (a) and (b) in supp.).

The underlying reasons for the unsatisfactory performances of semantic classes on the ScanNet and S3DIS datasets are distinct. For the ScanNet dataset, XMask3D processes only a corner of the scene in each forward pass, whereas PLA directly feeds the entire scene into the model. This segmentation approach negatively impacts categories that cover large areas, such as wall, ceiling, and floor, because cutting these categories into smaller pieces inevitably undermines segmentation performance by losing long-range relational information.

For the S3DIS dataset, in addition to the aforementioned partial input issue, we encounter a trade-off between computational resource consumption and segmentation performance. The validation samples in S3DIS are densely populated, with an image view potentially corresponding to over 500,000 points, which causes out-of-memory errors on a 24GB NVIDIA 4090 device and slow inference speed. Consequently, we discard views exceeding a threshold of 260,000 points for higher efficiency. However, this view selection process results in hollow regions in the merged outputs, primarily in the corners of rooms, and these regions typically correspond to the categories wall, ceiling, or floor. We conducted an ablation study, increasing the selection threshold from 260,000 to 500,000 points, as shown in the following table. When the threshold is raised, the segmentation performance for ceiling, floor, and wall consistently improves.

Following the weaknesses section, do the authors have any additional insights into why this phenomenon occurs with the fused embeddings? Could the 2D regularization terms ($L_{seg}^{2D}$, $L_{view}^{2D}$) be introducing too much bias towards the 2D visual modality, thereby causing geometric discontinuities in the output fused features?

As discussed above, the primary reasons are the partial point cloud input and the trade-off with memory consumption. We believe that the 2D regularization term does not significantly bias the fused features towards geometric discontinuities. As illustrated in Figures 1, 4, and 6 of our main paper, the fused features of wall and floor fully leverage the continuous geometry from the 3D branch. Additionally, we provide a qualitative analysis in the following table. The 2D branch performance is significantly worse than the others. The fused features perform better than the 3D features for wall and only slightly worse for floor. The numerical results also demonstrate that the performance of fused features in segmenting wall and floor is not significantly affected by geometric discontinuities from the 2D regularization terms.