UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior

Dear Reviewer T5Kw,

We thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns.

We hope to further discuss with you whether your concerns have been addressed or not. If you still have any unclear parts of our work, please let us know. Thanks.

Best, Authors

Q1: Since the backbone is modified from xMUDA, the oracle performance should also be provided.

A1: "Oracle" means solely training on the target domain, except in the "Day/Night" case where it uses a 50%/50% source/target batches to prevent overfitting due to the small target size, thereby serving as the upper bound for DA3SS.

Q2: The task settings in this study are derived from xMUDA (TPAMI), while the prompt tuning and corresponding experimental settings are based on VPT. Therefore, this proposed method can be considered somewhat incremental.

A2: Our method is groundbreaking in introducing the prompt-tuning concept into the universal model for DG3SS and DA3SS. In VPT-Deep, each prompt is a learnable token randomly initialized and directly insert into input space of all ViT layers. In contrast, our method first leverages sparse depth as a point-level prompts inserted to each ViT layers. It not only learns spatial distance perception prompts from point clouds but also learns invariance to sample perturbations. Secondly, our method uses the learnable token as query for seeking matched prompting after encoding in the ViT layer. It bridges the discrepancy between the information of pre-training dataset and the target scene.

Q3: Please explain why token length is equal to 100, the best performance can be obtained. The average performance along with the standard deviation also should be computed and reported.

A3: Consistent with all competitors, for each run of the experiment, we select the best model based on the validation set results (Via trial and error, when token length is set to 100, the result is stable and up to peak value), and use it for the final inference on the test set. All experimental settings are conducted at least 3 times and we select the best performance of all runs. The table below reports the 2D performance with standard deviation over 4 runs when the token length is set to 100.

Q4: Please explain the main contribution compared with "Learning to Adapt SAM for Segmenting Cross-domain Point Clouds".

A4: Above work leverages whole SAM to generate instance mask, guiding the alignment of features from diverse 3D data domains into a unified domain. On the contrary, our proposed UniDSeg does not directly use instance masks generated offline, but instead uses only the pre-trained image encoder of VFMs (e.g., CLIP and SAM) to extract visual prior information. UniDSeg is groundbreaking in introducing the prompt-tuning concept into the universal model for DG3SS and DA3SS. We propose a learnable-parameter-inspired mechanism to the off-the-shelf image encoder of VFMs, which maximally preserves pre-existing target awareness in VFMs to further enhance its generalizability. Just as the following A5 response, our method not only requires nearly 2% fine-tuning parameters of the visual backbone but also achieves superior segmentation performance.

Q5: Since VFM and ViT are utilized as encoder, the model parameters and computational costs should be comprehensive reported.

A5: Thank you for your suggestion. We have reported model parameters of CLIP: ViT-B, CLIP: ViT-L, and SAM: ViT-L for the VFM based encoder along with all trainable parameters. Note that, the entire ViT backbone in our VFM-based encoder is frozen during downstream training for DA3SS and DG3SS. Only two layer-wise learnable blocks, MTP and LST, are trainable. Hereby, we provide supplementary information on the trainable parameters for MTP and LST in the following table, where "Cost" means only need percentage trainable parameters compared to fine-tuning whole encoder consume.

Q6: Did the authors notice that when the point labels projected onto 2D images, there are slight mismatch between these labels and 2D image pixels? Please discuss the potential reasons.

A6: The mismatch can be caused by a variety of factors, including calibration error, geometric error, image resolution, etc. These factors are beyond the scope of this work. Following xMUDA, we assume that the calibration of LiDAR-Camera is available for both domains and does not change over time. Since images and point cloud are heterogeneous, according to extrinsic matrix from dataset, only the points falling into the intersected field of view are geometrically associated with multi-modal data (i.e., 3D-to-2D projection).

Dear Reviewer 4rFB,

We thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns.

We hope to further discuss with you whether your concerns have been addressed or not. If you still have any unclear parts of our work, please let us know. Thanks.

Best, Authors

Q1: This section only explains the steps of the method and lacks theoretical explanation of the method.

A1: Firstly, sparse depth map as a perspective projection representation of point clouds presents an unnatural image that can be processed by the image encoder in VFM. This is an additional prior derived from the point cloud. Lee et al. [Ref1] have been proved that source data internally know a lot more about the world and how the scene is formed, which called Privileged Information. This Privileged Information includes physical properties (e.g., depth) that might be useful for cross-domain learning. Secondly, though depth information is easy-to-access and tightly coupled with semantic information, Hu et al. [Ref2] argue that depth clues that complement colors are hard to deduce from color images alone, thus directly deep encoding fail to capture valid geometric information. To this end, we consider utilizing depth as point-level prompts inserted into ViT layers, enabling the model to learn spatial distance perception. This complements 2D representations, as LiDAR-derived depth information is less affected by domain variations compared to images, which can be easily influenced by changes in lighting and other factors. Experimental results in Table 8 further demonstrate the effectiveness of our method. For convenience, we reshow the table below:

[Ref1] Lee K H, Ros G, Li J, et al. Spigan: Privileged adversarial learning from simulation. arXiv preprint arXiv:1810.03756, 2018.

[Ref2] Hu S, Bonardi F, Bouchafa S, et al. Multi-modal unsupervised domain adaptation for semantic image segmentation. Pattern Recognition, 2023.

Q2: The novelty of LST.

A2: The innovation of our method is in introducing learnable-parameter-inspired mechanism to the off-the-shelf VFMs, which is guided by point-level prompts from 3D information. To achieve this, we place layer-wise MTP and LST blocks to take full advantage of semantic understanding of diverse levels and modalities. Among them, the affinity matrix computed in LST is a common solution to capture the associations between learnable tokens and 2D representations. The insight of learnable token originally comes from Visual Prompt Tuning (VPT) [Ref3].

[Ref3] Jia M, Tang L, Chen B C, et al. Visual prompt tuning. European Conference on Computer Vision, 2022: 709-727.

Q3: As a pioneer method, it is recommended to verify its effectiveness on VFMs designed for segmentation tasks such as SAM [3] and SEEM [4].

A3: We have supplemented the experimental results for DA3SS and DG3SS under another visual backbone, SAM: ViT-L. Notably, in this work, we only utilize the off-the-shelf image encoder of VFMs (e.g., CLIP (w/o text encoder) and SAM (w/o prompt encoder and mask decoder)). As shown in the table below, SAM-based UniDSeg exhibit better performance on "USA/Sing." scenario. Due to rebuttal time limitation, we will supplement all SAM-based experimental results subsequently.

Q4: Why does sparse depth have scenes at different scales?

A4: We are sorry to make you misunderstand. Here, "different scales" refers to different receptive fields of adjacent pixels. The complementarity of the features extracted by the 2D and 3D branches is also tightly correlated to the different information processing machinery, i.e., 2D and 3D convolutions, which makes networks focusing on different areas of the scene with different receptive fields. When sparse depth is projected onto the image plane and processed via a 2D network, it can focus on the scope of scenes at different scales. For example, adjacent pixels in the image may have similar pixel values, but their corresponding depth values (distance) in the depth map can differ significantly.

Q5: Experiments are needed to verify the impact of the number of layers of MTP and LST.

A5: In the following table, "Half layer" indicates that the MTP and LST blocks are inserted every other layer. Due to rebuttal time limitation, we will supplement the experiments of other layer combinations subsequently.

Q6: How long does it take from the start of training to completion?

A6: All experiments are conducted on one NVIDIA RTX 3090 GPU with 24GB RAM.

Q7: "Avg" is ambiguous. This leads to misleading averages of 2D and 3D mIoU results.

A7: Initially, xMUDA used "2D+3D" to denote the final results, but we found that "2D+3D" might be misleading as it implies an average of 2D and 3D. Therefore, we follow MM2D3D and VFMSeg by using "Avg" to denote the final result. Perhaps "Ensemble Result" (short for "Ens") would be more appropriate. Notably, in all your answers, we have changed "Avg" to "Ens".

Dear Reviewer KKSy,

We thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns.

We hope to further discuss with you whether your concerns have been addressed or not. If you still have any unclear parts of our work, please let us know. Thanks.

Best, Authors

Q1: The improvement does not seem to be strong enough.

A1: Our method enhances the performance of both DG3SS and DA3SS. Notably, all loss functions are the same as those in xMUDA, and no typical data augmentation or style transfer methods are introduced to address domain shift issues. The following table shows that UniDSeg, when combined with cross-domain distillation loss from Dual-Cross, achieves best performance compared to SOTA MM2D3D. We believe that more cross-domain learning constraints can provide better model.

When combining with MM2D3D, a parallel reinitialized encoder is added to the 2D backbone to process the sparse depth. We have shown the comparisons in Table 8. On Sing./USA scenario, compared to (a), (b) not only requires 0.6% fine-tuning parameters of the 2D backbone but also achieves superior segmentation performance, making it flexible to expand to various 3D semantic segmentation tasks with multi-modal learning. For convenience, we reshow the table below:

Q2: Analyze the reasons behind the SOTA performance.

A2: VFMSeg essentially utilizes SAM to generate instance masks for both the source and target images, applying mixing augmentation to the source and target domain data. This is a typical and useful method used to address domain shift in DA3SS. In contrast, our proposed UniDSeg exploits the image encoder of VFMs (e.g., CLIP and SAM) to extract robust visual prior information. UniDSeg proposes a learnable-parameter-inspired mechanism to the off-the-shelf image encoder of VFMs, which maximally preserves pre-existing target awareness in VFMs to further enhance its generalizability.

Q3: The motivation is not clear enough.

A3: Thank you for your suggestion. The motivation of this work is to study a universal framework based on VFMs to enhance the generalizability and adaptability of cross-domain 3D semantic segmentation, demonstrating the effectiveness of the visual foundation model priors. Different from VFMSeg, which utilize SAM to generate instance masks for both the source and target images, applying mixing augmentation to the source and target domain data. Our proposed UniDSeg is groundbreaking in introducing the depth-guided prompt-tuning concept into the image encoder of VFMs, which can be applied to various downstream tasks, including DA3SS, DG3SS, SFDA3SS, and fully-supervised 3SS.

Q4: Why not try a more advanced model?

A4: We supplement the experimental results for DA3SS and DG3SS under SAM: ViT-L. As shown in the table below, SAM-based UniDSeg exhibit better performance on "USA/Sing." scenario.

Q5: Some detailed explanations could be seen.

A5: (1) "source-domain data discrimination power" refers to the scenario in the DG3SS task where learning is conducted solely access to source domain data, enabling the model to develop the ability to discriminate domain-specific and domain-agnostic features; (2) Spatial: introducing spatial information (depth) into the prompt space to supplement 2D images; Temporal: since most scenes are captured smoothly, we learn the temporal invariance from the temporal correlations between the adjacent frames; (3) "pre-existing target awareness": In fine-tuning stage, the weights of VFMs are conventionally used to initialize source models and subsequently discarded. However, VFMs have diverse features important for generalization, and finetuning on source data can overfit to source distribution and potentially lose pre-existing target information.

Q6: What's SAM's performance on the datasets?

A6: In Table 2 of our paper, the experimental results demonstrate that, whether freezing or fine-tuning the image encoder of the VFM, it cannot directly solve the domain shift issue present in DG3SS and DA3SS. We supplement the experimental comparison between fine-tuning and our method, implemented on SAM image encoder serving as a visual backbone. The motivation please refer to A3. For other high-level tasks, they are not the focus of this work.

Q7: Pseudo-label difference and improvement compared to VFMSeg.

A7: Based on the VFMSeg experimental results, this method did not provide helpful pseudo-labels in "Day/Night" scenario, and there was even a decrease (-0.4) in performance. VFMSeg uses the averaging the probabilistic prediction of pretrained 2D network and SEEM to generate pixel-wise pseudo-labels, while our method follows the common practice adopted by most DA3SS methods in using offline 2D pseudo-label.

Dear Reviewer DeKz,

We thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns.

We hope to further discuss with you whether your concerns have been addressed or not. If you still have any unclear parts of our work, please let us know. Thanks.

Best, Authors

Q1:  What does Samp. in Figure 1 means, the corresponding explanation is not provided in the manuscript.

A1: "Samp." means sampling of 2D features. Only the points falling into the intersected field of view are geometrically associated with multi-modal data (i.e., 3D-to-2D projection). After "Samp." operation, we can obtain point-wise 2D features for cross-modal learning with 3D features.

Q2: Why choose CLIP as 2D backbone, since there is no language information for the task, other Visual Foundation Models (e.g. Swin Transformer, SAM, …) seem more suitable for the 2D information extraction.

A2: Thank you for your reminder. The motivation of this work is to study a universal framework based on VFMs to enhance the generalizability and adaptability of cross-domain 3D semantic segmentation, demonstrating the effectiveness of the visual foundation model priors. To this end, we utilize the off-the-shelf image encoder of VFMs as our visual backbone. UniDSeg can be applied to various downstream tasks, including DA3SS, DG3SS, SFDA3SS, and fully-supervised 3SS. Following your suggestion, we have supplemented the experimental results for DA3SS and DG3SS under another visual backbone, SAM: ViT-L. As shown in the table below, SAM-based UniDSeg exhibit better performance on "USA/Sing." scenario.

Furthermore, we supplement the experimental comparison between fine-tuning and our method, implemented on SAM image encoder serving as a visual backbone. Due to rebuttal time limitation, we will supplement all SAM-based experimental results subsequently.

Q3: Though SparseConvNet is examined as 3D network, it would be more convincing as "universal" if authors could provide results utilizing other 3D backbones.

A3: Thank you for your suggestion. We have supplemented the DA3SS experimental results for xMUDA and our proposed UniDSeg under another 3D backbone network, MinkowskiNet. Due to rebuttal time limitation, we will supplement all MinkowskiNet-based experimental results subsequently.

The authors' response has clearly addressed my concerns. After checking the peer review comments and the author's responses, I decided to raise the given score for this work.