SAM-Guided Masked Token Prediction for 3D Scene Understanding

We sincerely appreciate your thorough review and valuable suggestions. We have addressed your questions as follows.

Q1, apply on SOTA 3D detectors.

In the main paper, we did not apply our method to other state-of-the-art detection methods because those works often incorporate specialized design structures different from the pre-training backbones, making comparisons less fair. For pre-training tasks, comparisons are typically made with previous pre-training methods using the same structures. For example, Seal [1] only conducts training on the 3D U-Net and compares its results with previous outdoor pre-training methods. The overall performance of Seal, as reported in the paper, is also lower than state-of-the-art 3D semantic segmentation methods like 2DPASS [2] and LidarMultiNet [3] with full supervision on the nuScenes dataset.

However, we understand that applying our method to state-of-the-art detection methods can further demonstrate its generality. Therefore, we applied our approach to the state-of-the-art methods CAGroup3D [4] and VDETR [5]. Since CAGroup3D utilizes a specifically designed BiResNet backbone, we could only apply our SAM-guided knowledge distillation and representation re-weighting method to it. For VDETR, which reports results with both a modified ResNet34 encoder and a plain transformer encoder, we replaced its encoder with our pre-trained encoder and compared it to the transformer backbone results reported in the original paper. The experiments in Table 1 indicate that our pre-training strategy enhances the performance of these state-of-the-art 3D detection methods. Furthermore, with the proposed SAM-guided tokenization and two-stage masked autoencoder, the performance improvement in VDETR is larger than in CAGroup3D, demonstrating the effectiveness of our method.

Table 1: 3D object detection results on ScanNet dataset based on CAGroup3D and VDETR.

Q2, comparison with Seal.

Please refer to the first part of the general response at the beginning.

Q3, comparison with other structure pre-training techniques like PPT.

Please refer to the second part of the general response at the beginning.

Q4, domain gap between the single-view point clouds and the complete room-level ones.

This is a great question. Previous 3D scene understanding pre-training strategies like PointContrast [6], DepthContrast [7], and PiMAE [8] all pre-train on single-view point clouds before fine-tuning on complete room-level point clouds. As demonstrated in PointContrast, direct pre-training on complete room-level point clouds yields poorer results compared to single-view point clouds. This may be due to several factors, including a more abundant and diverse set of training samples and the regularization effect of natural noise from camera instability.

The limited domain gap between pre-training on single-view point clouds and fine-tuning on room-level multi-view point clouds can be attributed to several reasons. Pre-training on single-view point clouds helps the network learn detailed local features and geometric structures, which makes the network align well with the of room-level point clouds. This phase also enhances the network's robustness to various perspectives and partial visibility, facilitating effective knowledge transfer.

Q5, applicable to the online 3D perception setting.

Our method can also be applied to the online 3D perception setting. To demonstrate the generality of our approach, we applied it to the online 3D perception work [9]. However, due to differences in the backbone architecture, our pre-trained encoder cannot be directly integrated with the 3D detection method FCAF3D [10] used in [9]. Therefore, we replaced FCAF3D with VDETR [5] for the online detection. The experimental results in Table 2 below indicate that our method enhances performance in the online 3D perception setting, demonstrating the generality and effectiveness of the proposed pre-training approach.

Table 2: Online 3D object detection results on ScanNet dataset.

---

[1] Segment Any Point Cloud Sequences by Distilling Vision Foundation Models, NeurIPS 2023

[2] 2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds, ECCV 2022

[3] LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perception, AAAI 2023

[4] CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds, NeurIPS 2022

[5] V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection, ICLR 2024

[6] PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding, ECCV 2020

[7] Self-Supervised Pretraining of 3D Features on any Point-Cloud, ICCV 2021

[8] PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection, CVPR 2023

[9] Memory-based Adapters for Online 3D Scene Perception, CVPR 2024

[10] FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection, ECCV 2022

Thanks for your positive and insightful comments! We have addressed your questions as follows.

Q1, visual comparison with other mask generation methods.

We have added both visual and results comparisons with other mask generation strategies, such as superpoint and superpixel, to justify the effectiveness of SAM. Visualizations of these comparisons are included in the attached PDF file. Table 1 below indicates that utilizing SAM for mask generation achieves the best performance.

Table 1: Ablation study for different mask generation methods including superpixel, superpoint, and SAM.

Q2, what is training time for fine-tuning? How about memory consumption compared to Bridge3D?

We have included the training times in Table 2 below. Training Bridge3D requires approximately 31GB of memory, our method requires a total of 36GB of memory.

Table 2: Fine-tuning times for 3DETR and GroupFree3D on SUN RGB-D and ScanNetV2 datasets with 2 A100 GPUs.

Q3, application in the outdoor scene understanding.

One key motivation for our method is to address the 2D-3D alignment challenges posed by traditional KNN-based tokenization methods. In outdoor scenarios, point clouds are often very sparse, especially at long-range distances, making KNN-based tokenization methods unsuitable. Current transformer-based methods for outdoor environments utilize voxel or pillar methods for feature extraction, treating these voxels or pillars as 2D images. Due to significant differences between outdoor and indoor environments and the sparsity of point clouds, our SAM-guided tokenization may not be suitable for outdoor scenarios. However, other parts of our method, such as representation re-weighting and two-stage masked token prediction, can be effectively applied in outdoor contexts.

We have chosen BEV-MAE [2] as our baseline method to distill 2D features into 3D BEV maps using the proposed re-weighting strategy. In the second stage, we enable the student network to reconstruct the masked BEV features obtained from the first-stage teacher models. As shown in Table 3 below, our re-weighting strategy and two-stage masked token prediction also benefit outdoor scenarios. Due to time constraints, we were unable to fully optimize the training parameters, indicating that there is still room for performance improvement.

Table 3: 3D detection results in the outdoor Waymo dataset.

W1, typos, and format issues.

Thanks for pointing out! We will correct the typos and update the reference style in the revised version of the paper.

---

[1] GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds, CVPR 2023

[2] BEV-MAE: Bird's Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios, AAAI 2024

We sincerely appreciate your thorough review and valuable suggestions. We have addressed your questions as follows.

Q1, innovation of method.

Please refer to the first part of the general response at the beginning.

Q2, the insight of the two-stage design.

Masked token prediction has proven effective for single-modality learning [4]. Our method extends this concept to cross-modality knowledge distillation by designing an efficient two-stage framework for cross-modality masked feature prediction. Our two-stage design works as follows: In the first stage, we perform knowledge distillation using SAM-guided tokenization and SAM masks to seamlessly transfer 2D region-level information to 3D. In the second stage, we use masked-view inputs to predict contextualized 3D representations within a latent space aligned by a teacher model with complete-view inputs. This ensures that student models learn well-aligned and contextualized representations, leveraging the proven efficiency of masked feature prediction for multi-modality self-supervised learning.

The ablation study presented in Table 5 of the main paper (also shown in Table 1 below) demonstrates that the two-stage design outperforms the one-stage. In the one-stage design, the 3D encoder directly predicts masked parts of the 2D features from foundation models, resulting in poorer performance. This underperformance is likely due to the significant domain gap between 2D and 3D models, which causes the network to learn sub-optimal features when applying one-stage masked 2D feature prediction.

Our two-stage design addresses this issue by separating the process into two distinct phases: the first stage focuses on 2D to 3D knowledge distillation, and the second stage handles 3D masked token prediction. This approach effectively reduces the domain gap between 2D and 3D representations, leading to better representations and improved performance.

Q3, reasons for two stage design instead, three, or even four stages.

In the second stage, the teacher model is frozen, and the student model is trained to predict the masked feature tokens. Since the teacher model's weights remain unchanged, adding more stages would only extend the training duration for the student encoder in the second stage without providing additional benefits. Therefore, adding additional stages to train the student encoder, such as a third or fourth, is unnecessary. The experiments shown in Table 1 below indicate that introducing a third or fourth stage yields similar results to the two-stage setting even with more training time.

Table 1: Ablation study for stage settings.

Q4, why not utilize momentum updating?

When utilizing momentum updating, two augmented positive views are sent to both the trainable encoder and the momentum encoder to obtain positive representation pairs or pseudo-label pairs. The weights of the momentum encoder are then updated with the weight of the trainable encoder through the contrastive loss or the supervision loss via pseudo-labels. However, our method does not require two augmented views for training in either stage; therefore, momentum updating is not applicable to our approach.

Q5, why group-balanced re-weighting will work.

Group-balanced re-weighting is introduced to address the long-tail problem inherent in the natural imbalance of object class occurrences. As discussed in the paper [5], foundation models provide well-represented patch-level features that can effectively identify different classes through clustering-based methods due to their training on large datasets, making it effective as a zero-shot learner.

In this paper, we take advantage of the foundational model to extract object-level features and apply a clustering method to compute distribution statistics. These statistics are less noise-prone because the foundational model provides robust feature representation for both head and tail class objects. We then normalize the distribution factors as weights to regularize the distillation process, offering an effective solution to the long-tail issue.

Q6, comparison with CLIP2Scene and Seal.

Please refer to the second part of the general response at the beginning.

---

[1] Segment Any Point Cloud Sequences by Distilling Vision Foundation Models, NeurIPS 2023

[2] Bridging the domain gap: Self-supervised 3d scene understanding with foundation models, NeurIPS 2023

[3] Videoprism: A foundational visual encoder for video understanding, ICML 2024

[4] Self-supervised learning from images with a joint-embedding predictive architecture, CVPR 2023

[5] Deep ViT Features as Dense Visual Descriptors, ECCVW 2022

Thanks for your positive and insightful comments! We have addressed your questions as follows.

Q1, why SAM is chosen as the distillation source for tokenization:

We chose SAM [1] to guide tokenization because it generates great zero-shot masks that provide boundary regularities and effectively facilitate region-level knowledge distillation. In contrast, other foundational models, such as Stable Diffusion [2] or DINOv2 [3], cannot generate comparable zero-shot masks.

For feature distillation, we use DINOv2 as the teacher model, enabling the student 3D models to predict the same region-level features obtained from DINOv2. We conducted an ablation study comparing features obtained from other foundational models like SAM, Stable Diffusion, and CLIP [4] to extract visual features. The results in Table 1 below indicate that utilizing DINOv2 achieves the best performance.

Table 1: Ablation study for the choice of foundation models for representation distillation.

Q2: other tasks where the proposed representation could be useful.

Reviewer icGN suggests that our method could also be applied in online 3D perception settings. To demonstrate the generality of our approach, we applied it to the online 3D perception method described in [5]. However, due to differences in the backbone architecture, our pre-trained encoder cannot be directly integrated with the 3D detection method FCAF3D [6] used in [5]. Therefore, we replaced FCAF3D with the transformer-based VDETR [7] for online detection. The experimental results in Table 2 indicate that our method enhances performance in the online 3D perception setting, demonstrating the generality and effectiveness of the proposed pre-training approach.

Additionally, by replacing the DINOv2 teacher model with CLIP, our method could potentially enable 3D zero-shot semantic segmentation by calculating the feature similarity between the predicted features and the CLIP text features. Furthermore, since our method distills features from 2D foundation models into 3D networks, it can enhance various 3D tasks that benefit from 2D features, such as 3D scene classification, 3D reconstruction, and 3D instance segmentation. However, due to time limitations, we were unable to apply our method to these tasks in the current study, but we plan to do so in the future to further demonstrate the generality of the proposed method.

Table 2: Online 3D object detection results on ScanNet dataset.

Q3, some definitions.

What is the input to the $F_{2D,i}$ and $F_{3D,i}$?

As illustrated in the main paper (lines 174 to 177), $I$ represents pixel-level features obtained from raw images processed by foundation models and feature interpolation. $H_i$ represents the SAM-guided tokenized features obtained from the raw point clouds with the $i$-th masks generated by SAM. In Equation 1, we utilize the $i$-th masks to group the pixel-level image feature representations $I$ into the $i$-th object-level features to obtain $F_{2D,i}$ and utilize one MLP project layer to project $H_i$ to the $F_{3D,i}$. Therefore, the inputs for $F_{2D,i}$ and $F_{3D,i}$ are the raw images, point clouds, and the $i$-th masks generated by SAM.

Where is the i sampled from?

$i$ represents the $i$-th masks generated by SAM.

What is $n_{min}$ and $n_{max}$?

$n_{min}$ and $n_{max}$ represent the minimum and maximum numbers of feature clusters.

$\frac{1}{O_i}$ in Eq (1) should probably be written as $\frac{1}{|O_i|}$

Thanks for pointing out! We will correct $\frac{1}{O_i}$ in equation (1) to $\frac{1}{|O_i|}$.

---

[1] Segment Anything, CVPR 2023

[2] High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022

[3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024

[4] Learning Transferable Visual Models From Natural Language Supervision, ICML, 2021

[5] Memory-based Adapters for Online 3D Scene Perception, CVPR 2024

[6] FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection, ECCV 2022

[7] V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection, ICLR 2024

Thank you for being a reviewer for NeurIPS2024, your service is invaluable to the community!

The authors have submitted their feedback.

Could you check the rebuttal and other reviewers' comments and start a discussion with the authors and other reviewers?

Regards, Your AC