Multi-View Representation is What You Need for Point-Cloud Pre-Training

We sincerely appreciate the reviewer's insightful feedback and positive comments. In response to the concerns raised, we offer the following clarifications and additional information.

Q: All previous approaches have used the pre-trained weights from the encoder and decoder.

We thank the reviewer for raising this question. We would like to clarify this question here for different downstream tasks.

In the shape classification task, only the encoder is employed across all pre-training methods. This is because the classification task does not necessitate a decoder.

Similarly, for the object detection task, pre-training methods exclusively utilize the encoder. The detection decoder is typically custom-designed and differs significantly from those used in pre-training.

Regarding the semantic segmentation task, while the decoder could theoretically be reused, some recent pre-training methods like DepthContrast[1] do not incorporate a decoder during pre-training. Consequently, these methods do not offer a decoder component that can be initialized for downstream tasks.

In our experimental analysis, we observed that the encoder weights play a pivotal role. As demonstrated in the table below, when transferring pre-trained decoder weights, our model consistently outperforms the second-best method on semantic segmentation tasks.

Q: Masking ratio ablation experiment.

We express our gratitude to the reviewer for the meticulous attention in pointing this out. We intend to provide a more detailed explanation in our revised manuscript. Our method involved masking the input point cloud guided by the corresponding RGB image. Following the conventional practice in masked autoencoder-based pre-training [2], we divided the image into regular non-overlapping patches. Then we sample a subset of patches and mask (i.e., remove) the remaining ones. The remaining part is projected onto the point cloud, forming the input for our method. We employ this masking operation as a data augmentation technique, which assists in the development of more robust features during pre-training. As evidenced in Table 4(b) (referenced below), an appropriate masking operation during pre-training could enhance performance in downstream tasks.

[1] Zhang, Zaiwei, et al. "Self-supervised pretraining of 3d features on any point-cloud." In ICCV 2021.

[2] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." In CVPR 2022.

We are deeply grateful to the reviewer for their insightful feedback and positive comments. In addressing the concerns raised, we offer the following clarifications and additional information.

Q: Masking ratio ablation experiment.

We express our gratitude to the reviewer for the meticulous attention in pointing this out. We intend to provide a more detailed explanation in our revised manuscript. Our method involved masking the input point cloud guided by the corresponding RGB image. Following the conventional practice in masked autoencoder-based pre-training [1], we divided the image into regular non-overlapping patches. Then we sample a subset of patches and mask (i.e., remove) the remaining ones. The remaining part is projected onto the point cloud, forming the input for our method. We employ this masking operation as a data augmentation technique, which assists in the development of more robust features during pre-training. As evidenced in Table 4(b) (referenced below), an appropriate masking operation during pre-training could enhance performance in downstream tasks.

Q: Important recent related works.

We are thankful to the reviewer for bringing these related works to our attention. We will include a comprehensive discussion of these works in our revised manuscript. Mask3D introduces a method of pre-training in 2D by harnessing 3D priors, with the aim of enhancing 2D task performance. Our approach, conversely, seeks to apply 2D knowledge to augment 3D tasks, establishing a reversed transfer of knowledge. Given the differing objectives, a direct comparison with Mask3D is not straightforward.

MM-3DScene has proposed an innovative masking model for point cloud pre-training. We have conducted a comparative analysis with their method on the object detection task, and the results are presented in the table that follows. Our method outperforms MM-3DScene in most cases.

We also compare our method with MM-3DScene on the semantic segmentation task. Since MM-3DScene employs Point Transformer as the encoder backbone, we ensured a fair comparison by implementing Point Transformer as the backbone during our method's pre-training phase. The results, which are presented in the subsequent table, demonstrate that our method outperforms MM-3DScene under the same experimental setting.

Q: The number of params and FLOPS.

We are grateful to the reviewer for highlighting this matter. During pre-training, our model comprises a total of 110M parameters and operates at 31 GFLOPs. As a comparison, a preivous state-of-the-art method, 4DContrast, contains 71M parameters. It is important to note that while contrastive learning-based methods typically necessitate large batch sizes, our method does not require this. Consequently, our approach offers greater flexibility across various hardware configurations.

Q: Pre-processing discussion.

We extend our gratitude to the reviewers for bringing this to our attention. We acknowledge the significance of this point and will address it, adding a corresponding note to the limitations section in our revised manuscript. The process of projecting RGB-D scans to point clouds requires approximately 10 hours on a 64-core CPU machine. Storaging the generated point cloud would require ~600G extra space.

Q: Qualitative/visualization results.

We are thankful to the reviewer for the suggestion. In response, we have included additional qualitative results for the semantic segmentation task in the updated supplementary material.

[1] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." In CVPR 2022.

I appreciate the detailed response from the authors.

Most of the responses addressed my concerns, so I will keep my score of 6.

Please also note that, the qualitative results shown in the supplementary material are still not enough, more results are expected to be provided in the final version.

To summarize, I lean to accept this paper under the condition of providing more qualitative results and adding the missing discussions of related works and experimental results listed in the rebuttal.

We extend our heartfelt thanks to the reviewer for the positive feedback. In addressing the concerns raised, we are pleased to provide the following clarifications and additional information.

Q: Zero-shot experiments.

We extend our thanks to the reviewer for highlighting this aspect. Our method, not being designed to align learned features with the language domain, faces challenges in direct application to zero-shot experiments like PointCLIP [1] or ULIP [2]. Nevertheless, to demonstrate the effectiveness of our approach, we have included results from few-shot learning experiments conducted on ModelNet40. These results can be found in Table 3 of the Supplementary Material. The table below also presents additional results pre-trained on ShapeNet for a comprehensive view.

To further prove the efficacy of our method, we conducted the label-efficient learning experiments on the scene-level SUN RGB-D object detection task. During these experiments, we trained our model using varied proportions of the dataset, specifically 20%, 50%, and 100%, and subsequently evaluated the performance on the same test dataset. We reported the mAP @0.5 results in the table below. Our observations indicate that our pre-training method yields more significant gains with reduced amounts of labeled training data. Notably, with only 50% of the training data, our method surpasses the performance achieved by training from scratch using the full 100% dataset.

We hope that these relevant experiments will effectively address the reviewer's concerns.

[1] Zhang, Renrui, et al. "Pointclip: Point cloud understanding by clip." In CVPR 2022.

[2] Xue, Le, et al. "ULIP: Learning a unified representation of language, images, and point clouds for 3D understanding." In CVPR 2023.

We express our sincere gratitude to the reviewer for the insightful feedback. In response to the concerns raised, we offer the following clarifications and additional information.

Q: Adopt various backbone models for point cloud learning.

We deeply appreciate the reviewer for the insightful suggestion. We totally agree that a generic pre-training scheme should have the ability to enhance feature extraction in various point cloud backbones. In our paper, we have employed SR-UNet as the primary backbone for scene-level experiments and Transformer for shape-level tasks. We contend that this demonstrates the adaptability of our pre-training scheme to at least two different point cloud backbones.

To further validate the efficacy of our method, we have conducted additional experiments incorporating other encoder backbones for the rebuttal. We selected two point cloud encoders for this purpose:

1. PointNet++-based encoder: For this encoder type, network features are aggregated through multiple set abstraction levels. We utilized PointNeXt [1] as the representative encoder backbone in this category, which is an advanced version of PointNet++.
2. Graph-based encoder: In this approach, each point in a point cloud is treated as a graph vertex. Directed edges are generated based on the neighboring points. For this category, we selected the commonly-used DGCNN [2] as our test encoder backbone.

The results, as detailed in the following table, clearly demonstrate significant improvements in both encoder backbones when pre-trained using our method.

[1] Qian, Guocheng, et al. "Pointnext: Revisiting pointnet++ with improved training and scaling strategies." In NeuIPS 2022

[2] Wang, Yue, et al. "Dynamic graph cnn for learning on point clouds." ACM Transactions on Graphics (tog) 38.5 (2019): 1-12.