Point Cloud Self-supervised Learning via 3D to Multi-view Masked Leaner

We sincerely appreciate your thorough review and valuable suggestions. We have addressed your questions as follows.

W1: Grammatical error, and font size.

Thank you for pointing that out. We have corrected the grammatical errors and increased the font size in those figures.

Q1: Code open-source.

Thank you for acknowledging our contribution. We will release the code upon acceptance.

Q2: Novelty and comparison with I2P-MAE and TAP.

We have included a detailed discussion on the novelty of our approach and a comparison with I2P-MAE, TAP, and other methods in the revised supplementary materials, Sections A and B. The discussion is summarized as follows:

The novelty and comparison with I2P-MAE are provided in the general response.

TAP uses a pre-trained VAE to reconstruct 2D images from 3D inputs but fails to leverage multi-view information effectively. In contrast, our method introduces a unified approach that uses masked point clouds to reconstruct both multi-view 2D images and the original point clouds, ensuring a more comprehensive understanding of 3D geometry while effectively utilizing the multi-view attributes of the 3D data. Furthermore, we propose MSMH decoder to better global and local features and a two-stage self-training method to learn well-aligned representations.

Q3: Include more works in the comparative experiments.

Thanks for your suggestions, we have added ACT, I2P-MAE, and ReCon to Tables 1, 2, and 3 of the revised main paper about comparative experiments.

Q4: MSMH has been explored in Point-M2AE and more unsupervised methods.

The multi-scale multi-head (MSMH) attention mechanism described in this paper aims to capture more comprehensive features of the network. While this idea has also been explored in Point-M2AE and other unsupervised methods, our approach has several distinguishing aspects.

Point-M2AE introduces an attention structure to extract both local and global features. However, our method differs in two significant ways. Firstly, the implementation details of our attention mechanism differ from those in Point-M2AE. Point-M2AE's network structure is similar to a a U-Net structure, and its attention mechanism is akin to Pyramid Attention. In contrast, our MSMH attention operates across multiple scales within each attention layer, subsequently concatenating these features to enhance the extraction of both global and local attention more effectively. Secondly, our MSMH attention is applied exclusively to the decoder, which allows us to retain the encoder’s original structure. This approach not only preserves but also enhances the generalizability of the model, as it requires minimal modification to the encoder.

Q5: Feature-level images in the abstract represented in the main text.

Unlike previous methods such as Pi-MAE and Joint-MAE, which utilized masked 2D images as input and reconstructed the original image, our approach directly uses point clouds to reconstruct multi-view depth images. A key challenge in our method is the translation of 3D tokens to 2D tokens. Directly using an MLP for this projection is not feasible, as 2D tokens and 3D tokens are not directly aligned. Therefore, we propose a novel method to map 3D tokens to 2D tokens, thereby generating feature-level images through a 3D-to-2D multi-view feature projection (described in line 193 in the main paper and Figure 2 (a). Those 3 images in Figure 2 (a) are feature-level images). Specifically, we project the point cloud token features to 2D image tokens with a token size of 16, resulting in a 14x14 grid of tokens for a 224x224 image. After projection, we obtain 196 (14x14) tokens for each multi-view projection and subsequently use these feature-level images to reconstruct the original 2D depth images.

Q6: Ablation study for the view information.

These experiments are provided in the supplementary materials. In Table 2, we present ablation studies on the effectiveness of the pose pool size (i.e., the total number of multi-view images used). In Table 3, we provide ablation studies on the number of reconstructed views in the decoder at the same time. In Table 5, we investigate different view configurations (Circular, Spherical, Spherical, and Circular, Random). Our ablation studies are thorough and exhaustive in investigating the impact of view information on experimental results.

Q7: Qualitative visualizations.

For the second-stage design, our method focuses on feature reconstruction. Therefore, visualizing the reconstruction across the entire two-stage process poses significant challenges. To address this, we provide visualization results by directly integrating MAE into the stage-one framework, as detailed in Table 7 of the main paper. We directly let the masked point cloud inputs to reconstruct the original point clouds and multi-view images. The visualization results are presented in Figure 1 in the supplementary materials.

We sincerely appreciate your thorough review and valuable suggestions. We have addressed your questions as follows.

Q1: For the task part segmentation and few-shot learning, the proposed method achieved little increase. It would be better if more explanation and analysis could be given.

The results on the few-shot learning task, particularly in the 5-way and 10-way settings, show that our proposed method achieves near-ceiling accuracy for this benchmark. Given this strong baseline, the observed improvements, though limited, are significant because further advancements in such high-performing scenarios are inherently challenging, emphasizing the robustness of our approach. Additionally, the updated Table 2 in the main paper demonstrates that our method outperforms even those models distilled from foundation models, further underscoring its effectiveness.

Please feel free to reach out if you have any further questions or need additional clarification.

We sincerely appreciate your thorough review and valuable suggestions. We have addressed your questions as follows.

Q1: The weakness in the novelty

Please see the general response.

Q2: Motivation of the method

Please see the general response.

Q3: Why only focus on 3D geometric learning given the power of multi-modal foundation models?

With the advancement of multi-modal foundation models, incorporating them to distill knowledge from other modalities will undoubtedly lead to significant improvements. However, we believe that focusing on 3D geometric learning without relying on foundation models is also crucial. Due to the current limitations of available 3D data, there are no true 3D foundation models. Nevertheless, as data availability improves, it is likely that we will eventually see a foundation model specifically designed for 3D geometric learning. Moreover, 3D geometric learning is not opposed to features distilled from foundation models but instead complements them, as demonstrated in Table 1 of the supplementary material. Therefore, enhancing geometric features remains a valuable and essential step toward advancing the community's understanding and capabilities in 3D learning.

Q4: No reason to avoid multi-modal foundation models.

As stated in the general response, the primary motivation of this work is not to avoid using multi-modal foundation models but rather to enhance 3D geometric representation learning independently. Existing methods trained with multi-modal foundation models, such as I2P-MAE and ReCon, focus on two types of features: (1) geometric features obtained using existing 3D MAE techniques, and (2) novel knowledge distillation mechanisms introduced by these methods.

While I2P-MAE and ReCon propose novel approaches for knowledge distillation, they largely rely on existing frameworks like Point-MAE and Point-M2AE for learning 3D geometric features. In contrast, our approach is dedicated to improving 3D geometric learning without leveraging foundation models. We identify key limitations in prior 3D geometric learning methods and introduce innovative solutions to overcome these challenges. As demonstrated by our ablation study in Table 1 of the supplementary material, the 3D geometric features learned through our approach are complementary to those distilled from foundation models, underscoring the value and generalizability of our proposed method.

Q5: No evidence to back up the claim incorporating both 2D and 3D during training is redundant.

In this paper, we emphasize that incorporating both 2D and 3D inputs for 3D geometric learning is redundant. This claim is supported by our ablation study in Table 6 of the main paper. Training with only 3D inputs to reconstruct both multi-view 2D images and 3D point clouds yields the best performance. Using both 2D and 3D as inputs to reconstruct the original 2D and 3D information provides only a limited improvement compared to the 3D-only baseline.

Q6: The claim from Line 49 to Line 53.

As mentioned in lines 43 to 45 in the main paper, problems in this claim happen in Joint-MAE and Pi-MAE. In those approaches, both masked images and point clouds are taken as inputs to reconstruct the original point clouds and images. This process provides the network with visible 2D information to reconstruct 2D data.

The key issue here is that using 2D data as input encourages the network to rely heavily on reconstructing 2D information based on visible portions, which inherently focuses more on learning semantic information rather than geometric information. This is because, as per the original MAE papers, 2D-to-2D reconstruction tends to prioritize semantic understanding (i.e., inferring context from visible parts) rather than focusing on 3D geometric properties. Additionally, the presence of visible 2D data results in some "view information leakage," as the network can use visible parts to directly predict masked areas, reducing the emphasis on comprehensive geometric understanding.

In contrast, our method uses only 3D point clouds as input. With the given pose information, we guide the network to "imagine" what the corresponding depth image should look like. This approach significantly benefits geometric learning since the network must understand the 3D structure and predict how it would appear from different viewpoints. This emphasis on reconstructing depth images from only 3D inputs encourages a deeper understanding of the geometric features of the object.

I hope this clarifies the context and key ideas we are trying to communicate. If further elaboration is needed, We would be happy to provide more details.

We sincerely appreciate your detailed feedback. Below, we address each of your questions:

Q1: Comparison with ReCon with the single modality setting.

It is important to note that ReCon modifies both the fine-tuning and pre-training stages of Point-MAE to enhance its performance. In our main paper, we used the exact same settings as the original Point-MAE, which makes a direct comparison with ReCon less straightforward. To address this, we conducted experiments by applying our method using the Point-MAE settings employed by ReCon. Our results indicate that our method outperforms ReCon in single-modal settings, showcasing its effectiveness in 3D geometric feature learning. Moreover, we conducted few-shot learning experiments using the same Point-MAE settings as ReCon. Results show that, even without leveraging foundation models, our method performs comparably to ReCon, and even surpasses it in the 10-way settings, further emphasizing the strength of our approach.

It is also worth mentioning that we reproduced the single modality results from ReCon, and observed that the achieved scores were lower than the reported results (as mentioned in this issue and this issue). While ReCon selects the best seeds to produce optimal results, we did not have sufficient time to perform an exhaustive search for the best training parameters.

Here, we would like to reiterate our key motivation. In this paper, we aim to explore how to effectively leverage 2D information in 3D geometric learning through the use of masked autoencoders. Therefore, our main comparisons should be with methods that incorporate 2D information in 3D geometric learning via masked autoencoders, such as Pi-MAE, Joint-MAE, and TAP. We have demonstrated that our approach introduces a novel direction for leveraging 2D information in 3D geometric learning using MAE (3D to 2D & 3D reconstruction) and achieves state-of-the-art performance in this area.

Q2: Question on claim "incorporating both 2D and 3D during training is redundant for 3D geometric learning."

In the main paper, we demonstrated that incorporating both 2D and 3D during training is redundant 3D geometric learning with rendered images in the ShapeNet dataset. As ShapeNet does not provide rendered image, We were unable to conduct experiments with real images in ShapeNet, so we instead used the ScanNet dataset with downstream detection tasks for further evaluation. The results show that our claim—that incorporating both 2D and 3D during training is redundant—also holds true for real images. We believe this finding presents an intriguing opportunity in the design space for developing learning strategies for 3D representation learning.

Q3: More comparison with GVA.

Thank you for your suggestions. To better demonstrate the effectiveness of MSMH, we provide comparisons across both the ScanNet and ModelNet40 datasets using different splits. Results indicate that MSMH achieves the better performance.

We sincerely appreciate your thorough review and valuable suggestions. We have addressed your questions as follows.

Q1: The motivations of the proposed method.

Please see the general response.

Q2: What is the role of depth images within the model?

In our method, the network exclusively takes point clouds as input. During the 3D to multi-view image reconstruction process, we project these point clouds into multiple views to compute the reconstruction loss, thereby promoting a deeper multi-view geometric understanding of the learned representations.

Q3: Difference between depth images and rendered images.

Both rendered and projected images are commonly used in 3D representation learning. Rendered images contain RGB color information, which makes them valuable for semantic feature learning, whereas projected images provide depth information, making them more suitable for 3D geometric feature learning. Since our method focuses on 3D geometric learning, we use projected images for calculating the reconstruction loss. Our ablation study, shown in Table 4 of the supplementary material, further demonstrates that utilizing projected images leads to the best performance.

Q4: Projection for 2D depth images is time-consuming.

As stated in the main paper line 149, we use PyTorch3D to project point clouds into multi-view images. This process is highly optimized in PyTorch3D, allowing for real-time performance, which makes the time consumption negligible.

Q5: Not address or optimize the identified drawbacks.

As detailed in lines 45-55 in the main paper, A major limitation of previous methods to learn 3D geometric features is the reduced effectiveness of using 2D images as input to the network for 3D geometric learning with MAE. Including 2D images as the input to the network in the 3D-to-2D reconstruction process causes the network to over-rely on visible 2D information for predicting masked content, rather than fully comprehending the multi-view geometric relationships, ultimately degrading representation learning. In contrast, our approach does not use 2D images as input to the network. Instead, we only utilize them to compute the reconstruction loss, enabling a 3D-to-multi-view reconstruction that relies solely on 3D inputs.

Q6: No IoU (%) results for each category and the visualization results.

Thank you for pointing this out. We have included the IoU (%) results for each category in the part segmentation tasks, which can be found in Table 9 of the supplementary material. Additionally, reconstruction visualization examples are provided in Figure 1 of the supplementary material.

We sincerely appreciate your thorough review and valuable suggestions. We have addressed your questions as follows.

Q1: Insufficient evidence or ablation studies to substantiate the claim.

In Table 6 of the main paper, we present an ablation study analyzing the input and output modalities. Additionally, Table 4 in the supplementary materials provides an ablation study comparing different 2D image types (depth images versus rendered images). Those results support our claims and demonstrate that using point clouds as the sole input to reconstruct original multi-view depth images and point clouds yields the best performance.

Q2: Some previous works have often utilized only 3D inputs.

As mentioned in the paper, our focus is on geometric learning through MAE without relying on foundation models. While previous methods like I2P-MAE and ReCon also use only 3D inputs, their primary objective is to distill semantic or textual features from foundation models into the 3D modality. In contrast, our work exclusively targets 3D geometric features, which is complementary to features distilled from foundation models. To the best of our knowledge, this is the first work to reconstruct both 3D and 2D information solely from 3D inputs for geometric learning without incorporating foundation models. As demonstrated in Table 1 of the supplementary material, the learned geometric features are complementary to those derived from foundation models, showcasing the value and generalizability of the learned 3D geometric representations.

Q3: The question on the training epoch and times.

In both stages, we utilize 300 epochs. To demonstrate that the improvement is not simply due to an extended training period, we also train the baseline Point-MAE for 600 epochs. The results show negligible differences compared to those at 300 epochs. Furthermore, as shown in Table 7 of the main paper (also presented below), training with the proposed one-stage method for just 300 epochs already outperforms the baseline and previous methods. Adding the two-stage training further enhances performance, indicating that our improvements are not attributable to additional training epochs alone. Furthermore, although the two-stage structure double the training time, but it is still lower than other methods like I2P-MAE and Recon.

Q4: It is better to demonstrate the individual effectiveness of each component in Table 5

Thank you for the suggestion. In the revised version, we have included an ablation study for each component, which can be found in Table 5 of the main paper.

Q5: Additional results on more complex and larger datasets, such as Objaverse, would be valuable.

Thank you for the suggestion. However, currently there are few 3D representation learning works that have been evaluated on the Objaverse dataset. Comparative methods such as Point-MAE, TAP, I2P-MAE, and ReCon have also not been tested on Objaverse. Although Point-GPT was pre-trained on the Objaverse dataset using ViT-B settings, its evaluation was conducted on ScanObjectNN and ModelNet40. To demonstrate the generality of our method, we pre-train our method on the Objaverse and provide the fine-tuning results in ScanObjectNN and ModelNet40 datasets.

Q6: Typos.

Thanks for pointing out those typos, we have corrected them in the revised version.