MV3D-MAE: 2D Pre-trained MAEs are Effective 3D Representation Learners

Thank you for your thoughtful review of our submission. Your feedback is invaluable, and we are grateful for your constructive comments, here is the answer to your questions:

For Comment 1: Italicized mathematical symbols and more formulas for 2D-3D and 3D-2D.

Thanks for your suggestions! We will soon update the notation to use uppercase Roman symbols in the paper. Additionally, we will clarify the projection process more explicitly using equations.

For Comment 2: More experiment results.

In line with the previous reviewers, we have included comparisons with ACT, Point-JEPA, and PointGPT. We also extended our experiments by testing three ScanObjectNN experiments based on the original setup.

Table 6. More performance comparison of all additional.

We also tested three ScanObjectNN experiments based on the existing framework. The results on ScanObjectNN OBJ-B, OBJ-ONLY, and PB-T50-RS are as follows:

Table7. More performance comparison on ScanObjectNN.

As shown in the table, I2P-MAE achieves the best performance across all tasks. However, our method consistently delivers competitive results, surpassing all other methods, including ACT, Point-MAE, Joint-MAE, and Point-M2AE. The performance of our model is very close to I2P-MAE, and while we do not outperform I2P-MAE, our method still achieves remarkable results.

For Comment 3: How λ influences the performance.

To test how λ influences the performance of the pre-training. We do more experiments in training. We use the CD (Chamfer Distance) metric to compare the 3D reconstruction results. The results are shown in the table below:

Table 8. Performance on Reconstruction of Point Cloud with Different lambda.

When we set the value of λ to 0, 3D reconstruction does not occur. However, when we set λ to 25, the best metrics are achieved for reconstruction. As we increase the weight of λ, the performance of the 2D reconstruction model deteriorates, which in turn leads to a decline in 3D reconstruction results.
And the Table 8 proves that the performance of 3D reconstruction impacts the MAE component's effectiveness in downstream 3D tasks. This further highlights the importance of the learning process of the 3D reconstruction module in the MV3D-MAE model.

For Question 1: Can we introduce RGB image to the pipeline?

We really appreciate your question, as it aligns with one of our main concerns. After completing this work, we conducted further discussions regarding this issue.
If you wish to adopt the same paradigm as our network—first projecting 3D into 2D, performing MAE-related operations on the 2D images, and finally fusing the 2D images back into a 3D model—this approach is not limited to 3D point clouds and could be applied to other formats.
During this work, we were unable to find a suitable method to effectively merge multiple RGB images into a cohesive 3D object. However, we have recently come across methods, such as 3D Reconstruction with Spatial Memory, that could potentially achieve this.
We are currently conducting related research in this area and hope to achieve even better results in the future.

For Question 2: how the maxpooling work in the Mv-Swin?

Thank you for this question! We adopted the max-pooling strategy to extract the most salient feature for each object. In our approach, the input consists of multi-view images with 10 viewpoints per object. Therefore, the number of samples in each batch is always a multiple of 10. For instance, 40 depth images represent 10 views of 4 distinct objects. When extracting features for each object, we first compute the most salient feature across the 10 views and replicate it 10 times to maintain the batch size. This replicated feature is then used as the query. While the replication may appear as repeating the same feature 10 times (with no change in color), the final feature is obtained by applying max-pooling across the views. Not sure if we misunderstood your point, but we believe there are no errors in the colors in Figure 3.

Thank you for your thoughtful review of our submission. We appreciate your recognition of our pre-training method for learning effective representations from point clouds and the validation of our approach using 2D pre-trained models. Your feedback is invaluable, and we are grateful for your constructive comments, here is the answer to your questions:

For comment1: Why we not fine-tuning on ShapeNet55-34 and long-tail data.

Our pretraining experiments were conducted on ShapeNet55-34, so we may not conduct further fine-tuning on this dataset. However, our segmentation task is carried out on the ShapeNetPart dataset, as shown in Table 3.

There is limited research on long-tail data in point cloud datasets. After analyzing the existing data, we found it difficult to construct a long-tail dataset that meets the conditions for training or fine-tuning. We appreciate the reviewer's suggestion and will consider further research on this topic in the future.

For comment2: More comparison with Point-JEPA and PointGPT-S.

Compared with Point-JEPA:

We compared the performance of our model with Point-JEPA and PointGPT, as shown below:

• On the ModelNet40 classification task:
  • Our model achieved 94.1 accuracy without voting, compared to Point-JEPA's 93.8 (without voting).
• On the ScanObjectNN classification task:
  • Point-JEPA achieved approximately 86.6 ± 0.3 accuracy (as reported on its GitHub page), which is lower than our model's performance of 89.85.
• On the segmentation task:
  • Our segmentation performance reached 85.5, slightly lower than Point-JEPA's 85.8 ± 0.1.
  • This slight decrease is expected, as our method requires projecting 2D multi-view features back into 3D space in the latent space for segmentation, introducing additional complexity.

Compared with PointGPT-S:

We compared our model with PointGPT-S, which has the closest parameter count to our model:

• On ModelNet40:
  • PointGPT-S achieved 94.0 accuracy, close to our model's performance.
• On ScanObjectNN:
  • PointGPT-S achieved 90.0 accuracy, comparable to our result.
• On the segmentation task:
  • PointGPT-S only achieved 84.1, which is significantly lower than our model's 85.5, demonstrating our superior performance in this task.

Table5. More performance comparison.

For comment4: Failure analysis

Analysis of Classification Errors for the Top 5 Categories Table4. Misclassification Analysis Table

In our analysis of the 2,468 test objects, with an overall classification accuracy of 94.1%, a total of 145 objects were misclassified. Among these errors, the Flower Pot category consistently exhibited the highest misclassification rate. And also the model is hard to classify the Night Stand and dressers. So we illustrate the figures of Flower Pot and Plant, the Night Stand and Dresser in the Figure2 link

Error Analysis and Observations

The Flower Pot category frequently suffers from misclassification, often being confused with Plant, Vase, or Cup. This misclassification stems from the inherent similarity between Flower Pot and Plant, as both typically consist of two main parts: a vase or pot and a plant. Figure 2 in the referenced link highlights this issue, showing that many objects in the Flower Pot category share indistinguishable features with objects in the Plant category, making it challenging for the model to classify accurately. Moreover, ambiguities in the dataset's labeling process further exacerbate this issue, as the distinction between Flower Pot and Plant categories is poorly defined.

A similar challenge arises between the Night Stand and Dresser categories. These two categories share significant visual overlap, and size often serves as a critical factor in human judgment for distinguishing them—dressers are generally larger than nightstands. However, the preprocessing step in the dataset normalizes object sizes by resizing them to a uniform scale, which removes this crucial contextual information. Consequently, the model struggles to differentiate between these two categories due to the loss of this distinguishing factor.

Conclusions and Thinking Solutions

From the above analysis, we conclude that existing methods face significant challenges when classifying objects in point cloud datasets without color information. Categories like Flower Pot vs. Plant, Night Stand vs. Dresser, and Chair vs. Stool are highly similar, and in some cases, objects are virtually identical but classified into different categories. This ambiguity in the dataset undermines classification accuracy. Furthermore, the dataset's resizing process eliminates size-related distinctions, which are particularly critical for differentiating categories like Dresser and Night Stand. To address these challenges, we suggest two potential solutions. First, incorporating color information into point clouds could significantly improve classification performance, as color often provides critical cues for distinguishing between similar categories. For instance, green hues could help identify plants, while unique patterns or colors could differentiate flower pots from vases. Second, leveraging Vision-Language Models (VLMs) could further enhance classification accuracy by integrating visual features with semantic textual descriptions of objects. By combining color information with VLMs, models can better generalize to ambiguous cases and improve classification in real-world settings.

Thank you for your thoughtful review of our submission. Your feedback is invaluable, and we are grateful for your constructive comments, here is the answer to your questions:

For Comment1: Why do we need to reconstruct the point cloud and the revision of “no pose”.

Thank you for your comment. The design of a novel Mv2D depth image-to-3D point cloud structure was carefully considered. The mapping process of converting a 3D point cloud object into a smooth point cloud structure is irreversible, as it involves not only projection but also steps like densification and smoothing (see Appendix A.2). Using a simple multi-view 2D-to-3D projection and merging process would make it difficult for depth images reconstructed through MAE to form a coherent 3D object. This would undermine the supervision of the 3D loss and impair the model’s capability to learn the relationship between 2D and 3D structures. Hence, our 2D-to-3D projection module is essential.

Regarding "no pose information requirement," we used fixed parameters for stable 3D synthesis and applied random rotations during pretraining to improve robustness. We acknowledge your critique and will revise our wording to clarify: "Our model can perform 2D-to-3D mapping for depth images captured under fixed relative positions, without requiring explicit pose conditioning." This will be updated in the paper.

For Comment2: Parameter amount and FLOPs.

In the pre-training stage, we mainly train the parameters in GroupAttn and the Add layers in MAE, with a total parameter count of 28.35 million. However, when using the MV-Swin encoder-decoder for 3D reconstruction, the model's parameter count reaches 46.25 million, which means that the 2D-3D model have 17.9 million parameters.

Point-MAE and Point-BERT have a parameter count of 22.1 million. These models do not involve 2D-3D conversion and mapping processes, resulting in a smaller parameter size.

For downstream tasks, we do not need to use the MV-Swin structure for 3D reconstruction, so the speed is relatively fast. However, the segmentation head requires mapping dense features to point clouds. Although this part has only 11 million parameters, the parameter-free 2D image to 3D point cloud mapping process still takes a lot of time, causing the segmentation head to still require 90.8ms. The training time cost for each part in different stages can be referred in Table 3:

Table 3. Training time cost for each part.

As the comparison, the Point-MAE and Point-BERT training pipeline cost ~94ms and ~106ms. The classification head cost ~1ms, segmentation head cost ~86ms. Our model’s MAE module has an efficient throughput time.

For comment3: The analysis of partial point cloud experiments.

Based on Table 2 in the review, we further analyze why our performance surpasses that of Point-MAE and I2P-MAE on incomplete datasets.

First, we examine the data patterns in the PCN dataset. Since this dataset originates from the real world and is obtained using depth scanners, the missing patterns in the data are often distinguishable to the human eye.

As shown in the Figure1 in the link, when our model has access to three or more visible viewpoints, its performance remains consistently reliable. This demonstrates that, after fine-tuning, our model can classify objects based on certain visible viewpoints of an incomplete object.

For the ModelNet40-Crop dataset, which contains incomplete data, some viewpoints remain visible. After fine-tuning, our model achieves good performance. However, without fine-tuning, none of the three methods can deliver usable performance.

Therefore, we believe that all existing methods struggle to directly generalize to incomplete and noisy point cloud datasets. To better adapt to incomplete point clouds in real-world scenarios, in addition to direct data collection, we are particularly interested in learning a pattern of incomplete point clouds from real-world data. This pattern could be used to degrade complete point clouds into incomplete ones, thereby aiding point cloud classification algorithms in better generalizing to real-world environments.

Thank you for your detailed review and valuable feedback on our paper. We highly value your constructive comments:

For comment1: Why we use 2D pre-trained MAE for 3D tasks?

Our goal is to leverage 2D pre-trained models to enhance the performance of 3D tasks. Acquiring 3D objects directly is often complex and challenging, whereas obtaining uniformly formatted 2D images is significantly more convenient. Therefore, exploring how 2D models can assist 3D models in completing tasks more effectively is a highly valuable research direction.

In our experiments, the pre-trained 2D MAE has already learned extensive prior knowledge about images, including object edges and texture information. The features learned by the 2D MAE also help our model capture the textures and object features in depth images.

As shown in Table 5 of the ablation study, there is a significant performance gap between the model without the pre-trained 2D MAE and the one with it. This highlights the effectiveness and necessity of using 2D pre-trained models in 3D tasks. Moreover, the use of pre-trained 2D MAE is scalable. Although we achieved the current performance using only a ViT-base model, due to resource limitations, we have not yet tested larger MAE models. We plan to further extend our work in this direction.

For comment2，Compare with I2P-MAE and ACT.

Our method indeed has some performance gaps compared to I2P-MAE on ScanObjectNN, but it achieves superior performance on ModelNet40 with an accuracy of 94.1 compared to I2P-MAE's 93.7. Moreover, our proposed method introduces a distinct 2D-3D training paradigm different from I2P-MAE, allowing for better utilization of 2D MAE capabilities and offering stronger scalability. Regarding the ACT method, our classification model outperforms ACT, which achieves 93.50 on ModelNet40. Furthermore, compared to ACT's 89.16 performance on ScanObjectNN without data augmentation, our model demonstrates an advantage with an accuracy of 89.85.

Table 1. More comparison on downstream tasks.

For comment3，Further comparison with I2P-MAE on partial point cloud dataset.

In addition to using Point-MAE, we have also included I2P-MAE for further comparison on the partial point cloud dataset. The performance comparison is as follows:

Table2. More comparison of performance on Incomplete Datasets.

This demonstrates that our model achieves performance comparable to I2P-MAE on the PCN dataset. Furthermore, after fine-tuning on the ModelNet40-Crop dataset, our model outperforms I2P-MAE. However, when using a generalized model, the performance of our model drops to an unusable state, with accuracy falling below 70%.

For Question: How we fine-tuned and what components are retained?

Thanks for your questions！

The specific fine-tuning architectures for classification and segmentation are detailed in the final section of the appendix. We further explain how we perform fine-tuning during the process as follows:

During fine-tuning, we retain the 3D-to-2D projection module and the 2D encoder module of MV3D-MAE, while the 2D decoder module and the 2D-to-3D reconstruction module are discarded. All parameters are trainable during the classification and segmentation processes. The descriptions of the classification and segmentation heads have shown in the Appendix A.7.