Exploring Vision Semantic Prompt for Efficient Point Cloud Understanding

We would like to thank the reviewer for the detailed review and valuable suggestions.

Mitigation of local and global ambiguity: We understand the reviewer's concern and appreciate the opportunity to clarify. During the classification of 100 airplane and chair samples, we recorded which view the 2D prompt selected by max pooling originated from, and counted the number of times each view was chosen.The statistics in the Table below are based on the ModelNet40 dataset.

We observed that when class tokens from three views are simultaneously fed into the network, the trained model tends to favor specific views for different categories—for instance, the top view for airplanes and front/side views for chairs. These preferred views typically exhibit the least local ambiguity. Besides, our approach enables the model to adaptively associate each object with its most informative view, thereby mitigating global ambiguity. We will include more detailed technical explanations in the revised version.

Comparison of inference time: We agree with the reviewer that inference time is an important metric. Just in case, we respectfully clarify that the primary goal of "efficienct" in parameter-efficient fine-tuning is to mitigate the storage overhead caused by the exponential growth of parameters in large foundation models. To make our work more comprehensive and complete, we will add a new Table for comparison of inference time on Modelnet40 based on your suggestions.

Performances gain on segmentation and few-shot classification: We understand the reviewer's concern regarding the few-shot and part segmentation performances. The ShapeNetPart and ModelNet40 (few-shot for PETL) are nearly saturated datasets, where multiple methods in recent years have achieved similar performance. Additionally, the ShapeNetPart contains some commonly recognized annotation errors, leading to discontinuous performance improvements.

Concerns regarding 2D pretrained models processing depth maps: We understand the reviewer's concern regarding the difficulty for 2D models to extract semantic information from depth maps. 2D foundation models are pretrained on large-scale image datasets and possess generalizable feature representations. In our method, we convert single-channel depth maps into three-channel pseudo-RGB images by repeating the depth values across channels. Besides, as shown in Figure 7, the proposed 3D-to-2D projection strategy mitigates issues such as texture loss and boundary blurring. According to our feasibility experiments, the pretrained 2D model can effectively extract semantic cues from the depth maps we constructed. In addition, prior works such as [PointCLIP: Point Cloud Understanding by CLIP, CVPR2022] and [Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders, CVPR2023]have explored the use of CLIP to extract semantic information from depth maps generated by projecting point clouds.

Effectiveness of 2D semantics for point cloud understanding: We acknowledge the reviewer's concern about whether 2D semantics can meaningfully enhance 3D point cloud representation. Currently, Transformer-based point cloud foundation models are pretrained on ShapeNet, a dataset composed of background-free, single-object point clouds. Besides, the depth maps used in our method are generated directly from the 3D point clouds through a globally consistent projection, ensuring a correspondence between pixels and 3D points.

Real-world applications: We understand the reviewer's concerns regarding our application scenario and appreciate the opportunity to provide further clarification. We agree that scene-level applications are an important future direction. Existing Transformer-based methods are mainly pretrained on ShapeNet, while large-scale scene-level models remain limited. For a fair comparison with previous PEFT methods, we validate our method on the same datasets. Our proposed method has promising applications in various tasks. For example, in robotic interaction, the first step involves cropping the target object followed by classification without background, and it is necessary to understand the semantic parts of the object for interaction.

We hope our responses have addressed your concerns and helped clarify any confusion. We truly welcome any additional questions or suggestions that could further improve our work.

We greatly appreciate the reviewer's thoughtful feedback and sincerely thank the reviewer for the positive recognition of our work.

Feature inconsistency affects downstream task performance：We appreciate the reviewer's question regarding the impact of feature inconsistency on downstream performance. There exists a mismatch between the representations emphasized by the pre-training task and those required by the downstream task. Pre-training often focuses on local structures and exhibits a strong positional correlation. For example, the wing features of an airplane are different before fine-tuning due to they are not adjacent in 3D space, even though their structures are similar. However, downstream tasks rely more on semantic features, which require modeling long-range dependencies — capturing the relationship between the two wings of an airplane.

Ablation on number of views and impact of viewpoints: We greatly appreciate the reviewer's attention to the impact of view count—this is indeed a very important aspect for analysis. We additionally conduct experiments using only the top view, six views, and nine views. The results show that increasing the number of views does not lead to performance improvement. In feasibility experiments, we observed that orthogonal views yield the best 2D classification results. It provides a balanced and comprehensive representation of the 3D structure while maintaining geometric fidelity. Therefore, redundant views are filtered out by max pooling and do not contribute to the final performance.

Regarding the question on different viewing directions, in datasets such as ModelNet40, objects are aligned orthogonally but their orientations are not consistent — there can be angular differences like 90 or 180 degrees among objects within the same category. As a result, the choice of viewing direction has minimal impact on the final performance. Additionally, we explored projecting 3D point clouds onto cylindrical and spherical surfaces. However, these projections introduced significant distortions, which degraded the quality of the rendered views and adversely affected 2D models' performances.

Confusing description in Table 1: We acknowledge that the calculation of the percentage of tunable parameters in Table 1 has caused confusion. The percentage listed for "Ours" refers to the model after incorporating the 2D model. We will provide a more detailed explanation in future versions to clarify this point. Thank you for your helpful suggestion!

Unintuitive ablation table: We agree that the original ablation may not be sufficiently intuitive. Therefore, we have revised the ablation study of the main modules and present the updated results in the following table:

We sincerely thank the reviewer for the positive recognition of our work. We truly value your comments and would appreciate any further questions or suggestions that can help us improve the paper.

We sincerely thank the reviewer's for the constructive comments and valuable suggestions.

Performance gain on segmentation: We understand the reviewer's concern and appreciate the opportunity to clarify. The ShapeNetPart is a nearly saturated dataset, where multiple methods have achieved similar performance in recent years. Additionally, the dataset contains some commonly recognized annotation errors, leading to discontinuous performance improvements. However, our method achieves better-fine-tuned performance than baseline (full fine-tuning) on the datasets with less trainable parameters.

Computational cost and practical applications: Thank you for raising this important point regarding the computational cost of our approach; we present the FLOPs of selected models in Table 8, and we will add a new Table for comparison of inference time on Modelnet40 in the future version. Besides, we will enrich Table 1 with the comparison of FLOPS.

Our proposed method has promising applications in various tasks. For example, in robotic interaction, typically, the first step involves cropping the target object, followed by classification. For interaction, it is necessary to understand the semantic parts of the object, i.e., part segmentation. The 2D pretrained models used in our work have been widely deployed in real-world scenarios, including Google's multimodal robotic system PALM-E [PALM-E: An Embodied Multimodal Language Model].

Ablation on the number of views: We greatly appreciate the reviewer's attention to the influence of view count—this is indeed an important aspect. We additionally conduct experiments using only the top view, six views, and nine views. The results show that increasing the number of views does not lead to performance improvement. In feasibility experiments, we observed that orthogonal views yield the best 2D classification results. Therefore, redundant views are filtered out by max pooling and do not contribute to the final performance. Using only a single view may be suboptimal, as the most informative viewpoint varies across object categories. For example, the top view provides the richest semantic information for airplanes, while the side or front view may be more informative for chairs. Therefore, providing three orthogonal views and allowing the network to adaptively select the most relevant one offers a more robust and effective solution.

During the classification of 100 airplane and chair samples, we recorded which view the 2D prompt selected by max pooling originated from, and counted the number of times each view was chosen. The statistics in the Table below are based on the ModelNet40 dataset.

Qualitative Description of the Effectiveness of 2D Semantic Prompt: Thank you for pointing out the need for a qualitative description. In the Discussion section of our paper, we visualize the features in our proposed HAA through PCA. As shown in Figure 4, with Semantic Transfer, we can transfer the 2D semantic cues to 3D features, enriching the association of the same components (wings of the airplane) and improving model performance in downstream tasks.

CLIP outperforms DINOv2: We agree with your point. In our feasibility experiments, frozen CLIP outperforms DINOv2 when classifying using depth maps obtained via our proposed projection. We attribute this to CLIP's strong zero-shot classification capability, and we believe this stems from its image-text alignment training.

Larger 3D foundation model: We utilize CLIP-L to enhance PointGPT-L [PointGPT: Auto-regressively Generative Pre-training from Point Clouds, NIPS2023] 3D model, and still observed consistent performance improvements on ModelNet40.

Citing PointCLIP： PointCLIP is one of the representative works in point cloud understanding that explores the integration of 2D semantic information into 3D point cloud tasks. We appreciate the reviewer's suggestion and will include a discussion of PointCLIP in the Related Work section.

We hope our clarifications have been helpful, and we would be glad to address any further questions you may have.

Thanks for the reviewer's careful reading and detailed comments. First, we respectfully clarify that all the pretrained models in our paper are trained on ShapeNet, these works aim to explore effective approaches for building 3D foundation models. ModelNet40, ScanObjectNN, and ShapeNetPart are used only in downstream tasks for fine-tuning. We will add this clarification to the Implementation Details section in the revised version.

Clarification on PCA and knowledge forgetting: We understand the reviewer's concern and appreciate the opportunity to clarify. PCA is used to illustrate the difference between pre-trained features and downstream task features, and it is unsuitable to explain knowledge forgetting. The goal of pretraining is to learn generalizable point cloud representations, while downstream tasks focus on task-specific features. Full fine-tuning adjusts all parameters to fit the downstream objective, which may lead the network to lose the general features acquired during pretraining. In contrast, parameter-efficient transfer learning (PETL) updates only a small subset of parameters while keeping most of the pretrained weights frozen. This allows the model to adapt to new tasks while preserving the knowledge learned during pretraining. [Overcoming catastrophic forgetting in neural networks, PNAS2017].

Confusing description of Table 9: We acknowledge that our explanation of Table 9 may have caused confusion. In future versions, we will provide a more detailed description of the cross-attention mechanism in Table 9, where T$_{mixed}$ serve as V and K, and T as Q. In Table below, we present experiments directly applying cross-attention between 2D prompts and 3D tokens. In PEFT, simply using cross-attention does not effectively fuse 2D information with 3D features. To address this, we design a simple ST module and leverage hybrid attention for effective feature fusion.

Marginal improvements over SOTA: Our method proposes a parameter-efficient transfer learning framework that can be applied to existing self-supervised representation learning methods. We respectfully note that a direct comparison with fully supervised methods or self-supervised point cloud representation learning methods may not be entirely fair, as those methods often involve significantly more tunable parameters and different architectural designs. Compared to the previous SOTA method (DAPT), our method achieves higher performance on the same pretrained models. Additionally, unlike full fine-tuning, our approach requires significantly fewer parameters for downstream adaptation.

Lack of computational analysis: We also consider the analysis of computational cost to be highly important. We have presented FLOPs of selected models in Table 8, and we will add a new Table for comparison of inference time on Modelnet40 based on the reviewer's suggestions. Besides, we will enrich Table 1 with the comparison of FLOPs.

Scene-level applications: We agree that scene-level applications are an important future direction. However, existing Transformer-based methods are mainly pre-trained on ShapeNet, while large-scale scene-level pretraining methods remain limited. For a fair comparison with previous PEFT methods, we validate our method on the same datasets. We also look forward to the development of Transformer-based scene-level pretraining methods. Besides, our proposed method has promising applications in real-world applications. For example, in robotic interaction, the first step involves cropping the target objects, followed by classification without background. Then, for interaction, it is necessary to understand the semantic parts of the object, i.e., part segmentation. The 2D pretrained models used in our work have been widely deployed in real-world scenarios, including Google's multi-modal robotic system PALM-E [PALM-E: An Embodied Multi-modal Language Model].

Comparisons with classical multi-modal fusion methods: PointPainting and EPNet are classical multi-modal fusion methods. They utilize 2D semantic information to enhance scene-level 3D object detection tasks. In contrast, our work focuses on parameter-efficient transfer learning for 3D foundation models. We will include a discussion in the Method section to highlight the key differences between our approach and these methods.

We hope our replies have resolved the reviewer's concerns and provided sufficient clarification. We sincerely welcome any further questions or feedback.