From Language to 3D Worlds: Adapting Language Models for Point Cloud Perception

First of all, we express our gratitude for your comprehensive review and the valuable insights you have provided. Your feedback has prompted a closer examination of our work, and we have updated our manuscript to better reflect the novelty and impact of our approach.

• Originality and Novel Contributions:

We thank the reviewer for the constructive suggestion, and we have discussed the difference between these parameter-efficient methods including LLaMa AdapterV2 and Point-Bind & Point-LLM (Sec 2.3). Furthermore, in response to your concerns about the originality of our work, we have elaborated on the distinct aspects of LAMP in Section 3.2, emphasizing its unique contributions beyond attention-based adaptation. Our approach introduces a new alignment strategy for the data distribution of 3D point clouds with language models, which is distinct from the methodologies proposed in the studies you mentioned. This strategy demonstrates a different take on how pretrained language models can be effectively applied to 3D point cloud perception tasks, especially in long-tailed and out-of-distribution problems, which has not been addressed by LLaMA-Adapter V2 or Point-Bind & Point-LLM. Meanwhile, our method achieves outstanding performance jointly learning point cloud and natural language in the 3D visual grounding. These unique contributions distinguish our method from others.

• Methodological Distinction:

We have included a detailed comparison of our methodology with related works in Section 2.3. Here, we describe the unique aspects of our cross-modal self-attention learning and cross-modal cross-attention learning strategies. Unlike the attention mechanisms used in previous studies, our approach employs a novel projection network designed to harmonize point cloud embeddings with the parameters of language models, thus enabling more effective feature extraction and modality bridging.

• Evaluation on Realistic Indoor 3D Datasets:

As suggested, we conducted experiments on the Shapenet-Core55, PartNet, Objaverse-LVIS, and Scan200 datasets (New Table 7 & Sec.4.5). We hope that these experimental results can address your concerns.

In conclusion, we hope that the revisions and additions to our manuscript, particularly in Sections 4.4 and 4.5, address your concerns regarding the originality and applicability of our work. We believe these enhancements underscore the novelty of LAMP and its contributions to the field of 3D point cloud perception.

We look forward to your further comments and suggestions.

Thank you for your thoughtful and detailed review of our paper. We appreciate the time you spent analyzing our work and have made careful revisions to address the concerns and questions you raised. Below is our response to each point.

1. Performance Comparison with Point-MLP Elite:

We acknowledge your concern regarding the performance comparison with Point-MLP elite, especially as mentioned in Table 2. To address this, we have expanded our discussion in Section 4.1 and included additional analyses in Table 3 and Table 5, which demonstrate that LAMP outperforms advanced models like Point-MLP in terms of mean IoU scores in the dense prediction tasks of semantic segmentation and part segmentation. Meanwhile, LAMP also outperforms Point-MLP, particularly in long-tailed classes and out-of-domain recognition tasks​​ (Table 6 and Table 7). We add a detailed discussion between LAMP and Point-MLP in Section 4.

2. Evaluation on More Challenging Datasets:

We agree that evaluating LAMP on more challenging datasets would strengthen our paper. In response, we have included the results of ShapeNetCore, PartNet, and challenging Objaverse-LVIS, Scan200 datasets are as follows (New Table 7 in the Manuscript):

We hope that the experimental results on the newly added 4 benchmarks will address your concerns.

3. Comparison with Lightweight Models like RepSurf:

We acknowledge the great contribution of RepSurf and are willing to perform a comparison between RepSurf (as detailed in Page 18 Table 10). We compare LAMP with the method as follows, meanwhile, we have further discussed the difference in learning focus between LAMP and RepSurf in the manuscript.

4. Impact of Increasing Trainable Parameters:

Your question about the impact of increasing trainable parameters is insightful. In response, we have conducted additional experiments to explore this aspect. For multimodal tasks, the trainable language model backbone can significantly improve performance while increasing trainable parameters (as detailed in Sec 4.6).

5. Typographical Error in Section 3.3:

Thank you for pointing out the potential typo in Section 3.3. Upon review, we have corrected the notation from $W^{T}_q$ to $W^{L}_q$, aligning it with our intended representation.

6. Clarification on Inference Time in Table 8:

We appreciate your feedback for clarity on the 'Infer.(s)' column in Table 8. We have revised this table to present the inference time in milliseconds per sample 'Infer.(ms)', providing a more intuitive and direct comparison of the models' efficiency.

We hope these revisions and additional analyses adequately address the concerns you have raised. We are grateful for your feedback, which has significantly contributed to improving our work.

We greatly appreciate the time and effort you have dedicated to reviewing our paper. We have carefully considered your insightful feedback and have made substantial revisions to our manuscript. Please find below our responses, including references to the relevant sections in the revised paper.

1. Multimodal Tasks and Experiment Focus:

We thank you for your constructive comment on the focus of our experiments. We have expanded our experimental section to include more multimodal tasks, particularly those involving text-3D tasks and 3D visual grounding. As detailed in Section 2, LAMP demonstrates its effectiveness and versatility on both unimodal and cross-modal tasks, validating the approach’s superiority over traditional methods​​. Besides outstanding experimental results on point cloud analysis, LAMP also delivers a new state-of-the-art performance on 3D visual grounding task.

2. Suitability of LLM+Adapter for 3D Point Cloud Analysis:

We acknowledge your concerns about the use of LLM+adapter in traditional 3D point cloud analysis. We have included a comprehensive discussion in the manuscript (Table 1) to address this issue. We discuss the impact of the language model to understand the point cloud. It also reveals that a plain language model can read point clouds. Furthermore, it is more about the multimodal nature between nature language learning and point cloud analysis. Therefore, we propose a new approach for this task. Specifically, in Section 3, we describe how LAMP leverages language-pretrained models to transfer knowledge to 3D point clouds, exploring an under-utilized modality - language - for 3D tasks​​.

3. Concerns Over Model Overfitting and Dataset Size:

We have conducted additional experiments with larger datasets to address the concern of overfitting. Our results, which are now detailed in Section 4, indicate that LAMP achieves outstanding performance without overfitting, outperforming other competitive methods on the large-scale S3DIS dataset​​. Moreover, we demonstrate the robustness of LAMP in relieving the long-tail problem and out-of-distribution problems with substantial improvements over existing models as shown in Table 3 and Table 4​​.

Moreover, we are willing to conduct more experimental results on additional datasets including ShapeNetCore, PartNet, and challenging Scan200 datasets as follows:

4. Paper Writing and Clarity:

We have made significant revisions to the paper to improve clarity and presentation. The revised sections clearly articulate the methodology and experimental results, ensuring that the paper conveys our contributions effectively.

Questions Regarding the Paradigm of LLM+Adapter:

We have elaborated on the paradigm of LLM+adapter in our revised manuscript. In Section 5, we discuss how our architecture extends LAMP to multimodal scenarios, such as 3D visual grounding, demonstrating the framework's ability to unify encoding across different input modalities​​. This is further evidenced by the new state-of-the-art performance on the ScanRefer dataset​​. We believe that LAMP's approach to modality divergence in vision-language tasks opens a promising direction for future research.

We trust that these revisions address the concerns you have raised and strengthen the paper. We value your feedback, which has been instrumental in enhancing our work. Thank you for your consideration.