SPA: Enhancing 3D Multimodal LLMs with Mask-based Streamlining Preference Alignment

We sincerely thank Reviewer 94BN for their thorough review and valuable feedback. We believe the proposed changes, such as restructuring the introduction and related work, clarifying the method descriptions, and improving the presentation of figures and results, will significantly enhance the clarity and impact of our paper. We look forward to submitting a revised version that addresses these concerns in greater detail.

Weaknesses:

1. Writing and organization could be improved:

   Thank you for your feedback. We agree that the writing and organization can be improved to make the paper clearer and more accessible to the readers. To address this, we plan to restructure the introduction and related work sections to create a more logical flow, ensuring the key contributions are highlighted at the beginning. Additionally, we will revise some of the terminology and rephrase complex explanations to ensure clarity. We'll make sure these changes are updated at the end of this discussion, but for now we want to focus on other areas.

Questions:

1. The abstract should start by explaining what the addition of 3D features into LLMs actually achieves:

   Thank you for this suggestion. We will revise the abstract to clearly state the primary contribution of adding 3D features into LLMs, specifically how it enhances their ability to reason about and manipulate 3D representations. We will emphasize the improvements in understanding the world, extracting and editing 3D data, and the related performance gains. After presenting the overall achievements, we will discuss the challenges related to model size and the limitations of increased training and enhanced 3D encoders, as you suggested. Also, we'll make sure these changes are updated at the end of this discussion, but for now we want to focus on other areas.

2. Related work should follow the introduction and precede the proposed method:

   We appreciate your observation regarding the organization of the paper. We will move the related work section to directly follow the introduction, as per your suggestion, to provide proper context for the proposed method. This will help readers better understand the background before diving into the specifics of our approach. Also, we'll make sure these changes are updated at the end of this discussion, but for now we want to focus on other areas.

3. Unclear explanation of "traditional two-stage posttraining methods":

   Thank you for pointing this out. We will clarify the term "traditional two-stage post-training method" in the revised version. Specifically, we will define these stages more precisely, explaining that they usually refer to the post-training process, which requires the first stage to complete the data construction, generate preference text data, and complete the SFT training of the reference model, and the second step to perform DPO-like policy preference learning based on the generated data. (Changes are shown in blue text)

4. Rephrase "Preliminary" to "Preliminaries":

   We appreciate this suggestion. We have changed this one in latest version.

The papers you mentioned aligns with my argument regarding the relationship between CLIP and LLaVA. Clearly, in terms of conversational abilities for 3D understanding, these pre-trained models exhibit significant limitations in generalization and dialogue capabilities. Our model supports a maximum of 1024 tokens of text input, while the models you mentioned are only effective with inputs under 100 tokens, and they struggle with understanding more human-like conversational patterns, such as vague instructions. I believe that in the 2D domain, we need both GPT-4-V and CLIP, and this is consistent in the field of 3D understanding as well.

Thank you for your review. I would like to clarify that our method does not require directly generating additional QA pairs. As stated in the training objective outlined in Equation (5), we compute the probability difference between the normal point cloud input and the noisy point cloud input under the ground truth labels. In other words, our approach still constructs the training process based on contrastive learning from the SFT training data, without the need for extra new training data pairs.

Additionally, your point about the broader applications of works such as 3D-LLM is indeed valid. However, we also considered that our method focuses on improving existing approaches like 3D-LLM and PointLLM, which are LLMs base model for 3D understanding. Specifically, our focus is on enhancing their semantic understanding and conversation capabilities. The transformation of downstream tasks such as grounding and navigation, as you mentioned, is not in conflict with our work. On the contrary, it is facilitated by a stronger and more scalable 3D understanding backbone or base model in LLMs, which is essential for advancing future research.

Therefore, I believe it is important to reemphasize the focus of our research. We are not primarily targeting downstream tasks but rather focusing on the improvement of the 3D understanding backbone in LLMs. Our algorithm addresses the performance bottleneck in current 3D understanding models, which is the core contribution of our work.

Just like in the 2D domain with models like LLaVA [1], testing and evaluation typically focus on tasks such as VQA (Visual Question Answering), which indirectly reflect the model's semantic understanding and conversational abilities. There is usually no direct evaluation based on downstream task benchmarks, which is a conventional approach. Therefore, we aim to clarify our contributions clearly. At the same time, in image understanding, purely evaluating classification and localization capabilities may show that LLaVA is not as strong as pre-trained models like CLIP after fine-tuning. However, with a large language model as the backbone, its generalization ability and potential for improvement are extremely strong. This is why these related studies are meaningful, such as PointLLM. Our goal is to address the gaps and shortcomings of VQA MLLMs like LLaVA in 3D understanding.

[1] Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2024). Visual instruction tuning. Advances in neural information processing systems, 36.

Similarly, for many CAD application scenarios (which are also of significant importance), traditional point cloud understanding methods are no longer effective, and it is essential to introduce LLM-based approaches [1,2,3,4].

[1] Yuan, H., Xu, J., Pan, H., Bousseau, A., Mitra, N. J., & Li, C. (2024). CADTalk: An Algorithm and Benchmark for Semantic Commenting of CAD Programs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3753-3762).

[2] Khan, M. S., Sinha, S., Sheikh, T. U., Stricker, D., Ali, S. A., & Afzal, M. Z. (2024). Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Prompts. arXiv preprint arXiv:2409.17106.

[3] Xu, J., Wang, C., Zhao, Z., Liu, W., Ma, Y., & Gao, S. (2024). CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM. arXiv preprint arXiv:2411.04954.

[4] Zhang, Z., Sun, S., Wang, W., Cai, D., & Bian, J. (2024). FlexCAD: Unified and versatile controllable CAD generation with fine-tuned large language models. arXiv. https://arxiv.org/abs/2411.05823

We are glad to receive your constructive comments and questions, which will help us to further improve the manuscript.

Weaknesses:

1. SPA's generalizability beyond specific 3D datasets and tasks is unclear:

   Thank you for raising the point about the generalizability of SPA. While the experiments in this paper primarily focus on the 3DCQA dataset, we recognize the importance of validating SPA’s performance across different 3D data types and environments. In fact, additional experiments using cross-domain datasets (such as complex real-world scenes and various 3D data from practical applications) will be a focus of our future work (similar to the relationship between LLaVA and LLaVA-Plus), as these tasks are more engineering-driven. Our primary focus has been on improving the underlying algorithms to address the performance bottlenecks of existing 3D LLMs/MLLMs base models, which is the main contribution of our work.

2. The primary contribution, SPA, builds largely on established methods rather than introducing fundamentally new ideas:

   We acknowledge that some of SPA’s ideas are built on mature technologies. However, the contribution of improving existing post-training methods to adapt to 3D data is undeniable. Specifically, transitioning from the high-resource consumption required to generate additional pair post-training data to the current approach of constructing stable pair inputs on SFT data using contrastive learning is a highly valuable advancement. Moreover, we empirically demonstrate the close connection between this approach and NCE methods, which is well-supported by solid theoretical foundations.

Questions:

1. Conduct more granular ablations, particularly on individual components of SPA:

   We greatly appreciate your suggestion for more detailed ablation studies. In the current work, we have already conducted some ablation experiments on various components of SPA, such as the post-training scheme (Table 4) and the method for constructing negative examples (Table 2). The results of these experiments are clearly analyzed in the main text, and we hope you can review them further and continue to provide valuable suggestions!

2. Add experiments on datasets outside the 3DCQA benchmark:

   We fully agree that while the 3DCQA dataset is valuable, it has limitations in representing all types of real-world 3D data. Therefore, as mentioned in our response to W2, we would like to reiterate the focus of our current work. We hope that this embodied system can be adopted by more researchers in future work for real-world interactions, although this is not the primary motivation or contribution of our current method.

If you have any further suggestions, please feel free to let us know. If you feel that we have addressed your concerns, you may also consider revising the score. We look forward to further discussions.

Dear Reviewer ozMT,

We have noticed that you have changed your score from 6 to 3 without any respones or reasons, we are trying to follow your suggestions and active supplements in experiments. Thus, would mind to give us some feedback about why you want to change your score?

Thank you for reaching out regarding the score change. I appreciate your efforts in actively supplementing the experiments and following my suggestions. The adjustment in the score was primarily due to the lack of updates or responses for an extended period, which left some concerns unresolved on my end.That said, I am open to reconsidering my score if the necessary experimental results or relevant data can be provided to address the points I raised. Additionally, I noticed that the methodology of this work appears to share similarities with Self-Taught Evaluators. Could you kindly clarify the significant differences between the two approaches? A detailed explanation would be greatly appreciated, as it would help me better understand the unique contributions and advantages of your work compared to Self-Taught Evaluators.