Towards 3D Molecule-Text Interpretation in Language Models

Thank you for your time and constructive feedback! To address your concerns, we present the point-to-point responses as follows. We have carefully revised our paper, taking all your feedback into account.

Q1. Failure case study. The paper misses to include some failure cases (e.g., hallucination) of the proposed method. For instance, in Table 3(b), it seems the proposed 3D-MoLM wrongly interprets Globostellatic acid B (C34H48O6 while the ground truth is C33H48O7); the paper needs to discuss when the model tends to predict the wrong results.

Response: Thank you for your insightful suggestion. We agree that discussing failure cases is crucial for a comprehensive understanding of our model's performance. In response, we have revised our submission to include two failure case studies in Appendix D. Specifically, we show two mistakes of 3D-MoLM: 1) it confuses between Globostellatic acid C and Globostellatic acid B; 2) it fails to generate a molecule's correct chemical formula. We attribute these two mistakes to the following limitations:

1. The discernment of 3D-MoLM at a more refined granularity requires further enhancement.
2. Existing language models, including GPT-4, are not yet fully capable of accurately counting the number of atoms and providing the correct chemical formula.

These cases underscore areas where 3D-MoLM can be improved. Thank you again for your constructive critique, which has significantly contributed to enhancing the quality of our paper.

Q2: Performance of 3D-MoLM using smaller base LMs. I think the comparison in Table 3 might be somewhat unfair due to the different model sizes used for evaluation. Specifically, MolT5-Large has 800M parameters, while 3D-MoLM mainly uses LLaMA-7B models. In this respect, I suspect the improvements might simply come from the simple usage of language models with a large number of parameters and large training data rather than considering the 3D geometry of molecules. A good supporting experiment can be conducted by using a smaller LM for experiments (e.g., FLAN-T5-large).

Response: Thank you for your insightful comments. We understand your concern about the potential bias in the comparison due to different model sizes. To address this, we have included additional experiments by replacing Llama2 with MolT5 in Appendix F.

Specifically, we have transplanted the 3D molecule encoder and the projector onto MolT5. This allows us to make a fair comparison with the same base LM. As shown in the table below, MolT5s with 3D perception consistently outperforms those without 3D perception across all scales. This result substantiates our claim that the incorporation of 3D geometry perception enhances the molecular understanding of LM.

Table 1: Comparison between 3D-MoLM and MolT5 for molecule captioning. X: without 3D perception. Y: with 3D perception.

We hope this additional experiment adequately addresses your concern and further underscores the value of considering 3D geometry of molecules in our model.

We sincerely appreciate your constructive and thorough comments. Your suggestions, especially the ones about including a generalist model and clarifying training stages, have significantly improved our presentation. To address your concerns, we provide responses as follows. If you have additional concerns, we would be pleased to discuss them with you.

Q1: Novelties and contributions. The novelty of this work is actually hard to say enough. It is clear that the framework is almost the same as BLIP2, the pre-training tasks and the fine-tuning stages are also the same.

Response: We appreciate your comments, but wish to respectfully emphasize our novelties and contributions. Our adaptation of Q-Former to 3D molecule-to-text generation tasks is nontrivial, offering a new dimension to 3D molecule understanding.

Further, we have made significant modifications to BLIP-2 for adaptation to the molecule domain, and devise a dataset for 3D-molecule centric instruction tuning. Our novelties and contributions, compared to BLIP-2, are summarized below:

• First 3D molecule-text modeling method. 3D-MoLM is, to our knowledge, the first method to explore 3D molecule-text modeling. We believe 3D-MoLM advances the field by enabling text-based understanding of 3D molecules.
• Multi-task Generalist Model through Instruction-tuning. Unlike BLIP-2, 3D-MoLM includes a multi-modal instruction-tuning stage. This instruction tuning phase aims to align 3D-MoLM to human preference, and develop a generalist model that can perform various downsteam tasks. The performance of the generalist 3D-MoLM are reported in our updated Table 3 and Table 4. See our response to your Q4 for more details.
• Instruction-tuning Dataset. We present 3D-MoIT for 3D molecule-text instruction tuning. 3D-MoIT is, to our knowledge, the first 3D molecule-centric instruction tuning dataset. We also propose the GPT-3.5 enriched version of PubChem for 3D molecule-text alignment. We believe these datasets can facilitate further research and development in this area.

We hope this clarifies the unique contributions of our work and differentiate it from existing models like BLIP2.

Q2. The first question is about the Q-former (this may be the same as the Q-former in BLIP2, but the process is not clear here). From Figure 2, the caption task and the contrastive task will use the text branch for text processing. These tasks are only used in pre-training, hence the question is that in the inference stage, the text branch in Q-former will be ignored (if I understand correctly). Therefore, the pre-training and the fine-tuning are mismatched, since we will only use llama2 for text generation, instead of the text branch in Q-former. What's the performance if we remove these two pertaining tasks and also the text branch in Q-former?

Thank you for the valuable question and comment. We have divided this question into three sub-questions and prepared answers separately:

Q2.1. Is Q-Former's text-branch ignored in inference?

Response: Q-Former's text-branch is crucial in the inference of molecule-text retrieval, alghough it is not used in molecule-to-text generation. Specifically, Q-Former's text branch is used in both molecule-text contrasting (MTC) and molecule-text matching (MTM) -- the two essential components for molecule-text retrieval [1]. In MTC, the text branch generates text embeddings to calculate cosine similarities for contrasting; In MTM, the text branch processes text tokens, from which the query tokens obtain text information. Therefore, removing the text-branch will impede the molecule-text retrieval functionality.

On the other hand, removing the text branch will not affect molecule-to-text generation inference. However, the performance of molecule-to-text generation can be hurted, which is explained in the response to Q2.3.

Q2.2. The text generation task uses Q-Former's text branch, which is ignored during the inference of text generation. Therefore, the pre-training and the fine-tuning are mismatched, since we will only use Llama2 for text generation, instead of Q-Former's text branch.

Response: Thank you for kindly identifying the ambiguity! We would like to clarify that the pretrainnig (using Llama2 for text generation in stage 2 and 3) and the inference (using Llama2 for generation after stage 3) is aligned, instead of mismatched. Our original Figure 2 may give the wrong impression that only the Q-Former's text branch is used for text generation. This issue is fixed in our revised Figure 2. The correct training strategy is: Q-Former's text branch is trained for text generation in stage 1, and Llama2 is trained for text generation in stage 2 and 3. Therefore, Llama2 is fully optimzed for generation before inference, and the pretraining and inference is aligned.

Q2.3. How will 3D-MoLM perform if we remove the text-branch and the objectives of molecule-text contrasting and molecule captioning in training stage 1?

Response: It is important to note that removing the text-branch eliminates all three objectives in training stage 1: the molecule-text mathcing objective is eliminated because it relies on the text-branch to process text tokens [1]. Therefore, removing the text-branch means to remove training stage 1 and use randomly initialized Q-Former for stage 2. Then, the question boils down to stage 1's influence to text generation performance, which is shown below:

Table 1: Molecule captionging performance on PubChem dataset.

We observe that including stage 1 pretraining significantly improves molecule captioning performances. This is because stage 1 pretraining acts as a "warmup" training before stage 2 and 3 [1]. It prepares the Q-Former's ability of extracting molecule features that are the most relevant to the texts, which is achieved by the three multi-modal objectives in stage 1.

Thank you for your valuable comments and suggestions on our submission. Your suggestions to 1) clarify the novelties and contributions of this work, 2) illustrate and evaluate the functionality of Q-Former's text branch, 3) clarify the training objective functions of the three training stages, and 4) include a generalist model for generations tasks of molecule captioning and open-text QA, have helped us to substantially improve the clarity, significance, and comprehensiveness of our submission. We sincerely hope that these improvements can be taken into consideration.

Now we are approaching the end of the discussion period on November 22nd. If our response has resolved your concerns on our paper, we will greatly appreciate it if you could re-evaluate our paper. Should you have any further questions or need additional clarification, please know that we are eager and prepared to continue our discussions.

Thank you for the thorough review and positive feedback! Your meticulous suggestions have helped us to improve the clarity and enhance the comparison to baselines. To address your concerns, we present the point-to-point responses as follows.

Q1. The motivation and practical application of studying molecule-text modeling. The main criticism I have is that the motivation and needs for molecule-language multimodal model remains unclear. Of course, the multimodality between molecule and language domain is a promising research direction for machine learning researchers and practitioners. However, let us consider the situation where scientists actually conduct research using the proposed model. In what scenarios can the proposed model be utilized and how can it lead to the synthesis or retargeting of novel/existing compounds? The molecular domain is the task for domain experts (i.e., chemists and biologists), not the general users. Thus, discussion on the use case of the molecule-language multimodality for domain experts is crucial for setting the ultimate goal for this line of research. Unfortunately, however, this paper and other work in this field, seems to avoid answering this pivotal question. I hope that the authors address this question in the rebuttal or the revised manuscript.

...

First of all, as mentioned in the weakness section, I would like to ask the authors about the motivation and need for molecule-language multimodal learning. What are the specific use cases of the proposed model molecule-language model? How would scientists or practitioners (i.e., domain experts) be able to harness the proposed method in their research or business?

Response: Thank you for the insightful suggestion. We believe studying molecule-text multi-modal learning have values for scientists in (but not limited to) two aspects: 1) solving scientific tasks that are multi-modal in nature; 2) improving uni-modal scientific tasks by leveraging the science knowledge or reasoning ability of LMs. We now elaborate:

• Solving multi-modal scientific tasks. These tasks are multi-modal in nature and cannot be solved without multi-modal methods. For example, experimental procedure prediction [1] requires reading chemical reactions (molecule modality) and output experimental instructions (text modality). This task has significant practical values for automating chemical synthesis experiment design, and 3D-MoLM can be a foundation model for fine-tuning on this task. Another example is molecule-text retrieval, where chemists can specify desired chemical properties (e.g., biological activity and toxicity) using natural language, and retrieve molecules from database that includes unannotated molecules. This task can help retargeting existing molecules.

• Improving uni-modal scientific tasks. Uni-modal tasks (e.g., molecule property prediction) can potentially be improved by using the chemistry knowledge in LMs. [2] shows an example where joint multi-modal molecule-text learning improves molecule property prediction performance. [3] shows an example where joint multi-modal learning between an LM and a protein encoder improves the protein encoder's protein classification performance.

Q4: After carefully reading the response from the authors, I generally agree with their perspective.

Nonetheless, the proposed method and dataset do not seem to be capable of dealing with the scientific tasks mentioned in the response at the moment. While I acknowledge that this study is a first step towards the development of a multi-modal scientific language model, I think it is important that the authors specify the limitations of their work. This will not only clarify the contribution of this work, but also provide a guide for future research directions in this area.

Considering these points, I will maintain my initial rating.

Response: Thanks for your meticulous review and insightful feedback. In response, we have revised our submission and prepared responses below:

Regarding the capability of solving multi-modal scientific tasks.

• First, we want to highlight that 3D-MoLM have shown promising results for some (although not all) multi-modal scientific tasks mentioned in our previous response. Specifically, 3D-MoLM achieves significant and consistent improvements for molecule-text retrieval (Table 3) and predicting descriptive molecular properties (Table 4a); it achieves comparable performances for predicting computed molecular properties (Table 4c), when contrasted with Uni-Mol.

• Second, we acknowledge the limitations of the existing model and dataset, and want to propose the directions for future improvement:

  • More diverse and high-quality datasets. As you might have noticed, while some tasks are interesting and valuable (e.g., experimental procedure prediction [1]), they lack publicly available datasets (as per our submission time) to assist open-source studies. Additionally, existing tasks like molecular captioning and molecule-text retrieval can be further developed by introducing new datasets that focus on more complex and novel chemical properties (e.g., molecular spectra).

  • More fine-grained alignemnt between 3D molecules and texts. In Appendix D, we have included failure case studies showing 3D-MoLM's limitation in discerning fine-grained small molecular structures. Overcoming this issue is crucial for enhancing performance in tasks that demand a precise understanding of 3D molecular structures. One potential direction is to curate a new 3D molecule-text dataset with explicit 3D coordinate references within textual descriptions. Another direction is to explore more powerful 3D molecular encoders and 3D molecule-text projectors.

Regarding the limitation section. In response to your suggestion, we have revised and expanded the limitation section in Appendix G (page 18). This revised section now includes discussion about expanding to more molecule-text modeling tasks and fine-grained 3D molecule-text alignment. We also have a section discussing 3D-MoLM's failure cases in Appendix D, suggesting the exploration of better architectures and datasets for a more fine-grained alignment between 3D molecular geometries and textual concepts.

Your insights have significantly contributed to enhancing the completeness of our work. We hope our explanations and revisions can resolve your concerns about our submission. Should you have any other concerns, we are more than happy for discussions on openreview.

Reference:

[1] Inferring experimental procedures from text-based representations of chemical reactions. In Nature Communications 2021.

We sincerely thank you for your valuable comments and positive feedback! They have helped us to enrich the analysis of 3D-MoLM for a better understanding. To address your concern, we present the following response.

Q1: More insight and analysis. The analysis of the proposed 3D-MoLM is not enough. The proposed method is interesting and straightforward, any insight or analysis that could be offered to read to have a good understanding?

Response: Thank you for your constructive feedback. We agree that an in-depth analysis can help readers better understand 3D-MoLM's performances, and identify potential directions for future research and improvement.

Firstly, we have updated our paper to include a comparison between the performance of specialist 3D-MoLMs, which are fine-tuned for each task individually, and the generalist 3D-MoLM, which is a unified model trained on all tasks and evaluated using a consistent checkpoint. We also summarize the performance of them in the table below:

Table1: One checkpoint for captioning, open-text QA and computed properties.**

Table 1a: Molecule captioning performance.

Table 1b: Descrptive open-text QA performance.

Table 1c: Computed properties QA performance.

The results show that while the generalist model slightly underperforms in comparison to the specialist models, it still exhibits a performance gain over other baselines. This underscores its versatility and capability to effectively handle multiple tasks. For more detailed information, please refer to the performance tables in the revised paper (Table 3 and Table 4 in the revised paper) and the training details in Appendix C.

Secondly, we have revised our draft to include failure case studies in Appendix D, serving as preliminary examinations of 3D-MoLM's limitations. We identify two main limitations:

• 3D-MoLM's ability to discern fine-grained small molecular structures can be improved. In the revised Table 6, we show an example that 3D-MoLM confuses between Globostellatic acid C and Globostellatic acid B, where the only difference is the position of a methoxy group.

• We show that 3D-MoLM, and also GPT-4, cannot accurately count the number of atoms and generate the chemical formulas.

We are committed to continuously refining 3D-MoLM to address these issues. For a more detailed analysis and failure case study, please refer to Appendix D in the revised paper. We hope this analysis provides a more nuanced understanding of 3D-MoLM's capabilities and prompts future enhancement.