Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models

We sincerely thank you for your insightful feedback. We have addressed your concerns below and hope our responses provide clarity:

1. Quality control

In light of your valuable feedback, we have now expanded Section 3.2 to include a more comprehensive explanation of the quality control procedures we implemented (Highlighted in Section 3.2, Page 4).

2. More domain-specific baselines

In response, we have incorporated two new domain-specific Large Language Models (LLMs) into our evaluation: Galactica-6.7B[1] and PMC_LLaMA_13B[2]. This is because BioMedLM[3] and PMC_LLAMA_7B[4] are primarily suited for text-generation tasks. When prompted with questions and options, they tend to generate more options rather than providing answers, due to their lack of instruction tuning. To address this, we included PMC_LLaMA_13B, an instruction-tuned version of PMC_LLAMA_7B, as a new baseline in our study.

The updated results reflecting these expansions can be found in Tables 3 and 4, as well as Figures 5, 6, 8, 9, 10, and 11 in our revised manuscript.

3. More challenging datasets

Regarding the inclusion of the ScienceQA[5] dataset, we note that the majority of ScienceQA's biology-related content is multimodal, which does not align well with the current focus of Mol-instructions on text and molecular/protein sequences.

Incorporating multimodal data from ScienceQA would require a substantial extension of the Mol-instructions dataset to support formats beyond text and sequences. While this falls outside the scope of our current research, we recognize the potential value of such an expansion. It represents a promising direction for future work, where Mol-instructions could be enhanced to support a broader range of data types, including multimodal data. This would allow for a more comprehensive evaluation of LLMs in biomolecular research and is a pathway we are keen to explore in our future endeavors (Highlighted in Section 6, Page 9).

4. Factuality analysis

Thank you for highlighting the importance of a more in-depth analysis of the factuality of the generated text. The main challenge we encounter is the lack of a unified and widely accepted standard for evaluating tasks related to molecular/protein understanding. Consequently, we have relied on measurement methods used in previous NLP research.

We also attempted to use GPT-4 with certain predefined rules for scoring, but this approach did not yield stable scores. On the other hand, expert review, while potentially more accurate, is extremely time-consuming. Acknowledging these limitations, we are actively exploring more feasible and effective methods to analyze the factuality of the generated text, aiming to improve the robustness and reliability of our evaluation process.

Thank you once more for your invaluable suggestions, which have significantly helped in enhancing the rigor of our experiments and the completeness of our paper!

[1] https://huggingface.co/facebook/galactica-6.7b
[2] https://huggingface.co/axiong/PMC_LLaMA_13B
[3] https://huggingface.co/stanford-crfm/BioMedLM
[4] https://huggingface.co/chaoyi-wu/PMC_LLAMA_7B
[5] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. 2022.

Thank you very much for your valuable suggestion. Below are our responses:

Evaluation methods

Indeed, we have made several attempts and efforts in this regard. For instance, we customized rules to enable GPT-4 to score molecules. However, the outcomes were not as satisfactory as we hoped. GPT-4 consistently produced unstable scores, failing to accurately assess the quality of the molecules. Alternative methods like wet-lab experiments or expert reviews, while potentially more accurate, are prohibitively expensive and time-consuming. We are actively seeking more feasible and efficient solutions to address this evaluation challenge.

Thank you again for your insightful feedback!

Thank you very much for your insightful comments. Below are our responses:

Impact of Reduced Data on LLM Performance

The main premise of LIMA[1] is that almost all knowledge is acquired during the pre-training phase, and only a limited amount of instruction tuning data is needed to guide LLMs in generating high-quality outputs. For the domain-specific data encompassed in Mol-instructions, LLMs may not have encountered such data during the pre-training stage. Therefore, using fewer instruction data might not adequately equip the model to grasp the response format of this particular dataset. Our goal in constructing Mol-instructions was to comprehensively cover various biomolecular tasks, thereby infusing the model with relevant knowledge. This ensures that even users focusing on specific tasks can effectively utilize this dataset for subsequent research, benefiting the entire community. Indeed, the instruction tuning process may not necessarily require the full set of instruction data. I believe your suggestion provides an excellent starting point for future work, which could focus on discussing the most effective selection methods for sampling the most beneficial alignment data.

Thank you again for your constructive suggestions! We hope our responses address your concerns.

[1] LIMA: Less Is More for Alignment. 2023.

We sincerely thank you for your insightful feedback. Below are our detailed responses to your concerns:

1. More domain-specific baselines

Based on your recommendations, we have expanded our set of baselines to include Text+Chem T5[1], Galactica[2], and MolT5[3] as these models have undergone instruction tuning, aligning them more closely with the objectives of our study.

Regarding the baselines [4,5,6], we have chosen not to include them in our comparison as they focus on pretraining purely on molecular data without any instruction-following. This difference in approach makes a direct comparison somewhat unfair. However, we would like to clarify that our evaluation methods for reagent prediction, forward reaction prediction, and retrosynthesis are consistent with those used by Text+Chem T5[1], which we hope alleviates some of your concerns.

For protein-related tasks, the most closely related baseline we found was [7]. However, due to the unavailability of their code, we were unable to include it in our direct comparison. Instead, we have incorporated Galactica[2] as a baseline for these tasks, considering its relevance and accessibility.

The updated results reflecting these expansions can be found in Tables 3 and 4, as well as Figures 5, 6, 8, 9, 10, and 11 in our revised manuscript.

2. Lack of data processing details

Due to page constraints, a detailed description of our data transformation process, from the original sources to instruction data, was not feasible within the main text. However, to ensure transparency and reproducibility, we have included an extensive explanation in Appendix B (Pages 15-22). This section thoroughly details each task's definition, data sources, and the methodology employed to convert this data into the intended instruction format. We believe this provides a clear and comprehensive understanding of our data processing approach. Moreover, we have also added further details on quality control measures (Highlighted in Section 3.2, Page 4).

3. Insufficient tasks

We apologize for any misunderstanding and would like to clarify that our experiments comprehensively cover the 17 tasks listed in Figure 3. These include:

• Molecule-oriented Tasks: Molecular Description Generation (Figure 5), Description-guided Molecule Design (Table 4), Forward Reaction Prediction (Table 4), Retrosynthesis (Table 4), Reagent Prediction (Table 4), Property Prediction (Table 3).
• Protein-oriented Tasks: Domain/Motif Prediction (Figure 5), Functional Description Generation (Figure 5), Protein Function Prediction (Figure 5), Catalytic Activity Prediction (Figure 5), Protein Design (Figure 9)
• Biomolecular Text Tasks: Open Question (Figure 6), Multi-choice Question (Figure 6), Chemical-protein Interaction Extraction (Figure 6), Chemical-disease Interaction Extraction (Figure 6), Chemical Entity Recognition (Figure 6), True or False Question (Figure 6).

We believe that the variety of tasks covered by Mol-Instructions provides a substantial representation of common biomolecular challenges. Nevertheless, we recognize that there are aspects of biomolecular research that we have not yet explored. These represent opportunities for further enhancement in future research, and we are committed to continually expanding and refining our dataset to encompass an even wider range of biomolecular challenges.

Thank you again for your valuable suggestions! Your input has significantly contributed to enhancing the fairness of our experimental comparisons and the overall completeness of our paper.

[1] Unifying Molecular and Textual Representations via Multi-task Language Modelling. In ICML 2023.
[2] Galactica: A Large Language Model for Science. 2022.
[3] Translation between Molecules and Natural Language. In EMNLP 2022.
[4] Root-aligned SMILES: a tight representation for chemical reaction prediction. In Chemical Science 2022.
[5] Chemformer: a pre-trained transformer for computational chemistry. In Mach. Learn.: Sci. Technol. 2022.
[6] Reagent prediction with a molecular transformer improves reaction data quality. In chemical science.
[7] ProteinDT: A Text-guided Protein Design Framework. 2023.

Thanks for your response. It has solved most of my concerns. I have one more question to add. As I read through the revision, in Table 4, new baselines of MolT5 and Text+CHEM T5 are divided from other baselines. Is this presentation appropriate and fair? Considering MolT5 and Text+Chem T5 are also language models, I do not see specific differences to other models.

Authors are encouraged to revise the presentation in Table 4, and I will raise the rating accordingly.

1.8 Q&A and Information Extraction

1.9 Open Question

Mol-Instructions enhances the molecular understanding capabilities of LLMs, demonstrating notable improvements in every metric compared to baseline models, including even domain-specific smaller models. However, the molecule generation capabilities of LLMs still exhibit a discernible gap when compared to specialized smaller models, mainly because LLMs are designed to handle a wider range of tasks at the expense of the specialized performance seen in more focused models. For protein understanding, the model fine-tuned with Mol-Instructions outperforms other LLMs. And for biotext NLP tasks, it even surpasses PMC_LLaMA, which was specifically tuned with instructions on medical texts.

2. Data Processing and Quality Control

Due to page constraints, the detailed description of our instruction data construction process for each task is thoroughly outlined in Appendix B (Pages 15-22). Additionally, we have augmented Section 2.3 with further details on the quality control of task descriptions.

In conclusion, we extend our heartfelt thanks to all the reviewers for their invaluable suggestions, which have significantly enhanced the rigor and completeness of our work. We welcome further discussion and are more than willing to answer any additional questions you may have. Thank you very much!

Sincerely,
Authors

[1] Unifying Molecular and Textual Representations via Multi-task Language Modelling. In ICML 2023.
[2] Galactica: A Large Language Model for Science. 2022.
[3] Translation between Molecules and Natural Language. In EMNLP 2022.
[4] PMC-LLaMA: Towards Building Open-source Language Models for Medicine. 2023.