InstructProtein: Aligning Human and Protein Language via Knowledge Instruction

We would like to express our great appreciation to reviewer modh for their comments on our paper. The unclear content have been revised in manuscript, and the response to your concerns is as follow:

It is mentioned in the text that the higher the absolute value of the pLDDT score, the better. However, I believe it is not that straightforward. The pLDDT score is an indicator of the quality of a protein's structure, and in the text, protein sequences are generated using a language model. The unstructured regions in the protein sequence have a significant impact on the protein's function, and correspondingly, the pLDDT score may be lower. To compare the quality of generated protein sequences, it's important to evaluate whether the distribution in multiple dimensions aligns with real proteins as shown in Figure 5. In addition, the visualized results should incorporate real protein visualization results.

Regarding Figure 5, our evaluation focuses on the model's proficiency in adhering to structure-related instructions. In this context, the pLDDT score serves as a suitable evaluation metric. Specifically, when presented with an instruction such as 'I would like an all-alpha protein', the sequences generated by the model should demonstrate an increased proportion of alpha-helix and a decreased proportion of Intrinsically Disordered Protein (IDP) regions and other secondary structures. The high pLDDT score effectively reflects the low proportion of IDP regions, thus serving as a pertinent evaluation metric. Furthermore, Figures 5 (b,c) visually represent the embedding and predicted structures, highlighting significant differences in protein embeddings generated by distinct instructions. Notably, the folded structure aligns consistently with the provided instructions. In response to your suggestion, we've incorporated visualization results of real proteins, depicted in Figure 13. Upon observation, we note a striking similarity between the distribution of generated protein representations and that of real proteins. This congruence signifies the model's remarkable instruction-following ability. We will add these analysis in the final draft.

In addition, There are alternative evaluation metrics for evaluating the generated sequences, such as assessments based on αβ structures, barrel helices, thermostability, metal element binding, etc. From an AI perspective, tools like Biopython can also be used to analyze biological aspects, such as pI values and electrostatic potential.

Thanks for providing alternative evaluation metrics for the generated sequences. The experiments is Section 4.3 are designed to demonstrate the model's instruction-following ability rather than merely assessing sequence quality. Therefore, the evaluation metrics need to correspond to the instructions (i.e., structure-related instruction with structure-related metrics, etc). Furthermore, in response to your suggestions, to assess generated sequences based on metal element binding, we additionally design three proteins with the instructions of ``I would like a protein that enables metal ion binding''. However, their binding ability cannot be concluded without wet experimental verification. So we compared it to existing protein to evaluate them as shown in Figure 15.

When conducting baseline comparisons, it is more convincing to compare against Mol-Instructions in addition to LLaMA and Alpaca. Mol-Instructions is a dataset constructed from text data of proteins based on UniProtKB, and fine-tuned on LLaMA-7B. Therefore, this paper should give the comparison results against Mol-Instructions.

We leverage the checkpoint from "zjunlp/llama-molinst-protein-7b" and report the results as shown below:

From the results, we found three shortcomings of this model. Firstly, we have observed that employing a naive data cleaning strategy does not offer immunity to label imbalance issues. This observation further emphasizes the superiority and effectiveness of our proposed knowledge instruction methodology. Secondly, due to the lack of diversity of Mol-Instructions templates (excluding true/false questions), the model cannot understand the questions in the GO task and answer them accordingly. We propose that the cooperation of KG and LLMs to ensure the diversity of data is key for text and protein alignment. Finally, Mol-Instructions uses BPE to tokenize proteins, which makes it more difficult for the model to distinguish the nuances of proteins than amino acid-based tokenization, resulting in poor results on Fold Rank. Further, we leverage their model to design proteins as shown in Figure 14. The results demonstrate the model lacks the ability to follow structural instructions.

We would like to express our great appreciation to reviewer DyDT for their comments on our paper. The response to your concerns is as follow:

Regarding the data-collection. Prompt each triplet to ChatGPT and get the results as the instruction is an important step. But will this assume that ChatGPT can already be prompted to understand the protein input and make output correctly? Especially, in the Figure 4 which is an example of how ChatGPT is prompted, the ChatGPT can infer the information 'it is in the leptin family' and this information is important to pose the causal knowledge. How you guarantee that the output is correct and useful? 'ChatGPT' can understand protein is not a desiable assumption.

The methods shown in Figure 3a and 3b assume that LLMs understand proteins and are able to come up with novel tasks and output desired answers. Compared to them, our innovation lies in the removal of such an assumption regarding LLMs possessing inherent protein-related knowledge, as depicted in Figure 3c. We retrieve the facts needed to generate the answer through KG and prompt LLMs how to generate tasks through KGC templates. All LLMs need to do is to faithfully convert the triplet to natural language.

In Figure 4, our retrieval process involves not only acquiring the triplet (MRCPG..., function, hormone) but also capturing associated knowledge (MRCPG..., family, leptin) that exhibits a causal relationship with the triplet. By utilizing these two interlinked pieces of information, we prompt ChatGPT to generate instructions. Figure 8 offers a detailed breakdown of the process involved in converting a triplet into instruction data. This breakdown encompasses inputs (prompt) and outlines ChatGPT's corresponding outputs. We will add these analysis in the final version.

Lacking of details. Regarding the data-collection, more details about the knowledge-graph need to be provided. You claim knowledge-graph can provide causal knowledge. The motivation can be more straghtforward if you tell what relationships the KG can provide and help the downstream tasks. Regarding the sampling, how the protein KG embedding is initialized? and what are the training details?

Thanks for your advice. I would like to elaborate on the components involved:

(1) Knowledge Graph: Our knowledge graph comprises triples denoting (protein, relation, annotation). As delineated in Section 4.1, the annotations encompass various attributes such as superfamily, family, domain, conserved site, active site, binding site, location, function, and involved biological processes of proteins. Notably, proteins utilized in downstream tasks have been excluded from the training data to maintain integrity.

(2) Causal Knowledge: The mentioned annotations are distinct and independent entities. In the realm of protein function exploration by biologists, the customary approach involves commencing with easily studied features and deducing inferences based on the interconnections among these features. The objective of integrating causal knowledge is to equip language models with the capability to reason based on the relationships among annotations. This causal knowledge specifically encompasses the causal relationship from family and key sites to protein function and biological processes (A protein with an anaphylatoxin/fibulin domain indicates that it is probably located in an extracellular region).

(3) KG Embedding: Following the TransE approach, we initiate embeddings for entities and relationships through a random initialization procedure [1]. We employ the SGD optimizer with a learning rate of 1.0. The dimensions of entities and relations' embeddings are set to 200. After 1000 epochs, the loss eventually converges to 0.168. The L2 distance utilized for clustering proteins is set to 1.4. We will add these details in the final draft.

We will add these details in the final version.

We thank Reviewer Ex2Y for reviewing our paper and providing thoughtful feedback on our work. We have revised the manuscript to make this paper easier to follow. Here, we provide details on comments below.

For the first conclusion in 4.2: The results (comparing with OPT, LlaMa, Alpaca) demonstrate that training with the corpus where proteins and natural language coexist is beneficial to LLMs, enhancing their proficiency in protein language understanding. However, I think this argument can not be concluded based on these results. Because the performance of the InstructProtein is contributed by both pre-training and finetuning. Without finetuning LLMs on the instruction corpus, we can not conclude that the coexist of proteins and natural language is beneficial (even this conclusion is quite intuitive).

Thank you for highlighting the potential misunderstandings that the conclusion might induce. The essence of this conclusion lies in highlighting the limitation of the current Natural Language Processing (NLP) corpus in furnishing protein comprehension abilities to Large Language Models (LLMs). To further substantiate the advantages of a corpus integrating both proteins and natural languages, we are providing additional experimental results, presented below:

The pre-training is conducted with the UniRef100 and Pubmed datasets that respectively contain proteins and biomedical literature, while the finetuning is conducted using the proposed Knowledge Instruction approach with aligned natural language and protein language corpora. We can observe that the improvement of the performance is mainly due to the instruction tuning stage. We will add this results to the final draft.

Do we have any quality evaluation on the generated instruction dataset?

We advocate assessing the quality of the generated instruction dataset based on both correctness and diversity. To mitigate the potential generation of erroneous data by Large Language Models (LLMs), we construct a Knowledge Graph (KG) based on the well-maintained UniProbKB database and regared it as a reliable knowledge source. Then, we sample from this KG and convert the sampled triples to instructions. In this way, the correstness of the instruction dataset can be tracked and guaranteed by the UniProbKB. Diversity is quantified with information entropy. For instance, considering subcellular location-related instructions, Figure 2 highlights annotation imbalance. The original data distribution's information entropy is calculated as $H(x)=-\sum_x p(x)\log p(x) = 3.17$. Through our proposed KG triplet retrieval method, this entropy increases to $3.68$. This increment in information entropy denotes enhanced diversity within the dataset, illustrating increased coverage and richness of information in the generated instructions. We will add the above analyses to the final draft.

Thanks for your constructive and valuable comments, which have helped us improve the paper considerably. We will address these confusion and move this to the main text in the final version to make our paper stronger.

Before the point-by-point response, we would like to underscore the contributions of our work. Firstly, our study identifies and highlights the prevalent issue of annotation imbalance within the current protein corpus, as elaborated in Section 2. We propose a comprehensive framework designed to generate high-quality text-protein-aligned instruction data sourced from an imbalanced knowledge base. We believe this instruction dataset will greatly faciliate future LLM research in biology. Secondly, we introduce InstructProtein, a model specifically trained on our constructed instruction data, demonstrating proficiency in comprehending the nuances of protein language.

It is crucial to note that this article's primary focus centers on the translation between natural language and protein language. While our framework enables interaction with users through text-based communication, broader interaction akin to ChatGPT in general domain exceeds the current scope of our study.

Additional baselines should be incorporated for a more comprehensive analysis. The current comparison lacks an evaluation against more advanced Large Language Models (LLMs) such as ChatGPT , GPT-4, and Claude-2. It is crucial to understand the performance disparities between the proposed method and these well-established models.

To comprehend the performance disparities between InstructProtein and these established models, we conducted the localization annotation experiments of 100 proteins. Presented below is the statistical results as well as a specific case along with the corresponding response:

User: What cellular component is the protein located in? Please give only one prediction. MKLAIGRLLSPLFLKNPQKPLIITKRFYSTPGERRIKDILTEKLSPSSLRVIDVSGGCGSMYQVAIKSKAFQGKNTLAQHRLVNSILKEEIRNMHGLNLSTEVEDDISAGGSTTSS

ChatGPT: Based on the provided amino acid sequence, the prediction for the cellular component of the protein is the cell membrane. (Wrong)

GPT4: Based on the sequence provided, I would predict that this protein is located in the mitochondrion.  Here's the rationale for my prediction: ...(Wrong)

Claude-2: Nucleus. My rationale: (1) It lacks a signal peptide, indicating it is not a secreted protein. (2) It does not contain any predicted transmembrane domains, suggesting a soluble localization. (3) It has abundant lysine and arginine residues providing a positive charge. (Wrong)

InstructProtein: The protein is located in mitochondrion. (Correct)

The results yield three pivotal observations. Firstly, we are surprised to discover that ChatGPT and Claude-2's predictions solely encompassed cytoplasm, nucleus, cell membrane, and secreted locations. This finding underscores that annotation imbalance impacts even these well-established models. Notably, the performance degradation due to data imbalance persists despite model enhancements or increased training data, further emphasizing the significance of our contributions of the high-quality instruction dataset. Secondly, while the responses from them may initially appear plausible, their prediction accuracy is notably deficient. This highlights a crucial aspect regarding the precision of their predictions, indicating a large performance disparity. Additionally, we employed ChatGPT and Claude-2 to design the globin-like, immunoglobulin-like, and tim beta-alpha-barrel protein, and the outcomes are visually presented in Figure 14. We observed that Claude-2 ignores all instructions and output proteins with similar structures no matter what instructions were given. ChatGPT, on the other hand, can only understand basic instructions (globin-like, immunoglobulin-like). Only InstructProtein can faithfully follow instructions to generate desired proteins.

Here, we would like to explain why we did not consider ChatGPT or GPT-4 as baselines before. Until the day we submitted this manuscript to ICLR, ChatGPT often refused to generate protein sequences, as shown in Figure 1. We have tried our best to write the appropriate prompt, but for some cases, these LLMs still tend to decline protein sequence generation, which makes automated benchmark comparison impossible.