Large Language and Protein Assistant for Protein-Protein Interactions prediction

Thank you for your recognition of our work, and for your valuable suggestions. We will respond to your questions one by one.

These issues of "leakage" may also impact the comparisons against existing methods, which depending on how they are trained, may not have access to all of this information.

We have considered this issue. Specifically, UniprotQA has 1,593 duplicate sequences with SHS27k and 4,781 duplicate sequences with SHS148k. We removed the QA data for these sequences during pre-training, so there is no data leakage. The only potential source of data leakage is the protein encoder ESM-3B, as it was trained on UR50, which includes 48 million protein sequences. We added ESM2-3B as a control group, fixed its parameters, and trained a multi-classifier for experiments on the mPPI task. The results show that without fine-tuning the parameters of ESM2-3B, the model struggles to achieve ideal performance. In contrast to the superior performance of LLaPA, this suggests that any potential bias from data leakage in ESM2-3B is minimal.

In Section 3.4 lines 258-260, it is mentioned that the cosine similarity between protein embedding are used to retrieve the most similar proteins in the PPI network. What are the retrieval criteria exactly? Is there a score-based threshold or maybe a limit on the number of similar proteins that can be retrieved?

There are several works that address predicting sequence similarity/homology at the sequence level using ESM-2 and cosine similarity. It may be worth checking out PLMSearch (Liu et al; 2024) as one example of this.

Additionally, it may be worth checking out other protein database sequence similarity search tools like HMMER (Accelerated profile HMM searches (Eddy; 2011)), since UPINN only contains ~30K unique sequences. This would be very fast (probably faster than doing comparing the unseen sequence to all nodes in the graph) and could be used to provide more fine-grained control over which sequences are returned (considering where within a sequence matching regions are identified or the domain architecture of a returned sequence).

We initialize the node features of UPPIN using ESM2, and then select the node with the highest cosine similarity based on the target protein pp and the features of each protein in UPPIN, obtaining its topological information. Instead of using a threshold, we directly add the obtained topological information and the corresponding cosine similarity value as supplementary information to the prompt.

Thank you for providing information about the efficient sequence comparison tool. We believe that considering the matching regions or domain architecture within sequences could provide a more interpretable and expressive model, which might be significantly beneficial. We will carefully consider your suggestions and aim to improve LLaPA in future work, striving to ultimately build a general, practical, and highly interpretable multimodal protein natural language model.

Dear Reviewer, we have submitted the rebuttal version of our manuscript. For ease of reading, we have marked the revised content in red and used strikethroughs for the deleted content. We have temporarily moved Figure 3 and Figure 5 to the appendix to create more space.

We greatly appreciate the valuable time and effort you have invested in evaluating this work, your support for our work, and the insightful suggestions you have provided. We agree that considering matching regions or domain architecture within sequences could provide a more interpretable and expressive model. However, this requires more refined data processing and model training, with many details to consider, which we are unable to achieve in the current version. LLaPA is just a beginning, not an end. We believe that there is still a lot of room for exploration and potential research value in integrating protein features with LLMs, and we will carefully consider your suggestions in our future work.

Regarding the MA task you are interested in, we have placed the latest experimental results in Appendix A.10 of the revised version. The data we previously responded to you with had some omissions and inaccuracies due to parsing issues.

Once again, thank you for your trust and support for our work.

Thank you for your recognition of our model framework and experimental results, we also understand your concerns, and we will respond to your concerns one by one.

W1. Lack of Comparative Discussion and Limited Contribution: The most critical issue of the paper is that it ignores relevant protein LLM works which obviously can address the proposed challenges, i.e. relying on structural input and handling multiple protein input. To illustrate, ProtLLM should be discussed and compared. It models a joint space of protein and language, which directly addresses the proposed challenges. Also, there is another work ProLLaMA that does the similar thing. These two works are neither discussed in related work nor are they compared in the experiments.

Contributions: Our main contributions are as follows:

1. We have revealed a significant shortcoming of the existing Multi-label PPI type prediction (mPPI) works and proposed our solution to push the mPPI task towards a more practical direction.
2. We propose treating the PPI network as external knowledge and injecting it into LLMs through Retrieval-Augmented Generation (RAG) to assist downstream tasks. We also constructed a more general PPI network called UPPIN.
3. We have proposed a multimodal large language model (MLLM) that integrates protein features and PPI network information, achieving state-of-the-art performance in mPPI tasks and also showing promising results in Affinity prediction tasks (MA) tasks.

More Related Works: Since LLaPA is a MLLM for proteins, we agree that citing and discussing some works that integrate large language models for proteins can enhance the completeness of this work. Therefore, we will add a section introducing MLLMs for proteins, including related works such as ProtLLM[1] and Prollama[2]. Although both works utilize proteins and LLMs, LLaPA has significant differences from them in terms of model architecture and specific tasks. We will place the detailed discussion in the last paragraph of this response and include it in the revised version of the manuscript.

Challenges: The two methods you mentioned do not fully address the challenges we have proposed. We have identified three main challenges:

1. Existing methods have some unreasonable aspects when utilizing PPI networks, specifically using the topological information between the tested protein pairs during testing, which makes their high performance difficult to apply in real-world scenarios. ProtLLM and ProLLama do not use PPI network information and do not focus on mPPI task, so they cannot address this challenge.
2. Handling unseen proteins. We obtain topological information from the PPI network based on sequence alignment concepts, and inject the topological information of the PPI network into the LLM through RAG. ProtLLM and ProLLama do not address this issue and do not provide a solution.
3. Predicting relationships between multiple protein complexes. Although these two works do not discuss this issue, from a model architecture perspective, ProtLLM can indeed handle multiple protein sequences. ProLLama, while limited by its context length, can also process a certain number of protein sequences. We will revise this section accordingly.

Differences between LLaPA, ProtLLM, and ProLLaMA

1. ProtLLM designed a pre-training method that interleaves protein and natural language, and validated its effectiveness on several downstream tasks. Although it evaluated on a simple binary classification PPI task, it did not address more challenging mPPI tasks or MA tasks. In these tasks, relying solely on protein features is insufficient for achieving ideal performance; the position of the protein within the PPI network is also crucial. Our work focuses on using LLMs to solve more practically significant problems. Our pre-training strategy is distinctly different from ProtLLM. Additionally, we introduce PPI network features to better predict mPPIs tasks and MA tasks. We also propose leveraging the complementarity of protein information and PPI network information to accelerate the alignment of PPI network features with large natural language model features.

2. ProLLama focuses on how to effectively train protein sequences directly using large language models. It performed an additional pre-training and an Instruction Tuning based on LLaMA. Besides having a significantly different model architecture from our work, it also did not address mPPI tasks or MA tasks, nor did it utilize PPI network information.

[1] Zhuo, Le, et al. "Protllm: An interleaved protein-language llm with protein-as-word pre-training." arXiv preprint arXiv:2403.07920 (2024).

[2] Lv, Liuzhenghao, et al. "Prollama: A protein large language model for multi-task protein language processing." arXiv e-prints (2024): arXiv-2402.

Dear Reviewer, we have submitted the rebuttal version of our manuscript. For ease of reading, we have marked the revised content in red and used strikethroughs for the deleted content. We have temporarily moved Figure 3 and Figure 5 to the appendix to create more space.

We greatly appreciate the valuable time and effort you have invested in evaluating this work, as well as the insightful questions and suggestions you have provided. We have addressed all of your concerns, and in response to your queries, the following revisions have been made in the manuscript:

1. We have modified some qualifiers in the introduction to clarify which works the stated limitations are referring to;
2. We have added brief descriptions of ProtLLM, ProLLama, Prot2Text, and ProteinGPT in the introduction, and clarified that LLaPA is not the first work to address Challenge 3;
3. We have added three new baselines, including ESM2-3B(fixed), ESM2-3B(ft), and ProtLLM;

We hope that all the responses and experimental data provided can alleviate your concerns about this work and further support our efforts.

As there are still a few days left until the end of the discussion, if you have any other questions or concerns, please feel free to raise them, and we will do our utmost to address them.

Thank you for your partial recognition of our work, we also understand your concerns, and we will respond to your comments one by one.

Please describe in detail why the training is divided into two phases, and why there is a need for the first phase. It is suggested that the authors could add an experiment to illustrate the need of the first training phase.

Two-phase training is adopted by many multimodal large language models [1-5]. The primary purpose of the first phase is to use a large amount of task-agnostic data to align the feature space of the encoder output of other modalities (in this work, proteins) with the input feature space of the large natural language model. The second phase of training is fine-tuning for downstream tasks. The first phase of training is very important and can significantly improve the model's performance, as demonstrated by works [1-5]. Additionally, in our 5.3 ABLATION STUDY, we conducted experiments showing that without the first phase of training, the F1 score decreased by 7.34 on the SHS27k dataset under the DFS setting.

[1] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).

[2] Li, Chunyuan, et al. "Llava-med: Training a large language-and-vision assistant for biomedicine in one day." Advances in Neural Information Processing Systems 36 (2024).

[3] Chen, Jun, et al. "Minigpt-v2: large language model as a unified interface for vision-language multi-task learning." arXiv preprint arXiv:2310.09478 (2023).

[4] Liu, Zhiyuan, et al. "MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

[5] Zhang, Mengmei, et al. "GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Tasks." Proceedings of the ACM on Web Conference 2024. 2024.

In the prompt, the authors included text representations of the protein sequences and indexes. Please demonstrate the reason.

The input consists of three parts: the prompt, the protein sequences, and the position of each protein sequence in the protein-protein interaction network (UPPIN). The prompt is encoded by the LLM's encoding layer, the protein sequences are encoded by ESM2, and the protein indexes are used to find the corresponding proteins' positions in UPPIN and obtain the encodings of these nodes in UPPIN. Finally, the protein encodings and node encodings are fused with the prompt encoding at the positions of special tokens in the prompt, forming the final input. We apologize for any confusion caused by the representation in Figure 3, and we will add more detailed explanations in the caption to clarify this.

For training, there are 2 L_LLM. Please demonstrate their difference.

Are you referring to the two LLM losses in Figure 1? These two LLM losses correspond to two different training stages. As mentioned in the first question, we conducted an ablation study on the first training stage in section 5.3. The LLM loss in the second stage is essential; without it, the model cannot converge.

Why do the authors use SGC as graph encoder? What happens if you use gnn like GNN-PPI, e.g. GCN to encode networks? Compared to SGC, it seems like GCN just have one extra step of using a nonlinear activation function.

We initially used GNN-PPI. In fact, GNN-PPI as an encoder can provide better performance. However, GNN-PPI has higher complexity, requiring more memory and longer training time, especially after we constructed a larger PPI network. We hope to build a more general PPI network in future work, which may be even larger than the current UPPIN. Therefore, we aim to make some trade-offs between performance and efficiency. GCN is the most classic graph encoder, and we considered this method as well. However, after research, we found that SGC can achieve a better balance between performance and efficiency, so we chose SGC.

Dear Reviewer, we have submitted the rebuttal version of our manuscript. For ease of reading, we have marked the revised content in red and used strikethroughs for the deleted content. We have temporarily moved Figure 3 and Figure 5 to the appendix to create more space.

We greatly appreciate the valuable time and effort you have invested in evaluating this work, as well as the insightful questions you have raised. We have answered all of your concerns, and your queries have led to the following three revisions in the manuscript:

1. Added a more detailed explanation for Figure 3;
2. Explicitly stated that the Homo sapiens subset of the STRING dataset was used;
3. Corrected the formula for LLM loss;

Additionally, we have included more related works on protein-LLMs; added ESM2-3B (fixed weights), ESM2-3B (fine-tuned), and ProtLLM as baselines; clarified the definition of the MA task; and provided MA experimental results considering all sequences rather than unique sequences (Appendix A.10).

We hope that all the responses provided can alleviate your concerns about this work and further support our efforts.

As there are still a few days left until the end of the discussion, if you have any other questions or concerns, please feel free to raise them, and we will do our utmost to address them.

Dear Reviewer pBue,

Thank you for the time and effort you have dedicated to evaluating our work.

We have responded to your questions point by point, but we have not yet received your feedback. We are very eager to know if we have addressed your concerns regarding this work.

Although there are still three days left for the discussion, the weekend is approaching, which might be your personal time. Therefore, the actual time available for discussion may be limited.

I sincerely hope to receive your response to let us know if we have resolved your concerns.

Additionally, if you require any extra supporting materials, we still have time to prepare them.

Thank you once again, and we look forward to your reply.

Thank you for your partial recognition of our work. Your concerns are also valid, and we will address your questions one by one.

Usually, for a set of chains, a single affinity value does not exist; rather, it is dependent on the choice of two binding partners.

According to the definition of the dissociation constant, the affinity of a protein complex depends on the choice of the binding partner. There are indeed some ambiguities when defining the MA task. The correct definition of the MA task should be: given a complex $C=(B,T)$，where $B$ refers to the binder and $T$ refers to the target, predict its logarithmic dissociation constant $\log{Kd}=\log\frac{[B][T]}{[BT]}$. $B$ and $T$ can each be a single protein sequence or a complex containing multiple sequences. For example, in PDB 6ILM, the affinity describes the dissociation constant between the Fc receptor and Echovirus 6. However, the target Echovirus 6 is a complex containing four protein sequences. For the PDB2020 dataset, which includes 2852 complexes, accurately extracting the binder and target based on the given information is very challenging and requires manual analysis of the papers corresponding to each PDB entry. Therefore, we have simplified this task in the current version.

Regarding weakness #2, could you provide more theoretical or emperical evidence to justify the benefit of LLM?

You are absolutely right. Theoretically, LLM is not a necessity. For both the Multi-label PPI type prediction task and the Multi-sequence Affinity prediction task, we could use a smaller network architecture to achieve these tasks individually. LLM is one solution to the issues we raised, but it is not the only solution. However, using LLM has unique advantages:

1. LLM possesses excellent multitasking capabilities and flexibility. By constructing appropriate natural language instructions, we can unify various tasks into a generative task, using a unified framework and training method to build a multi-task unified model. Different downstream tasks can be guided through appropriate natural language instructions.

2. Additionally, small models fit different loss functions for different tasks, and the model itself does not know what it is doing. In contrast, large language models have a certain level of common sense and seem to have the ability to understand tasks. This means that LLM might know what task it is performing, what the task means, and even the hidden implications of the output. For example, if we ask gpt-4o-mini, "If the relationship between protein A and protein B is activation, what does it mean?", we would get: "If the relationship between protein A and protein B is described as 'activation,' it means that protein A enhances or stimulates the activity, function, or expression of protein B. This can involve various mechanisms, such as promoting a conformational change in protein B, facilitating its interaction with other molecules, or increasing its stability or availability within the cellular environment. In essence, protein A acts as a positive regulator of protein B's biological activity."

3. The LLMs demonstrates excellent few-shot learning and zero-shot learning capabilities [1-2]. By connecting the protein modality with the LLMs, it may enable the LLMs to understand proteins and provide a richer array of application scenarios.

More specific discussions can be found in references [1-6].

In this work, we started from specific practical tasks and hope to build a unified, practically valuable multimodal protein model in the future, which is why we used LLM in this work. However, in terms of performance, whether using LLM will necessarily be more effective than a specially trained small model for specific tasks is indeed a topic worth discussing and requires more comprehensive work for validation.

[1] Kojima, Takeshi, et al. "Large language models are zero-shot reasoners." Advances in neural information processing systems 35 (2022): 22199-22213.

[2] Gruver, Nate, et al. "Large language models are zero-shot time series forecasters." Advances in Neural Information Processing Systems 36 (2024).

[3] He, Qianyu, et al. "Can Large Language Models Understand Real-World Complex Instructions?." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 16. 2024.

[4] Yuan, Quan, et al. "Tasklama: probing the complex task understanding of language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.

[5] Zhao, Zirui, Wee Sun Lee, and David Hsu. "Large language models as commonsense knowledge for large-scale task planning." Advances in Neural Information Processing Systems 36 (2024).

[6] Guo, Taicheng, et al. "What can large language models do in chemistry? a comprehensive benchmark on eight tasks." Advances in Neural Information Processing Systems 36 (2023): 59662-59688.

In the main text, the authors present results for the "/R" models where the test edges are removed. As these models are trained on the PPI graph, it is to be expected that the "/R" models will underperform significantly due to a significant gap in training and test settings. Outperforming these nerfed models is not a convincing achievement of LLaPA.

We understand your concern that this comparison might seem unfair. It is important to note that a key focus of our work is to push PPI prediction towards a more practical direction. Therefore, we included the '/R' setting in the comparison group, as not including '/R' would also be unfair. We can see that starting from GNN-PPI, methods based on PPI networks have shown a significant leap in performance on the multi-label PPI type prediction task. However, this improvement is inherently unfair because these methods have potential data leakage during testing. Therefore, we believe that removing test edges might provide a fairer comparison. Although removing the edges between test proteins only during testing unsurprisingly reduces the model's predictive ability, removing test edges during training results in even worse performance, especially under more challenging DFS and BFS splits, because a considerable number of isolated nodes cannot obtain any information from the PPI network. Excluding these PPI network-based methods, PIPR might be the best mPPI prediction method currently available. In the revised version, we have added ESM-3B and ProtLLM [8] as baselines. For ESM2-3B, we froze its parameters (as our model also has frozen parameters) and trained a multi-classifier on top of it. The experimental results are shown in the table below. As can be seen, ESM2-3B performs poorly when only a multi-classifier is trained with frozen parameters, and ProtLLM shows similar results.

We hope the above response can alleviate your concerns and further endorse our work.

We greatly appreciate the time and effort you have invested in evaluating this work. We have added a new baseline, which involves full parameter fine-tuning of ESM2-3B. We modified the EsmClassificationHead in EsmForSequenceClassification from the transformers library transformers/src/transformers/models/esm/modeling_esm.py at v4.46.3 · huggingface/transformers to support the mPPI task. For each task, we trained for 15 epochs (same as LLaPA), and the other hyperparameters of the experiment were also consistent with LLaPA. On 8 A100 40G, each SHS27k-related task required about 1 hour of training; each SHS148k-related task required about 5 hours. After full parameter fine-tuning, ESM2-3B provided a stronger baseline on SHS27k (dfs), SHS148k(dfs), and SHS148k(bfs), but it did not perform better in other experimental settings. LLaPA still achieved the best performance under various task settings.

Additionally, we have listed the ratios of protein pairs tested under each experimental setting that fall into the categories of 'both have been seen' (BS), 'either one protein has been seen' (ES), and 'neither one has been seen' (NS), as shown in the following table:

We hope that the addition of this new baseline can further substantiate the effectiveness of LLaPA and alleviate your concerns. We will update these experimental results in the rebuttal version of the manuscript and submit it before the deadline. If you have any other questions or concerns, please feel free to raise them. As there are still a few days left before the discussion deadline, we have the opportunity for more in-depth discussions.