Illuminating Protein Function Prediction through Inter-Protein Similarity Modeling

Thanks for your insightful comments and golden suggestions! We respond to your concerns as below:

Q1: The paper lacks a dataset section. The evaluation procedure is not adequately described in the paper.

Thanks for the questions. During rebuttal, we’ve added more dataset details and evaluation setup in App. C.

Q2: The paper claims that their approach reduces computational requirements while needing less running time, yet this is not supported by experiments or any comparisons to other methods.

Thanks for the question. In the revised version, we include the time comparison of different methods in App. E.3.2.

Q3: The paper uses inter-protein similarity to model and predict function, but it does not compare this approach to important and state-of-the-art methods that model inter-protein similarity.

Thanks for bringing our attention to these works. We’ve added a paragraph to discuss the relation between our work and this stream of works in App. A.

To clarify, instead of focusing on inter-protein similarity, these works use protein-protein interaction networks for function annotation, such as STRING. These networks map both direct physical and indirect functional interactions among proteins. A key limitation of these methods is that they are not able to make predictions for newly sequenced proteins absent in existing PPI networks. Moreover, knowing protein-protein interactions is essentially a more difficult challenge, as it requires a more comprehensive understanding of protein properties. These problems make this line of work hard to use in real-world settings. Therefore, to ensure a fair comparison, we do not include these baselines in our benchmark.

Instead, during rebuttal, we add CLEAN paper [1] and other corresponding baselines on function annotation for comparison, which is more related with our work. The results are shown in App. E.1, which clearly demonstrates the effectiveness of our methods.

Q4: The authors assume that there is independence between the function labels. This assumption does not hold for protein function prediction using GO terms. The authors should better justify why they make this design decision.

This is a good point! We simply treat the function annotation problem as a multiple binary classification problem following common practice in machine learning. This is a reasonable assumption as most functions can be determined by the protein sequence and structure. Also, we want to emphasize that the focus of this paper is to study general function annotation tasks without any assumption on the function-specific characteristics. Nevertheless, we admit the effectiveness of modeling the tree structure of GO terms and leave this as future work.

Q6: In section 2, authors claim that "A notable advantage of these neural retrievers over traditional methods is their flexibility in fine-tuning for specific functions, as will discuss in next section". Is not clear to me where this is discussed or pointed, could you please elaborate this further?

Thanks for the question! One advantage of neural retrievers is that they can be fine-tuned for specific functions using contrastive supervised learning, as done in the CLEAN paper [1]. This is exactly what to do to refine the retriever in the ProtIR framework.

[1] Yu, Tianhao, et al. "Enzyme function prediction using contrastive learning." Science 379.6639 (2023): 1358-1363.

Thanks for your insightful comments and golden suggestions! We respond to your concerns as below:

Q1: I would worry about robustness to changes in hyperparameters for the EM finetuning process (eg. epoch size learning rate etc.).

Thanks for the question! We’ve added a section to discuss the hyperparameter setting and tuning in App. E.4. For ProtIR, we can directly inherit the best hyperparameters from existing works during pre-training. During the refinement stage, we tune the hyperparameter on the validation set to avoid looking at the test performance during training. We need to ensure the model converge at each interaction and avoid overfitting. To maintain this balance, we closely monitor performance metrics on the validation set and halt training when no further improvements are observed.

Q2: How much do you think this method (iterative refinement between similarity and classification models) do you think is general? What other kinds of data might it extend to? Are there similar examples in other domains?

The method is quite general, as it can be applied on texts, images or graphs. To the best of our knowledge, there is little existing work on other data formats. But there are some works using the idea of iterative refinement for mutual enhancement of LLM and GNN-based methods [1].

[1] Zhao et al. “Learning on Large-scale Text-attributed Graphs via Variational Inference” ICLR (2023)

Thanks for your insightful comments and golden suggestions! We respond to your concerns as below:

Q1: Innovation Clarification: The authors should clearly articulate any novel contributions of their work, particularly if there are innovative aspects beyond the combination of predictive and retriever-based methods.

Thanks for the question! We add a paragraph in the Introduction and Related Work sections to discuss the contribution of this work.

Specifically, we find that there is still a lack of systematic study of modern deep learning methods and retriever-based methods for function annotation. The most closely related work should be CLEAN [1], where they benchmark their proposed supervised contrastive learning methods with several retriever-based methods. However, there are still many deep learning baselines not considered in that paper. In this paper, we include more modern baselines in our benchmark and add another experiment on the CLEAN setup to show the advantage of our method in App. E.

From method perspective, there are many existing works using retriever-based and transductive learning methods for other machine learning applications, but there has been little effort put in the protein function annotation task. One related work along this direction is a concurrent work - RSA [2], which studies other sequence-based property prediction tasks. Our work is among the first to study how to use retriever-based methods for enhancing predictor-based methods.

[1] Yu, Tianhao, et al. "Enzyme function prediction using contrastive learning." Science 379.6639 (2023): 1358-1363.

[2] Ma, Chang, et al. "Retrieved Sequence Augmentation for Protein Representation Learning." bioRxiv (2023): 2023-02.

Q2: Theoretical Insights: A section discussing the theoretical implications of the findings would enrich the paper. This could include a discussion on the conditions under which each method excels or fails.

We want to argue that this paper is an application-oriented paper, where we put our focus on a practical function annotation setting. Our proposed method is principal as derived in Sec. 3 and achieves good performance on our benchmark. We believe the contribution of this paper is enough for a conference paper and we leave the theoretical analysis as future work.

Q3: Broader Impact and Ethical Considerations: The manuscript would benefit from a dedicated section on the broader impact and ethical implications of the research, particularly in the context of bioinformatics and potential applications in drug discovery.

Thanks for the suggestion! We’ve added a paragraph to discuss the potential negative impact of the paper in the App. A during rebuttal.

Q4: Compared to structure prediction, what's the most significance of function prediction of proteins?

Function prediction is clearly an important problem in the protein domain, as the ultimate goal in this domain is to design proteins with desired functions. This problem has been studied for decades [1,2,3]. Structure prediction is just an intermediate step for us to understand protein functions, as structure directly determines protein functions.

[1] Ashburner et al. "Gene ontology: tool for the unification of biology. The Gene Ontology Consortium". Nature Genetics (2000). 25 (1): 25–9.

[2] Webb et al. “Enzyme nomenclature 1992: recommendations of the Nomenclature Committee of the International Union of Biochemistry and Molecular Biology on the nomenclature and classification of enzymes.” Academic Press (1992)

[3] Yu, Tianhao, et al. "Enzyme function prediction using contrastive learning." Science 379.6639 (2023): 1358-1363.

Q5: Is it feasible to integrate the methodology proposed in the manuscript with AlphaFold 2 to create an end-to-end system for protein function prediction?

Thanks for proposing this idea. We’d like to argue that there is no need to combine with AlphaFold2 for an end-to-end system. First, AlphaFold2 focuses on the structure prediction problem, which has been almost solved and is not relevant to the problem we studied in this paper. We can directly take the experimentally determined or predicted structures as input to our model, instead of connecting with AlphaFold2. Second, using AlphaFold2 during training will be extremely time-consuming, which violates the goal of this paper to improve function annotation efficiency with little additional training.

Thanks for your insightful comments and golden suggestions! We respond to your concerns as below:

Q1: In the experimental section, the authors mainly present the experimental results, but the analysis is not sufficient.

Thanks for the suggestion! We’ve revised the experiment section to add more analysis about the benchmark results. Please refer to Sec. 5.2.

Q2: In the experiments section, the description of the datasets is not sufficient.

We’ve followed your suggestion to add a section for detailed dataset description in App. C during rebuttal.

Q3: The authors emphasize the improvement of computational efficiency, but in the experiment, the authors only provided a rough explanation and did not provide specific improvement results..

Sorry for the confusion. We’ve added a section to discuss the training time of different methods in App. E.3.2 to show the efficiency of proposed methods.

Q4: In the first paragraph of the section 2.1, how is the representation of protein x defined? How are the sequence and structural information of proteins integrated?

We use $x$ to denote a protein instead of its representations. To obtain its representation, we will feed its sequence and structure to any protein encoder, e.g., ESM-2, GearNet or CDConv. We’ve simplified the notation in the revised version.

Q5: In the last paragraph of the section 2.3, there was no analysis of the setting of parameter τ. How the different settings of this parameter will affect performance?

Thanks for the question! We’ve added a new section discussing the effect of the temperature $\tau$ and the number of neighbors $k$ in App. E.4.

Q6: In the section 3.3 on page 4, what is the meaning of const in the formula? The authors did not provide an explanation.

The const in the equation denotes the term irrelevant with the predictor $\psi$, which can be omitted in the E-step. We’ve added an explanation in the revised version.

Q7: The authors should improve the standardization of the paper. For example: (1) In the section 3.3, the label of the first formula is missing. (2) In Figure 2 of the section 5.3, the labels for the axes are missing.

Thanks for pointing out these! We’ve fixed these in the revised version.

Thanks for your insightful comments and golden suggestions! We respond to your concerns as below:

Q1: Please cite the original BLAST publication (Altschul 1990) rather than the 2004 paper cited here. BLAST has nothing to do with functional inference, per se. It identifies evolutionary relationships.

Thanks for the suggestion! We’ve followed your suggestion to replace the citation of BLAST to the original paper.

Q2: I found retriever/predictor terminology confusing at first. Please define these terms explicitly.

Thanks for pointing this out! We’ve added a definition in Sec. 2.1 and 2.2 in the revised version.

Q3: Why BLAST is used here instead of PSI-BLAST? Have you explored using alternatives to BLAST as your retriever? Why did you use the alignment score rather than the E-value from BLAST?

Table A: F_\max for the performance of three sequence retrievers on EC and GO. (#/#/#) shows the results on the split (30%/50%/95%), respectively.

Thanks for the suggestion! We’ve follow your suggestions to include both baselines during rebuttal, the results of which are shown in Table A.

About PSI-BLAST: As shown in the table, PSI-BLAST does not perform better than BLAST on most tasks. Here is our explanation. PSI-BLAST, or position-specific iterated BLAST, is equivalent to running one round of BLASTp and use the resulting MSA to build a profile and thus position-specific score matrix (PSSM) to imply conservation patterns. The PSSM can guide another round of search, which can be repeated iteratively. However, it requires searching against large sequence database to build an effective PSSM, which performs badly in our settings since our retrieval database is simply the training set of ~20k sequences. The same rationale also applies to HHsearch/HHpred, and we note that these methods are good for searching high-quality MSA but not a good kernel for retrieval w.r.t. a limited database.

About e-value: We’d like to emphasize that e-value and alignment scores have strong correlation, which may result in the similar set of retrieved proteins. The minor difference here is how to use these scores for weighting different samples. For e-value, we need to take the logarithm and then tune the temperature to rescale the similarity. As shown in the table, there is no obvious conclusion whether e-value or alignment scores are better for function annotation.

Q4: (1) The related work is almost comically dense, as is unfortunately required by this conference format. Rather than a laundry list of existing methods, I'd prefer that you cite a couple of reviews and then use the space to explain how your method relates to the state of the art. (2) Almost all of the related work is from the last few years. In fact, people have been working hard on this problem for decades. (3) What are the two or three most closely related prior methods for solving this problem, and how does your method differ from them?

Thanks for the suggestion! We’ve followed your suggestions to reorganize the related work section and added more related work in App. A. We also discuss the contribution of this work in the end of Introduce and Related Work section.

We find that there is still a lack of systematic study of modern deep learning methods and retriever-based methods for function annotation. The most closely related work should be CLEAN [1], where they benchmark their proposed supervised contrastive learning methods with several retriever-based methods. However, there are still many deep learning baselines not considered in that paper. In this paper, we include more modern baselines in our benchmark and add another experiment on the CLEAN setup to show the advantage of our method in App. E.

From method perspective, there are many existing works using retriever-based and transductive learning methods for other machine learning applications, but there has been little effort put in the protein function annotation task. One related work along this direction is a concurrent work - RSA [2], which studies other sequence-based property prediction tasks. Our work is among the first to study how to use retriever-based methods for enhancing predictor-based methods.

[1] Yu, Tianhao, et al. "Enzyme function prediction using contrastive learning." Science 379.6639 (2023): 1358-1363.

[2] Ma, Chang, et al. "Retrieved Sequence Augmentation for Protein Representation Learning." bioRxiv (2023): 2023-02.

Thanks for your careful reading! While some questions may be due to misunderstanding, we find that your suggestions are valuable and very helpful for us to improve the quality and clarity of our paper! We respond to your concerns below:

Q1: What data set you had available for EC and GO training and testing?

Thanks for the question. In the revised version, we add more description about the datasets and splits in the App. C.

Q2: Without knowing on what data this happens and what is the relationship between training and test data, it is difficult to imagine what is happening in the experiments. For example, is are fold classification and protein function data sets coming from different distributions? If so, what is the consequence on the experimental results?

Thanks for the suggestion. We did include the source of datasets in the original version and we’ve elucidated more about the dataset in the revised version in App. C. Here the fold classification dataset is collected from SCOPe, which is a different dataset with the Swiss-Prot dataset for function annotation.

Q3: The paper is vague when it comes to what neural networks are used as encoder model and predictor model. Only deep into the experimental sections it becomes clear that the same neural network (GerNet or CDConv) is used for both. But, does that mean that there are two copies of the neural net -- one serving as an encoder and another as a predictor?

We want to clarify that it is not necessary to use the same neural network for both predictors and retrievers. Our framework is very flexible so that you can use arbitrary learnable predictors or retrievers. It is just our choice to use the same neural network (GearNet and CDConv) for predictors and retrievers in our experiments to make this simple. Also, the retriever and predictor do not share weights, as distinguished by the different notations $p_\phi$ and $q_\psi$.

Q4: The paper is inconsistent when it comes to explaining hyperparameters for the experiments

We add more explanation about hyperparameter setting in App. E.4. Could you elaborate more about where the inconsistency you find in the hyperparameter setting?

Q5: Using the structure encoders is a major limitation of this approach because it is only applicable on proteins with known structure. Proteins with known structure are a highly biased sample of the protein space and are typically well studied with a good understanding of their functional properties. This, the usefulness of proposed approach is limited.

We want to argue that relying on protein structures should not be a limitation of this work. With the advent of AlphaFold2, it is much easier and efficient to obtain accurate structures for most proteins. Also, there have been large amount of protein structures already released in AlphaFold Database, which can be easily searched for.

To illustrate this, we test our structure encoders on the CLEAN datasets in App. E.1. For most proteins in the training (retrieved) sets (Swiss-Prot), we only have the protein sequences instead of structures. To solve the problem, we simply retrieve the predicted structures from AlphaFold Database according to the UniProt ID. Although there are a small portion of proteins without structures, our structure retriever can still outperform the CLEAN method on NEW-392 dataset. This shows that our method can work well on unseen proteins with AlphaFold2-predicted structures.

Thanks for your insightful comments and great suggestions! We respond to your concerns below:

Q1: (1) Missing important baseline and citation. "Enzyme function prediction using contrastive learning" is a recent Science paper about retrieval-based predictors for protein functions. (2) Missing evaluations on realistic datasets, which are referred to paper "Enzyme function prediction using contrastive learning". (3) The key advantage of retriever-based methods is the capability to annotate unseen proteins or unseen functionality. Investigations are needed to support the effectiveness of the proposed methods.

Thanks for the suggestions! We followed your suggestion to discuss more related work and include the test sets in CLEAN paper for evaluation during rebuttal. The results are shown in App. E.1, where the results of all listed baselines are included. As it is not feasible for us to fine-tune predictor-based methods on the Swiss-Prot training set within such a short time, we only consider several retriever-based methods. It can be observed that even without any supervised training on the swiss-prot training set, our proposed retrievers can outperform CLEAN on both the NEW-392 and Price-149 datasets. Also, to utilize our structure retriever on retrieving Swiss-Prot datasets, we retrieve the predicted structures from AlphaFold Database according to the UniProt ID. Although there are a small portion of proteins without structures, our structure retriever can still outperform the CLEAN method on NEW-392 dataset, which clearly demonstrates the potential of retriever-based methods.

Q2: Missing the ensemble baseline. If a simple ensemble of predictor and retriever can achieve a good performance, then there is no need to do iterative refinement between predictor and retriever.

Thanks for the suggestion! We’ve added the ensemble baseline in App. E.3.1. As shown in Table 6, ProtIR clearly outperforms a simple ensemble baseline on all considered tasks. Only with ProtIR, can these predictors achieve comparable performance with the state-of-the-art protein language model-based methods.

Q3: Beyond existing protein encoders, there are no specific biological designs for protein function annotation problems. More problem-specific characteristics like the tree structure of go terms should be considered in the modeling.

Thanks for the suggestion. We agree that including biological design would be helpful for protein function annotation problems. We want to argue that this will not contradict with the general idea introduced in the paper. Our strategy to train neural structure retrievers and iterative refinement framework are flexible for arbitrary protein encoders, which can include any biological design, such as GearNet and CDConv. As we want to focus on more general function annotation problems and make it able to generalize to unseen functions, we do not focus on function-specific characteristics like the tree structure for GO terms. We leave this direction as future work.

Q4: For the experiments, multiple runs are needed to showcase the prediction stability.

The experiments are run on quite large training and test sets, where over 1,000 proteins are used for evaluation. For retriever-based methods, as the retrieved proteins remain the same under different random seeds, there is no need to run multiple times. For predictor-based methods, we run GearNet and CDConv on EC multiple times, where the standard errors for all these methods are below 0.003. This shows the stability of our evaluation.

Thanks for your appreciation of our work! We respond to your questions and concerns below:

Q1: One significant drawback of this paper is that it doesn't compare its approach to recent standard methods.

Thanks for the suggestions! We followed your suggestion to include the test sets in CLEAN paper for evaluation during rebuttal. The results are shown in App. E.1, where the results of all listed baselines are included. As it is not feasible for us to fine-tune predictor-based methods on the Swiss-Prot training set within such a short time, we only consider several retriever-based methods. It can be observed that even without any supervised training on the swiss-prot training set, our proposed retrievers can outperform CLEAN on both the NEW-392 and Price-149 datasets. Also, to utilize our structure retriever on retrieving Swiss-Prot datasets, we retrieve the predicted structures from AlphaFold Database according to the UniProt ID. Although there are a small portion of proteins without structures, our structure retriever can still outperform the CLEAN method on NEW-392 dataset, which clearly demonstrates the potential of retriever-based methods.

Q2: The title of Section 5.2 should be revised to accurately reflect its content.

Thanks for the suggestions! We’ve revised the experiment section to better reflect the content and re-analyzed the experimental results. We believe the questions have been solved in Sec. 5.2.

**Q3: The impression conveyed by Table 1 is that predictor-based methods are unquestionably the superior choice for functional prediction. **

Sorry for the confusion! In the revised version, we emphasize that retriever-based methods are comparable or better than predictor-based methods without pre-training. When using protein language models, predictor-based methods still outperform retriever-based methods. This can be understood because deep learning techniques efficiently leverage large pre-training datasets, enabling neural predictors to capture more evolutionary information.

Q4: The subsection titled 'Predictors vs. Retrievers' in Section 5.2 lacks clarity and fairness in its comparison. The comparison is skewed because retriever-based models like GearNet and CDConv leverage structural information that is not utilized by their predictor-based counterparts.

Sorry for the confusion. We’ve revised our expression in the latest version.

Q5: It appears that the ProtIR framework may enhance predictor-based performance; however, the authors have not opted to demonstrate this in comparison to the best predictor-based method.

This is a good point! One advantage of the ProtIR framework is that it can significantly enhance those predictors without massive pre-training. For PLM-based predictors, it would be expensive to do multiple refinement iterations to fine-tune these methods. Instead, we propose another method to improve PLM-based predictors by ensembling with a PLM-based retriever with structural insights, as shown in Sec. 5.4. We’ve made this point more clear in the revised version.

Q6: Although Section 5.4 is intriguing, it appears to have a somewhat weak connection to the primary focus of this research.

Thanks for the suggestions! We rewrite the experiment section to make its logic clear. The results in Sec. 5.4 are very important and interesting, as it presents a way that we can greatly improve PLM-based retrievers by simply pre-training it with structural information.