($\texttt{PEEP}$) $\textbf{P}$redicting $\textbf{E}$nzym$\textbf{e}$ $\textbf{P}$romiscuity with its Molecule Mate – an Attentive Metric Learning Solution

We sincerely thank the reviewer for providing the valuable feedback. We have provided responses to your questions and we hope that they can address your concerns.

Weakness 1: Technique Contributions? We respectfully disagree. While we do utilize MoCo in the protein domain, our key innovations lie in integrating biological insights into various facets of our approach. These aspects are not explicitly considered in previous machine learning algorithms for protein functional annotation. In our framework, we have leveraged the attention mechanism to capture residue importance, introduced a hierarchical loss term customized for functional annotations, and integrated ligand embeddings to enrich the representations. These elements collectively contribute to our framework's efficacy. Our detailed experimental results (summarized below) underscore that these incorporations, rather than using MoCo alone, are instrumental in achieving the performance enhancements we have observed.

Weakness 2: Sizes of datasets? We would like to clarify that the datasets utilized in our experiments are not characterized as small. Specifically, for the training sets, we worked with a substantial number of protein sequences, with counts of 8K/10K/220K when employing different identity cutoffs of 10%/30%/100%, respectively. As for the testing set, which includes the Price, New, and CATH subsets, the respective sequence counts are 149/392/98. It is important to highlight that these dataset sizes align with those commonly encountered in the existing literature and are considered challenging, as demonstrated in the recent study by Yu et al. (2023).

Dear Reviewer QFnk,

We thank Reviewer QFnk for spending time to review our work and providing valuable feedbacks. We have provided responses to your question regarding the technical contribution. As the rebuttal deadline is drawing near, we hope you could take a look at our responses and see if they have addressed your concerns. Thank you again for your time and effort.

Best,

Authors

We genuinely thank the reviewer for saying that our idea is interesting. We have provided answers to your questions in detail.

Weakness 1: Lacking explanation? Thank you for your valuable feedback. In the manuscript, we have included a paragraph explaining the EC number in Section 3.1. In response to your suggestion, we have also incorporated an introductory mention of the EC number in the second paragraph of the revised draft to provide additional context.

Weakness 2: Difference between CLEAN and PEEP. The primary disparity between our approach and CLEAN lies in our incorporation of several techniques that account for biological priors across various dimensions: (1) at the input level, we fuse the representation of ligands to provide substrate-scope information; (2) at the model level, we adopt attention mechanism to learn functional residues; (3) at the objective level, we propose a metric learning objective to capture hierarchical nature of EC numbers. These enhancements collectively set our method apart from CLEAN and underline the motivations behind our approach.

Question 1: How to use attention modules to learn functional residues? The transformer layer was chosen to replace global average pooling used by CLEAN based on the insight that the residues responsible for endowing an enzyme function make up a small amount of the sequence, thus, global average pooling will significantly weaken this signal. By using attention, we instead learn a weighted pooling that weights residues, and their importance, towards enzyme function.

In Figure 2 and 4, we have provided evidence of the concordance between these attention values and the predicted significance of residues. As such, these introduced modules play a crucial role in enhancing the learning process for functional residues.

We thank the reviewer for acknowledging the domain knowledge embedded in our designs. We have provided answers to your questions below in detail and hope they can address your concerns.

Weakness 1: Novelty of our framework? While some of our techniques resemble some prevalent ones, it is also important to note that our proposed method has been recognized as novel by Reviewer nNj8 and QFnk. The novelty of our method primarily stems from its design, which is driven by specific challenges in the biological domain. This includes three key innovations: (1) the functions of proteins are highly correlated with their active sites; (2) the functional annotations have hierarchical structures; and (3) embeddings of ligands can be leveraged as additional sources of information for functional annotations. These elements, not explicitly considered in prior work, are central to our innovative approach. We have incorporated an attention mechanism to capture residue importance, introduced a hierarchical loss term customized for functional annotations, and integrated ligand embeddings to enrich the representations. Empirical evidence supports the effectiveness of these integrations in enhancing functional annotation performance.

Weakness 2: Significance of the performance improvement? We performed an additional analysis to assess the significance of the performance improvements on the Price-149 and CATH datasets, both of which were subjected to three separate experiments under a 10% sequence identity cutoff. Our results revealed error bars of approximately 0.007 and 0.011 for Price-149 and CATH, respectively, demonstrating the statistical significance of the observed improvements.

Weakness 3: Performance Comparison in Table 2. Thank you for your observation! It is worth noting that when comparing the results between the 4th and 5th rows of Table 2, we observe improvements in four metrics out of six. As such, we continue to believe that the combination of all techniques remains the most favorable choice.

Question 1: The fairness of comparison with baselines. In our study, we have ensured a fair comparison with baseline methods. For the training phase, we have used identical training datasets to (re-)train all models. For the evaluation, all the models including baselines, are evaluated on identical sets of test data. This ensures the fairness of our comparison. We have incorporated this benchmarking setup description into the draft to enhance its clarity and presentation.

Question 2: Motivation for de-duplicating training data. The motivation of using different clustering thresholds is to decrease the similarity between the training and the testing set and evaluate the model’s ability to generalize. This is in line with your understanding on the importance of the difference between training and testing examples in changing the difficulty of extrapolation tasks. In response to your inquiries, we conducted an assessment of relative Levenshtein distances (i.e., divided by the length of the sequence) between sequences in the training and testing sets. For every sequence in the testing set, we average the distance between it and 5 samples from the training set with the smallest Levenshtein distances. The table presented below illustrates that as the thresholds decrease, the distances between these sequences tend to increase. This indicates that adjusting the clustering threshold effectively enhances the distinctiveness of the training and testing data.

Question 3: Clarifying the statement in introduction. Thank you for pointing out this ambiguity in our statement. Yes, this number refers to the number of sequences with experimental annotation in UniProt SwissProt. We will clarify that this refers to SwissProt in the main text.

Question 4: Quantitative evaluation of active sites detection. Following your suggestion, we have established another evaluation protocol to judge the performance of active sites detection, using the accuracy and the precision at residue levels. Our method achieves an average accuracy of 48% and an average precision of 73% on the New-392 dataset. We have also benchmarked ProteInfer, using the class activation mapping (CAM) technique to identify functional localization, on New-392, which shows an average accuracy of 49% and an average precision of 54%. Note that both methods are trained on the same split of data (10% identity), and the comparisons are conducted on test samples that are in the training set, a prerequisite required by ProteInfer. Our results exhibit much higher precision while having the same level of accuracy. We have included these new results in the revision.

We appreciate the efforts made by the reviewer for providing the detailed and valuable comments. We have provided responses in detail and we hope they can address your concerns.

Weakness 1: Concept of Promiscuity We appreciate and agree with your comments. Your understanding of enzyme promiscuity is correct. While we have proteins with multiple EC numbers in our testing datasets, we are not directly tackling the promiscuity problem. We have updated the title of the paper and also the draft to make it purely about enzyme function and not enzyme promiscuity.

Weakness 2: Motivation of using attention? We would like to clarify that our motivation for deploying attention mechanisms is based on specific scientific reasoning and not post-hoc. In the context of protein analysis, it is well-established that not all residues contribute equally to the protein's function. Traditional methods that assign uniform weights to all residues often overlook the nuanced contributions of key functional residues. Our choice of attention mechanisms is driven by their ability to dynamically emphasize these functionally significant residues. This is not merely following a common trend, but a strategy to enhance model accuracy. These attention modules allow our model to adaptively focus on residues that are crucial for the protein's activity, thereby offering a more precise representation of functional sites compared to uniform weighting. Empirical evidence supports the effectiveness of attention mechanisms. In our experiments, deploying the attention mechanism consistently demonstrated superior performance. This improvement is attributable to the model's ability to learn and prioritize residues that are more relevant to the protein's function. Thus, our adoption of attention mechanisms is a design that aims at addressing the specific challenges in protein function prediction.

Weakness 3: Hierarchical label training is not biology-specific We appreciate the reviewer's point that hierarchical labels are not exclusively used in biological contexts. However, we demonstrate that EC numbers’s ontology is well-suited for the hierarchical label strategy and we demonstrate that it leads to improved performance. Thus, demonstrating the use of domain knowledge via hierarchical labeling is beneficial in prediction protein function from sequence. An insight that could benefit the protein machine learning community.

Here are a few examples on how combining hierarchical labels with domain knowledge is beneficial:

• For example, [r1] incorporates class hierarchy from WordNet [r2] as a source of domain knowledge to improve the performance of visual classification and [r3] uses label hierarchy to improve concept classification. Our technique applies this concept to the domain of functional annotation. By adjusting the loss function to reflect protein similarities based on their functional hierarchy, our model effectively integrates domain-specific knowledge.

In summary, while hierarchical labeling is a technique used across various fields, its implementation in our model is specifically tailored to address the complexities of biological data. This approach aligns with established practices in the literature, demonstrating our model's effective absorption and utilization of domain-specific knowledge.

[r1] Integrating Domain Knowledge: Using Hierarchies to Improve Deep Classifiers

[r2] WordNet: A Lexical Database for English

[r3] Domain-Specific Knowledge Acquisition and Classification using WordNet

Dear Reviewer siYE,

We extend our gratitude to Reviewer siYE for dedicating time to review our work and for the valuable constructive comments provided. We have meticulously addressed your questions in a point-by-point fashion. As the rebuttal deadline is drawing near, we hope you could take a look at our responses and see if they have addressed your concerns. Thank you again for your time and effort.

Best,

Authors

We greatly appreciate the reviewer's recognition of the quality of our paper's writing and its empirical performance. We have prepared detailed responses to your comments, and we hope that they can address your concerns.

Weakness 1: revision of the writing? Thank you for your insightful feedback. We have revised the manuscript and adjusted the sentence to enhance clarity, addressing your suggestion as follows: “To meet the goal, we customize a self-attention mechanism (Figure 1, c) to model the residue importance within protein sequences,” has been modified to “To meet the goal, we deploy a self-attention mechanism (Figure 1, c) to model the residue importance within protein sequences,”.

Weakness 2: lack of novelty? While some of our techniques resemble some prevalent ones, it is also important to note that our proposed method has been recognized as novel by Reviewer nNj8 and QFnk. The novelty of our method primarily stems from its design, which is driven by specific challenges in the biological domain, including three key aspects: (1) the functions of proteins are highly correlated with their active sites; (2) the functional annotations have hierarchical structures; and (3) embeddings of ligands can be leveraged as additional sources of information for functional annotations. These elements, not explicitly considered in prior work, are central to our innovative approach. We have incorporated an attention mechanism to capture residue importance, introduced a hierarchical loss term customized for functional annotations, and integrated ligand embeddings to enrich the representations. Empirical evidence supports the effectiveness of these integrations in enhancing functional annotation performance.

Question 1: more information about ligands? Thank you for your feedback. To address your concern, we have included a histogram illustrating the distribution of ligands associated with proteins in our training dataset in Appendix. We can see most EC numbers have less than 10 associated ligands.

We would like to emphasize that our random selection process is performed iteratively at each training step, ensuring that all ligands are utilized in the training procedure. We have also conducted multiple experiments specifically on the Price-149 dataset, employing a sequence identity cut-off of 10%. These experiments reveal a negligible error bar of 0.007, underscoring that the random selection process introduces only minor fluctuations in the results.

Question 2: methods for fusion? We have chosen the second option, which involves fusing protein representations with zero vectors. For a detailed performance comparison between the two methods, you can refer to Appendix A3.2. To make it more convenient, we have included the results in the table below: