ProtGO: Function-Guided Protein Modeling for Unified Representation Learning

Dear Reviewer qMp2,

We are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:

Q1 Protst[1] also utilizes function information of proteins, should also be listed as a baseline.

A1 Thank you for your valuable feedback! In actuality, ProtGO does not function as a pre-training model, making it unjust to draw comparisons with pre-training methodologies. Consequently, we integrate ESM-2 (650M) [1] with ProtGO, denoted as ProtGO-ESM, where the ESM embeddings serve as a component of graph node features. Our evaluation juxtaposes ProtGO-ESM against pre-training techniques in tasks like protein function prediction and EC number prediction. This includes a spectrum of methods: sequence-based approaches such as ESM-1b [2] and ESM-2; sequence-function based models like ProtST [3]; and sequence-structure based methodologies like GearNet-ESM [4], SaProt [5], and GearNet-ESM-INR-MC [6]. The comparative findings are detailed in Table 1 of the one-page rebuttal pdf. Notably, our proposed model, ProtGO-ESM, emerges as the top performer across sequence-based, sequence-structure based, and sequence-function based pre-training strategies.

Q2 It's confusing to find, in ablation study, ProtGO still outperforms all the baselines without teacher. The authors should explain the difference between backbone GNN model and other baselines. Furthermore, the improvement of teacher-student module is trivial compared with the improvement of the backbone GNN. This needs to be clarified.

A2 Thank you for your informative reviews! (1) The student model is meticulously crafted as a Graph Neural Network (GNN) model capable of encoding protein sequences and structures simultaneously, demonstrated in Eq.4 of the manuscript. This robust protein encoder adeptly integrates protein sequences and structures, outperforming alternative sequence-structure methodologies. A sequence pooling layer is utilized to condense sequence length effectively, facilitating the aggregation of crucial patterns. In this GNN architecture, each pair of message-passing layers is succeeded by an average sequence pooling layer, totaling eight message-passing layers in the model. (2) The sequence average pooling functions execute tailored average pooling operations on input tensors based on calculated indices, dividing the sequence length by two and flooring the result. These functions aggregate and synthesize information from input tensors through scatter operations to generate output tensors. Following each average pooling layer, the number of residues is halved, expanding the radius threshold $r_s$ to $2r_s$ post-pooling. This extension enables center nodes' neighbors to encompass progressively distant and infrequent nodes, concurrently reducing computational complexity. These operations enable the model to capture both local and global features effectively. (3) The ablation study's Table 5 showcases the student model's autonomous performance without teacher guidance, demonstrating its ability to independently model protein sequences and structures. (4) Additionally, Figure 3 in the manuscript illustrates the enhancements resulting from incorporating functional information, affirming the advantages of this augmentation.

Q3 A question is that if it's necessary to introduce domain adaptation. The ablation study about this is missing.

A3 Thank you for your reviews! The theory of domain adaptation forms the foundation for deriving Eq.10 and Eq.11, as detailed in Appendix F of the manuscript. In the realm of domain adaptation, employing a supervised loss is preferred when the student model operates with distinct task labels [2].

Thank you again for all the efforts that helped us improve our manuscript! We have tried our best to address your concerns as it is important for my graduation; we respectfully thank you for supporting the acceptance of our work. Also, please let us know if you have any further questions. Look forward to further discussions!

[1] Xu, M., et al. Protst: Multi-modality learning of protein sequences and biomedical texts. In International Conference on Machine Learning. 2023.

[2] Berthelot, D., et al. Adamatch: A unified approach to semi-supervised learning and domain adaptation. arXiv preprint arXiv:2106.04732.

Dear Reviewer oeK7,

We are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:

Q1 The proposed method only demonstrate results on several small benchmarks. More results on larger scale benchmarks can be useful, e.g., residue prediction, mutation effect prediction, etc. Functions are related with the folding / reaction / GO term / EC numbers, I wonder whether it is helpful to demonstrate the performance on other problems, to demonstrate the generalization of the proposed method.

A1 Thank you for your valuable feedback! Protein design involves the computational creation of amino acid sequences that can fold into a specific protein structure. Methods like ESM-IF1 [1], PiFold [2], and VFN-IF [3] are dedicated to protein design, distinct from protein representation learning for function prediction. By focusing on protein inverse folding, we apply our approach to CATH 4.2 dataset, as detailed in Table 2 of the one-page rebuttal pdf. Our method demonstrates exceptional performance in this context, which demonstrates the generalization of the proposed method.

Q2 The proposed method should learn better functions, therefore, binding affinity prediction may be a better downstream or zero-shot task to demonstrate the model performance.

A2 Thank you for your informative reviews! We use the binding affinity prediction dataset used in [4] and [5]. DNA Binding Site Prediction Result trained on DNA-573 Train, tested on DNA-129 Test. RNA Binding Site Prediction Result trained on RNA-495 Train, tested on RNA-117 Test. We use the AUC for evaluation. Our proposed achieves best results on the binding affinity prediction, illustrating its generalization ability.

Q3 The experiment results can introduce more about the error bars.

A3 Thank you for your reviews! The mean values are reported, we have calculated the ariane of results on protein function prediction. Our results have low error bars.

Q4 I wonder how the authors avoid data leakage. What's the sequence and structure similarity (which can use AlphaFold DB to measure) between the functional annotation dataset and the other datasets.

A4 Thank you for your questions! (1) Employing the teacher-student framework, the student model learns from the teacher model's latent embeddings through knowledge distillation loss. To ensure data integrity, the test sets utilized in downstream tasks are excluded, addressing concerns regarding data leakage. (2) For the functional annotation dataset used in the teacher model and downstream datasets used in the student model, the sequence similarity between it and EC number prediction dataset is only 25%, the structural similarity is 19%; the sequence similarity between it and GO term prediction dataset is 75%. (3) Adhering to the GearNet [9] protocol, the test sets for GO term and EC number prediction exclusively comprise PDB chains with less than 95% sequence identity to the training set, allowing for the generation of varied cutoff splits.

Thank you again for all the efforts that helped us improve our manuscript! We have tried our best to address your concerns as it is important for my graduation; we respectfully thank you for supporting the acceptance of our work. Also, please let us know if you have any further questions. Look forward to further discussions!

[1] Hsu, C., et al. Learning inverse folding from millions of predicted structures. ICML, 2022.

[2] Gao, Z., et al. PiFold: Toward effective and efficient protein inverse folding. ICLR, 2022a.

[3] Mao, W., et al. De novo protein design using geometric vector field networks. arXiv, 2023.

[4] Zhang, C., et al. Us-align: universal structure alignments of proteins, nucleic acids, and macromolecular complexes. Nature methods, 2022.

[5] Zhuang, W., et al. Pre-Training Protein Bi-level Representation Through Span Mask Strategy On 3D Protein Chains. ICML. 2024.

[6] Su, H., et al. Improving the prediction of protein–nucleic acids binding residues via multiple sequence profiles and the consensus of complementary methods. Bioinformatics, 2019.

[7] Wu, Q., et al. Coach-d: improved protein–ligand binding sites prediction with re- fined ligand-binding poses through molecular docking. Nucleic acids research, 2018.

[8] Xia, Y., et al. Graphbind: protein structural context embedded rules learned by hierarchical graph neural networks for recognizing nucleic- acid-binding residues. Nucleic acids research, 2021.

[9] Zhang, Z., et al. Protein representation learning by geometric structure pretraining. ICML, 2022b.

Dear Reviewer kUCf,

We are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:

Q1 In table 1 and table 2, the authors may consider add ESM-1b/ESM-2/ESM-3 as sequence modality baselines.

A1 Thank you for your valuable feedback! In actuality, ProtGO does not function as a pre-training model, making it unjust to draw comparisons with pre-training methodologies. Consequently, we integrate ESM-2 (650M) [1] with ProtGO, denoted as ProtGO-ESM, where the ESM embeddings serve as a component of graph node features. Our evaluation juxtaposes ProtGO-ESM against pre-training techniques in tasks like protein function prediction and EC number prediction. This includes a spectrum of methods: sequence-based approaches such as ESM-1b [2] and ESM-2; sequence-function based models like ProtST [3]; and sequence-structure based methodologies like GearNet-ESM [4], SaProt [5], and GearNet-ESM-INR-MC [6]. Notably, the outcomes of ESM-3 [7] are pending due to resource constraints. The comparative findings are detailed in Table 1 of the one-page rebuttal pdf. Notably, our proposed model, ProtGO-ESM, emerges as the top performer across sequence-based, sequence-structure based, and sequence-function based pre-training strategies.

Q2 Also table 1 and 2, the authors may consider add ESM-IF1 as structural modality baselines.

A2 Thank you for your informative reviews! Protein design involves the computational creation of amino acid sequences that can fold into a specific protein structure. Methods like ESM-IF1 [8], PiFold [9], and VFN-IF [10] are dedicated to protein design, distinct from protein representation learning for function prediction. By focusing on protein inverse folding, we apply our approach to CATH 4.2 dataset, as detailed in Table 2 of the one-page rebuttal pdf. Our method demonstrates exceptional performance in this context, nearly outperforming other approaches in this task.

Q3 Also table 1 and 2, the authors may consider add GO/sequence+GO modality as input, add DeepGO-SE as baselines.

A3 Thank you for your reviews! We have compared our model, ProtGO-ESM, with pre-training methods with sequence, sequence-structure and sequence function as inputs; the results are shown in Table 1 in the one-page rebuttal pdf.

Q4 I am interested in the downstream task performance differences for proteins with GOs (where the teacher can teach the student well) and proteins without GOs (where the teacher model may fail). Moreover, GO terms consist of 3 parts, molecular functions, cellular components, and biological processes, I wonder whether the authors could explore the effectiveness of each part through comprehensive downstream experiments.

A4 Thank you for your reviews! (1) The effectiveness of the teacher model in instructing the student is notably higher for GO terms with a higher frequency. Conversely, our experiments reveal that when the frequency of a GO term falls below 50, the teacher model may struggle. For instance, in the case of the GO term GO:0030027, denoted as lamellipodium, the performance of the teacher model is suboptimal. (2) Through experimental analysis, we segmented GO terms into three categories: molecular functions (MF), cellular components (CC), and biological processes (BP). By utilizing only one category of GO terms as input for the GO encoder of the teacher model, we aimed to assess the efficacy of each category. Our findings indicate that focusing on a single category, such as MF, primarily enhances predictions related to MF in subsequent tasks. While specialization in a specific category can improve accuracy within that domain, it may not directly translate to improved predictions in the other two categories (BP and CC). This divergence stems from the distinct biological aspects represented by each category, each with its unique characteristics and interrelations.

Q5 Minor: In line 4, the authors wrote "including sequence, structure, domains, motifs, and...", generally domains and motifs are structural annotations, they should not be listed together with sequence and structure.

A5 Thank you for your suggestion! We will rectify this in the revised version.

Thank you again for all the efforts that helped us improve our manuscript! We have tried our best to address your concerns as it is important for my graduation; we respectfully thank you for supporting the acceptance of our work. Also, please let us know if you have any further questions. Look forward to further discussions!

[1] Lin, Z., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.

[2] Rives, A., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2021.

[3] Xu, M., et al. Protst: Multi-modality learning of protein sequences and biomedical texts. ICML, 2023.

[4] Zhang, Z., et al. Protein representation learning by geometric structure pretraining. ICML, 2022b.

[5] Su et al. SaProt: Protein Language Modeling with Structure-aware Vocabulary. ICLR, 2024.

[6] Lee, Y., et al. Pre-training sequence, structure, and surface features for comprehensive protein representation learning. 2023.

[7] Hayes, T., et al. Simulating 500 million years of evolution with a language model. bioRxiv, 2024.

[8] Hsu, C., et al. Learning inverse folding from millions of predicted structures. ICML, 2022.

[9] Gao, Z., et al. PiFold: Toward effective and efficient protein inverse folding. ICLR, 2022a.

[10] Mao, W., et al. De novo protein design using geometric vector field networks. arXiv, 2023.