Learning Complete Protein Representation by Dynamically Coupling of Sequence and Structure

Dear Reviewer JwZm,

We are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:

Q1 Comparisons with state-of-the-art protein language models and their structure-aware versions.

A1 Thank you for your valuable feedback! Factually, CoupleNet is not a pre-training model; it is not fair to compare it with pre-training methods. Thus, we combine ESM-2 (650M) [1] with CoupleNet, named ESM-CoupleNet, using generated ESM embeddings as one part of graph node features. We compare ESM-CoupleNet with pre-training methods on protein function prediction and EC number prediction, including sequence-based methods, ESM-1b [2], ESM-2; sequence-function based methods, ProtST [3]; sequence-structure based methods ESM-S [4], GearNet-ESM [5], SaProt [6]. The comparison results are provided in Table 1 of the one-page rebuttal pdf. From this table, we can see that our proposed model, ESM-CoupleNet, achieves the best results among sequence-based, sequence-structure based, and sequence-function based pre-training methods.

Q2 Writing quality.

A2 Thank you for your suggestions! （1）In the following sections, we have definitions about the edge $\varepsilon_{ij}$, it is important to ensure the preciseness of the graph definition. (2) As shown in Section 3.1, in the definition of Invariance and Equivariance, the transformations $T_g$ and $S_g$ represent the actions of the symmetry group (like rotations and translations). (3) The range of $\mathcal{F}(\cdot)$ includes the geometric representations of the input 3D graphs, which are as listed in the paper, like $\Phi, \Psi, \Omega, \omega, \theta, \varphi$. (4) We present Eq. 4 as we update the message-passing mechanism in Eq. 10, and we will rectify our descriptions. (5) Eq. 5 is the definition of the local coordinate system as first presented in [7], which is shown in Figure 7(a) in the appendix. The symbol $\times$ represents the cross product operation. (6) As shown in Appendix F, the input/output dimensions of the FC layer in Eq.10 are 256. We do not update the edge features $\boldsymbol{e}_{ij}$ by the neighboring node features, which is different from GearNet [8].

Q3 On line 133, it is mentioned that the positions can be recovered from the geometric representations.

A3 Thank you for your reviews! The definitions of complete geometric representations are preliminaries proposed by [9]; this is correct as the geometric representations $\mathcal{F}(G)$ are complete, the structures can be calculated by these features if we use complete geometries.

Q4 Questions about average pooling.

A4 Thank you for your valuable reviews! As shown in Section 3.4, Appendix F and Figure 2, we employ complete message passing and sequence pooling layers to obtain the deeply encoded graph-level representations. Every two message-passing layers are followed by an average pooling layer. There are eight message-passing layers in the model. The sequence average pooling functions perform customized average pooling operations on the input tensors based on the calculated indices (dividing the length of the sequence by 2 and flooring the result). It aggregates and summarizes information from the input tensors using scatter operations to produce the output tensors (torch_scatter.scatter_mean). (2) After one average pooling layer, the number of residues reduces by half; for example, only one of the two consecutive graph nodes ($v_i, v_{i+1}$) remains. We reconstruct the graphs by Eq. 9 by expanding the radius threshold $r$ to $2r$ after once pooling, which makes neighbors of center nodes gradually cover more distant and rare nodes, also reducing the computational complexity.

Q5 On lines 227-228 in Section 3.4...

A5 Thank you for your comments! In Lines 227-228, we stated that the message passing mechanism only executes on nodes in CoupleNet instead of on nodes and edges alternately used in GearNet. CoupleNet is our proposed model.

Q5 The lack of sufficient structural data as compared to sequential data for proteins...

A5 Thank you! For some proteins, only sequential data is available, we can consider using the AlphaFold predicted structures. This model is only developed for modeling protein sequences and structures concurrently. In the era of AlphaFold, it is a trend to modeling protein sequences and structures, using the structural information to enahce the model's performance.

Thank you again! In case our answers have justifiably addressed your concerns, we respectfully thank you that support the acceptance of our work. Also, please let us know if you have any further questions. Look forward to further discussions!

[1] Lin, Z., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.

[2] Rives, A., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2021.

[3] Xu, M., et al. Protst: Multi-modality learning of protein sequences and biomedical texts. ICML. 2023.

[4] Zhang, et al. Structure-informed protein language model. arXiv, 2024.

[5]  Zhang, et al. A systematic study of joint representation learning on protein sequences and structures. arXiv, 2023.

[6] Su et al. SaProt: Protein Language Modeling with Structure-aware Vocabulary. ICLR, 2024.

[7] John, et al. Generative models for graph-based protein design. NeurIPS, 2019.

[8] Zhang, et al., Protein Representation Learning by Geometric Structure Pretraining. ICLR. 2023.

[9] Liu, et al. Spherical message passing for 3d molecular graphs. ICLR, 2021.

Dear Reviewer ANTD,

We are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:

Q1 The experiment results do not include error bars and related analysis about multiple trials.

A1 Thank you for your valuable feedback! We have done multiple trials. The performance is measured with mean values for five different initializations. We report the mean (variance) for the proposed CoupleNet:

Q2 The proposed method looks quite general. Applying the architecture to more real-world problems, e.g., residue design / engineering may be helpful. Or, including some discussions about these topics may be useful.

A2 Thank you for your informative reviews! Protein design is the computational approach to designing amino acid sequences that fold into a predefined protein structure. ESM-IF [1], PiFold [2], and VFN-IF [3] are methods aiming to do protein design, where the task differs from protein representation learning of function prediction. Removing the sequence information, we do protein inverse folding by our method on CATH 4.2, the results are shown in Table 2 in the one-page rebuttal pdf. Our method also achieves almost the best results on this task.

Q3 For the structure module, is the model robust to perturbations? Or, the model can aware of small changes in the structures? NMR PDB files which includes multiple models can be a test set.

A3 Thank you for your reviews! We have conducted the noise analysis experiments, the results are shown in Section 4.2. Proteins in the test set are categorized into four groups based on their similarity to the training set, 30%, 40%, 50%, 70%, not by the default split rate of 95%. The results indicate that even when there is a low similarity between the training and test sets, our model also has the highest scores, which demonstrates the robustness of the proposed model. The model can be aware of small changes in the structures, as we have constructed the complete structural representations to ensure global completeness; this has been demonstrated in Appendix H. Such global completeness is theoretically guaranteed to incorporate 3D information completely without information loss, while the local view would miss the long-range effects of subtle conformational changes happening distantly.

Q4 The structure files are not available for real problems. I wonder what's the difference (in the model output space) between Alphafold structures or other folded structures and the real structures? Especially, on multiers or interface data.

A4 Thank you for your valuable reviews! The proposed model is a basic framework to model protein sequences and strucutres, if the real structures are not available, we can use Alphafold predicted structures. We have not consider the difference between predicted structures and the real structures, as in experiments, the strucutres we used are all from PDB files. The proposed model can also model protein complex.

Thank you again for all the efforts that helped us improve our manuscript! In case our answers have justifiably addressed your concerns, we respectfully thank you for supporting the acceptance of our work. As you know, your support holds great significance for us. Also, please let us know if you have any further questions. Look forward to further discussions!

[1] Hsu, C., et al. Learning inverse folding from millions of predicted structures. ICML, 2022.

[2] Gao, Z., et al. PiFold: Toward effective and efficient protein inverse folding. ICLR, 2022.

[3] Mao, W., et al. De novo protein design using geometric vector field networks. arXiv, 2023.

[4] Zhang et al. Protein Representation Learning by Geometric Structure Pretraining. ICLR, 2023.

Dear Reviewer fLEX,

We are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:

Q1 I suggest the authors concisely compare those works and elaborate on the advantages of the proposed method.

A1 Thank you for your valuable feedback! (1) As shown in Lines 221-229 in the manuscript, there are only two different types of graphs in the proposed CoupleNet: radius graph and sequential graph; we did not use the k-nearest graph, as some nodes have distant neighbors when having $k$ neighbors. (2) The threshold in the radius graph in GearNet is set to be constant, but we change the threshold of radius dynamically to learn different distance relationships. (3) The message passing mechanism only executes on nodes in CoupleNet instead of on nodes and edges alternately used in GearNet. CoupleNet performs convolutions on nodes and edges simultaneously with several pooling layers. (4) ESM-GearNet combines ESM embeddings with GearNet embeddings; SaProt also uses ESM embeddings; it only uses the $C_\alpha$ coordinate at the structure level, but CoupleNet models the coordinates of all backbone atoms completely.

Q2 Comparisons with the related works.

A2 Thank you for your informative reviews! Factually, CoupleNet is not a pre-training model; it is not fair to compare it with pre-training methods. Thus, we combine ESM-2 (650M) [1] with CoupleNet, named ESM-CoupleNet, using generated ESM embeddings as one part of graph node features. We compare ESM-CoupleNet with pre-training methods on protein function prediction and EC number prediction, including sequence-based methods, ESM-1b [2], ESM-2; sequence-function based methods, ProtST [3]; sequence-structure based methods ESM-S [4], GearNet-ESM [5], SaProt [6]. The comparison results are provided in Table 1 of the one-page rebuttal pdf. From this table, we can see that our proposed model, ESM-CoupleNet, achieves the best results among sequence-based, sequence-structure based, and sequence-function based pre-training methods.

Q3 The ablation study is not related to the core techniques of this paper.

A3 Thank you for your reviews! (1) In this paper, a novel two-graph-based approach for modeling sequential and 3D geometric features is proposed, ensuring global completeness in protein representation, which has been demonstrated in the appendix. We have done the ablations on the sequential and radius graphs by removing different input features, as shown in Table 3 in the paper. (2) CoupleNet performs concurrent convolutions and sequence poolings on nodes and edges. Convolution and pooling operations consist of the network. We remove the pooling operations in the network, and the results are shown in the following table. We can see without pooling operations the model's performance degrades significantly.

Besides, in Figure 4 of the manuscript, CoupleNet demonstrates its capability to capture long-range relationships; higher accuracies are observed for relatively large proteins with sequence lengths surpassing the mean length. This also illustrates the effectiveness of the dynamic pooling operations.

Q4 Questions about pooling strategy.

A4 Thank you for your valuable reviews! (1) As shown in Section 3.4, Appendix F, and Figure 2, we employ complete message passing and sequence pooling layers to obtain the deeply encoded graph-level representations. Every two message-passing layers are followed by an average pooling layer. There are eight message-passing layers in the model. The sequence average pooling functions perform customized average pooling operations on the input tensors based on the calculated indices (dividing the length of the sequence by two and flooring the result). It aggregates and summarizes information from the input tensors using scatter operations to produce the output tensors (torch_scatter.scatter_mean). (2) After one average pooling layer, the number of residues reduces by half, and we expand the radius threshold $r$ to $2r$ after once pooling, which makes neighbors of center nodes gradually cover more distant and rare nodes, also reducing the computational complexity. These operations make the model capture the local to the global features.

Q5 Could you explain the advantages of this work from the difference with existing methods in Section 3.4?

A5 Thank you! We have stated the differences in Section 3.4. We will rectify it to explain extra advantanages in the revised version. For example, our model are more memory-efficient as we adopt dynamic (hierarchical) pooling to reduce the seuqnce length, the dynamically changed graphs can better model the node-edge relationships. We model the sequential and 3D geometric features, ensuring global completeness in protein representations.

Thank you again! In case our answers have justifiably addressed your concerns, we respectfully thank you that support the acceptance of our work. Also, please let us know if you have any further questions. Look forward to further discussions!

[1] Lin, Z., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.

[2] Rives, A., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2021.

[3] Xu, M., et al. Protst: Multi-modality learning of protein sequences and biomedical texts. ICML. 2023.

[4] Zhang, et al. Structure-informed protein language model. arXiv, 2024.

[5]  Zhang, et al. A systematic study of joint representation learning on protein sequences and structures. arXiv, 2023.

[6] Su et al. SaProt: Protein Language Modeling with Structure-aware Vocabulary. ICLR, 2024.