VN-EGNN: E(3)- and SE(3)-Equivariant Graph Neural Networks with Virtual Nodes Enhance Protein Binding Site Identification

We thank the reviewer for acknowledging our efforts to advance the state of the art in methods for protein binding site identification. We thank the reviewer for stating our main contributions so clearly and hope that we can answer their remaining questions in a satisfying way.

Questions

Question 1: Confidence Score's Role in DCC/DCA Metrics ...

With our architecture, we obtain two output values for each virtual node: a confidence score and a predicted binding site location. Since, the DCC/DCA metric computation require as many predictions as experimentially found binding sites, we provide the site location predictions from the highest-scoring virtual nodes for computation of DCC/DCA. To achieve high DCC/DCA values, both the prediction of the binding site location and the prediction of the confidence score must work well. Conversely, it applies, that in case confidence predictions would not work well, DCC/DCA would be low. A further aspect concerning the confidence score with respect to potentially weak signals is, that even if lower-confidence predictions would accurately identify binding sites correctly, they are excluded from the metric calculation. In light of having obtained new state-of-the-art results, our argumentation would be that confidence prediction works well.

Question 2: Virtual Node Clustering Implementation ...

The clustering process is done exclusively at inference time (we make this clear in the updated version of the manuscript). Since the number of virtual nodes used, is in almost all cases larger than the number of binding sites of the considered protein, it’s likely that several virtual nodes might get very close to the true binding site and to each other (with negligible numeric differences in the locations of the virtual nodes). Figure I1 (right) illustrates such a case, while Figure I1 (left) shows location predictions for another protein, which are spatially more distributed. In general, the case of using multiple virtual nodes to converge to nearby locations is more frequently observed for smaller proteins.

Question 3: Practical Implications for Binding Site Identification ...

From a practical point of view, we expect that adjusting the number of virtual nodes towards a value, which is slightly larger than the number of usually observed binding sites, might be interesting to users. There could be strong binding interaction events with large confidence values, but maybe also weaker ones with lower confidence scores, which might however be still interesting for further investigation.

Question 4: Overall a few sentences about the scaling in memory and computation according to protein size could be interesting ...

Since we work on neighborhood graphs and the neighborhood as well as the number of virtual nodes are usually quite limited (assumed to be a constant value), we expect that scaling properties behave linear in the protein size. We explicitly observed linear memory increase with a growing number of virtual nodes. The majority of the proteins in our dataset have not more than 8 annotated binding sites, which are modeled as virtual nodes in our architecture and which justifies using a constant value of 8 virtual nodes. We provide insights on memory allocation for an increasing number of virtual nodes in Figure M1.

Question 5: Finally, do the authors have any ideas ...

The reason why we specifically considered up to 5 layers, seems to have been caused by initial experimentation, where we followed the general paradigm "the deeper, the better" and obtained decent results, but we agree with the reviewer's thought that other hyperparameter choices could lead to competitive performance. The reason why five layers perform well might not be due to enhanced expressiveness, but rather could be because the learning behavior improves with additional layers. We add an additional ablation (see Figure L1) with different numbers of layers, which shows that decent results are achieved even with fewer layers.

I would like to thank the authors for their rebuttal to my initial review. In general, my opinion of this paper is unchanged: I think the practical contributions of this work are clearly demonstrated through the authors' experiments, though from a theoretical (and data curation) perspective, there is certainly room for improvement for this problem domain. Even though previous works have used datasets that are not necessarily filtered to strictly reduce overlap between training and test splits, I think the authors should consider revisiting this concern in follow-up work if possible. As such, I will hold my score at a weak accept (6) for now.

We thank the reviewer for providing a balanced review highlighting both the strengths and weaknesses, and for the possibility of potentially increasing the score. Below are our responses to the weaknesses and the question. We are grateful for making us aware of typos and even sharing thoughts about promising future directions.

Weakness 1: The authors' claim...

The work the reviewer mentioned has already referenced our research. At the time of their citation, our work was available in a preprint format.

Weakness 2: In Appendix G.1...

Here we adopted benchmarks from the community working on machine learning methods for binding site identification, especially from EquiPocket, which was recently published at this year’s ICML. This means that for comparability reasons, we used scPDB for training/hyperparameter selection and PDBBind, COACH420, and HOLO4K for testing. While we did not explicitly check for overlaps, it is worth mentioning that these datasets exhibit different characteristics. As shown in Figure J1 the only datasets comprising multiple chains are PDBBind2020 and HOLO4K. Further, the only datasets comprising multiple binding sites are COACH420 and HOLO4K. The training dataset does neither contain multiple binding sites nor multiple chains.

Weakness 3: Based on their...

The reviewer brings up an important topic: sensitivity to chirality. First of all it should be noted that the general scheme of VN-EGNN inherits the E(3) transformation equivariance/invariance properties of EGNN, which does especially mean predictions might get invariant to global reflections. For proteins, reflection invariance is not necessarily a desirable property since protein characteristics might change under reflection (which they usually are not considered to do under translation or rotation). To avoid reflection invariance in the context of protein graphs, it is sufficient to change node embeddings in case reflections occur, i.e., use other embeddings when the protein is reflected. In VN-EGNN, we encode amino acids by an ESM encoder. This encoder has been trained on tokens, which represent L amino acids. D amino acids have possibly been ignored in the ESM encoder, since they occur very rarely. We extend the ESM encoder by tokens representing D amino acids and initialize the embedding matrix randomly. Using an extended ESM encoder in that way, we encode L- and D-forms differently, and since amino acid residues change from an L-form to a D-form or vice versa under reflection, reflection invariance is broken.Thus reflection sensitivity is achieved. Additional comments:

• Such a random extension of an ESM encoder does not necessarily provide a very meaningful encoding of D amino acids, but allows to break reflection symmetry.
• On the other side there is lack of training data anyway for D amino acids, such that there is the question whether a more meaningful (than random) D amino acid encoding could be trained at all.
• Randomly extending the encoder for D amino acids should allow to break reflection symmetry for global reflections. In practice the occurrence of global reflections of proteins might however not at all be a relevant case. The aspect of chirality is more interesting with respect to local subparts of proteins. For this, global reflection symmetry of the architecture itself, is however not a problem.

Weakness 4: Concerning line 416...

We agree with the reviewer that requiring knowledge of the true number of binding sites for evaluation presents a general challenge in the field of binding site prediction. This limitation is indeed not specific to our work but affects all current approaches. However, this approach still allows a fair quantitative comparison between different methods in this research setting. The reviewer is right that there might be other more appropriate measures like IoU instead of computing a success rate with a predefined number of binding pockets.

Weakness 5: Regarding the fifth...

We thank the reviewer and adopt the sentence to the suggestion of the reviewer.

Weakness 6: If possible, releasing...

We provide an anonymized version of our source code at: https://anonymous.4open.science/r/vnegnn-code-3D77

Question:

Thanks for making us aware of that; we changed to "such that all eigenvalues of all other eigenvectors"

D5. Utilization efficiency of virtual nodes

From Figs. 2 and I1, we can see that there may be multiple virtual nodes converging to the same binding site. Can we improve the utilization efficiency of virtual nodes by introducing MMD Loss or other optimal transmission Loss like FastEGNN?

D6. Difference between E(3)-equivariance and SE(3)-equivariance

In fact, if the authors look up the entry on chirality in Wikipedia [b], they will find that chirality does not affect physical properties. The difference in biochemical properties is due to the fact that the human body does not undergo the same Euclidean transformation, that is, the entire system is not geometrically isomorphic to the original system. The task of this article does not seem to have such a quantity, so I think the chirality explanation here is far-fetched.

[b] https://en.wikipedia.org/wiki/Chirality_(chemistry)

We thank the reviewer for the encouraging words on scalability and hope that our responses to weaknesses might help to reconsider their initial assessment and view our paper more favorably for acceptance.

Weakness 1:

The reviewer is right that the suggested GNN architecture might also be suited for other applications than just Protein Binding Site Identification. An important aspect in creating a machine learning model with high prediction accuracy is however also the design of the neural network architecture itself as well as associated new learning schemes (extension of loss function, extension of learning algorithm), . We consider exactly this to be the exploitation of prior knowledge as questioned by the reviewer. In the manuscript we tried to argue why exactly this network architecture might make a lot of sense for protein binding site identification and we would like to point out that more generic network architectures such as GAT, SchNet, or, EGNNs showed worse performance.

It is important to acknowledge that there is often a trade-off between generic learning approaches and those that are more specialized and application-specific. The effectiveness of our suggested network architecture may be attributed to the limited amount of available training data in the field of protein binding site identification, which might make it necessary to use a more specialized network architecture to obtain state-of-the-art results. Whether the usage of virtual nodes and an associated loss function as we suggested, e.g., might be competitive for application fields with hundreds of thousands of data points is questionable.

For the mentioned reasons, we would like to remark that we (1) consider our paper to be an application paper (i.e., we submitted it to ICLR to the category “applications to physical sciences (physics, chemistry, biology, etc.)”), and, (2) that we discuss the limitation to the specific application domain in the Limitations-section of the paper. We add to the limitations section, that the reason why our network architecture works especially well for binding site identification might be the relatively low number of training data points.

Weakness 2:

We try to improve the text. What we actually wanted to bring over is:

• From an application point of view, having strict equivariance is a desirable property for a binding site identification algorithm in principle. In its core our suggested architecture is able to fulfill this property.
• However from a practical point of view it seems advantageous not to restrict the learning architecture too much. Therefore, we came up with a more approximate version combined with data augmentation (random rotations during training) of our architecture, which we are using in practice. We found this approach led to a more diverse spatial distribution across the protein surface while maintaining model performance.
• In a wider context, we would like to remark that AlphaFold 3 dropped model architecture restrictions with respect to equivariance for protein structure predictions. Compared to AlphaFold 2, which used a more strict form of built-in equivariance, structure prediction performance could be further increased. From a general point of view however, having SE(3)-equivariant predictions is still a desirable property for structure prediction.

Weakness 3:

We are happy to cite MEAN as the potentially first virtual node method in the context of EGNNs. We thank the reviewer for making us aware of the publication and will remove that "To the best of our knowledge VN-EGNN is the first E(3)-equivariant GNN architecture using virtual nodes." The aim of our work is similar to that of MEAN in using an adapted network architecture for a specific application in mind. We thereby tried to avoid some of the disadvantages (as, e.g., discussed in Joshi et al, 2023) of vanilla EGNNs and used a neural representation layer update scheme which might not have existed before. We were not sure, whether we could correctly identify the part, where the reviewer found an analogy of our work to AbDiffuser, but are open to cite it, in case the reviewer points us more directly to this analogy. One further work, which the reviewer mentioned, has already referenced our research. At the time of their citation, our work was available in a preprint format. Another suggested further work, mentioned by the reviewer, missed to reference this preprint.

We thank the reviewer for acknowledging the strengths of our research and hope that our response on the brought up weaknesses and questions can address the raised concerns and questions.

Weakness 1:

Yes, when we apply a one-sided Wilcoxon test, with the null hypothesis that VN-EGNN is worse with respect to the DCC metric than the scond-best method (P2Rank), we obtain p-values < 0.05 on the datasets PDBBind and COACH420. For HOLO4K the p-value is about 0.06. When we apply the same test for the DCA metric instead of the DCC metric, we do not observe significantly better prediction performances. The test results are in accordance to Table 1, for which we slightly changed the definition of the bold markings to be in accordance with the suggested tests.

Weakness 2:

Yes, we applied the same approach. The domain shift here is that HOLO4k predominantly contains multi-chain protein complexes with multiple binding sites, which is in contrast to the scPDB training dataset with single-chain proteins. To be able to provide binding site predictions for multi-chain protein complexes with k (experimentally resolved) chains in a meaningful way, we first split the protein up into multiple chains. We then apply our model, which was primarily trained on single chain proteins, individually to each single chain, i.e., we apply our model k times to get 8 predictions (where 8 is the number of virtual nodes, which is a hyperparameter of our model) for each protein chain together with the self-confidence scores for the individual virtual nodes. Finally, we merge the predictions and sort the 8*k predictions by their scores according to our confidence model.

To evaluate, we consider the k last (i.e., the k highest-ranked) predictions to be the identified binding sites by VN-EGNN. This is in accordance to the performance evaluation of Equipocket and maintains comparability of the results.

Question:

A figure (former Figure I1) in the appendix should have shown the correlation between predicted virtual node scores and proximity to the true binding pocket, with nodes nearest to the actual pocket receiving higher scores. We now moved this figure into the main part of our manuscript to make an interpretation how well our method might perform more accessible to readers.

We thank the reviewer for their effort and think that we can address the reviewer's concerns sufficiently to recommend acceptance.

Weakness 1:

The reviewer is right that this is the more interesting ablation. We provide exactly this ablation as “VN-EGNN (homog.)” as an ablation to “VN-EGNN (full)”. We rename “VN-EGNN (VN only)” to “EGNN+VN” to make clear that this ablation is just plain EGNN with virtual nodes. We also try to make the related text more clear and thank the reviewer for pointing that up.

Weakness 2:

We try to improve the text. What we actually wanted to bring over is:

• From an application point of view, having strict equivariance is a desirable property for a binding site identification algorithm in principle. In its core our suggested architecture is able to fulfill this property.
• However from a practical point of view it seems advantageous not to restrict the learning architecture too much. Therefore, we came up with a more approximate version combined with data augmentation (random rotations during training) of our architecture, which we are using in practice. We found this approach led to a more diverse spatial distribution across the protein surface while maintaining model performance.

Weakness 3:

We thank the reviewer for the suggestion and provide it in Table H1.

	Dice loss	IOU
COACH420	0.397 (0.015)	0.437 (0.005)
HOLO4K	0.584 (0.031)	0.263 (0.025)
PDBBind2020	0.357 (0.010)	0.477 (0.003)