EBMDock: Neural Probabilistic Protein-Protein Docking via a Differentiable Energy Model

Thank you for your time and valuable feedback. Below is the detailed response.

Q1: "The empirical results of the proposed method are not as strong as expected, as it fails to outperform traditional methods and AlphaFold-multimer. This raises questions about the effectiveness of the proposed approach in comparison to existing techniques."

The AlphaFold method, which is based on multiple sequence alignments (MSAs), is difficult to improve in terms of efficiency and cannot be used for large-scale virtual screening. Even if their performance is good, there are significant limitations when dealing with large-scale data. Our approach aims to improve accuracy while maintaining high inference efficiency, although there is currently no method that can achieve this effectively. Therefore, even though our work may not match AlphaFold's, our work has made progress and provided new insights for researchers.

Q2: "The use of only 100 samples for the DIPS-Het test set is questionable. It is unclear if there are any constraints that prevent the application of the proposed method to larger datasets. The authors should provide more information on this aspect and, if possible, test their method on a more extensive dataset."

We select 100 samples for testing because methods like PatchDock, HDock, and AlphaFold-Multimer have too long inference time. These baselines limit us from conducting a fair comparison on a more extensive dataset. We have shown that our method is highly efficient and relatively stable, allowing scalability to large-scale datasets in our paper. Here, we present our results on the entire DIPS-Het test set (1153 samples). We apologize for the misunderstanding caused and hope that our clarification can alleviate your concerns about our method.

Rigid protein-protein docking results on samples from the DIPS-Het test set.

Q3: "The omission of xTrimoDock[1] from the baselines is a notable oversight. According to [1], xTrimoDock can outperform both HDock and AlphaFold-Multimer, and it is also relatively fast. The authors should consider including xTrimoDock in their comparison to provide a more comprehensive evaluation of their proposed method."

We conduct comparative testing on the antibody-antigen dataset released in the xTrimoDock[1]. Unfortunately, we are unable to access their publicly available code, which results in differences in the training data and may lead to some unfair comparisons. Since we can not reproduce their experiments, the results related to xTrimoDock[1] in the table are directly extracted from their paper. Also, the inference time is not listed due to the different machines we used and it's not reproducible.

Results on antibody-antigen dataset released in xTrimoDock (68 samples). Note that DIFFDOCK-PP fails on 4 samples (CRMSD metric exceeds 50Å). The results we presented are obtained after removing these outliers.

Q4: "While the performance of EBMDock-interface is impressive, it relies on the ground-truth binding interface, which might not be a fair comparison to other baselines. This could lead to misunderstandings regarding the method's actual performance. I suggest that the authors clarify this point and consider presenting the results of EBMDock-interface separately in a standalone table to avoid potential confusion."

Thanks for your nice suggestion, we will modify this part to avoid confusion. Furthermore, the highlight of our work lies in proposing and utilizing learnable energy for docking, as well as further integrating EBM training and Langevin dynamics sampling for structure prediction. We have not dedicated our efforts to enhancing the interface recognition algorithm. Therefore, we provide results under the assumption that we have a rough knowledge of the interaction interface.

[1] Luo, Yujie, et al. "xTrimoDock: Rigid Protein Docking via Cross-Modal Representation Learning and Spectral Algorithm." bioRxiv (2023): 2023-02.

Thank you for acknowledging our work. Your rating has provided us with great encouragement, and your feedback has been immensely helpful. Below is our detailed response.

Q1: "What is the performance on binding site and contact prediction, before pose prediction and docking? How does this performance compare to binding site and contract prediction classifiers?"

As for the performance of the binding site and contact prediction, the details can be seen in Appendix B. As for other methods typically designed for predicting the binding site of the interface, such as SCANNET、MASIF, we are sorry for not being able to compare with them fairly. Since they only consider the binding sites on one protein, not considering the docking counterpart.

Q2: "Do the authors have any ideas about what explains the performance gap to HDock and how they could improve the EBMDock protocol? The results suggest that, based on DockQ scores, DiffDock-PP, EquiDock, and EBMDock without providing the ground truth binding site are not usable for pose prediction. This is a valuable result, but it also motivates proposals for substantial improvements."

HDock, PatchDock, and AlphaFold-Multimer use different training data compared to our models. They might have utilized parts of our test sets for extracting templates or as training samples, which may lead to the optimistic results. Therefore, we select 68 antibody-antigen samples from the Protein Data Bank which were released after October 2022 and have not been used to train AlphaFold-Multimer or as templates in HDOCK. The result here demonstrates that both HDOCK and AF-Multimer exhibit a significant decrease in performance on unseen cases. Although EBMDOCK's results also decline, they are close to those of AF-Multimer and surpass HDOCK in CRMSD. Additionally, when EBMDOCK is able to obtain rough docking interface information, the docking results are far superior to both AF-Multimer and HDOCK.

We acknowledge that the results are currently not usable, but the protein-protein docking is still a problem far from being solved. Our energy-based learning framework can be improved in many aspects.

The HDOCK and AF-Multimer achieve good performance on old proteins because they directly use information from training sets such as templates and sequence alignments. We can design new differentiable energy terms to measure how well the docking poses conform to the templates. We can also design energy terms based on physics, for example, van der Waals dispersion/repulsion, directional hydrogen bond energy, Coulomb electrostatic energy, and desolvation free energy. The calculation of these terms may be expensive, but we can approximate them using deep learning methods. They can be incorporated into our framework easily. Besides, our backbone can be easily replaced by new representation learning methods such as those based on the protein surface. For a particular category of docking problems such as antibody-antigen docking, we can train EBMDock on these proteins to learn a more refined energy function, leading to a usable model. We will explore these possibilities in future work.

Q3: "It would be helpful to define all of the variations (Equidock+plugin, EBMDock (5), etc.) in a single place before the results are presented. The naming conventions could also be made clearer and more specific."

Thanks for your nice suggestion, we will add a table of the naming convention. The highlight of our work lies in proposing and utilizing learnable energy for docking, as well as further integrating EBM training and Langevin dynamics sampling for structure prediction. We have not dedicated our efforts to enhancing the interface recognition algorithm. Therefore, we provide results under the assumption that we have a rough knowledge of the interaction interface. During inference, the number of poses sampled from the energy model is in parentheses.

Q3: "In the case where ground truth is not available, we still need to use AF2 to predict the structure of each protein. As a result, the entire workflow for EBMDock won't be much faster than AFM."

The setting of our task is to predict complex structures from monomeric structures. While it is true that protein structure is determined by its sequence, in many cases, it is not always necessary to start with the sequence to predict the monomeric structure. Experimental techniques conducted in wet lab can also be employed to determine the structure, and the accuracy achieved through experimental methods undoubtedly surpasses that of prediction. In fact, the monomeric data used for training and testing in our study are derived from experimental measurements rather than predictions. Moreover, to the best of my knowledge, traditional docking software(HDock, etc), including those utilizing deep learning approaches(DIFFDOCK-PP, etc), typically initiate their predictions from monomeric structures, and these endeavors are indeed meaningful.

Q4: "Gradient of rotation $\nabla_RE(R,t)$ is not defined clearly， In practice, how do you calculate the gradient $\nabla_RE(R,t)$?"

We are sorry we didn't make it clear in the paper. We will explain it in detail below. In practice, we use the Euler Angle to characterize the rotation. Let $R=[\alpha, \beta, \gamma]^T$, the rotation matrix $R_M$ is calculated as follows: \begin{aligned} R_M &= R_{z}(\gamma)R_{y}(\beta)R_{x}(\alpha) \\ &= \begin{pmatrix} \cos(\gamma) & -\sin(\gamma) & 0 \\ \sin(\gamma) & \cos(\gamma) & 0 \\ 0 & 0 & 1 \end{pmatrix} \begin{pmatrix} \cos(\beta) & 0 & \sin(\beta) \\ 0 & 1 & 0 \\ -\sin(\beta) & 0 & \cos(\beta) \end{pmatrix} \begin{pmatrix} 1 & 0 & 0 \\ 0 & \cos(\alpha) & -\sin(\alpha) \\ 0 & \sin(\alpha) & \cos(\alpha) \end{pmatrix} \end{aligned}

The coordinates of the unbound ligand and receptor are $X_L$ and $X_R$, respectively. The docking pose is $(R_M X_L+t, X_R)$. For a residue pair $(li, rj)$ on the binding interface, the distance of them is $d_{i,j}= ||R_M X_{Li} + t -X_{Rj}||^2_2$. Then the energy can be calculated by equation (8) and (9) in the paper. After that, we can get $\nabla_{R}E(R,t)$ by backpropagation.

Q5: "Equiformer-PP architecture is not described."

We are sorry that we have not elaborated on the architecture of EquiformerPP. We add this part in Appendix E.1, see new pdf for details.

Thank you for your time and valuable feedback. Below is the detailed response.

Q1: "The performance of EBMDock is much worse than AlphaFold-multimer (AFM) (DockQ: 0.16 vs 0.48). Although EBMDock runs much faster than AFM, the performance is far from AFM. As a result, EBMDock is not going to be useful in practical setting since its prediction is mostly inaccurate."

The AlphaFold method, which is based on multiple sequence alignments (MSAs), is difficult to improve in terms of efficiency and cannot be used for large-scale virtual screening. Even if their performance is good, there are significant limitations when dealing with large-scale data. Our approach aims to improve accuracy while maintaining high inference efficiency. Even though the performance may not match that of AlphaFold, our work has made progress and provided new insights for researchers.

Also, we have claimed that HDock, PatchDock, and AlphaFold-Multimer use different training data compared to our models. They might have utilized parts of our test sets for extracting templates or as training examples, which may lead to an optimistic result. Therefore, we select 68 antibody-antigen samples from the Protein Data Bank which were released after October 2022 and have not been used to train AlphaFold-Multimer or as templates in HDOCK. The result here demonstrates that both HDOCK and AF-Multimer exhibit a significant decrease in performance on unseen cases. Although EBMDOCK's results also decline, they are close to those of AF-Multimer and surpass HDOCK in CRMSD. Additionally, when EBMDOCK is able to obtain rough docking interface information, the docking results are far superior to both AF-Multimer and HDOCK. We will add these results to our manuscript and hope they can ease your concerns.

Q2: "EBMDock is only evaluated on rigid-body docking setting, where the ground truth structure of each protein is given. Therefore, its comparison with AFM is not fair because it does not assume the binding site is known and it folds both proteins from sequence directly."

We acknowledge that there may be some unfair comparisons when comparing with AlphaFold, but this is due to the inherent differences in our settings. In fact, other structure-based prediction methods also struggle to make a fair comparison with AlphaFold. When it comes to protein docking, we usually have known monomer structures and predict the resulting conformation after docking, rather than predicting the overall structure based solely on the sequence. Although AlphaFold has outperformed all other methods so far, its efficiency makes it challenging for large-scale data. Additionally, while AlphaFold can predict the folded structure even for two proteins that cannot dock, EBMDock identifies the difficulty of docking by observing high energy levels after docking. It is worth noting that as the length of protein sequences increases, the speed of AlphaFold further slows down, and the accuracy noticeably decreases. However, our method primarily incurs increased memory usage, while other metrics remain relatively stable. We hope these points will ease your concern.

Additionally, we would like to emphasize that we are not listing AFM here as a competitor that we have to outperform, but rather as a baseline since it is a well-known SOTA method. In fact, we explore new deep learning based methods aiming to significantly improve efficiency while maintaining a certain level of accuracy. Apart from us, there is also a series of works [1][2][3] making similar efforts after AFM. Even though all the performance may not match that of AlphaFold, our work has made progress and provided new insights for researchers.

Besides, we can leverage AFM to further improve our energy model for docking. Specifically, protein sequence information is easily accessible and abundant, while structural information is scarce. Given sufficient time, we can leverage AF to predict complex structures and generate new training data to further expand our approach (the training set now contains 10,000 samples derived from experimental measurements). In this way, high-quality structure prediction using AFM can expand the training set, while inference on large-scale data using our method can still maintain high efficiency. We believe the existence of AFM should not negate the exploration of other new methods. As reviewer AgXx said, "All deep learning approaches for docking have failure modes in real-world applications, and exploring new ideas and being forthcoming about performance are both needed for the field to progress."

[1] M Ketata, et al."DiffDock-PP: Rigid Protein-Protein Docking with Diffusion Models" ICLR 2023 MLDD [2] Y Luo, et al. "xTrimoDock: Rigid Protein Docking via Cross-Modal Representation Learning and Spectral Algorithm." bioRxiv (2023): 2023-02. [3] R Wang, et al. "Injecting Multimodal Information into Rigid Protein Docking via Bi-level Optimization" Neurips 2023

I would like to thank the authors for their response. I am concerned about the additional comparison with AFM for two reasons. First, the new test set only includes antibody-antigen complexes. It should include a diverse set of protein-protein interactions. It is pretty well known that AFM performs poorly on antibody-antigen complexes than other protein-protein complex types, so i unfair to just compare your EBMDock with AFM on antibodies. Second, reporting only the mean DockQ is also not fair. The protein-protein docking community also care about success rate, i.e., how many test cases with acceptable/medium/high docking quality (the acceptable quality threshold is DockQ > 0.23). If you look closely at the model performances, we see that AFM DockQ has a large standard deviation ($0.108 \pm 0.172$), which means some of them have a relatively high DockQ score. In contrast, the standard deviation for EBMDock is low ($0.05 \pm 0.02$). Therefore, it is likely that the success rate for EBMDock is nearly zero and much lower than AFM. Taking all these results into consideration, I would like to keep my original score.

Underperforming industry-level AFM does not mean an academic reject. You claimed that our work "is not going to be useful" because we can't match the performance of AlphaFold-Multimer (AFM) from giant company DeepMind. It is challenging for the academic community to compare with industry models in terms of accuracy due to limited resources. But we make significantly efforts in terms of efficiency. AFM is based on multiple sequence alignments (MSAs), leading to difficulty in improving the inference speed, and cannot be used for large-scale virtual screening (at least if given only limited budget). However, we construct a differential energy function and utilize an efficient sampling method to prevent exhuastive search (and also some space for future improvement as also appreciated by Reviewer AgXx, 94kF).

Also, you said that "EBMDock needs to use AF2 to predict the structure of each protein in the case where ground truth is not available and the entire workflow won't be much faster than AFM". However, EBMDock as well as other docking methods [1,2,3,4,5] begin with structure are not designed for this case, we are designed for given monomers structures achieved from experimental techniques. It is biased to deny the significance of our efforts to improve efficiency with the case from another different setting.

[1] Y Luo, et al. "xTrimoDock: Rigid Protein Docking via Cross-Modal Representation Learning and Spectral Algorithm." bioRxiv (2023): 2023-02.

[2] M Ketata, et al. "DiffDock-PP: Rigid Protein-Protein Docking with Diffusion Models" ICLR 2023 MLDD.

[3] R Wang, et al. "Injecting Multimodal Information into Rigid Protein Docking via Bi-level Optimization" Neurips 2023.

[4] O, Ganea, et al. "Independent SE(3)-Equivariant Models for End-to-End Rigid Protein Docking" ICLR 2022

[5] Y, Yan, et al. "The HDOCK server for integrated protein–protein docking" Nature Protocols 2020

Thanks for your feedback. Regarding the two points you mentioned, here is our response.

1. Due to time and finance constraints, it is challenging to obtain results on AFM in the limited rebuttal window, thus we will include a diverse set of protein-protein interactions before camera ready.

2. We agree that AFM has a higher success rate compared to our approach (at the cost of much higher training overhead in terms of computing resources and data/labels, as well as inference overhead including computing resource and time latency -- see also some details in the below table). However, the larger variance also indicates that while AFM provides higher mean accuracy, it is relatively unstable. From this point of view, this instability can be compensated by utilizing our proposed method. Specifically, when visualization shows that the predicted results from AFM are completely irrational, we can use energy model to optimize them.

From all of your feedback, you have been emphasizing the performance gap between our model and AFM, as well as the unfairness in comparing them, while admittedly overlooking our efforts and explanations about the different application scenarios. The table below provides a comprehensive comparison between our approach and AFM.

*the number is quoted from https://github.com/google-deepmind/alphafold

Thank you for your time and valuable feedback. Below is the detailed response.

We apologize if you felt being offended by our previous response. There may be some misunderstandings since you have been emphasizing the gap between EBMDOCK and AFM. We may partially distort what you mean.

1. We used the data set collected from xTrimodock, the results of Hdock and AFM were quoted from xTrimodock since they were time-consuming to reproduce. They did not report success rate. If the pdf could be further updated, we will reproduce the experiment and report their success rates. Here we only provide the success rates of EBMDock, 1 / 68 for EBMDOCK(5) and 42 / 68 for EBMDOCK(5)-interface.
2. We fully understand your concern about our performance, which also need further efforts for the community. We admit that we do not perform well when blind docking, but for biologists, they can get a roughly surface knowledge when facing accurate docking (such as finding surface docking concave candidates). And in this case, our performance will increase extremely (42 / 68 success cases), which indicates practical and verifies the validity of our energy function. If our method is used to do large-scale virtual screening, it can also screen out the absolutely irrational structure even under low performance.
3. We only claim that we surpass HDock on CRMSD, and HDock do outperform EBMDock on dockQ. But it should also be noticed that the inference speed of HDock is relatively slow, almost 50x slower than EBMDock as well as other deep learning methods.
4. Thank you for proposing another baseline, since we did not come across their article during our research for this work. We will compare with them in the subsequent updates.

In summary, we respect your valuable feedback and time, we will improve the paper for our final version based on your comments. Could you please kindly reconsider your rating if our feedback could ease some of your concerns (also seeing other reviewers comments).

Thank you for your time and valuable feedback. Below is the detailed response.

Q1: "The method only considers the position of alpha carbon in each residue, which can be a crucial limitation. Being able to model the geometry of full backbone atoms or even sidechain atoms is necessary in real-world scenarios."

Since internal structures of residues of the same class are relatively similar and modeling on residue level can be computationally efficient, we choose to use residue-level modeling in our experiment. Our method can be easily extended from residue level to atom level. To extend EBMDock to the atomic level, we only need to modify the nodes in the protein graph from representing a residue to representing an atom. Instead of learning the statistical potential of residue distances, we would learn the statistical potential of atomic distances. The rest of the network structure and training methods remain unchanged. Also, the challenge of protein-protein docking always lies in the prediction of the backbone structure, and we can use existing tools such as Rosetta to complete the side chain conformation. We hope these points will ease your concern.

It should be noticed that most deep learning methods, such as Equidock and DIFFDOCK-PP(our baseline), are also based on modeling at the residue level.

Q2: "The authors include the DockQ score as one of their evaluation metrics. The original DockQ score includes the percentage of native contacts preserved in the predicted structure as one of the criteria. However, the current method ignores all backbone atoms except for alpha carbon. The author should make this point clear."

The original definition of DockQ indeed involves the use of all-atom representation, but here we substitute it with alpha-C and calculate it in the same manner for all baseline methods. This means our comparison is still fair although we used an alpha-C version of DockQ. Thanks for pointing this out, we have modified our manuscript to make this point clear.

Q3: "The proposed method falls short of the AlphaFold-Multimer on both evaluation tasks. Although the authors have mentioned that the results of AFM might be optimistic (the test set may overlap with the training set of the AFM), it would be better if they could report the results of both models on another benchmark set composed of proteins released after the cutoff date of the AFM's training set."

We select 68 antibody-antigen samples from Protein Data Bank which are released after October 2022 and have not been used to train AlphaFold-Multimer or as templates in HDOCK. Since the template library constructed by HDOCK may have encountered the test set mentioned in the article, we also include it here for a relatively fair comparison.

The result here demonstrates that both HDOCK and AF-Multimer exhibit a significant decrease in performance on unseen cases. Although EBMDOCK's results also decline, they are close to those of AF-Multimer and surpass HDOCK in CRMSD. Additionally, when EBMDOCK is able to obtain rough docking interface information, the docking results are far superior to both AF-Multimer and HDOCK. We will add these results to our manuscript and hope they can ease your concerns.