Learning to design protein-protein interactions with enhanced generalization

We thank the reviewer for the detailed analysis of our work and for providing us with highly valuable feedback.

Weaknesses

(1) My major concern is that neither the code nor the data is publicly available. Are authors willing to release them? If possible, I would recommend authors upload the necessary part of the method to an anonymous GitHub (https://anonymous.4open.science/) during the review phase.

Thank you for raising this point. We definitely plan to release all the code, trained models and data once they are finalized and incorporate all comments / input from the review process.

(2) I value this study's effort in formulating a more comprehensive evaluation paradigm for mutation effect prediction, particularly in dataset splitting. However, the new splitting strategy is somehow unclear. For instance, the test set has only 5 samples, which are too small. And 3 cross-validation does not align with 5 test folds.

We apologize for the confusion. Please see our response to question 1 for detailed clarification. The 5 test samples from SKEMPI v2.0 are difficult to expand due to the high redundancy of the dataset, and these samples are independent from the 3-fold cross-validation. We firstly set aside these 5 representative and independent PPIs for testing, and then split the remaining data into 3 folds.

(3) In the task of retrieving 5 human antibody mutations effect against SARS-CoV-2, the performance of PPIFormer is not good enough, it only outperforms RDE-Net in 3 of 5 samples and is worse in P@5% and P@10% than RDE-Net.

Please note that while the SARS-CoV-2 benchmark proposed by [4] and [5] is an invaluable contribution to the field, one needs to be a bit careful making strong conclusions based on comparison of exact values on the benchmark. In this benchmark, only 5 of 494 mutations are annotated as favorable, while the effects of the other 489 are unknown. For example, according to the benchmark, RDE-Net retrieves the LH104F mutation with lower rank (the predicted rank is 5.47, which is better than the 10.93 predicted by PPIformer). However, we don’t know whether or not the hits predicted with better ranks by PPIformer are actually incorrect. Those mutations could also be favorable. The same holds true the other way around for high scoring mutations of RDE-Net. We will make a note about overinterpreting these results (both directions, in favour or against PPIFormer) in the paper.

Additionally, there is a chance that the performance of RDE-Net, as well as DDGPred, on this benchmark may be slightly overoptimistic. The pool of the experimentally-validated mutations, including the 5 benchmarked ones, was preselected according to DDGPred predictions, and RDE-Net was trained in a similar way as DDGPred (i.e. on a PDB-disjoint split of SKEMPI v2.0). This means that the performance of the methods may be overoptimistic due to the potential circular data leakage. In other words, the methods are tested on the data, that were preselected by the same or a similar method.

Questions

(1) I am a little confused about the evaluation of test folds. As mentioned in Appendix B.2, the author splits the dataset into 3 cross-validation folds based on the "hold-out-proteins feature" and then sets aside 5 random outlier PPIs to form 5 distinct test folds. Firstly, what is the "hold-out-proteins feature"? Please explain that term. Secondly, why are there 3 cross-validation folds but 5 test folds? Commonly, cross-validation would result in training sets as well as validation sets, and usually do not have a so-called test fold. In my understanding, these 5 extra outliers are excluded from both the training and validation sets and are used to evaluate models alone. So here comes the question: how does the author compute the for these 5 outliers? Notably, 3 cross-validation will lead to 3 sets of models that achieve the lowest losses in 3 different validation sets. Does the author just take an average of these 3 models' output to obtain the binding affinity change? If not, please specify the computational process

We appreciate the detailed analysis of our data preparation and apologize for the confusion. We have now significantly expanded Appendix B.2 “Test Datasets” in the updated paper to address all the raised questions. The intuition of the reviewer about the parts that were unclear is correct, and we briefly answer the questions below.

The “Hold out proteins” is the column in the SKEMPI v2.0 dataset, which specifies similar PPIs. We obtain our three folds by splitting the dataset by the values from this column. This ensures the PPIs are not leaked across folds. After 3-fold cross-validation, we, indeed, average three models for test predictions.

We appreciate the reviewer’s valuable feedback and address the comments below.

Weaknesses

When considering the data, it's important to note that the DIPS dataset may not have broad applicability for Protein-Protein Interaction (PPI) tasks. This is because PPI tasks involve not only binding, as observed in DIPS, but also encompass other aspects such as reaction, Ptmod, and activation labels, as described in reference [1]. [1] Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction

We appreciate this important comment and agree that DIPS (as well as Protein Data Bank in general) does not extend beyond protein binding. Please, however, note that the focus of our paper is on designing protein-protein interactions for higher binding energy, and we do not study other, indirect ways of cooperation and mutual regulation between proteins. In this setup, the DIPS dataset (as well as our improved PPIRef) provides an important resource for self-supervised pre-training. Nevertheless, we believe that the reviewer’s comment is very stimulating for potential future work. Extending PPIformer to also learn from PPI networks, where nodes are proteins and edges are interaction types, may lead to improved performance, inspired by [1, 2]. We will cite and discuss the suggested reference.

Questions

1. In the optimization of a human antibody against SARS-CoV-2 and engineering staphylokinase for enhanced thrombolytic activity, how were the mutation pools obtained?

For the SARS-CoV-2 test set, the authors of [3] selected a pool of all 494 possible single-point mutations on the heavy chain CDR region of the antibody. Out of these, five mutations were labeled as positive based on wet-lab experiments (please see [3] for the details on experimental setup). We obtained all the data from the official RDE-PPI repository.

For the staphylokinase test set, we used the results of [4]. The authors of [4] measured the effects of mutations of staphylokinase on its thrombolytic activity. 80 of these annotated mutations are located at its interface with microplasmin, where 28 mutations reportedly increased the protein's thrombolytic activity, while the remaining 52 decreased it. We use these mutations as an another independent test set. Please see Appendix B.2 "Test datasets" in the updated paper for more details.

2. Can you explain the process used to identify the five PPI outliers mentioned in Datasets?

We select PPI outliers based on their similarity as defined by the authors of the SKEMPI v2.0 dataset [5]. Specifically, PPIs are considered similar if they share an interacting partner or a homologous binding site. Then, the outlier PPIs are defined as PPIs that do not have other similar PPIs. These outliers are depicted in Figure 2 of the original SKEMPI v2.0 publication [5] as connected components of two nodes (proteins), along with several other connected components mentioned in footnote 2 (on page 19) of our updated manuscript.

We identified five outlier PPIs by manually analyzing these examples from the figure, aiming to select those with both positive and negative ddG annotations. We succeded at selecting the five outliers only after examining the vast majority of candidates, which highlights the difficulty of choosing a reasonable test set from highly biased SKEMPI v2.0. Lastly, we confirmed that the selected PPIs do not have structural homologs among the other (training) PPIs by calculating interface similarities with iAlign. For more details, please see Appendix B.2 "Test datasets" in the updated paper.

References

[2] Gao, Ziqi, et al. "Hierarchical graph learning for protein–protein interaction." Nature Communications 14.1 (2023): 1093.

[3] Shan, Sisi, et al. "Deep learning guided optimization of human antibody against SARS-CoV-2 variants with broad neutralization." Proceedings of the National Academy of Sciences 119.11 (2022): e2122954119.

[4] Laroche, Yves, et al. "Recombinant staphylokinase variants with reduced antigenicity due to elimination of B-lymphocyte epitopes." Blood, The Journal of the American Society of Hematology 96.4 (2000): 1425-1432.

[5] Jankauskaitė, Justina, et al. "SKEMPI 2.0: an updated benchmark of changes in protein–protein binding energy, kinetics and thermodynamics upon mutation." Bioinformatics 35.3 (2019): 462-469.

We thank the reviewer for a detailed analysis of our work, and the valuable feedback! Below we address their comments.

Weaknesses

IDIST could significantly improve the computation time needed to assess dataset redundancy, which can make redundancy tests a part of every dataset curation. However, I believe that establishing IDIST as part of dataset curation, requires more validation than comparing it against iAlign and for 100 pdbs. You should compare with more methods (eg Tm-align and US align) and repeat it for more pdbs (especially because you define the threshold of IDIST from the iAligh comparison)

We appreciate the reviewer’s idea to establish iDist as a standard tool for dataset curation. We have performed a preliminary experiment according to the reviewer’s suggestion and demonstrate that iDist accurately approximates USalign as well (details below, and illustrated in the new supplementary Figure 7 in the updated version of the manuscript). We plan to extend the benchmark to more than 100 PDB entries (2,7M PPI pairs) and potentially other methods for the camera-ready version of the paper. It was hard to do a large-scale analysis in the limited time of the rebuttal. Please note that we have not benchmarked iDist against TM-align, as suggested, because iAlign is an adaptation of TM-align to the domain of PPIs developed by the authors of TM-align (i.e. the group of Jeffrey Skolnick). Hence, we expect iAlign is better suited for PPIs than TM-align.

In detail, iDist achieves high precision (99%) and recall (97%) with respect to near-duplicate PPI structures detected using USalign. Notably, this is a similar level of performance we achieved when benchmarking our iDist against iAlign. However, the iAlign and USalign methods modify TM-align differently, and with different constants, which introduces a shift in the marginal distributions of their alignment scores. We found that the threshold of 0.7 we used for the iAlign score corresponds approximately to a threshold of 0.8 on the USalign score (the threshold of iDist does not need to be adjusted for accurate approximation; see Figure 7 in the supplementary material). Regarding the computational time, we find that iDist is roughly 100 times faster than USalign, confirming the need for iDist for large-scale PPI analysis.

The novelty is a bit limited, especially in the model architecture.

In our work, we have developed a new ddG predictor along with a reliable evaluation protocol. The novelty of our predictor (PPIformer) primarily lies in a newly developed data representation (coarse-grain representation based on virtual beta-carbons), training objectives (structural masked modeling for pre-training combined with log odds ratio for fine-tuning), and high-quality training data (PPIRef) prepared with our new method (iDist). To establish a new evaluation protocol, we have rethought the data splits and evaluation metrics that have been used for more than a decade (please see Section 2, Related Work for the references). Our evaluation scheme enables benchmarking ddG predictors in a way that goes beyond overfitting to near duplicate PPIs and is a step towards a new class of ddG predictors that will generalize to new unseen protein-protein interactions. We believe this work has the potential to significantly push forward the research community in this area.

Appendix Figure 3/B: You mistakenly have the same pdb code in both subfigures.

Please note that this is not a mistake. Both PPI interfaces (E-M and N-D) come from the same PDB entry with the code 2V6A. The complex is a 16-mer composed of two unique proteins. It is important to analyze the composition of interfaces within a single complex because symmetric complexes introduce a lot of redundancy (see also the caption under Figure 1 in the updated paper, which has been rewritten to be clearer).

We thank the reviewer for their constructive and valuable feedback on our work. Below, we address the reviewer’s comments.

Weaknesses

1. Evaluations on a broader range of datasets are preferred in terms of PPI and mutational effect predictions. Currently, all the test samples are selected from a larger benchmark dataset, making it skeptical that the results might be cherry-picked.

Thank you for this important comment. Please note that in our work we use three independent test datasets (see Section 5.1 Datasets). First, we select 5 outlier (i.e. not having a training homologue) test PPIs from the SKEMPI v2.0 dataset. SKEMPI v2.0 encompasses the vast majority of published experimental ddG studies. This dataset, indeed, represents a selection from a larger benchmark dataset. We selected the test samples once to evaluate the generalization of our model before seeing the results. We did not overfit our method to the test set by exploring any other subsets. Please see Appendix B.2 “Test Datasets”, which now describes in detail the construction of the SKEMPI v2.0 test set.

Moreover, we evaluate our method on two additional (and separate) datasets, which are not part of SKEMPI v2.0 and come from independent studies. The “Optimization of a human antibody against SARS-CoV-2” provides an independent evaluation on the benchmark proposed by [1] and [2], the studies proposing DDGPred and RDE-Network (ICLR 2023). Finally, to compare generalization of different methods even more robustly, we curated a novel independent test set for the problem of “Engineering staphylokinase for enhanced thrombolytic activity”. This dataset was mined from [12].

We hope that our test setup involving three mutually independent test sets provides enough evidence that the results are not cherry-picked. Please also see our comment on attempts to obtain more test data in the answer to the ZbAC reviewer’s Question 4. Overall, it is challenging to find additional independent and high-quality test data.

2. Comparison of baseline methods is limited. At least some other deep learning methods should be included. For instance, for mutational effect prediction, the authors might refer to https://github.com/OATML-Markslab/ProteinGym.

Thank you for your important comment. Please note that, while the number of evaluated baselines is not excessive, we have initially selected methods among the most representative methods in four available categories:

1. Physics-based simulators. Flex ddG is the most performant and elaborate force-field-based protocol for binding ddG prediction.
2. Evolutionary methods. MSA-Transformer is a well-established method that might benefit from using multiple sequence alignment (MSA) compared to other baselines (MSA is crucial, for example, for AlphaFold2).
3. Unsupervised models. ESM-IF is a state-of-the-art inverse folding method, trained on the extensive AlphaFold DB. It was shown to effectively score single-point mutations from SKEMPI v2.0.
4. Supervised models. RDE-Network is a recent, high-quality, and state-of-the-art method fine-tuned on SKEMPI v2.0.

We value the suggestion to consider additional baselines from ProteinGym. Consequently, we have now added the evaluation of GEMME, the best publicly available model in ProteinGym (GEMME ranks second among 34 methods and is publicly available, unlike the top-ranked model, TranceptEVE). We find that GEMME performs worse on most metrics than our PPIformer (please refer to the table below). However, it is worth noting that GEMME, as well as other ProteinGym methods, was not trained for PPI design. Instead, it was trained to optimize the global fitness of a protein with respect to deep mutational scanning experiments.

Thanks for updating these extensive experimental results and detailed explanations in such a small time window. The majority of my concerns are well-addressed. While the proposed method does not always achieve top performance (which is usually expected for a conference paper to be accepted), I reckon many investigations and analyses are valuable. I have thus raised my score to 6.

Also, since the authors have made great efforts during the rebuttal, it would be nice if these investigations could be integrated into the final version (if it gets accepted).

Reminder: Please revise your paper to fit into 9 pages as required.

We greatly appreciate the valuable feedback provided by the reviewer. Below, we answer the reviewer's questions.

Questions

1. Would be nice if you could also compare your methods with baselines on the normal PDB-disjoint split, as this is also useful in real world cases. If PPIformer also performs best, the results would be more consistent and persuasive.

We appreciate the reviewer's comment, but we would like to again emphasize that the PDB-disjoint splitting of SKEMPI v2.0 does not reflect the desired generalization in real-world scenarios. For instance, the first two PDB interaction codes present in the dataset, 1CSE_E_I and 1ACB_E_I, come from different PDB entries (1CSE and 1ACB) but have nearly identical 3D structures and sequences. Additionally, they are both labeled with the same six mutations, with slightly different ddG values. Despite this, they are divided into separate folds in the PDB-disjoint split by [5], exemplifying the data leakage problem. In SKEMPI v2.0, there are three times more unique PDB codes than PPIs, leading to most test PPIs having a near-duplicate PPI in the training set. Additionally, every third test mutation is located on a PPI present in the training set (please see Appendix B.2 "Test Datasets" for details).

Following the request of the reviewer, we have now re-trained our identical PPIformer architecture in the RDE-Net setup and examine its performance below. We would like to emphasize that for our method we use the same hyperparameters (e.g. architecture, loss and data representation) and do not tune the hyperparameters for these leaked splits. We only adapt the batch size from 32 to 8 (we also experimented with different batch sizes when training RDE-Net on our splits). Overall, PPIformer has competitive performance. More specifically, PPIformer performs reasonably on the overall evaluation (although it is not the best method, it often is the second best). In addition, PPIformer is the best method according to some of the metrics on the challenging set-up of multi-point mutations, which may better reflect its generalization capabilities. Please also see the review by Sjjh highlighting the importance of multi-point mutations and our corresponding response for some other details.

Overall performance:

We thank the reviewer very much for the high attention dedicated to the submission. We would like to sincerely apologize for the delay with answering the reviews, we are trying our best to run as many experiments as possible to address all the suggestions proposed by the 5 reviewers. We plan on submitting our official answer to all the reviews today.