Towards Stable Representations for Protein Interface Prediction

We sincerely appreciate the reviewer's thorough review and insightful feedbacks. We also appreciate the reviewer's recognition for the core idea and contribution of this paper. We will respond to these comments point-by-point.

[Cons. 1 Problem Formulation] Thank you for your professional comments, which we fully agree with. We will make some adjustments to avoid confusion. To address your concerns, we make the following efforts.

1. Firstly, we clarify that existing research related to our task falls into three categories: partner-specific interface prediction (PS-IP), partner-specific binding site prediction (PS-BSP), partner-agnostic binding site prediction (PA-BSP).

   PS-IP is the problem of our paper, which is to predict whether there is an interaction between a pair of residues of two different proteins. In many cases (e.g., NEA, SASNet, BIPSPI, HOPI), this task is referred to as 'protein interface prediction'.

   PS-BSP, as mentioned in your comment, predicts whether the residues belong to the interface. For this task, the interface of a protein is conditioned on its partner.

   PA-BSP predicts all possible binding sites of a protein. It is the main setting of ScanNet and PesTo, which does not assume knowledge of the partner.

2. Although the applications of the three tasks differ, they are strongly related. We will mention the above three tasks and their differences in the abstract and introduction sections. Importantly, in the related work section, we will provide descriptions of baseline methods belonging to each.

[Cons. 2 More Ablation Study] Thanks for the valuable suggestion. To address the your concerns, we show ablation results on stability regularization (SR) in Table 3,4 in the one-page PDF.

We find that:

(1) In the Native-Bound scenario, SR is ineffective, while in the remaining (real-world) scenarios, SR significantly improves all four cases.

(2) The gain of SR on the SGC encoder is relatively small among the four cases, which may be due to the direct reduction of the polynomial order $K$ in SGC. Therefore, the expressive power is explicitly diminished.

[Cons. 3 More Baselines] Thanks for the constructive feedback. Regarding the five papers provided, we find that except for Pair-EGRET, they are all designed for the task of 'partner-agnostic binding site prediction'. However, we find that PesTo can perform the PIP task after simple code modifications.

In this way, we consider Pair-EGRET, PesTo, ESM-Fold and AF2 as additional baselines.

The results are shown in Table 7 in the one-page PDF. We find that ATProt can still outperform AF2 (actually AF-Multimer) and ESM-Fold.

[Cons. 4.1 Use AF2 to Generate Training Data] This is a very worthy topic for discussion. Frankly speaking, this is our initial attempt. Table 7 in one-page PDF shows the results. The results indicate that using the suggested methods is difficult to achieve SOTA.

Regarding this, our insights are:

1. There may still be distribution shift between the training and testing sets. This is because the cropping process of AF2 considers monomers and multimers. This results in AF2's training set being a mixture of bound and unbound structures. Thus, using AF2 to generate both training and test sets may not necessarily guarantee consistent distribution.

2. It is not very practical to use AF2 to generate training structures on a large-scale dataset (e.g., DIPS with over 40,000 dimers), as it would be very time-consuming.

3. Our method allows training only once and then testing on various versions of structures. This uniformity and wide applicability are also a potential advantage.

[Cons. 4.2 Relax the Bound Structures] Performing force field optimization based on First Principles is a very insightful idea and has been proven effective[1]. However, molecular dynamics (MD) often requires huge computational cost. Thus, applying MD to large-scale datasets is a challenge.

[1] Benchmarking Refined and Unrefined AlphaFold2 Structures for Hit Discovery. ACS, 2023.

[Cons. 4.3 Representation Shift] Thanks for providing this literature called SAO. SAO indicates that there is room for improvement when training and testing both on AF2-structures (i.e., its TonP baseline). Although tasks are different, the deep motivations of ATProt and SAO are similar: both to enable effective inference using structures from various source.

[Q1. How to Change Structures] We strongly agree with your point that flexibility cannot be arbitrarily generated. We applied a dynamics guided method called ProDy. It can sample structures that differ from the initial structure by a specific value (RMSD).

[Q2. About dMaSIF] The main point of dMaSIF is to generate point cloud representations. However, the dMaSIF paper also contributes a geometric convolutional model, enabling it to address structure-related downstream tasks. Importantly, the PIP task is implemented in the dMaSIF paper.

[Q3. ProtMD] Thanks for providing this valuable literature, which has a user-friendly open-source nature.

In Table 7 of the one-page PDF, we present the experimental results of ProtMD. We consider the results of fine-tuning (ProtMD-FT) and linear probing (ProtMD-LP). Surprisingly, its performance slightly exceeds the baseline specifically designed for PIP.

[Q4. Pre-training with dMaSIF] Thanks for the thorough findings. The quality of the DIPS and DB5.5 dataset differs slightly. The latter is an expert-annotated "gold standard dataset", while the dimers mined in DIPS are mainly from multimers in PDB, which do not reflect pure binary interaction knowledge. Facing such gap, negative transfer (where pre-training has a negative impact) is common for some models because they are not designed for the pretraining-finetuning paradigm.

Let me add that I believe the method proposed by the author is a form of regularization rather than an adversarial training approach. If I am correct, please adjust the relevant writing accordingly.

Also, please add the ablation study and baselines (e.g. EBMdock) as reviewer RNrt mentioned.

We thank the reviewer RLcQ for acknowledging our method as "meaningful and interesting", the quality of our paper as "very high" and our methods as "sound". Many thanks for the constructive feedback especially for the questions about the writing part. These all help us to further enhance the readability and the quality of our paper.

[Cons. Description of Adversarial Training]

Our explanation:

Thanks for the constructive feedbacks. We strongly agree with you that adversarial training (AT) typically involves two or more conflicting training objectives. Many cases of literature indicate that a more accurate statement is: Lipschitz continuity regularization is a popular and effective method for Adversarial Robustness (AR).

From the formulation of the training objectives, two losses only exhibit intuitive conflicts, as $\mathcal{L}_S$ acts by constraining the slope of the GNN filter, which limits the expressive power of the GNN (as mentioned in the Limitation section). Moreover, from the empirical perspective, the regularization loss slightly reduces the model's performance on the 'Native-Bound' structure, implying that there may be an implicit conflict in the training objective. Therefore, we acknowledge that the two objectives do not have explicit or theoretically provable conflicts.

In our work, robustness is the model's performance when facing such flexibility 'attack', that is, when we test on unbound, ESM-Fold or AF2 structures. Therefore, while our training objectives ($\mathcal{L}_I$ and $\mathcal{L}_S$) have implicit conflicting nature, we believe that AR will be a more conducive expression for promoting readability.

Our adjustment:

1. Firstly, as shown in Table 3,4 in one-page PDF, we note that the proposed regularization slightly impairs performance in the standard, i.e., Native-Bound scenario. Thus, we validate the conflicting nature of the objectives in the empirical perspective.

2. Additionally, we will supplement the formulation analysis of the objective conflicts, i.e., how constraining the slope of the GNN filter affects its expressive power.

3. We will adopt the use of "adversarial robustness" instead of "adversarial training" which will involve the Preliminaries section. As a result, the training objective for protein representation stability will have a more rigorous description.

[Cons. Ablation Study] Thanks for the valuable suggestions.

About baselines:

We have added various baselines, and all results can be found in Table 7 in the one-page PDF. We have added the following baselines: EBMdock, Pair-EGRET, PesTo, ProtMD-FT, ProtMD-LP, ESM-Fold, AF2 and AF2-Train.

About regularization:

We perform ablation study on the effectiveness of the stability regularization. The results are shown in Table 3,4 in the one-page PDF.

We will include all newly added experiments to our revised manuscript

Dear RLcQ,

Have you had the chance to read the authors’ rebuttal, and does it address the concerns you raised, and has it influenced your evaluation of the paper?

Best regards, AC

We sincerely appreciate the reviewer kBXn's recognition of our paper and valuable comments. We will respond to reviewer's insightful suggestions point-by-point.

[Cons 1. Novelty of Model's Architecture] Thanks for the valuable comment. We sincerely clarify that the novelty of this paper is three-fold.

1. Various protein graph encoders: We fully agree with your feedback that we did not pay enough attention to the design of new protein representation models. However, the proposed method (strategy) is plug-and-play. Although we only demonstrate four cases, the GNN encoders applied for protein-related tasks are far more numerous [1,2]. The concept of stability regularization (SR) has the potential to be widely applied to various existing encoders to address the challenges posed by protein flexibility.

2. Various protein-related tasks: Through empirical and theoretical validation, we show that in the protein interface prediction (PIP) task, we can avoid simulating the exact distribution of protein flexibility and instead use simple yet effective regularizations. In other protein-related tasks, this strategy might also address distribution shifts caused by structural changes. For example, SR has the potential to resolve the distribution shift between AlphaFold 2 and native structures mentioned in [3].

3. New proposition for stable BernNet: BernNet is a widely applied model with strong expressive power. Here, we introduce the Lipschitz regularization for BernNet for the first time (to our knowledge), achieving state-of-the-art performance on the PIP task. Due to the analytical difficulties caused by the complex polynomial structure of BernNet, we respectfully believe it is a relatively fundamental contribution to the field of stable GNNs.

[1] Protein representation learning by geometric structure pretraining. ICLR, 2023.

[2] Structure-based protein function prediction using graph convolutional networks. Nature communications, 2021.

[3] SaProt: Protein language modeling with structure-aware vocabulary. ICLR, 2024.

[Cons 2. Need More Ablation Experiments] We really appreciate the reviewer's detailed comments. We include the added experimental results in Table 3,4 in one-page PDF.

From the results, we can see that:

(1) In the Native-Bound (ideal and virtual) scenario, the proposed SR is ineffective, while in the remaining (real-world) scenarios, SR significantly improves all four cases of encoders.

Furthermore, we have made a new discovery that:

(2) The gain of SR on the SGC encoder is relatively small among the four cases, which may be due to the direct reduction of the polynomial order $K$ in SGC caused by SR. Therefore, the expressive power is explicitly diminished.

We will add all of these results to our revised manusrcipt.

[Cons 3. Residue-Level Visualizations] Thank you for your insightful comments regarding this issue. To address your concern, we add the residue-level visualization, and the figure is included in one-page PDF. Due to space constraints, we select two out of the 25 samples from the DB5.5 testset.

The baseline method we select is NEA in our manuscript. First, we categorize the misclassified samples by NEA into false negatives (FN) and false positives (FP). Due to the imbalanced nature of the PIP problem, we observe that there are notably more FP samples than FN samples. Therefore, in the added figure, we visualize all FN samples and some of the FP samples. Please kindly note that all of the visualized pair samples are misclassified by NEA but correctly classified by ATProt.

From the figure we add, it can be seen that the residues involved in the shown samples (especially the FN samples), exhibit significant structural changes.

In addition, we quantify the average structural changes of residues corresponding to correct and incorrect predictions for both NEA and ATProt. This is performed with all of the DB5.5 testset samples. For NEA, the average structural changes of correctly and incorrectly classified residues are 0.563 and 0.972, respectively. For ATProt, the respective values are 0.589 and 0.597. In simple terms, the structural changes impair the performance of NEA but have little impact on ATProt.

This insightful comment highlights the advantages of the proposed method, and we will include this interesting part in our revised manuscript.

[Cons 4. Some Hyperparameters are Missing] Thanks for your very thorough review. Table 5 in one-page PDF is a comprehensive summary of the hyperparameters. The bolded rows are newly added.

[Question 1. Effectiveness of SR] Thanks for the feedback, and we have added more experimental results in the response to [Cons 2].

[Question 2. The Prediction Changes after Using SR] Thanks for the valuable comment. In the response to [Cons 3], we conducted visualizations and quantitative experiments, both of which show the robustness of ATProt to structural changes.

[Question 3. Hyperparameter Tuning Process for Weight of $\mathcal{L}_S$] Thank you very much for this question. For the overall training objective $\beta\mathcal{L}_I + \gamma\mathcal{L}_S$, we fix $\beta$ at $1.0$ and search for $\gamma$ in the range of $0.1$ to $10$. For fairness, we ensure that all baselines and our methods conduct hyperparameter search with n_trials=50 in the Optuna API.

We provide the results on the DB5.5 dataset in Table 6 in one-page PDF.

We observe that all three cases exhibit a similar pattern: with the increase of $\gamma$, the performance first increases and then decreases. The difference is that the optimal weight ranges for the three types of encoders are not the same.

[Question 4. More Detailed Hyperparameters] Thanks for your detailed suggestion, and we have provided more hyperparameter results in the response to [Cons 4]. The complete hyperparameter table will be added to our revised appendix.

Dear kBXn,

Have you had the chance to read the authors’ rebuttal, and does it address the concerns you raised, and has it influenced your evaluation of the paper?

Best regards, AC