Predicting mutational effects on protein binding from folding energy

Thank you for your review. We appreciate your thoughtful suggestions and have conducted additional experiments that provide additional insights to our method.

Fine-grained evaluation: We examine StaB-ddG’s performance on different subsets of SKEMPI.

Complex size: total number of residues of a PPI.

(PS RMSE: Per Structure RMSE)

Interface structural rigidity: percent of near-interface residues (defined as within 10A of another chain) with secondary structure type = loop.

We observe that StaB-ddG does worse on bigger complexes and more flexible interfaces. We believe that this information provides valuable additional insight to our method and will include it in the appendix.

Learned linear parameters (negative means destabilizing):

$\alpha = 0.24, \phi_0 = -0.19$

Amino acid offsets

Since both the fine-tuning losses and ddG predictions are invariant to additive shifts that might be applied to all per-amino-acid offsets, we report the differences relative to Alanine. We see that the learned $\phi_0$ is negative, indicating that most mutations in the dataset are destabilizing. We note that the value of the bias does not affect the correlation metrics, and find that adding the bias term lowers the overall RMSE from 1.88 to 1.79. We also note that the learned offset for TRP, a bulky hydrophobic residue expected to have larger effects on stability, is the second largest in magnitude and is negative (destabilizing).

Two-stage finetuning: Our rationale for our two-stage finetuning is that first fitting the linear parameters can correct for any initial scale mismatch in our predictions that might inflate initial gradients and destabilize training. When we view the linear model as the final layer of a deep predictor, this choice coincides with standard practice in transfer learning [1]. In our revision we will elaborate on this choice.

Comparison to ThermoMPNN: We believe the driving factor of the performance difference between our model and ThermoMPNN is (1) the architectural difference, and (2) we train on multiple mutations as well while ThermoMPNN are trained only on single mutations.

$\sqrt{M_n}$ : Thank you for your suggestion to try our loss with $\sqrt{M_n}$ . We re-trained the model with this loss and did not observe a meaningful difference with our original loss, we will include the results below in the appendix:

Bias and variance in per-structure metrics: For structures with a small number of variants computed correlations are subject to both significant bias and high variance (see e.g. [2]). Though non-exhaustive, we feel our figure 5 in our submission gives some indication of these effects as we have no reason to expect performance to be different on structures with few mutants.

Typos: Thanks for pointing these out! We will fix them accordingly in the revision.

[1] Kumar et al. Fine-tuning can Distort Pre-trained Features and Underperform Out-of-Distribution. ICLR 2022.

[2] Bishara, Anthony J. and Hittner, James B. Reducing Bias and Error in the Correlation Coefficient Due to Nonnormality. Educational and Psychological Measurement.

Thank you for your review. We appreciate your recognition that our use of folding energy data to improve binding prediction is novel and well-motivated. We found your suggestions helpful and address them below.

Baselines: We agree with the reviewer’s comment that the lack of deep learning baselines was a limitation of our submission. This concern was shared by reviewer yFSX, and we address it by including four new baselines. The results show that on our stricter split these additional methods provide similarly poor generalization performance. Please see our reply to yFSX for details.

Reproducibility and hyperparameters: Thank you for pointing out these missing details, which will be included in the revision. In brief, the network architecture and initialization are inherited exactly from ProteinMPNN (1.6M parameters, 3 layers for both encoder and decoder with hidden state size = 128) [1]. We fine-tune on the megascale stability dataset using the ADAM optimizer with learning rate 3e-5 for 150 epochs with a batch size of 50,000 tokens. We fine-tune on SKEMPI using the ADAM optimizer with learning 1e-6 for 50 epochs with a batch size of 50,000 tokens. To further ensure reproducibility our final version will include training and inference code.

Value of alpha: alpha is a scaling term learned from folding stability data and we will include them in the appendix (negative means destabilizing)

$\alpha = 0.24, \phi_0 = -0.19$

We see that the learned bias ($\phi_0$) is negative, indicating that most mutations in the dataset are destabilizing.

Per-residue patterns: we report the RMSE by mutant residue type for single mutants on SKEMPI below.

We note that StaB-ddG achieves lower RMSEs for many bulky hydrophobic residues (F, W, Y, M).

Size of ensemble over noise: we performed an ablation experiment and will include a figure for the impact of the size of ensemble in the appendix. We report it partially here:

We find that ensembling over 10 predictions with random permutation orders and backbone noise leads to near optimal performance. The results in table 4 are obtained with ensemble size = 20.

Monomers for Skempi: are obtained by splitting the PDB files by chain (holo structures). We will update the text to clarify this.

Standard errors and significance testing on per-structure and overall metrics: we’re glad that the reviewer appreciates that we reported standard errors on the per-structure metrics. For the revision we will include cluster-bootstrap standard errors for overall metrics, and (because their interpretation is non-trivial) a discussion of them in the appendix.

In brief, we initially chose not to report intervals for overall metrics because it was not clear what the standard error should represent. For each per-structure metric, the standard error represents uncertainty in the expected value of that metric for a collection of ddG measurements for mutants of some new structure typical of those in the test set (i.e. sampled iid from the same distribution). For each overall metric, we can compute an analogous standard error as the standard deviation of that metric on cluster-bootstrap resample of the test set [2] where on each bootstrap sample we draw full clusters from the test-set clusters with replacement. These standard errors approximate the variability in the overall metrics owing to the choice of structures included in the test set. We will include these standard errors in our revision.

Fine-tuning on dG: we thank the reviewer for this suggestion. We have not yet explored predicting direct dG’s because we expect this task to introduce additional complications; for example, the log-likelihood initialization is biased by the length of a sequence such that longer sequences will be predicted to be more stable than shorter sequences. However, we hope to explore this direction in future work.

Boltzmann-Alignment and ProteinMPNN-DDG: we will include a discussion in the relevant works section.

We hope that addressing these concerns helped strengthen our submission!

[1] Dauparas et al. Robust deep learning-based protein sequence design using ProteinMPNN. Science.

[2] Cameron, A. Colin and Miller, Douglas L. A Practitioner’s Guide to Cluster-Robust Inference. Journal of Human Resources.

Thank you for your review. We appreciate you pointing out our novelty of using folding energy data for binding ddG prediction. We address your comments/questions below.

Including multiple mutation sites: We have now evaluated folding ddG performance on multi-mutants in the megascale test set and report the results below.

We find that StaB-ddG performs worse on multi-mutants. We suspect that this is because of the limited number of multi-mutants in the training data. We will include these results in the appendix. We clarify that the results on SKEMPI in our submission include multiple mutations.

Why ProteinMPNN for stability prediction: The reviewer observes that since the problem is fundamentally a stability prediction task, other models that can predict stability (e.g. ThermoMPNN and sequence models) could be used instead of ProteinMPNN. This is indeed the case. In our submission we chose ProteinMPNN for

1. Simplicity of applying to complexes: ProteinMPNN accommodates multi-chain complexes natively, whereas the other methods described are implemented only for monomers. Adapting these alternative methods would require heuristics such as adding a glycine linker or a residue gap that might negatively impact performance,

2. ProteinMPNN is light-weight. Compared to ProteinMPNN, ThermoMPNN includes an additional transfer-learning module, and MutateEverything is built on either ESM2 or AF2 which have >10X more parameters and longer runtime compared to ProteinMPNN for a forward pass,

3. The thermodynamic properties of the resulting predictor. Unlike StaB-ddG, a predictor that uses ThermoMPNN or MutateEverything would be inherently asymmetric and so would not satisfy properties 1 and 2 in our Proposition 1, and

4. Strong zero-shot performance: Inverse folding models (including ProteinMPNN) provide stronger zero-shot stability performance than sequence models, and so provide a stronger starting point for fine-tuning [1].

In our revision we will make clear in the text the reasons for this design choice. Additionally we will include an evaluation of StaB-ddG with ESM-IF model as the stability predictor in our revision to confirm our result that including folding stability data improves binding prediction is not specific to our choice of ProteinMPNN.

[1] Notin et al. ProteinGym: large-scale benchmarks for protein fitness prediction and design. Neurips 2023.

We thank the reviewer for their constructive comments and appreciate that they find it interesting that StaB-ddG allows folding ddG data to improve binding ddG prediction. We hope addressing the comments has helped strengthen our submission.

Baselines: We agree with your comment that the paper will be made more convincing by including more baselines. To address this we have re-trained DiffAffinity, Prompt-DDG, ProMIM, and VQ-MAE on our SKEMPI splits, and will include the results below in table 4:

PS: per structure

We find these methods perform worse than both StaB-ddG and Stab-ddG zero-shot on 5 of the 7 metrics (all but RMSE). We suspect this better performance by StaB-ddG is due to its folding energy-based parameterization, which provides improved generalization. Out of distribution generalization is particularly important for our stricter data splitting.

Filtered and clustered SKEMPI splits: Thank you for pointing out that these clusters were not specified in our submission. We include these below, with each cluster on a separate line:

Train:

1AK4

1B2S 1B2U 1B3S 1BRS

1C4Z

1E50 1H9D

1F47

1FFW

1IAR

1KBH

1QAB

1YCS

2AW2

2B42

2C5D 2C5D 4RA0

1EMV 2WPT

2HRK

2J0T

2KSO

2O3B

2VN5

3BP8

3BT1

3EG5

3EQS 3EQY

3F1S

1ACB 1AHW 1BJ1 1CBW 1CHO 1CSE 1CZ8 1DQJ 1DVF 1EAW 1FC2 1FCC 1GC1 1JRH 1MHP 1MLC 1N8O 1N8Z 1NCA 1NMB 1PPF 1R0R 1SMF 1TM1 1UUZ 1VFB 1XGP 1XGQ 1XGR 1XGT 1XGU 1YQV 1YY9 2B2X 2BDN 2FTL 2NY7 2NYY 2NZ9 2SIC 3BDY 3BE1 3BN9 3BX1 3G6D 3HFM 3L5X 3MZW 3N85 3NGB 3NPS 3SE8 3SE9 3SGB 3W2D 4GXU 4JPK 4KRL 4NM8 5C6T

1A22 1BP3 3MZG

3Q8D

3SE3 3SE4

3SZK

3VR6

4HFK

2DVW 3AAA 4HRN

4J2L

4JEU

4K71

4OFY

4PWX

4RS1

1OHZ 4UYP 4UYQ 5M2O

4Y61

5CXB 5CYK

5E6P

5F4E

5K39

Test:

1B41 1FSS 1MAH

1EFN 1GCQ

1C1Y 1GUA 1HE8 1K8R 1LFD 3KUD 4G0N 5TAR 5UFE 5XCO

1KTZ 1REW 2QJ9 2QJA 2QJB 3B4V 3BK3 3HH2 3SEK

1S1Q 1XD3 2OOB 3M62 3M63

1A4Y 1Z7X

4CPA

1JTD 1JTG 2G2U 3QHY

2PCB 2PCC

2AJF 3KBH

3S9D 3SE4

3WWN

4B0M

4CVW

4E6K

1AO7 1BD2 1JCK 1LP9 1MI5 1OGA 1SBB 2AK4 2BNR 2P5E 2PYE 3C60 3HG1 3QDG 3QDJ 3QIB 4FTV 4JFD 4JFE 4JFF 4L3E 4MNQ 4N8V 4OZG 4P23 4P5T 5E9D

4FZA 4NZW 4O27

3SF4 4WND

4X4M

4YFD 4YH7

Our final version will include dataset filtering/splitting scripts.