Boltzmann-Aligned Inverse Folding Model as a Predictor of Mutational Effects on Protein-Protein Interactions

Q1: Could the authors explain the reasons for the relatively low MAE reported in Table 2?

A: As shown in Equation 10, the unsupervised estimation of $\Delta\Delta G$ relies not only on the conditional probabilities modeled by the inverse folding model but also on $k_B T$ (the product of Boltzmann constant and thermodynamic temperature). In the supervised version, we treat $k_B T$ as a learnable parameter, whereas in the unsupervised version, we simply set $k_B T = 1$. While this unknown scaling factor does not impact correlation-based metrics, it does affect absolute metrics like MAE, which may account for the relatively low MAE reported in Table 2.

Q2: Could the authors clarify how they determined the unbound state approximation for independent chain probabilities (Equation 9)? The paper provides some explanation on this point, but it lacks sufficient detail.

A: Thanks and addressed. As illustrated in Fig.1-Left, for a single chain (A) extracted from a complex (AB), we mask both the sequence and structure of the complex AB according to chain A's mask. The resulting masked information is then input into the protein inverse folding model, ProteinMPNN. For the unbound state, each chain is evaluated independently—chain A is processed in isolation, disregarding chain B, and vice versa. Conversely, for the bound state, chains A and B are processed together in ProteinMPNN, preserving all inter-chain interactions (represented as inter-chain edges in ProteinMPNN's graph representation).

Q3: The transformation between energy and probability, along with the approximation of unbound state probabilities, partially explains the high correlation between the inverse folding task and ΔΔG prediction. However, within the neural network, the sophisticated design of model doesn’t appear to provide additional information gain.

A: Thanks for your comment. In our opinion, we do not introduce any sophisticated model design; the only additional learnable parameter compared to the inverse folding model is a scalar $k_B T$ (the product of the Boltzmann constant and thermodynamic temperature). Our main contribution lies in incorporating physical inductive bias to more effectively leverage the inverse folding model for $\Delta\Delta G$ prediction. By incorporating the thermodynamic cycle and Boltzmann distribution, we can achieve better performance than previous $\Delta\Delta G$ predictors (including previous inverse-folding-based methods).

Dear Reviewer:

Thank you for your thorough review and constructive comments! We sincerely hope our responses fully address your questions.

W1.1: The method makes strong assumptions, particularly in treating the unbound state as independent probabilities for each chain, which oversimplifies the inter-chain interactions.

A: Thanks for your constructive comment. The unbound state typically refers to a condition with no significant inter-chain interactions. We acknowledge that weak inter-chain interactions, such as transient and solvent-mediated interactions, may exist in the unbound state. However, these interactions generally have minimal impact on free energy, making it a common and effective assumption to treat the unbound state as independent probabilities for each chain, thereby simplifying the problem with negligible cost. In contrast, inter-chain interactions are much stronger in the bound state. To account for this, we compute its probability by inputting the entire complex into ProteinMPNN, preserving all inter-chain interactions represented as edges in the model’s graph structure.

W1.2: Additionally, the predictions of inverse folding model’s do not fully align with the predicted probabilities (not the ground truth), introducing biases when single-chain probabilities are combined to approximate complex states. This approximation may result in significant deviations in some situations.

A: Thanks for your insightful comment. We agree that, although the pretrained inverse folding model we use is probabilistic, its predicted probabilities do not reflect the ground truth because it is simply trained using a cross-entropy loss on biologist-resolved structure-sequence pairs. However, our proposed Boltzmann Alignment establishes a bidirectional connection between the inverse folding model's predicted probabilities and the $\Delta\Delta G$ values, enabling mutual enhancement. Experimental results demonstrate that this approach not only enables the use of the inverse folding pretrained model to predict $\Delta\Delta G$ but also allows the fine-tuned inverse folding model to generate sequences with higher affinity (Section 4.4.3).

We are not sure if we have correctly understood and addressed your question here. If you have any further questions, please feel free to let us know.

W2: The method has only been validated on a limited dataset, which lacks sufficient evidence to demonstrate the generalizability of the approach and its underlying assumptions.

A: Thanks for pointing this out. (1) SKEMPIv2 remains the largest and most well-established benchmark available for $\Delta\Delta G$ prediction. (2) Due to the high cost of measurements and the lack of high-throughput techniques, experimentally determined $\Delta\Delta G$ data is limited in availability. (3) We also hope to see more public datasets in this field. Recent efforts, such as AlphaSeq, are promising, though their whole dataset is not yet publicly available.

W3: It seems that the paper lacks sufficient discussion on how improvements in the inverse folding model would translate to gains in this approach. It’s also unclear which specific module in this sophisticated approach contributes most significantly to the information gain.

A: Thanks. Our main contribution lies in incorporating physical inductive bias to more effectively leverage the inverse folding model for $\Delta\Delta G$ prediction. We do not introduce any sophisticated model design, except for a single learnable parameter, $k_B T$. Therefore, in our opinion, we consider the ablation study (on thermodynamic cycle and supervision method) in Section 4.3 to be clear and sufficient. However, we would appreciate any further suggestions or specific ablation experiments you think could enhance the clarity of our results.

Dear Reviewer:

Thank you for your constructive comments and kind support! All your concerns have been carefully addressed as below. We sincerely hope our responses fully address your questions.

W1: Method is structure based, it requires at least experimentally resolved or confidently predicted complex structure.

A: Thanks. Similar to previous state-of-the-art methods on SKEMPIv2, our approach is also structure-based. With recent advances in robust structure prediction methods, we are able to reduce our reliance on crystal structures, thus enhancing the applicability of our approach.

W2: The representation of protein is rather coarse, only backbone. Side-chain atoms and solvent molecules are omitted yet they are obviously important drivers of protein-protein interactions.

Thanks. We agree that side-chain atoms and solvent molecules are important drivers of protein-protein interactions.
Our framework can be further extended to consider side-chain conformations. In Equation (1), the probabilities can be represented as $p(X, R | S)$ instead of $p(X | S)$, where $R$ represents side-chain conformations. According to the chain rule of probability, this can be expanded as: $p(X, R | S) = p(R | X, S) \cdot p(X | S).$ Based on this, Equation (8) can be modified to jointly leverage the probabilities on backbone and side-chain conformations to estimate $\Delta\Delta G$.
We employ RDE to estimate the side-chain probability $p(R | X, S)$, here are the results:

The results show that BA-DDG with RDE achieves improvements on overall metrics but shows a slight decrease in per-structure metrics. This outcome may stem from challenges in side-chain conformation modeling or limitations in our current implementation. While we acknowledge the reviewer's insightful comment, we have not yet made progress in addressing these challenges, highlighting this as an important direction for future work.

W3: No representation of dynamics (structures are represented as static snapshots), which can affect the method performance for mutations inducing significant conformational changes.

A: Thanks for pointing this out. We agree that for mutations inducing significant conformational changes, simply using the wild-type structure as the mutant structure may affect the results. Exploring the representation of dynamics could be an important direction for future work.

W4: The described method seems to excel at point mutations and there's a trend towards decreased performance in more complex multiple mutations. Majority of those weaknesses are highlighted and discussed by the authors in the appropriate sections across the paper.

A: Thanks. As reported in Table 6, BA-DDG performs better on single-point mutations compared to multi-point mutations, which is also observed in other methods. This could be due to multi-point mutations being inherently more challenging, or it may indicate potential areas for improvement in our approach.

Q3: In the typical design scenario, as the problem is hard and the reported performance of the method is not perfect, one can also consider the uncertainty of the prediction which is an important and frequently used feature in e.g. protein folding models. It could potentially strengthen the paper if authors could discuss and provide more information on how uncertainty estimation could be introduced to the model.

A: Thank you for this insightful comment. We agree that uncertainty estimation is an important yet challenging problem. While metrics such as pLDDT in protein folding models provide valuable confidence scores, they are also known to suffer from false positives. Extending uncertainty estimation to our approach would require careful consideration of similar issues, and we see this as an interesting direction for future work. We have considered three possible directions for introducing uncertainty estimation:

1. Inference Ensembles: Using multiple models or parameter sets during inference to assess the variability in predictions.
2. Dropout as a Bayesian Approximation: Applying dropout during inference to approximate a posterior distribution over model predictions.
3. Inverse Folding Model Uncertainty: Extending the inverse folding model to output uncertainty metrics similar to pLDDT in protein folding models. These uncertainty estimates could then be integrated into BA-DDG for uncertainty estimation of $\Delta\Delta G$.

We hope to explore these ideas in our future work.

Q4: In appendix B - authors write "We compare BA-DDG with several representative methods under single-point, multi-point, and all-point mutations on SKEMPI v2". It's not clear to me what is the difference between multi-point and all-point mutations. Is the all-point a union of point- and multi-point mutations (I'd assume so by looking at the following Table) ? For clarity authors can consider to explain this more in the paper.

A: Thanks and addressed. Your understanding is correct: "all-point" indeed represents a union of point- and multi-point mutations. We will clarify this distinction in the revised version of the paper to improve clarity.

Q5: The discussion about computational (non-ML) methods for ddG estimation is scarce. Even though it's purely ML venue, I miss a bit more thorough discussion of how the proposed method relates to force-field methods utilising more nuanced thermodynamical cycles / approaches like MM-GBSA/PBSA, FEP, TI, etc.

A: As we are less familiar with these methods, and given that FEP can be computationally expensive, we are still working on this. We hope to provide some reliable results by Nov 26.

Update (Nov 27): We have attempted to conduct $\Delta\Delta G$ estimations using non-ML methods, including MM-PBSA and FEP, on a subset of the SKEMPIv2 dataset. However, the results were lower than expected, likely due to our limited experience with these non-ML methods. We are continuing our efforts and will share reliable results if obtained before the discussion period ends.

Q1: In section 4.4.3 authors utilise dataset from Shen et al (2022) however focus only on point mutations. This dataset is, I think, much richer and contains double and triple mutants from multiple optimisation rounds. I believe it would be important to expand this section to show proposed model performance more extensively. This, I believe, has a potential to boost the impact and trust in the model in a typical antibody design scenario.

A: Thanks for your suggestion. Results for multi-point mutations are as follows, using the same setting as described in Section 4.4.3.

Q2: Following on sec 4.4.3 - preference probabilities are a good metric but they in my view they are a bit abstract. I think a simpler metric of e.g. how the method ranks particular mutation would be more interpretable (as they relate to simple consideration of how many designs one can screen in the wet lab before finding a particular hit).

A: Thanks for your suggestion. For a CDR region of length $L$ with $K$ mutations, the number of possible mutants is $C(L, K) \times 19^K$. For double and triple mutations, this results in 117,325 and 17,833,400 mutants, respectively. Here we report the ranking results only for single-point mutations.

Table R3. Spearman correlation of different mutation types.

Table R2. Pearson correlation of different mutation types.

Table R1. Frequencies of different mutation types.

Dear Reviewer:

Thank you for time and effort! We sincerely hope our responses fully address your questions.

Q1: Are there any specific types of mutations or protein interactions where the authors have observed the method to be particularly effective or challenging? Sharing such insights could help guide future research directions.

A: Thanks for your constructive suggestion. While we have explored this, we have not yet obtained conclusive results. Nevertheless, we are happy to share our detailed analysis results.

Analysis of different protein interaction types:

The protein interaction type annotations in the SKEMPIv2 dataset are incomplete, with approximately 3000 data points labeled, with one-third for antibody-antigen (AB/AG), enzyme-substrate (Pr/PI), and TCR-pMHC interactions, respectively. Here we report the average per-structure correlation metric for each protein interaction type, as shown in the table. Differences in metrics across types are observed, but we cannot rule out the impact of data distribution differences.

Analysis of different mutation types: We report the frequencies of different mutation types for single-point mutations in the SKEMPIv2 dataset, along with the correlation metrics of BA-DDG, calculated separately for each mutation type. The rows represent the wild-type amino acid, and the columns represent the mutant amino acid. Mutation types with frequencies greater than 10 are highlighted in bold.(When the frequency is too low, the correlation metric may be either uncalculable or meaningless.) Due to uneven data distribution, we cannot determine whether BA-DDG's lower performance on specific mutation types is due to a lack of training samples or the inherent difficulty of these types.

Thanks for your recognition of our work. If you have any further questions, please feel free to raise them during the discussion phase.

We sincerely appreciate your time and effort in reviewing our submission! With the discussion period ending soon on Dec. 2nd, we kindly welcome any further comments or questions you may wish to share. We are happy to engage in any further opportunities for clarification.

Dear Reviewer:

Thank you for your thorough review and insightful comments! In the following, we will address each issue in detail and provide the necessary experimental results.

W1: The code is not available, it should be open-sourced upon publication.

A: Thanks for your suggestion. We are currently organizing and refactoring the code, and we plan to release our code upon publication.

W2 & Q1 & Q2: In Section 4.1, Dividing the dataset into 3 folds by structure cannot strictly prevent data leakage, different protein complexes might have similar protein binding structures, can you elaborate more about the method you used to split data and prevent data leakage? Could you explain more about data partition process, why it can help to reduce data leakage process?

A: Thanks for your insightful comment. (1) Our decision to randomly divide the dataset into 3 folds by structure was primarily to ensure consistency and fair comparison with previous works, including RDE, Surface-VQMAE, DDGPred, Prompt-DDG, DiffAffinity and ProMIM. (2) From the perspective of $\Delta\Delta G$ labels, structure-based split does not introduce data leakage. (3) We have the same concern with you that different protein complexes might share similar protein binding structures. A more rigorous approach, such as clustering structures based on TM-scores, could be considered in the future. However, this has not yet been explored for SKEMPI v2 in existing studies.

Q3: The SKEMPI dataset includes both single-point mutations and multiple-point mutations. Given the varying complexity, predicting binding affinity for single-point mutations may be less challenging and could potentially yield more accurate prediction results compared to multiple-point mutations. This raises the question: why not consider training separate models tailored for single and multiple mutation binding affinity prediction tasks, rather than using a single model for both?

A: Thanks for your insightful suggestion. We explored training separate models for single- and multiple-point mutations but observed no significant improvement in performance compared to a unified model under the same training conditions. We speculate that, despite the increased complexity of multiple-point mutations, both tasks fundamentally address the same underlying problem, and leveraging shared data does not seem to harm performance.

Thank you for your insightful comments once again! As the discussion period is coming to close (Dec. 2nd), if you have any further questions, please feel free to raise them. We are happy to engage in any further opportunities for clarification.