Learning conformational ensembles of proteins based on backbone geometry

We thank the reviewer for their feedback and constructive critique. We are glad to read the reviewer finds our paper well-written, conceptually reasonable and the efficiency improvement noteworthy. Below we address weaknesses mentioned in the review.

 

A4.1 - Addressing W1: Rare events sampling

We agree with the reviewer that sampling conformational changes triggered by rare events is interesting for practitioners. However, these typically occur on MD timescales well beyond 300ns, which is the MD timescale of the ATLAS dataset. As we state in section 1.1, A.3, 'limitations' and 'Further baselines' in the original submission, sampling such large conformational changes and alternative states is a task that is different from the one solved by AlphaFlow and BBFlow.

In AlphaFlow and BBFlow, MD is emulated under fixed, standardized settings. The ground truth ensembles strongly depend on these settings (temperature, timescale, forcefield). Thus, evaluating distributional accuracy quantitatively (and not rather qualitatively as in BioEmu) is only sensible if these settings are fixed. This becomes apparent in the poor performance of BioEmu in all metrics on ATLAS as done in Table A.2 of the original submission and on de novo proteins in Table R16. This poor performance is expected because BioEmu does not claim to solve this specific task, conversely it can be expected that BBFlow and AlphaFlow would perform poorly if evaluated on BioEmu's task.

We would like to note that not only the prediction of rare events and far alternative states but also emulation of nanosecond-timescale MD has a lot of applications. These kinds of MD simulations are employed on a regular basis by practitioners for multiple different purposes. To state a few concrete examples: MD on the 300 ns timescale can be used for assessing opening-closing movements, hinge movements or flexibility of binding pockets and is useful for rational enzyme design. [1, 2, 3].

---

[1] Raubenolt, Bryan A., et al. "Molecular dynamics simulations of the flexibility and inhibition of SARS-CoV-2 NSP 13 helicase." Journal of Molecular Graphics and Modelling 112 (2022)

[2] Kiss, Gert et al. "Molecular dynamics simulations for the ranking, evaluation, and refinement of computationally designed proteins." Methods in Enzymology. Vol. 523. Academic Press, 2013. 145-170.

[3] Wijma, Hein J. et al. "A computational library design protocol for rapid improvement of protein stability: FRESCO." Protein engineering: Methods and protocols. New York, NY: Springer New York, 2017. 69-85.

 

A4.2 - Addressing W1: Contact metrics in appendix

As we argue in section A.2.1, we focus on metrics that are established in the field of biomolecular dynamics and provide the contact metrics, which were only recently introduced in AlphaFlow, for completeness in the appendix. As stated in that section, the metrics contain arbitrary thresholds of 8 Å and 10%. BBFlow indeed performs slightly worse than AlphaFlow but still better than the distilled model (Table A.4). We regard the distilled model as relevant baseline since it is the only AlphaFlow model that comes close to BBFlow in terms of efficiency, demonstrating BBFlow's favourable performance in the tradeoff between efficiency and accuracy. Notably, BBFlow also performs better in the contact metric than BioEmu (Table A.4), which is commonly considered as good model for off-equilibrium sampling. However, we included BBFlow's slightly worse performance in the contact metrics as limitation in the 'limitations' section in the main text.

We would like to note that we ran an additional experiment during the rebuttal phase that shows that there is no clear trend, which model is generally better at sampling off-equilibrium states. In addition to the contact metrics, we introduce another metric that measures the difference between equlibrium structure and generated samples. Instead of using contacts as difference measure between states, we explicitly calculate the RMSD between samples and equilibrium structure for each protein and report the values in Table R13. We find the reversed trend - BBFlow being better than AlphaFlow-T at capturing off-equilibrium states. In terms of RMSD, BBFlow samples states that are further away from the equilibrium structure, while AlphaFlow-T samples closer to the equilibrium structure. We believe this demonstrates that slight performance advantages may depend on the metric definition and that there is no clear candidate that is generally better at sampling larger conformational changes.

Table R13. Deviation from the equilibrium structure on the ATLAS test set. Closer to MD is better.

 

A4.3 - Addressing W2: Omission of BioEmu from the main text

As discussed in answer A4.1, we find reporting BioEmu’s performance on the ATLAS test set would place it in an unfavourable evaluation setting, as BioEmu is not designed to emulate 300ns of MD with distributional accuracy. It is rather designed to sample off-equilibrium states by training on longer MD timescales and fine-tuning on folding transitions with experimental data. We think this makes a comparison to BBFlow and other models trained on ATLAS dataset (300 ns) unfair (for BioEmu) and misleading and should not be emphasized in the main text. We stress this in Section 4 of the original submission under Further Baselines.

Appendix Tables A.2 and A.5 of the original submission illustrate that BioEmu, indeed, performs poorly across nearly all metrics on the ATLAS test set in standardized settings. We also predicted ensembles for de novo proteins with BioEmu during the rebuttal period and report its performance in Table R14. Probably due to shallow evolutionary information for de novo sequences, BioEmu performs even worse for those proteins. Conversely, it can be expected that BBFlow and AlphaFlow would perform poorly if evaluated on BioEmu's task.

Table R14. Performance of BioEmu (and BBFlow) on the de novo proteins from Table 2 in the original submissions. Units and settings as in Table 2. Full set of metrics in Table R15.

 

A4.4 - Addressing W3: Provide full set of metrics for de novo proteins

We thank the reviewer for their suggestion. We report the full set of metrics for de novo proteins in Table R15.

Table R15.: Evaluation on the de novo proteins from Table 2 including the metrics introduced in AlphaFlow. Units and settings as in Table A.2 and Table 1, respectively.

We thank the reviewer for the time they invested in reading the paper and their helpful feedback. We are happy that the reviewer finds our method technically compelling, sees it as state-of-the-art on a widely accepted benchmark and recognizes its applicability to multi-chain systems. For resolving the concerns raised in the review, we conducted additional experiments, which we will include in the final version of the paper. In the following, we address all points individually.

 

A3.1 - Addressing W1: Performance improvements over AlphaFlow-T are modest

We would like to note that performance is not only assessed solely via accuracy but also efficiency has to be considered. In this combination, the tradeoff between accuracy and efficiency, BBFlow's improvement over AlphaFlow-T is large: BBFlow is 40 times faster, at comparable accuracy. Additionally, it covers multi-chain systems, an important class of proteins for which AlphaFlow is not applicable. Both of these points are also recognized by other reviewers.

 

A3.2 - Addressing W1: The empirical impact of equilibrium structure encoding is limited

As the reviewer states, removing the conditional prior indeed reduces performance (Table 3, line b). But removing the equilibrium structure encoding and keeping the conditional prior (line e) reduces performance even stronger. Thus, both of these innovations are needed to achieve state-of-the-art performance. Of course, line b suggests that the conditional prior applied within AlphaFlow-T (i.e. without eq. structure encoding but with MSA and Evoformer) might lead to the same performance in terms of accuracy, however, efficiency would be greatly reduced. Thus, also the equilibrium structure encoding can be regarded as crucial, especially for efficiency.

 

A3.3 - Addressing W2: Robustness under noisy or bad quality equilibrium structures

Short answer:

If the quality of the equilibrium structure is reduced, BBFlow's performance only decreases slightly, indicating robustness under noisy equilibrium structures.

Long answer:

We agree with the reviewer that robustness under changes of the equilibrium structure is a good consistency check and also relevant in practice. In Table A.1 of the original submission, we reported the performance of BBFlow if AlphaFold2-predicted structures, instead of the equilibrium structures from the ATLAS dataset, are passed to BBFlow. In this setting, we find only a slight decrease in performance compared to the equilibrium structures used in ATLAS, which can be expected since the MD simulations are not long enough (not converged to statistical equilibrium) for assuming independence of the starting structure. Importantly, BBFlow is still more accurate than the other baselines that have no access to the equilibrium structure from ATLAS, such as AlphaFlow without templates.

While we regard AlphaFold2-predicted structures as the most relevant class of equilibrium structures in practice (AlphaFold2 is heavily used by practitioners and AlphaFold2-predicted structures are available for millions of naturally occurring proteins), we also ran an additional experiment during the rebuttal period, where we distort the equilibrium structures by adding noise. To this end, we added Gaussian noise to the Euclidean backbone coordinates of the ATLAS test set proteins and reran inference with BBFlow. We find that the performance remains strong and decreases only slightly. We report the results in Table R9 below.

Table R9. Performance of BBFlow on distorted equilibrium structures. We add Gaussian noise with the respective standard deviation to the backbone atoms equilibrium structure, or choose random MD conformations as equilibrium structure, and evaluate each model on the ATLAS test set. Units and settings as in Table 1.

 

A3.4 - Addressing W3: Evaluation on short-timescale MD simulations, not on more complex dynamics

It is correct that also longer MD timescales and larger conformational changes are interesting. However, as we state in section 1.1, A.3, 'limitations' and 'Further baselines' in the original submission, this task is different from the one solved by AlphaFlow and BBFlow, in which MD is emulated under fixed, standardized settings. The ground truth ensembles strongly depend on these settings (temperature, timescale, forcefield). Thus, evaluating distributional accuracy quantitatively (and not rather qualitatively as in BioEmu) is only sensible if these settings are fixed. This becomes apparent in the poor performance of BioEmu on ATLAS as done in Table A.2 of the original submission and on de novo proteins in Table R10. This poor performance is expected because BioEmu does not claim to solve this specific task, conversely it can be expected that BBFlow and AlphaFlow would perform poorly if evaluated on BioEmu's task.

Lastly, we would like to note that not only the prediction of far alternative states but also emulation of short-timescale MD under standardized setting has a lot of applications. Shese kind of MD simulations are employed on a regular basis by practitioners all around the world. To state a few concrete examples: MD on the 300 ns timescale can be used for assessing opening-closing movements, hinge movements or flexibility of binding pockets [1, 2, 3].

Table R10. Performance of BioEmu (and BBFlow) on the de novo proteins from Table 2 in the original submissions. Units and settings as in Table 2. Full set of metrics in Table R16.

---

[1] Raubenolt, Bryan A., et al. "Molecular dynamics simulations of the flexibility and inhibition of SARS-CoV-2 NSP 13 helicase." Journal of Molecular Graphics and Modelling 112 (2022)

[2] Kiss, Gert et al. "Molecular dynamics simulations for the ranking, evaluation, and refinement of computationally designed proteins." Methods in Enzymology. Academic Press, 2013.

[3] Wijma, Hein J. et al. "A computational library design protocol for rapid improvement of protein stability: FRESCO." Protein engineering: Methods and protocols, 2017. 69-85.

 

A3.5 - Addressing W4: No direct discussion or comparison with AlphaFlow-Lit

In AlphaFlow-LIT inference time is sped up by calculating the evoformer output only once. We refrained from including AlphaFlow-LIT as baseline because, although published as workshop paper at ICML 2024, no code, model weights or generated samples are publicly available. These would be required for evaluating the model. This would be necessary since AlphaFlow-LIT [1] does not report all metrics from AlphaFlow [2] and since in Table 1 of [1], inconsistent values for AlphaFlow are reported (see Table R11), raising the question whether the same metrics are being used by [1].

We will include AlphaFlow-LIT as related work in the background section and explain, as above, why it cannot be evaluated.

[1] Li S et al. Improving AlphaFlow for efficient protein ensembles generation. arXiv preprint arXiv:2407.12053, 2024. [2] B. Jing et al. AlphaFold Meets Flow Matching for Generating Protein Ensembles. ICML 2024

 

A3.6 - Addressing Q1: How sensitive is the model with respect to the hyperparameter xi?

The hyperparameter xi was chosen by brief, manual optimization on the validation set.

We thank the reviewer for suggesting an ablation study, which we performed during the rebuttal period and report the results in Table R12. We observe that diversity, as measured by median RMSF and pairwise RMSD, increases with smaller xi, in the sense that xi=0.1 and xi=0.2 gives better diversity than xi=0.4. For the inference-xi-ablation (Table R7 in answer to QtLV), we observe a tradeoff between accuracy (RMSF r, RMSF MAE, PCA W2) and diversity.

We also observe that training converges faster with larger xi, but we did not perform extensive analysis on this effect since it requires training several models per xi and we regard the importance of speeding up training as not too high, compared to speeding up inference.

Table R12. Ablation of models trained with different choice for the hyperparameter xi. In the paper, we chose xi=0.2 based on manual optimization on the validation set. Units and settings as in Table 1.

 

A3.7 - Addressing Q2: Why are metrics better for multi-chain systems reported in A.3?

We thank the reviewer for bringing this potential source of confusion to our attention. The reason for this effect is that some proteins are more challenging than others, thus the metrics between different sets of proteins cannot really be compared on an absolute scale, rather different baselines on the same set of proteins. Also, we would argue that there is no clear trend but it is rather mixed: RMSF r, RMSF median and DCCM r are worse than in Table 1 of the submission.

We cordially thank the reviewer for their feedback on our paper. We are happy to read the reviewer finds our paper well-written, the core idea elegant and the experiments comprehensive. Below, we address these points raised in the review and present corresponding additional experiments that we ran during the rebuttal period.

 

A1.1 - Addressing W2/Q1: Sensitivity of the model to the quality of the equilibrium structure

Short answer:

If the quality of the equilibrium structure is reduced, BBFlow's performance only decreases slightly, indicating robustness under noisy equilibrium structures.

Long answer:

We agree with the reviewer that robustness under changes of the equilibrium structure is a good consistency check and also relevant in practice. In Table A.1 of the original submission, we reported the performance of BBFlow if AlphaFold2-predicted structures, instead of the equilibrium structures from the ATLAS dataset, are passed to BBFlow. In this setting, we find only a slight decrease in performance compared to the equilibrium structures used in ATLAS, which can be expected since the MD simulations are not long enough (not converged to statistical equilibrium) for assuming independence from the starting structure. Importantly, BBFlow is still more accurate than the other baselines that have no access to the equilibrium structure from ATLAS, such as AlphaFlow without templates.

While we regard AlphaFold2-predicted structures as the most relevant class of equilibrium structures in practice (AlphaFold2 is heavily used by practitioners and AlpaFold2-predicted structures are available for millions of naturally occurring proteins), we also ran an additional experiment during the rebuttal period, where we distort the equilibrium structures by adding noise. To this end, we added Gaussian noise to the Euclidean backbone coordinates of the ATLAS test set proteins and reran inference with BBFlow. We find that the performance remains strong and decreases only slightly. We report the results in Table R1 below.

Table R1. Performance of BBFlow on distorted equilibrium structures. We add Gaussian noise with the respective standard deviation to the backbone atoms equilibrium structure, or choose random MD conformations as equilibrium structure, and evaluate each model on the ATLAS test set. Units and settings as in Table 1.

 

A1.2 - Addressing W2/Q1: Escaping the energy well around the eq. structure

We can understand the concern of the reviewer that BBFlow might be biased towards predicting states that are close to the equilibrium structure since our method heavily relies on that structure. However, already from the median RMSF column reported in Tables 1 and 2 from the original submission, it can be seen that BBFlow predicts off-equilibrium states to a similar extent as MD, while AlphaFlow-T samples too close to equilibrium, as indicated by the smaller median RMSF. We will stress this important point more in the final version.

To make the observation more explicit, we introduced an additional analysis during the rebuttal period. We calculated the average RMSD of the generated samples from the equilibrium structure for each protein and report the results in Table R2 below. We find the same trend as in the original submission: BBFlow samples states that are on average as far away from the equilibrium structure as those explored by MD, while AlphaFlow-T samples states that are closer to the equilibrium structure.

Table R2. Deviation from the equilibrium structure on the ATLAS test set. Closer to MD is better.

 

A1.3 - Addressing Q2: Effect of hyperparameter xi on training time and diversity of ensembles

We thank the reviewer for suggesting this experiment, demonstrating the effect of xi, which we performed during the rebuttal period and report the results in Table R3. We indeed observe that diversity increases with smaller xi, in the sense that pairwise RMSD and median RMSF of the model trained with xi=0.4 is smaller than that of the xi=0.2 and xi=0.1 models. For the inference-xi-ablation (Table R4), we observe a tradeoff between accuracy (RMSF r, RMSF MAE, PCA W2) and diversity.

As anticipated by the reviewer, we also observe that training converges faster with larger xi, but we did not perform extensive analysis on this effect since it requires training several models per xi and we regard the importance of speeding up training as not too high, compared to speeding up inference.

Table R3. Ablation of models trained with different choice for the hyperparameter xi. In the paper, we chose xi=0.2 based on manual optimization on the validation set. Units and settings as in Table 1.

Table R4. Ablation of the model from the paper, trained with xi=0.2, for inference with different values of xi. Units and settings as in Table 1.

 

A1.4 - Addressing Q3: Evolutionary information encoding dynamics via fitness

Indeed, evolutionary information (e.g. MSA) can encode information about functionally or structurally important sites in proteins and their dynamics.

As we demonstrate in our paper, evolutionary information is not strictly required for emulation of 300ns MD simulation if abundant training data (such as the ATLAS dataset) is present. We expect that evolutionary information might become a necessity for predicting conformational changes occurring at longer timescales, where the task becomes increasingly more challenging, especially given the lack of large datasets covering micro- and millisecond protein dynamics. This is, however, out of scope of the paper as we describe in sections 1.1, A.3 and 'limitations' of the original submission. We added the above discussion on the role of evolutionary information to section A.3.

 

A1.5 - Addressing W3: More detailed explanations on models and algorithms cited from other papers

We thank the reviewer for making us aware of this. We will add an overview on the respective resources to the appendix of the final version.

We cordially thank the reviewer for the time they invested in reading the paper and their constructive critique. We are happy that the reviewer finds our core ideas interesting, the experiments well-executed and highlights the valuable capability of BBFlow to sample ensembles of multimers! In the following, we address all points raised in the review. We will include the respective additional experiments and explanations in the final version of the paper.

 

A2.1 - Addressing W1/Q2: Inference time comparison is not fair

Short answer:

This point is technically true, however, changing the settings to a strictly fair comparison only changes the values marginally. In the paragraph below, we provide a table with the respective values and remarks on the fairness of the comparison, which we also include in the final version of the paper. We thank the reviewer for bringing this potential misunderstanding to our attention, which allows us to clarify this in the final version as below.

Long answer:

AlphaFlow and ESMFlow first predict the backbone in the SE(3) frame representation, similar to BBFlow. As in AlphaFold2, the sidechain positions are then predicted in a post-processing step with a separate small neural network that only takes a fraction of the backbone-prediction inference time to run. Without considering sidechain conformations, AlphaFlow-T's inference time (per conformation on the protein 7c45A from Tab. 1 in original submission) is reduced from 32.6s to 32.5s while BBFlow only requires 0.8s. Thus, also in the strictly fair setting of backbone-only generation, BBFlow outperforms AlphaFlow-T in terms of efficiency, with virtually the same relative margin as reported in the original submission. In Table R5, we report the runtime of the individual modules of AlphaFlow and compare with BBFlow.

Table R5: Inference time by module of a single AlphaFlow and BBFlow forward pass on a 300 residue protein. The sidechain module is entirely separate from the backbone module and only takes a fraction of the time.

Lastly, we would like to note that AlphaFlow, during post-processing of the predicted backbone structure, uses the same sidechain module as AlphaFold2 without introducing methodological changes or ablating performance of the sidechain module. In AlphaFlow, as in our paper, the focus is on the backbone structure, which encodes larger conformational changes and the overall structure of the protein. If practitioners are interested in sidechain configurations of the generated backbone ensembles, it is, in principle, possible to combine the sidechain module from AlphaFold2 also with BBFlow, as done in AlphaFlow. However, this requires re-training and is out of scope for the paper at hand.

 

A2.2 - Addressing W2: Lack of Ramachandran analyses and sidechain energy landscapes

We added Ramachandran histograms of backbone atom dihedrals to the paper, comparing BBFlow, AlphaFlow-T and MD for four randomly sampled proteins from ATLAS. We find that the Ramachandran histograms of both BBFlow and AlphaFlow-T indeed show overlap with MD (we cannot upload the plots during rebuttal since the global response with PDF is not supported this year and links are not allowed). The Ramachandran plots are possible because backbone dihedrals can be calculated from BBFlow's predicted Euclidean coordinates for the backbone atoms. For energy landscapes, however, the sidechain conformation is required in order to apply a force field. While further sidechain analyses would be interesting indeed, they are out of scope for both our paper and also AlphaFlow. Instead, the main contribution of both papers is predicting ensembles of backbone structures. The backbone structure captures overall conformational changes and dominates computational cost of state-of-the-art models like AlphaFold2 or Alphaflow, as shown above. Thus, also the authors of AlphaFlow (the most relevant baseline) do not perform analyses on the recovery of the sidechain conformations, Ramachandran plots or energy landscapes, possibly because the sidechain prediction method is not novel but done as post-processing step with the same network as in AlphaFold2, as described above.

 

A2.3 - Addressing Q4: Ablation study on the xi hyperparameter

We thank the reviewer for suggesting this experiment, which we performed during the rebuttal period and report the results in Table R6. We observe that diversity, as measured by median RMSF and pairwise RMSD, increases with smaller xi, in the sense that xi=0.1 and xi=0.2 gives better diversity than xi=0.4. For the inference-xi-ablation (Table R7), we observe a tradeoff between accuracy (RMSF r, RMSF MAE, PCA W2) and diversity.

We also observe that training converges faster with larger xi, but we did not perform extensive analysis on this effect since it requires training several models per xi and we regard the importance of speeding up training as not too high, compared to speeding up inference.

Table R6. Ablation of models trained with different choice for the hyperparameter xi. In the paper, we chose xi=0.2 based on manual optimization on the validation set. Units and settings as in Table 1.

Table R7. Ablation of the model from the paper, trained with xi=0.2, for inference with different values of xi. Units and settings as in Table 1.

 

A2.4 - Addressing Q5: Analysis on sample diversity

We can understand the concern of the reviewer that BBFlow might be biased towards predicting states that are close to the equilibrium structure since our method heavily relies on that structure. However, already from the median RMSF column reported in Tables 1 and 2 from the original submission, it can be seen that BBFlow predicts off-equilibrium states to a similar extent as MD, while AlphaFlow-T samples too close to equilibrium, as indicated by the smaller median RMSF. We will stress this important point more in the final version.

To make the observation more explicit, we introduced an additional analysis during the rebuttal period. We calculated the average RMSD of the generated samples from the equilibrium structure for each protein and report the results in Table R8 below. We find the same trend as in the original submission: BBFlow samples states that are on average as far away from the equilibrium structure as those explored by MD, while AlphaFlow-T samples states that are too close to the equilibrium structure.

Table R8. Deviation from the equilibrium structure on the ATLAS test set. Closer to MD is better.

 

A2.5 - Addressing Q3: Intuition for removal of index encoding

Intuitively, removing the index encoding reduces the potential for overfitting and incentivizes the model to generalize better. If the index of a certain (challenging) structural motif in the training set is available, the model might use the combination of motif and index in order to memorize the dynamics in the training set. The index should have no physical meaning but only structural context and amino acid identity, removing it as input can thus be interpreted as physical inductive bias. We will add a comment on this to the final version and thank the reviewer for making us aware of the need for clarification.

 

A2.6 - Addressing Q1: Comment about the necessity of weights of a folding model or evolutionary information

We agree with the reviewer and identify this as one of the core ideas that underlie our method. We would like to note that also all other reviewers find the idea 'elegant', 'technically compelling' or 'conceptually reasonable'.

 

A2.7 - Addressing Q6: typo

We corrected the typo and thank the reviewer for pointing this out!