Dynamics-Informed Protein Design with Structure Conditioning

Dear Reviewer E2Vz,

Thank you for your valuable time and feedback, including points on figures 1, 2 & 3 – they help us improve the content and clarity of the manuscript.

On evaluation:
We address your concern on “weak evidence to support the method works” in the general response. Briefly, in addition to visual inspections and evaluating the NMA loss, the current quantitative evaluation, we newly add a self-consistent NMA (scNMA - explained in the general response) evaluation to our evaluation pipeline and are currently running experiments to assess it.

On your detailed points:

• “The filtering in section 5.1 is very ambiguous [...]” Apologies for any confusion caused. Out of 300 samples, for strain targets conditioned samples, 18 samples failed with NaNs, and 63 violated distance filter. Out of 300 samples, for random targets, 38 samples failed with NaNs, and for 62 samples violated distance filter. The unconditional samples never failed with NaN, but about 40-50 violated the chain distance filter. The NaN values occur due to what we previously referred to as “blow up”, explained in more detail in the next point.

• “What does ‘blow up’ mean?” We apologise for the use of informal language and updated this statement in the manuscript. By ‘blow up’ we refer to a process where a coordinate increases by orders of magnitude along the sampling trajectory, or even becomes NaN. This phenomenon tends to happen when the conditioning pushes a sample’s coordinates outside of the realm of observed samples for which the denoiser was trained (and therefore it predicts mostly random scores there). This phenomenon underlines that finding a right balance between the NMA-loss driven part of the score and the unconditional part of the score requires striking a delicate balance, one of our contributions. This “divergence” effect is not unique to the denoiser we use and it has been generally found that many other large diffusion models working with image data suffer from diverging features in the sampling. In images they usually combat this with the ‘thresholding trick’ - they clip the feature values to never exceed some pre-defined range. Within the image domain that results in samples with too much colour saturation. For instance, in Stable Diffusion, which uses weights to balance conditional score and the unconditional score, too much weight on the conditional score leads to mode collapse and out of distribution samples [https://arxiv.org/pdf/2311.00938.pdf].

• Q: “Remarkably, Genie outperformed other models such as [..] Why is this remarkable?” - Apologies for the confusing phrasing. We meant for it to be read more like “Note that”. This is updated in the manuscript. We mention the capabilities of Genie model as compared to other models to explain our design choice to use Genie in the experiments. In this work we were not investigating how to improve the sample quality in the unconditional generation, but we needed to use a reasonable unconditional model to test the conditioning methodology. We chose Genie because of its satisfactory performance that according to the Genie paper [https://arxiv.org/abs/2301.12485] outperforms other models, as well as because of its code availability.

• Q: “ the mean chain distance that should be close to 3.8 A Where does 3.8 come from?” - thank you for this comment, we should have clarified that we refer to the distances of subsequent alpha carbons in the backbone here. The alpha carbon atoms (CA) in subsequent amino acid residues in a protein are physically constrained to lie in a very narrow range of around 3.8 Angstrom [c.f. e.g. https://www.wiley.com/en-us/Biochemistry,+4th+Edition-p-9780470570951]. Here we consider the mean $C_\alpha-C_\alpha$ distance, that is known to be close to 3.8 A.

• Q: “Do the values of NMA loss differ for strain or random targets? Do you expect them to?” – Good question! As illustrated in Figure 2, the loss values are indeed fairly comparable between the two situations, however, we find it hard to give an intuition as to what to expect in these general cases. Instead, we would like to highlight that “strain” and “random” are meant mostly as more general demonstrations that the conditioning works for a variety of target definitions and substructures and that we use two kinds of targets in order to show that NMA-loss can be minimised regardless of how we choose the target. The “hinge” target formulation on the other hand is meant to capture the more application relevant situation of trying to design a certain motif with a targeted low mode behaviour.

• Q: ““In the analysis of the remaining samples, we considered the distributions of NMA-loss and scTM-score (Figure 5). Isn’t the second figure RMSD?” - apologies for the confusing phrasing here. The distribution of NMA-loss is presented at Figure 5, scTM-score is discussed later in the text. We updated the phrasing in the text.

Dear Reviewer 65X8,

Thank you for your comments and suggestions on how to improve the paper.

We hope that we address your comment about the novelty in the overall response. Briefly, we agree with you that our approach utilises the existing guidance framework (reconstruction based guidance). Yet we would like to highlight the simplicity of the method as an advantage rather than a drawback of our work. We would also like to emphasise that this particular choice of guidance strategy and schedule is not arbitrary, but was carefully chosen to match the strengths of NMA: In contrast to classifier-based guidance, reconstruction guidance results in more “protein-like” starting points for NMA analysis, for which it is more applicable. This is manifested by predicting the current estimate of the denoised position $\hat{x}_0$ via Tweedie’s formula before applying NMA instead of using the more noisy $x_t$, which would be used in standard classifier-based guidance and would require training an additional network to predict dynamics based on noisy version of the sample. We further tuned the guidance schedule to start only after a “burn-in” period in the sampling trajectory, after which a sufficiently protein-like structure is formed.

We also discuss the points about evaluation in the general response, since they were of interest to multiple reviewers. For a short summary here, in addition to visual inspections and evaluating the NMA loss, the current quantitative proxy evaluation you mention, we newly add a self-consistent NMA (scNMA - explained in the general response) evaluation to our evaluation pipeline and are currently running experiments to assess it. Akin to the self-consistency metrics which are used in structure conditioning, this metric will evaluate how well the targeted mode is captured by the inverse folded & re-structure-predicted protein. We agree that evaluations which include molecular dynamics simulations will be a further, more powerful test that are out of scope for this current work, but that we plan to perform in the future.

Please find here the answers to your other questions.

1. The guidance scales were indeed fine-tuned per target type, as is also common elsewhere in the field [1,2]. The same guidance scales were used for all samples with strain targets (when multiple sampled proteins had varying lengths), the same for all random targets (again, multiple sampled proteins with varying lengths), and there were different guidance scales for each of the hinge targets (per each target, many proteins, same lengths). Please find references to the works showing that fine-tuning guidance scale is a common practice in the field.

2. Why is the guidance scale tuned on in the middle of sampling? In our loss-guided diffusion, in the sampling at step t>0 we rely on a reasonably good prediction of sample $\hat{x}_0$ at t=0 via Tweedie’s formula. This is important to be in the realm of applicability of NMA, which is robust to slight changes in coordinates but not backbone topology. Early in the sampling the model estimate of the sample at t=0 is poor and fluctuates strongly such that starting conditioning too early has a negative impact on sample quality. We therefore tune the guidance scale on midway through the sampling process, when the sample has become more protein-like. We note that similar techniques (i.e. timestep dependent losses) are used by other works in the field, such as in FrameDiff [https://arxiv.org/abs/2302.02277], where the auxiliary loss on the atoms distances was switched on only when t<0.5*T_max.

[1] Dhariwal and Nichol, “Diffusion Models Beat GANs on Image Synthesis”, https://arxiv.org/pdf/2105.05233v4.pdf

[2] Chung et al. , “Improving Diffusion Models for Inverse Problems using Manifold Constraints”, https://arxiv.org/pdf/2206.00941.pdf

Dear Reviewer FtGW,

Thank you for thoroughly reviewing our manuscript and giving your insightful comments. We now improved the clarity of our manuscript according to your suggestions. Where suitable, we added pointers to the modifications in the manuscript that show where your concern has been addressed.

Please also note the general points on evaluation and novelty that we discuss in the general response, since some questions were raised by several reviewers.

Specifically to your point on how we envision this method to be used:
If we have a selected motif (e.g. an enzymatic active site or pocket, binding cleft/pocket) that performs backbone rearrangements that are well captured by NMA during biological function (e.g. hinge or shear type backbone rearrangements), then this could be “grafted” onto a new scaffold that not only preserves the geometry of the motif, but also creates a scaffold backbone that promotes the relative motion of the motif that is needed to achieve the biological function. This idea is driven by the observation in the NMA community that some protein backbones pre-determine them for certain types of motion [https://pubmed.ncbi.nlm.nih.gov/16143512/], and that this motion is – for selected classes of proteins – the biologically relevant motion.

We envision this to be useful to transfer functional motifs to different backbones which might have different additional binding domains to confer specificity, or lack thereof to promote promiscuity, while preserving the topology required for the functional motif’s primary motion. Besides coming from an existing protein, a functional motif with a dynamic behavior (e.g. a pincher motion) could also be the result of a protein engineering or design hypothesis.

Below is a point-to-point response to your further detailed comments: (minor clarity/typo related points are not listed here, but addressed in the revised manuscript and highlighted with blue boxes)

• Page 4 eq (8) - in the current model specification, a weight factor in the loss function does not affect the final result Consider our loss in Equation 1 after we cast it to the form required in the conditional score (Equation 15) with an added temperature $\gamma$:

with the guidance-scale $\lambda_t$. After reparametrisation $\hat{\lambda}_t = \frac{\lambda_t}{\gamma}$

the temperature $\gamma$ has no effect as $\hat{\lambda}_t$ is tuned.

• Page 5, "x_C_M \in ... is the expected position of conditioned residues at t=0 as sampling progresses" - We rephrase that entire sentence in the updated manuscript, but please find additional clarification here. At each step in the sampling, which progresses from t=T_max to t=0, in order to calculate the NMA-loss or the structure loss we must obtain the model prediction of the residues of interest in the final backbone at t=0. By that we mean the “expected positions of conditioned residues at t=0“. Importantly, this is what differentiates the guidance paradigm we use here (reconstruction based guidance) from general classifier-based guidance – where a predictor network (or function) is needed that can predict using noisy data. While NMA is somewhat robust to noise, by predicting the current estimate $\hat{x}_0$ via Tweedie’s formula, we apply NMA to a more protein-like structure ($\hat{x}_0$ instead of $x_t$), moving us closer to the realm in which NMA is valid.

Dear Reviewer M3zP,

Thank you for your comments and in particular the note on Section 3.1 with a suggested read. We now added the paper you recommend to the discussion of the design choices in Section 3.1.

Regarding your concern on the novelty of our method/content of the paper:
We agree with you that our approach utilises the existing guidance framework (reconstruction based guidance). Yet we would like to highlight the simplicity of the method as an advantage rather than a drawback of our work. We would also like to emphasise that this particular choice of guidance strategy and schedule is not arbitrary, but was carefully chosen to match the strengths of NMA: In contrast to classifier-based guidance, reconstruction guidance results in more “protein-like” starting points, for which NMA is more applicable. This is manifested by predicting the current estimate of the denoised position $\hat{x}_0$ via Tweedie’s formula before applying NMA instead of using the more noisy $x_t$, which would be used in standard classifier-based guidance. We further tuned the guidance schedule to start only after a “burn-in” period in the sampling trajectory, after which a sufficiently protein-like structure is formed.

The problem of conditioning on dynamics has been widely discussed, but bringing the utilising and evaluating normal mode analysis as a conditioning target is a novel contribution of our work. Furthermore, we do not only propose this method but empirically demonstrate that we can generate samples that are still valid proteins but fulfil the targeted normal mode characteristics at the same time.

Please find more discussion of these important points you raise also in the general response above.

Regarding your concern on the evaluation:
We agree with you that this work should be followed up by more in-depth evaluations via e.g. molecular dynamics simulations, however for this work we wanted to focus on establishing the principle behind our method. We are however currently running additional evaluation experiments to further support our claims (“scNMA” - self-consistency of NMA characteristics after inverse folding); for more information on this, please see the general response.

Dear Reviewers,

We have now updated the manuscript with the additional experiments in the Appendix G. We hope you will find them helpful in addressing any of your remaining questions.

Please find a quick summary below:

• The additional experiments extend our evaluation framework with a self-consistency NMA score (scNMA). This metric quantifies whether the structure obtained from the pipeline: Genie $C_\alpha$ backbone -> ProteinMPNN inverse-folding -> ESMFold folding has the NMA-loss close to the value derived from the backbone.
• In the conducted experiments we focused on a target that illustrates a biologically relevant scenario: a segment of 22 residues containing the lysozyme active site, that is involved in the lysozyme hinge motion. To evaluate the sample quality and the method’s success, we used the previously described scRMSD, scTM, pLDDT and pAE criteria.
• We found that for the chosen target, the structure conditioning problem is a challenging task itself. Motif-only conditioned samples had low designability scores. Joint motif+NMA conditioning surprisingly improved the designability score slightly
• The extra evaluation confirms that indeed our methodology allows for generating raw $C_\alpha$ backbones with the targeted dynamics, which we believe is a major contribution of our work: Only 2/150 ($\sim1.3$%) of motif-only backbones achieve $\text{NMA-loss} < 0.5$, as opposed to 10/43 ($\sim23$%) for motif+NMA -- corresponding to a roughly 17x-fold enrichment, while achieving comparable motif scaffolding performance (c.f. Fig. 11 & 12).
• However, much of this benefit seems to disappear in the process of inverse-folding and the subsequent re-folding. After inverse-folding and re-folding, joint motif+NMA conditioning still increases the relative chance of obtaining a sample with a low scNMA-score ($3/43 \sim 7$% of samples below $0.5$) as compared to motif-only conditioning ($3/150 \sim 2$% below $0.5$), roughly 3-fold, but the difference is much less pronounced than for the NMA-loss of the designed backbone (original NMA-loss).

We would like to again highlight the contribution of our work which is a methodology for dynamics-informed backbone design. This is a first work aiming at directly integrating dynamics information to the protein diffusion model and we hope it will be of benefit to a wider scientific community.

We hope you find the new analysis satisfactory and encouraging for the score reconsideration. Thank you for your constructive feedback that has been invaluable in advancing our evaluation pipeline.