RNA FrameFlow: Flow Matching for de novo 3D RNA Backbone Generation

Dear Reviewer FB4S,

Thank you for your time and valuable feedback on our paper. We appreciate the criticisms which go into improving the quality of our submission. We respond to the questions and weaknesses brought up above. We have also uploaded a new PDF for your reference.

We ablate the number of sampling steps $N_T$ in Table 1 in Section 4.1. To summarize, increasing the number of sampling steps beyond an optimal amount does not yield any performance improvement. The number of sampling steps is conventionally a hyperparameter in flow matching that is tuned for a specific dataset and model – it is known that having more steps does not always mean higher quality samples – and we found the optimal steps to be $N_T = 50$.

Could we please clarify the difference between “number of sampling steps” vs “number of timesteps”? For our Flow Matching model, the number of sampling steps during inference is equal to the number of interpolation steps during training, which is the common practice.

Could you conduct ablation studies to demonstrate how the added auxiliary losses contribute to performance improvements?

Indeed. We provide ablations of the auxiliary losses and their impact on performance (ie, the local and global metrics) in Table 6 in Appendix C2. To do so, we drop/include each loss from our total training loss for each re-run and compute the evaluation metrics. We also visually show artifacts from generated samples when ablating each loss in Figure 8 in Appendix C2.

To summarize, we observe that including ALL the losses offers the best performance. As for the relative importance of each loss, we believe it is in this order: $L_{SO(3)}$ to place the frames with the right orientation in 3D, $L_{bb}$ to ensure the frames are contiguous, and $L_{dist}$ to ensure intra-frame atoms are placed properly with minimal clashes.

Could you compute the RMSD between the generated structure and the folded structure for a more comprehensive evaluation?

We include the scTM and scRMSD scores in Section 4.1 for our best-performing model. However, we believe RMSD might not be the best metric since our structure predictor, RhoFold, predicts backbones that may not perfectly correspond to the crystal structure, as shown in Figure 7 in Appendix B.2.

The reliance on folding and inverse-folding models, which have their own biases, affects RNA-FRAMEFLOW’s overall performance and the validity of generated designs.

We include an assessment of the loss in validity/self-consistency due to using gRNAde and RhoFold in Appendix B.3. We notice that RNA-FrameFlow generates backbones that closely match the validity of gRNAde sequences.

We hope this clarifies the questions. Please let us know!

Hello Reviewer FB4S,

We additionally include the scRMSD computed using the frame atoms {C4', O3, C3'} to factor in the rotation component, not just the translation:

We observe that these scRMSD values correlate well with those in our original submission. For sequence lengths with high % validity, we see relatively lower scRMSD values and higher scTM scores.

Do let us know if we can clarify anything else!

Dear Reviewer Bgio,

Thank you for your time and valuable feedback on our paper. We appreciate the criticisms which go into improving the quality of our submission. We respond to the questions and weaknesses brought up above:

RoseTTAFoldNA constructs frames based on the phosphate group (P, OP1, and OP2), this paper bases its construction on C3′−C4′−O4′…the authors do not provide compelling reasons…

We pick the atoms {C4’, C3’, O4’} to create the frame comprising the rigid rotations and translation of the C4’ atom. Aligning with the medicinal chemistry literature [1], we wanted the atoms to be close to the centroid of the nucleotide to prevent the accumulation of errors when autoregressively placing the non-frame atoms. It also allows us to implicitly capture intra-nucleotide motions like ring puckering that occurs in the ribose sugar ring containing our frame atoms. We document the choice of frame atoms in more detail in Section 2.1.

RF2NA’s choice of frame is actually completely arbitrary (no biological motivation) and has the following drawbacks: Assuming rigid/fixed geometry at P, OP1, OP2 is not supported by literature, unlike our choice of C4’, C3’, O4’ [1]. Placing the remaining atoms via predicted torsion angles w.r.t. the P atom can lead to large error accumulation as the P, OP1, OP2 (phosphate group) is on one extreme of a nucleotide.

I think this paper needs to highlight some other insights, such as what problems arise when applying frame flow to RNA and how they should be addressed.

In Appendix B.3., we now have a section on the upper bounds of performance of bringing over the self-consistency pipeline seen in protein design (as used by RFDiffusion [2] and FrameDiff [3]) and porting it to RNA design. Given the current state of RNA folding and inverse folding tools, we observe that the generated backbones from RNA-FrameFlow closely retain the validity and self-consistency of gRNAde sequences.

We also include ablations on the auxiliary losses that capture the more complex nucleotide structure (with 13 atoms) in Appendix C.1., documenting how naively applying FrameFlow to RNA design might not immediately yield results and requires introducing RNA-specific inductive biases that account for structural differences to proteins.

We hope this clarifies the questions. Please let us know!

[1] Ribose puckering: structure, dynamics, energetics, and the pseudorotation cycle. Harvey & Prabhakaran. 1986.

[2] De novo design of protein structure and function with RFdiffusion. Watson, J.L., Juergens, D., Bennett, N.R., Trippe, B.L., Yim, J., Eisenach, H.E., Ahern, W., Borst, A.J., Ragotte, R.J., Milles, L.F. and Wicky, B.I.. Nature, 620(7976), pp.1089-1100. 2023

[3] SE(3) diffusion model with application to protein backbone generation. Yim, J., Trippe, B.L., De Bortoli, V., Mathieu, E., Doucet, A., Barzilay, R. and Jaakkola, T.. 2023

Dear Reviewer Dcv1,

Thank you for your time reading our submission and providing feedback on enhancing the quality of our work. We respond to the critiques brought up above. We have uploaded a new PDF for your reference.

Does the selection of frame atoms involve some rigid assumptions?

Yes. We pick the atoms {C4’, C3’, O4’} to create the frame comprising the rigid rotations and translation of the C4’ atom. Aligning with the medicinal chemistry literature [1], we wanted the atoms to be close to the centroid of the nucleotide to prevent the accumulation of errors when autoregressively placing the non-frame atoms. It also allows us to implicitly capture intra-nucleotide motions like ring puckering that occurs in the ribose sugar ring containing our frame atoms. We document the choice of frame atoms in more detail in Section 2.1.

Could you provide the average scTM for each method in addition to the validity metrics?

We have included the average scTM and scRMSD metrics in Section 4.1 (colored red). We have also included the per-sequence-length scTM and validity in Figure 8 in Appendix B.3..

It would be beneficial to provide the training distribution and compare it with the TM-score distribution across different sequence lengths for better analysis.

We provide this in Figure 3 in Section 4.1. We show the scTM scores across sequence lengths. Do let us know if there’s something specific we can elaborate on here. We also include the average scTM and scRMSD scores for our generated backbones in Section 4.1.

One approach could involve sampling a similar distribution of structures from the test set and computing self-consistency scores using the inverse folding–structure prediction pipeline.

We thank the reviewer for this suggestion. We have now included a section on assessing our self-consistency pipeline in Appendix B.3 (colored red), quantifying how well gRNAde and RhoFold influence the validity of samples. We apply our pipeline to ground truth samples from RNAsolo to measure recovery.

We hope this clarifies the questions. Please let us know!

[1] Ribose puckering: structure, dynamics, energetics, and the pseudorotation cycle. Harvey & Prabhakaran. 1986.

Dear Reviewer WjH5,

Thank you for your time reading our submission and providing feedback on enhancing the quality of our work. We respond to the critiques brought up above. We have also updated our submission PDF.

Could you elaborate on the limitations and potential errors of gRNAde that might impact the self-consistency evaluation of backbone generation?

We include an assessment of gRNAde and RhoFold on the RNAsolo training set to see how it influences self-consistency in Appendix B.3. We observe that the generated backbones from RNA-FrameFlow closely retain the validity and self-consistency of gRNAde sequences.

Additionally, gRNAde has been experimentally validated in the wet lab with a success rate of 50% at forming the target backbone structure across diverse RNAs; see https://openreview.net/forum?id=lvw3UgeVxS – this is close to our computationally determined validity rates at different sequence lengths.

Include the results of the 50/50 split calculation of the training set as a reference to provide a clearer understanding of the EMD's numerical scale.

We now include EMD scores for the local structural descriptors from the 50/50 training set random split in Table 2 in Section 4.2 (colored red). We see very high similarity and low variance in the statistics from both these sets.

Consider incorporating results from multiple structure prediction models to provide a more comprehensive evaluation of your method's performance.

We use Chai-1 [1] as an alternative structure predictor and include the self-consistency results in Table 7 in Appendix C.3. We see similar validity scores to RhoFold across sequence lengths.

Additionally, several recent benchmarks of 3D RNA structure prediction (latest example) have found that open-source models such as RhoFold, trRosettaRNA, RF2NA, etc. all perform very similarly due to architectural and training data similarities, with minor differences for specific targets/families.

We have also recently obtained the academic-use weights for AlphaFold3 and are incorporating it into our pipeline.

How frequently do the generated 3D backbones exhibit clashes, and what is the average proportion of nucleotides involved in these clashes?

We include a detailed analysis of steric clashes in Appendix D.4. We describe how we define a steric clash (aligning with popular tools like MolProbity [2]) and measure the clashes from RNAsolo and the generated backbones across sequence lengths.

A more effective approach might be to explore whether the generated structures encompass diverse RNA types or families (e.g., by calculating structural or sequence similarity)

We also use a metric called pdbTM to show how structurally different our generated backbones are from existing samples in RNAsolo, which aligns with the reviewer’s comments on “calculating structural similarity”. The best-performing RNA-FrameFlow model has an average pdbTM of 0.54 (1 being perfect). We define how we measure pdbTM in Section 3 on evaluation metrics.

can be further enhanced by further checking whether the generated RNA structures contain some common tertiary interaction motifs, such as pseudoknots and base multiplets, rather than just base pairing and stacking.

These interaction motifs like pseudoknots, bulges, etc are determined by interactions between the atoms in the bases, and not the backbone atoms. The reviewer’s comment on this is valid but we cannot compute this as we only deal with backbone atoms, not base atoms. We have included structural clustering as a data augmentation that only deals with the tertiary structure and not any base interactions.

We hope this clarifies the questions. Please let us know!

[1] Chai-1: Decoding the molecular interactions of life. Chai Discovery, Boitreaud, J., Dent, J., McPartlon, M., Meier, J., Reis, V., ... & Wu, K. 2024.

[2] MolProbity: More and better reference data for improved all-atom structure validation. Williams et al. 2018.

Thank you for your time and consideration! Do let us know if there's anything we can clarify.

Dear Reviewer Vgi9,

Thank you for your time reading our submission and providing feedback on enhancing the quality of our work. We respond to the critiques brought up above. We have also uploaded a new PDF for your reference.

The model’s training set, derived from a limited pool of 3D RNA structures, restricts RNA-FRAMEFLOW’s generalization ... limiting its novelty in unseen scenarios.

The points raised in the Weaknesses are valid, however, we believe they can be characterized as general limitations of our current pipeline which we have honestly brought to light in our submission. We hope the broader community will work towards addressing some of these interesting scientific questions about data preparation and rigorous evaluation of generative models for relatively niche, non-protein biomolecules like RNA.

How might explicit modeling of RNA base pairing and stacking interactions (e.g., incorporating hydrogen bonding) impact RNA-FRAMEFLOW’s ability to produce realistic structures?

We have tried incorporating base stacking and pairing by including the base atom N1/N9 in our auxiliary backbone losses. This leads to the model’s loss emphasizing more on the coordinates as well as pairwise distances between nearby N atoms (which signify the starting position of the base) – hence explicitly emphasizing base pairing and stacking interactions. The results are included in Table 5 in Appendix C.1. For the loss including the Nitrogenous base atom, we see an increase in validity (ie, realism) from 41.0% to 46.7% but with a decrease in diversity and novelty. We believe combining such an approach with data augmentations and novel folds could mitigate the hit to diversity and novelty.

Given the length bias of RhoFold, do the authors have plans to explore alternative evaluation tools that might yield less length-biased results?

Yes. Recently, two open-source RNA structure prediction tools have emerged, Chai-1 [1] and Boltz-1 [2]. We compare to Chai-1, a recent open-source structure prediction tool with similar results to AlphaFold2. We include these self-consistency results in Table 7 in Appendix C.3. We do not notice a significant change in validity scores across sequence lengths. In fact, several recent benchmarks of 3D RNA structure prediction (latest example) have found that open-source models such as RhoFold, trRosettaRNA, RF2NA, etc. all perform very similarly due to architectural and training data similarities, with minor differences for specific targets/families.

We have also acquired AlphaFold3’s academic-access weights and are incorporating it into our self-consistency pipeline.

Could the clustering and cropping data augmentation methods be adjusted to avoid introducing invalid folds?

Yes, we are currently experimenting with a cropping augmentation that refolds the extracted subsequence during batching rather than blindly snipping it from the original structure. We believe this may introduce more diverse and realistic folds into the training set. However, this is still limited by the availability of an accurate structure predictor.

We hope this clarifies the questions. Please let us know!

[1] Chai-1: Decoding the molecular interactions of life. Chai Discovery, Boitreaud, J., Dent, J., McPartlon, M., Meier, J., Reis, V., ... & Wu, K. 2024.

[2] Boltz-1: Democratizing Biomolecular Interaction Modeling. Wohlwend, J., Corso, G., Passaro, S., Reveiz, M., Leidal, K., Swiderski, W., ... & Barzilay, R. bioRxiv, 2024-11. 2024.

[3] Accurate structure prediction of biomolecular interactions with AlphaFold 3. Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., ... & Jumper, J. M. Nature, 1-3. 2024.