gRNAde: Geometric Deep Learning for 3D RNA inverse design

... The authors should provide a reasonable explanation or correct all descriptions in the paper that claim "without considering 3D geometry."

The first line of the abstract is more of a stylistic choice than a technical assertion. Our intention is to illustrate that most research on RNA inverse design has focussed on 2D structure except Rosetta (2010), until recent works like RDesign and gRNAde revisited the task from the 3D perspective. This review supports our sentence: https://academic.oup.com/bib/article/19/2/350/2666340 (all the methods they benchmark are 2D-only)

We have not made the claim that other methods do not consider 3D geometry anywhere else in the paper or that we are the first to do so.

In the abstract, gRNAde should be compared with RDESIGN rather than Rosetta, which would be more reasonable.

We found Rosetta to outperform RDesign in the single-state benchmark of 3D RNAs of interest identified by Das et al. (2010) in Figure 2, so we thought it was reasonable to state metrics w.r.t. the next best performing method.

Rosetta is also very well known in the broader biomolecule design community, so we think this will help readers better contextualize our work if they only skim the abstract.

Figure 3(a) shows the performance of gRNAde under different numbers of conformations. This experiment also needs to be compared with RDESIGN. Although RDESIGN can only accept one conformation, the authors should use the result of RDESIGN with a single conformation as a baseline for comparison.

Thanks for the suggestion - we have added the result for RDesign into the revised manuscript Figure 3. We found that gRNAde (0.48-0.53) outperforms RDesign (0.38) in the multi-state design benchmark, too.

The experimental section lacks a comparison with RDESIGN, which is also deep-learning-based.

We have directly compared gRNAde to Rdesign. See Figure 2a, Appendix Table 2, as well as Figure 6. We also added RDesign results to Figure 3 as per your suggestion. Overall, we found that gRNAde outperforms RDesign.

Because the author added a comparison with RDesign, I increased my score.

However, regarding the issue of "without considering 3D geometry," I still believe it has not been resolved. First, I acknowledge that the author's statement, "We have not made the claim that other methods do not consider 3D geometry anywhere else in the paper or that we are the first to do so," is correct. Strictly speaking, this may not be considered a real-time error. However, my points are as follows:

1. Writing in this way in the abstract is misleading and can lead readers to believe that this paper aims to address the issue of past methods not considering 3D geometry. Since this paper is not the first to solve this problem, I think it is inappropriate to write it this way.
2. I do not consider considering 3D geometry to be an interesting innovation, and the negative impact of writing it this way outweighs the positive impact. Even ignoring RDesign, inverse folding based on 3D structures has already been a very classic method in proteins, such as ProteinMPNN. It is clear that such methods can be transferred to RNA, so I do not think considering RNA 3D geometry is an interesting innovation. What truly impressed me about this paper is the idea of considering multiple conformations, which is the main content. The authors should emphasize the idea of considering multiple conformations more fully. However, in the abstract, the authors also emphasize considering RNA 3D geometry, which distracts the reader's attention. It shifts the reader's focus from a very interesting idea (multiple conformations) to a relatively uninteresting point (3D geometry). Therefore, I believe it is a loss.

1. Does the model incorporate RNA modifications, cellular localization, or folding kinetics?

No, gRNAde or any other inverse folding models that we are aware of (whether for proteins or RNA) do not consider modified nucleotides/residues nor cellular localization yet. There is very little data in the PDB with modified nucleotides/residues. Cellular localisation may not always be relevant for inverse folding depending on the application (eg. biotechnology/sensing applications). There is almost no data on folding kinetics of RNAs except studies of individual RNAs by J. Lucks' lab.

In summary, we don’t have sufficient data for all 3 of those to be able to train ML models, as far as we are aware.

2. On metrics apart from sequence recovery and perplexity

We have included 2D and 3D structural self-consistency metrics in addition to the above. We have also discussed pros/cons of the various metrics in Section 2.3 and practical considerations of ranking designs in Appendix G.

Regarding functional motifs – in practical RNA design applications, if specific functional/structural motifs are required as part of the design, gRNAde allows users to fix the identities of specific positions that participate in these motifs. The notebook design.ipynb in the anonymous codebase shows this in cell #4.

3. Different RNA backbone representations

We did not try the consensus all-angle conformers nor modular string nomenclature.

When we started development of gRNAde, we did preliminary experiments on the choice of backbone atoms to be used during featurization (Figure 13). Anecdotally, we found that using a 7-bead representation (over a 3-bead representation used in our final model) tended to fit training data better but produced the same or worse performance on the held-out test set. There is evidence from RNA structural biology literature that the 3-bead representation describes RNA backbones completely in most cases while reducing the size of the torsional space to prevent overfitting (Wadley et al., 2007).

We haven’t formally ablated this design choice but thought you may find this interesting.

4. Failure cases and limitations

In experimental validation, we see that both gRNAde and Rosetta do not succeed on the GMP Riboswitch in its bound state (with its partner ligand) extracted from a longer mRNA, as well as the RNA component of the Telomerase enzyme (a protein-RNA complex). In both cases, the OpenKnot score of the Wildtype sequence also falls below 80. This suggests two failure modes:

1. Lack of biological context: Designing sequences for RNAs that have been extracted/removed from their broader biological context (a partner ligand and longer mRNA, or a protein-RNA complex) is more challenging than RNA-only design scenarios. We have noted this point as a limitation and avenue for future work in the Conclusion section, too.
2. Dynamics: The fact that the Wildtype sequence has tendency to not fold into the given target backbone structure suggests that the RNA may be more likely to be conformationally flexible (particularly known for Riboswitches). This was also a general observation from our computational benchmarks in Appendix C – the multi-state test set was significantly more challenging than the single-state test set in terms of all metrics.

In summary, gRNAde and other structure-based design tools are best suited for tasks where folding stability is highly correlated with function. In addition to folding stability, natural RNAs are subject to other functional constraints, especially binding of partners, which we hope to incorporate into the next version of gRNAde.

5. RNA sequence lengths affect on performance

We have seen overall minor variations in sequence recovery with length. For instance, on the single-state benchmark:

6. Will the authors benchmark against the RNAInvBench?

We weren’t aware of this work as it is pretty recent, but we have now cited the benchmark in the Related Work section in the revision. RNAInvBench focuses on 2D structure-based design (emphasis on base pairing) while our work focuses on 3D geometry (emphasis on non-canonical interactions and tertiary motifs).

Any 3D RNA structure is folded into a specific shape due to tertiary interactions such as pseudoknots, kissing loops, or G-quadruplexes. Therefore:

1. All the RNA structures that we train and evaluate on from RNAsolo/PDB are already in the ‘pseudoknot-inclusive’ regime.
2. As far as we know, there are no ‘pseudoknot-free’ 3D RNA structures that we can use, b/c pseudoknots and other tertiary interactions are the drivers of RNA folding into 3D structures.

My impression is that current versions of Rosetta do not support RNA design. Why is that? ... Is it a strong enough baseline to be compare against?

As far as we understand, there has not been enough support to maintain the Rosetta RNA tools since early 2010s so they have not been included in the latest Rosetta builds. Recent deep learning-based tools like gRNAde represent a revival of interest in this problem: they are GPU-based/fast, more broadly accessible, and the designed RNAs have a higher experimental success rate than Rosetta! (We are now actively working on new biotechnology applications with experimental collaborators.)

3D structure-based RNA design has been a niche topic until recently, so we are not aware of other, better computational tools for this task. Thus, Rosetta is the state-of-the-art in physics-based RNA inverse folding to the best of our knowledge.

Rosetta is also very well known in the broader biomolecule design community, which helps readers better contextualize our work.

... methodological similarities/differences vs. RDesign in the main paper ... Can you provide a short piece of text that you'd include in a new version of the main paper describing the overlap?

See page 7, line 330 onwards in the revised manuscript. We have included the following short piece of text summarizing key methodological differences: “RDesign is a concurrent deep learning-based 3D inverse folding model which uses invariant GNN layers and non-autoregressive decoding without sampling; see Appendix B for details.”

Dear Reviewer,

We are reaching the end of the discussion period (November 26 at 11:59pm AoE): please check the author rebuttal and post your response at your earliest convenience. Please also update your review accordingly, even if your view of the paper has not changed (to acknowledge that you have read the rebuttal).

Thank you, --Your AC

...briefly summarize the technical components of Rosetta

We have updated the Related Work section’s description of Rosetta and how gRNAde is different in page 15, line 763 onwards.

...focus their discussion on the technical similarities and differences to previous work, without negativity

We have revised the statements in page 16, line 836 onwards to not be negative.

There is no wet lab comparison to RDesign, and the authors should explain why this is not possible

The Eterna platform is independent to us and we are restricted in the number of designs we can submit and when they can be submitted. We have chosen to prioritize our model and have compared it to Rosetta (with the help of Eterna organizers). Rosetta outperforms RDesign in computational benchmarks, which further justifies prioritizing Rosetta.

Please explain what happens to the other non-successful examples from Eterna...

Both gRNAde and Rosetta do not succeed on the GMP Riboswitch in bound state extracted from a longer mRNA, as well as the RNA component of the Telomerase enzyme (a protein-RNA complex). In both cases, the OpenKnot score of the Wildtype sequence also falls below 80. This suggests two failure modes:

1. Lack of biological context: Designing sequences for RNAs that have been extracted/removed from their broader biological context (a partner ligand and longer mRNA, or a protein-RNA complex) is more challenging than RNA-only design scenarios. We have noted this point as a limitation and avenue for future work in the Conclusion section, too.
2. Dynamics: The fact that the Wildtype sequence has tendency to not fold into the given target backbone structure suggests that the RNA may be more likely to be conformationally flexible (particularly known for Riboswitches). This was also a general observation from our computational benchmarks in Appendix C – the multi-state test set was significantly more challenging than the single-state test set in terms of all metrics.

In summary, gRNAde and other structure-based design tools are best suited for tasks where folding stability is highly correlated with function. In addition to folding stability, natural RNAs are subject to other functional constraints, especially binding of partners, which we hope to incorporate into the next version of gRNAde.

Would the method generalize to unseen datasets/sources?

Yes, we have been careful to evaluate gRNAde on test sets which evaluate for generalization to new RNA structures and folds unseen during training. Details are provided in Section 3, ‘Splits to evaluate generalization’. The Ribozyme in Section 4.3 is also unseen and gRNAde’s perplexity is useful for ranking its mutants’ fitness.

Do these properties affect performance results when the model is evaluated?

Yes, the length and structural flexibility of an RNA backbone do impact gRNAde’s performance.

• Flexibility: See Figure 3 (b) and the multi-state design benchmark – we find that performance is decreased in structurally flexible regions of RNA backbones, as characterized by increasing RMSD between the available states as well as by undergoing base pairing changes.
• Length: We have seen overall minor variations in sequence recovery with length. For instance, on the single-state benchmark:

Are there any biases of the model...

The model reflects the biases present in its training data (from the PDB), which is ‘a feature and a bug’. Thus, designs from gRNAde are likely to be ‘PDB-like’ sequences which are biased towards thermal stability. In practical scenarios, GC content can be enforced via logit biasing. This involves adding or removing a small bias term to the output logits during autoregressive decoding (the model samples the choice of base from its logits at each position during inference).

The notebook design.ipynb in the anonymous codebase actually shows how one can bias gRNAde’s sampling in practice – see cell #4.

Will this new dataset also be released?

Yes, all the datasets are available in processed ML-ready format for anyone to develop new ideas on top of (RNAsolo provides raw PDB files). We have also released scripts which ingest the raw PDB files and prepare processed data. See ‘Downloading and Preparing Data’ section of README.md in the anonymized code. Our datasets have already been used by other researchers. We initially didn’t consider this major enough to be listed as a contribution, but thanks for noticing this.