gRNAde: Geometric Deep Learning for 3D RNA inverse design

Thank you for your encouraging and actionable review! We believe our revised paper will be strengthened by incorporating your suggestions. We hope our rebuttal further addresses your questions and concerns.

Question 1

• We have ablated the inclusion of long (primarily ribosomal) RNAs in gRNAde’s training data, which also serves to tell us how # training samples and maximum length impact performance. See Appendix D, Ablation Study and in particular the results shaded in yellow under ‘Max. train RNA length’. Appendix Figure 15(a) is also relevant to show the length distribution.
• Number of training samples corresponding to maximum length cutoffs:
  • cutoff @ 500 → 2607 samples
  • cutoff @ 1000 → 2876 samples
  • cutoff @ 2500 → 3467 samples
  • cutoff @ 5000 → 4022 samples
• Overall, we found that (somewhat unsurprisingly) using more data and learning from ribosomes generally improves performance. At cutoff length 500/1000, there is noticeable drop in performance. We eventually chose to report our main results for models trained on all data (cutoff at 5000) as it lead to models with the lowest perplexity, which is akin to ability to ‘fit’ the data distribution.
• We will include these additional details from the rebuttal in the ablation study in our revised manuscript.

Question 2

• See global response on ‘Why are performance improvements for multi-state gRNAde marginal’. In summary, the performance gains are marginal b/c of our challenging split, lack of very dynamic RNA in the training set, and difficulty of the task itself. Despite this, we believe there is some signal that multi-state architectures improve performance in both single-state and multi-state settings by explicitly extracting information about RNA dynamics.
• We have also included results for more expressive multi-state pooling methods (Deep Symmetric Sets) in the new PDF attached – unfortunately, it did not improve performance but we believe this will be interesting to readers.

Weakness 1

• In the new PDF attached with the rebuttal, we have added regression plots measuring gRNAde perplexity vs. sequence recovery as well as 3D self-consistency metrics. We made plots for both the 14 RNAs from the Rosetta benchmark in Figure 2, as well as across all 100 RNAs from the test set (16 designs each → 1600 points in each plot). We measured correlation coefficients and MAE/RMSE of regression in each plot.
• We found weak positive/negative correlation (depending on metric) as measured by Pearson/Spearman correlation coefficients of +- 0.4 to 0.5. Recovery is more correlated than structural self-consistency (which is also harder to measure due to limitations of the structure predictor itself).
• Besides correlation, the visualizations also suggest that perplexity < 1.2 is indicative of good designs in terms of both recovery and structural self-consistency. There are clearly far more good designs at low perplexities below 1.2 compared to higher values.
• Thanks for highlighting this – we think it will make for an important addition to the revised manuscript.

Weakness 2

• We acknowledge your point that we could have used stronger baselines than random, but our goal was to show a new capability of gRNAde that (to our knowledge) has not been explored at all for RNA previously in the literature.

On limitation point 2

• We have noted that paucity of RNA structural data is a major limitation and challenge for modellers in our Introduction.
• Obviously, more data is better, but the inverse folding task is inherently local so we should be thinking about quantity of data in terms of # of unique tokens/nucleotides (different from structure prediction, which is a more global task). In that regard, we feel 4K RNAs with an average length of 100-150 nucleotides is a sufficiently large dataset to develop at least a useful inverse folding tool for the community (eg. outperforms Rosetta and can be useful for fitness ranking, too).

On limitation point 3

• That’s a fair point – we will append the societal impact section of the NeurIPS paper checklist (point #10) to make a statement on potential dual-use: “We hope that our tools contribute to the development of RNA-based therapeutics towards improving health outcomes and biotechnology applications. However, it is worth noting that generative models for biomolecule design can be misused for the development of harmful molecules for negative use cases.”

Thank you for your actionable review. We think details in our appendix and rebuttal responses address several of your questions and concerns – please do consider revising your score if you find the responses satisfactory and let us know if there is something we can further clarify.

Weakness 1 and Question 3

• See global response on ‘Lack of architectural novelty’.
• The other reviewers have positively noted the following new technical contributions:
  • Careful data preparation and experimental setup as well as evaluation (vioU, h3v6)
  • New design capabilities: Improved inference speed and accessibility over physics-based tools for RNA design (h3v6); Zero-shot mutant fitness ranking (E3ka, QQ46)
• Please reconsider the suitability of our work for NeurIPS. The official Call for Papers clearly states: “We invite submissions presenting new and original research on topics including but not limited to the following: … Applications, Machine learning for sciences.” We believe our work is well within the scope of NeurIPS.

Weakness 2 and Question 1

• For the experiments in Figure 2, Rosetta and the others methods are not ML models but rather physics based software which do not require training. We reported the numbers stated in Das et al. 2010, Supplementary Table 1. In more detail:
  • As noted in line 236-237, we have followed the evaluation protocol established by Rhiju Das et al. in their Nature Methods paper which first pioneered 3D RNA design: (1) we have evaluated our model on the same set of 14 high quality RNA 3D structures of interest from the PDB, and (2) for the physics-based baselines, we have reported the numbers from Das et al. 2010. See Appendix E, Table 2 for full results per RNA.
  • The ViennaRNA 2D-only baseline has been evaluated by us using the ViennaRNA python package. This is a thermodynamics-based method which does not involve any training, either. We will make a note of this in our revision – thank you!
  • In summary, the only method requiring training is gRNAde (our work), and we have carefully prepared data splits to evaluate for generalization and ensure fair comparison to these classical baselines (detailed in line 201 onwards).
• All the other experiments involve baselines and models that have been trained from scratch by us using the same experimental settings and evaluation protocols as our final model.
• Please let us know any other minor/major experimental details that you felt were missing. We will promptly clarify – we want to make the experimental protocol and setup completely transparent and reproducible (along with a detailed and documented codebase).

Question 2

• Your understanding of the multi-state setting is exactly correct!
• Yes, we show that multi-state gRNAde architectures marginally improve performance for multi-state RNAs in Figure 4:
  • Figure 4(a) shows improvements in aggregated performance across 100 test set samples (all of which have multiple conformational states).
  • Figure 4(b) shows why there is improvement: at a per-nucleotide level, multi-state gRNAde has better performance for nucleotides that are locally more flexible, undergo changes in base pairing within multiple states, and are on the surface.
• We would also like to share another interesting finding: multi-state gRNAde also improves performance on the single-state test set. See Appendix D, ‘Ablation Study’, line 600 onwards (in the table, see the columns for sequence recovery as well as 2D/3D self-consistency for the ablated variants highlighted in red). Thus, even for test set RNAs that have only one state available, training gRNAde in a multi-state manner can lead to better inverse folding performance.
• What do we understand/take away from these results? Multi-state training seems to allow gRNAde to better understand RNA dynamics and conformational changes.

Thank you for your review – please see our detailed responses below – we believe we have addressed several of your concerns and questions. Please let us know what further information we can provide to make you reconsider your vote to reject the paper.

Question 1

• See global response ‘On marginal improvements of multi-state gRNAde’. In summary, the performance gains are marginal b/c of our challenging split, lack of very dynamic RNA in the training set, and difficulty of the task itself. Despite this, we believe there is some signal that multi-state architectures improve performance in both single-state and multi-state settings by explicitly extracting information about RNA dynamics.

Question 2

• The 14 RNAs that are used to compare to Rosetta in Figure 2 are indeed a decade old, but we believe they are still a great benchmark because:
  • They are extremely high resolution crystal structures (even by today’s standards for RNA).
  • They were handpicked by Rhiju Das in their Nature Methods paper as being RNAs which are interesting for biologists to design and develop new versions of.
• In addition to the above, in Appendix D, we report a more comprehensive set of results on our full test set of 100 RNAs in total (Single-state split). This includes the 14 from Figure 2 as well as all recently released versions of the same RNAs and their structural homologues (see line 201 onwards for precise description of the splits).
• In summary, the model’s performance does not degrade when tested on more recent and greater quantities of RNA.

Weakness 1

• See global response on ‘Lack of architectural novelty’. We have also included results for more expressive multi-state pooling methods (Deep Symmetric Sets) in the new PDF attached – unfortunately, it did not improve performance but we believe this will be interesting to readers.

Weakness 2

• Some comments on your suggestions:
  • Geometric deep learning of RNA structure – this is a structure ranking model that takes as input an RNA structure + RNA sequence, and outputs a score for how closely that structure resembles the true structure (it actually predicts RMSD) – such structure prediction and ranking models cannot be used for inverse design.
  • Physics-aware Graph Neural Network for Accurate RNA 3D Structure Prediction – similarly, this is also a model for structure prediction/ranking and it is inapplicable to the inverse folding problem. We will cite it in our revision.
  • RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design – please see global response on ‘Comparison to RDesign’ for why apples-to-apples comparison to their work is not possible.
• We have done a literature review in Appendix A where we have discussed deep learning for RNA structure modeling, why current tools cannot be used for inverse design, and how gRNAde is contextualized within the broader literature.
• We believe Rosetta is a highly relevant baseline as it is the state-of-the-art in physics-based modeling of RNA. Yes, we realize that the Rosetta results are a decade old, but the reason for this is that 3D RNA design has not received as much attention from the Rosetta community as 3D protein design.

Weakness 3

• Unfortunately, it is not possible to use RNA design recipes in the latest Rosetta builds (we did try).
• A major limitation of Rosetta recipes is that many of them do not use GPUs (and this is a major advantage of new deep learning based alternatives). Rosetta RNA design recipes do not use GPUs, which is why they are so slow. Most of the Rosetta recipes are also just inherently slow b/c they use MCMC sampling and need to iterate until convergence, whereas deep learning models are one-shot predictors/generators.
• Tmol is an ongoing effort by Institute of Protein Design to port Rosetta functionality to GPUs in a differentiable manner. However, as you can see from the github this is a very early effort which is still in development without any documentation.
• This is all the more reason for releasing gRNAde in an open source manner and easy to access via notebooks and tutorials. We hope that making these datasets and tools more broadly accessible will invite renewed attention to 3D RNA design.

Another issue that makes direct comparison with RDesign impossible is that their model does not implement sampling during inference.

• Their model class contains a method called sample(): https://github.com/A4Bio/RDesign/blob/master/model/rdesign_model.py#L76 -- however, this method does not actually implement any sampling from the probability distribution outputted by the model. It just directly outputs the probability distribution.

• This would explain why RDesign's perplexity numbers are non-sensical (Appendix E.3 of their paper, as well as just the training logs for their model).

• Not being able to perform sampling means that the model will be useless in real design scenarios (especially as it is non-autoregressive/one-shot independent decoding per token), because it will keep outputting the same probability distribution and inherently cannot be used to generate diverse sequences as a result.

Is the reviewer somewhat convinced now that:

1. Our work brings something new to the community and is a valuable contribution?
2. Direct comparison to RDesign is not possible given the limitations of both their experimental methodology as well as their reproducibility?

We also wanted to add an additional note: After your comment, we are now trying to run RDesign to see if we can load the datasets and checkpoint, as well as run inference with it.

1. There are no installation instructions available in the repository: https://github.com/A4Bio/RDesign/

2. The dataset files released by them are corrupted (we tried loading them on a Macbook, a linux server, and on Github Codespaces). You can even try it quickly on your end on your browser:

   • Open a GitHub Codespaces on their repository.
   • Download the dataset files they have released: https://github.com/A4Bio/RDesign/releases/tag/data
   • Upload any of the files to the Codespaces.
   • Open a terminal or create a new jupyter notebook in the Codespaces and type the following:

import torch
torch.load("<path-to-data>/<data_file>.pt")  # or torch.load("<path-to-data>/<data_file>.pt", map_location='cpu')

...which will always lead to the following error:

RuntimeError                              Traceback (most recent call last)
Cell In[3], line 1
----> 1 torch.load("val_data.pt", map_location='cpu')

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1040, in load(f, map_location, pickle_module, weights_only, mmap, **pickle_load_args)
   1038     except RuntimeError as e:
   1039         raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None
-> 1040 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1264, in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
   1262 magic_number = pickle_module.load(f, **pickle_load_args)
   1263 if magic_number != MAGIC_NUMBER:
-> 1264     raise RuntimeError("Invalid magic number; corrupt file?")
   1265 protocol_version = pickle_module.load(f, **pickle_load_args)
   1266 if protocol_version != PROTOCOL_VERSION:

RuntimeError: Invalid magic number; corrupt file?

This happens for every data file they have provided.

We will keep you posted if we manage to run the code, but these two points should already tell you why direct apples-to-apples comparisons are so difficult in this situation. Without being able to load the processed dataset files, and without any documentation in the README or within the code as to what is the expected format of input data to their model, it is not possible to re-train or do inference with this model.

Thank you for your actionable review. We hope that our rebuttal answers your questions sufficiently and makes you reconsider some points that you noted as weaknesses. Please do consider revising your score if you find the responses satisfactory and let us know if there is something we can further clarify.

We would also like to incorporate your suggestion on arbitrary decoding order into our revision -- thank you for it!

Question 1

• We can certainly do that and we imagine this will improve models’ performance at designing interfaces. We are developing the next version of gRNAde which goes beyond just scalar features and instead considers nodes from the interaction partners together with the RNAs nodes. We hope to use this model for binding-related tasks such as aptamer design. However, this is still work in progress.

Question 2

• It is very simple to do arbitrary decoding orders for autoregressive decoding b/c we can simply permute the inputs during training and inference. And we agree that arbitrary decoding order is extremely useful in real design scenarios where we are usually given partial sequences.
• In the new PDF attached with the rebuttal, we have added results for gRNAde trained with random decoding order (shaded in green) and seen that this leads to only minor decreases in sequence recovery and 3D self-consistency. Perplexity gets significantly worse as it is more challenging for the model to fit the training data. The same observations were made in the ProteinMPNN paper.
• Thanks for the suggestion – we will include these results in our updated manuscript.

Weakness 1

• See global response ‘On architectural novelty’. We have also included results for more expressive multi-state pooling methods (Deep Symmetric Sets) in the new PDF attached – unfortunately, it did not improve performance but we believe this will also be interesting to readers and want to include it in the ablation study.

Weakness 2

• Obviously, more data is better, but the inverse folding task is inherently local so we should be thinking about quantity of data in terms of # of unique tokens/nucleotides (different from structure prediction, which is a more global task). In that regard, we feel 4K RNAs with an average length of 100-150 nucleotides is a sufficiently large dataset to develop at least a useful inverse folding tool for the community. As evidence of this:
  • gRNAde performs better than Rosetta, the best physics based tool for 3D RNA design.
  • gRNAde was found useful for ranking Ribozyme fitness in our retrospective study.
  • Based on the Ribozyme retrospective study, we felt confident enough to send gRNAde’s designed sequences for the same Ribozyme for experimental validation to our collaborators.
• AlphaFold 3 only came out 1 week prior to the NeurIPS deadline – we will certainly consider using their ideas in our future work but please don’t hold it against us for not using RNAs without 3D structure!

Weakness 3

• We have already done what you have asked. We elaborate below:
• Figure 2 does use only 14 RNAs to compare to Rosetta, but we believe they are still a great benchmark because:
  • They are extremely high resolution crystal structures (even by today’s standards for RNA).
  • They were handpicked by Rhiju Das in their Nature Methods paper as being RNAs which are interesting for biologists to design and develop new versions of.
• In addition to the above, in Appendix D, we report a more comprehensive set of results on our full test set of 100 RNAs in total (Single-state split). This includes the 14 from Figure 2 as well as all recently released versions of the same RNAs and their structural homologues (see line 201 onwards for precise description of the splits).

Weakness 4

• See global response ‘On marginal improvements for multi-state gRNAde’. In summary, the performance gains are marginal b/c of our challenging split, lack of very dynamic RNA in the training set, and difficulty of the task itself. Despite this, we believe there is some signal that multi-state architectures improve performance in both single-state and multi-state settings by explicitly extracting information about RNA dynamics.

On the stated limitation

• We agree that inverse folding generative models are not suitable for representation learning/predictive tasks. It has also been seen for proteins that models for inverse folding are not useful for representation learning (eg. this paper by Kevin Yang) -- our goal is inverse design and not representation learning.

Thank you for acknowledging the rebuttal. We hope you will reconsider, given that we put significant effort to address your review and we think it did improve the paper overall.

Adapting existing architectures to new problems, doing the experiments rigorously, setting the first benchmarks for a field, and releasing open-source resources so others can build upon them is a valuable contribution. The NeurIPS Call for Papers states: “We invite submissions presenting new and original research on topics including but not limited to the following: … Applications, Machine learning for sciences.” Doing applied work well requires strong understanding of the application domain as well as the deep learning architectures and evaluation, but may not always involve new architecture ideas.

We addressed as many of your weaknesses and questions as we could via the rebuttal.

• Weakness 1: We tried more expressive set pooling methods, but it often happens in structural biology applications that complex architectural ideas do not generalize to o.o.d. test sets. It would obviously have been nice for the novelty of our paper if we could have proposed an exciting, new multi-state fusion method. However, we benchmarked many ideas rigorously and did not find them to bring real improvements.
• Weakness 2: We basically processed and prepared ML-ready datasets from all the structured RNAs available publicly on the internet; we couldn't use the AlphaFold3 self-distillation idea b/c it was not available when we did the work. We hope this is not counted as a weakness in your assessment.
• Weakness 3: We actually did use a lot more test samples and reported results with confidence intervals in the appendix. We hope this is also not counted as a weakness in your assessment.
• We can use our models with arbitrary decoding order, too. We did this experiment that you asked for, too.

Thank you for your encouraging review! We think that incorporating your comments will strengthen our revised paper and we hope our responses address your questions in the best way possible.

Question 1:

• To clarify, ‘perfect’ in this sentence is from a machine learning context, not from an applied context. So given an RNA backbone as input, a perfect ML model will output its groundtruth sequence from the PDB, which it would have memorized (leading to perplexity = 1).
• However, such a perfect ML model is completely useless from an applied perspective, because we never want to perfectly recover the groundtruth sequence from the PDB. We want to sample a diverse set of designs which are reasonably close to the PDB sequence while remaining interesting/non-trivial.
• Perplexity = 4 would mean that the model is randomly selecting from the 4 bases for each position in the sequence, so your understanding is correct.

Question 2:

• We re-visited basic chemistry definitions and agree that ‘conformer’ is a term reserved for conformational states at which a molecule's energy is at a local minimum. We will revise our paper to not use this term – thank you!
• The multiple conformations that we use are from multiple deposited structures for a given PDB entry, or multiple PDB entries for the same RNA, or both. The answer to whether they are at local energy minima really depends on who one asks, in our opinion. Our current understanding is that it remains an open question whether crystal structures of biomolecules deposited in the PDB are truly energy minimas or just some frozen states/artifact of crystallization.
• See Appendix Figure 15 for some statistics on # of multi-state RNAs, # of states per RNA, whether # of states is correlated with length (no), etc.

Weakness 1 and Limitation 1:

• Please see the global response on ‘Comparison to RDesign’ for why apples-to-apples comparison is not possible. We have also done extensive ablation studies of our architecture as a way to understand the impact of different components on performance.
• 2D-only design techniques are meant for different types of RNA design problems than what Rosetta/gRNAde are intended for. They can not incorporate 3D information about kinks and turns and motifs explicitly (b/c they optimize for only maintaining the same base pairing in designed sequences; they are also not evaluated for ‘recovery’ but rather for a ‘success rate’ of retaining the same exact base pairing).
  • You can see this by comparing gRNAde variants to ViennaRNA in Appendix Table 1: ViennaRNA actually has very good 2D self-consistency score but poor 3D self-consistency as it is simply not designed to account for 3D structure. All other 2D inverse folding methods we are aware of also cannot account for 3D interactions, so we think it is reasonable to expect similar performance.

Weakness 2:

• AlphaFold3’s paper was published one week before the NeurIPS deadline and its code or weights are not yet available. Our framework is flexible to swap out RhoFold for AF3 as soon as possible (we are also eager to move on as there are limitations to RhoFold's performance; see the 'Groundtruth sequence prediction baseline' in our ablation study for an upper limit on our test set).
• RF2NA is not a suitable choice as it is primarily for protein-NA complexes, not solo RNAs. We also think (intuitively) that RhoFold’s use of a language model makes it less reliant on MSAs than RF2NA (we almost never have MSAs for designed/synthetic RNAs).
• RhoFold was used in AIchemy_RNA which was used as a baseline in AlphaFold3 as ‘the top performing machine learning system’ on CASP15 targets. See Extended Data Fig. 5 in AF3 paper, where AF3 outperforms RhoFold but not outright (RhoFold is better on some targets). So we think RhoFold is a reasonable choice at present.
• We chose EternaFold b/c it has actually been evaluated on designed/synthetic RNAs in the wet lab before and is broadly accessible for benchmarking. We are aware that it is not the best 2D structure predictor for naturally occurring RNAs, but believe that it does a good job at recovering at least the correct base pairings (which is the goal of 2D self-consistency evaluation).
• Nobody has conducted a self-consistency score based benchmark for 3D RNA design before us, so we are unable to report self-consistency numbers for Rosetta/other baselines (it is not possible to use the RNA backbone design recipe in the latest Rosetta builds).

Weakness 3:

• Our choice of a TM-score cut-off of 0.45 to determine whether RNAs are in the same global fold follows US-align and other similar works on RNA structure modeling (eg. see recent CASP evaluation for RNA structure prediction where the 0.45 cutoff was used). Whereas it is conventional in the proteins literature to cluster protein structures according to a 0.5 TM-score cut-off, given the wider and more flexible nature of RNA compared to protein structures, a slightly lower structural similarity cut-off for RNA is warranted.

Weakness 4 / lines 258 and 259:

• For riboswitches, the aptamer domain that is interacting with the ligand partner is usually conserved, and we currently want to use gRNAde to re-design scaffolds around this binding domain that may be more thermally stable while retaining the switching mechanism. We think gRNAde can be somewhat useful for this already.
• But we agree that ligand conditioning during sequence design will provide greater context for multi-state design, and we are actively developing a version of gRNAde which incorporates ligand partners.

Thank you for responding.

Re. RDesign

• As you noted, 2 other reviewers were concerned with the comparison to RDesign. Short of all options, we have been trying to use RDesign but we simply cannot reproduce its data format or run inference with their model, at present (eg. no installation instructions, dataset files seem corrupted, model does not have an actual sampling mode). We hope you would agree that the community needs good open source training code, datasets, and reproducibility in order to further develop this new research direction (RNA design with deep learning).
• Also, please see our ablation study for apples-to-apples comparisons to RDesign's architectural ideas vs. gRNAde's (majorly, non-autoregressive/one-shot vs autoregressive), benchmarked fairly on our split and experimental settings. We think this brings insights to readers, b/c the question of autoregressive vs. one-shot models is also relevant for other biomolecule design tasks.

Re. publication date:

• The decision notifications for ICLR were released in January privately to authors. We would imagine that the actual publication date is when a camera-ready version of a paper is released to the public audience beyond Authors, Reviewers, ACs, PCs (which was April).

Re. consistency of self-consistency metrics: That's a great point and we do agree that these metrics are not perfect. These metrics (as with most metrics for generative models) are supposed to be proxies for actually evaluating these designs in real world experiments. So maybe they are not perfect, but the way we actually use them for RNA design is as filters where really poor self-consistency means that the design is very likely a poor one (but good self-consistency does not automatically mean its a good design). We have actually discussed this and caveated our results at several instances in the paper.

Re. AlphaFold3's server:

• We are restricted to 20 jobs per day, and it is not possible to submit batched jobs.
• Given these restrictions, and the fact that AF3 server came out after the NeurIPS submission deadline, would you agree that actually using AF3 for our evaluations is impossible?
  • Here is a rough scale of how many times a structure predictor is called during our evaluation protocol:
    • For example, take 100 test RNA backbones.
    • Design 16 sequences for each backbone.
    • Fold the 16 x100 designed sequences = 1600 calls per model evaluation (per one entry in the table in Appendix D)
  • Assuming there are on average 5 authors in our team, evaluating one variant of our model would take 16 days via AF3 server. We have run experiments on ~25 models (not including further results we will be adding after rebuttals to other reviewers).
  • Also note that there is no programmatic way to run AF3 server -- we would have been manually doing data entry.

Re. 2D structure predictors: Okay, we will now be rushing to try one of these out and will report back the results honestly. Please stay tuned.