Beyond Sequence: Impact of Geometric Context for RNA Property Prediction

Thank you very much for the authors' replies. I still have some questions and hope to get their explanations.

D1. Need to add original improvements to the model.

The facts stated in the authors' response are indeed the core issues in this field. High-quality RNA datasets are very rare, so the benchmark compilation contribution of this article is indeed very important.

However, I and several other reviewers are very concerned about the fact that ICLR, as a conference in the field of machine learning, may not contribute enough to just propose benchmarks and test existing baselines (although this article belongs to the field of "datasets and benchmarks"). Although this article compares a large number of baselines, it lacks the authors' own innovation in model architecture. Generally, AI for Science benchmarks are more or less designed for special scenarios (e.g. GVP-GNN and GNS). Can the authors take advantage of the particularity of RNA to add some more. For example, introduce some small sample learning techniques (such as active learning, etc.) based on the lack of data? Or can they improve FastEGNN based on the sequence structure of RNA?

D2. About FastEGNN settings.

Can the authors list the specific parameters of FastEGNN? For example, how many virtual nodes are used? Is there any additional edge deletion?

As far as I know, it seems that the virtual nodes of FastEGNN are introduced without prior knowledge, and the point set distribution of the experimental data set in the original paper does not show obvious serialization characteristics like RNA. Can the poor performance of FastEGNN on RNA data be considered to be due to the contribution of virtual nodes to irrelevant real nodes? Can the authors further analyze such phenomena?

In response to this problem, can the authors modify the initialization of virtual nodes (for example, set one for each RNA substructure) and the way virtual nodes are connected to real nodes (delete edges with a long distance or introduce radial basis functions) as their customized model on the benchmark? I think that designing a methodology that uses RNA prior knowledge to modify the general model into a customized model can be a good contribution to match the benchmark to make the paper more acceptable.

D3. More baselines.

The method of using high-degree steerable features is also very important for 3D geometric graph networks. As a benchmark, I think it would be more beneficial to the quality of the article to choose one from TFN, NequIP, MACE, EquiformerV2 for testing. I recommend using TFN and using the code in https://github.com/chaitjo/geometric-gnn-dojo/blob/main/models/tfn.py, which is easier to modify. In addition, it may not be particularly easy to migrate special models such as MEAN and dyMEAN. I can also understand that the authors do not have enough time to design them specifically, so I will not ask for them.

We thank the reviewer for their feedback and appreciating the value and importance of our newly contributed RNA datasets for advancing the field of RNA property prediction. We address the questions and comments raised by the reviewer point-by-point:

RW1: Lack of explanation for RNA's uniqueness:
We thank the reviewer for raising this point. RNA differs fundamentally from proteins in both the availability of data and the nature of practical challenges in modeling. Unlike proteins, which benefit from extensive high-quality databases such as PDB [1] containing sequences, experimental structures, and experimentally measured property labels in benchmarks such as FLIP [2], RNA datasets are far more limited [4,5]. Existing RNA resources, like RNAsolo [3], provide 3D structures but lack property annotations, leaving a significant gap in benchmark datasets for RNA property prediction. Moreover, RNA is particularly susceptible to sequencing noise due to variability in platforms and quality [6, 7], a challenge less prominent in protein studies where experimental characterization techniques are more advanced. These distinctions mean that while the role of structural information is well-understood for proteins, it remains under-explored for RNA. Thus, the evaluation setups for protein and RNA cannot be compared directly. With regards to DNA, similar limitations hold as for RNA and hence we are not aware of high-quality structural datasets with property labels for DNA either and recent literature only chooses to model DNA sequences alone.

References:
[1] Protein Data Bank: the single global archive for 3D macromolecular structure data, Nucleic acids research, 2019. [2] FLIP: Benchmark tasks in fitness landscape inference for proteins, Dallago et al., NeurIPS 2021
[3] RNAsolo: a repository of cleaned PDB-derived RNA 3D structures, Adamczyk et al., Bioinformatics 2022
[4] Translating rna sequencing into clinical diagnostics: opportunities and challenges, Byron et al., Nature Reviews Genetics, 2016
[5] Reducing costs for dna and rna sequencing by sample pooling using a metagenomic approach, Teufel & Sobetzko, BMC genomics, 2022.
[6] Rna sequencing: advances, challenges and opportunities, Ozsolak & Milos, Nature reviews genetics, 2011.
[7] Accuracy of next generation sequencing platforms, Fox et al., Next generation, sequencing & applications, 2014.

RW2: Evaluation with more recent 3D methods:.

Following the reviewer's suggestion, we have now added a comparison of four additional 3D models (GVP, DimeNet, and recent FAENet and FastEGNN) in Table 1 and Figs. 2, 3, 5, 6 in the main text (highlighted in blue). However, we still find that all these models show similar performance on RNA property prediction tasks with 3D models failing to outperform 2D models. We further chose FastEGNN for all subsequent experiments and observe the similar trend as reported earlier with EGNN and SchNet.

Please let us know if there are any further questions. Thanks a lot!

I have carefully read the author's feedback and think that the author's explanation makes sense. I have combined the opinions of both parties and put forward the following suggestions for improvement. If the author can do this, I will consider recommending that this article be accepted.

D4. About models using high-degree steerable features

I still think it is necessary to use these models based on high-degree steerable features as baselines. After all, they are also an important part of equivariant models. Authors can avoid OOM problems by adjusting batch_size or reducing channels. Alternatively, authors can consider using the recently emerged SO3krates [a] or HEGNN [b] to introduce high-degree steerable features while avoiding the complexity of tensor products. I understand that the remaining time in the current discussion stage may not be enough to complete the experiment, so in the discussion stage, I only require the authors briefly discuss the possible benefits of high-degree steerable features in the article. Presumably, there is still enough time from acceptance to submission of camera ready for the authors to supplement the experimental results (and also the results of models like SaVeNet, LEFTNet and MEAN).

[a] Frank J T, Unke O T, Müller K R, et al. A Euclidean transformer for fast and stable machine learned force fields[J]. Nature Communications, 2024, 15(1): 6539.

[b] Cen J, Li A, Lin N, et al. Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?[J]. arXiv preprint arXiv:2410.11443, 2024.

D5. Discussion on the inapplicability of FastEGNN

I think the authors' response makes a lot of sense, and I hope to include this discussion in the manuscript. I think FastEGNN is an excellent model, although its CoM virtual node initialization may not be suitable for this problem that clearly has a sequence prior. It would be great if the authors could combine this to explain why FastEGNN performs poorly and give suggestions for improvement. I believe this will not only improve the value of this article, but also be a respect for the original author of the FastEGNN model.

D6. Discussion on considering noise in model design

Since most studies in this field do not explicitly address noise when designing model architectures, could the authors discuss how to effectively account for noise during model design and what methods might be employed to minimize its impact?

Dear authors, thank you for your responses. Apart from the misplaced references of SO3krates and HEGNN in Line. 1088, I think your summary is quite accurate. I raise my rating to "Weakly Accepted".

We thank the reviewer for raising their score and championing for our work highlighting our contributions as novel and meriting publication owing to the utility of the real-world datasets and experiments for the field!

I carefully compared this article with other benchmark articles according to the requirements of the benchmark and reviewed it again. In fact, the amount of work in this article is sufficient, and as far as I know, it is the first to study the properties of RNA in a real-world scenario. In particular, large-scale long-chain RNA datasets are rare, and the author's integrated representation modalities and combed research directions are valuable. By reading 'Author Responses to Reviewer JSSw', I was further convinced of the value of this article, so I upgraded the rating to 'Clear Accepted'.

We thank the reviewer for their feedback and appreciating the thoroughness and comprehensiveness of our experiments and for highlighting the value of our newly annotated RNA data. We address the questions and comments raised by the reviewer point-by-point:

RW1: Lacking methodological innovation:
We appreciate the reviewer's observation about the simplicity of the proposed framework. We would like to emphasize the our paper sits under “datasets and benchmarks”, a subject area highlighted in ICLR call for papers. Our main contributions are: 1) introducing first of its kind RNA datasets with all 1D, 2D and 3D structures and property labels; 2) providing a modular unified testing environment for benchmarking 1D, 2D and 3D property prediction models; 3) studying existing 1D, 2D and 3D models to assess the impact of geometric information in a range of real-world scenarios and establish baselines for future research in this direction. No such study exists to date despite the importance of RNA as therapeutic modality.

RW2: The article merely compares various metrics, and the key points are not sufficiently emphasized.
In this work, we provide a comprehensive comparison beyond just RNA property prediction accuracy, which includes a range of real-world challenges of modeling RNA: noise robustness, OOD generalization, data and label efficiency. We derived several key insights including: 2D spectral methods outperforming 1D and 3D methods in low-to-moderate noise regimes (highlighted in bold in line 259 and line 373); 3D models outperforming 1D models under limited data regime even despite the structural noise (highlighted in bold in line 383); 1D sequence methods require 2x-5x more training data to match the performance of 2D and 3D methods (line 379), but excel in high noise regime (highlighted in bold in line 462); simple Transformer1D2D with soft structural prior outperforms the rest of the models even in high noise regimes while still utilizing structural information (lines 472-476). We consider these insights significant since they cover a range of real-world scenarios that have not been investigated in prior work. We would appreciate the suggestions from the reviewer on which key points require further elaboration and more emphasis.

RW3: Impact of uncertainties in secondary and tertiary structures for property predictions:
We wish to clarify that this is intentional and corresponds to the real-world setting reflecting the challenges practitioners face when modeling RNA properties. We are not aware of any dataset containing triplets of 1D RNA sequences, experimentally determined 2D and 3D structures, and corresponding experimentally measured property labels. This is due to the high cost and technical difficulty of experimentally determining RNA structures and measuring their properties. Consequently, practitioners must rely on predicted structures or fall back to 1D sequence models. Thus, the structural uncertainties are inherent and inevitable in practice and including them into analysis better reflects practical reality.

RQ1: Results with RNA language models:
Good suggestion! We have now benchmarked 2 SOTA RNA language models: RNA-FM [1] and SpliceBERT [2] and added the results in Table 1 in the main text (highlighted in blue). We find that for 2 out of 4 datasets (Covid and Ribonanza-2k), RNA foundation models perform worse than the supervised transformer baseline wheres for the other two datasets (Tc-Riboswitches and Fungal), they achieve similar performance (results reported n MCRMSE, lower is better). This is consistent with recent papers in multiple biology domains demonstrating generalized foundation models are yet to surpass specialized supervised baselines (see [3,4,5]). Thus, all our presented conclusions hold.

RQ2: Criteria for selecting secondary structure prediction tools:. We choose EternaFold as the tool for secondary structure prediction because of the reported SOTA performance of EternaFold in recent works ([6,7,8]) and has been explained in lines 143-146 of the main text.

RQ3: Explain the motivation behind the five task settings:
We appreciate the reviewer's feedback and recognize the importance of clearly explaining the motivation for the selected tasks. These tasks were specifically chosen to simulate real-world challenges encountered in RNA property prediction, focusing on diverse practically relevant aspects of model evaluation. The tasks aim to address critical questions such as:
Task 1: how effectively models leverage structural information of RNA which so far has not been explored in literature in the context of RNA property prediction;
Tasks 2 and 3: how well they perform with limited training data or partial labels, a common constraint in experimental settings due to the costs of obtaining large experimental databases;
Tasks 4 and 5: the robustness of the models under sequencing noise, which mirrors variations and errors produced by RNA sequencing platforms and methods.
In Sec 3 (pages 4 and 5), we have cited relevant works which highlight exactly these scenarios for RNA data. We have now also added detailed descriptions and motivations for these tasks in the Appendix D (pages 19-20 and highlighted in blue) and referred to it in main text (lines 248-249).

Please let us know if there are any further questions. Thanks a lot!

References:
[1] Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions, Chen et al., arXiv 2022. [2] Self-supervised learning on millions of primary RNA sequences from 72 vertebrates improves sequence-based RNA splicing prediction, Chen et al., Briefings in Bioinformatics, 2023. [3] Specialized Foundation Models Struggle to Beat Supervised Baselines, Xu, Gupta et al., FM4Science@NeurIPS 2024
[4] Convolutions are competitive with transformers for protein sequence pretraining, Yang et al., cell Systems, 2024
[5] Assessing the limits of zero-shot foundation models in single-cell biology, Kedzierska et al., bioRxiv, 2023
[6] RNA secondary structure packages evaluated and improved by high-throughput experiments, Wayment-Steele et al., Nature Methods, 2022. [7] Deep learning models for predicting RNA degradation via dual crowdsourcing, Wayment-Steele et al., Nature Machine Intelligence, 2022
[8] Ribonanza: deep learning of RNA structure through dual crowdsourcing, He et al., bioRxiv, 2024

Dear Reviewer,

Thank you for your valuable feedback. We have carefully responded to all your concerns and would be grateful if you could consider raising your score.

As the rebuttal deadline approaches, we would greatly appreciate your feedback at your earliest convenience. Thank you again for your time and thoughtful suggestions!

Best regards,
The Authors

We thank the reviewer for their feedback and appreciating the originality, thoroughness, and comprehensiveness of our experiments that highlight the value of our newly annotated RNA datasets. We also appreciate the reviewer's observation that our work presents a first study of its kind for RNA property prediction in range of real-world scenarios.

Now we address each weakness and question raised by the reviewer.

RW1. Experiments with more 3D models:
Following the reviewer's suggestion, we have now added a comparison of four additional 3D models (GVP, DimeNet, and recent FAENet and FastEGNN) in Table 1 and Figs. 2, 3, 5, 6 in the main text (highlighted in blue). However, we still find that all these models show similar performance on RNA property prediction tasks with 3D models failing to outperform 2D models. We further choose FastEGNN for all subsequent experiments and observe the similar trend as reported earlier with EGNN and SchNet.

Regarding 3D models such as ARES and PaxNet, we note that they are meant for RNA 3D structure prediction and ranking and do not support RNA property prediction out of the box. Similarly, methods such as MEAN are highly specialized for modeling antibody domains by design and adapting them for RNA property prediction will be a non-trivial contribution in itself.

RW2 & RW3: Pooling strategy and dealing with long-range dependencies:
We appreciate the reviewer's observation about the simplicity of the proposed framework. We would like to emphasize the our paper sits under “datasets and benchmarks”, a subject area highlighted in ICLR call for papers. Our main contributions are: 1) introducing first of its kind RNA datasets with all 1D, 2D and 3D structures and property labels; 2) providing a modular unified testing environment for benchmarking 1D, 2D and 3D property prediction models; 3) studying existing 1D, 2D and 3D models to assess the impact of geometric information in a range of real-world scenarios and establish baselines for future research in this direction. No such study exists to date despite the importance of RNA as therapeutic modality.

With this, substantially modifying existing or developing novel 3D methods is outside the scope of this work. At the same time, we agree with the reviewer that better utilization of 3D information is important, and our work will serve to foster future research in this direction.

RW4: Conclusions from data efficiency experiment in Section 4.2:
The experiments in section 4.2, investigating models' data efficiency, not only reveal that the performance improves with increasing training data, but also uncovers novel insights of 2D spectral models excelling in low data and partial label regimes, and 3D models outperforming 1D models under limited data regime despite the structural noise. To the best of our knowledge, these insights are novel. We consider these insights significant in the real-world scenarios where large fully-labeled datasets are unavailable.

RW4: lower prediction accuracy of existing methods for 3D structures:
To the best of our knowledge, no datasets exist containing triplets of 1D RNA sequences, experimentally determined 2D/3D structures, and corresponding property labels, as discussed in Appendix A. This is due to the high cost and technical difficulty of experimentally determining RNA structures and measuring their properties. Consequently, practitioners must rely on predicted structures or fall back to 1D sequence models. Thus, the structural inaccuracies are often inherent and inevitable in practice. At the same time, as the reviewer rightly noted, 3D data contains more information than 1D and 2D data. However, before our work, it was unclear whether this extra, potentially noisy information would add value for property prediction compared to 1D or 2D data. No other prior work has investigated this trade-off between richer information content and potentially more uncertain 3D structures.

RW5: Conclusions from sequencing noise experiment in Section 4.3:. We want to highlight that the goal of experiments in Section 4.3 was to assess the models under realistic sequencing noise. To this end, we sampled the noise profiles practically observed in real-world sequencing platforms (line 430-434). Since no prior work has studied the impact of realistic sequencing noise on quality of predicted structures, it was not clear how different classes of property prediction models perform in this realistic scenario and to what degree their performance deteriorates under realistic noise. Our experiment not only reveals that Transformer1D is more robust under high sequencing noise, but also that 2D models still perform the best under low-to-moderate noise regimes (now highligted in blue in lines 479-480). Additionally, we reveal that simple Transformer1D2D with soft structural prior outperforms the rest of the models even in high noise regimes while still utilizing structural information (lines 473-476). We consider these insights significant since they cover real-world deployment scenarios and no prior work has investigated the impact of realistic sequencing noise on 1D, 2D and 3D property prediction methods.

RQ1. Have the authors considered finding a 3D RNA dataset, such as the RNAsolo?
We thank the reviewer for the suggestion. However, note that RNAsolo dataset does not include any property labels associated with these structures and hence cannot be used for our purpose of studying the impact of sequence and structural information for property prediction tasks. Currently, we are not aware of any dataset containing triplets of 1D RNA sequences, experimentally determined 2D and 3D structures, and corresponding experimentally measured property labels per data point. This also highlights the importance of our work in a real-world setting where both experimentally determined geometric structures and property labels are unavailable.

RQ2. What specific properties are being referred to, and what are the units for the predicted values?
We have described the properties in Subsection 2.1 of the main text. To summarize, the properties we model are Riboswitch switching behavior for the Tc-Riboswitches dataset, nucleotide degradation for the COVID-19 dataset, reactivity for the Ribonanza-2k dataset, and expression for the Fungal dataset.

For Tc-Riboswitches, the labels are percentage reflecting switching behaviors. For COVID-19, Ribonanza-2k datasets, the labels degradation and reactivity labels are normalized intensity (hence without units) and for Fungal dataset, the expression labels are measured in transcripts per million (TPM) per kilobase million (RPKM).

Please let us know if there are any further questions. Thanks a lot!

Dear Reviewer JSSw,

We noticed that your review mentions, “the overall quality is marginally acceptable,” yet the final score assigned is “reject.” We kindly ask if you could check the score to ensure alignment with your comments, and consider our rebuttal at the same time.

Thank you for your time and consideration.

Best regards,
The Authors

Dear Reviewer,

Thank you for your valuable feedback. We have carefully responded to all your concerns. Could you kindly review your rating and consider our rebuttal? We would be grateful if you could consider raising your score.

As the rebuttal deadline approaches, we would greatly appreciate your feedback at your earliest convenience. Thank you again for your time and thoughtful suggestions!

Best regards,
The Authors

We thank the reviewer for raising their score based on our response. If the reviewer can point us to which concerns are still remaining so that we can address them from our side to help raise their score even further. Thank you!

In this paper, we collect RNA sequences from publicly available datasets and generate the 2D and 3D structures from publicly available tools. We cite all the sources in our paper, please refer to Section 2.1 and 2.2 in our paper. Therefore, we don't see any potential ethical issues of the data used in this paper.

We have also mentioned this in the "Ethics Statement" part in our paper. If all your concerns are addressed, could you please consider raising your score as only few hours of discussion period remain?