KinPFN: Bayesian Approximation of RNA Folding Kinetics using Prior-Data Fitted Networks

We thank the reviewer for the helpful feedback. If there are any further questions or any clarification needed, we are happy to answer those. If not, we would appreciate it if the reviewer could increase our score.

With kind regards,

The authors

Dear Reviewer FZT1,

Thank you for your positive feedback and highlighting the soundness of our approach.

Why not train on real passage time data that you can simulate? Why synthetic?

Generating a sufficiently large dataset for training a deep learning approach is currently infeasible due to the exponential increase in computation time required by Kinfold (and other kinetic simulators) as the RNA sequence length grows linearly. This is caused by an exponential growth of the conformational space S_n with the sequence length n, roughly following S_n \approx 1.86^n. Additionally, a dataset consisting of simulation data from a single simulator might be strongly biased, and the trained deep learning approach would potentially struggle to generalize across different simulators. However, our initial approach involved training a PFN using real first-passage time data that we generated by running Kinfold on a large set of very short sequences. However, training a PFN on real simulation data resulted in substantially worse performance compared to KinPFN, even when evaluating on data obtained from the same simulator.

Generally, the generation of synthetic data for biological applications is challenging because we cannot make any assumptions about the underlying data generating process. We, therefore, decided to approach the problem by directly learning the distribution of first passage times in an in-context learning setup that allows us to condition on previous observations (e.g. obtained from simulator data). Besides providing improved performance, this approach makes KinPFN broadly applicable to RNA folding kinetics independent of the sequence length, the simulator used for generating the context data, the RNA type, the energy differences between the start and stop structure, and even the specific problem addressed by the approximations (as shown by our generalization to gene expression data).

We think that training on synthetic data could solve many current issues connected to the usage of deep learning methods, particularly in the life sciences. These issues e.g. include overfitting due to limited amounts of available training data,unbalanced training data, as well as fairness problems. With KinPFN, we demonstrate that even with a relatively simple synthetic prior, we could generate synthetic data that represents a biological problem rather well, leading to strong approximation quality, avoiding the risk of overfitting, while learning from unlimited amounts of data that provides full control over data parametrizations.

The prior described by the synthetic data is so simple it may be easy to just run MCMC. Why use a language model at all?

Monte Carlo methods are regularly used in RNA folding kinetics and we agree that it would be beneficial to exclusively train on MCMC-generated data. However, the runtime of MCMC depends on the chain length, and thus on the number of structural states. Therefore, running MCMC on long RNAs is computationally infeasible.

Secondly, [1] recently showed that PFNs clearly outperform MCMC in terms of runtime and accuracy on the task of learning curve extrapolation, highlighting that generating sufficient samples with MCMC to reliably approximate the posterior may impose significant overhead (see Section 4 in [1]).

We, therefore, decide to base our approach on PFNs rather than MCMC, and to train on synthetic FPT distributions instead of simulation data.

[1] Adriaensen, S., Rakotoarison, H., Müller, S., & Hutter, F. (2024). Efficient bayesian learning curve extrapolation using prior-data fitted networks. Advances in Neural Information Processing Systems, 36.

Language models as priors can have some pathologies. How does the model behave as the amount of data becomes large? In figure 11 it looks like even with a lot of data the PFN doesn't converge to the true CDF.

We agree with the reviewer that the context can have a substantial impact on the resulting approximations. We tried to account for this by reporting means and standard deviations across 20 different contexts for all the approximations shown in the manuscript. Generally, we observe that the approximation performance is better with more context. In Figure 11 mentioned by the reviewer, the largest context size is 75, which arguably is still relatively small compared to the 1000 (or typically even more) simulations required to obtain reliable CDFs from Kinfold. However, also in Figure 11, we observe a clear trend of improved performance with more context data available, which is also a general pattern shown in Table 6 where we compare KinPFN with different alternative approaches. However, we agree with the reviewer that it might be interesting to see KinPFN’s behavior for much larger context sizes. We, therefore, add a Table with the results for context sizes of up to 1000 to the Appendix H.2 of our revised manuscript.

Dear Reviewer PRsD.

Thank you for your valuable Feedback. We address your questions and concerns in the following.

I would have liked to have seen Table 6 comparing all the different methods in the main paper. Plus, I would have liked to have seen comparison on MAE across methods, including in Table 1

We thank the reviewer for this useful comment and moved all results from Table 6 to the main paper. Additionally, we add MAE and Kolmogorov-Smirnov (KS) statistic results for the respective experiments in Appendix H.2 due to space limitations in the main body.

Reliance on multimodal Gaussians

We agree with the reviewer that different distributions might further improve KinPFNs performance. We think that we adequately address this limitation in the discussion on future work in Section 6. However, as the first deep learning approach for RNA folding kinetics, our results indicate that assuming a Gaussian distribution for first passage times is reasonable and already leads to strong performance. Nevertheless, we plan to explore the potential of alternative distributional assumptions in future work.

Reliance on only two metrics to measure accuracy of the metric.

The negative log-likelihood is commonly used to assess the performance of PFNs across different tasks and we think that it captures the performance of KinPFN arguably well. However, we are happy to include additional metrics that offer new insights or increase the understanding of strengths and weaknesses of our approach. For the specific case, we added MAE and KS to the analysis of the performance of KinPFN (results shown in Appendix H.2).

Why not show the comparison between the sequence length and FPT in Table 1? I think that might provide more insight as to where KinPFN works better over other methods unless sequence length isn't an important variable, which it seems to be since it's the subject of study in Figure 3

We are currently analyzing the influence of sequence length along with other RNA features for all predictions and will add the requested analysis to the appendix, as we think that Table 1 might be overloaded otherwise. The number of potential structural states increases exponentially with the sequence length of the RNA. The reviewer, therefore, is right that simulations for longer RNAs require substantially longer runtimes of the simulators and are particularly challenging. As a pure in-context learner, KinPFN’s performance, however, is independent of the sequence length (similar to fitting KDEs or GMMs), which is one of the major advantages of our approach because it offers substantial speed-ups specifically for the case of long RNAs. The analysis shown in Figure 3 was performed to confirm this independence claim for KinPFN.

Why not put the legend in Figure 5 in the appendix? I'm not going to be able to copy/paste that to check, anyway.

We included the three sequences in the figure legend to clarify that this experiment compares three different RNAs, each folding into the same secondary structure but undergoing distinctly different folding processes and efficiencies. We, therefore, disagree with the reviewer that the sequences provided in the legend of Figure 5 are invaluable for understanding the results and would like to keep the respective figure as it is.

Why not use the KS test between CDFs as another comparison? This would help capture maximum discrepancy and add nuance to the experimental analysis.

We thank the reviewer for this helpful suggestion and we add the KS statistic results to Appendix H.2 in the revised version of our manuscript.

We hope that we addressed all your questions and would like to thank the reviewer again for the valuable feedback. We are happy to answer further questions. If all your concerns have been addressed in the response, we would like to kindly ask you to increase our score.

With kind regards,

The authors.

Compared to the dynamic changes in RNA secondary or tertiary structures, the folding ratio provides very coarse-grained information about RNA folding dynamics, which seems still far from practical applications. The paper needs to further elucidate how this study can contribute to solving RNA biology problems.

We agree with the reviewer that RNA folding kinetics and dynamics can be captured at different levels of granularity. However, we disagree with the reviewer that first passage times are far from practical applications. In our initial submission, we already show an interesting application, assessing the folding efficiency of different RNA sequences that fold into a common minimum free energy structure (Section 5.3). This kind of analysis is particularly useful for RNA drug discovery, where the rates of obtaining the functional molecular folds could be essential to rank different candidates. In addition, the first passage times of systems play a substantial role in biology, chemistry, and medicine (see e.g. [1]). While KinPFN was primarily developed to study RNA folding kinetics, our results for gene expression data suggest that KinPFN might generalize to other data sources of FPTs as well, with the potential to impact different areas of biology besides RNA folding kinetics.

In Addition, many published examples show the direct application of kinetic folding simulations to biological data, e.g. the original Kinfold paper (doi: 10.1017/s1355838200992161) or studies that investigate riboswitch folding (doi:10.1021/jacs.6b10429)

That said, we add a sentence to the Introduction of the revised version of our manuscript to clearly highlight fields of applications of KinPFN.

[1] Polizzi, N. F., Therien, M. J., & Beratan, D. N. (2016). Mean first‐passage times in biology. Israel journal of chemistry, 56(9-10), 816-824.

In the experimental section, the results from Kinfold are used for validation, but the inherent error of Kinfold needs rigorous demonstration, which diminishes the persuasiveness of the results. Is it possible to use collected or published wet lab data for the evaluation of this problem?

We agree with the reviewer that we use simulation data obtained from Kinfold as ground truth. However, we also analyze the performance of KinPFN for simulation data from another simulator (Kfold, see experiment 5.1), showing no decrease in the performance of KinPFN. Since KinPFN was not trained using simulation data but a synthetic prior, the performance of KinPFN is independent of the underlying data-generating process. That said, we would expect that KinPFN is capable of predicting first-passage time distributions for wet-lab kinetics data as well. This is also supported by our experiments for gene expression data which arguably requires more transfer capabilities than experimentally obtained first passage times compared to simulation data. However, we are not aware of any resource for obtaining RNA folding kinetics wet-lab data.

tRNA and rRNA are the most common and numerous types of RNA. It would be better to test a broader and more diverse range of RNA types.

While we agree with the reviewer that tRNA and rRNA are common and well studied RNAs, this was exactly our motivation to use these two types of well-known, structured non-coding RNAs as a reference for KinPFN’s behavior on eukaryotic RNAs. However, in response to the reviewer’s concerns, we are currently running simulations for the following three RNAs:

• https://rnacentral.org/rna/URS00002F3927/224308
• https://rnacentral.org/rna/URS0000BA5588/9606
• https://rnacentral.org/rna/URS0000759FB2/9606

We hope that we can obtain the required number of simulations within the next few days.

Would an additional evaluation on these samples resolve the reviewers concerns regarding evaluations for different RNA types?

There is a lack of research and discussion on deep learning methods suitable for the data in this problem. The paper only presents the prior-data fitted network for deep learning-based probability density estimation.

We thank the reviewer for the useful comment. We add a section on suitable deep learning methods to the related work sections in the main paper and the appendix and further briefly discuss the usage of AI-based methods for MD simulations.

We thank the reviewer again for the valuable feedback and helpful comments. We hope that our response clarified the questions and solved the reviewers’ concerns. If there are any further questions or clarifications required, we are happy to answer those. Otherwise, we would appreciate it if the reviewer could increase our score.

With kind regards,

The authors

Dear Reviewer DNAm,

Thank you for your valuable feedback and for highlighting the novelty and reproducibility of our approach. In the following, we address your concerns and questions in detail.

The current writing does not facilitate quick comprehension of the research problem for readers from diverse backgrounds. It would be beneficial if the author could include a figure illustrating specific data and formalization when introducing the problem. For instance, depicting the relationship between the RNA folding process and the corresponding change in folding fraction could enhance clarity.

We thank the reviewer for this helpful comment. We are happy to include an overview figure to increase the clarity of our proposed approach. We add a draft for the figure in the Introduction of our revised manuscript.

Would a full version of this figure solve the reviewers’ concerns regarding quick comprehension of the research problem?

The unique challenges RNA folding kinetics pose are not adequately summarized in the introduction. Additionally, the paper directly employs prior-data fitted networks to model the CDF without additional enhancements. Highlighting the improvements made to address the specific issues in this field would enhance the paper's contribution.

We thank the reviewer for the useful comment. We update the Introduction to clarify our contributions more and to avoid confusion. However, we disagree with the reviewer that we directly employ PFNs to model the CDF without additional enhancements. In contrast to previous work that use PFNs e.g. to extrapolate learning curves, we do not predict the PPD of a target y for a given quantile x conditional on a dataset D, but learn the entire PPD of y (in this case, the first passage times) without knowledge about the quantiles (x is always a zero-vector), effectively representing the absence of further information.

This approach is motivated by the underlying problem structure, where we do not have access to the true quantiles of the context first-passage times. Therefore, we cannot treat the task as a standard regression problem but directly learn the PPD of first passage times conditional on a (data)set of context first passage times but without requiring quantile information which is novel in the field of PFNs.

We add a small part at the end of the Background section of the revised manuscript to point out this novelty more clearly.

It would be clearer to explicitly state in the introduction or background section whether the paper focuses on RNA's tertiary or secondary structure, and how the folding ratio is calculated.

We include the clarification that the first passage times discussed in the paper are consistently derived from folding simulations that specifically focus on secondary structure formation in the Background section of our revised manuscript.

However, we would like to note that the underlying structural information, be it tertiary or secondary structure based, is more related to the simulators used. As a pure in-context learning approach, KinPFN is well-suited to generalize across different simulators and we would expect that the usage of a tertiary-structure-based simulator for the generation of context FPTs would only marginally impact the FPT approximations of KinPFN.

To clarify the calculation of the folding fraction: the folding fraction is represented by the cumulative distribution function (CDF) derived from a given set of FPTs. To compute the CDF using an available dataset of FPTs—referred to in the paper as the ground truth CDF over a statistically reasonable number of 1000 available FPTs—the process is as follows: The RNA folding times are first arranged in ascending order. For each time point in this sorted list, the fraction of folding events completed by that time is determined by dividing the number of folding times less than or equal to that time by the total number of events. This yields cumulative proportions at each time point, providing a stepwise function that describes the progression of completed folding events over time.

Moreover, since the models currently use Kinfold simulations, it could be limited by the accuracy of the NN energy model, which has limitations on longer RNAs. Maybe KinPFN offers new perspectives to deal with this challenge?

We agree with the reviewer that KinPFN offers new perspectives to deal with limitations of the underlying simulators, particularly those limitations that are connected to sequence length which requires substantially longer simulations. Since KinPFN is a pure in-context learner with the simulation's first passage times as context, we cannot improve the accuracy of the NN energy model. However, we can significantly reduce the time required to get accurate approximations, completely independent of the sequence length. With our experiments across different sequence lengths (Section 5.1), we took a first approach in this direction, however, the experiment is limited to RNAs with a length of < 150nt due to the very long runtimes of the simulators.

In addition, we would like to emphasize that we also use Kfold simulations (see Section 5.1) and achieve similar performance. This indicates that KinPFN’s predictions are independent of the underlying data-generating simulator but only depend on the provided context. We, therefore, expect KinPFN to work similarly well across different simulators which do not necessarily have to be secondary structure-based. This, however, is currently not empirically validated and requires further evaluations and larger-scale data acquisition.

Does this answer the reviewer's question?

The performance of KDEs seem very close to KinPFN, and I wonder if the author(s) could provide more context in the main body to discuss the significance of the results (e.g., from supp mat H.2)

We agree with the reviewer that, while outperformed by KinPFN, the performance of KDEs appears close to KinPFN. Nevertheless, we think that KinPFN has several advantages over KDEs, most obviously, it is not limited to multi-modal Gaussians for the formulation of the prior. As already stated by the reviewer, KinPFN is an early pioneering approach and there is still room for improvement. However, we will likely explore further applications of KinPFN for RNA kinetics in the future, including different distributions for the prior formulation. Regarding the explicit results of KDEs, we add a more comprehensive analysis including different metrics to Appendix H.2 and update the discussion of the results in the main body.

It may be out of the scope of this manuscript, but I would be interested to understand better how other features such as the nucleotide composition, MFE value, number of base pairs or loops, etc. impact the results

We are preparing the requested analysis and will add it to the Appendix.

However, we do not expect substantial changes in KinPFN’s ability to predict FPTs for sequences that have particularly low MFEs or exceptionally high GC contents. RNA structure is highly dependent on the sequence context but we do not expect any bias toward a particular shape of the FPT distributions as a result of the sequence/structure traits mentioned by the reviewer. While simple, our prior seems to cover a broad range of possible FPT distributions, and thus these traits should not have a strong influence on KinPFN’s prediction quality. Nevertheless, we are also curious to see the aforementioned results and will share them here when available.

How does KinPFN perform on multi-stable RNAs

Multi-stable RNAs would typically produce CDF data with one or more plateaus, depending on the modalities of the FPT distribution. Since we train KinPFN with up to five modalities, the behavior of multi-stable RNAs is generally captured by our prior. Given a set of context first-passage times of a multi-stable RNA, we thus expect KinPFN to achieve similar performance as demonstrated in the paper.

We hope that our responses clarified all the questions of the reviewer and we are happy to answer further questions if necessary. If all your concerns were addressed, we would be very thankful if you would increase your score.

With kind regards,

The authors

Dear Reviewer TUhM,

Thank you for your valuable feedback and for highlighting the soundness and novelty of our approach. In the following, we provide answers to your questions and address your concerns in detail.

Is there any further conceptual novelty in this submission that I might be missing?

We would like to further emphasize an additional unique aspect of our method: its ability to operate PFNs without requiring quantile information. Unlike traditional PFN approaches that predict the PPD of a target y for a given quantile x conditional on a dataset D, our method learns the entire PPD of y (in this case, the first passage times) without knowledge about the quantiles (x is always a zero-vector), effectively representing the absence of further information.

This approach is motivated by the underlying problem structure, where we do not have access to the true quantiles of the context first-passage times. Therefore, we cannot treat the task as a standard regression problem but directly learn the PPD of first passage times conditional on a (data)set of context first passage times but without requiring quantile information which is novel in the field of PFNs.

We add a small paragraph to the end of the background section to explicitly mark this change.

As far as I understand it, the structural model is based on secondary structures and uses (for simulations) the nearest neighbor (NN) energy model. The application to secondary structure is only partially described in the paper. It is an important limitation, and I am concerned that readers may miss this information. I would suggest updating Fig. 2 (reproduced from Muller et al, 2022) to include more details on the sampling (e.g., using a secondary structure model and Kinfold) and eventually on the benchmark too. This is just a suggestion as the author(s) may prefer to include this information at other places of the manuscript (e.g., expand the background section to describe the folding model).

We agree with the reviewer that we do not discuss the secondary structure aspects in detail in the initial manuscript. However, we design KinPFN to be independent of the simulator, and therefore also of the underlying secondary structure folding model. While we do not want to speculate about it without empirical evidence, we think that KinPFN as a pure in-context learner would also be applicable to very different simulation data. In this regard, our experiments with Kfold simulations, different start and stop structures (both Section 5.1), as well as the application to gene expression data (Section 5.4) could be seen as the first evidence.

Does this clarify the reviewers' concerns?

A claim of the paper is to dramatically accelerate folding simulation. I agree this is a nice feature, but I also wonder to what extent it is currently a major bottleneck (for biologists) or what new applications it will enable. I think the manuscript could benefit from further justifications/motivations For instance, the usefulness of KinPFN for design applications sounds very promising, and I would have liked more discussion (or experiments?) related to this topic.

We agree with the reviewer that the speed-ups achieved by KinPFN are a major contribution of our work. These accelerations indeed offer the opportunity for novel applications, in particular but not limited to, the field of RNA design. However, while we also agree that further assessment of this strength would be very interesting, we also think that the development of a kinetic RNA design algorithm (and the required experiments) is out of the scope of this work. That said, a different application of KinPFN is shown in the case study on the folding efficiency of different RNAs in Section 5.3. In this regard, KinPFN could e.g. be used to identify switching states in RNAs. Nevertheless, we agree with the reviewer that we could discuss the potential benefits of our approach more in our manuscript and we add a short section to the Introduction of the revised version of the paper.