Generative modeling for RNA splicing code predictions and design

Part3:

Finally, to demonstrate that the above results are not tied to a specific dataset/species, we repeated this analysis with TrASPr over the mouse 6 tissues dataset (MS-6) where ~23% of the sample pairs across tissues showed dPSI >= 0.15. We repeated this for the subset of changing samples (MS-6C).

     |  TrASPr  
MS-6 | 0.79,0.45 | 
MS-6C| 0.60,0.57 | 

While performance here is different (likely due to different quality in terms of #reads and #samples), the trends are the same. Taken together, we believe that we added a substantial amount of analysis/results that clearly support our assertion that (a) There is benefit in tissue specific splicing prediction (b) We indeed achieved SOTA results for this task.

Traspr appears to excel at predicting constitutive and non-constitutive exons but performs poorly for exons that fall in between (the truly alternatively spliced exons). It might provide more insight if the tested exons were categorized accordingly (e.g., see Fig. 1C in the SpliceAI paper).

The above comment seems to reflect a fundamental misunderstanding of the analysis/data which is a clear indication we failed to explain it well. Alternatively, there is a lack of clarity regarding the terms “constitutive” and “non-constitutive” used above by the Reviewer. Either way, let us try to clarify this important point. First, ALL the data presented in this work is for alternative cassette exons. Alternative exons are defined as those for which we have evidence of being alternative based on the annotation and/or read data as processed by the MAJIQ algorithm for the specified tissues. Conversely, “Constitutive” are those that appear as such in the annotation and don’t have read data supporting otherwise. Notably, these labels are by definition somewhat noisy and context specific: A constitutive exon may become alternative in some other context and an alternative one may have very little reads (or just annotation) supporting it as such while in practice it is very much “constitutive”. Distinguishing between alternative and constitutive exons (originally based on annotation alone) is considered a relatively easy task in the splicing field with AUC well over 90% in SVM papers dating back to 2005 (Dror et al Bioinformatics) and later in Splicing code papers as well (~97% AUC, Barash et al Genome Biology 2014). This is NOT the task we are handling here as all exons are “alternative” by the above definition. In fact, if we were to introduce such constitutive exons into our test set our performance would skyrocket (as discussed in the GB 2014 paper above). This is also not about scoring (cryptic/weak) splice sites vs ‘main’ ones and mutations that activate/decrease those (as in the SpliceAI paper). Consequently, referring to Fig 1C in the SpliceAI the reviewer pointed us to, we see a significant difference: The vast majority of inclusion rates in that figure (>85%) are 0.9-1.0. In contrast, the data in the GTEX-3 set we described above, based on alternative exons in those tissues, is distributed more evenly with ~50% at the 0.9-1.0 range and the rest are roughly evenly distributed between the 0-0.1 and the 0.1-0.9 ranges.

Another option is that the Reviewer made here a general comment that high inclusion and high exclusion are generally easier to predict. We would be the first to agree (see above stats from scoring constitutive vs. alternative exons) and there are of course biological reasons for that - easier to detect a crappy splice site or a very strong (aka constitutive) exon. Furthermore, “in between” cases as the Reviewer calls them are more likely to be those that are shifting in a tissue specific manner - these are exactly the ones we are trying to better predict here.

In conclusion, predicting alternative vs. constitutive is a much easier task then predicting exact PSI for alternative only events (which is the case here), and predicting ‘in between’ cases specifically is harder as these are harder to detect and many times reflect tissue specific signals. However, we strongly disagree with the negative conclusion “performs poorly” - compared to what? For sure we could do batter, but we believe we show strong evidence we actually perform better than previous algorithms on those in between values (see for example the GTEX-3C evaluations above when things by definition shift from 0 and 1 and classifying tissue specific changes).

(continued in the next part)

Part2:

The authors focus on differential splicing and use splicing profiles across multiple tissues and species to train their model. However, they do not present any immediate benefits. For instance, they do not provide information on how much SpliceAI (Jaganathan et al., 2019) loses in performance, or how much Traspr gains, by having a universal (tissue-agnostic) model for splicing.

Trying to address the above concern we see two potential arguments being made here by the Reviewer. The first potential argument being made here is that there is no point (“immediate benefits”) in modeling tissue specific splicing. For this, we would point to decades long research about the functional importance and overall abundance of tissue splicing, how it is tightly regulated and how changes in tissue specific splicing (or targeting it) are key in disease and for therapeutics. We are happy to make that point more clear in the revised version but do not believe this requires additional proof in the scope of this paper.

The second argument, which seems more likely to follow from the Reviewer’s suggestion to compare to SpliceAI, is that even if predicting tissue specific splicing is desirable (see above point), a strong tissue agnostic algorithm such as SpliceAI may do just as well. We agree we should have better addressed this concern and therefore performed the tests described below. Before we describe those in detail below, it is important to note the following: If you take a random set of AS events (as was done here and in many other studies) across a few tissues, most events are NOT tissue specific. Thus, a good predictor of their non-tissue specific PSI should do well (making the point about SpliceAI comparison). However, being able to predict tissue-specific events is nonetheless an important task even if those cases are the smaller set (see point above). Now for the additional analysis and results:

We ran the three algorithms in question (SpliceAI, Pangolin, TrASPr) on AS events from the test chromosomes as described in the paper, for the 3 tissues that matched those in the Pangolin paper. We term this dataset GTEX-3. The results of these tests were as follows (all results include two numbers: first Pearson, then the Spearman correlation)

       |  TrASPr   | Pangolin  |  SpliceAI
 GTX-3 | 0.91,0.81 |  0.8,0.67 |  0.69,0.6

We find that Pangolin (note the numbers here reflect the bug fix described above), which uses the SpliceAI architecture but is trained with tissue information, indeed offers improvements over SpliceAI in PSI prediction in those tissues. TrASPr significantly outperforms both models. The improvement of TrASPr over Pangolin is also much more significant than Pangolin over SpliceAI. Also, note that as described above we also tried re training Pangolin with MAJIQ PSI values for these 3 tissues, with similar results. Overall, these results suggest that the change in model architecture is key to achieve this improved PSI prediction performance across the 3 tissues.

While the above shows significant improvement by TrASPr, most events in the GTEX-3 test set do not change PSI significantly between a pair of conditions/Tissues. Specifically, in the GTEX-3 test set only ~25% of the tissue pairs comparison in this set show a change of dPSI >= 0.15 so performance on this is governed to a large extent by a good fit for PSI values more than tissue specific effects. To zoom in on those important subset of events we defined the test set GTEX-3C: The same test set as above but now focusing only on the ~¼ of pairwise cases that exhibit dPSI >= 0.15.

       |  TrASPr   |  Pangolin  |  SpliceAI
GTX-3C | 0.75,0.73 |  0.59,0.55 |  0.41, 0.39

Here we see more clearly the benefit of tissue specific modeling when events are indeed changing, at least to some extent (dPSI >= 0.15). Still, this test reflects PSI prediction on changing events and does not measure directly the ability to classify changes. We thus computed AUROC and AUPRC (since there is class imbalance) for classifying both events that are changing “Up” (dPSI+) between two tissues and those that are changing “Down” (dPSI-) between two tissues. Since this is the GTEX-3 dataset (i.e. the 3 tissues available in Pangolin) we have 3 pairs of tissues so the stats below are the average of those (first number is for dPSI+ second number is for dPSI-)

TrASPr:

AUPRC: [0.37 0.50]

AUROC: [0.83 0.88]

Pangolin:

AUPRC: [0.15 0.23]

AUROC: [0.60 0.68 ]

We see a significant improvement with TrASPr. Note that by definition SpliceAI can not be applied here.

(continued in the next part)

We thank the reviewer for noting the promise in the BOS algorithm for designing sequences for RNA splicing and the significant amount of work that went into this manuscript. Due to the character limitation of each response, we will respond in multiple comments.

Part 1:

Weaknesses:

This manuscript attempts to address two major problems but, unfortunately, does not succeed in either case. Here are the main issues:

The manuscript indeed addresses two major problems: (1) Tissue specific splicing predictions (2) Generative model for sequences that would exhibit desirable (tissue specific) splicing profile. We hold that we are indeed able to successfully address those problems in this work. We believe this rather strong language/statement by the reviewer is a compound result of (a) misunderstanding some of the results/analysis presented (b) an error we made in one of the results (“correlation 0.17”) which made it look suspicious to the reviewer, and consequently led them to doubt the entire manuscript (c ) additional evidence which the reviewer requests and that we believe we adequately address below. We break down our response to each specific point raised by the reviewer below. We apologize in advance for the length of this response - the comments and strong disbelief we felt reading those required a lot of work and analysis which we list below.

In Section 4.1, the authors themselves acknowledge that "[...] predictions for tissue-specific splicing changes were not very accurate, and we, therefore, did not include them here."

This seems to be a misunderstanding - this comment in the manuscript was referring to a point made by the Pangolin authors about the fact Pangolin is not very accurate in predicting condition specific splicing changes - it is not a comment about TrASPr or about predicting tissue specific splicing changes in general. Regardless, we include those results now and as we clearly show below Pangolin does offer improved tissue specific splicing predictions over SpliceAI (after the above bug was fixed…) and TrASPr significantly outperformed both of those in this task.

Furthermore, the approach for calculating cassette exon inclusion in Pangolin, as described in the manuscript, may not be optimal, as it ignores the upstream and downstream exon junctions. A more appropriate method can be found in the Methods section of the SpliceAI paper.

This seems to be a misunderstanding by the reviewer regarding how Pangolin computes inclusion. Pangolin uses SpliSER which does indeed take into account competing upstream/upstream junctions and computes a ratio between those going each splice site. SpliSER can be seen as a generalization of the direct approach the reviewer mentions in the Methods section of the SpliceAI paper, where multiple upstream/downstream splice junctions can be accounted for. Regardless, for the 3 GTEX tissues analysis described below for Pangolin we also retrained the model with MAJIQ’s PSI values (i.e. the same input used for TrASPr) and got no distinguishable differences on that dataset (changes of 0.01 and 0.02 in Pearson and Spearman correlations). Thus, we conclude that this is not an issue in our analysis.

The manuscript lacks convincing performance comparisons. It is challenging to believe that Traspr outperforms Pangolin (and subsequently SpliceAI) by such a significant margin (Pearson correlation 0.81 vs. 0.17). SpliceAI and other methods have undergone extensive testing by independent researchers.

We believe this issue is likely at the heart of the reviewers general skeptical and negative assessment and therefore would like to address it first. A small correction to the Reviewer’s point is that the claimed 0.17 was Pearson (not Spearman). We wrote: “Pearson correlation for PSI prediction (0.81 vs 0.17 see Fig2)”. We actually did exactly the same mistake ourselves which was Error1 by us: The scatter plot shown in Fig2 had Pearson correlation of 0.34. The 0.17 value was the Spearman value, with rank correlation suffering more from the bulge of points at the top of the scatter plot. This “bulge” leads us to the much more severe Error2 we made. This has to do with differences in input data processing between SpliceAI and Pangolin which messed up the Pangolin performance. We only caught this mistake after submission and we deeply apologize for not catching it before. The actual numbers are indeed much higher, and the scatter plot does look better (will be included in the final version, if given the opportunity). Again, we believe this error set the tone for much of the review and we deeply apologize for not catching it early. We hope both the correction and the many additional analyses detailed below will address the “ lacks convincing performance comparisons” view of the reviewer.

(continued in the next part)

First, I need to thank the authors for the immense amount of time that they have put on answering reviewers' questions. And I do believe most of them should have been in the paper to begin with. For example, the focus on dPSI >= 0.15 might be a much more convincing experiment than what is already in the paper.

Some of the explanations help me understand the paper better, but I feel my original comments still apply. The authors attempt to solve two hard computational biology problems, which results in suboptimal results for both. This might be better suited for a Comp Bio journal rather than this venue.

To summarize my comments and reiterate my points, let's focus on Section 4.1 and Fig. 2. Here the goal is predict PSI in different tissues and it is essentially the main benchmark.

• What does each bin in the 2D histogram show? Are all (tissue, AS event) included in this plot? How does the tissue-stratified PSI prediction error look like?
• Looking at Traspr results and if one excludes PSI < 0.1 and PSI > 0.9, correlations look much weaker, judging by the density of points at the small and large PSI values. Again, focusing on dSPI rather than PSI could be helpful here.
• Tissue-specific splicing can also be switch-like: an exon is always included in one tissue and never included in other tissues. Showing dSPI instead of PSI can be helpful.
• It looks peculiar that Pangolin does such as bad job here. It predicts an almost-zero PSI for hundreds if not thousands of events, that even basic algorithms don't. Is PSI zero in one tissue or in all tissues? I think the devil is in how splice site usage is translated into PSI.

I will finish by saying that the BOS method looks super promising and I am confident with the right experimental setup it can be an impactful method.

We thank the reviewer for the time and effort spent to review and comment on our work. We were very pleased to find such a nice summary of key points as points of strength in our work - Thank you!

We focus below only on the main points raised that require fixing/clarifications.

The ability to assess the validity of the mutated sequences is rather limited. The authors were able to show that alternative methods for exploring sequence space to find mutated sequences are less effective at that.

Yes, we agree this is a limitation. A more comprehensive method is of course to have such data generated by a high-throughput assay but this is well beyond the scope of a short ML conference paper.

Although overall the manuscript is clear and easy to read, it contains many typos and grammatical errors - see below.

We thank the reviewer for taking the time to point those out. We will be happy to correct all of those in a revised version, if given the opportunity.

When it comes to in-silico mutations, I am not sure the authors' experiment shows all that much. A better approach might be to see how well the model predicts splicing QTLs as was done in the Pangolin paper (and using integrated gradients works better than mutagenesis in our experience).

The reviewer raises a good point here we should elaborate on. To be clear, this work aims to achieve two objectives: (a) build a model that achieves SOTA performance for tissue specific splicing predictions (b) develop a new BO method for designing RNA sequence with user defined condition specific splicing profile (and formulating the task as such). We are NOT aiming here to prove SOTA in predicting the effect of mutations for say disease diagnostics. We believe such applications are exciting future directions, as we describe in the discussion section. Thus, the in-silico mutation analysis in this work is only aimed to show the model is able to capture tissue specific regulatory elements. Yes, we very much agree overlapping the model predictions with sQTL is a great way to show the ability to predict the effect of mutations but again this is not the focus of this work and many sQTL hits are not tissue specific. We did not have an easy solution/set to assess against (for example, the set of variants used in the recent Wagner et al Nat Gen 2023 is not accessible to the best of our knowledge) and running an extensive analysis for say sQTL that are also tissue specific is beyond the scope of a short ML conference paper. Please also see a much more lengthy discussion of this point, including additional data and analysis we added for another mutation dataset in response to reviewer Bzy6.

We hope the extensive responses we included to all reviewers will convince the reviewer the revised manuscript will safely pass the acceptance criteria.

We thank the reviewer for the time and effort spent to review and comment on our work. We focus below only on the main points raised that require fixing/clarifications.

The literature review concerning splice site prediction is phrased in a confusing way. On pg 2, when discussing DNABERT, SpliceAI, etc. authors mention sequence length limit (e.g. 10kbp for SpliceAI) as a key challenge these method face, and yet the method proposed in the paper uses short window, 400bp for each of the four Transformers. Recent tools such as SpliceBERT are not mentioned.

We will be sure to improve the literature review, framing the works better in terms of tasks and adding SpliceBERT. Regarding the usage of only short windows - please see a detailed discussion about this point in response to reviewer Bzy6 which raised a similar question. This again is a reflection of lack of clarity in the review section about what each such model/work tries to achieve (which is very different!) and design choices accordingly.

The description of the “condition-specific” nature of the method is somewhat vague, and should be described/discussed in Introduction and in the relevant Methods section in more detail. For example, it seems (pg. 3, bottom paragraph) that the method does not take condition information (RBP knockdown) on input directly, but “simulates it” via sequence modification. On the other hand, “condition” relating to tissue specificity seems to be used as input to the model as part of the Event features; rationale for this approach and its impact on usage of the method e.g. for splicing-altering diseases should be discussed.

That is a good point which we definitely should make more clear. To address the question raised, the model is “condition specific” in the sense that it learns to predict condition specific splicing outcome. The “condition” can be anything the user decides to train on - tissue type, specific cell line, a specific cell line with an RBP KD (as done for example by ENCODE for hundreds of RBPs), a cellular condition (e.g. stress response in a cell line of interest) etc. This definition is very much inline with previous works on splicing code prediction algorithms, dating back to the first splicing code (Barash et al Nature 2010). In this work we focused on “condition” as “tissue” in two key organisms (human, mouse) as this is a main focus of much current research and relates to therapeutics as well. In this setting we are not trying to mimic an RBP KD as an objective - we only did the in silico motif removal to illustrate that the model is indeed learning condition specific regulatory motifs. Similar analysis showing ability to infer new biology for splicing regulation when modeling RBP KD as a “condition” was shown in Jha et al Bioinformatics 2017 for CELF vs. FOX antagonistic regulation, which was followed by extensive experimental validation in Gazzara et al Genome Research 2017. We will make sure to improve our description and framing of these tasks in the revision, if given the opportunity.

We hope the extensive responses we included to all reviewers will convince the reviewer the revised manuscript will safely pass the acceptance criteria.

We thank the reviewer for the time and effort spent to review and comment on our work. We focus below only on the main points raised that require fixing/clarifications.

Unfortunately, I am not convinced by the manuscript that the goal is achieved. First, I am not sure what meta data information could be utilized to predict condition-specific splicing, assuming the input RNA sequences are from the same species thus are very similar. I feel it will be useful to give a concrete example to motivate the proposal.

There seems to be some lack of clarity regarding what is “meta data”. As we describe in the text, this data is simply information regarding things like exon length, intron length, tissue type etc. This information is known to relate to splicing outcome (e.g. micro exons that are neuro specific) and are thus useful. We clearly show are useful in the ablation study (Table2). It’s true that in some species splicing may be very different (e.g. in plants) but we are not exploring those cases here.

Second, I think it will be nice to expand the prediction part with more experiments to validate the basic assumptions. For example, if we replace the input sequences with random sequences or sequences from non-splicing genes, but with the same meta-data, we expect to see that psi is close to 0. Assuming the network has successfully captured the regulatory mechanism, we expect to see the psi will decrease to 0 if we remove the U2 elements (suppl. Fig. 7d) from the input sequences. Since the splicing mechanism is conservative between human and mouse, we expect to see that NN trained on human data should exhibit similar predictive power on mouse homolog genes. If the prediction part works with high precision, then it is worth designing novel RNA sequences.

We have worked very hard to add analysis and results that would support our claims that (a) we achieve SOTA performance in tissue specific splicing prediction. We refer the reviewer to the extensive details we included in the response to reviewer Bzy6. Regarding the specific suggestions: Abolition of the splice signal (e.g. U2 binding area) leads to immediate destruction of the splicing - this is part of the pre-training of the Transformer on splice sites which is an easy task shown in many previous papers (and why we didn’t emphasize it). More generally, see our discussion in response to Bzy6 regarding the much easier task of constitutive vs. alternative exons. We do indeed see similar predictive power between mouse and human as we show in the paper (and see again Bzy6 response).

It will be really helpful for me to read the ms if the authors could highlight the main ideas and assumptions and hiding unnecessary details.

We will make sure to improve the writing and better flush out the main ideas/assumptions, given the opportunity. With that said, we respectfully request the reviewer to carefully consider their rejection score given the above and the many details/clarifications we supplied to the other reviewers. As it stands we did not find any specific claim made by the reviewer that warrants a rejection score. If they still hold the paper should be rejected (score of 5) we would like to have specific arguments about where we fail to develivr on the tasks we set out, namely: (a) develop SOTA predictor for tissue specific splicing (b) use it for a new BO algorithm for designing RNA sequence with tissue specific splicing constraints. We thank the reviewer again for the time and effort on this.

I think it is equally important to clearly convey the underlying assumptions and show some intuitive examples to the readers beyond SOTA performance, which I have not seen from the authors' feedbacks.