Omni-DNA: A Genomic Model Supporting Sequence Understanding, Long-context, and Textual Annotation

We sincerely appreciate your detailed feedback and constructive remarks on our paper. Below, we provide a detailed response to your questions and comments, along with additional ablation results from your suggestion.

Q1. The authors discuss several approaches for extending pre-training vocabularies (Lines 142-149). Which is used in the experiments?

There are two tasks which introduce new vocabulary to the Omni-DNA: Next-Token Prediction approach for 10 tasks (section 4.1 Multitasking) and Sequence-to-Function Generation. The key-token replication is only used for multi-tasking; while NEFTune (i.e., adding isotropic Gaussian noise) is used for both tasks.

Q2. ...the $logK$ factor would not be that untenable?

While the theoretical difference between $O(N\log K)$ and $\Theta(N)$ might appear marginal, the practical efficiency gains of our method are impactful due to the following reasons:

• 1. As SeqPack needs to be integrated into both finetuning and inference, when N is very large, logK*N could lead to differences as well.
• 2. Memory footprint is another concern. When we perform global sampling, we need to maintain relatively large tensors in memory, while the splitting approach allows the sequential/parallel processing of each segment.
• 3. Implicit prior, by encouraging the compressor to sample from each segment, we are enforcing an implicit uniform prior, which could help to avoid the potential issue of collapsing the learned mass on a single region.

Q3. Doesn’t applying top-k destroy differentiability?

If we directly use the top-K selection method, we get hard binary choices (either a token is selected or not) and this creates a problem: since the selection is discrete, there's no gradient information flowing back through the selection process. We use Gumbel–Softmax to build a soft, differentiable selector $y_{\text{soft}}$ from the scores, then apply a straight-through estimator: the forward pass uses hard one-hot indices for behavior, while the backward pass uses the soft probabilities for gradients. Thus, the model learns from a smooth relaxation even though the forward selection is discrete. At inference, we switch to ordinary discrete selection (sampling or top-k).

// Inputs
// X: (B, L, D) token embeddings
// weights: (L,) non-negative scores (normalized internally)
// k: number of tokens to keep
OUTPUT: selected_token_embeddings(B,k,D)

IF training_mode:
    // Gumbel-Softmax for differentiable selection
    logits = expand(log(weights), to=(k,L))
    gumbel = sample_gumbel_noise(shape=(k,L))
    y_soft = softmax(logits + gumbel )
    
    // Straight-through: hard forward, soft backward
    y_hard = one_hot(argmax(y_soft))
    y_st = detach(y_hard - y_soft) + y_soft
    
    // Apply selection and return
    RETURN einsum('kl,lbd->kbd', y_st, transpose(sequence))
ELSE:
    // Standard multinomial sampling for inference
    indices = multinomial(weights, k, no_replacement=True)
    RETURN sequence[:, sort(indices), :]

Q4. Can the authors confirm that all the models from Section 4.2 (Table 4) have 450kbp input sequences. Additionally, would it be possible to provide error bars for Table 4?

Yes, we confirm that all the models are evaluated with the input sequence of length 450,000 bp. We use the original benchmark from [1]. Following your suggestion, we ran the code from the original benchmark to obtain the new table with error bars as shown below, and we will integrate this into the paper.

Q5.What is the positive class ratio of the final datasets used in the experiments from Section 4.3.

The positive: negative ratio is 1:1.

Q6. Additional ablation on the SeqPack Parameter with different Compression Ratio

To abalate the parameter of SEQPACK compression ratios (c = $\frac{L_{original}}{L_{pruned}}$) and assess SEQPACK's effectiveness against established methods, we adapted Dynamic Context Pruning (DCP)[2] to work with Omni-DNA 116M on the Enhancer-Target Prediction task. The results demonstrate SEQPACK's substantial advantages.

At extreme compression ratios (220×, 150×), DCP performance degrades severely, barely outperforming random baselines, while SEQPACK maintains strong performance (AUROC > 0.89). Even at moderate compression (50×), SEQPACK outperforms DCP (0.907 vs 0.880 AUROC).

Q7. The computational overhead of SeqPack

Compressing inputs (25,600–819,200 tokens) to a fixed 4,096 with SeqPack, we report (i) SeqPack operator time and (ii) Omni-DNA-1B forward time in seconds on the compressed sequence. As input length grows 32×, SeqPack time rises only 3.56× (inference: 0.00898 to 0.03197 s) and 5.95× (training: 0.015 to 0.091 s). Forward time is effectively flat. SeqPack’s share of end-to-end time remains low and increases gently with the compression ratio 6.25× to 200×: 1.2% to 4.0% (inference) and 4.3% to 21.6% (training).

Inference

Training

Q8. Additional Ablations on why Omni-DNA works

We generally believe the transformer's learning capability is the reason why the model works. To see the effect of the training paradigm (BERT vs. NextTokenPrediction), the additional ablations are conducted, with detailed results presented in our response to Q1 of reviewer 1(rvnr). We could see that the BERT pretraining, in terms of the data source, we found that the number of training samples is a key. But NTP training provides the model with the natural ability for sequence generation, which is generally preferred and has substantial downstream use cases. This is the reason why we use the NTP as the default training scheme. But due to the limit of computational resources, we may not be able to explore all the architectural choices.

Q9. Finding 2

Finding 2 is to emphasize that the lower loss does not guarantee better performance in downstream tasks. We can see several bumps in Figure 3, which means such relations are not guaranteed. We should be cautious about using the loss as the only criterion for selecting the best model.

Q10. Regarding a few suggestions of the introduction section.

We appreciate your kind suggestion. The below is the reframed paragraph to replace line 31-40 of the original manuscript.

While GFMs have demonstrated utility on specific genomic tasks, they still exhibit important limitations: (i) Limited optimization of architectural components for genomic tasks: Prior work has explored design choices including tokenization (DNABERT vs. DNABERT-2), positional encodings (NT vs. NTv2), and genomic-specific modules (GPN-MSA, Caduceus), but a more comprehensive, task-specific search of architectures and pretraining strategies tailored to genomic inductive biases is still lacking. (ii) Limited exploration of multi-task synergies: Many approaches fine-tune on a single downstream task, missing the opportunity to exploit shared biological signal and achieve cross-task performance gains, despite evidence of such benefits in genomic assay prediction (e.g., AlphaGenome, Enformer). (iii) Insufficient multimodal capability: Most existing GFMs restrict vocabularies to nucleotide base pairs, predefined labels, or learned subwords with the BPE tokenizer, limiting tasks that require mapping DNA sequences to functional annotations in natural language (DNA2Func). While ChatNT can process (DNA, text) inputs, it uses separate tokenizers and a two-stage stack consisting of NTv2 as the DNA encoder and Vicuna-7B as the decoder, resulting in less unified handling of multimodal information. (iv) Context-length adaptivity challenges: Transformer-based models (e.g., DNABERT-2, NTv2) remain constrained by the quadratic cost of self-attention. This limits the application of transformer-based models to long-context tasks such as enhancer–target prediction.

[1] Cheng, Wenduo, et al. "Dnalongbench: a benchmark suite for long-range dna prediction tasks." bioRxiv 2025.

[2] Anagnostidis, Sotiris, et al. "Dynamic context pruning for efficient and interpretable autoregressive transformers." NeurIPS 2023.

Thank you for your positive feedback of and constructive suggestion for our manuscript. Below, we address each of your questions and comments in detail.

Q1. Did you mask the intervening bases between the enhancer and how was this masking accomplished?

We adopt the curated benchmark from Cheng et al. [1] and use the same masking strategy to mask the intervening enhancers. In the original implementation, the positions of the intervening enhancers in K562 cell lines are recorded in a text file as a blacklist. This file includes the chromosome, start, end, and strand of the enhancers that need to be masked. During data loading with selene_sdk (a python library), the masked positions are replaced with [0.25, 0.25, 0.25, 0.25]. In our case, we replace these positions with 'N' for Omni-DNA to consume.

Q2. Genetic disease variant-type classification

Objective: Conventional training on benign vs. pathogenic labels is valuable but inherently coarse: it often fails to capture nuanced mutation mechanisms—such as how mutation direction varies across diseases and how these differences map to distinct disease phenotypes within the same gene. Our objective in this task is to move beyond generic assessments of deleteriousness and enable the model to learn disease-related patterns that reflect fine-grained functional changes. Disease-type labels serve as a form of weak supervision for learning mutation directionality—something that a simple benign-vs-pathogenic objective cannot reliably disentangle.

In real-world workflows (e.g., cardiac clinics), clinicians need to prioritize variants specific to the target disease in the form of patient reports, rather than flagging all pathogenic variants across conditions. Current practice relies heavily on rule-based gene–disease associations from resources such as OMIM[2] and the Clinical Genomic Database (CGD)[3] as disease-focused filters. We demonstrate that by fine-tuning general sequence models with disease-type labels, we can rapidly recover these associations and highlight the most relevant variants for the disease of interest—thereby aligning model outputs with established clinical triage pipelines.

Draw the negative controls from non-pathogenic variants: Nevertheless, we are happy to include the additional results in the appendix, based on sampling negatives from non-pathogenic variants, as shown below. Specifically, we sample the negatives from benign variants with an AF_TGP (allele frequency from the 1000 Genomes Project) less than 0.01, using the same version of ClinVar. The classification F1 is generally higher than that of the disease-type classification task (as shown in Table 5 of the manuscript).

What does the model learn: In order to examine whether the model learns disease–variant associations by memorizing gene identity or other non-coding regions, we performed a simple ablation study for lung cancer. In this setup, genes were masked with 'N' (based on the human gene annotation file), and the model was still able to achieve reasonable performance. Note that implicitly including gene position is not necessarily undesirable, as this aligns with how hospitals typically prioritize variants using rule-based systems.

[1] Cheng, Wenduo, et al. "Dnalongbench: a benchmark suite for long-range dna prediction tasks." bioRxiv 2025

[2] https://www.omim.org/ OMIM (Online Mendelian Inheritance in Man)

[3] https://research.nhgri.nih.gov/CGD/ CGD (Clinical Genomic Database)

Thank you again for your constructive feedback on our submission. We're writing to follow up with the rebuttal. If there is any follow-up questions, we are happy to answer.

We appreciate your feedback on the paper. Below, we provide detailed responses to your questions.

Q1. MLM vs. NTP as pretraining objective

We agree with your view if one considers the genome as an object that needs to be modelled as a whole. However, there is no de-facto paradigm for pretraining genomic foundation models. Some recent models, such as Evo[1] and Evo2[2], adopt next-token prediction (NTP), while others, such as DNABERT2 and NTv2, use masked-language modeling (MLM); as do others for genomic foundation models, e.g., contrastive learning in Orthrus [3]. Our work does not depend on a specific pretraining objective, and the choice is largely orthogonal to our contribution.

To address this point, we added a small ablation as shown below: we pretrain the same backbone (Omni-DNA 116M) with either NTP or MLM under the same token budget and model size, then fine-tune without a causal mask for the discriminative downstream tasks (so both settings see bidirectional context at fine-tuning time). We observe task-dependent differences: NTP matches or exceeds MLM on several tasks (Promoter and Enhancer), while MLM is competitive on others (Splice); importantly, our main conclusions are unchanged.

A practical side note is that NTP-pretrained models also retain generative capability, which can be convenient for sequence design use cases; however, this is not central to our claims.

Q2. Losing nucleotide level precision

We intentionally do not apply SeqPack during pretraining. Instead, SeqPack is an optional, task-configurable module used at fine-tuning/inference time. This separation preserves a fully resolution-retaining base model while allowing practitioners to choose compression only when it benefits a specific task.

For tasks that require base-level fidelity (e.g., SNP-focused tasks), SeqPack’s compression ratio can be set to 1× (i.e., no compression). In this setting, there is no information loss, so sensitivity to short motifs is unchanged. For long-sequence tasks, when inputs exceed the model’s native context window (due to memory/time constraints), SeqPack enables efficient processing with minimal performance loss by compressing long contexts while preserving salient information. This allows Omni-DNA to handle long-range tasks within practical compute budgets.

To demonstrate SeqPack's effectiveness compared to other context length compression algorithms, we evaluated it against Dynamic Context Pruning (DCP) [4] on the Enhancer–Target Prediction task. In the table below, we applied DCP to OmniDNA-116M and compared its performance with SeqPack. DCP performed significantly worse than SeqPack across compression ratios ranging from 100 to 200, barely outperforming the random baseline model.

Q3. Unsatisfactory results on cCRE and splicing

We appreciate the concern. Our reading is that this comment refers to the NT downstream benchmarks in the Table 2 of the original manuscript.

Overall, Omni-DNA attains the best score on 13/18 tasks. The remaining tasks which Omni-DNA does not performs the best are related to promoter (cCRE-related) and splicing categories.

The promoter classification task includes two subtasks:

• NONTATA: Omni-DNA-116M trails the top model (NTv2-500M) by 0.1%.
• ALL promoters: Omni-DNA-1B is 0.2% shy of the best result.

These deltas are marginal and within typical variance; they do not indicate a substantive weakness. For splicing tasks, Omni-DNA-1B achieves an average of 0.959 across the three splicing tasks, ranking just behind NT-2.5B and NTv2-500M, and ahead of all other baselines. Given reported standard deviations, the differences are modest rather than “unsatisfactory.”

The evidence shows consistent, near-SOTA performance on promoter/cCRE and strong top-tier results on splicing, alongside state-of-the-art performance on the majority of tasks. We therefore believe the characterization “unsatisfactory” is not supported by the data.

Q4. What heads does the proposed method use in this paper? Have the authors evaluated the performance of different heads?

As stated in section 3.1 (L125–128), we attach a lightweight 1-hidden-layer MLP (“linear-MLP”) head per task on top of the model’s sequence representation, following prior GFMs (e.g., NTv2, DNABERT2). Concretely, with last-token $r \in \mathbb{R}^d$, the head is, $z = W_2\sigma(W_1 r + b_1) + b_2 \in \mathbb{R}^{C}, \qquad \hat{y} = \mathrm{softmax}(z)$, where $C$ is the number of classes. As we update the transformers' weights during fine-tuning; increasing head complexity did not yield performance gains in our checks (linear vs. 1-hidden-layer vs. deeper), so we keep this lightweight head for all results.

Q5. Results with HyenaDNA

We notice the gap between the original Hyena paper and the reproduction experiments, hereby providing a detailed discussion of the problem in the Appendix section E. In summary, the difference arises from two factors. 1) Evaluation Settings: our evaluation setting follows Caducess [5], which allows a maximum fine-tuning epoch of 20 for NT Downstream tasks and 10 epochs for Genomic Benchmark; while the original paper allow the max finetuning epoch to a few hundreds. 2) the original HyenaDNA uses the EMA (Exponential Moving Average), while our evaluation does not use it. Note that the reported results in our manuscript are consistent with the prior work, such as Caducess[5] and the follow-up work[6].

[1] Nguyen, Eric, et al. "Sequence modeling and design from molecular to genome scale with Evo." Science 2024

[2] Brixi, Garyk, et al. "Genome modeling and design across all domains of life with Evo 2." bioRxiv 2025

[3] Fradkin, Philip, et al. "Orthrus: towards evolutionary and functional RNA foundation models." bioRxiv 2024

[4] Anagnostidis, Sotiris, et al. "Dynamic context pruning for efficient and interpretable autoregressive transformers." NeurIPS 2023

[5] Schiff, Yair, et al. "Caduceus: Bi-directional equivariant long-range dna sequence modeling." ICML 2024

[6] Wu, Wei, et al. "GENERator: a long-context generative genomic foundation model." arxiv 2025

Based on your review, we have added several new experiments and clarifications:

• An NTP vs. MLM ablation study under matched compute budgets.
• SEQPACK vs. DCP comparisons on long-range association tasks.
• Clarification on the lightweight nature of the per-task MLP heads, including checks on their complexity.
• A note detailing the reproduction settings for HyenaDNA.

If you have a moment to review our updates, we would be grateful for any follow-up questions before the deadline.

Thank you for the detailed feedback. Below, we first address each of your questions, then clarify Section 3.3.1 and present a revised Contributions section aligned with your suggestions.

Q1. SeqPack compresses a sequence of length K into a much smaller embedding of length L. Does this lead to a loss of resolution, thereby limiting OmniDNA’s sensitivity to very short sequence tasks?

We intentionally do not apply SeqPack during pretraining. Instead, SeqPack is an optional, task-configurable module used at fine-tuning time. This separation preserves a fully resolution-retaining base model while allowing practitioners to choose compression only when it benefits a specific task.

For tasks that require base-level fidelity (e.g., SNP-focused tasks), SeqPack’s compression ratio ($\frac{L_{original}}{L_{pruned}}$) can be set to 1× (i.e., no compression). In this setting, there is no information loss, so sensitivity to short motifs is unchanged. For long-sequence tasks, when inputs exceed the model’s native context window (due to memory/time constraints), SeqPack enables efficient processing with minimal performance loss by compressing long contexts while preserving salient information. This allows Omni-DNA to handle long-range tasks within practical computational budgets.

To demonstrate SeqPack's effectiveness compared to other context length compression algorithms, we evaluated it against Dynamic Context Pruning (DCP) by Anagnostidis et al.[1] on the Enhancer–Target Prediction task. In the table below, we applied DCP to OmniDNA-116M and compared its performance with SeqPack. DCP performed significantly worse than SeqPack across compression ratios ranging from 100 to 200, barely outperforming the random baseline model.

Q2. In Table 3, Caduceus-PH seems to perform comparably to Omi-DNA despite having 300x fewer parameters. Could the additional performance shown by Omni-DNA simply be a result of the much larger parameter space it explores?

This performance comparison can be explained at multiple levels. First, the fundamental architectural differences between Mamba and Transformers (state space models vs. dense self-attention) play a crucial role. According to the original Mamba papers [2][3], Mamba-based models are computationally efficient, demonstrating advantages with lower computational requirements (smaller models and reduced training). However, when scaling both Mamba and Transformer architectures to billion-parameter scale, empirical results [4] show that large Mamba-based models (at Billion Scale) trained on fewer than 1 trillion tokens perform significantly worse than transformer-based models on sequence understanding tasks. This motivates the use of transformers as the backbone in this work. Second, Caduceus-PH benefits from Reverse Complementarity (RC) data augmentation during training. This augmentation technique enables the model to learn effectively with substantially fewer parameters by leveraging the inherent symmetry in DNA sequences. Notably, when comparing OmniDNA to other transformer-based models like DNABERT2, OmniDNA consistently demonstrates superior performance across most tasks, suggesting that the performance gains extend beyond simply having more parameters and reflect architectural and training improvements.

Q3. What do you mean by “output all tasks” in line 235?

We thank the reviewer for pointing this out, it is a typo. Line 235 should be "Omni-DNA (multitask,multi-head) and NTv2 (multitask,multi-head) outperform their single-task counterparts on all tasks".

Q4. Does the choice of tasks impact the benefit one could derive from this approach?

Yes, the choice of tasks would impact the benefits derived from this approach. In the multitasking framework presented in this work, a shared model backbone processes all tasks, followed by lightweight task-specific linear MLPs. When tasks are too dissimilar or contradictory to each other, the shared backbone may struggle to learn useful hidden representations. However, this limitation can be addressed by incorporating more expressive task-specific decoders. Such decoders would provide greater flexibility for the model to adapt its representations to diverse tasks, potentially enabling effective joint learning among seemingly unrelated tasks.

W1. The position of Section 2.2

We are happy to move the position of ablation studies in Section 2.2 for a better flow. Thank you for the suggestion on that.

W2. Additional Baselines for sequences to functions

We have added ChatNT[5] as an additional baseline for sequence-to-function tasks. ChatNT employs NTv2 500M as its DNA sequence encoder and a 7B language model as the language decoder, processing (DNA/RNA, text) inputs to generate textual outputs. Following the methodology from the original paper, we fine-tuned ChatNT using our Seq2Func training split. ChatNT@zeroshot represents direct application to the function prediction task without fine-tuning, while ChatNT@ft is the fine-tuned version trained on our evaluation split for 10 epochs. The results demonstrate that ChatNT, despite being specifically designed for handling (text, sequence) inputs and generating textual outputs, does not perform well on function prediction tasks. In zero-shot evaluation, although the model was instructed to output function descriptions, it consistently produced a fixed formatted response: "This RNA exhibits a motif of <number>." The fine-tuned version (ChatNT@ft) provided more reasonable answers but still lagged behind OmniDNA's performance. These results show that even compared to ChatNT, which uses separate tokenizers and models for DNA and text processing, OmniDNA presents a simpler yet more effective solution for sequence-to-function prediction.

W3. Clarification on Section 3.3.1

If we directly use the top-K selection method, we get hard binary choices (either a token is selected or not) and this creates a problem: since the selection is discrete, there's no gradient information flowing back through the selection process. We use Gumbel–Softmax to build a soft, differentiable selector $y_{\text{soft}}$ from the scores, then apply a straight-through estimator: the forward pass uses hard one-hot indices for behavior, while the backward pass uses the soft probabilities for gradients. Thus, the model learns from a smooth relaxation even though the forward selection is discrete. At inference, we switch to ordinary discrete selection (sampling or top-k).

Inputs
 X: (B, L, D) token embeddings
 weights: (L,) non-negative scores (normalized internally)
 k: number of tokens to keep
OUTPUT: selected_token_embeddings(B,k,D)

IF training_mode:
    // Gumbel-Softmax for differentiable selection
    logits = expand(log(weights), to=(k,L))
    gumbel = sample_gumbel_noise(shape=(k,L))
    y_soft = softmax(logits + gumbel )
    // Straight-through: hard forward, soft backward
    y_hard = one_hot(argmax(y_soft))
    y_st = detach(y_hard - y_soft) + y_soft
    // Apply selection and return
    RETURN einsum('kl,lbd->kbd', y_st, transpose(sequence))

W4. Clarification on "Junk Regions" and SeqPack's Treatment of Non-coding DNA

We appreciate the reviewer's concern regarding our terminology and its implications for non-coding regions. To clarify, our use of "junk regions" specifically refers to repetitive sequences and regions with no currently identified functional significance, not regulatory elements or other functionally important non-coding DNA. As explicitly stated in line 75, regulatory elements are not categorized as junk regions in our framework. Importantly, SeqPack does not impose any algorithmic bias against non-coding regions. The algorithm operates in a task-agnostic manner during the packing process, preserving sequence information regardless of coding status. During task-specific fine-tuning, the model learns to identify and weight functionally relevant elements—including those in non-coding regions—based on their importance for the particular downstream task. This approach allows Omni-DNA to maintain sensitivity to regulatory elements, enhancer sequences, and other functional non-coding elements when they are relevant to the prediction task. We acknowledge that our terminology could be clearer and will revise it to avoid potential misinterpretation.

W5. Writing (Lines 31–40)

Thank you for the suggestion. We have revised lines 31–40 to improve clarity, concision, and flow. For the full updated rationale, please see our response to Reviewer MarC (Q10).

[1] Anagnostidis, Sotiris, et al. "Dynamic context pruning for efficient and interpretable autoregressive transformers." NeurIPS 2023

[2] Dao, Tri, et al. "Transformers are ssms: Generalized models and efficient algorithms through structured state space duality." ICML 2024

[3] Gu, Albert, et al. "Mamba: Linear-time sequence modeling with selective state spaces." COLM 2024.

[4] Waleffe, Roger, et al. "An empirical study of mamba-based language models." arXiv 2024.

[5] de Almeida, Bernardo P., et al. "A multimodal conversational agent for DNA, RNA and protein tasks." Nature Machine Intelligence 2025

We hope you are well. We are gently following up on our revised submission. We appreciated your detailed feedback and have incorporated several updates in response:

• Clarified that SeqPack is optional during fine-tuning/inference and incurs no loss when set to 1×.
• Revised the language around “junk” DNA regions to better emphasize sensitivity to non-coding and regulatory sequences.
• Added the ChatNT baseline for the sequence-to-function task.
• Clarification on Section 3.1.1

We would welcome any further feedback you might have. An updated assessment before the discussion period closes would be greatly appreciated. We are happy to provide more details on any of these points.

We thank the reviewer for carefully considering our point-by-point response and for the upward adjustment of the score! Regarding the removal of the junk regions, please let us clarify a bit. There is no prior hard removal in SEQPACK. Instead, it used a dynamic learning approach to decide which regions are important based on the task-specific learning target. In other words, SEQPACK compresses unrelated regions based on the training annotations. As shown by the code snippet in Clarification for Section 3.1.1, weights parameter is learned through the fine-tuning process and back-propagation.

As for the model size, Omni-DNA generally shows superior performance across tasks with fewer parameters, compared to transformer-based models such as DNABERT2 and NTv2. The state space models are more parameter-efficient, but they are data-demanding, as detailed in the study [1]. We believe Omni-DNA does a good trade-off between performance and data/computing resources.

[1] Waleffe, Roger, et al. "An empirical study of mamba-based language models." arXiv 2024.

We appreciated for your feedback. We have provided additional experiment results in response to your questions, which include further ablations on pretraining scheme and data to the downstream model performance and an comparison to the other sequence pruning methods. In the following, we also discuss the quality of SEQ2FUNC dataset and the significance of multi-tasking.

Q1. Does the performance gain of Omni-DNA come from pretraining recipe, data or optimized backbone?

We acknowledge that our holistic framework approach made it challenging to isolate individual contribution sources. To address this concern and provide the requested ablation analysis to disentangle the key factors contributing to Omni-DNA's performance. Our original paper included ablations for architectural choices (model size, normalization layers, and positional encoding). To isolate the remaining factors you identified, we designed two controlled experimental sets using the 116M-parameter Omni-DNA.

Experiment 1: Pre-training Objective Analysis

• Masked Language Modeling (MLM) using HuggingFace's default flow and configurations [7]
• Next-Token Prediction (NTP) using our original training framework
• This directly isolates the impact of our pre-training recipe choice

Experiment 2: Corpus Scaling Analysis

• Systematic evaluation across 30B, 60B, 90B, and 120B nucleotides
• This quantifies the contribution of dataset scale independent of other factors

These controlled ablations reveal two findings: (1) For Promoter and Enhancer Classification tasks NTP consistently outperforms MLM across all data scales, but for splice classification MLM wins (2) Performance scales with data size; This indicates that Omni-DNA's performance is potentially attributed to synergy of the pre-training recipe, architecture and data scale.

Q2. SEQPACK and other token pruning methods used in NLP

We acknowledge there are existing token pruning methods in NLP. However, SEQPACK's unique way of compressing the input sequence with geometric splitting schema is not used by existing token pruning methods in NLP. Below, we highlight these factors and provide additional experimental results. We will add these into a new Appendix section in our revised paper later to illustrate key differences.

Key Differentiating Factors: SEQPACK operates at significantly higher compression ratios (c = $\frac{L_{original}}{L_{pruned}}$) than typical NLP pruning methods. In the Enhancer-Target Prediction task, SEQPACK achieves compression ratios of c = 100-220, whereas most existing pruning methods (e.g., LazyLLM[1], Dynamic Context Pruning in NeurIPS 2023[2]) typically operate at $c = 4$ to $5$ in their original NLP applications. Additionally, many existing token pruning methods are designed specifically for generation tasks (i.e., decoding)[3][4] rather than sequence understanding (i.e., the full-sequence encoding), limiting their applicability to our genomic prediction benchmarks.

To directly assess SEQPACK's effectiveness against established methods, we adapted Dynamic Context Pruning (DCP) to work with Omni-DNA 116M on the Enhancer-Target Prediction task. The results demonstrate SEQPACK's substantial advantages:

At extreme compression ratios (220×, 150×), DCP performance degrades severely, barely outperforming random baselines, while SEQPACK maintains strong performance (AUROC > 0.89). Even at moderate compression (50×), SEQPACK outperforms DCP (0.907 vs 0.880 AUROC). These results highlight SEQPACK's unique capability to maintain genomic sequence understanding at compression levels that render existing NLP methods ineffective.

Q3. Quality of LLM-based Dataset

We appreciate the concern about training on LLM-generated text. In our pipeline, the LLM is not asked to infer function from a sequence. Instead, for each record we first retrieve structured fields from NCBI (e.g., molecule type and canonical gene/product names). The LLM is then constrained to rephrase those fields into a readable description to minimize the hallucination. An example is shown below.

Examples:

NCBI Annotation: mRNA, regulator of G-protein signaling 20

LLM Annotated: RGS20 mRNA encodes a regulator of G-protein signaling that acts as a GTPase-activating protein (GAP) for Gα subunits—especially Gαz and Gαi—thereby accelerating GTP hydrolysis and terminating GPCR signaling (dampening pathways such as μ-opioid receptor signaling).

Quality control includes a dual-pass agreement check (two independent generations must match on key entities). These controls aim to ensure the rewrite remains a faithful, human-readable rendering of the underlying NCBI fields rather than a speculative annotation. For the test split, we recruited a genomic domain expert to examine 10% of the dataset (around 2700 records) to cross-validate on the quality, and each of the selected records does not have hallucination.

LLM-based weak/synthetic labeling—now routine in NLP [5], multimodal learning [6], and clinical AI [7] —can provide reliable, performance-boosting supervision when properly constrained and filtered. We extend this to genomics by converting structured database fields into readable supervision to train task-ready models at scale.

Q4. The Novelty of the Multi-Tasking

While we acknowledge that multi-task learning techniques are established in NLP, our contribution lies in demonstrating their unexplored potential in genomic foundation models.

Key Contribution: Our work reveals that multi-task learning can achieve remarkably high performance on standard genomic benchmarks—a finding not systematically explored in prior genomic foundation model research. This identification of multi-tasking's effectiveness in genomics represents an important empirical contribution to the field. Applying mature techniques to a new domain is valuable when it yields new evidence and usable practice. Recent work in genomics (e.g., AlphaGenomep[8]) likewise achieves strong results by emphasizing training strategy over architectural novelty. Our experiments reinforce this trend: careful multi-tasking—not a new model class—drives the improvements, offering the community actionable guidance for more effective GFMs.

Q5. "limitation is the immense complexity ... difficult for other researchers to reproduce"

We acknowledge the concern about reproducibility and have prioritized accessibility in our implementation. To ensure broad adoption and ease of use, we provide comprehensive resources that integrate seamlessly with mainstream frameworks.

Accessibility Features: All components of our framework—including pretrained models, SEQPACK implementation, and training code—are fully compatible with standard libraries such as Hugging Face Transformers[9] and TRL[10]. This represents a significant advantage over existing DNA foundation models that often require substantial adaptation or custom implementations.

Ease of Reproduction: Researchers can reproduce our results or build upon our work with minimal setup. The model can be loaded and fine-tuned for sequence understanding tasks with just a few lines of standard code, or directly used as a DNA generation model without modification. This streamlined approach removes typical barriers to entry that have limited adoption of genomic foundation models. As detailed in our supplementary materials, we provide the complete codebase. If accepted, the codebase would be hosted as an open-source repository for full reproducibility, and we will keep track of the issues. And we will release all pretrained model weights, and detailed documentation.

[1] Lazyllm: Dynamic token pruning for efficient long context llm inference. arXiv 2024.

[2] Dynamic context pruning for efficient and interpretable autoregressive transformers. NeurIPS 2023.

[3] Saliency-driven dynamic token pruning for large language models. arXiv 2025.

[4] H2o: Heavy-hitter oracle for efficient generative inference of large language models. NeurIPS 2024.

[5] Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv 2022.

[6] Visual Instruction Tuning. arXiv 2023.

[7] Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework. JMIR Medical Informatics 2025

[8] AlphaGenome: advancing regulatory variant effect prediction with a unified DNA sequence model. bioRxiv 2025.

[9] Huggingface's transformers: State-of-the-art natural language processing. arXiv 2019

[10] TRL: Transformer Reinforcement Learning

Thank you again for your constructive feedback on our submission. We're writing to follow up with the rebuttal, in which we aimed to address your points directly. The key updates include:

• Controlled ablations on the pretraining objective and corpus scale.
• Head-to-head SEQPACK vs. DCP results, particularly at high compression ratios.
• Clarifications on the quality controls used for SEQ2FUNC.
• Clearer framing of the novelty and reproducibility of our multi-tasking approach.

As the discussion deadline is approaching, please let us know if anything remains unclear; we are on standby to clarify. Many thanks for your time.