Fast and Low-Cost Genomic Foundation Models via Outlier Removal

The updated manuscript can be accessed anonymously at link.

Reviewer's Comment: a detailed comparison to ...

Response: We thank the reviewer for the feedback and the opportunity to clarify our evaluation of outlier suppression techniques. We would like to highlight that we have already included a detailed comparison of GERM with alternative outlier suppression methods in Table 23 and Appendix E.8 of the supplementary material. We are happy to provide further clarification if needed.

Reviewer's Comment: Definition of the outlier…

Response: We appreciate the opportunity to clarify the definition of outliers in our work and how it differs from the classical statistical definition. A more detailed explanation is provided in the response to the reviewer AD8a.

Reviewer's Comment: The theoretical analysis in Appendix B assumes…

Response: Thank you for the helpful feedback. We acknowledge that our assumptions may not hold universally. However, we believe they remain practical in real-world GFMs for three reasons:

1. Weight matrices in large foundation models rarely become singular. Overparameterization and regularization typically prevent this.

2. The low-rank assumption may not hold in every scenario. Yet it helps us show the strong expressiveness of LoRA tuning transformers with outlier-free layers. Many theoretical analyses use similar assumptions, so they form a reasonable setup.

3. Under these conditions, our results show that LoRA-tuned outlier-free transformers match or exceed the expressiveness of softmax-based architectures. These assumptions may not apply everywhere, but they do not undermine our theoretical insights.

Reviewer's Comment: LoRA Experiments...

Response: We appreciate the reviewer’s suggestion to disentangle the effects of LoRA and quantization for a more granular analysis. In our revised manuscript, we add additional experiments and analysis for LoRA+Quantization in Table 24 and Appendix E.9.

Reviewer's Comment: larger models available?...

Response: Thank you for your advice to include larger models for a more comprehensive evaluation. In our study, we select NT2.5B as the largest model for the classification task because it represents the most suitable model architecture designed for genomic sequence classification. While larger models such as Evo and GeneOcean exist, they are fundamentally designed as generation models rather than classifiers. These models prioritize sequence generation capabilities rather than directly optimizing for classification accuracy. As a result, they differ significantly in architecture, objective function, and training strategy. So, making a direct comparison with GERM in a classification context is less appropriate.

Reviewer's Comment: Unclear practical impact of outlier metrics...

Response: Thank you for the comment. We clarify that the outlier metrics, such as kurtosis and max infinite norm, are empirically shown to correlate with model quantizability—that is, robustness to performance degradation under quantization [1,3]. Prior studies [2,4] demonstrate that outliers significantly reduce quantized model performance, and thus reducing these metrics has direct implications for improved performance of quantization.

[1] Bondarenko, et al. "Quantizable transformers"

[2] Wei, Xiuying, et al. "Outlier suppression"

[3] Chmiel, Brian, et al. "Robust quantization"

[4] Dettmers, Tim, et al. "Gpt3. int8 ()"

Reviewer's Comment: other GFMs should include in literature review...

Response: We appreciate the reviewer's suggestion to expand our literature review by including additional GFMs. In response, we incorporate Evo in the related work section of our revised version.

Reviewer's Comment: Incomplete discussion of limitations?...

Response: We thank the reviewer’s suggestions regarding the need for a more complete discussion of GERM-T’s limitations. We update the discussion of GERM-T's limitations in our revised manuscript, specifically in Appendix F.

The updated manuscript can be accessed anonymously at link.

Reviewer's Comment: does not clearly explain what an outlier is or how to quantify it...

Response: We thank the reviewer for the helpful comments and the opportunity to clarify our definition of outliers and provide theoretical analysis for their mitigation.

Definition of Outliers in Our Work

The definition of outliers in our work differs from the classical statistical definition. In traditional statistics and machine learning, outliers are typically defined as data points that fall outside the modeled distribution. In contrast, in our context, we define outliers as tokens or activations that disproportionately influence the attention mechanism, despite containing little or no meaningful information. A more detailed explanation is provided in the response to the reviewer AD8a.

Q2: How Does softmax_1 Mitigate Outliers?

In our paper, we introduce Softmax1 equation, a modified softmax function designed to mitigate the effects of outliers:

$$\text{Softmax1}(S)_i = \frac{\exp(S_i)}{1 + \sum_j \exp(S_j)}$$

Where $S = QK^\top / \sqrt{d}$ represents the scaled dot product in the attention mechanism.

Key Improvements in Softmax1:

1. Suppression of Low-Value Tokens:

Unlike standard softmax, which assigns non-zero probabilities to all tokens — even those with highly negative scores — Softmax1 allows low-information tokens to receive near-zero probabilities. This behavior is crucial in genomic models where repetitive sequences or spacer elements resemble no-op tokens.

2. Controlled Attention Distribution:

Softmax1 suppresses the broadening of the attention distribution, ensuring that the model remains focused on biologically relevant regions rather than noisy patterns.

---

Theoretical Support for Softmax1’s Outlier Resistance

• Standard Softmax Behavior:

Standard softmax assigns non-zero probabilities to all tokens, even those that ideally should receive no attention. In extreme cases:

$$\lim_{x_1 \to -\infty} \ldots \lim_{x_k \to -\infty} \text{Softmax}(x)_i = \frac{1}{k} > 0$$

• Softmax1 Behavior:

In contrast, Softmax1 ensures that tokens with highly negative scores are assigned probabilities that collapse to zero:

$$\lim_{x_1 \to -\infty} \ldots \lim_{x_k \to -\infty} \text{Softmax1}(x)_i = 0$$

This limiting behavior ensures that low-information tokens, such as repetitive motifs or spacer tokens, are effectively ignored, stabilizing the attention mechanism.

---

Empirical Evidence for Outlier Mitigation

To validate that Softmax1 effectively mitigates outliers, we provide both quantitative and qualitative evidence:

• Quantitative Evidence:

As shown in Table 1, GERM achieves a 92.14% reduction in kurtosis and an 82.77% reduction in the maximum infinity norm compared to DNABERT-2. These reductions demonstrate that GERM effectively suppresses extreme values associated with outliers.

• Qualitative Evidence (Attention Distribution Plots):

Visualizations in Appendix E.10 in revision illustrate that DNABERT-2 exhibits sharp, irregular attention spikes corresponding to outlier tokens, while GERM maintains a smoother, more stable attention distribution.

---

Conclusion

We appreciate the opportunity to clarify our definition of outliers and their impact on genomic foundation models. Our theoretical analysis highlights how the attention mechanism illustrates the formation of outliers, while the Softmax1 equation mitigates their influence by reducing the amplification of low-information tokens. The combination of improved theoretical design and strong empirical evidence reinforces GERM’s ability to suppress outliers, enhancing its stability and performance in genomic modeling tasks.

The updated manuscript can be accessed anonymously at link.

Reviewer's Comment: what would lead to an outlier in a GFM...

Response: Thank you for your insightful question about the nature of outliers in genomic foundation models. In our work, we define outliers as tokens or activations that disproportionately influence the attention mechanism, despite containing little or no meaningful information. These outliers emerge when softmax amplifies the attention probabilities of tokens that ideally should receive minimal or zero focus. We use the following attention mechanism to analyze this behavior:

$$\text{Output} = \text{Residual} \left( \text{Softmax} \left( \frac{Q(K)^\top}{\sqrt{d}} \right) V + X \right)$$

As shown in (Hu et al., 2024), if the attention input $X$ already contains sufficient information, the attention mechanism within the residual connection should ideally behave like an identity transform, producing near-zero attention outputs:

$$\text{Softmax} \left( \frac{QK^\top}{\sqrt{d}} \right) V \approx 0$$

In such cases, tokens with high values in $V$ — which may represent biologically significant features — should still receive near-zero attention probabilities.

Why Classic Softmax Fails

The problem arises from how the softmax function normalizes probabilities. Softmax enforces that probabilities sum to 1, which inherently magnifies the attention probabilities assigned to low-value tokens. This unwanted amplification broadens the attention score distribution and introduces outliers — tokens that exert a disproportionate influence despite their low information value. Genomic models such as DNABERT-2 face a critical challenge from outliers - including repetitive elements, low-complexity regions, and non-coding segments - similar to no-op tokens. Despite their limited biological significance, these regions receive disproportionately high attention weights through standard softmax operations, consequently suppressing the model's focus on biologically relevant genomic features.

Intuition Behind Outliers in Genomic Models

Outliers in genomic models typically arise from sequence patterns that produce anomalous query-key interactions in the attention mechanism. Though genomic data lacks traditional "words" like language models, certain biological patterns produce similar effects. Key examples include:

1. Low-Complexity Regions (e.g., Poly-A or Poly-T Sequences)

Genomic sequences frequently contain repetitive base patterns (e.g., AAAAA..., TTTTT...). These sequences carry minimal unique information yet can produce large, uniform dot-product values in the attention mechanism. This causes softmax to assign exaggerated probabilities to these low-information tokens, effectively making them outliers.

2. Repetitive Motifs and Tandem Repeats

Genomic repetitive elements such as microsatellites and tandem repeats, which contain patterns (e.g., (CA)n, (GAA)n repeats) that generate artificially inflated attention scores due to their inherent self-similarity. However, such regions lack corresponding biological information, often resulting in softmax overemphasizing them as if they were biologically significant.

3. Boundary and Spacer Elements (e.g., Alignment Padding or Non-coding Spacer Sequences)

In genomic datasets, artificial padding sequences, non-coding segments, or spacer sequences are sometimes introduced to ensure proper sequence alignment. These tokens are intended to have no biological relevance, yet softmax’s behavior inadvertently amplifies their attention scores, creating noise that distorts meaningful patterns.

Impact on Genomic Analysis

In genomic foundation models like DNABERT-2, these outliers negatively impact performance by:

• Increasing Error Rates: Outliers divert attention away from biologically meaningful regions, reducing prediction accuracy for tasks like mutation site identification.
• Destabilizing Fine-Tuning: During fine-tuning, excessive focus on low-information tokens increases noise in gradient updates, limiting convergence stability.
• Masking Important Features: Outliers may overshadow rare but critical genomic patterns, reducing the model’s capacity to detect subtle but meaningful biological signals.

Reviewer's Comment:.One possible criticism is….

Response: We appreciate the reviewer’s feedback on the importance of innovation. We also appreciate the reviewer's comments, acknowledging that our paper makes a solid contribution as an empirical study that others can build upon. This paper presents experimental evidence demonstrating that outlier-free methods offer an efficient solution for improving low-rank adaptation and quantization. We restate our contribution in the response to the reviewer sdac.

The updated manuscript can be accessed anonymously at link.

Reviewer's Comment: However, the motivation ….

Response: We thank the reviewer for suggesting a direct attention score analysis to validate the outlier hypothesis. To address this point, we add detailed visualizations of the attention score distributions in Figure 3 and Appendix E.10, where we compare DNABERT-2 and GERM. The results demonstrate that GERM suppresses attention outliers, leading to more focused and efficient attention patterns. This analysis provides direct evidence supporting our hypothesis. We clarify this connection in the revised manuscript and invite the reviewer to examine Figure 3 and Appendix E.10 for a detailed analysis of these findings.

Reviewer's Comment: While statistics like ..

Response: We sincerely appreciate the reviewer’s insightful suggestion about the need to more rigorously establish the connection between attention outliers and downstream performance degradation. In response, we have added detailed attention score distribution visualizations in Appendix E.10 and Figure 3, including per-layer heatmaps comparing DNABERT-2 and GERM. For a formal definition of “outliers”, we respectfully refer the reviewer to our response to Reviewer AD8a. We also discuss in our response to Reviewer 8UsM how existing literature has demonstrated the negative impact of attention outliers on model performance. We believe these additions strengthen the theoretical and empirical motivation of our work and clarify the causal relationship between attention outliers and performance.

Reviewer's Comment: Limited novelty…

Response: We appreciate the reviewer’s feedback on the importance of novelty. While our approach builds upon established methodologies, we would like to highlight our key innovation: we are the first to integrate outlier removal to simultaneously enable (1) robust quantization and (2) accelerated low-rank adaptation for genomic foundation models. This contribution is important because genomic data presents unique challenges—such as extreme sparsity, high variability, and frequent outliers in attention mechanisms—which differ significantly from those in traditional NLP.

1. First to achieve accelerated LoRA for Genomic Models via Systematic Outlier Removal: Our work pioneers the use of LoRA in genomic foundation models like DNABERT-2. Applying LoRA directly to DNABERT-2 results in performance degradation due to genomic data-specific outliers. This novel integration of the outlier-free Hopfield mechanism enables effective low-rank adaptation alongside robust quantization, achieving a 37.98% improvement in fine-tuning performance compared to DNABERT-2.
2. Adapting Techniques for Genomic Challenges: The genomic domain demands unique adaptations due to sparse and highly variable tokenization methods like k-mer and BPE. The outlier-free Hopfield layer required significant adjustments to mitigate domain-specific outliers, reducing kurtosis and infinity norm values by 92.14% and 82.77%, respectively, across 27 genomic datasets. This ensures both robust quantization and efficient fine-tuning in resource-constrained settings.
3. GERM-T for Continual Learning: Beyond LoRA, GERM-T introduces a novel continual learning strategy that avoids training from scratch while effectively leveraging outlier-free layers. This approach focuses on resource-constrained genomic research, making adaptable fine-tuning without compromising performance possible.
4. Empirical Validation: Our work provides the first empirical evaluation of integrating outlier mitigation into LoRA fine-tuning and quantization for genomic models, achieving a 64.34% improvement in quantization robustness compared to DNABERT-2. These results validate the effectiveness of our proposed modifications and their impact on genomic tasks.

By adapting and extending these methods, we address domain-specific challenges while advancing genomic modeling. We hope this response clarifies the novelty and significance of our contributions, and we are happy to provide further details or analyses if needed.

Reviewer's Comment: What is the operational definition …

Response: We appreciate the opportunity to provide more details about the significance of outliers in transformer-based models. A more detailed explanation is provided in the response to the reviewer AD8a.

Reviewer's Comment:.Further reference ….

Response: We appreciate the reviewer's suggestion to expand our literature review by including additional GFMs. In response, we incorporate Evo in the related work section of our revised version.