dnaGrinder: a lightweight and high-capacity genomic foundation model

Major concerns

• The biggest question is how well this foundation model performs its primary function: predicting masked tokens. It seems possible that these models are simply memorizing the genome, which, relatively speaking, is not that large. The authors mention that (line 1,000), “For instance, after achieving 60% accuracy on chromosome 1, the model performed poorly on other chromosomes with the same MLM task.” Carefully assessing the model's performance on held-out data, at least on held-out chromosomes, is key. Although there are many homologous sequences across chromosomes, this approach is a good start.
• In reviewing the performance metrics (e.g., Table 3), it’s unclear how meaningful these differences are or whether they will yield any practical biological outcomes. There are numerous new techniques, architectures, and paradigms that improve foundation models in one way or another. A clear objective is necessary for developing a foundation model rather than simply applying different “module” combinations.
• Comparing performance metrics between two methods cannot rely solely on point estimates. Providing a measure of uncertainty, such as confidence intervals or standard deviations, would help. Since all these methods run on the same test datasets, using paired tests like paired bootstrapping would be even better.
• There’s some uncertainty around excluding more than half of the genome (repeats). Although many repeats are simple, they are vastly heterogeneous; many are actively silenced and regulated and play crucial roles in development and disease.

Minor Points

• The sentence on line 49 requires citations for each application.
• In Fig. 1b, it would be helpful if sequences before tokenization were numbered or labeled, or if arrows were added to indicate that sequences are reordered, not padded.
• Lines 326–330 are unclear: Is the proposed model further pre-trained using 0.41B tokens or 0.176B tokens?

The following points are my major concerns regarding the highlighted contributions in the paper:

• As a method development paper, the algorithmic and engineering innovations presented are minimal. While the authors position this work as introducing a novel lightweight genomic foundation model, the primary architectural differences from DNABERT-2 are:

  • Flash Attention 2 replacing Flash Attention Triton,
  • ALiBi positional encoding instead of RoPE,
  • and GEGLU activation in place of SwiGLU.

  These modifications are sensible engineering adjustments for language model development but do not constitute genuine innovations.

• While Sequence Length Warmup (SLW) has shown advantages in training decoder-only models by introducing progressively longer, more complex sequences akin to curriculum learning, its relevance for encoder-only genomic models is uncertain. Given that dnaGrinder’s tokenized sequences vary only moderately in length (700-2300 tokens, as reported), simpler alternatives—such as truncating sequences to a max_token_length or using balanced sampling (e.g., LengthGroupedSampler) for length-consistent batches—could likely achieve similar benefits.

• The introduction of ME-BPE in dnaGrinder seems aimed at solving a memory efficiency problem that may have limited relevance in practical genomic modeling. While training the BPE tokenizer can indeed be memory intensive, applying it during pretraining and inference requires minimal memory overhead. Even if a system lacks the memory to load the entire corpus, cloud vendors provide high-memory instances at reasonable costs, making this less of a barrier. Moreover, in NLP, tokenizers are often trained on a representative subset of the corpus (using options like --input_sentence_size=<size> and --shuffle_input_sentence=true in sentencepiece), rather than the full dataset used for language model training. This subsampling approach effectively reduces memory requirements without compromising vocabulary quality or requiring the additional complexity of ME-BPE’s iterative adjustments. As a result, ME-BPE seems to address a problem that is not particularly impactful in real-world scenarios.

• The authors claim that this work is the first to apply repetitive sequence removal, showing a clear disregard for the existing literature. Numerous models have already addressed repetitive elements using various techniques: down-weighting (GPN [1], PlantCaduceus [2]), downsampling/removal (SpliceBERT [3], GPN-MSA [4], hgT5 [5]), and fragment deduplication (gLM2 [6]). Notably, hgT5 also utilized RepeatMasker to identify repeats. The proposed method aligns with what should be considered standard practice, rather than an innovation.

Minor issues:

• The repetitive element removal method is introduced in the Human Reference Genome Dataset section, but the authors should clarify whether it was applied only to the human reference genome or to the entire combined dataset. If the latter is the case, this removal method would be better placed in the Methods section for consistency and clarity.

• Encoder-only models like dnaGrinder are non-causal and do not require KV caching for inference or embedding extraction. The authors' claim that Flash Attention 2’s inference optimization is "especially beneficial for our model...with most not exceeding 1,000 bp" is factually incorrect. In BERT-style models, inference is achieved through a single forward pass without the need for KV caching, which is specifically beneficial for autoregressive generation—a context irrelevant to dnaGrinder. Thus, the supposed inference advantage of Flash Attention 2 does not apply here.

• Some suggested improvements to the paper:

  • In Figure 1a, the plot displays two masked token embeddings, yet the tokens in “ME-BPE Tokenization” are not masked. Aligning these would improve clarity.
  • In the last paragraph of Section 2.2.2, Further Pretraining, the phrase “our model performed 100,000 steps” should be corrected to “DNABERT-2 model performed 100,000 steps.”
  • DNABERT-2 should be recognized as the state-of-the-art model in the benchmark alongside dnaGrinder, as it not only holds the highest combined number of top-1 and top-2 rankings but also has the highest average scores.
  • In the results table, the “Params” header is marked with a ↓ symbol, explained as “↓ indicates that a lower value is better.” However, the question of whether smaller or larger language models perform better for genomic modeling remains unresolved. It would be beneficial for the authors to discuss this open question.

References:

1. Benegas, G., Batra, S. S., & Song, Y. S. (2023). DNA language models are powerful predictors of genome-wide variant effects. Proceedings of the National Academy of Sciences, 120(44), e2311219120.
2. Zhai, J., Gokaslan, A., Schiff, Y., Berthel, A., Liu, Z. Y., Miller, Z. R., ... & Kuleshov, V. (2024). Cross-species modeling of plant genomes at single nucleotide resolution using a pre-trained DNA language model. bioRxiv, 2024-06.
3. Chen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., & Yang, Y. (2024). Self-supervised learning on millions of primary RNA sequences from 72 vertebrates improves sequence-based RNA splicing prediction. Briefings in Bioinformatics, 25(3), bbae163.
4. Benegas, G., Albors, C., Aw, A. J., Ye, C., & Song, Y. S. (2023). GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction. bioRxiv.
5. Ioannidis, N. (2024, March). GUANinE v1. 0: Benchmark Datasets for Genomic AI Sequence-to-Function Models. In Machine Learning in Computational Biology (pp. 250-266). PMLR.
6. Cornman, A., West-Roberts, J., Camargo, A. P., Roux, S., Beracochea, M., Mirdita, M., ... & Hwang, Y. (2024). The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling. bioRxiv, 2024-08.
7. Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., & Boomsma, W. (2023, November). Bend: Benchmarking dna language models on biologically meaningful tasks. In The Twelfth International Conference on Learning Representations.

Conceptually, the proposed points of novelty are mostly incremental advancements. Therefore, it is important to provide convincing empirical evidence. While the model shows promising performance on the benchmarks, it is unclear how much each modification contributes to the improvement. Ablation studies should be conducted to analyze the individual contributions of each component, which is vital to support the claims. More specifically:

a. Flash attention 2: This seems like a minor contribution, especially since DNABERT-2 already integrated flash attention.

b. Sequence length warm-up: HyenaDNA used this technique. Why is being the first encoder-only model to use it considered notable? While adapting it for variable-length BPE-tokenized sequences is a nice idea, an ablation study would be more convincing to show how much performance gain comes from this technique.

c. ME-BPE: This appears to be a minor modification of BPE, which is already used by DNABERT-2. There is also no assessment of how much memory this strategy actually saves.

d. Population data augmentation: it appears to be a small modification to the NT 1kG dataset. Though I acknowledge it to be a unique contribution. Again, it is unclear how much of the performance gain is attributed to this modification.

1. The work seems to lack substantial novelty on the method design. Since most modules incorporated are previously proposed (e.g. the Flash Attention, SLW and ALiBi).
2. While dnaGrinder demonstrates efficiency improvements, it performance on some tasks (e.g. Epigenetic Marks Prediction, Core Promoter Detection and Promoter Detection in Table 3) is not competitive enough.
3. Given the work relies on multiple components described in the paper, it's better to have an ablation study to illustrate the impact of each component.