The Genomics Long-Range Benchmark: Advancing DNA Language Models

We thank the reviewers for their comments and suggestions and for recognizing the utility of our benchmark and analyses. We address the concerns and questions in detail below.

Concern 1: Why do certain DNA LMs perform well on specific tasks?

The models we test vary along several axes, including context length, parameter count, length of pre-training, architecture, and tokenization. While teasing apart the effect of each of these factors would fall outside the scope of this work, some of the factors we believe are driving the results are model capacity (e.g. the NTv2 with 500M parameters performs best among all the models within the NTv2 family) and the scope of pre-training. DNABERT and HyenaDNA were both pre-trained for 200-260B tokens, whereas NTv2 was pre-trained for 900B tokens. This disparity is made even larger by the fact that NTv2 uses k-mer tokenization. Overall we find that NTv2 is the best performing LM (even looking at the smaller model sizes for NTv2 in Tables 11 and 12). The scope of the pre-training is potentially a significant driver of this.

Concern 2: Size mismatch (e.g., between NTv2 and other models) provides a potentially unfair comparison

For each of the family of DNA LM (i.e., DNABERT, NTv2, and HyenaDNA) we evaluated several variations of each model. The full results are presented in Tables 11 and 12 in the appendix. Due to space constraints, in the versions of these tables for the main text, we only report the best model from within each family of models. Although the largest NTv2 model is significantly bigger than some of the other models in the benchmark, our goal was to provide a robust accounting of important works in this field, and so we used the available models that have been pre-trained and made available to the community. Note, we did not re-pre-train any of these models, but rather only used existing published weights.

Concern 3: Caduceus and Evo need to be added to evaluations.

This is a great suggestion. We are actively working on getting these results for Caduceus (we provide the initial results below and will update here once the full set is available). We also plan to add this model to the TSS context analysis in Figure 2. For the Evo model, given it was pre-trained on prokaryotic and phage genomic sequences and is a substantially larger model than any of the ones we have run for the current benchmark, we have restricted results to the zero-shot prediction tasks.

Pre-trained Caduceus results

Evo results

Concern 3: Included relevant benchmarks like GenBench [4] and BEACON [5] to improve the coverage of related literature.

Thank you for this suggestion. Below we include a discussion of of GenBench and BEACON that we will add to our revised manuscript:

GenBench: This suite is composed of 43 different datasets split between “short” and “long” range tasks, where long-range tasks are defined by having a sequence length of greater than 1000 base pairs. The tasks in GenBench, spanning multiple species, are primarily binary, sequence level classification tasks but also include multi-class classification and regression tasks. The authors evaluate six different genomic language models covering both attention and convolution-based architectures. While GenBench provides a comprehensive evaluation, it lacks critical tasks like variant effect prediction in non-coding regions and zero-shot evaluations. It also omits comparisons to long-context models like Enformer and is limited in its evaluation of long-range tasks, with the longest sequence length capped at 30,000 base pairs.

BEACON: This benchmark introduces the first unified evaluation framework for RNA modeling, encompassing 13 tasks across structural analysis, functional studies, and engineering applications. It evaluates 29 models, ranging from pre-trained RNA language models to naive supervised models, and examines the influence of tokenization strategies and positional embeddings on performance. While BEACON is a valuable resource for assessing RNA-focused models, its scope is distinct from genomic benchmarks, as it targets RNA-specific tasks rather than genomic applications like regulatory element prediction, variant effect prediction, or gene expression prediction.

Concern 4: Typo in ProteinGym citation

Thank you for catching this. We’ve corrected it in our manuscript.

References:

Liu, Zicheng, et al. "GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models." arXiv preprint arXiv:2406.01627 (2024).

Notin, Pascal, et al. "Proteingym: Large-scale benchmarks for protein fitness prediction and design." Advances in Neural Information Processing Systems 36 (2024).

Nguyen, Eric, et al. "Sequence modeling and design from molecular to genome scale with Evo." Science 386.6723 (2024): eado9336.

Ren, Yuchen, et al. "BEACON: Benchmark for Comprehensive RNA Tasks and Language Models." arXiv preprint arXiv:2406.10391 (2024).

Schiff, Yair, et al. "Caduceus: Bi-directional equivariant long-range dna sequence modeling." arXiv preprint arXiv:2403.03234 (2024).

We clarify that all of the results presented in our original manuscript include mean $\pm$ standard deviation for test set performance from 5 separate fine-tuning runs. For each run, we use a different validation chromosome and do early stopping on the validation loss and then compute performance on the test set. Given the limited timeframe during the rebuttal period, the newly reported results from the rebuttal have not had 5 runs collected, but we are actively working on doing so. For our camera ready revision we will also have mean $\pm$ standard deviation for the new experiments conducted during the rebuttal period as well (using the same protocol described above).

We identify a number of specific factors affecting the performance of DNA LMs across tasks: model input context length, model size, training data (including data quality and number of tokens), and the expressivity of the architecture. We provide analysis and supporting experiments for each factor below, drawing from both our work as well as results reported in the literature.

Context Length

Context length generally improves DNA LM performance. For example, gene expression can be regulated by regions up to 100K bp away: accurately predicting gene expression requires the model to process inputs containing both the gene and its regulatory regions, i.e. a DNA input of up to 100K bp. Similar arguments can be made for variant classification tasks (e.g., identifying eQTLs).

Below, we report experiments showing that across architectures, context length increasingly improves performance on multiple tasks.

We’ve seen evidence for this claim in other works, as well. For example, in the Caduceus paper (Schiff et al. 2024) Figure 4, we see that a Caduceus model, despite being orders of magnitude smaller, with larger context size of 131k bps, outperforms a much larger NTv2 model on a version of the eQTL variant effect expression task. Numbers from that figure are reproduced below:

Model Size

As a rule of thumb, all else equal, larger models yield improved performance. Larger models are more expressive and fit the data better. Empirically, better fitting data also yields representations that perform well on downstream tasks. More formally, a good data fit means the model more accurately identifies conserved vs. non-conserved regions, a useful predictive feature that improves downstream performance. This is especially true for tasks where sequence conservation is an important feature, e.g., variant effect prediction.

In our benchmark, the NTv2 family of models generally performed best. Below we examine more closely how performance varies across model sizes for this family, comparing the 50M to 500M models (these numbers are taken from Tables 12 and 13 of our manuscript, where we see the larger model consistently outperform the smaller one on the downstream tasks:

Expressivity of the Architecture

Given a fixed number of parameters, certain DNA LM architectures represent a more expressive hypothesis class or have better inductive biases, which improves performance for a fixed model size and training budget. For example, attention can express more complex functional relationships than convolutions, resulting in improved performance on both language and genomics tasks. Other architectures incorporate inductive biases that would otherwise have to be learned from data, such as reverse complement (RC) equivariance, which improves performance and data efficiency.

Below, we report performance across convolutional, Hyena, and Mamba architectures controlled for model size and context length. More expressive Hyena and Mamba architectures (variants of RNNs) outperform simpler convolutions. Caduceus models further add bi-directionality, and RC equivariance; each step further improves performance over multiple tasks (results taken from Table 1 of Schiff et al. 2024; best values bolded, second best are italicized):

Training Data: Number of Tokens

For a fixed model and architecture, training the model longer on more tokens typically improves performance. This is primarily because longer training further minimizes the loss and improves the data fit. A better fit to the data yields better representations for reasons described above. Typically overfitting is not a problem as the pre-training datasets of most models are often so large that seeing each token more than 3-4 times is computationally intractable. For example, Figure 3 in Caduceus (Schiff et al. 2024) and Figure S4 in Evo (Nguyen et al. 2024) show that models’ pre-training loss on the test set continues to decrease even as training progresses.

Training Data: Quality

The quality of the training data can matter even more than its quantity. This is especially true for DNA LMs, given that a large percentage of many genomes consists of repetitive regions that have little or no functional role. Training on these regions may be at best a waste of computation, and at worst may bias the model towards certain less important repetitive regions of the genome at the expense of others. In both protein language models and natural language applications, sequence deduplication is a key step. We think that in DNA LMs, data pre-processing will be at least as important.

While many DNA LM (e.g., the Hyena and NT families) train on all genomic data, recent work (e.g., GPN; Benegas et al 2023) has reported significant performance improvements from sampling different genomic regions with different frequencies while keeping training budget constant. Similarly, recent PlantCaduceus (Zhai et al. 2024) models have observed significant performance improvements from training on a larger diversity of plant genomes. We think genomic data curation is an under-explored area that will significantly impact DNA LMs.

Fine-tuning Algorithms

Once a model is trained, it has to be used on downstream tasks, oftentimes by being fine-tuned on a small labeled dataset using gradient descent. Most works fix the output embeddings from the model and train a simple classifier on top of them. Alternatively, we may fine-tune the weights of the full model on the downstream tasks. The latter task yields a significantly more expressive hypothesis class for the supervised problem, and in our observations improves performance.

Below, we report experiments that demonstrate that across most tasks and models full fine-tuning yields significantly better results than freezing embeddings. We attribute the few cases where it doesn’t to the susceptibility of the full model to overfitting. We report the delta between full-finetuning and freezing the backbone embeddings (results reproduced from Table 15 of our work):

Training Hyper-Parameters

Implementing training and fine-tuning also involves optimizing the model using gradient descent. This procedure is sensitive to hyper-parameters such as batch size and learning rate. We observed below that tuning these parameters has non-trivial effects on model performance, and requires careful consideration when applying a DNA LM.

References

Benegas, Gonzalo, Sanjit Singh Batra, and Yun S. Song. "DNA language models are powerful predictors of genome-wide variant effects." Proceedings of the National Academy of Sciences 120.44 (2023): e2311219120.

Nguyen, Eric, et al. "Sequence modeling and design from molecular to genome scale with Evo." Science 386.6723 (2024): eado9336.

Schiff, Yair, et al. "Caduceus: Bi-directional equivariant long-range dna sequence modeling." arXiv preprint arXiv:2403.03234 (2024).

Zhai, Jingjing, et al. "Cross-species modeling of plant genomes at single nucleotide resolution using

We thank the reviewer for their detailed feedback and for recognizing the novelty and importance of our work. Below we address the concerns and comments raised.

Concern 1: More analysis of dependence on context length is required.

This is a great suggestion. We have run ablations on the effect of sequence length by training supervised baselines on our “long-range” tasks with varying input context sizes.

1. CNN baseline (12 M param model with residual connections) that is trained in a supervised manner. The CNN is inspired by that used in GPN, Benegas et al. 2023, (with dilation removed).
2. Caduceus baseline (3.3 M param model) that is trained in a supervised manner to our evaluation. This model is trained from scratch on the datasets.

We present initial results below (note some runs for the Caduceus model have not completed.

While results are being collected, we do observe a positive association between context size and performance on these hypothesized long-range tasks for both architectures.

Details about the model / experiment : For the CNN, we use an 8 layer convolutional model with skip connections between layers and hidden dimension of 512. We use an input context of 2,048 base pairs. The same LR and batch size are used as for the DNA LM benchmarking, but since we train from scratch, we train the models for 10-20 epochs depending on the task (as opposed to 1-3 for the DNA LMs). For the Caduceus from scratch model, we use 8 layers and hidden dim 256 with input context size of 2,048 base pairs. The LR is set to 1e-4 with a linear warmup of 500 steps. We use the same number of epochs as when training the CNN baseline.

Concern 2: Add alignment-based model (namely GPN-MSA) to zero-shot variante effect task

We thank the reviewer for this suggestion as well. We have run GPN-MSA on the zero-shot tasks and are adding this model to our revised manuscript. The results for GPN-MSA are presented below (with CADD as a reference). We see that GPN-MSA outperforms all DNA LMs and is competitive than CADD, and the discussion regarding the importance of alignment and GPN-MSA as a useful baseline will be added to our updated manuscript.

Concern 3: Enformer is the wrong baseline for ClinVar task

This point is well taken. With the new GPN-MSA results that we will be reporting for the variant effect prediction tasks, we agree that removing Enformer as a baseline for this task is more appropriate and the combination of CADD and GPN-MSA can serve as strong watermarks against which to compare other DNA LMs.

Concern 4: Discussion of why some tasks are “short-range” is missing

This discussion is present in Appendix B. For each task that is deemed “short-range,” we include a discussion, similar to that for the “long-range” tasks in the main text, as to why we hypothesize long-context is less important in these settings. We originally opted to include the short-range discussion in the appendix due to page limit concerns.

Concern 5: Missense VEP should not be categorized as a “short-range” task

The reviewer raises an interesting question / discussion here. We will update our manuscript to reflect this hypothesis. Below we also report supporting evidence for the reviewer’s point by seeing the effect of context size on the zero-shot performance of 2 long range models for this task. We report results for HyenaDNA and Caduceus pre-trained models:

We thank the reviewer for their detailed feedback and for recognizing the value of the tasks we curate. Below we respond to the concerns and questions that the reviewer raised

Concern 1: Add length-dependence ablation experiments

This is a great suggestion. In response to this comment, we have run ablations on the effect of sequence length by training supervised baselines on our “long-range” tasks with varying input context sizes.

1. CNN baseline (12 M param model with residual connections) that is trained in a supervised manner. The CNN is inspired by that used in GPN, Benegas et al. 2023, (with dilation removed).
2. Caduceus baseline (3.3 M param model) that is trained in a supervised manner to our evaluation. This model is trained from scratch on the datasets.

We present initial results below (note some runs for the Caduceus model have not completed.

While the results are still being collected, we do observe a positive association between context size and performance on these hypothesized long-range tasks for both architectures.

Details about the model / experiment : For the CNN, we use an 8 layer convolutional model with skip connections between layers and hidden dimension of 512. We use an input context of 2,048 base pairs. The same LR and batch size are used as for the DNA LM benchmarking, but since we train from scratch, we train the models for 10-20 epochs depending on the task (as opposed to 1-3 for the DNA LMs). For the Caduceus from scratch model, we use 8 layers and hidden dim 256 with input context size of 2,048 base pairs. The LR is set to 1e-4 with a linear warmup of 500 steps. We use the same number of epochs as when training the CNN baseline.

Concern 2: Emphasize human-centricity of the benchmark

We will rename our paper to be titled “The Human Genomics Long-Range Benchmark: Advancing DNA Language Models” and have added an explicit reference to the tasks being human genome to our abstract. The updated abstract reads as follows (changes highlighted in bold):

… Here, we present the Human Genomics Long-Range Benchmark (LRB), which focuses on biologically meaningful tasks and supports long-range contexts. We complement our benchmark with fine-tuning recipes that meaningfully improve performance and affect model evaluation. We evaluate DNA LMs across nine compiled human genome tasks…

The human genome focus of our benchmark is also already highlighted in the intro and is explicitly marked in Table 1 as well.

Question 1: How is sqrt(L) context length extension performed

We follow the implementation / algorithm from Rabe et al. 2021. Below are more details about this method which we will add to our Appendix:

The algorithm in Rabe et al. 2021 leverages a “lazy softmax” approach where key-value pairs are processed sequentially, maintaining only two vectors in memory: one for the accumulated weighted values and another for the cumulative sum of weights. This method significantly reduces memory usage by avoiding the storage of all pairwise attention scores. To optimize performance on modern hardware accelerators, which rely on parallelization for efficiency, the implementation processes attention in chunks. Rabe et al. (2021) empirically determined that using a chunk size of $\sqrt(L)$ strikes a balance between memory savings and computational overhead. Larger chunks increase memory requirements, while smaller chunks can lead to excessive re-computation of activations during the backward pass. Additionally, the implementation is numerically stable and functions as a drop-in replacement for the standard attention module, making it highly practical for tasks requiring extended context lengths.

Question 2: How are train/test splits generated?

We provide a detailed overview of train/test splits Appendix B, Table 6. For most tasks in the benchmark, train and test splits are performed by chromosome (following previous works that introduced these datasets, e.g., Enformer for the eQTL variant effect prediction task) to minimize sequence overlap and ensure that performance reflects the models’ ability to generalize to unseen genomic regions. The exception is the CAGE task, where the split was done randomly in order to better compare to the results in the original Enformer paper that followed a similar protocol.

Question 3: Add Caduceus (Schiff et al. 2024) and Evo (Nguyen et al. 2024) models to benchmark

This is a great suggestion. We are actively working on getting these results for Caduceus (we provide the initial results below and will update here once the full set is available). For the Evo model, given it was pre-trained on prokaryotic and phage genomic sequences and is a substantially larger model than any of the ones we have run for the current benchmark, we have restricted results to the zero-shot prediction tasks.

New Baseline results

Pre-trained Caduceus results

Evo results

References:

Benegas, Gonzalo, Sanjit Singh Batra, and Yun S. Song. "DNA language models are powerful predictors of genome-wide variant effects." Proceedings of the National Academy of Sciences 120.44 (2023): e2311219120.

Nguyen, Eric, et al. "Sequence modeling and design from molecular to genome scale with Evo." Science 386.6723 (2024): eado9336.

Rabe, Markus N., and Charles Staats. "Self-attention does not need $O (n^ 2)$ memory." arXiv preprint arXiv:2112.05682 (2021).

Schiff, Yair, et al. "Caduceus: Bi-directional equivariant long-range dna sequence modeling." arXiv preprint arXiv:2403.03234 (2024).

We thank the reviewer for their feedback and recognizing the novel aspects of our work. Below we respond to the concerns and questioned raised in detail

Concern 1: Adding more supervised baselines.

We have added two new baselines:

1. CNN baseline (12 M param model with residual connections) that is trained in a supervised manner. The CNN is inspired by that used in GPN, Benegas et al. 2023, (with dilation removed).
2. Caduceus baseline (3.3 M param model) that is trained in a supervised manner to our evaluation. This model is trained from scratch on the datasets.

We present initial results below (note some runs for the Caduceus model have not completed).

New Baseline results

On most tasks, other than the gene expression and chromatin features tasks, we find that this supervised baseline performs competitively with the strongest performing baseline from our benchmark results, even outperforming models on the task of promoter identification. These results underscore the fact that DNA LMs are still in the early stages of development before being mature enough to replace traditional supervised methods.

Details about the model / experiment : For the CNN, we use an 8 layer convolutional model with skip connections between layers and hidden dimension of 512. We use an input context of 2,048 base pairs. The same LR and batch size are used as for the DNA LM benchmarking, but since we train from scratch, we train the models for 10-20 epochs depending on the task (as opposed to 1-3 for the DNA LMs). For the Caduceus from scratch model, we use 8 layers and hidden dim 256 with input context size of 2,048 base pairs. The LR is set to 1e-4 with a linear warmup of 500 steps. We use the same number of epochs as when training the CNN baseline.

Concern 2: Adding statistical comparison

To the best of our knowledge, it is standard practice to report mean and standard deviations (e.g., a similar practice is seen in the BEACON reference provided by the reviewer). We are not aware of works that also perform statistical / hypothesis testing to distinguish benchmarked model performance. If the reviewer has any references or specific tests they had in mind, we will do our best to perform these in the coming days.

References

Benegas, Gonzalo, Sanjit Singh Batra, and Yun S. Song. "DNA language models are powerful predictors of genome-wide variant effects." Proceedings of the National Academy of Sciences 120.44 (2023): e2311219120.

Schiff, Yair, et al. "Caduceus: Bi-directional equivariant long-range dna sequence modeling." arXiv preprint arXiv:2403.03234 (2024).