Diverse Genomic Embedding Benchmark for Functional Evaluation Across the Tree of Life

Thank you for your thoughtful review and suggestions. We appreciate you deeming DGEB to "raises the current standard in the genomics space" and our findings to "reveal interesting empirical evidence on the presence of scaling laws and performance gaps between existing AA and NA models"

I am concerned that the overall claims of the paper are too large in its current state.

Thank you for raising this. We acknowledge that our DGEB benchmarking suite is not comprehensive of all aspects of biological function. We have included a limitations section to highlight areas where DGEB currently does not support. In addition, in the introduction we add this following sentence highlighting where DGEB fits into the existing Benchmark landscape. "DGEB focuses on evaluating the representations of higher-order functional and evolutionary relationships of genomic elements, and is designed to complement existing benchmarks that focus on residue-level representations (Notin et al., (2023), Marin et al., (2024))."

Please see the added limitation section pasted below: "DGEB includes multiple zero-shot tasks, as ground-truth labels for biological function are sparse and biased. These tasks rely on embedding geometry to evaluate model performance. The assumption that models capturing important features of biological function have geometry directly matching the given tasks is not guaranteed. Future research could explore methods for identifying and leveraging relevant subspaces within model embeddings. For the EDS task, we acknowledge the limitation of Euclidean embeddings for representing phylogenetic tree structures and the possibility that certain regions of the phylogeny may be of low confidence (due to the inherent uncertainty in reconstructing the ground-truth phylogeny). However, this task provides a useful starting point for comparing model performance, and will be important for evaluating novel hyperbolic architectures, baselined by Euclidean embedding model results. While DGEB is designed to support both NA and AA models, current suite is biased towards coding sequences with only four tasks targeting non-coding elements (16S Arch, 16S Bac, 16S Euk, Ecoli RNA), limiting ability to evaluate NA model representations of regulatory elements (e.g. promoters, transcription binding sites). Furthermore, DGEB’s current evaluation suite focuses on single-element, inter-element, and multi-element scales of representations, and is designed to complement existing benchmarks that focus on residue-level representations (e.g. mutational effects Notin et al. (2023) Marin et al. (2024))."

I believe the authors are well positioned to further increase the relevance of their work by inclusion of baselines that are not derived from a foundation model.

Thank you for this suggestion. We have added a one-hot baseline, where the sequence is represented as one-hot vector per residue, and mean-pooled across the sequence dimension like all benchmarked models. The baseline results are shown in Figure 3 as dotted lines.

The contribution of the paper could be strengthened by benchmarking more gLM models.

We have added the DNABERT-2 model as an additional gLM model. Our benchmark now includes all commonly used gLM models referenced in [1] (Evo is a newer version of HyenaDna from the same author, trained on diverse genomic data, not just human).

Is there any way to provide a snapshot of the implementation via OpenReview or some other anonymized github repository?

We have added the anonymized code as a zip file in the OpenReview supplementary data.

The colour scheme for Figure 3's Model Type is not the most colourblind friendly.

Thank you for bringing this to our attention. We will update the colors in the camera-ready revision.

[1] https://www.biorxiv.org/content/10.1101/2024.08.16.608288v1

Thank you for the detailed responses. I believe the changes have improved the quality of the paper and I have raised my score. Overall, as the other reviewers also point out, adding rigour in the form of homology splitting etc. is an important benchmarking requirement that this paper could help establish in the genomics community.

I would like to re-iterate the sentiment shared in the original review and by other reviewers that extending this benchmark to evaluate models other than pre-trained foundation models is highly relevant to the larger community. While I acknowledge that papers have a finite scope, it is important to understand whether these pre-trained foundation models confer any advantage in the selected tasks compared to supervised deep learning on the labels from scratch. This point is reinforced by the surprising fact that NA models underperform a OHE in some tasks. Perhaps for the camera ready, the authors could add a supervised training benchmark using a simple sequence model (e.g. dilated CNN). Alternatively, I have anecdotally found that compressing the sequence using the Mamba architecture without any training at all (i.e. leveraging the structured state space initialization) is a strong naive baseline on genomic tasks.

Unfortunately, I must point out that the anonymous code is decidedly not anonymous and may violate double blind. I am unsure of how to resolve this and will defer to the AC.

For each of the datasets, it would be helpful to provide more detailed explanations of the intended learning objectives and their relevance.

We include the biological relevance of each dataset in section 3.3. For example: "BacArch BiGene: Similar to matching translated sentences between two languages, we curated functionally analogous pairs of sequences in a bacterial genome (Escherichia coli K-12) and an archaeal genome (Sulfolobus acidocaldarius DSM 639 ASM1228v1). ModBC BiGene: Identifying interacting pairs of ModB and ModC from sets of orthologs is a challenging task. ModB and ModC are interacting subunits of an ABC transporter. This dataset consists of pairs of ModB and ModC that are found to be interacting in the same genome. The goal is to correctly find the interacting ModC for each ModB given a set of orthologous ModC sequences (found in different genomes)."

What does the performance on these tasks/datasets reveal about the models tested?

The reviewer is correct in that "[o]ur results demonstrate that there is no single model that performs well across all tasks." This is similar to the conclusion drawn from analogous studies in text embedding benchmarks (e.g. MTEB (Massive Text Embedding Benchmark) [1]) where no single foundation model performs well across all tasks. DGEB, similar to MTEB, allows the ML practitioners to choose the most appropriate model for fine-tuning depending on the specific downstream task and modeling objectives. (e.g. for search capabilities, it would be more appropriate to fine-tune a foundation that performs well in retrieval tasks)

Given the large number of protein language models, what is the justification for benchmarking the models that were picked (over others such as AlphaFold, pLM-BLAST, etc)?

DGEB is an embedding benchmark that evaluates unsupervised biological sequence foundation models. AlphaFold is a structure prediction model that takes in an MSA and is supervised on structures. pLM-BLAST is a tool for remote homology detection built on ProtTrans (benchmarked in our study).

What measures are being taken to prevent data leakage in the tasks that have training data?

We conduct extensive sequence similarity based dereplication to prevent data leakage (e.g. "For tasks requiring train and test splits, datasets are split with a maximum sequence identity of 10%.")

For the convergent evolution classification dataset, how do you determine/enforce enzymes having different evolutionary histories?

Convergent evolution dataset consists of enzymes that belong to the same enzyme class (identical EC number) but have less than 10% sequence identity. (The typical threshold used for homology (shared evolutionary history) is 30% sequence similarity, see [2])

Given that phylogenies can only be inferred, how robust is the phylogeny construction to perturbations in parameters/underlying construction algorithm?

It is true that there is no absolute "ground truth" for phylogenies. We use IQ-tree, which tests multiple substitution models and parameters, for determining the most likely phylogeny values. Therefore, the final tree is robust to changes in parameters in the phylogeny algorithm. However, we acknowledge that certain regions of the phylogeny can be of low confidence in the limitation section: "For the EDS task, we acknowledge [...] the possibility that certain regions of the phylogeny may be of low confidence (due to the inherent uncertainty in reconstructing the ground truth phylogeny)."

[1] https://arxiv.org/abs/2210.07316

[2] https://academic.oup.com/peds/article-abstract/12/2/85/1550637?redirectedFrom=fulltext

Thank you for taking the time to review our work. We are glad that inclusion of "diverse range of species" to be an important and "biologically and evolutionarily relevant" to benchmarking gLMs and pLMs.

This dual focus weakens the depth of analysis, as attempting to tackle both classes leads to a more superficial investigation. A more focused approach, concentrating on a single class—such as AA models—would result in a stronger, more detailed exploration, particularly since most tasks appear to be better suited to this domain.

We agree that AA models and NA models traditionally draw from separate bodies of work. However, AA and NA models both fundamentally aim to represent the biological code, and more recently, the field has observed the convergence of two model types. For instance, Evo (Ngyuen et al 2024) and MegaDNA (Shao and Yan, 2024) claim that NA models can be used to generate protein coding regions and learn meaningful representations of protein sequences. Whether NA models are as efficient as AA models at learning representations of protein coding genes is an important question for the field, which could not be answered because no existing benchmark supported direct comparison of NA and AA models. We agree that the current version of DGEB features more AA tasks, we hope to expand upon NA tasks in our future DGEB releases. We believe the direct comparison of AA and NA models on representations of protein coding genes is an important contribution to the field, and could help guide machine learning practitioners in their design choices on the modality with which one models biology.

The dataset used in the paper appears ill-suited for NA models, as the tasks primarily target protein sequences rather than regulatory or non-coding regions of the genome.

The two NA foundation models benchmarked here (Evo and NT) have been proposed to model protein coding genes as well as regulatory regions of the genome. For example, Evo designs Cas proteins and Transposases, and is evaluated on protein fitness and gene essentiality tasks. Recent follow-up work (Boshar et al 2024) claims that NT can be used for protein downstream tasks. With the growing efforts and motivation to model the entire genomic regions including protein coding regions with DNA sequences only (other examples: megaDNA, plasmidGPT), we believe that it is important to evaluate NA models performance on protein tasks and compare directly with AA models. We however agree that more non-coding region tasks should be included (currently limited to three rRNA EDS tasks and one 16S rRNA clustering task). We hope to include more regulatory element classification tasks in future releases of DGEB and we include this as a current limitation with the inclusion of the following sentence: " While DGEB is designed to support both NA and AA models, current suite is biased towards coding sequences with only four tasks targeting non-coding elements (16S_Arch, 16S_Bac, 16S_Euk, Ecoli_RNA), limiting ability to evaluate NA model representations of regulatory elements (e.g. promoters, transcription binding sites)."

The DGEB datasets are relatively small and designed primarily for assessing zero-shot performance, where no training data is provided.

We deliberately prioritize zero-shot tasks in DGEB to address the problem of biased, sparse and often incorrect labels in biological evaluation datasets. Relying heavily on supervised benchmarks in biology can lead to biased evaluation that is not desirable for a foundation model.

This narrow scope restricts the benchmark’s utility to specific use cases, such as evaluating pre-trained models, rather than facilitating a comprehensive analysis of model performance across varied settings.

DGEB is designed as a benchmark for pre-trained foundation models, not fine-tuned models. We designed DGEB to address the gap, where no diverse suite of benchmarks exists to evaluate new types of foundation models. We hope that thorough benchmarking of foundation models across diverse tasks can guide fine-tuning efforts.

The benchmarking of models could be expanded to include a wider range of architectures.

We expanded the benchmarked models to include DNABERT2 in this revision. With this addition, we believe we have included most widely used and performant foundation models.

The authors argue that there is currently a eukaryotic bias in the evaluations available. I wonder if it's such a bad thing since most of the applications that we are interested in have to do with perturbing eukaryotic organisms?

We argue that "Labels are heavily biased towards model organisms (e.g. Human), therefore performance on species-specific evaluation tasks are not guaranteed to transfer to other organisms." Most of biotechnological breakthrough discoveries in the past decades were made in non-human systems (e.g. CRISPR-Cas (bacteria), RNAi (plants and worms), DNA Polymerase I for PCR (bacteria), GFP (Jellyfish), Antibiotics (bacteria)). Models that overfit on Human genomic evaluation sets will not contribute to enabling future high-impact discoveries in biotechnology that are likely to be found in diverse organisms.

Another question that is important to mention in any sequence based evaluation benchmark is that biotechnology is continuously improving and some of the measurement methodologies currently are flawed. What is the role of batch effects in constructing these datasets? How will these evaluation datasets continue to improve overtime as our ability to measure cell properties keeps improving?

This is a great point. We believe that evaluation benchmarks must continually improve over time. We explicitly mention and implement versioning: "We version both the software and the datasets and include versioning in the results, making the benchmark results fully reproducible." We also explicitly built a platform that can be improved over time: "[...] existing functional annotations must be continuously refined and expanded. DGEB supports simple extension of tasks and datasets. New or revised datasets can be uploaded to the HuggingFace Hub and new evaluation tasks can easily be added through GitHub pull requests." We expect there to be batch effects across versions as the granularity and accuracy of our sequence-to-function mapping improve. These batch effects are inevitable and will be tracked with thorough versioning of both the benchmarking software and datasets.

Thank you for the thoughtful review, we appreciate you highlighting DGEB's ability to handle different modalities and inclusion of diverse sequences beyond eukaryotes.

The authors don't propose simple baselines

Thank you for raising this point. We have added a one-hot baseline, where the sequence is represented as one-hot vector per residue, and mean-pooled across the sequence dimension like all benchmarked models. The baseline results are shown in Figure 3 as dotted lines.

There is no SNP evaluation and most of the evaluation consists of assessing homology relationships between sequences across different species or within.

Thank you for pointing this out. We acknowledge that the current evaluation suite in DGEB focuses on higher level representations (single-element, inter-element and multi-element) as we identified this to be a major gap in existing benchmarks. DGEB complements existing residue-level benchmarks (e.g. ProteinGym, BEND) that focus on token/residue-level representations. We believe that it would be valuable to incorporate more residue-level benchmarks to our suite in the future. We have added this point to the Limitation section: "DGEB's current evaluation suite focuses on single-element, inter-element, and multi-element scales of representations, and is designed to complement existing benchmarks that focus on residue-level representations (e.g. mutational effects (Notin et al 2023, Marin et al 2024)."

Overall I think a double blind venue isn't the right setting for evaluation of dataset papers since it requires assessment of the codebase. I think the quality of this work would be best assessed in a journal or a datasets and benchmarks track.

We have added the anonymized code as a zip file in the OpenReview supplementary data. This paper is submitted under Primary Area: "Datasets and Benchmarks" track.

The authors aren't very precise with their vocabulary: "For BiGene Mining, we curated functionally analogous sequences found in two phylogenetically distant taxa (e.g. Bacteria and Archaea) or interacting paralog pairs in sets of orthologous sequences. " Paralogs are homologous sequences related through duplication events. Orthologs are sequences related through speciation events. When the authors describe interacting genes do they mean physically interacting? If so what kind of evidence is used to support this claim?

Thank you for pointing this out. We recognize that this language is confusing. We have edited this sentence as "For BiGene Mining, we curated functionally analogous sequences found in two phylogenetically distant taxa (e.g. Bacteria and Archaea) or orthologs of interacting protein pairs across many species." The dataset we curated for this task is now renamed as "ModBC BiGene mining", where the physically interacting pairs of ModB and ModC are curated, and the task is to find the interacting ModC for a given ModB, in a set of ModC orthologs (ModCs found in different species).

"this metric cannot be used to determine how well a model can abstract evolutionary and functional relationships between non-homologous proteins." I don't really understand what the authors mean by this sentence.

Thank you for flagging this. We have edited this sentence to "this metric cannot be used to determine how well a model can represent evolutionary and functional relationships between non-homologous proteins." The context of this sentence was to highlight that while DMS benchmarks are appropriate for assessing fitness effects of mutations in a homologous set of sequences, they do not assess how well a model represents relationships between distant and non-homologous sets of sequences.

"These are important properties, they are too coarse in scope to evaluate whether a model has learned biologically meaningful functional information." Is there evidence to support this claim?

We have modified this sentence to "These are important properties, they are too coarse in scope to evaluate whether a model has learned more granular functional information (e.g. enzymatic function, protein interaction)".

What would a reasonable, general framework for training and evaluating a simple regression model on top of these embeddings look like?

The DGEB benchmark is designed to be easily extendable with community contributions of datasets and tasks. The addition of regression tasks, like gene expression, would be very similar in implementation to the existing classification tasks, like EC Classification, which trains a sklearn LogisticRegression on the embeddings for the provided train/test split. In this case a LinearRegression layer should be used.

Same question, but for link prediction.

DGEB includes tasks which are considered link prediction, such as the operon prediction tasks which identify interacting pairs from non-interacting pairs. This could be extended to additional link prediction tasks with higher degree graphs, for example using nearest-neighbor link prediction.

Is it possible to measure/quantify, or at least demonstrate, the effect of not enforcing the phylogenetic train-test split?

We would like to clarify that the phylogenetic evolutionary distance similarity (EDS) task does not use a train-test split, as it is a zero-shot task.

Thank you for your review. We are glad you find the design of DGEB to be "carefully thought out and sensible" and the direct comparison between AA and NA modalities to be a useful contribution to the field.

Assumption that more capable models will have geometries that more closely match various problems

This is a great point, we agree that there is no guarantee that models will have geometries matching specific problems, and that this is an assumption based on empirical evidence. We have added this to the Limitations section: "DGEB includes multiple zero-shot tasks, as ground-truth labels for biological function are sparse and biased. These tasks rely on embedding geometry to evaluate model performance. The assumption that models capturing important features of biological function have geometry directly matching the given tasks is not guaranteed. Future research could explore methods for identifying and leveraging relevant subspaces within model embeddings."

Many of the benchmark tasks make assumptions about the metric structure of the embedding space, the correct distance function is likely to depend on the specifics of the model architecture

We agree that there is not necessarily a single correct distance function for any task. Because of this, we default to the commonly accepted cosine similarity as the metric for all tasks relying on embedding distances, except for evolutionary distance tasks, where there is less certainty in the best metric so we select the one with highest Pearson correlation. For clustering, we use k-means with euclidean distance, as other distance metrics are not supported by sklearn k-means implementation. Regarding pair classification, other distance metrics are computed, but only cosine similarity is selected as the main metric and the others are unused (will be removed in the next release).

Limitation of using Euclidean embeddings for phylogenetic distances

Thank you for highlighting this, and we agree that Euclidean space is not optimal for representing phylogenetic tree structures. While Euclidean embeddings have limitations, they still provide a useful starting point for comparing model performance, and we believe this benchmark will be important for evaluating novel hyperbolic architectures, baselined by Euclidean embedding model results. We add this acknowledgment to the limitations section.

It is not clear which layer is chosen as "mid" when the size of the model fluctuates. I also wonder how influential this choice of "mid" layer is on model performance

We evaluate all models using embeddings from the middle and last layer to account for layer specialization noted in previous studies (Section 4.2.1). The middle layer is extracted automatically for each model using its config (which specifies model depth). We do not benchmark each layer to avoid issues with multiple hypothesis testing, and to keep the evaluation fair over models with varying depth. Empirically, we do not observe discrete jumps in performance across consecutive layers, likely due to the residual stream in modern transformer architectures.

The presentation of the results across many subfigures, rather than a single benchmark table, is somewhat confusing. I recommend cleaning up the table in Appendix G and including it as the main result. (This may be more of a matter of personal preference).

We chose to represent the results in the main text as Figure 3 instead of the table in Appendix G, because the figure provides immediate ability to compare scaling performance across parameter count. The table cannot show the continuous axis of model scale, and is provided as reference to readers.

I would like to see the authors justify the specific design choices that go into the architectures they use as well

We would like to clarify that our benchmark focuses on evaluating publicly available pre-trained models. We did not train any new models in this study. Our primary objective is to assess existing models on biological function. Justifying the design of these pre-trained architectures is beyond the scope of our current work.