GenomeOcean: Efficient Foundation Model for Genome Generation

As the paper's contribution lies in its application to biology and genomics rather than novel methodological advancements, the biological accuracy of the experiments and generated sequences are of high significance to the overall impact of the paper. Several experimental concerns are highlighted below:

• The paper omits any statistical or textual description of the pre-training dataset, other than the method of collection, making it difficult to form intuition about the biological validity of experiments and judge the significance of the method. It is unknown how much of this pre-training dataset (and consequently GenomeOcean) overlaps with existing genome foundation models. As the authors note, the dataset will be released and will likely be an extremely valuable resource for the community, but as it stands it is difficult to assess the paper's contributions in this aspect.

• While the evaluation of common genomic FM modelling decisions is a valuable contribution to the community, the choice to evaluate performance on the Genome Understanding Evaluation benchmark may dilute the insights gained. Based on my understanding of the benchmark from the original paper, the train/val/test splits in the datasets are not made while accounting for homology or sequence similarity. This filtering step is common in other biological domains such as pLMs. Without this step, data leakage may occur, and it becomes unclear whether the models are truly capturing biological function / structure. Consequently, it is unclear whether the modelling decisions in the preliminary experiments contribute towards better representations.

• Although the proposed evaluation metric in Section 4.1 is novel to my knowledge (though perhaps it could be formalized into something resembling the inception score or FID) and interesting from a methodology perspective, using other genomic foundation models as a method to discriminate generated sequences may be a limitation, as the metric would suffer from inherent biases in the existing FMs. This may detract from the metric, given the discourse in community on the overall validity of genomic foundation models. A simple baseline agnostic of existing FMs such as phylogenetic distance between sequences would be interesting to see.

• While the proposed evaluation metrics have a reasonable setup, the tasks evaluated (e.g. species classification) lack strong correlation with the functional validity of the generated sequences, which is highly relevant for real world utility of GenomeOcean in my opinion.

• Minor, but claims in the introduction that genomic foundation models outperform classic supervised models are unsubstantiated. For splice site prediction specifically, none of the in-line citations provided mention splice site detection. Moreover, to my knowledge, there has been no comparison of genomic foundation models against more modern splice site predictors, nor on the more realistic splice site prediction task described in SpliceAI / Pangolin. As such, the overall claim may be pre-mature.

• Poorly motivated in terms of use cases (see my first question)
• Using GUE to determine which tokenization, model architecture, and training objective to be used for the generative model seems arbitrary and potentially incongruous. The representations learned using these modeling decisions will be optimized for the relatively narrow set of tasks presented in GUE, and may have very little generalizability.
• Given the prevalence of CNN-based models in genome sequence modeling, the authors should also consider CNN-based architectures such as GPN (Benegas et. al, 2023) or LOGO (Yang et. al, 2022).
• The results from the biological properties investigation are not significant
• As the authors correctly state, it is difficult to validate the biological plausibility of genome sequences. Thus, there needs to be a more comprehensive effort to evaluate the “goodness” of the generated genomes in this paper. Currently, there is too much emphasis on model design and not enough emphasis on validation.
• The main context benchmarks in 4.1 are inherently better suited to the biases of GenomeOcean, which has likely seen a larger diversity of sequences across domains, whereas Evo and GenSLM are only trained on prokaryotes and viruses, respectively.

Benegas, G., Batra, S. S., & Song, Y. S. (2023). DNA language models are powerful predictors of genome-wide variant effects. Proceedings of the National Academy of Sciences, 120(44), e2311219120.

Yang, M., Huang, L., Huang, H., Tang, H., Zhang, N., Yang, H., Wu, J., and Mu, F. (2022). Integrating convolution and self-attention improves language model of human genome for interpreting non-coding regions at base-resolution.Nucleic Acids Research 50, e81–e81.

However, the paper exhibits the following limitations:

• The paper lacks significant research contributions. The authors repeatedly claim high inference speed as a major contribution. While they demonstrate efficient code, they primarily utilize existing codebases. Consequently, this represents good engineering practice rather than a novel research contribution. A true contribution would involve proposing a new method for accelerating such models, specifically within the context of DNA generation. The second claimed contribution is the ablation study; however, similar ablations have been performed extensively in the literature, such as in NT, DNABert, HyenaDNA, and Evo. While the authors' slightly different setting justifies a new ablation study, the actual contribution remains limited. The third and primary claimed contribution is the introduction of a new dataset. However, the dataset is not adequately discussed, with minimal information provided. Given its central role in the work, the dataset requires a proper introduction and detailed description within the publication.

• The experimental protocol focuses on three specific tasks, while the model is intended to be generic. The authors should either (1) design a more comprehensive set of tasks across various domains to evaluate the model's generalizability or (2) focus on fewer tasks with more in-depth analysis to understand the specific genomic knowledge captured by the network. For example, the authors could investigate whether their model captures enhancer-promoter interactions better than existing models (see the insightful biological analyses in [1]) or if the attention maps can capture RNA secondary structures (see the recent work [2]). They could also leverage and adapt recent ideas from D3, a diffusion model for DNA sequences [3].

• Interpreting the performance differences between Evo, GenSLM, and GenomeOcean is challenging. Since all three models are trained with the same loss function and prompted similarly, the only differences lie in (1) their architecture and (2) pre-training data. The authors do not discuss the pre-training data or the extent of overlap between the downstream datasets and the pre-training datasets for each model, making it difficult to gain insights from the comparison.

While I acknowledge the high presentation quality, tutorial aspects, and strong engineering, the publication lacks sufficient contributions for a conference like ICLR. A major concern is the emphasis on the new pre-training dataset, which is not adequately discussed.

Figure 1 on it’s own is not a great selling point to me, as it does not tell me the quality of the results. random.choice([“A”,”T”,”C”,”G”]) with an associated single and bigram base frequency is most certainly faster and has fewer parameters!

The test described in Figure 2b is incongruent with the rest of the paper. Here the authors train a descrimitive model using different DNA representations as input. It’s obvious that the character level input will do poorly because there is no context window. In modern generative models, there’s a huge context window. It’s difficult to see how this discriminative test translates to the modern generative modeling case. Especially with what is shown in Figure 2c, why not just do that set of experiments for all different DNA tokenizers?

Since the speed of GenomeOcean is strongly advertised, it’d be great to have an ablation study of the improvements that made the model what % faster. All of these are listed in section 3.1. This is a relatively easy experiment to do, because you don’t have to train the model for this; it’s just inference speed.

For section 4.1 and Table 1, did you hold out those species or even representative genus while training the generative models?

I think the ORF length analysis is quite weak. The Ground Truth should be the dataset the model was trained on, so a comparison of the models can be done. Evo was trained on OpenGenome, which is just prokaryotic sequences. Did you filter in your “ground truth” dataset for this? Which did you use?