SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model

Thank you for your thoughtful review. Additional experiments are presented in https://anonymous.4open.science/r/charts-CDE6. We respond your concerns below.

Q1:Performance on Splicing Tasks and GUE Dataset

We clarify two key points:

• SPACE ​outperforms DNABERT2 across all splicing benchmarks (see Table 7 and Table 10). While SPACE does not yet surpass Nucleotide Transformer (NT) in absolute terms, it achieves ​substantial gains over Enformer:
  • ​GUE splice task: 0.8748 (SPACE) vs. 0.8593 (DNABERT2), 0.0593 improvement (relative to Enformer)
  • ​NT splice tasks: Average 0.917 (SPACE) vs. 0.851 (DNABERT2), ​0.161 average improvement (relative to Enformer)
• Our analysis shows that SPACE underperforms DNABERT2 only on a subset of Epigenetic Marks Prediction tasks in the GUE benchmark. This can be attributed to SPACE encountering a novel species (yeast) in these tasks. Notably, SPACE demonstrates significant improvements over Enformer under the same conditions, which provides strong evidence for its generalization capability.

Q2:Comparison to gLMs Baselines

• Grover: Directly referenced Grover's results on the GUE dataset. As Grover used only human-annotated datasets, we limited our comparisons accordingly (in section 1.1 of supplementary materials). Our results demonstrate significant outperformance over Grover across all these tasks. Furthermore, we fine-tuned Grover using the official hyperparameters on the revised NT downstream tasks (in section 1.2 of supplementary materials).
• Caduceus: All baseline comparisons were derived from Caduceus' reported performance on genomic benchmarks (as showed in their paper). Notably, Caduceus did not evaluate the Drosophila task, consequently, the CNN baseline results were extracted from genomic benchmarks (showed in ​Supplementary Materials Section 1.3).
• GPN-MSA: Compared SPACE against GPN-MSA on mutation effect prediction tasks within the BEND benchmark.While SPACE underperforms GPN-MSA, we emphasize that ​SPACE uses only reference sequences during inference, aligning with standard foundation model evaluation protocols. In contrast, GPN-MSA requires ​multi-sequence alignments (MSAs) as input, introducing an unfair advantage for this baseline.
• evo: While we acknowledge Evo's contributions, integrating its results during rebuttal faced challenges due to:(1) Evo's 7B parameters exceed the largest baseline capacities (NT-2.5B), and current resources cannot support timely replication;(2) No official benchmark data exists for Evo; (3) Evo focuses on generation tasks, lacking results on mainstream benchmarks.

Q3:Data Availability Challenges

We acknowledge that raw sequence data is more accessible than genomic profiles. However: (1) Multi-omic data is increasingly available, and we've demonstrated SPACE's advantages using only limited data, suggesting future improvements as these resources expand; (2) While profile data is needed during pre-training, downstream applications require only sequence input since profile understanding is embedded in the model parameters, unlike GPN-MSA which requires additional MSA input for both pre-training and inference.

Q4:Increasing capacity by decoder and species embedding

We appreciate your suggestion. We have incorporated this consideration in our new ablation studies, with the results presented in Section 2 of the supplementary materials. The corresponding analysis is provided in Q3 of Reviewer oB6U.

Q5:Task Performance

Yes, we hypothesize that SPACE demonstrates superior performance on tasks requiring modeling of gene regulatory relationships. However, for purely sequence-based tasks, SPACE may underperform compared to gLMs pretrained on extensive sequence data, as its training data is more limited in this regard.

Q6: Other Comments

We fully agree with the reviewer's perspective and will soften our claims. Our work primarily validates the potential of supervised pre-training for DNA foundation models, having already achieved SOTA. Both our results and those of GPN-MSA demonstrate that reference sequence pre-training alone is insufficient. However, we think that integrating extra information as supervision signals is particularly meaningful, as it enables downstream tasks to rely solely on raw sequences while implicitly embedding biological knowledge into model parameters, whereas GPN-MSA requires additional inputs. In the protein LM field, recent work has similarly adopted MSA information as supervision [1]. [2] has also demonstrated the advantages of using AlphaFold2's supervised-trained backbone for pLMs. Therefore, we believe incorporating additional information as supervision signals provides advantages for foundation models.

Detailed discussion on long sequence modeling is in our response to Q5 of Reviewer ceSM.

References:

[1]Evolution-inspired loss functions for protein representation learning

[2]Exploring evolution-aware &-free protein language models as protein function predictors

Thank you for your thoughtful review and for highlighting areas where our manuscript can be improved. We supplemented with additional experiments which are presented in https://anonymous.4open.science/r/charts-CDE6. Below, we address your concerns and outline revisions to strengthen the paper.

Q1:Category Operator

The category operator leverages biological priors derived from the Enformer dataset, which groups genomic profiles into four categories based on distinct measurement technologies:

1. Transcription factor (TF) chromatin immunoprecipitation and sequencing (ChIP–seq)
2. Histone modification ChIP–seq
3. DNase-seq or ATAC-seq
4. CAGE tracks

These categories reflect critical biological distinctions. For example, chromatin accessibility (DNase/ATAC-seq) and transcription initiation (CAGE signals) are functionally linked: chromatin accessibility establishes permissive 3D topological environments necessary for transcription initiation, with transcription start site (TSS)-associated promoters and enhancers overlapping spatially with accessible chromatin domains. This biological rationale guides expert specialization in the decoder—profiles with strong interdependencies (e.g., CAGE and ATAC-seq) share experts, while distinct profiles (e.g., TF ChIP-seq) activate specialized experts.

Q2:Necessity of the components

The evaluation of DNA language models (LMs) typically exhibits substantial variance, and no existing DNA LM achieves absolute dominance across all datasets. For instance, DNABERT2 adopted mean ranking as its primary metric in their publication. We note that while the ablation results in Table 3 of our original paper report averaged outcomes, the complete results in Appendix E (Table 11) demonstrate robust improvements over Enformer across nearly all evaluated tasks. To further validate the architecture, we conducted targeted ablation studies, which demonstrate that:

• The species-aware MoE encoder enables cross-species knowledge transfer.
• The profile-grouped decoder captures category-specific regulatory mechanisms.

These components are biologically grounded and critical for generalizability, particularly in data-scarce scenarios. To further strengthen our claims, we have included and analyzed new ablation results in Q3.

Q3:Ablation Studies

Additional ablation studies are documented in Section 2 of our supplementary materials. Section 2.1 presents comprehensive ablation results on Nucleotide Transformer downstream tasks. All ablation models were pre-trained with their hidden dimension parameters halved, as detailed in Section 4.6 of the main text.

SPACE demonstrates comparable or superior performance to the decoder-removed variant in ​14/18 tasks, with ​11/18 tasks still outperforming even when replaced by a parameter-matched MLP. Notably, for regulatory element classification tasks, SPACE achieves better results in ​4/5 datasets, with the only exception being the ​TATA box dataset—which primarily examines sequence motifs of TATA boxes and does not require complex regulatory mechanism understanding. This suggests that while our decoder does not explicitly improve direct chromatin profile prediction accuracy, the ​MoE architecture implicitly captures cross-profile regulatory interactions by modeling their dependencies. This capability provides critical advantages for tasks requiring integrated understanding of multiple profiles, such as regulatory element prediction.

To further validate cross-species generalization, we evaluated ablation variants on the GUE benchmark (yeast and virus tasks, detailed in Section 2.1). Results reveal that the MLP-based decoder variant shows markedly weaker generalization to novel species compared to SPACE with its enhancement decoder architecture.

Q4:Qualitative Analysis and Interpretability

In Section 4.4 of main text, we analyze expert specialization patterns:
• Encoder: Two shared experts capture cross-species mechanisms, while one species-specific expert adapts to unique genomic contexts.
• Decoder: Complex profiles (e.g., TF ChIP-seq) activate dedicated experts, whereas interdependent profiles (e.g., CAGE and ATAC-seq) co-activate shared experts—aligning with their biological relationships.

We agree that deeper case studies could strengthen interpretability and will prioritize this in revisions.

Q5:Broader Impact

We have re-examined the viability of ​supervised learning paradigms for DNA foundation models and proposed specifically designed optimizations. This approach effectively addresses the core limitation of traditional unsupervised pre-training — ​inadequate modeling of biological functional associations. We believe these insights could have profound implications for advancing research in ​genomic language models (gLMs).

Thank you for the constructive feedback. We appreciate your acknowledgment of our work’s potential contributions to multi-genome foundation models. We supplemented with additional experiments which are presented in https://anonymous.4open.science/r/charts-CDE6. Below, we address your points to clarify and strengthen the manuscript:

1. Response to the Question

Reviewer’s Question:
Are the authors using a pretrained model from Enformer or performing pretraining from scratch?

Response: SPACE is trained from scratch on genomic profile prediction tasks without initializing weights from Enformer. We sincerely appreciate this insightful inquiry, as it inspires us that initializing each expert FFN's weights with Enformer's pretrained parameters could be a promising direction for future exploration.

2. Significance of Improvements

Reviewer’s Observation:
Although the improvement over Enformer is not very significant in most cases, the experimentation is sufficient. Perhaps, the most notable improvement is in the underrepresented mouse experiments.

Response:
Our improvements yield significant gains over Enformer in cross-species generalization and new genomic tasks, demonstrating that SPACE learns more robust DNA representations. As detailed in Table 2 of the main text, we systematically quantify SPACE’s cross-taxa generalization superiority over Enformer. This is further empirically validated through the Drosophila melanogaster enhancer classification task (Section 1.3 of supplementary materials). Notably, on splice prediction benchmarks, SPACE demonstrates marked improvements of ​0.0593 on the GUE splice dataset and an average ​0.161 across NT splice tasks (Table 7 and Table 10 of main text). These quantitative enhancements provide rigorous evidence of SPACE’s enhanced generalization capacity on both evolutionarily divergent species and functionally distinct genomic tasks.

The strong performance on underrepresented species (e.g., mouse) highlights SPACE’s ability to generalize in data-scarce scenarios—a strength enabled by its species-aware encoder. By decoupling shared and species-specific regulatory mechanisms via MoE, SPACE effectively adapts to species with limited training data.

We sincerely appreciate the constructive feedback. We supplemented with additional experiments which are presented in https://anonymous.4open.science/r/charts-CDE6. We address each key issue and outline manuscript improvements below.

Q1:Generalization Across Species and Tasks

We concur that evaluating generalization capacity is paramount for supervised DNA foundation models.To comprehensively address this concern, we performed extended benchmarking using the Genomic Benchmarks dataset, which represents the only mainstream benchmark encompassing species beyond those investigated in our previous experiments, including Human-or-worm classification and Drosophila enhancer classification. The results are presented in Section 1.3 of supplementary materials. SPACE achieves robust performance on these tasks, demonstrating cross-species generalizability. While it doesn't achieve SOTA on the human-or-worm classification, it significantly outperforms Enformer.

Q2:Architectural Complexity vs. Performance Gain

SPACE's architecture is biologically motivated:

• ​Species-aware Encoder: Explicitly models shared and species-specific regulatory mechanisms to enable cross-species knowledge transfer.
• ​Profile-Grouped Decoder: Captures mutual regulatory influences among distinct genomic profiles.

SPACE consistently improves performance across ​most tasks while maintaining biological interpretability. And additional ablation studies on module effectiveness are provided in Q3 of Reviewer oB6U.

Q3:Missing Baselines

We acknowledge this limitation and resolved in ​Q2 of Reviewer ueQp.

Q4:Novelty and Contribution Clarity

We consider both aspects as key contributions:

As the first work to rigorously explore supervised pretraining for DNA foundation models, SPACE achieves SOTA over NT on most benchmarks despite using significantly fewer parameters and less training data. This validates the effectiveness of supervised pretraining for genomic tasks. While computational limitations and the scarcity of curated multi-species, multi-omics datasets prevent scaling SPACE to the extremes of NT or Evo, our work highlights supervised pretraining as a promising scale-up direction for future research.

The Species-Aware Encoder and Profile-Grouped Decoder collectively enable substantial improvements over Enformer, particularly in cross-species tasks and new biological applications(e.g., splice site prediction). The detailed experimental results presented in the Appendix D consistently demonstrate that our proposed module achieves superior performance compared to Enformer. The MoE design enhances interpretability: expert selection frequencies align with foundational biological principles. While we acknowledge the reviewer’s concern about limited downstream benchmarks, we have validated SPACE on additional benchmarks, including Genomic Benchmarks and BEND (see ​Q2 of Reviewer ueQp).

Q5:keys for developing DNA LMs

First, based on the scaling laws observed in NT [1] and Evo [2], expanding both model parameter size and training data scale — following practices in the LLM field — is critical for developing DNA language models.

Second, increasing the model's ​context window length is theoretically essential to prevent information loss by accommodating full DNA sequences. However, current DNA models have not yet empirically validated this advantage. As noted in Evo 1.5 [3], training with 8K context lengths outperforms 131K configurations. While longer input lengths technically allow processing full DNA sequences, they currently lack consistent performance improvements.

Additionally, most existing DNA LMs scale training data by aggregating reference genome datasets, which inherently fail to capture mutation patterns. This explains why DNA LMs underperform Protein LMs [5] in ​zero-shot variant effect prediction [4]. Protein LMs benefit from more diverse training data (e.g., multiple sequence alignments), suggesting that incorporating population-level mutation data into DNA LM pretraining — as proposed in [6] — holds significant promise.

Finally, as emphasized in our work, ​genomic profiles are indispensable for advancing DNA LMs. DNA sequence functionality is regulated through cell-type-specific interactions with diverse genomic profiles (e.g., chromatin accessibility, histone modifications). These profiles collectively shape chromatin environments that govern gene expression patterns across cellular contexts.

References:

1. Nucleotide Transformer: building and evaluating robust foundation models for human genomics
2. Sequence modeling and design from molecular to genome scale with Evo
3. Semantic mining of functional de novo genes from a genomic language model
4. GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction
5. Evolutionary Scale Modeling: Pretrained language models for proteins
6. Pre-training Genomic Language Model with Human Variants for Better Understanding Functional Genomics