JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model

Dear Reviewer BnYd,

We sincerely express our gratitude for your recognition of our work and your insightful, detailed feedback. We have carefully formulated the following responses:

W1: While I agree that the choice of dual-stream autoregression is appropriate, it'd be good to include a discussion about this point to contextualize this choice for genomics modeling.

Thank you very much for raising this important point. We are very thankful for the help in making this statement more clear.

Autoregressive modeling relies on unidirectional representations, where understanding and predicting a current pattern depends on preceding ones. This approach is extensively applied in Natural Language Processing (NLP), as the logical sequence of human language allows subsequent elements to be inferred from prior context. Such modeling naturally aligns with language generation tasks. However, DNA, as a complex biological "language," exhibits distinct patterns significantly different from human languages in regulating phenotypes. Unlike human languages, genomic processes frequently involve bidirectional interactions, with critical regulatory elements located both upstream and downstream of key genomic regions.

For example, bidirectional promoters initiate transcription in both orientations [1, 2]. Additionally, intergenic enhancers transcribed predominantly bidirectionally often function as weak promoters in both directions. Conversely, for elements with unidirectional transcription (both enhancers and promoters), transcription direction typically correlates with the orientation in which the element functions as a promoter in vivo [3].

Thus, bidirectional modeling is essential for comprehensively understanding genomic processes.

In addition to dual-stream autoregression, masked modeling is a classic method for bidirectional sequence modeling. However, due to its masking logic, only a subset of tokens contribute to model optimization, as discussed in our "Empirical Validation of Training Efficiency" section (Section 3.1).

Therefore, Janus modeling, the dual-stream autoregressive method proposed in our work, facilitates effective bidirectional DNA modeling while maintaining high learning efficiency, making it particularly suitable for genomic modeling.

We briefly addressed this topic in Section 1 (lines 55–60). Incorporating the above discussion will further enhance clarity and biological relevance. We will include this expanded content in the camera-ready version, as you suggested. Thank you again for highlighting this important point.

[1] Wei, Wu, et al. "Functional consequences of bidirectional promoters." Trends in Genetics 27.7 (2011): 267-276.

[2]Schulz, Daniel, et al. "Transcriptome surveillance by selective termination of noncoding RNA synthesis." Cell 155.5 (2013): 1075-1087.

[3]Mikhaylichenko, Olga, et al. "The degree of enhancer or promoter activity is reflected by the levels and directionality of eRNA transcription." Genes & development 32.1 (2018): 42-57.

W2: As the authors have discussed, Janus-DNA is only pretrained on Human Reference Genome and may lack the knowledge needed for modeling other species, but it'd be good to see some performance on these tasks (such as EPM and Mouse from GUE) to understand the performance gap.

Thank you very much for your interest in exploring further possibilities for our work; we greatly appreciate it. Currently, we are integrating genomic data from additional species, variants, and incorporating modalities, such as epigenetic states (e.g., chromatin accessibility, histone modifications) and single-cell transcriptomic profiles. These additions are vital for understanding cell-type-specific regulation and predicting chromatin-influenced phenotypes within the JanusDNA framework.

We fully agree that evaluating models pre-trained on the human reference genome across other species to determine performance gaps will help explore genomic similarities between species and contribute to future biological research. We will pursue this direction in our future work. We express our sincere gratitude for your insightful suggestions and the valuable research ideas proposed.

Thank you once again for your valuable feedback. Your comments significantly enhance the quality of our manuscript, and we are committed to making the necessary improvements to meet the high standards of the NeurIPS community.

Dear Reviewer yT7i,

We sincerely appreciate your detailed review and recognition of our work. Below, we address your thoughtful and constructive suggestions.

Q1: Detailed explanation of Eq. 6

Thank you very much for highlighting this point; we are happy to clarify.

First, the unidirectional representation follows an autoregressive structure, where, for example, the representation at position 0 predicts the token at position 1, the representation at position 1 predicts the token at position 2, and so forth.

Thus, it is understandable that the masks in the self-attention regions (top-left and bottom-right in Figure 1(E)) allow attention to themselves, as reflected by the first and second lines in Eq. 6.

Second, the cross-attention parts (top-right and bottom-left) indicate fusion of the two unidirectional representations. Similar to the forward autoregressive representation, the reverse representation uses the representation at position T to predict token T-1, position T-1 predicts token T-2, etc. Therefore, to predict a middle token (excluding tokens at positions 0 and T) using these bidirectional features, we account for an offset of "+2". For instance, when predicting token 5 (with indexing starting from 0), the forward representation at position 4 and the backward representation at position 6 are utilized. However, if here is "+1", for instance, when predicting token 5 (with indexing starting from 0), the forward representation at position 4 and the backward representation at position 5 are utilized, but the backward representation at position 5 includes the original information of token 5, causing information leakage, and training loss will decrease to 0 quickly correspondingly.

We arrived at the formulation illustrated in Figure 1(E), which we have now improved to clearly indicate that the diagonal in the top-left and bottom-right region should also be unmasked. We also removed the confusing phrase “future peeking” in the manuscript and now clarify that we want to prevent information leakage. Thank you again for bringing this to our attention.

Q2: Metrics of computational complexity (refine figure)

Thank you for suggesting a more common complexity metric. We now additionally present the training efficiency based on the clock time required for model training along with corresponding performance on the last-token prediction task.

Listing these details in a table may not be intuitive; therefore, in camera ready version of the paper, we update Figure 3 by plotting accuracy against training time to clarify this comparison. Thank you for this valuable recommendation.

Q3: Why is Janus pre-training faster than masked models (per 1000 steps)? On first looks, there appear to be more operations per training step in Janus modeling.

For this ablation experiment, to keep the comparison as fair as possible, we used the same architectures for both the Janus and the masked language model, except masks for the final attention layer.

The Janus model consists of one layer with two parallel unidirectional encoders followed by a FlexAttention layer that incorporates a specially designed sparse attention mask for Janus-type feature fusion. As depicted in Figure 1(E) (which we revised to make the masking and prediction indices more intuitive to understand), the two unidirectional features are concatenated into a tensor of shape (batch, sequence_length × 2, dim).

Compared to the Janus model, the architecture of the masked model is similar, where there is one layer of two separated parallel unidirectional encoders (same as Janus model) followed by a FlexAttention, but with no attention mask. This is because for masked training, partial tokens have been masked in the dataset loading process, causing the full attention requirement for bidirectional feature fusion. Meanwhile, to keep the comparison as fair as possible, the two unidirectional features from the layer of unidirectional encoders will also be concatenated as a tensor with the shape of (batch, sequence_length * 2, dim).

Therefore, the speed difference primarily arises from the attention stage, where the Janus model employs sparse attention, while the masked model uses dense, full attention. Additionally, the masking pre-processing for the masked model incurs some computational overhead. Consequently, the Janus model achieves faster performance than the masked model when the architectures are identical.

Importantly, an alternative implementation of the masked model would concatenate features along the feature axis rather than the sequence axis, resulting in final attention over a tensor with shape (batch, sequence_length, dim * 2). This approach would make the computational costs of Janus and masked pretraining nearly equivalent. However, we opted against this implementation in our ablation study because concatenating along the feature axis would alter the masked model's capacity, compromising the comparability between the two approaches. We have now clarified this point in the description of the ablation experiment.

Q4: Lines 200-204 makes a distinction between masked models and the Janus model. What is the difference between the T-th token being masked for masked models and it being omitted for the Janus model? Is this distinction meaningful?

Thank you for closely examining this point. There is indeed no distinction; all models receive identical inputs, where the last token is masked for prediction. We will revise this description in future versions to eliminate confusion. We appreciate your careful observation.

Q5: How is the JanusDNA model fine-tuned (line 317)?

For the DNALONGBENCH tasks, all models, including Caduceus and JanusDNA, were fine-tuned entirely (full model fine-tuning) using a learning rate of 4e-4, a batch size of 8, and one batch per GPU over three epochs. We specified these hyper-parameters explicitly in Appendix D.2, lines 579–583, and will ensure this is clearly stated in future revisions.

Thank you for pointing out this ambiguity.

S1: SSM is first referred to in line 61 but the full form of this term is not stated until later. I recommend defining the acronym here.

Thank you for identifying this oversight. We will define the acronym "SSM" upon its first mention in line 61 in the camera ready version of the manuscript.

S2: When the hybrid MoE is introduced in Section 3.2, it is not evident how this helps JanusDNA deal with longer sequences.

We appreciate your insightful comment. Our intention was not to claim explicitly that MoE handles longer sequences more effectively. Rather, MoE models achieve superior performance compared to dense models with an equivalent number of inference parameters. This can be observed in Figure 4, where MoE models consistently demonstrate improved perplexity compared to dense models, across both shorter (1024) and longer (131072) sequence lengths.

Meanwhile, for many downstream tasks of DNA language models, we need to run inference on the entire genome (3 billion nucleotide pairs), potentially for many individuals. Therefore inference performance is crucial, requiring better performance while less running time, and MoE helps with that. We will clarify this point in the discussion in the camera ready version of the manuscript.

Thank you once again for your valuable feedback. Your suggestions and comments will significantly enhance the quality of our manuscript, and we are committed to making the necessary improvements to meet the high standards of the NeurIPS community.

Dear Reviewer yT7i,

Thank you for your thoughtful feedback and for recognizing the improvements. We greatly appreciate your detailed comments, which have significantly strengthened our paper, and your raised score. Have a nice day!

Best wishes

Dear reviewer tdwT,

We greatly value your time and effort in providing this constructive review and recognition of our work. We carefully considered your suggestions and propose the following response for your further review:

W1&Q1: Training speed (Janus model vs masked model)

Thanks very much for pointing this out. For the Janus model, the architecture is one layer of two separated parallel unidirectional encoders followed by a FlexAttention with a specifically designed attention mask for Janus-type feature fusion. As shown in Figure 1(E) (which needs to be modified a bit that the diagonal of the left top part should be unmasked), two unidirectional features are concatenated shaped (batch, sequence_length * 2, dim) , and the attention mask of Janus fusion layer is sparse due to its design.

Compared to the Janus model, the architecture of the masked model is similar, where there is one layer of two separated parallel unidirectional encoders (same as Janus model) followed by a FlexAttention, but with no attention mask. This is because for masked training, partial tokens have been masked in the dataset loading process, causing the full attention requirement for bidirectional feature fusion. Also, to keep the comparison fair, the two unidirectional features from the layer of unidirectional encoders will also be concatenated shaped (batch, sequence_length * 2, dim).

Thus, the speed difference primarily arises from the attention stage, where the Janus model employs sparse attention, while the masked model uses dense, full attention. Additionally, the masking pre-processing for the masked model incurs some computational overhead. Consequently, the Janus model achieves faster performance than the masked model when the architectures are identical.

Importantly, an alternative implementation of the masked model would concatenate features along the feature axis rather than the sequence axis, resulting in final attention over a tensor with shape (batch, sequence_length, dim * 2). This approach would make the computational costs of Janus and masked pre-training nearly equivalent. However, we opted against this implementation in our ablation study as concatenating along the feature axis would alter the masked model's capacity, compromising the comparability between the two approaches. We have now clarified this point in the description of the ablation experiment.

W2: limited pre-training data

We fully agree that the data scope of this work is limited to the human reference genome to achieve the fairest possible comparison with existing methods since the primary contribution of this work is the model architecture and pre-training paradigm itself as you acknowledged. We truly appreciate that. We fully agree that, for many DNA language model applications, diverse pre-training data is crucial and we are actively working on extending the data scope to genomes of additional species, more individual human genomes, and incorporating modalities such as epigenetic information and different sequencing data types to make the JanusDNA architecture more generalizable and robust. Thanks very much for your suggestion.

W3&Q2: MLM, CLM, Janus modeling ablation upon the same model architecture?

Thank you for this insightful suggestion. We conducted an ablation comparing MLM and Janus modeling using the same architecture as in Figure 3. However, conducting a direct ablation between CLM and Janus/MLM is challenging due to inherent architectural differences. The core strength of Janus modeling lies in its bidirectional context understanding, requiring a bidirectional architecture. CLM, by definition, is strictly unidirectional, and adapting it to a bidirectional architecture would conflict with its fundamental principles.

Nevertheless, we acknowledge the value in exploring this further. We ran a new ablation experiment to compare the performance of a unidirectional CLM and a bidirectional Janus model. In this comparison, we expected similar performance for the prediction of the last token (where there is no bidirectional information, so both models have exactly the same information to predict the last token), whereas we expected Janus to outperform CLM for the prediction of a token in the center of the sequence (where Janus benefits from the bidirectional context). To maintain fairness, causal models were constructed with one layer of a single mamba encoder and a FlashAttention2 layer with a causal attention mask, ensuring comparable model sizes to the Janus models.

The results of this new experiment are presented in the table below and will be included in the camera ready version of the manuscript. We confirm that the Janus model consistently outperforms the CLM for tokens in the center of the sequence across all evaluated model sizes. To our surprise, Janus also slightly outperforms the CLM even on the last token prediction task, suggesting Janus models indeed learn richer DNA representations through bidirectional training. We will clarify this point further in the manuscript.

Q3: Computational overhead of RC and ablation on its value?

Thank you very much for raising this key point. We fully agree that clarifying the computational overhead and performance impact of RC is essential.

Including RC enables the model to more robustly and comprehensively capture DNA sequence patterns, especially for biologically important motifs (e.g., transcription factor binding sites) that are non-palindromic. Recognizing both forward and RC versions of such motifs (for instance, motifs like GATA and TATC) is challenging, as it effectively requires the model to learn two distinct representations. Nonetheless, the utility of RC depends heavily on the specific biological context. For instance, genomic elements such as splice sites are strictly defined by a single DNA strand, making RC inclusion potentially suboptimal or misleading.

Regarding computational overhead, RC is introduced only during inference by averaging the embedding representations of a strand and its RC prior to decoding. Thus, the additional computational cost is minimal, primarily involving an extra forward pass through the model for the RC strand.

To quantitatively assess the impact of incorporating RC, we conducted ablation experiments using the NT benchmark. We fine-tuned models under consistent experimental conditions (learning rate 1e-3, batch size 256), with and without RC during prediction. The results clearly indicate that models utilizing RC generally outperform those without RC across most tasks. Notably, the exception to this pattern was observed in splice site prediction tasks, where RC inclusion led to inferior performance, consistent with the biological reality that splice sites are inherently strand-specific.

Based on your insightful feedback, we will clarify this discussion further in Section 3.2 and include the ablation experiment results explicitly in the final manuscript. We sincerely appreciate your constructive comments, which have helped strengthen our paper.

Dear Reviewer hQUx,

We are very grateful for your detailed review and recognition of our work's contributions. We have carefully considered your comments and respond as follows:

W1: Narrow data scope. We fully agree that the data scope of this work is limited to the human reference genome to achieve the fairest possible comparison with existing methods since the primary contribution of this work is the model architecture and pre-training paradigm itself as you acknowledged. We truly appreciate that. We fully agree that, for many DNA language model applications, diverse pre-training data is crucial and we are actively working on extending the data scope to genomes of additional species, more individual human genomes, and incorporating more modalities such as epigenetic information and different sequencing data types to make the JanusDNA architecture more generalizable and robust. Thank you very much for your suggestion.

W2: Limited improvement in Genomic Benchmark. We greatly appreciate your recommendation of this reference paper. We have carefully reviewed it and acknowledge its powerful and novel approach and now include the ConvNova performance in our results tables.

Additionally, we made further improvements to our architecture and reran all experiments. Specifically, we noticed that the model benefits from additional MLP layers after the final attention layer that fuses the forward and backward information as an example shows below.

Sequential(
      (0): Linear(in_features=32, out_features=64, bias=True)
      (1): SiLU()
      (2): Linear(in_features=64, out_features=32, bias=True)
      (3): SiLU()
    )

This simple modification does not significantly change the model design or size, but most performance metrics improved considerably. We updated the manuscript with these new results for the cam-ready version.

We attached the updated results for your reference, where JanusDNA_mlp is the latest model:

Genomic Benchmark

Following the settings from Caduceus, we fine-tuned the models with a batch size of 256, obtaining the new results shown in columns 5 and 6, surpassing ConvNova on 4 out of 8 tasks. Since ConvNova did not specify a batch size, we performed additional parameter searches across batch sizes of 128, 256, and 512. The results are shown in columns 7 and 8, surpassing ConvNova on 7 out of 8 tasks. (5-fold cross-validation).

Nucleotide Transformer (NT)

Performance on NT also improved significantly, as shown in columns 6 and 7. Columns 4 and 5 represent performance without the two additional simple MLP layers. (Columns 8 and 9 are for addressing Q1 to save space about ablations on reverse complement design.) (10-fold cross-validation)

DNALONGBENCH

Performance also improved significantly after adding the two simple MLP layers.

Q1: Value of reverse complement design and ablation?

Thank you very much for raising this important point. We fully agree that clarifying the value of the reverse complement (RC) design is crucial.

As you correctly pointed out, DNA follows the complementary base-pairing principle, meaning each DNA strand has a reverse complement strand with equivalent genetic information. However, despite this theoretical equivalence, many biologically important motifs (e.g., transcription factor binding sites) are non-palindromic. Recognizing both the forward and RC versions of such motifs (for instance, motifs like GATA and TATC) is challenging, as it effectively requires the model to learn two distinct representations. Therefore, explicitly integrating RC information allows the model to more robustly and comprehensively capture DNA sequence patterns.

Nonetheless, the usefulness of RC information can depend heavily on the specific biological context. For instance, certain genomic elements such as splice sites are defined strictly by a single DNA strand (the template strand), making the inclusion of RC information potentially suboptimal or even misleading.

To quantitatively assess the impact of incorporating RC information, we have conducted additional ablation experiments using the NT benchmark. We fine-tuned all models under consistent experimental conditions (learning rate of 1e-3, batch size of 256) across 10-fold cross-validation, with the only difference being with or without of RC information during prediction.

The results (presented in the last two columns of the above Table on NT, following the main experiments) clearly indicate that models with RC information generally outperform those without RC across most tasks. Notably, the exception to this pattern was observed in splice site tasks, where RC inclusion led to inferior performance, reflecting the biological reality that splice sites are inherently strand-specific.

Based on your insightful suggestion, we will clarify and expand our discussion on this topic in Section 3.2 (Reverse Complement) and include this new experiment in the final manuscript. We truly appreciate your constructive feedback, which has enabled us to further strengthen our work.