Revisiting Convolution Architecture in the Realm of DNA Foundation Models

What is the intuition and rationale behind using dilation over self-attention in a DNA foundation model? And why is dilation outperforming self-attention?

We hypothesize that DNA modeling often prioritizes interactions between adjacent nucleotides, similar to how enzymes process DNA in a sliding window fashion. This local inductive bias of CNNs, particularly with dilation, may offer an advantage over self-attention, which lacks a built-in focus on local context. This is especially evident in tasks like H3K4me2. Additionally, dilated convolutions have lower computational complexity compared to self-attention, further contributing to their efficiency and performance in DNA modeling.

How is the receptive field for the dilated convolutions determined?

the receptive field can be calculated using the formula: $𝑅𝐹(𝑛)=𝑅𝐹(𝑛−1)+(𝑘−1)×𝑑$ where $𝑅𝐹(𝑛)$ is the receptive field of the current layer, $𝑘$ is the kernel size, and $𝑑$ is the dilation rate. If your question is about how the dilation rate ($𝑑$) is determined, we select it based on ablation study results (coming soon). Additionally, since ConvNova is designed as a DNA foundation model capable of handling long-range tasks, we choose a dilation rate of 4 to balance these requirements.

What factors contribute to the dual-branch design outperforming a single-branch approach? The dual-branch design facilitates independent feature extraction and promotes complementary representation learning. Because of the gating mechanism, dual independent convolution can add more expressiveness than single branch.

I found the organization of the paper somewhat confusing. For instance, the comparison of downsampling and dilation in Section 3.3 and Table 1 would be more appropriately placed in the experiments section. Additionally, given the variety of tasks with different state-of-the-art models and settings, I recommend adding a dedicated section that briefly outlines the experimental setup. This section should include the objectives of each task, descriptions of the baseline models, and the specific configurations used in each experiment.

Thank you for your feedback. We agree that the comparison of downsampling and dilation would be more appropriately placed in the experiments section. We will reorganize the paper as suggested and include a dedicated section outlining the experimental setup, including the objectives of each task, descriptions of the baseline models, and the specific configurations used. We will inform you once the revisions are complete. Thank you again for your valuable input!

Beyond demonstrating superior accuracy compared to Transformers and SSMs, it would be valuable to include an in-depth discussion and comparison with models like LegNet on block design, architecture, parameter count, and training schemes in the main text.

Thank you for the suggestion. As our main focus mentioned above, we do not prioritize detailed comparisons with specific models like LegNet in the main text. However, we appreciate the importance of such comparisons and have provided the following details for reference: .

Training Schema

1. LegNet's original training involved prediction of expression bin probabilities, but such tasks are absent in the foundation model benchmarks. Thus both models use original labels for supervision.
2. LegNet uses Lion optimizer in their original implementations, but for fair comparison, we use AdamW consistently across all models.
3. LegNet is trained in a fully supervised, from-scratch manner, whereas ConvNova leverages pretraining.
4. Even without pretraining, a from-scratch ConvNova outperforms LegNet. You can see with Table 13 and the results coming soon.

Parameter Count ConvNova has multiple versions to ensure fair comparisons with different models. For comparison with LegNet, we used a 1.7M parameter version of ConvNova, while LegNet has 2.1M parameters(with the output head).

Block Design

1. ConvNova incorporates dual-branch structures and dilated convolutions, which are absent in LegNet.
2. ConvNova’s gating mechanism is implemented using convolution, whereas LegNet uses an MLP for gating.
3. ConvNova maintains a fixed hidden dimension throughout, while LegNet adopts a progressively shrinking hidden dimension sequence: 256, 128, 128, 64, 64, 64, 64.
4. LegNet employs group convolutions to reduce parameter count, while ConvNova does not.

Actually, we do not see much similarity between these two models. We hope this helps clarify the distinctions.

We sincerely appreciate Reviewer 2bPc's thoughtful feedback, and here we provide corresponding responses to address these concerns.

While the authors present solid empirical evidence, the paper lacks a clear discussion on the intuition behind the design, making it more like a technical report. For example, ConvNeXt [1] builds on a well-known variant, ResNeXt [2], and details the accuracy impact of each modification. In contrast, ConvNova does not seem to establish a clear rationale connecting it to previous CNN-based DNA foundation models, which makes the design choices appear somewhat ad hoc.

In recent foundation model research, CNNs have been largely ignored and not compared with existing methods. In fact, previous CNN models, such as Basenji, performes poorly across tasks(See Table 13). We are the first to compare CNNs with other architectures, demonstrating that with the right design, CNNs can outperform them. Indeed, this answers the question: CNNs are still not surpassed by other architectures.

The motivation behind our architecture design are following several key considerations. The gating mechanism could enable dynamic feature selection. The dual-branch design facilitates independent feature extraction and promotes complementary representation learning. Dilation serves two purposes: (1) downsampling, and (2) investigating the local dependency condition of downstream tasks.

However, unlike convNext, which can use ImageNet a single unified metric, DNA-related tasks are diverse and lack such benchmarks. Thus, instead of detailed analyses like ConvNeXt's, we will provide ablation studies on dilation rates and kernel sizes, offering insights into their impact across tasks soon. We will inform you once available.

The main issue with self-attention is that it does not inherently focus on neighborhood sequences, which could contribute to their suboptimal performance in many DNA modeling tasks. To support our claim, we modify the NTv2 model by adjusting the RoPE's 𝜃 and initializing the bias of the 𝑞𝑘 linear layer to [0, 0, ..., 1], while keeping other initializations consistent with NTv2 (𝑠𝑡𝑑=0.02, 𝑚𝑒𝑎𝑛=0). This adjustment increases the attention map's focus on neighborhood sequences, enhancing the Transformer's inductive bias.

Our approach is similar to the methodology in [1]. When tested on H3K14ac, a task with strong local dependencies, the results improved significantly (34.42 vs. 46.65 with the enhancements).

However, this method provides only a naive way to add inductive bias to Transformers. It still does not surpass ConvNova trained from scratch. Further research is needed to fundamentally strengthen the Transformer's inductive bias for neighborhood modeling.

While this is not our main focus, we still hope this addresses your concerns.

[1] RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality. CVPR. 2022

Thank you for your insightful comments. However, the premise of your question may be problematic. CNNs have long been established in DNA modeling [1],[2],[3], while SSMs and Transformers emerged later. Additionally, the DNA foundation model works [4],[5],[6],[7] did not compare Transformers and SSMs directly with CNN-based foundation models when introduced. Therefore, there is no evidence to suggest that Transformers and SSMs outperform CNNs.

The key question in this field isn't why CNNs outperform Transformers and SSMs, but rather whether Transformers and SSMs truly surpass CNNs, and if so, why that might be the case. Given the current focus in the field on Transformer and SSM architectures, revisiting CNNs and including them in comparisons is both necessary and timely. Our paper addresses this by showing that under well-designed settings for CNNs, neither Transformers nor SSMs have surpassed CNNs in this domain.

However, we can still offer some insights into why CNNs can outperform Transformers and SSMs. CNNs inherently possess inductive biases for neighborhood modeling, which may be crucial for DNA sequence tasks. Additionally, CNNs have lower computational complexity compared to Transformers and SSMs, making them more efficient.

Regarding the kernel size and dilation rate experiments, these are not the central focus of our paper. They are included to ensure rigor and provide a more comprehensive evaluation.

We encourage you to revisit the settings and focus of our paper, which may further clarify our contributions. Your feedback is greatly appreciated, and we welcome continued discussion on this topic!

[1] Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks. Genome Res. 2016

[2] Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome Res. 2018

[3] Predicting effects of noncoding variants with deep learning-based sequence model. Nat Methods. 2015

[4] The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics. bioRxiv. 2023

[5] DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome. ICLR. 2024

[6] HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution. NeurIPS. 2023.

[7] Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling. ICML. 2024

Beyond demonstrating superior accuracy compared to Transformers and SSMs, it would be valuable to include an in-depth discussion and comparison with models like LegNet on block design, architecture, parameter count, and training schemes in the main text.

We have added the comparison between LegNet and ConvNova(trained from scratch) in Table 13 to showcase the superiority.

I found the organization of the paper somewhat confusing. For instance, the comparison of downsampling and dilation in Section 3.3 and Table 1 would be more appropriately placed in the experiments section. Additionally, given the variety of tasks with different state-of-the-art models and settings, I recommend adding a dedicated section that briefly outlines the experimental setup. This section should include the objectives of each task, descriptions of the baseline models, and the specific configurations used in each experiment.

We have moved the comparison of downsampling and dilation to Section 4.4, as you suggested. Additionally, we have made modifications and added further descriptions regarding the objectives, baseline models, and configurations of each task in Section A2.

How is the receptive field for the dilated convolutions determined?

If your question is how we determine the dilation rate, we have included relevant experiments and details in A3.2, Tables 11 and 12.

We appreciate your insights and hope these changes address your concerns. Please don't hesitate to reach out if you have any further questions.

Thank you for your message. We have included the relevant details in the paper revision. Please refer to A5, Figure 5, and Table 15 for the updated information.

Let us know if you need further clarification!

I would say that calling the model a “foundation model” is somewhat misleading as the model used in the paper has 7M parameters and was pretrained for 14 hours on a single GPU.

Thank you for the comment. Following prior works like HyenaDNA, DNABERT-2 and Caduceus, we adopt the term "foundation model." While our model is relatively small, we feel its evaluation across such diverse tasks justifies this terminology, though we appreciate differing interpretations.

Is the architecture inspired by the DNA task in some way?

Yes. The key factor influencing robust performance is the carefully designed dilation mechanism. We will provide the ablation for dilation, kernel size as well as local dependency for tasks soon. We will inform you once available.

We sincerely appreciate Reviewer SH7n's thoughtful feedback, and here we provide corresponding responses to address these concerns.

The novelty of the approach is somewhat limited, as CNNs have already been applied to this setting, and the training procedure is broadly the same as in the HyenaDNA paper.

The CNN architecture used is based on gated convolutions, which has already been proposed elsewhere, limiting the novelty.

To better understand the novelty of the approach, a benchmark comparing with a more standard CNN architecture would have been useful.

The paper seems to lack a clear motivation as to why this specific CNN architecture was proposed.

In recent foundation model research, CNNs have been largely ignored and not compared with existing methods. We are the first to compare CNNs with other architectures, demonstrating that with the right design, CNNs can outperform them. Indeed, this answers the question: CNNs are still not surpassed by other architectures. Thus, novel architecture is not the primary focus of our work.

The motivation behind our architecture design is following several key considerations. The gating mechanism could enable dynamic feature selection. The dual-branch design facilitates independent feature extraction and promotes complementary representation learning. Dilation serves two purposes: (1) downsampling, and (2) investigating the local dependency condition of downstream tasks.

Standard CNN models like Basenji and LegNet are designed for specific tasks and lack the generalizability required for DNA foundation modeling (See Table 13). This broader scope differentiates our work from prior efforts.

I would have expected to find a description of the pretraining data in the main paper.

Thank you for pointing this out. In our paper, we mention that our pretraining paradigm follows HyenaDNA, which implicitly includes a description of the pretraining data. However, we appreciate your suggestion and will explicitly clarify the details of the pretraining data in A.1 to ensure there is no ambiguity. We will inform you as soon as we've made the revision.

We have added a description of the pretraining data in Section 3.3 of the latest revision.

We also have added the motivation why we adopt the dual-branch design in Section 3.2. The formula in Section A.3.1 has been revised in the hope that it will help you discover how the dual-branch structure with two independent convolutions provides greater expressiveness compared to a single-branch convolution.

More ablation experiments are conducted in Section A.3.2, table 11 and table 12 to explain why we design the dilation rate mechanism.

We hope this provides the clarity you were looking for, and please feel free to reach out if you have any further questions.

Dear reviewer:

Here we provide the motivation to consider CNNs against other models for DNA modeling tasks, which has not been discussed in other work.

Besides the computational complexity and model sizes of CNNs, CNNs inherently possess inductive biases for neighborhood modeling, which may be crucial for many DNA sequence tasks. We have already added relevant experiments and analyses; please refer to A5, Figure 5, and Table 15 for the updated information.

Here we briefly explain how we support our hypothesis.

Specifically, the main issue with self-attention is that it does not inherently focus on neighborhood sequences, which could contribute to its suboptimal performance in many DNA modeling tasks. To support our claim, we modify the NTv2 model by adjusting the RoPE's 𝜃 and initializing the bias of the 𝑞𝑘 linear layer to [0,0,…,1], while keeping other initializations consistent with NTv2 (std=0.02,mean=0). This adjustment increases the attention map's focus on neighborhood sequences, thereby enhancing the Transformer's inductive bias.

Our approach is similar to the methodology in [1]. When tested on H3K14ac, a task with strong local dependencies, the results show significant improvement (34.42 vs. 46.65 with the enhancements). Similar results on other 2 tasks can be found in Table 15.

However, this method provides only a naive way to add inductive bias to Transformers. It still does not surpass ConvNova trained from scratch. Further research is needed to fundamentally strengthen the Transformer's inductive bias for neighborhood modeling.

We welcome any additional questions or feedback you might have and would be glad to engage in further discussion.

[1] RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality. CVPR. 2022.

We sincerely appreciate Reviewer PceR's thoughtful feedback, and here we provide corresponding responses to address these concerns.

The paper needs to do a better job, in the main text, of demonstrating what specifically about the model is novel, and what specifically contributes to the superior performance of this model compared to prior CNN-based DNA models. For example, is it just better pretraining data? Is it including the complement sequence? I saw dilations, and receptive field choice were shown to be a portion of this, and I saw the ablations on the different components.

However, the specific novelty of this design is a bit ambiguous to me. The figures need to be clearer to emphasize the novel components compared to prior CNN-based models. Specifically, just answering the question in explicit detail: CNNs have been around for a while, why didn't we find a high-performing model like this well before this? What new has been added that wasn't found before?

In recent foundation model research, CNNs have been largely ignored and not compared with existing methods. In fact, previous CNN models, such as Basenji, performes poorly across tasks(See Table 13). We are the first to compare CNNs with other architectures, demonstrating that with the right design, CNNs can outperform them. Indeed, this answers the question: CNNs are still not surpassed by other architectures.

Your points are insightful and highlight considerations that, while not our primary focus, remain important. Through additional ablation experiments(we will provide the results soon), we find that the most critical factor is the appropriate dilation mechanism, as deviations from optimal dilation rates lead to performance degradation. Other design choices, such as the dual-branch architecture, pretraining data, and gating mechanism, also contribute to performance improvements, as shown in our ablation studies Table 5.

As for the question, "why didn't we find a high-performing model like this well before this?" the main reason is that this is an underexplored area. We are the first to systematically evaluate CNNs across such a broad range of tasks. What's more, traditional CNN designs(like DeepSEA) often rely on pooling operations, which are generally unsuitable for masked language modeling. By addressing these gaps, we demonstrate that with thoughtful design, CNNs can perform robustly as DNA foundation models.

There should be some more qualitative analysis in the main text, mainly I would like to see an understanding of how the different model classes compare in the types of errors they make. For example, does the smaller receptive field contribute to certain types of errors for tasks where longer range reasoning is necessary?

This is an excellent question. From our experiments, we observe that sufficiently large pretrained transformers(like NTv2) excel at tasks with short sequences (a few hundred bp) but minimal local dependencies, such as splice site donor and acceptor prediction. However, transformers struggle on tasks like H3K4me2, which heavily rely on local dependencies. This may be due to the lack of an inductive bias in self-attention that emphasizes adjacent tokens, as CNNs naturally do.

SSM models, on the other hand, perform well on long-range tasks, and they do not exhibit specific type of error.

Traditional CNNs, such as LegNet, often underperform on targeted tasks like splice site prediction. However, with our design adjustments, CNNs achieve robust performance across most tasks, including long-range ones like gene finding(Table 2) and variant effect prediction:

We hope these clarifications and additional results address the concerns and demonstrate the value of our approach.

We have provided ablation studies on dilation and kernel size in A3.2, Tables 11 and 12. For a comprehensive understanding, we recommend reviewing these tables alongside Table 5 to observe the impact of our proposed components on performance. We hope this helps clarify the details, and please feel free to reach out if you have any further questions.

Dear reviewer:

Here we provide the motivation to consider CNNs against other models for DNA modeling tasks, which has not been discussed in other work.

Besides the computational complexity and model sizes of CNNs, CNNs inherently possess inductive biases for neighborhood modeling, which may be crucial for many DNA sequence tasks. We have already added relevant experiments and analyses; please refer to A5, Figure 5, and Table 15 for the updated information.

Here we briefly explain how we support our hypothesis.

Specifically, the main issue with self-attention is that it does not inherently focus on neighborhood sequences, which could contribute to its suboptimal performance in many DNA modeling tasks. To support our claim, we modify the NTv2 model by adjusting the RoPE's 𝜃 and initializing the bias of the 𝑞𝑘 linear layer to [0,0,…,1], while keeping other initializations consistent with NTv2 (std=0.02,mean=0). This adjustment increases the attention map's focus on neighborhood sequences, thereby enhancing the Transformer's inductive bias.

Our approach is similar to the methodology in [1]. When tested on H3K14ac, a task with strong local dependencies, the results show significant improvement (34.42 vs. 46.65 with the enhancements). Similar results on other 2 tasks can be found in Table 15.

However, this method provides only a naive way to add inductive bias to Transformers. It still does not surpass ConvNova trained from scratch. Further research is needed to fundamentally strengthen the Transformer's inductive bias for neighborhood modeling.

We welcome any additional questions or feedback you might have and would be glad to engage in further discussion.

[1] RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality. CVPR. 2022.

Thank you for your thoughtful feedback and suggestions. Based on your recommendations, we conduct additional explorations to investigate the qualitative differences in the types of errors made by different model classes. The conclusion from our analysis is that there are no significant differences in the error sequences between the model classes. The results of our analysis can be found in this anonymous link: https://anonymous.4open.science/r/Error-Analysis-AB32.

Specifically, we focus on the H3K14ac task, which exhibited significant performance differences among the models (70.71 vs. 60.84 vs. 57.22). We analyze the error sequences from ConvNova, Caduceus, and NTv2 using t-SNE to visualize the distribution of error points from various angles. However, no clear qualitative differences are observed in the distributions t-SNE figures.

To further ensure rigor, we train a linear SVM on the error sequences to verify whether a hyperplane could effectively separate the errors from different models(0 for ConvNova and 1 for Caduceus). The results are shown below:

Similar results are observed in another test where 1 represents error sequences from ConvNova and 0 from NTv2, as shown below:

These results suggest that no significant qualitative differences exist in the types of errors made by different model classes numerically.

We have also used traditional DNA sequence clustering algorithms including MMseq2, ALFATClust and MeShClust v3. However, these sequence-identity-based algorithms still do not yield other insights and conclusion.

We welcome further discussion and appreciate your valuable insights. Please do not hesitate to share additional thoughts or suggestions!

We sincerely appreciate Reviewer NQxy's thoughtful feedback, and here we provide corresponding responses to address these concerns.

For Weaknesses:

(W1) Despite the efficient design, the proposed ConvNova is somewhat simple and lacks novelty and support. The (dilated) convolution with gating branch is not a new design (proposed by MogaNet [1] in 2022 and well studied by StarNet [2] and MambaOut [3] in 2024), especially after Mamba variants came out (i.e., the implicit long convolution with gating). The authors should discuss these background works (discussing in the related work section or comparing with them), and provide more supports of why the proposed design is specially useful in DNA applications (refer to Q1 for details).

In recent dna foundation model research, CNNs have been largely ignored and not compared with existing methods. Previous CNN models, such as Basenji, perform poorly across tasks. We want to clarify that our primary contribution is not architectural novelty, but being the first to propose and compare traditional CNN-based DNA foundation model with other architectures, demonstrating that with the right design, CNNs can outperform them. Indeed, this answers the question: CNNs are still not surpassed by other architectures.

While the architectural design is not our main focus, we would like to address the concerns about the dilation with gating mechanism. The key factor driving the model's performance is not gating but rather the carefully chosen dilation design. We have compared U-Net-style downsampling approaches and performed ablation studies on different dilation rates, selecting the current design based on the requirements of foundation models for long-range tasks. Moreover, dilation help to investigate the down stream tasks' local dependency. For instance, larger dilation sizes (e.g., 4) improve performance on H3 (81.49 vs. 77.16), while smaller sizes (e.g., 1) are better for H3K4me3 (67.15 vs. 60.20).

As for the gating mechanism, while it is a known concept, its use in ConvNova is empirically validated for effectiveness in DNA modeling(Table 5). While our approach differs in several aspects, we acknowledge the contributions of MogaNet, StarNet, and MambaOut.

• (W2) Although the authors have compared with several well-known DNA models, some recently published DNA models and pre-training works are overlooked (e.g., GPN [4], VQDNA [5], and DNABERT-S [6]). Meanwhile, there are various DNA benchmarks that should be referred to and compared (classical benchmarks like GUANinE v1.0 [7], BEND [8], and GUE [9]), especially some long-range benchmarks [10] where SSM-based models work well. From my perspective, I am still not sure or not convinced that the dilated convolution with gating aggregation could consistently outperform self-attention or SSM architectures.

Thank you for the valuable feedback. We coduct additional comparisons with GPN and DNABERT-2 on the GUE benchmark. As shown in the following results, ConvNova outperforms the baseline in 19 out of 28 tasks.

Epigenetic Marks Prediction

Epigenetic Marks Prediction and Promoter Detection

Transcription Factor Prediction (Human) and Core Promoter Detection

Transcription Factor Prediction (Mouse) and Virus & Splice Detection

Regarding VQDNA, it is not included in the comparison as our work focuses on CNN-based models versus self-attention or SSM-inspired architectures. VQDNA uses vector quantized tokenization and the DNABERT-2 architecture, which falls outside the scope of this study. Additionally, as VQDNA lacks open-source code, future work may explore comparisons with VQ-based settings.

For the long-range benchmarks mentioned, the lack of open-source code presents a limitation. However, we complete the Variant Effect Prediction task across three tasks. As shown in the results, ConvNova successfully models long-range dependencies.

Variant Effect Prediction

For Questions:

(Q1) Is there any empirical analysis or theoretical support to verify that the dilated convolution is capable and more efficient than self-attention or SSM modules on the DNA tasks? For example, the authors could visualize the reception field or analysis the learned patterns to show that the dilated convolutions with gating learn better.

For efficiency, dilated convolutions possess lower computational complexity compared to self-attention and SSM modules. For capability, we have conducted extensive experiments with varying sequence lengths, and the results demonstrate the effectiveness of ConvNova across DNA tasks. As for a potential explanation, we hypothesize that, similar to how enzymes interact with DNA in a sliding window fashion, DNA modeling may prioritize adjacent nucleotides. In this sense, the inductive bias of CNNs might be more advantageous than self-attention models, which may not emphasize local context in the same way. This could be a key reason for the success of CNN-based models in this domain.

While we do attempt to visualize the receptive fields and analyze the learned patterns, we do not observe any particularly insightful results. We agree that interpretability is an important aspect, and this might require a more systematic and dedicated effort.

(Q2) Some questions about the hyper-parameters in the network. How to determine the kernel size and the dilated ratio in ConvNova? Are there any implementation details of the ablation studies and providing the corresponding configurations (in addition to Table 9)? Meanwhile, is there more ablation of different designs (like analysis in [1, 2, 3]) of the gating branches as mentioned in Eq. (1) and Appendix A.3?

We will provide additional ablation experiments soon.

(Q2) Some questions about the hyper-parameters in the network. How to determine the kernel size and the dilated ratio in ConvNova? Are there any implementation details of the ablation studies and providing the corresponding configurations (in addition to Table 9)? Meanwhile, is there more ablation of different designs (like analysis in [1, 2, 3]) of the gating branches as mentioned in Eq. (1) and Appendix A.3?

We have provided ablation studies regarding dilation and kernel size in A3.2, Table 11 and Table 12. In short, increasing the kernel size and dilation generally improves performance. Although some tasks may benefit from smaller dilations, we choose a dilation rate of 4 to guarantee long sequences modeling capability for the foundation model. To reduce the number of parameters, we select a kernel size of 9.

Regarding the gating and dual-branch designs, we have revised the ablation study in A3.1. The learning rate, optimizer settings and other settings for this ablation study are kept consistent with ConvNova.

As mentioned above, our goal is not to propose a highly novel CNN architecture, so we did not conduct further comparisons of different designs.

We hope this answers your questions!

Thanks for the detailed response, extensive experiments, and the updated manuscript. Although this manuscript is technically oriented with some straightforward design, it really provided a new insight and useful DNA foundation model, which are verified by comprehensive benchmarks. What I am still concerned about is W1. I believe the authors should further clarify the novelty of the design model with empirical analysis and well-arranged backgrounds to enhance the contribution of rethinking the pure convolution-based architectures for long-sequence tasks with DNA. Maybe provide a table or timeline figure to illustrate further the motivation that CNNs can still be competitive for the genomic tasks. Overall, after going through comments from other reviewers, I decided to raise my score to 6 and encourage the authors to further polish the manuscript and provide more ablation experiments.