Rethinking the bert-like pretraining for dna sequences

Q2: The authors say they progressively expand the masking boundary to prevent easy learning. Shouldn't it be the other way around? The reviewer believes that the masking boundary should progressively contract. What is the point of showing the shorcut first and complicating the learning? The model would already learn the shortcut if the masking boundary progressively expanded. An additional experiment on this needs to be conducted.

A2:

Curriculum learning was proposed by Yoshua Bengio in 2009. It is a training strategy that mimics the human learning process by presenting the examples in an easy-to-difficult order. It has been shown to improve the performance and convergence stability of various Large language model.

The paper in ACL2020 that explore curriculum learning for Large language Model, titled Curriculum Learning for Natural Language Understanding. This paper proposes a curriculum learning approach that reviews the training set in a crossed way to distinguish easy and hard examples, and arranges a curriculum for Large language Model on various NLU tasks. We wanted to increase the difficulty of the DNA sequence pre-training task and at the same time ensure the stability of the training, so we adopted a course learning solution.

In the following table, the best performance results are represented by boldface, and the second best performance results are represented by <u>underline marks</u>. We tried consecutive mask6, 12, 14, 28 and 40 tokens during the entire pre-training process. Finally, it was found that this RandomMask from mask 6 to mask 14 achieved the best performance.

We further tested both options from easy to hard and from hard to easy. In the table below, we divide pre-training into two stages equally. Mask 6 to 12 means masking 6 tokens continuously in the first stage and masking 12 tokens continuously in the second stage. Mask 12 to 6 means masking 12 tokens continuously in the first stage and masking 6 tokens continuously in the second stage. It can be found that the former has better performance.

Q1: In the description of Table 2, what are MCC and PCC? The reviewer is aware that they are later explained in Experiemnt section, but what they are and what they do need to be briefly explained in the description of Table 2 as well for the readers who are not in life sciences field.

A1: I have put the following content into the description of Table 2 or Supplementary Material.

Matthews Correlation Coefficient (MCC) and Pearson Correlation Coefficient (PCC) are both statistical measures used to assess the relationship between variables. The maximum values of MCC and PCC are both 100%. The larger these two indicators are, the better the model performance is.

1. Matthews Correlation Coefficient (MCC): MCC is primarily used in binary classification problems to measure the performance of binary classification models. The formula for MCC is:

   $$MCC = \frac{TP \times TN - FP \times FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}$$

   where:

   • TP = Number of True Positives
   • TN = Number of True Negatives
   • FP = Number of False Positives
   • FN = Number of False Negatives

   True Positives are instances where the model predicts the sample as positive, and it is indeed positive. True Negatives are instances where the model predicts the sample as negative, and it is indeed negative. False Positives are instances where the model predicts the sample as positive, but it is actually negative. False Negatives are instances where the model predicts the sample as negative, but it is actually positive.

2. Pearson Correlation Coefficient (PCC): PCC is used to measure the strength and direction of the linear relationship between two continuous variables. Let $\hat{y_i}$ be the model's predicted value, and $y_i$ be the actual value of the sample. The formula for PCC is:

$$PCC = \frac{\sum_{i=1}^n (\hat{y_i} - \bar{\hat{y}})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n (\hat{y_i} - \bar{\hat{y}})^2 \sum_{i=1}^n (y_i - \bar{y})^2}}$$

where:

• $\hat{y_i}$ = Predicted value of the sample by the model
• $y_i$ = Actual value of the sample
• $\bar{\hat{y}}$ = Mean of the model's predicted values
• $\bar{y}$ = Mean of the actual values

EMP, TF-M, TF-H, PD, CPD, and SSP are classification tasks, so the evaluation metric for the model on these tasks is MCC. EAP is a regression task, therefore the evaluation metric for the model on this task is PCC.

[Clarity]: The experiment and result section is relatively short. Some more clarifications would be helpful. For example, (a) when describing the baseline, the author mentioned “All models are trained on human genome and fine-tuned on the benchmark datasets with identical settings.” The “identical” setting does not bring clarity on how the mask is generated. Does this mean all the algorithms are trained and fine tuned with 15% token masked? This setting is slightly different than the original setting in the “Nucleotide Transformer” paper. (b) “Finetuning” section discussed the dataset in a detailed way; however, the details of the algorithm setup in the fine tuning is not discussed.

A to Clarity: We will additionally add detailed settings to the revised paper and open source our code.

a) "Pretraining": For Nucleotide Transformer, hyenaDNA, and DNABERT, we used the open source pre-training weights from huggingface. When we pretrain DNABERT using RandomMask (DNABERT+RandomMask), we just modified the strategy for masking with RandomMask. The Other settings, such as the model structure, training steps, and learning rate, remained consistent with the settings of DNABERT's paper. We have made changes to the introduction of pretraining in the experimental section.

b) "Finetuning": In the fine-tuning stage, the fine-tuning parameters for NT are detailed in the A.7 of supplementary material's final section. The fine-tuning parameters for DNABERT and DNABERT+RandomMask are identical and their specific values can be found in the A.7 of supplementary materials. HyenaDNA used the fine-tuning parameters, being consistent with HyenaDNA's paper, can also be found in the A.7 of supplementary materials.

[Quality] The value of RandomMark (or training with gradually increased difficulty) can be better verified through experiments. For example, the author presented the result using the training with 5 phases. It would be great if the author could compare the result with different numbers of phases: one phase with maximum difficulty, two phases with two different difficulties, and etc.

A to Quality:

(1) Two phases.

In the following table, we tested both options from easy to hard and from hard to easy. In the table below, we divide pre-training into two stages equally. Mask 6 to 12 means masking 6 tokens continuously in the first stage and masking 12 tokens continuously in the second stage. Mask 12 to 6 means masking 12 tokens continuously in the first stage and masking 6 tokens continuously in the second stage.

In the following table, the best performance results are represented by boldface

It can be found that the easy to hard strategy has better performance. This verifies the rationality of the course learning strategy from easy to difficult in BERT-like DNA pre-training.

(2) Compare one and two phase.

We tried consecutive mask 6, 12, 14, 28 and 40 tokens during the entire pre-training process. DNABERT + RM (RandomMask) means use RandomMask to pretrain DNABERT.

In the following table, the best performance results are represented by boldface, and the second best performance results are represented by *.

From the table above, we can see that the length of the continuous mask is not longer, the better. The optimal performance of various tasks is concentrated around mask 12.

From the above results, we designed a RandomMask strategy that gradually increases the task difficulty in five stages from 8 to 14.

Q1: Algorithm 1 is not very clear to me. For example, if the r <= p at position i, [i − m/2 + 1, i + m/2] will be masked. The following nucleotide should be i+1, or I+m/2+1? If the next one is i+1, the mask range could be overlapping and can grow to be very long.

A1: If the r <= p at position i, [i − m/2 + 1, i + m/2] will be masked. the following tokens should be i+1. In fact, the mask range can grow to be very long. We conducted the following experiments when selecting strategies.

In the following table, the best performance results are represented by boldface, and the second best performance results are represented by *. We tried consecutive mask 6, 12, 14, 28 and 40 tokens during the entire pre-training process. Finally, it was found that this RandomMask from mask 6 to mask 14 achieved the best performance.

When setting a continuous mask of 14 and mask rate p=2.5%, approximately 25% of the tokens in the entire sequence are covered. In other words, in this case, the maximum number of tokens that can be continuously covered is not too long, no more than a quarter of the sequence length.

It is not that the longer the continuous mask, the better. When the number of consecutive masks is greater than 14, it is possible that the mask may have too long consecutive tokens due to the overlap you mentioned, further leading to a decline in model's performance.

Q5: The paper's presentation could be improved in several ways:

1. The introduction is somewhat verbose, indirectly causing the first two weaknesses and making the paper hard to read.
2. Placing Figure 1 and Table 1 on page 1 would improve readability, given that the main content describing Figure 1 and Table 1 is in the first page.
3. The separate table on the left in Table 2 appears to be redundant.
4. The experimental settings in Section 3 lack detailed descriptions, potentially making reproduction difficult and potentially misleading.
5. A thorough proofreading could enhance the clarity of writing and word choice.

A5: Thanks for your valuable suggestions, we have considered all the suggestions and improved the paper's presentation as much as we can. Besides, we will ask senior researchers who have published more than ten top conference papers to revise the manuscript to ensure that the article is concise and easy to read.

Q6: It would be beneficial to further explore whether sparse attention is indeed a problem for DNA sequence representation. Sometimes, sparse attention can improve generalization [1]. This might depend on different sub-sequences and the various functions of different layers when modeling cross-attention. I would appreciate further elaboration on the limitations of sparse attention in DNA sequence representation.

[1] Correia, et al. "Adaptively Sparse Transformers." Conference on Empirical Methods in Natural Language Processing (2019).

A6: In A.5 of the Supplementary Material, we show that DNABERT faces the problem of fixed attention weight patterns rather than the problem of sparse attention weights. Specifically, for DNABERT-6mer with open source weights, the pattern of attention weights in intermediate layers such as layers 4 and 5 is very fixed (only focusing on [CLS] token). Such results reveal that the original DNABERT is not sufficiently trained. Thus, we propsoe RandomMask to alleviate this problem.

For sparse attention, we find some discussions in GENA-LM [2]. The following table results are from GENA-LM. As we can see, sparse attention does show significant gains in some down-stream tasks like enhancer-development and polyadenylation site prediction, but other tasks performed essentially the same or worse.

[2] Fishman, Veniamin, et al. "GENA-LM: A Family of Open-Source Foundational Models for Long DNA Sequences." bioRxiv (2023): 2023-06.

Q4: The comparison in Observation 1 does not seem to be an apples-to-apples comparison. Overlapping represents more patterns and creates longer sequences for the same DNA length. It would be beneficial to understand if the conclusion holds for different lengths of k-mer.

A4: Follow the reviewer's suggestions, we investigate the effect of sequence length. The following table is an expansion of Table 2 in the paper. The row named Same-lenth is to copy the Non-overlapping tokens five times to obtain the same sequence length as Overlapping. (i.e. Non-overlapping: token1 token2; Same-lenth: token1 token2 token1 token2 token1 token2 token1 token2 token1 token2 token1 token2.)

The best performance results are represented by boldface, and the second best performance results are represented by *.

1. An interesting phenomenon. In NT that uses non-overlapping 6mer for pre-training, stretching the sequence length will indeed produce obvious gains in TF-M, TF-H and CPD. Combined with Table 1 in the paper, the common feature of these three tasks is that the DNA sequence length is short. The DNA sequence lengths of EMP, PD, SSP and EAP are 500, 300, 400 and 250 nucleotides respectively. However, the DNA sequence lengths of TF-M, TF-H and CPD are 100, 100 and 70 nucleotides respectively, and these are shorter than others.

2. But in general, the overlapping tokenizer to obtain more diverse tokens get better performance than simply lengthening the sequence length (Same-lenth) both of overlapping pretraining (DNABERT) model and non-overlapping pretraining model (NT).

Q2 and Q3：The motivation behind the study is somewhat unclear. Although the authors identify three potential challenges -- rapid convergence, the risk of under-training, and the potential for sparse attention -- they do not adequately explain how RandomMask addresses or mitigates these issues. There is a lack of experimental analysis supporting the source of the observed improvements, which is crucial for substantiating the paper's main claims. For example, besides the quantative improvements, does the rapid convergence and under-training still exist after applying RandomMask?

A2 and A3: This study gives a comprehensive understanding and a specifically tailored approach for bert-like DNA pretraining. Based on a series of exploratory experiments, three insightful observations are given, followed by a simple-but-effective method RandomMask for solving the rapid convergence and under-training problems. Below, we give more experimental analysis supporting the observed improvements of RandomMask:

1. In Figure 3 of Supplementary Material, we can see that DNABERT's loss will quickly decrease to an extremely low value. If RandomMask is used, the loss will increase at the start of each stage, giving it enough space to decrease, so we will see a decrease at each stage in the loss curve with RandomMask. Reviewer 45WA affirmed this point. He emphasized that we enhanced the generalization of the model by increasing the difficulty of the task and reducing the overfitting problem caused by rapid convergence that gradually increasing the complexity during the training could mitigate the overfitting issue and achieve better performance by combining fast convergence and generalizability.
2. In the Figure 1 of Supplementary Material, we can see that the middle layer only pays attention to the [CLS] token, while the first and last layers capture a variety of features. Further, in Figure 2 of Supplementary Material, we see that all tokens in the DNABERT's middle layer pay excessive attention to [CLS]. Therefore the middle layer is not sufficiently trained to capture sequence features. However, as showed in Figure 2 of Supplementary Material, if pre-training with RandomMask, these layers of the model are activated. The attention weights of these layers learn to pay attention to other tokens, and the attention patterns are more diverse. Diverse attention weights indicate that these layers are mroe adequately trained to capture sequence features as the first and last layers do.
3. What RandomMask wants to solve is the fixed attention weight pattern of the middle layers in DNABERT. For sparse attention, different tokens and different layers usually show diverse sparse patterns. However, in Figure 1 and 2 of Supplementary Material, we see that the different layers in the middle of DNABERT and all tokens show the same attention weight pattern, that is, concentrated attention [CLS]. Thus, the problem we're trying to solve and sparse attention are two different things. In Figure 2 of Supplementary Material, we see that adding RandomMask will diversify the attention weight pattern of layers in the middle of DNABERT.

Q1: The authors might consider refining the focus of this work. The true contribution appears to be the improvement of the overlapping strategy tokenization for DNA pretraining, which diverges from the broader theme of "rethinking the pretraining for DNA sequence."

A1: We named our paper as "Rethinking the BERT-like Pretraining for DNA Sequences" based on the following two points:

1. The primary influences on BERT-like pretraining methods are the tokenizer and mask. Our paper discusses both of these key factors: 1) We discuss the impacts of DNA tokenizer in both pretraining and fine-tuning stages, as shown in Figure 1, Figure 2, Table 2 of manuscript and Table 1 of Supplementary Material. In Figure 1 of manuscript, we show the comparisons between commonly used BERT-like DNA tokenizer and NLP tokenizer. In Figure 2 of manuscript, we show the impact of overlapping and non-overlapping tokenizer on the pre-training process and the underlying representation of the model. In Table 2 of manuscript, we show the influence of overlapping and non-overlapping tokenizer on the fine-tuning stage. In Table 1 of the supplementary material, we also compare the performance of 6mer with BPE, with the support of RandomMask, the performance of 6mer is much higher than the most advanced BPE model. 2) We investigate the methods of mask in BERT-like pretraining. In Figure 1, we compare different mask approaches in DNA and NLP. In Table 4 of the supplementary material, we discussed the impact of the number of consecutive mask tokens on performance. In Table 5 of the supplementary material, we discussed pre-training mask task from easy to difficult and from difficult to easy. Based on these ablation experiments, we found the optimal setting strategy for RandomMask.
2. Our contribution includes three experimental observations related to BERT-like DNA pretraining and a simple yet effective method. Reviewer SHyo acknowledged the inspirational significance of these progressive observations, while Reviewer 45WA affirmed that our novel insights differ from traditional views, offering valuable insights for numerous related studies and applications. Based on these observations, our article introduces a simple yet effective method called RandomMask. Therefore, our contribution is not just in optimizing the tokenizer, but in providing three new observations regarding BERT-like DNA pretraining and presenting an effective method based on these insights.

Q3: How does RandomMask compare on the benchmark datasets listed in HyenaDNA?

A3: Follow the reviewer’s suggestion, we compare RandomMask on the benchmark datasets listed in HyenaDNA:

(1) 18 of the 25 downstream task datasets open sourced by HyenaDNA are already included in the GUE benchmark we used (including EMP, TF-M, PD, CPD, SSP and EAP). These results are showed in the Table 3 of the manuscript. The results of the remaining 7 datasets are as follows.

(2) In the table below, the first row shows the results of CNN on the 7 datasets. The second row is the result of two layers of Transformers. The third row is the result of two layers of Hyena. These three rows of results are all from the original paper of HyenaDNA. We also test open source DNABERT and our DNABERT + RM (RandomMask), and show the results in the fourth and fifth rows respectively. Because the results in the original HyenaDNA's paper are accurate to one decimal place, our test results in this table are also kept to one decimal place. The best performance results are represented by boldface, and the second best performance results are represented by *.

Comparing the last two rows, we can find that except Human vs Worm, DNABERT+RM is always better than DNABERT. And the average performance of DNABERT+RM far exceeds DNABERT. This proves the superiority of RandomMask. Compared to other methods, DNABERT+RM achieves the best performance on four of the seven datasets. DNABERT+RM achieve second-best performance on two of the three datasets where performance was not the best. And DNABERT+RM are the best in terms of average performance across seven datasets.

Q1: How does the RandomMask method compare with alternate tokenization approaches such as BPE (proposed in DNABERT-2)?

A1: Follow the reviewer's suggestion, we conduct comprehensive experiments to compare our RandomMask (RM) method with BPE and DNABERT-2. Here we name the open source DNABERT-2 model as DNABERT2-BPE to distinguish it from the DNABERT2-6mer that we trained.

(1) Compare DNABERT+RM and DNABERT2-BPE

In the table below, we compare our DNABERT + RM (RandomMask), with open source DNABERT and DNABERT2-BPE, the best performance results are represented by boldface, and the second best performance results are represented by *.

As you can see from the table below, the performance of our pre-trained DNABERT+RM is slightly better than DNABERT2-BPE.

(2) Compare DNABERT2+6mer and DNABERT2-BPE

In addition to using the BPE tokenizer method, DNABERT-2 also changed the model architecture, adopted a new activation function, and the pre-training data are larger multi-species DNA data.

In order to have a fair comparison, we trained DNABERT2-6mer and DNABERT2-6mer + RM (RandomMask) using 6mer tokenizer under the same architecture, pre-training data and pre-training hyperparameters of DNABERT-2's paper. The results are shown in the following table, the best performance results are represented by boldface, and the second best performance results are represented by *.

In the table above, we see the results in the first and second rows and find that if we just replace the BPE with 6mer, the performance of the model will decrease. The third line is the model performance after using RandomMask. It can be seen that the performance of the 6mer model has been greatly improved after using RandomMask, far exceeding DNABERT2-BPE.

Q2: How does the RandomMask improve the internal representations - Can you see visible differences in embedding representations for downstream tasks (e.g Biotype Embeddings shown in HyenaDNA)

A2: Since HyenaDNA does not open source the code and biotype dataset for this task. Therefore we tried it on our dataset.

Following the motivation of biotype, we selected EMP-Yeast, TF-Human, TF-Mouse and Prom-Human, covering three species and three functional originals. We used DNABERT and DNABERT + RandomMask to generate embeddings for these datasets, and performed t-SNE for visualization. We put the visualization results in A.8 of the Supplementary Material.

We can get the following conclusion. Compared with the original DNABERT, the model pre-trained using RandomMask can produce clearer boundaries and the separation between different species is more obvious.