BSM: Small but Powerful Biological Sequence Model for Genes and Proteins

Thank you very much for your valuable feedback. We address your questions point by point below.

Q1: The main motivation of this paper is to use the small language model (SLM) for biological sequences. But why do we need SLM for biological sequence analysis?

We would like to emphasize that our main contribution lies in addressing an important but often overlooked issue in current research: exploring rich and high-quality data, particularly cross-modal bio-sequence data, which has rarely been studied or utilized in this field. Existing work has focused on scaling models to billions of parameters to achieve strong performance. In contrast, we demonstrate that introducing cross-modal data can enable smaller models to achieve performance close to or even surpass that of billion-scale models. This highlights the importance of data.

Additionally, we show that scaling model size continues to improve performance with this data. Due to resource limitations, we were unable to further scale the model size, but our work strongly demonstrates the tremendous potential of expanding both cross-modal data and model size.

Q2: It may be convincing, if the SLM performs better than larger model due to the limited availability of data. However, as shown in various experiments, larger models perform better than smaller models. For example, Figure 3 (b) LucaOne performs better than BSM models, and BSM-270M model performs better than BSM-110M model.

For the statement, "It may be convincing, if the SLM performs better than larger model due to the limited availability of data," we would like to clarify that this is not true. Our model used 270B tokens, while larger models like LucaOne-1.8B[1] used 800B tokens. Therefore, we are approaching or surpassing these larger models not due to their use of less data, but because they did not use high-quality cross-modal data as we did. This data can greatly enhance learning efficiency and model performance. In Q3, we added more experiments to support this conclusion.

When we refer to large models, we mean models that are much larger than ours, such as Evo-7B[2], ESM-3B[3], and LucaOne 1.8B[1]. BSM-110M and BSM-270M are our own models, which are much smaller than the others. In comparison with these baselines, we conducted extensive experiments across 8 tasks, including gene, protein, and cross-modal tasks. Among these tasks, we achieved the best results in 4 tasks: ncRIP (note that ncRPI-LGAT is specifically tailored for this task and cannot be extended to others), PPI, ProtLoc, and Zero-shot ncRNA Fitness Prediction. For the remaining 4 tasks, although we did not achieve the best results, our performance was still very close to the best results.

The fact that BSM-270M outperforms BSM-110M demonstrates that our method has scalability, and continuing to scale model size with cross-modal data will lead to even stronger models.

Suggested by other reviewer, we compared our models with ESM-150M, which is similar in size to ours. The results show that for models of the similar size, our performance is significantly higher. Additionally, the baseline models we report for larger sizes perform much better than ESM-150M. By incorporating data, BSM-110M/270M can achieve performance close to or even exceeding that of much larger models, demonstrating the significant value of cross-modal data. (Added in Section 3.6)

Q3: I think we need more ablation studies on multi modal data to further demonstrate the necessity of multi-modal pre-training approach.

Thank you for your valueable suggestion. We ran additional experiments on the 110M model, training it on single-modality data with the same token budget as BSM_R2 and BSM_R3. The results show that BSM_R2 outperforms BSM_single_R2, and BSM_R3 outperforms BSM_single_R3. This confirms that performance gains come from incorporating cross-modal data. The results also show that without cross-modal data, BSM_single_R3 is far lower than billion-scale models. However, with cross-modal data, it matches or exceeds their performance. (Added in Section 3.5)

We also added the valid loss curve of BSM-110M-single to Figure 6, showing its valid loss consistently higher than BSM-110M. This aligns with Figure 1, confirming that introducing cross-modal data significantly reduces valid loss and improves single-modal representations.

Q4: How do you choose the data mixing ratio? Is it heuristically decided? How the model performance varies as mixing ratio changes?

In Chapter 2.3, section "Simulated Annealing & Data Mixing," we provide a detailed explanation of how we use the Simulated Annealing technique [4] to select the optimal data mix ratio. We evaluate various data mixing ratios by training the BSM-110M model for 1000/500 steps, with the learning rate (lr) linearly annealed to 0 (Round 2: lr 1e-5→0, Round 3: lr 1e-6→0). The ratio with the lowest validation loss is selected as the final ratio.

The table below shows the validation loss for different data mix ratios we recorded. Results indicate that including RefSeq in Round 2 reduces validation loss compared to excluding it, but further increasing its proportion (i.e., 10:4:0 vs. 10:2:0) does not improve performance, likely due to its limited size, where excessive sampling hurts results. Adding Gene Related Seq data (10:2:1) also improves performance. In Round 3, adding web data enhances performance, with further improvements observed as its weight is increased.

---

[1] LucaOne: Generalized Biological Foundation Model with Unified Nucleic Acid and Protein Language

[2] Sequence modeling and design from molecular to genome scale with Evo, Science 2024

[3] Language models of protein sequences at the scale of evolution enable accurate structure prediction, Science 2023

[4] The Llama3 Herd of Models

 

We want to express our sincere gratitude for your review. If you have any further questions or concerns, please feel free to contact us at any time. If our response has addressed your concerns, we sincerely hope you will consider raising your score.

Best regards,
All Authors

Dear Reviewer RZTE,

We sincerely appreciate your constructive feedback on our manuscript. Guided by your insightful suggestions, we have strived to address each point thoughtfully, which we believe have significantly improved the quality of our work.

Thank you once again for your time and valuable guidance. We're eager to hear your thoughts on these revisions.

 

Best regards,
All Authors

Thank you very much for your valuable feedback. We address your questions point by point below.

Q1: Making paper more concise...Section 4 is largely redundant...Data and experimental details would be better summarized in tables for clarity and ease of comparison..

Thank you for your suggestions. We have made some revisions based on your feedback. We have reduced the repetitive content in Section 4 and added some discussion, and will refine them further in the camera-ready version.

We'd like to explain our choice to present the results using figures instead of tables. Our experiments cover 8 different tasks, including protein, gene, and various cross-modal tasks, each with its own baseline methods. It is challenging to merge these results into shared tables, and presenting them in too many separate small tables would not be effective. We will continue to optimize the presentation of results in future work.

Q2: There are no details about the head used in the experiments in conjunction with the foundation models (FMs). These details are crucial and need to be carefully set for a head-to-head comparison with other FMs.

We apologize if we haven't fully understood your question. If by "head used in conjunction with the foundation models" you are referring to how two models are connected to handle tasks with two input sequences, these experimental results are reported by LucaOne [1], where they used a dual-tower structure. LucaOne conducted comprehensive experiments to compare different structures, and we reported the best results of these structures. In 3 out of the 8 tasks, a dual-tower structure is used for baseline models. We will include the final architecture used for each of these 3 tasks in the appendix in the future.

Our BSM, on the other hand, has the capability to handle any number of sequences. We simply connect the two sequences as one input in BSM, so our experiments do not involve complex architectural choices.

Finally, we would like to emphasize that our newly added ablation study, comparing performance with and without cross-modal data using the same model structure and token budget, shows significant benefits of adding cross-modal data. (Added in Section 3.5)

Q3: Additionally, for other FMs, there could be inverse scaling, where larger models do not consistently outperform smaller ones on all tasks (see McKenzie, et al. (2023). Inverse scaling: When bigger isn’t better. arXiv preprint arXiv:2306.09479). Therefore, it is difficult to conclude that the proposed BSM model performs the best, despite being the smallest, solely due to the introduction of additional data for training.

In all the baseline models we compared, some have only one size, such as LucaOne 1.8B and Evo 7B, while others have a family of models of different sizes. For those with models of different sizes, we want to emphasize that we report the best-performing model for each task.

To further address your concern, we compared our models with ESM-150M, which is similar in size to ours. The results show that for models of the similar size, our performance is significantly higher. Additionally, the baseline models we report for larger sizes perform much better than ESM-150M. By incorporating data, BSM-110M/270M can achieve performance close to or even exceeding that of much larger models. (Added in Section 3.6)

Q4: The differences in performance across models are mostly subtle in many experiments. Statistical analysis would be helpful here.

We have added variance to our experiments, while baseline results reported by [1,2] did not provide this information. Results prove the robustness and reliability of our experiments.

(a) Protein-Protein Interaction

(b)Prokaryotic Protein Subcellular Location

(c) Protein Stability Prediction

(d)Zero-shot Protein Fitness Prediction

We emphasize that our work is vital for advancing bio-sequence modeling. While current methods focus on scaling model size, we introduce a novel approach that greatly improves learning efficiency and model performance by incorporating cross-modal data. Our contribution is in presenting a new direction and proving its potential. Further exploration of cross-modal data and scaling models will lead to even stronger bio-sequence models.

---

[1] LucaOne: Generalized Biological Foundation Model with Unified Nucleic Acid and Protein Language

[2] Sequence modeling and design from molecular to genome scale with Evo, Science 2024

[3] ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction, NeurIPS 2023

 

We want to express our sincere gratitude for your review. If you have any further questions or concerns, please feel free to contact us at any time. If our response has addressed your concerns, we sincerely hope you will consider raising your score.

Best regards,
All Authors

Dear Reviewer FwnU,

We sincerely appreciate your constructive feedback on our manuscript. Guided by your insightful suggestions, we have strived to address each point thoughtfully, which we believe have significantly improved the quality of our work.

Thank you once again for your time and valuable guidance. We're eager to hear your thoughts on these revisions.

 

Best regards,
All Authors

Thank you very much for your valuable feedback; your input is very helpful in improving our work. We address your questions point by point below.

Q1: It is unclear to this reviewer what performance gains are due to the increased token budget vs from the different types of data included.

Q2:The authors kind of show something like this in Figure 1, but it's unclear to me exactly what the validation set is.

To address them, we first clarify the valid loss is computed on validation data from Round 1, which includes only single-modality data (DNA, RNA, and protein). The valid losses across all rounds are computed on the same data for consistency. This was explained in Figure 1’s caption and Line 95 of the original paper, and we have now revised the PDF to make this clearer, with changes highlighted in blue.

Figure 1 shows that introducing new cross-modal data in Round 2 and Round 3 significantly reduces valid loss compared to not adding such data, indicating improved representation across all single modalities.

To address Q1, we ran additional experiments on the 110M model, training it on single-modality data with the same token budget as BSM_R2 and BSM_R3. The results show that BSM_R2 outperforms BSM_single_R2, and BSM_R3 outperforms BSM_single_R3. This confirms that performance gains come from incorporating cross-modal data rather than longer training. The results also show that without cross-modal data, BSM_single_R3 is far lower than billion-scale models. However, with cross-modal data, it matches or exceeds their performance. (Added in Section 3.5)

We also added the valid loss curve of BSM-110M-single to Figure 6, showing its valid loss consistently higher than BSM-110M. This aligns with Figure 1, confirming that introducing cross-modal data significantly reduces valid loss and improves single-modal representations.

Q3: The Authors do not compare to the smaller models (e.g. the small ESM models) where applicable. Reporting their 110M and 270M compared to ESM-150M would help showcase how ther training setup and architecture compares to a model similar in size.

Q4: On all the protein tasks, the authors only report ESM-3B, except for zs-Protein Fitness Prediction where they use ESM-650M. Note: going down to ESM-650M only for that one task is odd.

For Q4, we report ESM-650M for zs-Protein Fitness Prediction because it is the best size for this task. We follow Evo [1], which provides a clear explanation: "We also included ESM 2 650M and ProGen2 large given that these models have sometimes shown better performance at function prediction than larger variants of these models (Notin et al., 2023) [2]."

For Q3, thank you for the suggestion. We compared our models with ESM-150M, which share similar size with ours. Results show that ESM-150M is far lower than its larger model, and both BSM-110M and BSM-270M significantly outperform ESM-150M, highlighting the advantages of our approach and the importance of cross-modal data. (Added in Section 3.6)

Q5: What sequence lengths are used, for models that are truncated, how are long sequences processed?

The BSM model supports a maximum context length of 1024 tokens (Line154 & Table4). Sequences longer than this are truncated.

Q6: What was the fine-tuning procedure for each of the models? How much does this type of processing affect different model performance? Architectural choices like these likely have some effect on the performance of each model observed, but it's currently unclear how much these types of decisions contributed to the observed performance and so it's difficult to disambiguate what the true effect of the data selection is.

We apologize for not being entirely clear on what is meant by the fine-tuning procedure here. If you're referring to the baseline models using a dual-tower structure for tasks that require two sequences as input, these experimental results are reported by LucaOne[3]. LucaOne conducted comprehensive experiments to compare different architectures, and we reported the best results of these architectures. In 3 out of the 8 tasks, a dual-tower structure is used for baseline models. We will include the final architecture used for each of these 3 tasks in the appendix in the future.

Our BSM, on the other hand, has the capability to handle any number of sequences. We simply connect the two sequences as one input in BSM, so our experiments do not involve complex architectural choices.

Finally, we would like to emphasize that the ablation experiment, conducted to answer Q1 above with the same model structure and token budget, shows significant benefits of adding cross-modal data.

Q7: More details are needed for each evaluation task. Appendix C and Table 4 are a good start, but there's details such as the number of instances of each class, the species the task was taken from, etc.

Q8: Most of the tasks only report accuracy or SRCC, but it's unclear if this is an adequate summary statistic for each task given this missing information.

For Q7, we have added the species information for the tasks in Appendix C. But we can't find the number of instances for each class in any related work.

For Q8, to maintain comparability with previous work, we used the same metrics as in [1][3], which are widely used.

Q9: Could the species you used for training also have an effect on downstream performance. Was there any logic to the species selected? Is it possible that any differences in performance are (at least partly) due to the species selected?

We used a mix of mammals (including humans) and fungi for our RefSeq species. The specific species are listed in Table 5 (old pdf Table 4). The species were randomly selected during data collection and were not specifically designed for downstream tasks. Our experiments include 8 diverse tasks from a wide range of species, including mammals, invertebrates, bacteria, viruses, prokaryotes, and more. So we believe our results are reliable.

Q10: How exactly were data leaks prevented? I understand that data from the downstream tasks was removed from pre-training but how exactly was this done and how are you sure that no leakage occurred by e.g. a very similar sequence, or having a gene whose transcribed protein is in the test set?

We removed sequences from the pretraining data that appear in the test set, without considering similar sequences like those involved in transcription. Please note that unlike LucaOne, our training only uses sequence data, without attribute data. For the 8 downstream tasks, the label information for 7 tasks (except Central Dogma) is not included in the training data. Five of these tasks are classification or regression, and the other two are ncRNA-Protein and Protein-Protein Interactions, which are also not included in the pretraining data. We don't think only using sequence data in pretraining would cause leakage for these tasks. However, we did remove these test sequences, as we did for the Central Dogma task. Therefore, we believe our results are fully reliable.

---

[1] Sequence modeling and design from molecular to genome scale with Evo, Science 2024

[2] ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction, NeurIPS 2023

[3] LucaOne: Generalized Biological Foundation Model with Unified Nucleic Acid and Protein Language

 

We want to express our sincere gratitude for your review. If you have any further questions or concerns, please feel free to contact us at any time. If our response has addressed your concerns, we sincerely hope you will consider raising your score.

Best regards,
All Authors

Table 2:

2. Why is the phase 3 data so effective? Phase 3 (the web data) seems to have a dramatic impact in a relatively few number of tokens.... I'm concerned that what you're showing isn't that "varied forms of data help", but rather that "pre-training on a set of sequences similar to those we find in the test set improves performances". While you've convinced me you're not leaking labels, I'm still concerned that your model is seeing very similar sequences in the pre-training data particularly in phase 3. The val ppl also decreasing rapidly may help point that phase 3 data actually does help the model learn something useful, but I'm not sure you can quite say this without a more rigorous exploration of what is in the validation data on how it affects different parts of it (e.g. does it decrease the ppl in only the protein portion of the val or uniformly across them)?. Perhaps I'm misunderstanding something about this and would appreciate some clarification on whether this is actually happening in practice or whether my concern is unfounded. I think this is worth discussion in much more detail in the main paper (what aspect of the web data is so effective and why are we sure our gains aren't from accidents leakage). Additionally, assuming that the web data is fine, why even bother with the first 2 phases then?

Regarding these concerns, we provide the following explanation.

1. Let's first talk about valid data.

• Our valid data is single-modal data comes from LucaOne [1], which is randomly split from 800B tokens containing 169,861 species, while our web data only has 33 million tokens. We believe there is no correlation or bias between the web data and the valid data, so the results of the valid data in Figure 6 demonstrate that Round 3 can significantly improve the model's performance.

• To address your question, "does it decrease the PPL in only the protein portion of the validation data or uniformly across all of them?", we have provided additional data below. We divided the valid data into protein-only and gene-only subsets. By comparing the differences in the Round 2 and Round 3 checkpoints across these three types of valid data, it is clear that both the protein and gene valid loss decreased through Round 3 training. In other words, the cross-modal data improved the representation for both modalities.

Table 3:

2. Analysis from downstream task test data: As shown in the table in our response to Q1, both R2 and R3 data improve the model's performance. It's not just the web data—R2 shows a significant improvement, approaching the performance of other billion-scale models , and R3 continues to show further improvement. We believe the results on the valid data demonstrate that web data contributes to enhancing the model's performance in all single-modal data, which is consistent with the improvements observed in downstream tasks. When combining both results, they validate the benefits of web data.

3. The Role of Cross-Modal Data in Round 2 and Round 3:

• In multi-modal models (such as text-image models), there are two commonly used types of data beyond single-modal data: image-text pairs and interleaved data (which consists of sequences of multiple images interspersed with text). The latter has gained increasing attention in recent research and has been widely shown to be effective [3, 4]. As an extension of image-text pairs, the interleaved format not only covers a broader range of data but also captures longer and more complex relationships.

• We introduced these two types of data in BSM. In Round 2, we introduced gene-protein pair data, which establishes semantic links between genes and proteins, enhancing the model's learning efficiency across different modalities. In Round 3, we used interleaved bio-sequence data obtained from the web, which consists of sequences of proteins interspersed with genes. This data allows the model to learn more complex and broader relationships between sequences over longer contexts, improving its performance on complex multi-modal tasks.

• Each type of data makes a unique contribution, and none is dispensable. By combining and leveraging these data types effectively, the model gains stronger and more comprehensive capabilities.

Thank you again for your highly valuable feedback. As with the last feedback, your input is very helpful in improving our work.

I may be wrong, but don't you have some list/file that contains the labels for each datapoint for each task? I was just asking for the value counts for each of classification tasks, and maybe some summary statistics for the regression tasks since I'm not familiar with all of the tasks you use. I understand you're using the same metrics as previous work, and I'm giving the tasks from previous work the benefit of the doubt that they're evaluated properly, but I would have imagined this is something pretty easy to calculate since you ran the evaluation of all your models on these tasks. I hope this was just a miscommunication during the review process, but if the authors mean to tell me that they are unable to calculate statistics like this, that would indicate to me some serious methodological problems with your evaluation setup and I am unable to increase my score at this time as a result.

We sincerely apologize for not providing you with this information earlier. As you mentioned, we can calculate this information. We have added the calculated information to the Table 1 below and included it in Appendix C. If you would like to know any information beyond this table, please let us know.

We use Spearman Correlation Coefficient (SRCC) for the regression tasks, and Accuracy for the classification tasks. We followed LucaOne/Evo [1,2] in using the SRCC metric, which computes the Spearman correlation between the values (e.g., protein stability for ProtStab, fitness scores for proteins and ncRNAs) and the sequence likelihood (for autoregressive language models) or the sequence pseudolikelihood (for masked language models).

Table 1:

Regarding your other three new concerns, we will address them one by one as follows.

1. Did the authors sample new tokens to continue training or did they use the data as phase 1 to continue training? I'm sorry if I missed this somewhere in the new work, but my concern is that if their experiment used the data as phase 1 (that is repeating sequences) then I would expect a marginal gain at best in model performance. If the authors sampled a new set of data from phase 1 (eliminating or minimizing the overlap with the original phase 1 data), then this helps alleviate the concern, although I'm still a bit unclear of the exact phase make-up for each phase (see concern 3).

We previously sampled a total of 220B single modal data from LucaOne [1] (Line 170-173 and Figure 2), with 100B used in Round 1 and 120B mixed with other cross-modal data in Round 2. In the new experiments, we sampled new 50B tokens from LucaOne, bringing the total to 270B single modal data. These data do not contain any repeating sequences. Therefore, BSM and BSM-single share the same 220B single modal data, with the difference being that BSM used about 50B of cross-modal data, while BSM-single used about 50B of extra new single modal data. The data used in each round is summarized in the Table 2 below.

I would like to emphasize that our two evaluation tasks, the zero-shot protein fitness prediction task and the zero-shot ncRNA fitness prediction task, are both zero-shot evaluations. In these tasks, we directly compute the Spearman correlation between the test sequence likelihood from the model and the fitness score. For zero-shot tasks like these, the model has never been fine-tuned on downstream tasks, and the model has never seen the label at any data at any stage, whether during pretraining or fine-tuning. Therefore, there is no possibility of any leakage. Our experimental results demonstrate significant improvement in both R2 and R3 for these tasks, proving the effectiveness of cross-modal data.

We use edit distance because, in your description, you used it to measure sequence similarity. For the calculation of edit distance, we directly compute the minimum number of edit operations required to transform one sequence into another, without applying any specific sequence alignment technique. Based on this approach, the "<20%" threshold is not considered "relatively small" but is actually an appropriate threshold for our analysis.

To clarify this, we can provide a table that summarizes the sequence count at various edit distance thresholds. This threshold ensures that the test set is appropriately divided. If the threshold is too small (e.g., 10% or 15%), the number of similar sequences would be too few (<100), which may lead to unreliable statistics. On the other hand, we avoid using too large a threshold (e.g., 30%) because we believe smaller thresholds more strictly define a subset that closely resembles the web data. If there were indeed a leakage issue as you suggested, the performance gain would be more apparent for this subset relative to the others; however, the results show that this is not the case.

Thus, by using a statistically meaningful subset (with count >100) and a stricter similarity measure (<20%), we ensure that although the subset may contain sequences very similar to those in the web data, the performance gain observed does not exceed that of other data subsets.

While you mentioned the use of other similarity measurement tools, we believe that using a strict edit distance threshold of <20% still defines a reasonable and meaningful subset with strong correlation to web data. The results derived from this strict criterion are reliable and support that our performance gains are not due to data leakage.

Based on all the evidence provided above, we firmly believe that our contributions, particularly the effective use of cross-modal data, are solid and the research significance is substantial.