NovoBench-100K: A large-scale protein dataset for in silico evolution of de novo TadA

Thank you for your thoughtful feedback and for recognizing the strengths of our work. We are grateful for your recognition of the dataset (addresses a crucial evaluation criterion), extensive study (impressive and commendable effort), and detailed comparison and analysis of BLMs (examining factors such as modality, model size, and K-mer). We truly appreciate your insights.

Below, please find our detailed responses to your concerns.

1 Bio Experiment

How is the read score representative of the evolution effectiveness? Is the least ranking sequence in the dataset completely ineffective at editing or all the 100K variants effective? Specifically I would like to know if you are also evaluating BLMs capability in predicting both effectiveness and ineffectiveness?

Thank you for your insightful question. As shown in Appendix A.1, the editing efficiency of TadA protein is directly determined by its enzymatic activity, which in turn influences the replication of bacteriophages. This leads to higher read counts for sequences with higher editing efficiencies. Rather than being a binary classification of “effective” versus “ineffective,” this is a continuous spectrum of activity levels.

In the dataset derived from NGS sequencing, the sequences ranked lowest in terms of read count generally have lower editing efficiency than those ranked higher. However, whether these sequences are entirely inactive is not always clear. This is precisely why we chose to construct the dataset using rankings instead of absolute activity values: rankings focus on the relative performance of sequences, making the data representation more robust against noise and experimental variability.

Thus, the dataset does not evaluate effectiveness as a binary trait but rather as a continuum of activity levels. This approach ensures that the benchmarking remains focused on distinguishing relative performance, which aligns better with the inherent variability of biological experiments.

What external evaluation can you provide to show your effectiveness scores are sound and consistent with previous work / distaste ?

Thank you for raising this important question. We agree that biological experiments inherently exhibit variability and system-specific fluctuations. In such cases, absolute calibration across different experimental setups is difficult to achieve. Thus, comparisons are typically made relative to well-established baselines or control groups.

In our study, we ensured the reliability of our evolutionary experiments through extensive preliminary trials to validate the experimental setup. Moreover, our experiments began with TadA8e—a globally recognized benchmark enzyme with high activity. This starting point serves as a robust and reliable control, reinforcing the soundness of our methodology. As a result, while absolute calibration may not be feasible, our scores are meaningful and consistent within the context of our controlled experimental framework.

We believe this approach demonstrates the validity and reliability of our findings, aligning with best practices in the field. Thank you for allowing us to clarify this point.

2 SEQ2RANK

Though Figure 4 is a helpful visualization, I think some kind of end to end pipeline of data construction and usage is helpful, currently the main framework is mixed up with the details provided.

Thank you for pointing this out. We appreciate your suggestion to clarify the framework further with an end-to-end pipeline illustration. While Figure 4 provides a detailed visualization, we acknowledge that separating the framework from the finer details can enhance the clarity and utility of our presentation.

It is important to note that our approach, SEQ2RANK, inherently serves as an end-to-end pipeline for ranking tasks based on biological sequencing data. From raw sequence input to ranked output, SEQ2RANK integrates data preprocessing, ranking extraction, and consistency maintaining within a cohesive framework. This design emphasizes its generalizability and adaptability to various biological sequencing-rank scenarios, ensuring a seamless workflow for the user.

Further discussion continues on the next page.

4 Learning to Rank

The paper introduces a listwise learning-to-rank method for all models and claims this as a contribution. However, have the authors evaluated this ranking loss against regression loss? In fact, findings from the 2011 Yahoo Learning to Rank Challenge indicate that learning to rank may not significantly outperform pointwise regression or classification losses. In some cases, regression loss could be more effective if specific label values are available. see [1] section 6.3. I do not think listNet is a powerful learning to rank algorithm. The author should report convincing results that listNet is better than basic regression loss. [1]https://proceedings.mlr.press/v14/chapelle11a/chapelle11a.pdf

Thank you for your valuable feedback and for pointing out the work from the 2011 Yahoo Learning to Rank Challenge. We agree that learning-to-rank (LTR) methods may not always outperform regression losses, especially when specific label values are available.

However, our motivation for using a listwise LTR approach was based on the fact that our dataset provides only relative ranking information (i.e., the order within a list), rather than explicit numeric labels. It is because relative ranking rather than absolute numeric values, helps mitigate random noise, experimental variability, and discrepancies across different rounds of evolution. Therefore, traditional regression approaches would not be suitable in this case, as they require exact target values for training.

In Table 2 of the Appendix, we explored several loss functions related to ranking, including Kullback-Leibler (KL) divergence and cosine similarity. Ultimately, we selected cross-entropy loss for our ranking tasks, as it provided a good balance between performance and theoretical grounding for listwise ranking.

Thank you for your thoughtful feedback and for recognizing the strengths of our work. We are grateful for your recognition of the large-scale dataset, extensive study, and well-written paper. We truly appreciate your insights.

Below, please find our detailed responses to your concerns.

1 Dataset Division

To rigorously evaluate a machine learning algorithm, it is necessary to prepare separate training, validation, and test sets. However, this work only partitions the dataset into training and test sets. It is not a good practice to use the test set to tune hyperparameters (e.g., batch size, early stopping, or training epochs). The test set here is actually a validation set. A good performance on the validation set does not mean that the results on the test set will be similar.

Thank you for your valuable comment. You are absolutely right—we used the test set for hyperparameter tuning in this work. Moving forward, as we collect more data across additional domains and rounds, we will ensure to properly separate validation and test sets to avoid this issue.

From another perspective, even though we performed hyperparameter tuning on the test set, the poor performance on out-of-domain (OOD) data highlights the severity of the OOD challenge, which remains a key limitation of the current models.

2 Evaluation Method

The paper primarily evaluates linear probing in the main text, but linear probing typically performs significantly worse than fine-tuning the entire model or the top several layers of the language model. I noticed in the appendix Table 6 that the authors only evaluated ESM2 using both linear probing and all-parameter fine-tuning, revealing a substantial performance gap (e.g., the SP metric increased from 0.138 to 0.208 and from 0.075 to 0.187). Given this difference of pLM, I suggest the authors present the main results for these pLM algorithms using parameter fine-tuning rather than a frozen embedding strategy.

Thank you for your insightful comment. We agree that fine-tuning generally yields better in-domain performance compared to linear probing. However, fine-tuning 80 models is computationally expensive, and we were not able to afford such a comprehensive evaluation within the scope of this work.

Also, our primary focus in this work is to compare the representational power of different large-scale models, and thus we chose to evaluate them using linear probing. This approach allowed us to focus on understanding the quality of embeddings across different models. Furthermore, in our experiments, fine-tuning did not significantly improve out-of-domain performance, which remains a key challenge. Therefore, we believe the conclusions drawn from our linear probing experiments still hold strong in terms of supporting our central argument.

3 Evaluation of Multiple Models

The authors claim to have evaluated up to 80 biological language models (BLMs), which seems quite surprised to me. Each model's hyperparameters should be meticulously tuned. How can the authors ensure that they are using each model correctly or with optimal settings? If suboptimal settings were used, the benchmark may not be reliable. For instance, SaProt is a protein language model that incorporates structural information, yet the paper does not specify what type of structures were used (PDB or AlphaFold2 structures with or without pLDDT). This information is not available in the supplementary materials.Evaluating more models is a good thing, but one should make sure that they actually understand and evaluate them correctly.

Thank you for your thoughtful feedback and recognition of the scope of our work. The core objective of our experiments was to evaluate the representational capabilities of different BLMs. As part of this, we primarily focused on extracting embeddings and then performing linear probing. To ensure we were using each model appropriately, we performed extensive hyperparameter tuning for each model. For linear probing, we carefully tested different learning rates, optimizers, and other hyperparameters, ultimately selecting the best-performing configurations for each model.

In particular, for SaProt, we followed the setting from the original paper’s Appendix E.3, which describes the “RESIDUE SEQUENCE-ONLY” setting. The original paper describes the setting as:

...all Foldseek structure tokens are substituted with “#”, resulting in “si#”. This scenario often arises in various protein engineering tasks where the fitness values of protein variants can be obtained through wet experiments, but experimental structures for them are unavailable.

Further discussion continues on the next page.

Thank you for your response. However, there still remain some concerns that need to be addressed:

1. Regarding Hyperparameter searching on the test set.

The current approach of using the test set for hyperparameter optimization raises significant concerns and potentially compromises the validity of the results. Your proposed solution of collecting additional data in the future does not address this issue with the current benchmark. As a benchmark study, ensuring the reliability and reproducibility of your evaluation framework is paramount.

2. Limited Evaluation Scope.

Your study evaluates 80 models but primarily under linear probing conditions. While this broad comparison is interesting, the absence of fine-tuning results - which typically yield better performance in practice - limits your conclusions. I suggest focusing on fewer models but evaluating them under optimal fine-tuning settings, as this would provide more actionable insights than testing many models under suboptimal conditions.

3. Baseline Issues.

Upon reviewing your evaluation details, let's say SaProt, there appears to be a discrepancy. The original paper specifies that SaProt works on residue-only sequences during the fine-tuning stage. In your implementation, the SaProt model remains frozen. As a structure-aware PLM claimed by the paper, simply removing its structure part without any fine-tuning may not make sense. It would be beneficial to: • Consider implementing the fine-tuning stage as described in the paper • Provide some necessary analysis of why some baselines perform differently under your specific conditions.

Comparing baselines under fair settings is important for a benchmark paper.

Thank you for your thoughtful feedback and for recognizing the strengths of our work. We are grateful for your recognition of the Novel Dataset, Robust Methodology, and Comprehensive Benchmarking. We truly appreciate your insights.

The creation of NOVOBENCH-100K fills a crucial gap in protein engineering, specifically for base editing.

The SEQ2RANK algorithm ensures data consistency and robustness, addressing typical noise and variability in biological data.

The inclusion of various BLMs and modalities provides a detailed performance landscape, especially in distinguishing in-domain from out-of-domain capabilities.

Below, please find our detailed responses to your concerns.

1 Application

Limited Real-World Application Validation: Although the dataset is extensive, there is limited discussion on how it performs in real-world applications beyond the in silico setting. Including preliminary results from practical applications or collaborations with experimental labs could strengthen the paper.

Thank you for your suggestion. The real-world applications of this dataset involve protein engineering and directed evolution. Researchers can use our dataset to develop protein evolution algorithms that better mimic real-world multi-round evolution processes.

In fact, using this dataset and the associated algorithms, we have successfully engineered proteins with higher activity than TadA8e. However, due to the scope of this work, we did not include these results in the NovoBench-100K paper. Additionally, we have no intention of using such results to enhance the recognition of NovoBench-100K, which is focused on the dataset and benchmark itself. Further application-focused outcomes will be released in subsequent work as part of this series. For those interested, we encourage following our future publications.

Our goal in this paper is to provide an open and versatile resource, enabling researchers to freely explore AI-guided protein engineering and related applications. We believe this approach will maximize the potential impact of our work in advancing real-world applications.

2 OOD

Out-of-Domain Generalization: The authors highlight the poor performance of current BLMs on out-of-domain rankings. However, the paper could benefit from a more in-depth analysis of why these models fail and potential strategies for overcoming these limitations.

Are there plans to address the out-of-domain challenges identified in the paper, perhaps by incorporating domain adaptation techniques or more diverse training data?

Thank you for pointing this out. We believe the vastness of the protein sequence space is a key factor contributing to the poor out-of-domain performance of current BLMs. When sequences from different rounds of evolution exhibit significant differences, it becomes challenging for these models to generalize effectively.

To address this, future strategies might include developing methods for constructing latent spaces or adaptive approaches that are inherently more robust to sequence variation. Such advancements could help reduce the impact of sequence discrepancies and improve generalization across rounds. We appreciate this suggestion and will explore these directions in our future work.

Additionally, we are continuously expanding our dataset. Currently, we have collected data from five rounds of evolution experiments, comprising over 300K sequences. We plan to update and release new data regularly to support further OOD research and improve model robustness.

3 More Models

Benchmarking Scope: Although 80 models are benchmarked, it would be beneficial to include additional model families or discuss other state-of-the-art models in more detail to contextualize their choices further.

Thank you for your feedback. We have benchmarked 80 models, including cutting-edge and influential models such as ESM3, EVO, Chai1, and OmniGenome. We believe this represents a comprehensive scope of current state-of-the-art models. We are more than willing to test additional models and greatly appreciate any specific papers or model families you could point us to. This would further enhance the diversity and impact of our benchmark.

4 SEQ2RANK Generation

Can the authors clarify if SEQ2RANK can be generalized to other protein datasets or if it’s specifically tailored for the TadA evolution?

Thank you for your question. SEQ2RANK is not specifically tailored to TadA evolution. It is a general framework designed to bridge the gap between raw biological sequencing data (e.g., from NGS) and ranked datasets. As such, it can be applied to any protein dataset where ranking based on activity or other metrics is required.

The method is protein-agnostic and aims to address the broader challenge of converting sequencing data into robust and usable rank-based datasets, enabling its use across various protein engineering and evolution applications.

Thank you for your thoughtful feedback and for recognizing the strengths of our work. We are grateful for your recognition of the

• extensive experiments (a wide range of BLMs, evaluation methods, and hyperparameters),
• ranks and SEQ2RANK (mitigate experimental noise and address scalability limitation),
• valuable dataset (valuable for research in this domain, OOD practical perspective on model’s robustness in the real-world applications),
• and paper writing (presentation is pretty good). We truly appreciate your insights.

Below, please find our detailed responses to your concerns.

1 Evaluation

The experiments primarily use BLMs as encoders, leveraging the embeddings for downstream ranking tasks. However, there are other common approaches, such as prompting and in-context learning, which are now widely used. The authors should consider including these methods in the evaluation.

Thank you for highlighting this important point. Our primary goal in this work was to broadly evaluate the representation capabilities of biological language models (BLMs). To this end, we focused on approaches like linear probing and fine-tuning, prioritizing experiments across a wider range of models.

We agree that prompting and in-context learning are valuable methods and offer exciting possibilities for leveraging BLMs. While they were beyond the scope of this work, we plan to explore these approaches in future studies. Thank you again for the suggestion!

There are additional aspects that could be included in the analysis, such as details about the model architectures (e.g., encoder-decoder or decoder-only). Additionally, providing more description of the initial design purpose of each model and how this aligns with the TadA evaluation would be helpful.

Thank you for emphasizing the importance of providing detailed model architecture and background information. We recognize that these aspects are critical for a comprehensive understanding of the models used. We have already compiled substantial details about the models, including architecture, training datasets, and design purposes. However, due to space constraints, we were unable to include them in the main text. To ensure transparency and provide these details to the community, we plan to make them publicly available on an open-source platform (e.g., GitHub).

Below is a sample of the compiled information we plan to release in markdown format:

Further discussion continues on the next page.

2 More Experiments and Details

It would be beneficial to conduct more rounds of experiments, as I am questioning whether the model's poor performance on out-of-domain cases is due to overfitting to a specific experimental round. I notice that the author plans to include additional rounds of wet experiment data, which I am looking forward to seeing.

Thank you for your thoughtful feedback. As of now, we have conducted 5 rounds of evolution experiments, resulting in 348,267 sequences and 344,497,666 edges in a directed graph. This extensive dataset allows us to evaluate the model’s performance across diverse scenarios, including out-of-domain cases. We are actively exploring additional ML experiments while collecting more data, and further results will be shared in our future work. We appreciate your interest and will ensure transparency in reporting updates as they become available.

The authors should include detailed descriptions of the wet experiment conditions as a reference for researchers who wish to replicate or expand this dataset.

Thank you for your suggestion. We have provided the wet experiment methods in Appendix A.1. However, the specific experimental parameters are currently proprietary to our collaborating company and are not being made publicly available at this stage.

We are committed to transparency and reproducibility, and as such, we will release the complete dataset, AI code, and BLM embeddings to support further research and replication efforts. Should the company decide to disclose these details in the future, we will ensure they are made available.

3 Benchmark Name

There is an existing benchmark names NOVOBENCH (without the 100k) (https://github.com/Westlake-OmicsAI/NovoBench). What is the relationship between NOVOBENCH and NOVOBENCH-100K?

Thank you for bringing this to our attention. We acknowledge that the name “NovoBench-100K” coincides with the existing “NovoBench” benchmark. This similarity is unintentional, and we apologize for any confusion it may have caused.

4 Bio Experiments

What are your insights on why all models fail in out-of-domain evaluation, and how can the models' robustness be improved for out-of-domain cases?

Thank you for this insightful question. We believe that the vastness of the protein sequence space poses a significant challenge, especially when sequences from different rounds of evolution exhibit notable differences. This divergence is difficult for current BLM approaches to fully address.

In the future, simpler downstream or adaptation methods for constructing latent spaces or adaptive approaches that are more robust to sequence variation may help mitigate the impact of these differences. Such methods could reduce the effect of sequence discrepancies across rounds and improve model performance in out-of-domain scenarios.

Why do conflicts arise in the rankings? If these conflicts are due to experimental errors, should the algorithm identify and exclude erroneous experiments from the data, rather than simply using credibility to mitigate the conflicts?

Thank you for this insightful question. Ranking conflicts can arise due to inherent variability and uncertainty in biological experiments, which are influenced by factors such as noise, experimental conditions, and measurement errors. Rather than excluding data outright, which risks losing valuable information, we opted to design a framework that assigns credibility scores to account for these conflicts.

To clarify, we conducted two separate rounds of large-scale evolution experiments (R1 and R2), each consisting of multiple turns over time (T1, T2, T3, …). The credibility score leverages the fact that E. coli growth rate is proportional to enzyme activity. Over time (T1, T2, T3, ...), this leads to an exponential increase in read counts for sequences with higher activity, which reduces the impact of initialization differences and random noise. This makes later-round data more reliable.

Figure 4 is not informative about Seq2rank and does not have good correspondence with algorithm 1 in appendix.

Thank you for your feedback. We will ensure better clarity by aligning Figure 4 with Algorithm 1 and improving its correspondence in the revised manuscript. Additionally, we will release all code for transparency and include key Python implementation details in the supplementary material to provide further clarity on Seq2Rank.

3 Bio Experiment Details

What if such prior does not exist and there is naturally uncertainty in two replicated experiments that needs to be considered equally?

What if there are experimental replicas that may have a batch effect but we don’t necessarily have a “reliability” measure of the replicas and want to consider them as equally important?

Thank you for this thoughtful question. Our experimental design couples enrichment levels with enzymatic activity, ensuring that over successive rounds of evolution, enzymes with different activity levels increase exponentially in number. This design helps mitigate uncertainty by leveraging the underlying biological principles.

We fully acknowledge that biological experiments inherently involve uncertainty and randomness, and no approach can entirely eliminate these challenges. Our method proposes a framework to assess and reduce variability across experiments by emphasizing reliability. However, we also recognize that in cases of perfectly replicated experiments yielding different results, the standard practice of averaging over multiple repetitions remains the most practical solution, albeit imperfect.

When talking about credibility, it mentions “later rounds are more reliable”. Is this suggesting that read counts across different rounds are compared? E.g., it might compare the read count of seq A in round 1 with the read count of seq B in round 2?

If seq A has a very high value in round 1, but is somehow close to noise low value in round 2, given that round 2 is always given higher reliability seq A will be ranked low? What if the round 2 data is noise, and such an assignment might create a lot of wrong ordering of sequences relative to seq A?

We apologize for the confusion caused by our wording. To clarify, we conducted two separate rounds of large-scale evolution experiments (R1 and R2), each consisting of multiple turns over time (T1, T2, T3, …).

Within a single evolution experiment, later rounds (e.g., T5) benefit from the coupling of enzymatic activity with the exponential growth of E. coli. This effect amplifies the distinction in read counts between enzymes with differing activity levels, making the data from later rounds more reliable. We will revise the manuscript to ensure this explanation is clearer.

Thank you for your thoughtful feedback and for recognizing the strengths of our work. We are grateful for your recognition of the dataset (scale, quality, and real-world wet-lab data), algorithm (for realistic challenges), and benchmarking effort (80 BLMs, novel OOD setting). We truly appreciate your insights.

The paper introduced a new benchmark dataset for protein function prediction which excel at its scale and data quality.

Such real-world wet-lab experiments data could be a valuable source of data for protein design community.

The algorithm for list construction does take into consideration several realistic challenges.

The benchmarking effort is also very extensive, covering 80 BLMs and both in-distribution and out-of-distribution. The OOD setting with different evolution round is a novel setup.

Below, please find our detailed responses to your concerns.

1 Technical Contribution

since it is considering one specific protein TadA and base editing enzymatic activity, the generalizability could be limited.

the majority of the work focuses on data creation and benchmarking, the technical contribution and novelty may be insufficient for an ML venue.

Thank you for pointing this out. We acknowledge that our work focuses on the specific case of TadA and its base-editing enzymatic activity. However, this choice was intentional to demonstrate the potential of integrating ML methodologies with biological data collection and interpretation in a concrete, well-studied context. While generalizability to other proteins is indeed an important direction for future work, our approach aims to establish a foundational framework. For example, the focus on real-world biological scenarios like out-of-distribution (OOD) protein evolution and robust ranking from sequencing data highlights the broader applicability of ML to address key challenges in biology.

We believe such integration is essential for enabling ML for Science, rather than merely applying ML techniques to isolated datasets. We emphasize that this effort reflects a deliberate integration of ML principles into the process of biological data generation and curation, rather than treating the two fields in isolation. In our view, separating biological data collection from ML research risks creating a “science + ML” dynamic, rather than fostering true “ML for Science.” Specifically, our contributions concerning dataset construction include:

• Designing a pipeline that captures realistic OOD protein evolution scenarios.
• Using ML-inspired strategies to ensure the robustness of ranking lists derived from noisy biological sequencing data.
• Demonstrating how such datasets can lead to meaningful insights and evaluations for ML models in scientific contexts.

2 Seq2Rank Details

Thank you for pointing this out. We will clarify these details in the revised manuscript. Here are short responses to each question:

The “GREEDY_SAMPLE” algorithm is too abbreviated and lacks details, such as how are the read count index sampled at each step, and with what “greedy” order it traverse the possible sequence space.

How do we determine what read count index to use in each step of GREED_SAMPLE?

The ‘GREEDY_SAMPLE’ algorithm selects reads based on the highest current read count at each step. We build the "read-seqs dictionary", Dict[float, List[str]]. Then, we choose the read index with the most number of seqs, i.e., GREEDY_SAMPLE (max(read_seqs_dict.keys(), key=lambda x: len(read_seqs_dict[x]))). Furthermore, once a sequence is selected to the list, it is removed from its corresponding read-seq dictionary, dynamically updating the read counts. This ensures that the algorithm maximizes the extraction of the most abundant and representative lists.

It is also not clear whether the generated list can cover similar value range, or some of them will be biased towards higher region. When adding sequences from different experiments into same graph, it implicitly tries to compare read counts across different experiments, which may not be ideal.

Thank you for raising this important question. Indeed, using raw values to construct datasets often introduces biases, as value distributions can vary significantly across experiments. To address this, we use ranked lists, ensuring that the relative activity within each list remains accurate. When combining rank lists from different experiments into the graph, we do not rely on absolute values; only relative order is preserved.

By maintaining the correctness of relative rankings within each list and leveraging Seq2Rank to ensure consistency across lists, we effectively mitigate the issue of inconsistencies in raw values from different experiments. While the rank-based approach inherently diverges from the original value distribution, it enables robust comparisons and avoids biases introduced by value range mismatches.

Further discussion continues on the next page.