Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks

Thank you for your positive assessment of our work. We appreciate your recognition that our approach is well-motivated and technically sound.

On the applicability of synthetic benchmarks to real-world cases

You raised an important question about whether our synthetic benchmarks apply to real-world cases. To address this directly, we conducted new experiments comparing optimizer performance on Ehrlich functions versus established lookup-based biological test functions (TFBind8, DhfR, TrpB -- see our response to reviewer b3Qx). We found strong rank correlations (0.61-0.89) between algorithm performance across these benchmarks, confirming that Ehrlich functions effectively capture the structure of real biological optimization problems.

This validation supports our benchmark design principles, which were carefully crafted to capture key properties of real biophysical sequence optimization:

1. Feasibility constraints: The vast majority of random sequences fail to express or fold properly
2. Epistasis: Non-additive effects between sequence positions
3. Position-dependent sensitivity: The importance of specific residues at specific positions
4. Motif constraints: The need for functional motifs to appear with proper spacing

By deliberately incorporating these properties, Ehrlich functions provide meaningful insights into algorithm performance on real-world biological optimization tasks while maintaining computational accessibility.

On the computational cost of LLMs

You also asked about the computational cost of using LLMs. We address this important practical consideration in Section 6.1:

"For relatively easy optimization problems, since the performance of various methods is similar, using a specialized model with 0.01% of the parameters of an LLM may be more practical."

Our experiments revealed an interesting nuance: the optimal choice between generalist LLMs and specialist models depends on problem difficulty. For medium-difficulty problems, LLMs with appropriate training can significantly outperform specialized models, potentially justifying their higher computational cost. For very easy or very difficult problems, however, specialized models offer comparable performance with substantially lower computational requirements. This insight provides practical guidance for practitioners choosing between approaches based on their specific constraints and objectives.

We have additionally conducted new LLOME-MargE experiments with a very small LLM (~226K params) trained from scratch (see our response to reviewer b3Qx). We found that even for a very small LLM with no pre-training, LLOME-MargE is significantly more sample efficient than LaMBO-2, despite having fewer model parameters than LaMBO-2. As such, the computational costs of LLOME need not always be a concern.

Thank you again for your thoughtful review. We believe our work contributes valuable insights to both the machine learning and biological sequence design communities, and we appreciate your recognition of its potential impact.

Thank you for your positive assessment and constructive feedback. We appreciate your recognition of our thorough experiments and well-defined benchmarks.

On testing with real-world biological datasets

You noted that our evaluation relies on synthetic Ehrlich functions rather than real-world biological datasets. While direct evaluation on real biological data would indeed be valuable, there are two key challenges:

1. Training data contamination: Using widely available biological datasets risks contamination with LLM training data, which would invalidate our assessment of LLMs as black-box optimizers.

2. Accessibility: Real biological datasets and simulators often require specialized software, significant computational or experimental resources, or domain expertise, creating barriers to reproduction and wider adoption.

To address these concerns while maintaining biological relevance, we conducted new experiments comparing optimizer performance on Ehrlich functions versus established lookup-based biological test functions (TFBind8, DhfR, TrpB -- see our response to reviewer b3Qx). We found strong rank correlations (0.61-0.89) between algorithm performance across these benchmarks, confirming that Ehrlich functions effectively capture the structure of real biological optimization problems.

This validation approach allows us to demonstrate biological relevance while avoiding contamination in LLM training data. It also preserves the computational accessibility that makes Ehrlich functions valuable for algorithm development.

On additional references

Thank you for suggesting additional relevant references. We will include them in the revised version to better situate our work within the growing literature on protein sequence optimization using LLMs.

We believe our work provides several unique contributions to this field:

1. A systematically designed benchmark that balances realism, computational accessibility, and difficulty
2. A bilevel optimization framework that effectively leverages LLMs for constrained optimization
3. A novel preference learning objective that outperforms SFT, DPO, and REINFORCE when rewards are observed.

These contributions advance both the theoretical understanding and practical application of LLMs for biological sequence optimization.

Thank you again for your thoughtful review and suggestions for improvement.

Thank you for your positive assessment and thoughtful questions. We're pleased you recognize the value of our contributions and appreciate your suggestions for strengthening our work.

On success rates and feasibility

You raised an important point about success rates in real-world settings with costly wet-lab validation. We directly address this through our measurements of feasibility rates over time in Figures 4 and 10, which show how each method improves in generating valid sequences as optimization progresses.

Figure 12 provides additional insight by analyzing the relationship between edit distance and feasibility, showing that LLOME-MargE achieves the best balance between exploration (making meaningful changes) and constraint satisfaction.

Our simple regret plots (Figure 3) also demonstrate sample efficiency, which is critical when optimization is constrained by laboratory resources. LLOME-MargE consistently finds high-quality solutions with fewer function evaluations than other methods, particularly on medium-difficulty problems.

On Ehrlich function applicability to real biology

We share your interest in validating Ehrlich functions against real biological data. To address this, we conducted new experiments comparing optimizer performance on Ehrlich functions versus established biological test functions (TFBind8, DhfR, TrpB -- see the response to Reviewer b3Qx). We found strong rank correlations (0.61-0.89) between algorithm performance on these benchmarks, confirming that Ehrlich functions effectively capture the structure of real biological optimization problems.

This validation approach allows us to demonstrate biological relevance while avoiding contamination in LLM training data, which could compromise benchmark integrity.

On unified vs. separate models for scoring and generation

You asked about the advantages of our unified approach compared to the traditional setup with separate scoring and generation models. The main benefits are:

1. Improved ranking and guidance: When a model is jointly trained on both tasks, its internal representations develop a better understanding of the relationship between sequence features and objective values. This improves the model's ability to generate and rank candidates effectively.

2. Computational efficiency and search depth: By unifying generation and evaluation in one model, we can perform deeper, more focused exploration with the same computational budget.

3. Overcoming distributional limitations: Traditional "generate-and-filter" approaches assume the desired outputs already exist in a mode of the generative model's training distribution. Scientific discovery inherently requires finding solutions in low-density regions or even outside the training distribution entirely. Our unified approach enables the model to progressively learn to generate such solutions.

4. Simplified training and deployment: Using a single model reduces engineering complexity and maintenance overhead in real-world applications.

Thank you for your valuable suggestions. We agree that further validation on real antibody maturation datasets would strengthen our work's impact, and we're actively pursuing this direction for future research. We believe our current contributions provide a solid foundation for advancing the application of LLMs to biological sequence optimization problems.

Thank you for your thoughtful assessment of our work. We appreciate your recognition of our technical contributions and would like to address your concerns.

On the choice of Ehrlich functions over existing benchmarks

To validate the real-world applicability of Ehrlich functions, we conducted new experiments comparing optimizer performance on Ehrlich functions vs. 3 established lookup-based test functions:

• TFBind8: DNA transcription factor binding optimization (8-base sequence)
• TrpB: Tryptophan synthase β-subunit protein optimization (4-amino acid sequence)
• DhfR: Dihydrofolate reductase DNA binding optimization (9-base sequence)

For each benchmark, we evaluated the median cumulative regret (estimated from 8 trials) of 64 variants of our GA with different hyperparameter settings. We then computed rank correlations between algorithm performance on these biological benchmarks vs. comparable Ehrlich functions. The strong rank correlations confirm that Ehrlich functions effectively capture the structure of real biological optimization problems:

On the design of Ehrlich functions

We developed Ehrlich functions after carefully analyzing the limitations of existing benchmarks. As we explain in Section A.2.2, current benchmarks fall into several categories, each with significant drawbacks. Database lookup benchmarks are costly to construct and unnecessarily restrictive of the search space. Empirical function approximations often have spurious optima that are easy to find but not reflective of real solutions for the biological problem. Physics-based simulations are slow to evaluate, difficult to install/run correctly, and admit trivial solutions that score well but are not desirable.

Instead, we designed the Ehrlich suite to have the following criteria: (1) low compute cost, (2) well-characterized solutions, (3) non-trivial difficulty, (4) similarity to real-life applications, and (5) not already seen in training data. To the best of our knowledge, this is currently the only biosequence optimization benchmark to possess all 5 attributes.

On the relationship between Ehrlich functions and biological sequence optimization

Ehrlich functions capture four key properties of real biophysical sequence optimization:

1. Feasibility constraints: The vast majority of random sequences are non-viable/non-expressible
2. Epistasis: Non-additive effects between sequence positions
3. Position-dependent sensitivity: The importance of specific residues at specific positions
4. Motif constraints: The need for functional motifs to appear with proper spacing As we illustrate in Fig. 6, these properties are directly related to antibody-antigen binding.

On the choice of comparisons

We have added a comparison against LLOME-MargE with a smaller LLM (~226K params), trained from scratch. Although pre-training should not offer any additional advantages due to the lack of overlap between Ehrlich functions and the pre-training data, we choose this setting to be similar in model size and training to LaMBO-2. We evaluate on the f2 function (i.e. Ehr(32, 32)-4-4-4). Despite being ~2/3 the size of LaMBO-2, this model achieves the same min. regret using several thousand fewer test function evaluations (please compare to LaMBO-2 results in Fig. 3b):

This illustrates the strength of LLOME-MargE, even in small models without any pre-training.

On trade-offs

You raise an important question about the trade-offs between LLMs and specialized models. We explicitly address this in Section 6.1:

"For relatively easy optimization problems, since the performance of various methods is similar, using a specialized model with 0.01% of the parameters of an LLM may be more practical."

Our results show that the choice between generalist and specialist models depends - for medium-difficulty problems, the performance improvements of LLMs may justify their cost. For very easy or difficult problems, specialized models offer comparable performance with much less compute.

On essential references

We have an extended Related Work section in A.1, which discusses many LLM + evolution works. Also, ProteinGym is not a sequence generation benchmark.