Likelihood-based Finetuning of Protein Language Models for Few-shot Fitness Prediction and Design

Thank you very much for your feedback - we appreciate the review. Thank you for acknowledging the simplicity and applicability of our fine-tuning approach as a strength, whilst other reviewers discounted those same traits. Responses to your comments are given below. If we have sufficiently addressed your concerns, we would kindly ask that you champion the paper during the reviewer discussion phase, or accordingly, please let us know if further clarification is needed.

Relationship to Kermut:

We thank the reviewers for highlighting Kermut! Kermut does indeed outperform ProteinNPT on the ProteinGym suite of fitness prediction tasks. There are a few reasons why we did not consider it as a fair baseline in our work:

1. Kermut introduces a novel “biologically meaningful” composite kernel function with which to compare two given proteins. However, this composite kernel utilizes vastly more information than our method (and the collection of baselines we ultimately chose) - in particular, they make use of an inverse folding model that provides structural information to three of their four kernel components. Whilst we agree with the reviewer that this is a neat approach, and one that would likely improve results, including structural information is orthogonal to our work and we instead solely focus on the valuable task of fine-tuning sequence-based PLMs.

2. Kermut is not SOTA without structural information. I.e. Their ablation demonstrates that the sequence-only kernel, with representations from ESM-2, is outperformed (in Spearman random data setting) by ProteinNPT (0.744 - 0.033 = 0.711 versus 0.73). We directly compare to, and outperform, ProteinNPT.

3. Ultimately, we see Kermut’s primary contribution as “how to incorporate the information contained in PLMs into GPs (e.g. to provide meaningful uncertainty estimates)”, an important research topic in its own right. However, we argue that this is orthogonal to our work, and potentially could be combined in future work. On this point, we have added further discussion to our paper making explicit these points.

Observed Trends:

As the reviewer correctly points out, it is somewhat surprising that the proposed ranking approach is still effective in some landscapes at N=512. We see a clear trend that as N increases, the performance gap of the proposed likelihood ranking approach shrinks relative to the standard parametric regression approach.

To support this claim, and the main claims made in the paper, please find additional fitness prediction results for N=24 and N=48 in the Reviewer vxYr (Experimental Analysis) section.

We will include per-landscape results in the Appendix, and also Error bars in the Spearman results.

Zero-shot values for base PLMs are provided in Table 11 in Appendix B.

Objective as the likelihood of a generative process:

Please could you expand upon how Pearson correlation could be used as a training loss?

Thank you very much for your review. Responses to your comments are given below. However, we highlight a number of misunderstanding and factual errors in the provided review. We kindly request that you re-evaluate your review in light of these clarifications. If we have sufficiently addressed your concerns, we kindly ask that you raise your score accordingly, or let us know if further clarification is needed.

One fundamental misunderstanding repeated multiple times is that “The paper claims that ranking-based finetuning outperforms supervised finetuning in fitness modeling, as shown in the experiment.”

To clarify, our proposed approach directly adapts PLM likelihoods via a ranking-based loss function and is in fact still a supervised fine-tuning approach. Both settings, regression and ranking, utilize a supervised learning paradigm via ground truth fitness scores in a ProteinGym DMS. Our results demonstrate that by directly applying ranking losses to the PLM likelihoods we can better utilize limited labelled data.

Furthermore, the two comments “... However, it's unclear if this claim also applies to auto-regressive models like esm and ProtGPT.” and “Auto-regressive PLMs should also be considered as baselines in these tasks.”

Contains a misunderstanding of the models used in our work. Our paper introduces fine-tining methods specifically for conditional autoregressive models, like the family-based PoET in Section 4.1 “Conditional scoring functions”, and for masked language models ESM-1v and ESM-2 in Section 4.2 “Scoring functions for masked PLMs. That is, ESM is not an auto-regressive PLM as the reviewer suggests.

Another misunderstanding is that our work targets open-ended optimization via search. In the final paragraph of our Introduction, we state “...we study the modelling problem in silico which mimics ground truth values being available from wet lab experiments.”

We agree with the reviewer that search-based methods are an important field in proteinomics, however, it is not the focus of our work. Therefore, we respectfully refute the reviewer’s claim that methods such as directed evolution, PEX, AdaLead and LatProtRL [1,2,3] are “essential references not discussed”, and “should be included and studied”.

Leveraging the large collection of datasets available in ProteinGym allows for more concise evaluation of the proposed methodologies. This is a common approach taken in ProteinNPT (Notin, P. et. al. NeurIPS 2023), PoET (Truong Jr & Bepler, NeurIPS 2023), Kermut (Groth, P.M. et al. NeurIPS 2024) and Metalic (Beck, J. et. al. ICLR 2025). Conversely, it is well known that creating a biological meaningful oracle model to provide feedback, as per the reviewer suggess, is arguably a more challenging task than the optimization itself [4, 5].

[4] Buttenschoen, M., et. al. Posebusters: Ai-based docking methods fail to generate physically valid poses or generalise to novel sequences. Chemical Science, 2024

[5] Surana, S. et. at. Overconfident Oracles: Limitations of In Silico Sequence Design Benchmarking, AI for Science Workshop at ICML 2024

“...discuss DPO used in protein design more…”:

We kindly thank the reviewer for their references and will expand upon our current Section 4.5. “Relationship to preference learning for LLMs” where we discuss the benefits and limitations of DPO for our setting, e.g. that the KL penalty term hinders PLM adaptation in low-data settings.

In fact, we do cite work suggested [2] in this section, and in Section 2 Related Work (Page 2). Whilst it is concurrent work, they focus on a non-conditional autoregressive model, limiting their application to the family-conditioned model PoET, and therefore will not achieve our SOTA results (Table 1 and Figure 1).

Furthermore, [1] introduces a structure conditioned generative LM, pre-trained using DPO. Whilst this is an important task, it is orthogonal to our task of fine-tuning pre-trained PLMs. Our focus (as in ProteinNPT, PoET and Metalic) is to adapt the performance of pre-trained models using limited (small-N) datasets. Our work makes no claims on the pre-training regimes.

Similarly, [3] introduces a preference-based fine-tuning scheme for a pre-trained structure-conditioned diffusion model. Whilst conceptually related at a high level, their goal is de novo generation of high binding antibodies, and they incorporate feedback from a physics-based energy oracle, more similar to the open-search setting discussed above.

For additional low-N fitness prediction results, please see Review vxYr (Experimental Analysis).

“No supplemental material”. This is not true. Other reviewers acknowledged the Supplemental materials.

Natural extension from ranking losses:

In fact, we feel this is a strength of our work: it is a natural and necessary extension of the current literature, and we have demonstrated it has wide ranging applicability to different families of PLMs, as per the comments from Reviewer AKmL and QopU.

Thank you very much for your detailed feedback - we appreciate the review. If we have sufficiently addressed your concerns, we kindly ask that you raise your score accordingly, or let us know if further clarification is needed.

Experimental Analysis

Clarification: We do provide a clear quantitative comparison of methods in the multi-round optimisation setting. We report the Area Under Curve (AUC) in all design experiment plots (in the legend), where methods are sorted by this metric. Also, full numeric values for all models, scoring functions and loss functions are provided in Table 13 in Appendix B.

Figure 1 presents our proposed ranking-based likelihood fine-tuning methods against a subset of baselines. This result is important and demonstrates that one can improve the multi-round performance of a “weaker” PLM to perform in-line with, or exceed, the recent SOTA baseline method ProteinNPT (Notin, P. et al. NeurIPS 2024).

To directly compare the fine-tuning strategies for a given PLM, regression (MSE) based PLM baselines are presented in Figure 2 (ESM-2) and Figure 3 (PoET) in Appendix B. These results demonstrate that for a given PLM, our proposed likelihood fine-tuning method is almost always preferable.

Please find additional low-N experiments in Reviewer vxYr (Experimental Analysis).

Baseline modeling

We respectfully disagree with the reviewer on this point. ProteinNPT is a SOTA general purpose PLM developed specifically for low-data regimes. An extract from their work: “we introduce ProteinNPT, a non-parametric transformer variant tailored to protein sequences and particularly suited to label-scarce and multi-task learning settings”. We demonstrate comparable (and in some cases better) performance than ProteinNPT with “weaker” PLMs via pair-wise ranking loss functions directly applied to the likelihoods. Furthermore, we tuned the learning rate and the number of training steps for all models evaluated.

Relation to Metalic:

This is concurrent work and takes a different approach to improve zero-shot predictions of a PLM. Metalic is an in-context meta-learning approach that “learns to learn” to adapt to available in-context sequences available at test time. It does so via a meta-training phase over a distribution of similar fitness prediction tasks. They leverage the same subset of ProteinGym landscapes (as per ProteinNPT and our work), and specifically target zero-shot adaption.

Section 6.2 is included in Section 6 to establish its contribution towards multi-round design. That is, we introduce PLM ensemble models that 1) explicitly model predictive uncertainty and 2) apply principled acquisition functions. Both advancements are aimed at performing BO in Result 2, rather than fitness prediction in Section 5.

Questions for Authors:

The assumption that our proposed methods achieved sub-optimal results in larger N data regimes is incorrect. Please allow us to clarify: Our motivation is driven by the multi-round design setting, where we follow the SOTA ProteinNPT experimental setup (Notin, P. et. al. NeurIPS 2023b). This consists of fine-tuning the PLM at each round and acquiring 100 new sequences per round, for a total of 10 rounds. This is a realistic biological design setting since many wet lab experimental platforms compute ~100 scores in a single “plate” in parallel. Therefore, our thesis is that improved fitness prediction at low-N datasets (as per Table 1), drives meaningful data acquisition and multi-round design performance (Table 13 and Figures 1, 2 and 3).

‘Evo tuning’

This is an orthogonal research direction. Indeed, test-time training methods have been demonstrated to improve model performance using an unsupervised objective on test sequences, or indeed on homologs [1]. Also, whilst PoET conditions on MSA sequences, and demonstrates a clear benefit doing so, our work focuses on demonstrating that directly adapting model likelihoods using small experimental datasets improves fine-tuning. In practice, both or a hybrid approach could be applied.

[1] Bushuiev, A. et. al. Training on test proteins improves fitness, structure, and function prediction 2025

Multi-round eval:

Yes, your intuition is correct. A random seed of initial data is sampled from the landscape (where the seed is fixed for all methods), so the initial training set is identical. Furthermore, we evaluate all methods across three seeds.

wt-marginal strategy is a standard approach to compute the likelihood of MLM mutations introduced in Meier J., et. al (NeurIPS 2021). Popularised due to its efficiency and the widespread adoption of ESM. Concretely, the method computes the log-odds ratio of the probability of the mutated amino-acid with the wild-type's amino-acid. It does this in a single forward pass of the wt sequence.

Applying the MSE loss directly to the likelihood scoring function results in unstable performance (Table 6 and Table 13 Appendix). We did not explore scaling the likelihoods.

Thank you very much for your detailed feedback. If we have sufficiently addressed your concerns, we kindly ask that you raise your score accordingly, or let us know if further clarification is needed.

Claims and Evidence

Your comment is correct, Figure 1 does not show that. In fact, Table 13 (Appendix) demonstrates that the ranking fine-tuning counterparts outperform regression methods for multi-round design tasks. We plot MSE-based PLM baselines in Figure 2 (ESM) and Figure 3 (PoET) in Appendix B.

To clarify, without modification, non-ensemble PLM methods do not estimate uncertainty. Thus in multi-round design experiments greedy acquisition strategies are used (based on the predictions only). We include ensemble baselines as they specifically provide a measure of uncertainty, allowing a principled way to trade off exploration and exploitation via (non-greedy) acquisition functions.

Experimental Analysis

The N in Table 1 refers to the number of sequences used for fine-tuning the PLM. We include fitness prediction results N=32, 128, and 512. Rationale for N=512 is to demonstrate (as you correctly point out the trend) that for higher N, the performance gap between ranking and regression shrinks. Our hypothesis is that in higher data regimes, the parametric heads have sufficient data to fully adapt and approximate the fitness, something which is not true in the critical low-N settings, such as N=32 or 128. We demonstrate that directly adapting the likelihoods (via ranking-based losses) in these settings achieves the best performance across model classes.

We agree with your suggestion of additional low-N experiments. Please find N=24 and N=48 fitness prediction results below which additionally support our hypotheses in low-N regimes, and the main claims made in the paper. We will include the Figure as suggested. Additionally, we will add individual landscape scores into the Appendix, and include errors in Table 1.

Additional N=24, N=48 single-mutant fitness prediction results

ESM-2 (650M) linear-head mse loss = (0.263, 0.308) vs wt-marginals ranking = (0.449, 0.466)

PoET linear head mse = (0.429, 0.472) vs likelihood ranking = (0.513, 0.541)

ProteinNPT (MSAT) = (0.394, 0.461) and ProteinNPT (ESM-1v) = (0.398, 0.422)

We note it is computationally expensive to fine-tune the broad suite of PLM methods in our paper and thus we are limited in the number of additional experiments. The set of eight single mutant landscapes comprise a representative set of DMS used as validation and ablation sets in ProteinNPT (Notin, P. et. al. NeurIPS 2023b), and Metalic, (Beck, J. et. al. ICLR 2025). Furthermore, we evaluate using five challenging multi-mutant landscapes that goes beyond those used in ProteinNPT. Is there a particular landscape the reviewer thinks would add additional support to our analysis? If so, we will endeavour to include it in our results. The fitness prediction and multi-round design are standard experimental setups and follow previous works, ProteinNPT, and Kermut (Groth, P.M. et al. NeurIPS 2024).

References

Regarding your observation that Brookes et al., 2024. is a missing reference, please kindly note that we cite and briefly discuss their work on Page 2. They do indeed introduce a theoretical perspective relative to elucidating epistasis, but their work is limited in the models they use. Whereas, the focus of our work is specifically on providing general recommendations for adapting the likelihoods of pre-trained PLMs, a setting which they do not address. We thank the reviewer for highlighting the subtlety with regards to Gordon et al., 2024, and we will update the manuscript.

Spearman

We believe there is a misunderstanding here: whilst ranking-based fine-tuning parameterises a Bradly-Terry model, the underlying PLM still predicts a sequence (or mutation) likelihood. It is these likelihoods that we compute the Spearman correlation with respect to the ground truth fitness scores in ProteinGym. This is standard practice in the literature, e.g. Krause, B. et. al. 2021.

Real-world situations

Our results demonstrate improved fitness prediction (Table 1) and improved multi-round design (Table 13) when leveraging ranking-based fine-tuning in low data settings across a broad range of popular PLMs. Recently, we have seen the proliferation of pre-trained PLMs across many protein engineering tasks, and our work provides actionable insights for practitioners to get better predictive performance from existing pre-trained models, with less data.

Likelihood-based title

In Table 6 (Appendix) we ablate methods that directly fine-tune the model likelihood, and those that apply a parametric head to the representations. The results demonstrate that ranking-loss applied to the output of the parametric (linear) head also performs poorly. Our key recommendation is that our results support directly fine-tuning model likelihoods using a ranking loss function.