We appreciate your time taken to engage with the work. Thank you for noting the new perspective we frame fitness with, the paper’s rigor in the derivation of claims, and clear structure. Below we aim to address the questions raised:

“Could you elaborate on how the choice of loss function and measurable value impacts the influence function results? Specific examples could enhance understanding.”

We will update our work by adding the following statement, “When calculating influence functions, the loss function must match that used at training time, as this ensures the influence calculations reflect the actual optimization landscape the model encountered during training. For example, in language models this is the cross-entropy loss and for regression models this could be mean squared error. The measured value computed from the model's prediction may be any arbitrary differentiable function, allowing us to analyze different aspects of model behavior. In our analysis, we chose likelihood as our measured value because of its predictive capability towards zero-shot fitness calculation. This choice helps us identify which training examples most strongly lead to our findings in Section 4.3.”

“Could the authors provide a more extensive comparison across different architectures (e.g., autoregressive vs. masked models) to see if the observed trends are generalizable?”

This is an important question and we aimed to address the differences in architecture by showing multiple model scales for commonly used masked and autoregressive models in Figure 2b and 2c respectively.

“Would results vary significantly with different pretraining datasets, especially those curated with diverse or balanced sequence representation?”

We also find this question very interesting and the next natural question to extend on this work. As we do not have the resources to train a model on well curated balanced data, we leave this as a question for future research. We conjecture that a balanced dataset would still exhibit power laws over data influence, but would hopefully see less low and high likelihood sequences improving overall performance.

“Have the authors considered methods to explicitly model epistatic interactions in conjunction with the proposed likelihood-based framework? This could potentially address the limitations associated with linear fitness predictions.”

This is an important future direction for the work. Most of the DMS substitution datasets that we utilized often rely on only single mutants, making the linear fitness prediction a successful metric on this test bed. Kantroo et al. 2024 [1], which proposes a single forward pass PLL metric relying on a finetuned ensemble, gets towards this ambition by using the entire sequences PLL to calculate a fitness score, but ultimately found it to underperform linear fitness prediction.

[1] Kantroo et al. 2024, “Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation”

Thank you for your review and feedback on the work. We appreciate you noting the generalizability of our analysis and the role of influence functions to causally link behaviors to training data. Below we detail our response to your questions and feedback.

We’ll make sure to fix line 230 to reflect Section 4.2 being in the main body of the text. Thank you for catching this.

“Line 347, why can only 10K protein sequences from UniRef50 approximate the distribution of training data of EMS-2?”

While 10K samples is much smaller than the entire sequence distribution of UniRef50, our goal was to understand the broad scale distribution of influences across the training dataset so this value was chosen as it both has sufficient sequences to explore many different parts of sequence space while being resource aware. Calculating influence functions in itself is quite expensive, requiring the estimation of an inverse hessian product that takes a significant amount of memory and FLOPs. We believe this value to be large enough to capture our distribution of interest fairly in a tractable manner and examine influence distributions against other subsets of identical size for fairness.

I read the response and thank the authors for their rebuttal.

I raised a number of concrete questions in my review. Can you please respond to them?

Note that many of the other reviewers gave positives scores for the paper, but their reviews were fairly short and high-level. It should be treated as gating for paper acceptance to respond to my review.

“I didn't understand figure 1B. [...] Why would the overall likelihood of the wildtype be correlated with the size of this set?”

The fluctuation in the size of plausible mutations varies with likelihood as the likelihood of wild type gives information about how much uncertainty the model has about residues. In particular, as the model gets more confident (higher likelihood) the number of mutations that are likely to be sampled goes down. When the log likelihood is zero, the model expects there to be no variation in the protein sequence at all. This figure extends Figure 1 of Hie et al. 2023 [8] adding more depth to the proposed hypothesis of efficient evolution.

[1] Ding et al. 2024, “Protein language models are biased by unequal sequence sampling across the tree of life” [2] Devlin et al. 2018, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” [3] Meier et al. 2021, “Language models enable zero-shot prediction of the effects of mutations on protein function” [4] Notion et al. 2023, “ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design” [5] Lin et al. 2023, “Evolutionary-scale prediction of atomic-level protein structure with a language model” [6] Kaplan et al. 2020, “Scaling Laws for Neural Language Models” [7] Weinstein et al. 2022, “Non-identifiability and the blessings of misspecification in models of molecular fitness.” [8] Hie et al. 2023, “Efficient evolution of human antibodies from general protein language models”

“Also, why is the single-pass approach actually necessary in your experiments?”

PLL has been an important metric within the pLM community with its extensive use within ESM-2 [4] and as a validation metric in protein sequence generation works. This mirrors the commonly used perplexity metric in NLP to assess characteristics of language models; since the ESM series of models are trained with the masked-language modeling objective, existing literature uses this O(L) variant. Please see Supplementary Materials, Appendix 2.2 in Lin et al., 2024 [5]. With an average length of 300 residues in these databases, our algorithm provides a 300x speedup in wall clock time democratizing the use of this computation.

To put this speedup into perspective, let's compare calculating PLL for a protein database to the costs of training a pLM from scratch. ESM-2 650M was trained for roughly 25 epochs (2M tokens per batch / 300 residues per protein * 270K steps / 65M sequences in UniRef50 = 25 epochs) [5]. Calculating PLL on all of these sequences at 300 residues per protein would be 300 inference passes over the entire dataset, using the estimation from Kaplan et al. [6] that a full training step is computationally equivalent to 3 inference passes, means that this would be the same as training for 100 epochs. In sum, if any researcher wants to calculate this statistic for a large database they need 4 times the resources required during the training of a modern foundation model.

“The paper focuses on DMS datasets that measure different attributes of a protein [...] yet it uses assay-independent models.”

We agree that this does upper bound performance, but this is the norm for studying zero-shot fitness prediction like in Meier et al. [3] and ProteinGym [4].

“What if, for example, the correlations you are seeing are due to the fact that low-likelihood proteins are from the sort of species where the DMS datasets are based on high-level observations about organismal survival, instead of low-level molecular biology measurements like binding?”

This motivates a good follow up experiment to disentangle this effect. We’ll recreate the panels in Figure 2, but split by selection criteria, then recalculate Spearman correlations to ensure that this trend holds over different families.

“I think influence functions are cool and I enjoyed seeing that you used them to study protein LMs, but I don't think the inclusion of the influence function results drives the paper's story enough to be included.”

Although our results reinforce a previous method, we found it important to increase the thoroughness of the paper and used the results to motivate our intervention. Influence functions provide a specific mechanism to study how training data impacts our feature of interest: likelihood.

“Can you please explain the connection to Qin and Colwell in more detail?”

Qin and Colwell observe a power law tail over the eigenvalues of the covariance matrix and we find one over the distribution of influence values. Moreover, Qin and Colwell found that the power tail signal harmed performance in parallel to how our high likelihood sequences perform poorly. This provides evidence towards the results of Weinstein et al. 2022 [7] where phylogenetic signals corrupts fitness signals. We will update the paper to better convey this.

“Can you elaborate on the result of sec 5.2? How is this observation actionable? How does it inform the other parts of the paper?”

Because sequence overlap improves likelihood most, we take the lowest E-value sequences as they would have the most overlap with the wild type during evo-tuning.

We appreciate the thorough review of this paper, and have attempted to address each of the points raised in the comment below. The feedback has assisted in creating better writing around key elements of the work.

“I struggled to see a concrete argument about why this is reflecting some specific issue regarding the structure/balance of protein LM pretraining datasets instead of just a basic impact of undertraining/overtraining. [...] What part of this observation is specific to proteins and why does it rely on a notion of 'preference'?”

This is a fair question that we will tighten our language on in the paper. We believe that preference best encapsulates our hypothesis. We’d first like to point out that the training procedures for the models samples roughly uniformly across a large protein database like UniRef50. If each sample has equivalent training emphasis, then why are some sequences presented with a lower or higher density than others after sufficient training time? Here lies the observation that the difference in observed probabilities must be a result of the data distribution, otherwise we’d expect all proteins to achieve a roughly equal likelihood.

The notion of “preference” in our work is that we examine how likelihoods are affected by data composition, and data composition originates from our human preferences when we decide what to sample and include (see Ding et al., 2024 [1]). Our work demonstrates that data has a big impact on pLM likelihoods; therefore, its inability to faithfully recapitulate the fitness landscape is implicitly due to humans’ preferences in aggregating the database. This is a helpful piece of feedback in improving the clarity, and we’ve updated the writing to reflect this.

Single Inference Pseudo Likelihood

We appreciate the feedback on language within the proof and will be updating our wording with the advice as follows: “Under a mask-consistent masked language model that sets masked tokens [...]” will become “Under a mask-consistent masked language model with a training scheme that sets training tokens [...]”. Our language comes from the original BERT paper from Devlin et al. 2018, in which they describe the algorithm as “If the i-th token is chosen, we replace the i-th token with (1) the [MASK] token 80% of the time (2) a random token 10% of the time (3) the unchanged i-th token 10% of the time.” In addition, we will introduce our notion of “mask-consistency” earlier.

“However, what exactly is [Pseudo Likelihood …]?”

We describe the exact algorithm in Section B.1 Algorithm 1. It is consistent with the proposed calculation in the comment.

“Do the new and old approaches provide the exact same output for all possible models [...]?”

Yes (under the typical assumption in ML that models have correctly converged during training). The output is not affected by the distribution that is examined, but on how tokens were perturbed during MLM training. This is why we needed the $\alpha$ and $\beta$ terms in our algorithm.

“Given that the actual inputs to the model are different, which means that the activations are different, it appears to me that the approaches would provide numerically different outputs [...]?"

We agree with the argument presented, and would like to highlight how our proof side steps this issue. Though the model would in fact see different activations, our proof relies on the downstream probabilities instead of activations. This makes it non-reliant on these activations.

Taking into account the feedback, we will make sure to make sure to emphasize the need for a model to be “sufficiently trained” in order to exhibit this capability.

We thank the reviewer for their constructive feedback and engagement throughout the discussion period. Given that the comment period closes tonight, we would appreciate the reviewer’s thoughts on whether our responses and clarifications have addressed their concerns regarding the initial score. If there are any remaining points we can clarify to help with this assessment, we would be happy to address them promptly.

We’re grateful for the positive reception towards our problem framing and simplicity. As noted in the review, there are many complicated methods that lead to marginal performance gains, so it was our aim to provide a simple well-motivated intervention.

“The experiment would be better if it includes more datasets to fully support some claims [...]”

While we agree that more experiments would add more confidence, we ran 217 unique different evo-tuning experiments to draw our conclusions and reinforce our findings.

“Also, I am wondering if Bert-like esm-3 or other smaller Bert-like pLMs may be examined.”

This is a good point, we’ll aim to include more results for ESM-3 [1] and ProtBert [2] in the camera-ready version. One minor technical note is that the substitution parameters for ESM-3 aren’t explicitly given in the work, so estimating PLL in a single pass will require us reaching out to the authors.

[1] Hayes et al. 2024, “Simulating 500 million years of evolution with a language model” [2] Elnaggar et al. 2021, “ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning”