Metalic: Meta-Learning In-Context with Protein Language Models

Thank you for your feedback and for noting our comprehensive experiments. We appreciate your careful reading of our paper, and being open to discussion. We provide responses and clarifications below. Please kindly consider raising your score accordingly, or let us know if any topic remains unclear.

Weaknesses:

The primary concern seems to be with the lack of novelty of in-context meta-learning in the zero-shot setting, or the lack of improvement in the many-shot (n=128) setting. We address this concern here.

While the largest gains from per-task fine-tuning appear in the many-shot setting (n=128), we actually saw significant improvement in the few-shot setting as well (n=16). For example, we ran this experiment and saw that fine-tuning improved the Spearman correlation from .476 (± .004) to .490 (± .003) in the single-mutant setting. The fact that the meta-learning helps more than the fine-tuning as n decreases is not surprising: The more data is limited, the more meta-learning prior beliefs from related data is useful.

Moreover, it is not the case that fine-tuning does not help in the zero-shot setting; rather, it is simply not possible to use fine-tuning in that setting, since no additional sequences are available for it to be applicable. However, we believe that we have clearly demonstrated that the novel combination of in-context meta-learning followed by fine-tuning is highly effective, and that in-context meta-learning alone can be highly effective, even in the zero-shot setting, which is a surprising and informative result unto itself.

Finally, we show that our method outperforms alternative forms of in-context meta-learning and gradient-based meta-learning, even in the many-shot (n=128) setting. This suggests that, while beyond the scope of this study, Metalic can be used more widely than just for protein fitness prediction in the future. (We believe the main reason that Metalic is not state-of-the-art in the many-shot setting has more to do with implementation details and scale than methodology.)

We believe that our method has been demonstrated to be highly effective in pushing the state-of-the-art for zero-shot and few-shot protein fitness prediction, has a novel combination of in-context meta-learning and fine-tuning that helps in both the few-shot and many-shot settings (n=16 and n=128), and outperforms alternative meta-learning methods even in the many-shot setting. For these reasons we believe the work will have a high impact both in meta-learning and in low-data protein fitness prediction communities.

Questions:

1. “Why did you stop at 8M?” We too had this question. We stopped at 8M because the model started to become impossible to fit on a single GPU with significantly more parameters. We did evaluate the 35M model, in response to this request, in the 128-shot setting, for 35,000 steps. The Spearman increased from 0.541 (± 0.004) to 0.550 (± 0.001), suggesting there is some gain possible, but that the improvement is modest. We hypothesize the reason the gain is not larger is that the meta-learning stage drives most of the improvements from our method, and there is not yet sufficient data in the meta-learning stage to take advantage of much larger models. We expect this will change as ProteinGym grows and more data becomes publicly available. Thank you for the question!
2. “How many tasks are presented in-context to generate the figures… Is there overlap with the data used during the meta-learning phase?” We evaluate over eight held-out single-mutant tasks (and five held-out multi-mutant tasks). There are 189 tasks total. Since the eight single-mutant (and five multi-mutant) tasks are held-out from meta-training, there is no overlap with the data used in the meta-learning phase. The data used in-context at evaluation time consists of a support and query set. These sets are subsampled from the specific task that is being evaluated, one task at a time.
3. “unsupervised adaptation." -> Is there any evidence in the paper that this occurs?” Thanks for asking! Yes, there is! Since we evaluate on held-out tasks, when we evaluate the zero-shot setting, the model has never seen any labels for the given task. We dive into this phenomenon in Section 5, where we show that even in the zero-shot setting, the model is attending to other proteins in the context. This means that other proteins are useful, despite having no associated labels ever given to the model! We found this phenomenon to be fascinating and highly useful. Future work could explore the mechanisms enabling this capability.

Thanks for your response. We would like to highlight three points:

1. Contribution in the 0-Shot Setting: Regular meta-learning methods do not typically address the 0-shot setting in supervised learning. In contrast, our work demonstrates the surprising emergence of unsupervised adaptation in this setting, which we believe is a significant novel contribution on its own.
2. Strong Improvements Beyond 0-Shot: The novel components of Metalic improve performance in other settings as well (16-shot and 128-shot). Metalic is also the strongest method in the 16-shot setting, which is only possible due to our contribution.
3. Reasonable Interpretation of Guidelines: Interpreting the ICLR guidelines to mean that any novel application (which we provide) must also introduce a novel method (which we also provide) and that the greatest improvement from that novelty must occur precisely where the method is strongest, seems like an unreasonably high standard that is unlikely to align with the intent of the ICLR guidelines.

We truly appreciate your thoughtful consideration of our work, and we hope these clarifications demonstrate the significance and broad impact of our contributions. We would be grateful if you might revisit your assessment in light of this added context.

Thank you for the review, the detailed feedback, and for noting the strong performance of our method. We respond to your comments below. Please do let us know if we have addressed your concerns, and please do increase the score correspondingly if we have. Thank you for your help.

Weaknesses:

1. “I anticipate a more in-depth analysis of the correlation between PLM probability and fitness values in the context of meta-learning, especially when the authors assume that this correlation may not be reliable.” We believe there may be some miscommunication here. Rather than assuming the correlation may not be reliable, we actually do measure this correlation. Specifically, each of our baseline methods, in the zero-shot setting, use the protein likelihood directly as the fitness prediction. The results given in the zero-shot setting are precisely the Spearman correlation for PLM probabilities. Note that the fact that all baselines must rely on the PLM probabilities is the exact reason that Metalic is able to outperform them in the zero-shot setting. Using the additional meta-learning stage, ours is the only method that is able to learn how to use the PLM probabilities rather than directly using them for the zero-shot setting.
2. “Although in-context learning is a strength, its effectiveness diminishes for tasks that deviate significantly from the distribution of tasks used during meta-training, potentially limiting generalization to out-of-distribution tasks.” All of our tasks are non-parametric and are held-out from the training set, and are thus OOD tasks. It is true that in-context learning can struggle here, which is precisely the reason that the combination with fine-tuning in Metalic is so effective.
3. “Does not extensively benchmark against other recent meta-learning approaches that incorporate different strategies for handling fine-tuning or leveraging unsupervised data.” We compare to Reptile, since it is a widely used method for making gradient-based methods more efficient, which is necessary given the computational demands. To leverage unsupervised data in the query set, we combine Reptile with Metalic in Metalic-Reptile, which can additionally condition on the unsupervised data in-context, and still find it not worth the extra computation. Are there other methods for unsupervised meta-learning that you had in mind and expect to work well here?
4. “Metalic shows strong performance on single-mutant tasks but falls short of the state-of-the-art on multi-mutant tasks … a more in-depth analysis of this limitation would be valuable.” We have added an additional plot of the performance as we change the number of multi-mutant training tasks, both with and without the single-mutant tasks. This additional analysis allows us to see that more multi-mutant data is useful in either case, strengthening our analysis and argument. The plot is available in the updated manuscript
5. “additional evaluations on other benchmarks would provide a more comprehensive understanding of Metalic's strengths” We use ProteinGym since it has become the standard aggregated benchmark in the field. Note that ProteinGym is comprehensive and is itself composed of roughly 200 individual tasks, each of which is its own dataset collected by different researchers. ProteinGym aggregates available tasks, and thus spans many relevant protein-related tasks. We remain open to suggestions for tasks that are not included in ProteinGym and would be beneficial for evaluation, but do not know of any such standard benchmarks.

Questions:

1. “The authors assume that the correlation between PLM likelihood and protein fitness may not be reliable. How does Metalic perform when the assumed correlation is weaker?” Please see answer 1) in the Weakness section.
2. “In-context learning relies heavily on the distribution of tasks used during meta-training. … are there strategies that could improve its generalization to such out-of-distribution tasks?" Please see answer 2) in the Weakness section.
3. “Given that Metalic does not account for fine-tuning during meta-training, how does this omission affect the model's ability to adapt to tasks with extensive fine-tuning requirements?” This is a great question, and one that we investigate extensively in Section 4.6. Specifically, we evaluate Metalic-Reptile, to account for the fine-tuning during meta-training. Metalic-Reptile is equivalent to Metalic, but adds the objective from Reptile that accounts for fine-tuning during meta-training. It turns out that even when fine-tuning, the improvement from Metalic-Reptile is marginal, and the increased computation is large. When controlling for the total number of gradient computations, Metalic-Reptile increases implementation complexity with no clear advantage.

Thank you very much for your review and feedback. Thank you for noting that the performance is strong and that the experiments are thorough. We respond to the weaknesses and questions below.

Weaknesses:

“This work is the limited number of datasets used in evaluating… a partial k-fold cross validation would be convincing” We agree that evaluating over diverse datasets is an important consideration. While a k-fold evaluation would be prohibitively expensive (approximately 192 days per seed with each evaluation over eight datasets), we do take care to supplement with multi-mutant evaluation and ensure that the single-task datasets match prior work and are diverse in terms of data and difficulty. The original justification, taken from Notin et al. (2023), is that “together, these 8 assays cover diverse protein families in terms of taxa, mutation depths, sequence lengths and MSA depth.” Moreover, the Spearman correlation of our models ranges from 0.139 to 0.852 over the 8 datasets, further suggesting a broad range of difficulty within our evaluation set.

“On a more minor note, the exact implementation of the algorithm would benefit from a slightly more detailed treatment” Thanks for noting the difficulty! To make the exact computation clear, we have added a figure with more architectural detail to the appendix. (Note that we will also release the accompanying codebase publicly upon acceptance to ensure that the exact computations are clear and reproducible.)

Questions:

1. “How does performance change when training and evaluating on a separate set of proteins?” We chose the evaluation set to align with prior work (Hawkins-Hooker et al., 2024 and Notin et al. 2023), and do not have the computational resources to run a k-fold evaluation, but appreciate the question.
2. “How do inference and training costs of Metalic compare to the Reptile ablations?” Reptile inference costs are similar, but the training is significantly more expensive. For every gradient update of Metalic, Reptile must use three times the number of gradient computations. (Note that if we wanted conditions to match exactly, Reptile would use 100 times the number of gradient computations, but this was impossible due to compute limitations.) For this reason, we originally compared to Reptile after training for only 15,000 steps. However, we have run more experiments for the rebuttal, and the Reptile experiments have now been trained for the entire 50,000 steps. We have updated the manuscript with these results.
3. “Vaswani et al. 2017 style figure in the appendix of the exact computations that occur?” Thank you for the suggestion, we have now added such a figure to the appendix in response. Note that we will also release the codebase to ensure reproducibility!

Thank you very much for your detailed feedback. We appreciate the review and thank you for noting the importance of the application area and the value of the analysis. Responses to your comments are given below. If we have sufficiently addressed your concerns, we kindly ask that you raise your score accordingly, or let us know if further clarification is needed.

Weaknesses:

1. “Can we get better zero-shot or few-shot performance via scaling pretraining parameters?” We did evaluate the 35M model, in response to this request, in the 128-shot setting, for 35,000 steps. The Spearman increased from 0.541 (± 0.004) to 0.550 (± 0.001), suggesting there is some gain possible from scaling the model, though the improvement is modest. We hypothesize the reason the gain is not larger is that our meta-learning stage drives most of the improvement over the baselines, and there is not yet sufficient data in the meta-learning stage to take advantage of much larger models. We expect this will change as ProteinGym grows and more data becomes publicly available.
2. “Results in Figure 3b and 3a don't seem too much better than baselines.” The multi-mutant results are indeed outperformed by ESM-IF1 and PoET; however, as we hypothesize in the manuscript, this limitation is likely due to available data and not due to methodological limitations. We provide evidence of positive scaling with the number of tasks in Table 4, and we have added an additional graph into the manuscript to depict this scaling. Moreover, our method is uniquely placed to take advantage of any newly released dataset in the future, since it is the only method that leverages a meta-training stage. This means that our method is the only method that will actually improve with more datasets released over time.
3. “More rigorous examination of the effects of varying query set size, etc. might be quite insightful.” We did perform some experiments to evaluate this, not reported in the paper, since they used older hyper-parameters. Primarily, we saw that in the zero-shot setting, performance did increase marginally from 0.461 (± 0.004) to 0.471 (± 0.005), when increasing the query size from 100 to 200. We could not increase the query size much more than this due to computational constraints. We also note that forming firm hypotheses from experiments on the query size is quite difficult, since the larger the query size the less variance there is in the gradient estimation.
4. “Some papers have pointed out that protein likelihoods don't always indicate protein fitness, and might be due to the training data biases.” This is exactly the reason that metallic is able to outperform the baselines. The baselines make this assumption in the zero-shot setting, whereas the meta-learning phase enables Metalic to NOT rely on likelihoods to predict fitness.
5. “Minor / meta comment…There has also been working showing that "evotuning" (finetuning on evolutionary neighbors)” We believe that fine-tuning on evolutionary neighbors could be used in conjunction with our method, in the zero-shot setting, not instead of it. Additionally, the ESM pre-training stage may already give our model a similar benefit, since it has had access to similar homologs in pre-training.
6. “From a writing perspective, authors might be able to increase impact by outlining real-world drug discovery scenarios where this is relevant.” We have added text to the introduction to clarify this point, discussing how protein fitness prediction can be used to optimize properties such as the binding affinity of a monoclonal antibody therapy to its target, or the thermostability of enzymes functioning at high temperatures.

Questions:

1. “Are there some tasks that are easier to learn vs. harder to learn?” Certainly some of the tasks are easier and some are harder. For example, in the 128-shot single mutant evaluation, the Spearman correlation for Metalic ranged from 0.139 (± 0.009) to 0.852 (± 0.020), with a median of 0.578. (The mean Spearman correlations, in order, are .139, .342, .445, .547, .610, .688, .794, .852.) The large range in Spearman correlation shows a broad range of difficulties for evaluation tasks.
2. “Do authors have hypotheses for why the performance gap between Metalic and baselines is so much more pronounced for zero-shot vs. few shot tasks?” Yes, the result was not surprising to us given the purpose of meta-learning. Meta-learning adds an additional training stage to learn prior beliefs and inductive biases from related data. The more limited the data is, the more useful it is to rely on prior data. We have made a note of this in Section 4.3.

Thank you for your thoughtful questions and detailed engagement with our paper. We address each point below, with specific attention to the applicability of zero-shot and few-shot settings, informed by discussions with our protein engineering domain expert. We hope these answers will help increase your confidence in our work.

1. Multi-Mutant Data: The limitation in performance is likely due to the difference in available datasets: We have 121 single-mutant datasets but only 63 multi-mutant datasets for meta-training (pre-training). We explore this hypothesis in Section 4.4, where we demonstrate that performance improves significantly with additional meta-training data, suggesting that the bottleneck is indeed the lack of sufficient multi-mutant data.
2. Real-World Application of Zero-Shot and Few-Shot Settings: Protein engineering often involves mutating residues to test or alter an aspect of fitness. To clarify the utility of zero-shot and few-shot predictions, we provide a practical example: Suppose a research lab discovers a monoclonal antibody with both reactivity and neutralization potential against a broad panel of viral strains. To investigate whether this monoclonal should be developed as a therapeutic, researchers may want to identify mutations within the binding pocket that could lead to viral escape. For a particular residue of interest, the researcher may need to mutate that residue to all 20 possible amino acids, requiring the design and expression of up to 19 additional protein variants—a significant investment of time and resources. Few-shot prediction models can save substantial effort and cost. For example, the researcher could assay fitness for 10 variants experimentally and use a few-shot prediction model to predict the fitness of the remaining 10 variants, reducing labor by half. This scenario corresponds to a 10-shot fitness prediction problem. Predictions improve with additional data, so incorporating variants from other residues in the binding pocket would further enhance accuracy. For instance, if escape profiles were already known (from previously published work) for 10 individual positions, predicting the escape profile of an 11th position becomes a ~200-shot prediction problem. (Note that a researcher could generate the entirety of this dataset themselves using a Deep Mutational Scanning Library. However, depending on the type of scanning library made, there may be a large upfront cost and the costs could scale significantly with the number of residues mutated or the length of the protein. So, it may be preferable to avoid a DMS altogether, or it could make sense to generate only a subset of these 11 residues within the library and then predict the rest.) Zero-shot prediction is even more impactful, as it eliminates the need for prior assays altogether, avoiding all upfront costs. While not all tasks lend themselves to few-shot or zero-shot prediction, many do (e.g., mutations sharing biophysical side-chain properties, such as electrostatic charge). Empirically, this is supported by our up to 85% Spearman correlation on held-out validation tasks. These scenarios were developed in consultation with a protein engineering domain expert actively conducting wet lab research, underscoring their relevance to real-world practice.
3. Task Difficulty: Differences in the amino acid sequence and the property being assayed produce distinct challenges for modeling. The tasks are derived from Notin et al. (2023), who note that they “cover diverse protein families in terms of taxa, mutation depths, sequence lengths, and MSA depth.” Additionally, the properties of these proteins vary significantly. Both the protein sequence and the assessed property will affect difficulty. For example, in tasks that involve predicting epistatic effects on viral escape, protein fitness will be a highly complex function of the protein sequence. In contrast, in tasks such as direct binding interactions, knowing the effects of one mutation gives considerable information about neighboring mutations, resulting in an easier protein fitness prediction task. Additionally, as mentioned in answer 1), there are known issues due to data constraints in the multi-mutant setting. While we understand the reason that some of these tasks are difficult, either due to data constraints or more complex relationships between the protein sequence and its associated fitness, systematically varying evaluation data by the protein sequences and properties would be a useful feature in a standard benchmark, and such an effort would represent a valuable future contribution.

Thank you for noting that you believe the ICLR community would benefit from this paper. We hope these responses address your concerns and further increase your confidence in our work. We greatly appreciate your engagement and detailed feedback.