MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering

Thank you for your appreciation of our model, dataset, and code. We address your concerns below.

Q1: Details of PLM representations and delta features

As detailed in Appendix A.1, the PLM representations used in this study are residue-level embeddings. Regarding the scale of $h_{\Delta}$, we demonstrate the following:

• We calculate the average L2-norm on the MutaPLM dataset, which is 9.90 for the wild-type representations $h_{wt}$ and 0.35 for the mutational representations $h_{\Delta}$. Since our dataset only involves single-site (missense) mutation, this indicates that a mutation with very few residues will NOT lead to $h_{\Delta}$ being too small.
• The protein delta encoder amplifies $h_{\Delta}$, resulting in an average L2-norm of 1.04 for its outputs (delta features $z_{\Delta}$). Additionally, we argue that the orientation of $h_{\Delta}$ plays a more significant role in elucidating the evolutionary directions of mutations.

Q2: Experiments on prior mutation-related benchmarks

Thanks for your suggestions. We have tested MutaPLM on several mutation downstream benchmarks, including Spike-ACE2 [1] and avGFP [2] involving protein-protein interaction (virus-receptor binding) and protein properties (fluorescence intensity). Details of the benchmarks and our implementation are presented in the global rebuttal R2, and the results and case analysis are presented in Table 3 and Figure 1 in the uploaded PDF. We observe that MutaPLM achieves competitive performance with fine-tuned mutation models on these realistic benchmarks. We plan to extend our experiments to more protein fitness benchmarks in a future revision of our paper.

Q3: Comparisons with AlphaFold representations

Thank you for your insights. We manually inspect 500 test samples from our dataset and observe that 23 of them describe specific alterations of the protein structure explicitly.

To explore the impacts of structural representations for mutation explanation, one can use the first row of MSA representations calculated by the EvoFormer of AF2 [3] as the residue-level protein representations and feed the representations of the wild-type and the mutant into an LLM. Unfortunately, calculating AF representations on our dataset would require approximately 4000 GPU hours. Hence, as stated in our limitations, we reserve analyzing alterations of 3D structures for future investigation.

Q4: This paper is more suitable for the benchmark and dataset track

We argue that our contribution are three-fold, including the protein delta network, training strategies, and the dataset. We would like to address the innovations of our methodology, as recognized by Reviewer eM8T, in the following:

• Compared with prior works, we are the first to model mutations explicitly with delta features.
• We propose a chain-of-thought strategy for explaining and engineering mutations, which has not been explored in previous multi-modal protein-text LLMs [4, 5, 6].

For mutation interpretation, we have validated the interpretability of our model with qualitative evaluations in Figure 2 and Figure A2 in our paper, as well as additional quantitative evaluations on actual protein fitness benchmarks. For mutation engineering, we have reported the fitness optimization results on 6 realistic datasets in Figure 5 in our paper. Hence, we believe that our work bears potential in real-world applications, and is suitable for the main track of the conference.

Refs.

[1] Shifting Mutational Constraints in the SARS-CoV-2 Receptor-binding Domain during Viral Evolution.

[2] Local Fitness Landscape of the Green Fluorescent Protein.

[3] Highly Accurate Protein Structure Prediction with AlphaFold.

[4] Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models.

[5] ProtLLM: An Interleaved Protein-language LLM with Protein-as-word Pre-training.

[6] ProLLaMA: A Protein Language Model for Multi-Task Protein Language Processing.

Thank you for your positive comments on our presentation, dataset, methodology, and application values. We address your concerns and answer your questions below.

Q1: Misleading statement of PLMs in mutation explanation and engineering

We apologize for this misleading statement in our abstract. We clarify it as follows:

• In our introduction, we argued that while PLMs have shown effectiveness in zero-shot mutation explanation and engineering [1, 2], their implicit modeling of mutations through evolutionary plausibility is not satisfactory for practical needs.
• The inherent limitations due to architectural design and lack of supervision arise in modeling mutations explicitly.

We will change this statement to:

"However, due to architectural design and lack of supervision, PLMs model mutations implicitly with evolutionary plausibility, which is not satisfactory to serve as explainable and engineerable tools in real-world studies."

Q2: Comparison between MutaPLM and MLDE-based methods

Thanks for your insightful consideration. If MLDE refers to Machine-Learning-based Directed Evolution, the discrepancies between MutaPLM and prior MLDE-based methods are as follows:

• Explicit modeling of mutations. MutaPLM models mutations explicitly with delta features by an encoder-decoder architecture. In contrast, existing MLDE-based models [2,3] either model mutations implicitly with evolutionary plausibility or focus on the wild-type sequence instead of the discrapancies between the wild-type and mutant.
• Textual supervision. MutaPLM is a general framework for mutations that allows knowledge transfer across different wild-type proteins through natural language supervision. However, prior MLDE-based models are fine-tuned on a single wild-type protein with a phenotype of interest.

We perform comparisons between MutaPLM and fine-tuned PLMs on both mutation explanation and engineering. Please refer to our global rebuttal R1 for details of these baselines, and the PDF file for experimental results. We observe that MutaPLM consistently outperforms fine-tuned PLMs on both tasks, which demonstrates the effectiveness of our approach. Unfortunately, MLAEP is not comparable under our experimental settings, as the model is specifically designed for calculating the binding affinities of SARS-Cov-2 and its receptors and antibodies.

Q3: Further justification for the role of textual descriptions

We argue that textual descriptions are indispensable for general protein mutation modeling due to the following:

• Texts connect mutational knowledge from diverse proteins. The mutational effects of proteins are diverse and complicated, and the number of samples with a phenotype of interest is often limited. Therefore, constructing a multi-label dataset will lead to excessive classes and extremely imbalanced label distributions. In contrast, using texts helps combine supervision signals from diverse wild-type proteins and allows knowledge transfer across different phenotypes. This corroborates the intuition of CLIP models [4, 5] using texts instead of conventional labels for cross-modal supervision.

• Texts provide a user-friendly interface for mutation explanation and engineering. As stated in our introduction, the evolutional plausibility calculated by PLMs cannot meet the practical needs in studying mutations. Texts allow MutaPLM to interpret novel mutational effects from multiple facets and engineer proteins with user-defined properties even when the phenotype of interest has not been seen during training. Such capabilities cannot be obtained on a multi-label dataset.

For experimental justification, we have shown in our ablation studies that textual instructions bring 5.8% absolute gains in Recall@50 in mutation engineering. We further validate that the delta features have captured mutational knowledge through textual supervision. Specifically, we fine-tune the delta encoder and a regression head on two protein fitness datasets introduced in global rebuttal R2. From the results below we observe that textual supervision brings significant benefits to mutation explanation.

Q4: Applying MutaPLM on regression tasks

Similar to existing LLMs [6], we acknowledge that directly applying MutaPLM on regression benchmarks may lead to sub-optimal outcomes. We discuss strategies for handling quantitative predictions are as follows:

• Discretizing numeric values into several segments and instructing the LLM to predict the discrete values [6].
• Performing comparison (which mutation leads to increased fitness) instead of regression, partly inspired by the reward model in InstructGPT [7].
• Fine-tuning a regression head using delta features as additional inputs. The motivation is that the delta features have captured mutational knowledge from massive biomedical texts that could benefit regression tasks.

We implement the third strategy (fine-tuning with delta features) on two datasets, including Spike-ACE2 and avGFP. Please refer to our global rebuttal R2 for implementation details. The results are displayed in Table2 in the uploaded PDF file, where MutaPLM achieves competitive performance with fine-tuned mutation models.

Refs.

[1] Language Models Enable Zero-shot Prediction of the Effects of Mutations on Protein Function.

[2] Learning Protein Fitness Models from Evolutionary and Assay-labeled Data.

[3] Low-N Protein Engineering with Data-efficient Deep Learning.

[4] Learning Transferable Visual Models From Natural Language Supervision.

[5] ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts.

[6] Tx-LLM: A Large Language Model for Therapeutics.

[7] Training Language Models to Follow Instructions with Human Feedback.

Thank you for your appreciation of our task, methodology, and writing. We address your concerns in evaluation as follows.

Q1: Additional supervised baselines.

We have added supervised baselines, including fine-tuned PLMs, for both mutation explanation and engineering. Please refer to our global rebuttal (R1) for more details and Tables 1 and 2 in the uploaded PDF file for experimental results. We observe that MutaPLM consistently outperforms supervised baselines on both tasks, thereby demonstrating the effectiveness of our approach.

Q2: Temporal evaluation.

We appreciate your insightful suggestion. We perform temporal splitting by extracting the publication dates of the corresponding literature from PubMed [1] for each mutation in our dataset. Mutations studied before 2022 are used as training and validation sets, while those studied in 2022 and 2023 comprise the test set. The train/valid/test set comprises 156K, 8K, and 1.6K samples, respectively.

The experimental results for mutation explanation are below:

The experimental results for mutation engineering are below:

We observe that MutaPLM achieves promising performance on the temporal split and outperforms strong baselines, showcasing its potential in assisting real-world scenarios. We are working on evaluating other baseline models on the temporal split.

Q3: GPT-4 evaluation of mutation explanations.

We take 500 samples from our test sets and recruit a postgraduate from a top university who majors in biology to assess the mutation explanations of MutaPLM following the same categorization protocol as GPT-4. Below is the confusion matrix for manual and GPT annotations.

We observe that GPT-4 evaluation is consistent with human experts on 82.8% cases, although it occasionally misclassifies accurate predictions into relevant, and relevant or opposite predictions into irrelevant. We will report and discuss the manual evaluation results in a future revision of our paper.

Q4.1: Hyperparameters

Given the computational expense of the experiments, we do not specifically tune our hyperparameters. The rationales for our hyperparameter settings are as follows:

• The learning schedule and LoRA rank are derived from prior LLMs [2, 3].
• The batch size is selected to maximize GPU memory usage.
• The number of pre-training steps is determined based on convergence observations.
• The number of fine-tuning steps is based on evaluating the validation loss every 10K steps.

Q4.2: Protein literature dataset

As detailed in Appendix B.1, the protein literature dataset is collected from the Publication entry of proteins within UniProtKB/SwissProt [4] and PubMed [1]. We plan to publicly release this dataset in the future.

Q4.3: Details about MutaDescribe construction

All mutations are collected from UniProtKB Reviewed (Swiss-Prot), ensuring each sample has undergone expert review. The mutation explanations are obtained from the Phenotypes and Variants -> Description entry for each protein.

Refs.

[1] PubMed: The Bibliographic Database.

[2] ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training.

[3] BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine.

[4] UniProtKB/Swiss-Prot, the Manually Annotated Section of the UniProt KnowledgeBase: How to Use the Entry View.

Thanks for your newly-added experiments and clarifications! I have updated my score.