Conditional Enzyme Generation Using Protein Language Models with Adapters

Our Response to (1): Thank you for your support of our study and for the insightful feedback. We would first like to clarify our dataset splitting details. In Table 2, "Swissprot Heldout 70%" and "Swissprot Heldout 90%" are held-out protein sequences used only for the evaluation of perplexity. By contrast, we used the "Swissprot Heldout ECs" split to evaluate generation quality from OOD EC numbers. This split was constructed by holding out all protein sequences associated with certain EC numbers. These sequences are rarely homologous to the training sequences (as they belong to different enzyme families entirely), and nearly all have <40% identity to the training set. Therefore, when we ask the model to generate EC numbers that are OOD (not present in the training set), this is a very difficult task, and we see underwhelming performance in Figure 4D. We must admit that there is a lot of room for improvement here; by being one of the first to explore this question, we encourage others to tackle it with different approaches.

Our Response to (2): As for the heldout ECs, we note that they are sampled across diverse EC numbers at level 4 (X.X.X.X), based on the "medium" split for Task 2 in the CARE benchmarks (https://arxiv.org/abs/2406.15669v1, details in Table 3 and Figure A.3B). In fact, these EC numbers are intentionally spread across functional space (details in CARE Benchmarks Appendix A.2). This essentially means that if 4.2.1.20 is held-out, there will still be sequences with other EC numbers falling under 4.2.1.- (similar functions) in the training set. You may also note that the "hard" split in the CARE benchmarks is more difficult because all EC numbers under a certain EC level 3 (X.X.X.-) are held out. Thus, the most similar EC numbers still present in the training set in that case would be X.X.-.- . However, this presents a more difficult task, so we only explored the "medium" split in this study. We hope this clarifies why there is no data leakage and that the held-out EC numbers were thoughtfully chosen to be spread across function and sequence space. Happy to explain further if there are remaining questions and/or concerns!

The reviewer also raises an interesting point about more detailed analysis of the generated sequences. It is correct that certain enzymes have important domains, such as active sites, which may be more highly conserved. However, because most enzymes have complex and poorly understood sequence-function relationships, we have not had a chance to examine these subdomains in detail and primarily focused evaluation on local sequence alignment with BLAST. That being said, we are currently working on structure evaluation of generated sequences (using pLDDT), which we be included in this revision as another metric to evaluate generation quality. We are also performing additional experiments to strengthen the overall study, as suggested by the other reviewers.

Please let us know if you have any further questions or concerns that we can address.

Our Response to (1): Thank you for taking the time to review our paper. We have made some significant revisions to our manuscript and uploaded the revision as a new PDF file. We would first like to clarify that we ensured splitting would not have data leakage. Namely, the OOD EC numbers evaluated during generation were entirely held out. The representations we used also do not have data leakage. The Onehot encodings and DRFP embeddings are fixed representations, thus they are not learned. While the CREEP representation is learned using contrastive loss, we clarify in the main text that “there is no data leakage in the CREEP representation, as it was trained with the same held-out EC numbers.” More details on the CREEP training procedure can be found at https://arxiv.org/abs/2406.15669v1.

Our Response to (2): This is a valid point, although we would like to note that we are one of the first people to explore the task of generation of OOD enzyme functions. An important contribution of our study is showing how difficult this task is, while also highlighting its importance to the community. We approached this task not knowing if OOD generalization is even possible, given publicly available data. We hope that future work will be able to tackle this question with different approaches.

That being said, we would like to underscore one of the main contributions of our study: showing that adapters are a framework for conditional generation from PLMs that is flexible to many types of conditioning. By contrast, previous studies using adapters for PLMs have been limited to conditioning on structure. To make the broader utility of ProCALM more compelling, in this revision, we have trained new ProCALM models that utilize natural-language descriptions to generate proteins more broadly, not just enzymes. After benchmarking against ProteinDT, we found that ProCALM generates higher quality sequences and generalizes better (see Figure 4).

Additionally, we’ve now evaluated the generated sequences with pLDDT and found that our generated sequences have structures with high predicted confidence and are comparable to or better than existing methods (see Figure 2C and Figure 4E). We hope that these revisions convey how our study is compelling and useful. Please let us know if you have any further questions or concerns that we can address.

Our Response to (1) and (2): Thank you for the detailed feedback. We have made some significant revisions to our manuscript and uploaded the revision as a new PDF file. In our revision, we have included a more detailed discussion of previous works in the context of our current work, including a discussion and citation of LM-Design (see Related Work). While previous studies using adapters have been largely limited to conditioning on protein structures (including LM-Design), we would like to highlight that ours (ProCALM) is the first study to explore flexible conditioning with multiple types of inputs and explore OOD generalization in function space. We have added the following text to the revision to better explain this: “LM-Design is particularly interesting because the authors also evaluate generalization toward unseen protein folds (Zheng et al., 2023). Conditioning on structure is useful, as structure often determines function, but a goal of protein engineering is often to find proteins with novel functions (such as for new-to-nature enzyme activity (Arnold, 2018)), without any known structure or sequence performing this function. Thus, there is a need to explore models that can condition directly on function and generalize to new functions.”

Our Response to (3): To better showcase the broad utility of our approach, we have added a significant additional task with new experiments and evaluations. Namely, we trained a ProCALM model that can conditionally generate from natural-language descriptions of proteins and found that it performs better than ProteinDT, an existing model that can perform this task (see Figure 4).

Our Response to (4): We would like to clarify that we did in fact calculate the sequence diversity among themselves: this is given by the metric called "90% clusters" in Figure 2B, for example. The metric is calculated as (number of unique clusters at 90% identity clustering)/(number of total sequences). We have updated the text to explain this more clearly. Evaluating with pLDDT is a good suggestion, and we have performed additional analyses and reported those results in the revision, finding that our structures are as confident or more confident that existing approaches (see Figure 2C and Figure 4E) . Regarding structure diversity, since sequence space maps to structure space degenerately, most structures for a given enzyme family should be the same rather than different. We have now confirmed this by evaluating structure diversity with Foldseek (https://github.com/steineggerlab/foldseek) clustering. We found that, for a given enzyme class, typically the (number of unique structure clusters at 90% coverage)/(number of total structures) is less than 5%, which is consistent between ProCALM and ZymCTRL and not too surprising. Thus, we chose not to include these results in the manuscript.

The suggestion to examine scaling effects is also interesting. We trained two new models with ProGen2 2.7B and 6.4B, but larger models did not necessarily result in improved performance. Please see Figure A.5 and A.6.

We hope that these revisions convey how our study is compelling and useful, reiterating that we are the first to explore the use of adapters for conditioning PLMs beyond structure. Please let us know if you have any further questions or concerns that we can address.

Thank you for your feedback and for pointing us to these highly relevant papers. We have made some significant revisions to our manuscript and uploaded the revision as a new PDF file. While previous studies using adapters have largely focused on conditioning on protein structures, we would like to highlight that ours (ProCALM) is largely the first to explore flexible conditioning with multiple types of inputs (functions), which is an important application for protein engineering. In our revision, we have included a more detailed discussion (and citation) of the previous works above, in the context of our current work (see Related Work).

In terms of technical novelty, we propose and demonstrate the utility of having "parallel adapters" for multiple types of conditioning (e.g.. taxonomy and enzyme family), which has relevant implications for protein generation. To better strengthen our main contributions, we have included a few significant additions in this revision:

1. To make the broader utility of ProCALM more compelling: We have trained new ProCALM models that utilize textual descriptions to generate proteins more broadly, not just enzymes. After benchmarking against ProteinDT, we found that ProCALM generalizes better and generates higher quality sequences (see Figure 4).
2. To provide additional insights on the ML side: We have explored scaling effects for ProCALM by training larger models and found that larger models do not necessarily result in improved performance (see Figure A.5 and A.6).
3. To make the evaluation more convincing: We have evaluated the generated sequences with pLDDT and found that our generated sequences have structures with high predicted confidence and are comparable to or better than existing methods (see Figure 2C and Figure 4E).

We hope that these revisions convey how our study is compelling and useful, reiterating that we are the first to explore the use of adapters for conditioning PLMs on function (EC number, natural language, etc.), not just structure. Please let us know if there are further experiments that could be included here to improve our paper.