NL2ProGPT: Taming Large Language Model for Conversational Protein Design

Q1: I found it very surprising that no recent papers from the Baker lab were cited (RFDiffusion, ProteinMPNN, etc) were cited. These are really important contributions to the field and highly related to your paper. Can you please comment on these?

A1: Our work focuses on text-to-protein generation, integrating structural information from protein representation models. (Refer to our “contributions”). RFDiffusion and ProteinMPNN are protein sequence design given protein backbones, which are not so-called highly related to our work.

Q2: I am extremely confused about why you did k-means on the 2-dimensional UMap representations. Can you provide more background about why this approach is more 'intuitive and reliable'?

A2: Directly clustering protein representation in high-dimensional space is unstable and hard to optimize. The UMap could reduce the dimensionality while maintaining the spatial relationship between proteins. In addition, our experiments also confirmed that this clustering result has certain biological significance, as shown in Table 2 and Table 3.

Q3: I don't understand how the rosetta energy function was used as a reward, since the energy needs to somehow be normalized by the energy of ground truth proteins with the desired attributes. You say, "Generally, protein structures with lower scores are more likely to be closer to the native structure." What is 'native structure' and how is it used?

A3: It is common knowledge that lower molecular energy corresponds to higher stability. For example, ProGen and ProtGPT2 both use Rosetta scores for evaluation (see the Related Works Section). For more details, please see (https://www.rosettacommons.org/demos/latest/tutorials/scoring/scoring). The native structure means the structure of natural proteins.

Q4: The rewards in eqs (9) and (10) have an optimum when the model just generates cluster centers, which will severely hurt diversity. When presenting your various eval metrics, I'm curious what would have happened if you had considered a simple baseline approach that just memorized a few exemplars.

A4: In fact, it is not generating the cluster center point that can obtain the optimal reward. Hitting the cluster representation means hitting the dimensionally reduced region (where the reward is the same within the region), not a single point. Therefore, such rewards do not hurt the generation's diversity.

Q5: I don't understand the overall evaluation setup. What does 'We randomly generate 1000 protein sequences from these models'. What metadata did you condition on? Was it 1000 different sets of metadata? How do you make this comparison fair when using models like ESM that don't have the ability to condition on metadata?

A5: We selected 10 text descriptions and obtained 1000 protein sequences. For ESM-2MR, we selected 10 text descriptions corresponding to protein sequences from the dataset, randomly masked 50%, and reconstructed the sequences.

Q6: The "Protein credibility prediction" paragraph should mention that progen also confirms wet-lab experiments.

A6: Thanks for your advice.

Conversational protein generation models, by offering a more intuitive, flexible, and interactive interface, provide a powerful tool for researchers in the field of biology, potentially driving deeper breakthroughs in related research. As discussed in the introduction's LLM section, they can serve as a tool for researchers in the field of biology, facilitating discoveries and advancing related research. As stated in our paper, "Consequently, LLMs offer unprecedented potential to advance protein discovery, particularly in the context of text-to-protein translation."

Hi,

Thanks for your response.

• Regarding Q3, I meant the 'ESM-2MR' model -- is closer performance to the GT (I'm sorry for the typo).
• Regarding Q4, the references for using text with protein design are the ProteinDT model (A Text-guided Protein Design Framework, Liu et. al. 2023. https://arxiv.org/pdf/2302.04611.pdf), or ProGen family (Large language models generate functional protein sequences across diverse families, Madani et. al., 2023, https://www.nature.com/articles/s41587-022-01618-2).

Q1: The method seems to use the same model in the framework for evaluation. For instance, they use ESM2 to embed the structure with reward constraints to an ESM-based cluster and use ESMFold (built on ESM2) for structure prediction evaluation. Also, they use the consistency (with Rosetta) reward in Step 3 to constrain the model and evaluation. This may raise the question of model performance benefits from the inductive bias of these pretrained models and tools.

A1: We sincerely do not agree that our evaluation is unfair because ESMFold is only used for structure prediction, and the real score is obtained by the Rosetta scoring function. In addition, we do not think that using RL is unfair for comparison. First, we use multiple evaluation metrics to verify our model’s superior performance than other methods as shown in Figure 2. Second, we also report the performance of our method without RL in Table 1, still showing a better performance than the baseline model.

Q2: The paper claims to embed the structural information into the description, but it’s doubtful how much the structure is preserved. First, though the ESM2 paper claims their embeddings have structural information, it is still implicit. Second, though the case study shows some insight into the cluster representation, it’s unclear how much information UMAP (into 2-D) and k-mean can preserve, as we know the loss of information after the dimension reduction and the difficulty of clustering.

A2: From Table 2, we can see that our method can preserve quite high structural information in the generated proteins. It also should be emphasized that we provide a new flexible way to incorporate structural information for the protein language model that has not been explored before.

Q3: In Figure 2, it doesn’t seem the proposed method achieves the best performance in any measure. For instance, a similar approach — the GT model performs better in the first one.

A3: It is important to clarify that in Figure 2, 'GT' represents the ground truth protein.

Q4: As NL2ProGPT is not the first approach combining natural language and protein (e.g., GTprotein), this raises the question of motivation in which scenario the proposed method is necessary.

A4: Could you give a reference of GTprotein or do you mean the ground truth model ?

Q6： Questions:

1. Our approach exhibits strong scalability, allowing for the incorporation of additional protein representation models in the future.

2. We exclusively leverage ChatGPT to enrich the textual description of proteins. Only text is inputted; protein sequences are not included.

3. Our methodology involves concatenating protein textual descriptions with protein sequences for unified input during self-supervised training. Our focus is solely on obtaining protein sequences through text. During inference, protein sequences are fully derived from the model through textual descriptions.

4. We utilize the official GPT-2 checkpoints provided by Hugging Face: https://huggingface.co/gpt2.

5. GPT employs sampling during generation, providing control over diversity by adjusting the sampling strategy.

6. The combination of Figure 4, Table 2, and Table 3 indirectly validates the effectiveness of our clustering approach.

7. Since our model is developed as a protein generation model within a conversational system, we refer to it as conversational protein design.

Q1： I don't quite understand how ChatGPT is used to generate the text descriptions. Are these training examples from the above process with ChatGPT? If so, how did you do any verification on the quality of this dataset?

A1： We enriched textual descriptions using ChatGPT, specifically by leveraging the generated text from templates through ChatGPT to diversify the textual descriptions. For instance, the original statement "Provides a protein that contains GNAT domain and belongs to ESM_51 and ONTO_38." was transformed into "This protein presents attributes encompassing the GNAT domain and falls under the classifications of ESM_51 and ONTO_38." This process focuses solely on enhancing the diversity of textual descriptions, ensuring that the quality of the dataset remains unaffected.

Q2: Given that some of them models use text as inputs like the authors' approach and some do not e.g. Progen I am a bit confused as to how all the models are compared e.g. is each model fed a different input and what are these inputs?

A2: In fact, we mainly compare the quality of generated protein sequences through three metrics, i.e., conformational energy distributions, foldability-measured sequence pLDDT distributions and self-consistency distributions. Therefore, for those methods without text inputs, the protein sequences are generated without text constraints.

Q3:  When evaluating for generality and consistency are the metrics used the same as that were used for RL? (in which case it would be unfair since the model would be overfitting on the reward). Some clarification would be great.

A3: We sincerely do not think that using RL is unfair for comparison. First, we do not find any overfitting evidence in our experiment results. Second, we use multiple evaluation metrics to verify our model’s superior performance than other methods as shown in Figure 2. Third, we also report the performance of our method without RL in Table 1, still showing the better performance than the baseline model.