STELLA: Leveraging Structural Representations to Enhance Protein Understanding with Multimodal LLMs

Questions:

4. Does STELLA show a significant enough improvement over other established models to justify its scientific meaning and contribution?
5. I’m curious about the model’s scalability because of the interactive demonstration. Can STELLA efficiently handle large-scale datasets or high-throughput predictions?

Response to Q4

• Sequence, structure, and function (text) are three core data modalities in protein biology. Structural data, in particular, plays a crucial role in uncovering biological functions. Despite the extensive structural data which are publicly available, such as the PDB and AFDB, there is still a pressing challenge to fully utilize these resources to delve into understanding of protein functions. Bridging the gap between structural data and functional or biochemical insights is essential to unlock their full potential for practical applications in both research and industry.
• To address this challenge, innovative approaches that integrate structural data with cutting-edge computational tools are urgently needed. STELLA represents a transformative solution, combining the strengths of LLMs with protein structure insights to deliver accurate, versatile, and interactive predictions across a wide range of protein-related tasks (as demonstrated in Appendix A.1 with Examples 1 and 2 in the revised manuscript). By achieving state-of-the-art performance in multiple tasks, STELLA demonstrates the potential of multimodal LLMs as a groundbreaking paradigm for advancing protein biology research. This advancement not only builds upon but also transcends the capabilities of traditional protein language models (pLMs), as highlighted by reviewers tXk1, cosX, and L7ds, who recognized the effective utilization of LLM capabilities as a key strength of this work.
• STELLA introduces a novel methodology for understanding proteins beyond the existing pLM frameworks and expands the boundaries of LLM applications in protein biology. This study paves the way for a new paradigm in protein science and computational biology, offering a more integrated and efficient approach to leveraging structural data for scientific discovery.

Response to Q5

• Scalability is a critical consideration for STELLA, especially given its interactive demonstration capabilities. Our model is designed with scalability in mind, enabling it to efficiently handle large-scale datasets and perform high-throughput predictions. Incorporating optimization techniques and leverage state-of-the-art frameworks such as vLLM[1] helps to accelerate inference. By leveraging the computational efficiency of LLMs and implementing optimization techniques such as batching and parallel processing, STELLA maintains robust performance even with increasing data volume. These optimizations can make STELLA a robust and efficient tool, suitable for both interactive exploration and high-throughput scientific workflows, paving the way for broader applications in protein science.

  [1] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J., Zhang, H. and Stoica, I., 2023, October. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles (pp. 611-626).

Thank you for raising several insightful questions for our work, which are addressed below.

Questions

1. While STELLA seems to enhance function prediction, I wonder if the model’s reasoning behind those predictions is easy to interpret. What measures, if any, have been taken to make its outputs understandable, especially for biologists who need to validate its findings?
2. Given that STELLA relies heavily on structured protein data, how does it perform when dealing with less common protein structures?
3. In my experience, we often work with proteins that lack precise structure or even sequence information in certain regions. How well could STELLA handle incomplete protein data?
4. Does STELLA show a significant enough improvement over other established models to justify its scientific meaning and contribution?
5. I’m curious about the model’s scalability because of the interactive demonstration. Can STELLA efficiently handle large-scale datasets or high-throughput predictions?

Responses to Q1

• We understand the critical importance of accurate protein function interpretation, especially when applied to biotechnological research. The potential consequences of false interpretations—such as leading research in the wrong direction or wasting resources—are indeed significant. While STELLA provides valuable insights into protein function predictions, we agree that incorporating confidence measures and uncertainty quantification in predictions is essential. This could include mechanisms for signalling the reliability of predictions, based on model confidence or uncertainty in the embeddings, as well as validating predictions through experimental feedback. We plan to explore these directions in future iterations of STELLA to improve the robustness and trustworthiness of the generated predictions and to make the STELLA's reasoning process even more transparent and accessible to domain experts.

Responses to Q2

• In the function prediction (FP) task, we employed a rigorous data splitting strategy based on sequence similarity, ensuring that the similarity between training and test sequences was below 40%. This approach was designed to simulate scenarios involving rare or uncommon proteins, providing a more realistic evaluation of STELLA's ability to generalize to novel and previously unseen cases.

Responses to Q3

• Thank you for highlighting this important consideration, as incomplete protein data is indeed a common challenge in protein biology research. To address this, we conducted an additional experiment to evaluate STELLA’s ability to handle incomplete protein structures. Specifically, for the testing data, we cut away the terminal 10% of the protein structures to simulate incomplete structural information and assess the model’s performance under these conditions. STELLA's performance, measured by the ROUGE-L score, shows a slight decrease from 0.5257 to 0.4915. Considering that the training was conducted using complete protein structures, this slight decline due to inconsistency still demonstrates the robustness of our model, indicating its capability to handle incomplete protein information in application scenarios.

Response to Q1

• Thank you for your insightful comments and constructive suggestions. We greatly appreciate your feedback, which not only highlights key areas for refinement and future exploration but also acknowledges the contributions of this work to the community.
• As for the technical details, we utilized the 1.4B parameter version of ESM3 (esm3_sm_open_v1) in our experiments. AFDB and PDB data were used as raw input, and embeddings were extracted using the official inference pipeline provided by the ESM3 framework. This ensured consistency, reproducibility, and alignment with established best practices.
• We recognize the importance of providing clear and comprehensive technical details and will incorporate this clarification into the manuscript to enhance its transparency and accessibility. Once again, thank you for your valuable feedback and thoughtful suggestions!

Thank you for raising 3 insightful questions for our work, which are addressed below.

Response to Q1

• Thank you for giving this construction suggestion. ESM3 underwent self-supervised learning in masked language modeling (MLM) mechanism using massive proteins across three modalities—sequence, structure, and function keywords—at the residue level. This residue-level multimodal interaction allows ESM3 to learn more fine-grained protein representations. In Figure 4, we provide a UMAP visualization showing ESM3's better representation ability than other protein encoders. Further details on the encoder can be found in Table 9. Also, ESM3 is widely recognized and utilized in the computational biology community, providing extensive validation and benchmarking. Its widespread use ensures that results can be easily compared and contextualized with other studies. In conclusion, ESM3’s balance of scalability, versatility, and high-quality representation learning makes it a compelling choice for protein-related research. We hope that these insights will assist future studies in selecting appropriate encoders for their specific objectives.

Response to Q2

• Thank you for your insightful question. We observed improvements across all metrics by extending the stage-2 training from 3 to 6 epochs. As a result, STELLA outperforms Prot2Text$_{LARGE}$, achieving state-of-the-art performance in the FP task.

Evaluation results of the FP task, comparing with existing work. Bold and italic indicate the best and the runner-up performance.

| Model/Method | BLEU-4 ↑ | BERT Score ↑ | ROUGE-1 ↑ | ROUGE-2 ↑ | ROUGE-L ↑ | |-----------------------------------------|-----------|--------------|------------|------------|------------| | Prot2Text$_{BASE}$ [abdine2023prot2text] | 0.3511 | 0.8430 | 0.5059 | 0.4271 | 0.4849 | | Prot2Text$_{LARGE}$ [abdine2023prot2text] | 0.3629 | 0.8520 | 0.5368 | 0.4560 | 0.5140 | | STELLA-ESM3-Llama-3.1-8B-Instruct (e3+e3) | 0.4024 | 0.8496 | 0.5218 | 0.4487 | 0.5041 | | STELLA-ESM3-Llama-3.1-8B-Instruct (e3+e6) | 0.4300| 0.8564 | 0.5423 | 0.4747 | 0.5257 | | Foldseek + Manual retrieval | 0.3627 | 0.8358 | 0.4799 | 0.4027 | 0.4586 |

• Although we have achieved SOTA performance with the above metrics in the FP task, we acknowledge that given the specialization and complexity of biological function descriptions, the quality of LLM responses cannot be fully captured by a single NLP metric. Therefore, we employ multiple metrics above to provide a more comprehensive evaluation of STELLA’s effectiveness. Additionally, recognizing the limitations of such conventional NLP metrics in protein-related tasks, we intentionally designed multiple-choice QA (MCQA) tasks to objectively assess STELLA’s performance.
• Regarding the concern about Table 3, to enhance clarity and conciseness, we have decided to remove the table and simplify the content for improved readability and flow.

Response to Q3

• Thank you for your constructive suggestions. We list several potential user usage scenarios and will include additional dialogue examples in the revised version of the manuscript to demonstrate these capabilities:
  • Providing comprehensive clarifications and addressing specific questions related to terminologies or concepts in biological processes.
  • Highlighting similarities and differences among proteins involved in the same biological process. (see example 1 in Appendix A.1 of the revised manuscript)
  • Suggesting novel biotechnological applications based on protein functionalities. (see example 2 in Appendix A.1 of the revised manuscript)
  • Offering iterative functional predictions according to domain experts' feedback or new experimental evidence.
• To sum up, scientific research is a long and iterative process of trials and errors. STELLA showcases transformative potential by creating a new LLM-based paradigm besides the protein language model (PLM) paradigm for protein biology research. STELLA is expected to act as a research copilot by offering efficient feedback, supporting continuous improvements, and empowering domain experts to make more informed decisions through dynamic, iterative interactions.

Thank you for raising several insightful questions (weaknesses) for our work, which are addressed below.

Response to Q1

• Thank you for highlighting this point. While it is true that PDB structures are derived from detailed research publications, the annotations in the PDB typically include only abstracts and keywords, which do not provide the comprehensive, expert-reviewed functional descriptions found in databases like Swiss-Prot. We have revised the phrasing in the manuscript to avoid any potential misunderstandings. Meanwhile, we want to emphasize the lack of functional annotations for proteins in the AlphaFold Protein Structure Database and to further enable exploration of the protein universe in which many of our proteins remain hidden. This work aims to develop new AI methods to discover the unknown knowledge of proteins. We appreciate your feedback, as it helps us refine and clarify our motivation more effectively.

Response to Q2

• We achieved state-of-the-art (SOTA) results in both the FP and EP tasks, with a ROUGE-L of 0.5257 in the FP task and an accuracy of 0.8885 in the EP task. Notably, in the FP task, the ROUGE-L score improved from 0.5041 to 0.5257 as we increased the stage-2 training epochs from 3 to 6. Similarly, in the EP task, the accuracy increased from 0.8806 to 0.8885 with the same adjustment in training epochs.

Response to Q3

• Thank you for your insightful feedback. We acknowledge that our original phrasing in the introduction may have unintentionally misled readers regarding the handling of “local and global structural factors.” To address this, we have revised the statement in the introduction to clarify our intent. Specifically, we aim to emphasize that enzyme function prediction often correlates with certain structural regions, such as active sites or binding pockets, rather than implying that our framework explicitly models local and global structural factors.
• Our focus in this work is on leveraging multimodal modeling to integrate structural information in a more general sense, without explicitly isolating or addressing these factors. We believe this clarification better aligns with the design and scope of STELLA and avoids potential misinterpretation. Thank you for helping us improve the clarity of our manuscript.

Response to Q4

• Thank you for your thoughtful question. First, we would like to clarify that STELLA does not completely remove sequence information. Specifically, the ESM3 encoder takes residue-level sequence information as its partial input. However, our study emphasizes the utilization of structural data as a complementary modality to sequence information, aiming to explore the potential of multimodal modeling based on structures in protein biology. This deliberate focus allows us to investigate how structural insights can enhance predictions and address challenges that sequence-based models alone may not fully resolve. We achieved SOTA results in the FP task, with a ROUGE-L = 0.5257，surpassing previous ROUGE-L=0.5140 of Prot2Text_Large.

Response to Q5

• We achieved a SOTA result with an accuracy of 0.8885 in the EP task by increasing the stage-2 training epochs to 6. Given the limited size of the Enzyme dataset, extending the training from 3 to 6 epochs allowed the model to better learn from the available data. This led to an improvement in accuracy from 0.8806 to 0.8885, achieving SOTA performance.

Response to Q6

• Thank you for your valuable suggestion. We have improved the clarity of the paper’s presentation. In response, we would like to clarify the roles of Figures 3 and 4: Figure 3 illustrates the length distribution of protein sequences in the dataset, highlighting the model’s applicability to functional descriptions across proteins of varying lengths. Besides the length distribution, we have also provided analysis of data label distribution of our OPI-Struc dataset in the Appendix. These analyses aim to offer a comprehensive understanding of the dataset’s characteristics. These efforts confirm that our dataset has been meticulously curated to provide a high-quality resource for the community, support robust model evaluation, and offer guidance for constructing future datasets. This attention to details reflects our commitment to advancing LLM-driven paradigm for protein biology research and laying a dependable foundation for future scientific endeavours.
• Figure 4 emphasizes the effectiveness of the three encoders in modeling proteins from the perspective of function annotation. Among them, ESM3 demonstrates superior performance, consistent with the model’s results on both the FP and EP tasks. Additionally, we have updated the referencing format for the figures as suggested.

> User: What are the main functions of this protein?[AFDB_ID: AF-Q9W3K5-F1-model_v4.pdb]
> STELLA: The main functions of this protein are to catalyze the ATP-dependent ligation of L-glutamate and L-cysteine, and to a lesser extent, of L-histidine and L-tryptophan.
> User: Compare the function of glutamate-cysteine ligase with that of glutathione synthetase. What structural or mechanistic features distinguish these enzymes, and how do they together regulate glutathione levels?
> STELLA: Glutamate-cysteine ligase and glutathione synthetase are both involved in the synthesis of glutathione, a critical antioxidant in cells. However, they have different functions and mechanisms of action. Glutamate-cysteine ligase catalyzes the ligation of L-glutamate and L-cysteine to form a peptide bond, while glutathione synthetase catalyzes the condensation of L-glutamate and L-cysteine to form glutathione. The two enzymes have different substrate specificities and reaction conditions, and they work together to regulate glutathione levels in cells.

> User: What are the main functions of this protein?
[AFDB_ID: AF-Q5KYR2-F1-model_v4.pdb]
> STELLA: The main functions of this protein are involved in the isomerization of 5-deoxy-glucuronate (5DG) to 5-dehydro-2-deoxy-D-gluconate (DKG or 2-deoxy-5-keto-D-gluconate).
> User: Describe potential industrial or medical applications of the isomerization catalyzed by this protein, specifically converting 5-deoxy-glucuronate (5DG) to 5-dehydro-2-deoxy-D-gluconate (DKG). How might this reaction be utilized in biotechnology or pharmaceutical production?
> STELLA: The isomerization reaction catalyzed by this protein has several potential applications in biotechnology and pharmaceutical production. For example, the conversion of 5DG to DKG could be used to produce novel compounds or to improve the efficiency of existing ones. Additionally, the reaction could be used to develop new biotechnological processes or to improve existing ones.
> User: Any further examples?
> STELLA: Yes, the isomerization reaction could also be used to produce novel antibiotics, anticancer agents, or other bioactive compounds. For example, the conversion of 5DG to DKG could be used to synthesize novel peptidoglycan structures, which are important for bacterial cell wall synthesis and could potentially be used to develop new antibiotics.

Regarding Q1, you have consistently failed to directly answer my question. Since you claim that your model can obtain information that is not available in the PDB database, I only ask you to provide one or two examples. Moreover, proposing a model and then asking the community to verify its broader applications does not seem to be a responsible approach. Furthermore, the novelty of your model is not apparent. Don't mention Reviewer CVwt to me, I question his professionalism. This work is clearly very similar to Prot2Text and ProteinGPT that I mentioned, and he irresponsibly gave a score of 10 without thorough investigation, which makes me question his motives. Also, Reviewer CVwt himself mentioned about your initial model version that "The model does not really push the state of the art" and is "missing technical details". I maintain a high level of skepticism about his high score of 10 under these circumstances.

Regarding Q2, I can accept that you achieved better performance after further training, although the improvement is quite limited, such as an increase of only 0.0035 on EP. Such a negligible improvement could be invalidated due to various factors, such as data partitioning and sampling strategies. However, this still clearly does not sufficiently demonstrate the role of 'iterative feedback.'

Regarding Q3, what is the fundamental difference between your model and Prot2Text and ProteinGPT? The difference between STELLA and Prot2Text is merely a change in the encoder and LLM backbone; you also said: "STELLA distinguishes itself through differences in dataset construction and model evaluation". I am curious why you brought up Reviewer CVwt again; I am almost certain that his/her review comments hold little reference value.

Response to follow-up Q3

• Thank you for raising this concern and giving suggestions to discuss Prot2Text[1] and ProteinGPT[2]. This allows us to clarify the unique advantages of STELLA.

• Prot2Text

  • Prot2Text employs a graph-based encoder and a GPT-2 decoder, designed to take protein structures as input and generate functional annotations as output. To ensure a fair comparison, we conducted experiments using the exact same dataset splits as Prot2Text for the FP task. Notably, this dataset split is highly rigorous, maintaining sequence identity < 40% between the training and testing sets, which we recognize as a significant contribution of Prot2Text. Additionally, we integrated Prot2Text’s encoder into our model architecture as part of an ablation study exploring different protein encoders and LLMs, as detailed in Section 5.3.2. Under these conditions and through extensive experiments, STELLA achieved SOTA performance, highlighting its clear advantages over Prot2Text in terms of accuracy and versatility.
  • While Prot2Text is a specialized model focused on static protein function prediction, it has inherent limitations in meeting the broader and dynamic needs of domain experts. In contrast, STELLA operates as a multimodal LLM capable of generating interactive functional annotations in a conversational format. This capability enables more dynamic and user-centric interactions. Moreover, STELLA goes beyond functional annotation by supporting additional tasks, such as enzyme prediction, which underscores its versatility and adaptability to diverse research needs.

• ProteinGPT

  • While ProteinGPT is also a multimodal LLM, STELLA distinguishes itself through differences in dataset construction and model evaluation:

  • Dataset construction:

    • ProteinGPT leverages abstract descriptions from the RCSB-PDB database as the text modality, while our OPI-Struc dataset is built using detailed annotations from the Swiss-Prot database. Swiss-Prot provides more comprehensive protein functional annotations curated and reviewed by domain experts, ensuring higher data quality and greater reliability for downstream tasks.

  • Model evaluation:

    • STELLA takes a more strict evaluation strategy with rigorous dataset split. ProteinGPT employs 7:3 split ratio for dividing the training and testing sets. In contrast, STELLA adopted a more stringent data splitting strategy, ensuring sequence similarity between training and test sets is below 40%. The dataset split strategy for STELLA better reflects real-world scenarios for annotating uncommon proteins.
    • STELLA takes protein-related specialized models to compare for more strict and responsible evaluation. Regarding the choice of baseline models for comparison, ProteinGPT selected general-purpose LLMs that are not specifically optimized for protein-related tasks. In our work, we compared STELLA against previous SOTA specialized models for both the function prediction (FP) and enzyme prediction (EP) tasks (e.g., Prot2Text_LARGE and CDConv), establishing a more strict and responsible evaluation.

• Overall, the evaluation strategy in our work ensures rigorous, reasonable and reliable assessment of STELLA’s performance, which is also recognized by Reviewer CVwt as one of our work’s strengths.

Dear Reviewer L7ds,

Thank you once again for your detailed and thoughtful feedback. Your comments have been instrumental in leading us to refine our work further. We also appreciate the opportunity to address your prior follow-up concerns.

---

Regarding Q1

• We sincerely apologize for any misunderstanding caused by the presentation that 'STELLA can obtain information that is not available in the PDB database' in our initial submission. To clarify, STELLA does not possess the capability to obtain information that is entirely unavailable in the PDB database. Instead, its strengths lie in leveraging structure-based representations to address two protein-related tasks as follows:
  • Functional description prediction. Using the structural data available in the AFDB, STELLA can help predict functions of proteins whose structures are known but whose biological functions remain unclear. This functionality is expected to be valuable for exploring unannotated proteins.
  • Enzyme-catalyzed reactions prediction. STELLA can utilize structural data to predict enzyme-catalyzed reactions, providing insights into potential catalytic activities based on structural features.

Regarding Q2

• Thank you for pointing out the need to clarify the role of 'iterative feedback' in our evaluation. Iterative feedback refers to the process that LLMs improve and adjust its responses based on previous outputs or additional user inputs. However, we acknowledge that quantifying this capability is currently challenging, as existing benchmarks for multimodal LLMs are not designed to assess dynamic and iterative feedback. Our quantitative evaluation of STELLA focused on static performance metrics, such as ROUGE-L and accuracy, which do not capture the iterative nature of this process. In our current quantitative evaluation of STELLA, we conducted only a single round of interaction for each task. We regret not adequately explaining these limitations in our original submission. Thank you again for your insightful review to help us identify areas for improvement.

Regarding Q3

• STELLA adopts a projection-based multimodal LLM architecture, inspired by widely recognized vision-language models like LLaVA[1]. The prior works such as Prot2Text[2] and ProteinGPT[3] also unitize a projection layer (e.g., MLP or Linear layers) to build the multimodal framework. These models combine embeddings from the biomolecular modality (e.g., sequences or structures, or their fusion) with the text modality, aiming to leverage the capabilities of LLMs to handle biomolecular tasks.

• The fundamental difference between STELLA, Prot2Text and ProteinGPT is the modality fusion strategy. Prot2Text and ProteinGPT integrate protein sequence and structure data using a late fusion strategy, where each modality is encoded separately before being fused. However, late fusion approaches have certain limitations, such as the potential loss of cross-modal relationships and increased complexity of encoder modules. In contrast, STELLA adopts an early fusion strategy inherited from the ESM3 encoder, where the sequence and structure modalities are jointly represented and fused into a unified representation at encoding stage. This early fusion strategy has the potential to both preserve the intrinsic relationships between modalities and improve computational efficiency.

[1] Liu, H., Li, C., Wu, Q. and Lee, Y.J., 2024. Visual instruction tuning. Advances in neural information processing systems, 36.
[2] Abdine, Hadi, et al. "Prot2Text: Multimodal protein’s function generation with gnns and transformers." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 10. 2024.
[3] Xiao, Yijia, et al. "ProteinGPT: Multimodal llm for protein property prediction and structure understanding." arXiv preprint arXiv:2408.11363 (2024).

Thank you for highlighting the important issues. We have revised the manuscript to present STELLA’s capabilities and limitations more precisely. Your feedback is greatly valued. We hope this revision effectively highlights STELLA’s unique features. Please let us know if further clarification is needed. Thanks a lot.

---

Best regards,
STELLA Team

Thank you for raising 3 insightful questions (weaknesses) for our work, which are addressed below.

Response to Q1&Q2:

• You are correct that MCQA is not commonly used in practical protein-related tasks. However, the purpose of designing this subtask is to serve as a supplementary evaluation method, providing a more comprehensive assessment of the model’s capabilities. The specialized and complex nature of biological function descriptions makes it challenging to fully capture the quality of LLM responses using conventional NLP metrics, such as BLEU, BERT and ROUGE scores. Recognizing these limitations, we intentionally included the MCQA subtask to objectively evaluate STELLA’s performance in a structured and quantifiable manner.
• For multiple-choice questions with four options, even random selection yields an accuracy rate of 25%, which reduces the performance differentiation between models. Furthermore, previous studies in multimodal LLMs revealed that LLMs may prefer to predict a certain choice among all given choices [1]. Based on above considerations, we introduce FP_{eval_MCQA_4X} as a complementary evaluation metric besides FP_{eval_MCQA_1X}.
• When calculating FP_{eval_MCQA_4X}, the model is deemed successful in solving a question only if it correctly predicts the answer across all permutations. This requirement is significantly more challenging than answering a single question correctly. For instance, if a model randomly selects one of the four options, the probability of correctly answering a single question is 25%, whereas the probability of answering all permutations correctly drops drastically to 25%^4 . This substantial gap has also been reported in [1]： switching the evaluation from 1X to 4X results in a performance drop of -4.7% to -34.1% across 21 models, underscoring the increased difficulty and the stricter evaluation standard.

  [1] MMBench: Is Your Multi-modal Model an All-around Player?

Response to Q3:

• Thank you for your interest in STELLA's ability to incorporate iterative feedback from domain experts. To further demonstrate this capability, we have provided two more examples below. In the example 1, STELLA responds to a domain expert's inquiry by comparing the similarities and differences in the functions of two proteins within the same biological process. This enables a deeper understanding of protein functionality. In the example 2, STELLA offers suggestions for biotechnological applications based on expert feedback, providing valid directions for further exploration.

Example 1

> User: What are the main functions of this protein?[AFDB_ID: AF-Q9W3K5-F1-model_v4.pdb]
> STELLA: The main functions of this protein are to catalyze the ATP-dependent ligation of L-glutamate and L-cysteine, and to a lesser extent, of L-histidine and L-tryptophan.
> User: Compare the function of glutamate-cysteine ligase with that of glutathione synthetase. What structural or mechanistic features distinguish these enzymes, and how do they together regulate glutathione levels?
> STELLA: Glutamate-cysteine ligase and glutathione synthetase are both involved in the synthesis of glutathione, a critical antioxidant in cells. However, they have different functions and mechanisms of action. Glutamate-cysteine ligase catalyzes the ligation of L-glutamate and L-cysteine to form a peptide bond, while glutathione synthetase catalyzes the condensation of L-glutamate and L-cysteine to form glutathione. The two enzymes have different substrate specificities and reaction conditions, and they work together to regulate glutathione levels in cells.

Example 2

> User: What are the main functions of this protein?
[AFDB_ID: AF-Q5KYR2-F1-model_v4.pdb]
> STELLA: The main functions of this protein are involved in the isomerization of 5-deoxy-glucuronate (5DG) to 5-dehydro-2-deoxy-D-gluconate (DKG or 2-deoxy-5-keto-D-gluconate).
> User: Describe potential industrial or medical applications of the isomerization catalyzed by this protein, specifically converting 5-deoxy-glucuronate (5DG) to 5-dehydro-2-deoxy-D-gluconate (DKG). How might this reaction be utilized in biotechnology or pharmaceutical production?
> STELLA: The isomerization reaction catalyzed by this protein has several potential applications in biotechnology and pharmaceutical production. For example, the conversion of 5DG to DKG could be used to produce novel compounds or to improve the efficiency of existing ones. Additionally, the reaction could be used to develop new biotechnological processes or to improve existing ones.
> User: Any further examples?
> STELLA: Yes, the isomerization reaction could also be used to produce novel antibiotics, anticancer agents, or other bioactive compounds. For example, the conversion of 5DG to DKG could be used to synthesize novel peptidoglycan structures, which are important for bacterial cell wall synthesis and could potentially be used to develop new antibiotics.

Thank you for raising 3 insightful questions for our work, which are addressed below.

Response to Q1

(1) Methods

• Thank you so much for suggesting Foldseek as a baseline for comparison. Taking Foldseek as a straightforward baseline for comparison includes the following steps:

Step 1. Structure retrieval: Using Foldseek, we search the structure databases to identify structurally similar proteins with structural similarity to the target protein.
Step 2. Function determination: Functions of the retrieved proteins are annotated by mapping them to their corresponding entries in the Swiss-Prot database.

• In our OPI-Struc dataset, the sequence similarity between the training and testing sets is below 40%, representing a challenging split that emulates real-world scenarios where models are required to generalize to predict functions for novel, unannotated proteins. In the first step, for the 4,203 protein structures in our test set, we used the Foldseek easy-search (https://github.com/steineggerlab/foldseek?tab=readme-ov-file#search) command with default parameters to search for similar protein structures within the training set for each test protein. For the e-value parameter, only matches with an e-value below 0.001 are considered and returned to the results. In the second step, the function prediction for proteins in the test set is assigned based on the functional annotation of the top 1 retrieved protein from the Swiss-Prot database. The median e-value of the top 1 retrieved proteins is 2.723e-20, indicating a high confidence in the retrieval results by Foldseek.

(2) Results

• Among the 4,203 protein structures in the test set, four structures did not have any similar matches in the training set identified by Foldseek. For the remaining 4,199 proteins, the ROUGE-L score achieved by Foldseek was 0.4586, which is significantly lower than the score of 0.5257 achieved by our model, STELLA-ESM3-Llama-3.1-8B-Instruct. For the four unfound proteins (Swiss-Prot/AFDB ID: P40532,O31983,P23487,Q15847), we double-checked using the Foldseek web interface (https://search.foldseek.com/search) and searched against the AFDB-SWISSPROT database, which contains detailed functional annotations. However, no similar structures were found for these proteins beyond the query structures themselves.
• Here, we provide a specific example. A protein from the test set (Swiss-Prot ID: P14721) and a protein from the training set (Swiss-Prot ID: P51109) share identical functional descriptions but exhibit globally dissimilar structures (TM-score: 0.48). Our model produced a completely correct prediction with a ROUGE-L score of 1, whereas Foldseek delivered an entirely incorrect result with a ROUGE-L score of 0.
• This highlights that our multimodal learning approach effectively captures more fine-grained relationships between protein structure and function, a capability that retrieval-based models inherently lack. The relationship between protein structure and function is highly complex, as demonstrated by a more detailed analysis of our dataset. Using Foldseek, we clustered the protein structures in the training set based on 90% structural similarity. Notably, only about 25% of proteins within the same cluster share identical functional annotations.

Response to Q2

• Thank you for your question. To ensure a fair and rigorous comparison with other enzyme classification models, we converted the EC numbers to their unique official names according to the BRENDA Enzyme Database (https://www.brenda-enzymes.org/). In other words, each label corresponds to a single unique name. A model output is considered correct only if it fully matches the reference official enzyme names.

Response to Q3

• Protein biology revolves around the interplay of three data modalities: sequence, structure, and function (text). Structural data plays a vital role in uncovering protein biological functions. Although extensive structural data including PDB and AFDB have been accumulated and publicly available, further efforts are needed to leverage these valuable resources for deeper understanding of protein biological functions. Fully realizing the potential of such plentiful structural data resource in protein understanding requires bridging the gap between structural data and the functional and biochemical insights essential for practical applications in both research and industry. To address this challenge, there is an urgent need to develop innovative approaches that integrate structural data with cutting-edge computational tools.
• By establishing structural and functional correlations by utilizing condensed knowledge in LLMs, STELLA fulfils accurate, versatile and interactive predictions across a range of protein-related tasks (see the examples 1 and 2 below, which have also been put in Appendix A.1 in the revised manuscript). STELLA achieves state-of-the-art performance across multiple tasks, signalling the transformative potential of multimodal LLMs as an innovative paradigm in advancing protein biology research besides the development of protein language models (pLMs). The effective utilization of LLM capabilities has been recognized by Reviewer tXk1, cosX and L7ds as a strength of this work.
• STELLA not only creates a new paradigm to understand proteins besides the development of pLMs, but also expands the boundaries of LLM capabilities in protein biology. This study will help drive the advancement of protein science and computational biology through LLM-based approaches, establishing a new paradigm beyond the PLM-based paradigms.

Example 1

> User: What are the main functions of this protein?[AFDB_ID: AF-Q9W3K5-F1-model_v4.pdb]
> STELLA: The main functions of this protein are to catalyze the ATP-dependent ligation of L-glutamate and L-cysteine, and to a lesser extent, of L-histidine and L-tryptophan.
> User: Compare the function of glutamate-cysteine ligase with that of glutathione synthetase. What structural or mechanistic features distinguish these enzymes, and how do they together regulate glutathione levels?
> STELLA: Glutamate-cysteine ligase and glutathione synthetase are both involved in the synthesis of glutathione, a critical antioxidant in cells. However, they have different functions and mechanisms of action. Glutamate-cysteine ligase catalyzes the ligation of L-glutamate and L-cysteine to form a peptide bond, while glutathione synthetase catalyzes the condensation of L-glutamate and L-cysteine to form glutathione. The two enzymes have different substrate specificities and reaction conditions, and they work together to regulate glutathione levels in cells.

Example 2

User: What are the main functions of this protein?
[AFDB_ID: AF-Q5KYR2-F1-model_v4.pdb]
> STELLA: The main functions of this protein are involved in the isomerization of 5-deoxy-glucuronate (5DG) to 5-dehydro-2-deoxy-D-gluconate (DKG or 2-deoxy-5-keto-D-gluconate).
> User: Describe potential industrial or medical applications of the isomerization catalyzed by this protein, specifically converting 5-deoxy-glucuronate (5DG) to 5-dehydro-2-deoxy-D-gluconate (DKG). How might this reaction be utilized in biotechnology or pharmaceutical production?
> STELLA: The isomerization reaction catalyzed by this protein has several potential applications in biotechnology and pharmaceutical production. For example, the conversion of 5DG to DKG could be used to produce novel compounds or to improve the efficiency of existing ones. Additionally, the reaction could be used to develop new biotechnological processes or to improve existing ones.
> User: Any further examples?
> STELLA: Yes, the isomerization reaction could also be used to produce novel antibiotics, anticancer agents, or other bioactive compounds. For example, the conversion of 5DG to DKG could be used to synthesize novel peptidoglycan structures, which are important for bacterial cell wall synthesis and could potentially be used to develop new antibiotics.