MMSite: A Multi-modal Framework for the Identification of Active Sites in Proteins

Dear Reviewer,

Thank you for your valuable feedback and constructive suggestions. Below is our point-by-point response to your comments.

Re - Weaknesses 1

It's true that MMSite rely on an external generative model to produce textual context for inference. This decouples the training of protein-to-text models from that of the active site prediction model. Training MMSite on top of the pretrained BLM and PLM models is very energy-efficient, so we could easily get a better active site prediction model once a better protein-to-text model is available.

Re - Weaknesses 2

We apologize for the confusion. In Section 4.1, we have described the implementation of each component of MMSite, and covered the training strategy for baseline methods in Lines 236-238.

Re - Weaknesses 3

When the split strategy is harsh, the baseline methods couldn't generalize, whereas MMSite can find meaningful patterns from both sequence and text and achieve good performance. We think their performances are closer at 50% threshold because (1) the dataset is easy enough for the baseline models to generalize, and (2) the performance is already very high, leaving little room for improvement. We will add this discussion to the revised paper.

Re - Questions 1

Thank you for your valuable suggestions. We will include a review of sequence-structure multimodal methods in the Related Work section, and emphasize the modalities of MMSite and compare it with similar sequence-text multimodal methods.

Re - Questions 2

While ProtST is a protein-text multimodal model, it aligns global sequence-level representations with textual descriptions, overlooking individual amino acid features, which reduces their effectiveness in residue-level tasks. In contrast, MMSite integrates text-empowered sequence representations directly at the token level, preserving the original features of amino acids. This ensures the model can benefit from the information provided by the text descriptions.

Re - Questions 3

In Lines 61-62, we have covered the inference strategy, but did not emphasize the source of embeddings, which may cause confusion. We will clarify both points in the future version. Thank you for your suggestion.

Re - Questions 4

The [CLS] token is used to aggregate the sequence-level representation of the entire input sentence, as first proposed in BERT. We will add this reference in the future version.

Re - Questions 5

We apologize for the confusion. Due to the character limit, we have provided detailed explanations about the dimensions in the Author Rebuttal. For Line 179, please refer to our response in the Re - Question 10 section.

Re - Questions 6

During the fusing step, we concatenate the text-enhanced sequence features with the original sequence features along the feature dimension. This combined feature is then fed into the prediction module to produce the final predictions. Detailed dimensions are provided in the Author Rebuttal. Additionally, for newly discovered proteins that lack text descriptions, we use an agent model to generate the necessary texts, which are then used alongside the sequences.

Re - Questions 7

Yes, we first extract amino acid-level embeddings from the baseline models, and then use a Transformer, followed by an MLP and a Sigmoid function to obtain active site probabilities. For the standard deviation, we trained each method 5 times with different random seeds.

Re - Questions 8

We have already included well-organized code, dataset and detailed instructions for replication in a zip file as Supplementary Material.

Re - Questions 9

We have tested the effect of removing manual prompting. The comparison is shown below:

The results show that manual prompting improves performance on most metrics. We believe this is because: (1) complete sentences provide richer context for the BERT-based BLM, (2) it reduces ambiguity in attribute meanings, and (3) it aligns better with BLM's pretraining, leveraging its knowledge more effectively.

Re - Questions 10

Traditional hard-label alignment assigns positive pairs a label of 1 and negative pairs a label of 0, pushing them apart in high-dimensional space. This isn't ideal for similar protein sequences $S_i$ and $S_j$, whose corresponding texts $T_i$ and $T_j$ are also similar. Thus, pushing $S_i$ away from $T_j$ may not be reasonable. To address this issue, we align the distribution $Q_i^{\rm s2t}$ towards $P_i^{\rm s2s}$ and $Q_i^{\rm t2s}$ towards $P_i^{\rm t2t}$ using KL divergence in Equation 8. This allows for a more nuanced alignment that respects the inherent similarities within the sequence and textual domains.

Re - Questions 11

Surface-based and sequence-structure methods require structural information, which can be costly to obtain. In contrast, functional information may be more accessible for de novo proteins.

Empirically, MMSite performing better at low clustering threshold demonstrates its generalization capabilities, but does not imply it is ineffective to de novo proteins. That said, we have not evaluated MMSite on de novo proteins, so we are not 100% sure.

Re - Questions Minor Comments

We will address all the issues you point out in the revised version to ensure they are correct and clear. Specifically:

• For Figure 2, please refer to our newly submitted PDF.
• For Figure 3, we use three colors to show the model's results: green for correctly predicted sites, blue for sites not predicted, and red for incorrectly predicted sites.
• For Figure 8, we will add more details the caption: "Each subfigure caption is the protein's Entry ID in the UniProt database. The colors mean the same as in Figure 3."

Thank you again for your time and consideration.

Best regards,

Authors

Dear Reviewer,

Thank you for your valuable feedback and constructive suggestions. Below is our point-by-point response to your comments.

Re - Weaknesses

To the best of our knowledge, ProTAD is currently the only dataset that simultaneously includes amino acid-level labels and detailed protein textual descriptions, making direct comparisons with other datasets challenging. Regarding the general protein-level tasks you mentioned, our MMSite framework incorporates comprehensive and rich textual descriptions of proteins as input. These descriptions inherently contain information about function and/or localization, making evaluations on these tasks somewhat unreasonable. In contrast, the annotation of active sites is not explicitly included in the input, which justifies the focus on active site prediction in our work. As for binding site prediction, it is not suitable to rely solely on protein sequence, as the binding sites often depend on specific interactions with various small molecules, which are not included in our model's input.

Re - Questions 1

Please refer to our response in the Re - Weaknesses section above.

Re - Questions 2

Thank you for mentioning an important related work! Here we compare our ProTAD dataset with ProtDescribe: Firstly, ProtDescribe includes 4 text attributes, whereas ProTAD comprises 17 text attributes, providing more comprehensive and detailed information. Secondly, ProTAD includes fine-grained amino acid-level labels, which are not present in ProtDescribe. These labels are crucial for the residue-level tasks such as active site identification from a multimodal perspective.

Re - Questions 3

ProtST-induced PLMs do indeed outperform vanilla PLMs in many downstream, sequence-level tasks, such as protein localization prediction and function annotation. This is because ProtST aligns the protein-level sequence embeddings with text during training, rather than token-level features, causing the representations to focus on the global function of the protein. We suspect that by doing so, the model loses sharpness in the individual characteristics of each amino acid, which reduces their effectiveness in residue-level tasks.

Re - Questions 4

In fact, we follow the standard practice of randomly sampling each batch from the training set. However, considering this biological principle, we use soft-label alignment to address the limitations of traditional hard-label alignment (such as CLIP [1]), where positive pairs are assigned a label of 1, and negative pairs are assigned a label of 0. The model then “pulls close” the positive pairs and “pushes away” the negative pairs in the high-dimensional space with equal emphasis to each pair. However, for similar protein sequences $S_a$ and $S_b$, their corresponding texts $T_a$ and $T_b$ may also be similar. Thus, pushing away $S_a$ and $T_b$ may not always be reasonable. To mitigate this issue, we employ soft-label alignment, which aligns the distribution $Q_i^{\rm s2t}$ towards $P_i^{\rm s2s}$ and $Q_i^{\rm t2s}$ towards $P_i^{\rm t2t}$ as described in Equation 8. This approach allows for a more nuanced alignment that respects the inherent similarities within the sequence and textual domains.

Re - Questions 5

Thank you for your valuable suggestion! During inference, if the protein has corresponding multi-attribute text descriptions, the process is the same as during training. However, if such descriptions are not available, we use the Prot2Text agent model to generate the text inputs. Below is a comparison of MMSite inference time (seconds per sample) between using pre-existing text descriptions from ProTAD and generated text with an agent model. For further context, we also compare against the BioMedGPT [2], another excellent protein-to-text model. The tests were conducted on a single GPU (NVIDIA GeForce RTX 4090) and a CPU (Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz).

We also compared the model performance using text generated by Prot2Text and BioMedGPT:

As the comparison shows, while using Prot2Text does increases the inference time of ProTAD, it provides comparable performance to BioMedGPT with a much smaller inference time cost. We will add this discussion to the revised paper.

Thank you again for your time and consideration.

Best regards,

Authors

Reference

[1] Radford A, Kim J W, Hallacy C, et al. Learning Transferable Visual Models From Natural Language Supervision. ICML, 2021.

[2] Zhang K, Yu J, Yan Z, et al. BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks. arXiv, 2023.

Dear Reviewer,

Thank you for your valuable feedback and constructive suggestions. Below is our point-by-point response to your comments.

Re - Weaknesses 1

Thanks for your concern regarding the inference efficiency of our model! During inference, if the protein has corresponding multi-attribute text descriptions, the process is the same as during training. However, if such descriptions are not available, we use the Prot2Text agent model to generate the text inputs. Below is a comparison of MMSite inference time (seconds per sample) between using pre-existing text descriptions from ProTAD and generated text with an agent model. For further context, we also compare against the BioMedGPT [1], another excellent protein-to-text model. The tests were conducted on a single GPU (NVIDIA GeForce RTX 4090) and a CPU (Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz).

Additionally, we also compared the model performance using text generated by Prot2Text and BioMedGPT:

As the comparison shows, while using Prot2Text does increases the inference time of ProTAD, it provides comparable performance to BioMedGPT with a much smaller inference time cost. We will add this discussion to the revised paper.

Re - Weaknesses 2

Thank you for your careful review and constructive suggestions! You are correct that MMSite should be classified as a Seq. & Text model, and we will revise the tables accordingly. We will address these issues in future versions. Thank you again for your valuable feedback.

Re - Weaknesses 3

Please refer to the responses in Re - Questions 1-3 sections.

Re - Questions 1

Thank you for your insightful question! While ProtST is also a protein-text multimodal model, it aligns global sequence-level features with textual descriptions during training, rather than focusing on token-level features. This approach overlooks individual amino acid characteristics, causing deviations that reduce their effectiveness in reflecting inherent properties, ultimately resulting in poorer performance. In contrast, MMSite integrates text-empowered sequence representations directly at the token level. By using the MACross module and skip concatenation, we preserve the original features of amino acids while enriching them with relevant textual context. This approach ensures that amino acid representations maintain their intrinsic properties and benefit from the additional information provided by the text descriptions.

Re - Questions 2

Thank you for your careful review of our paper! In the reference you mentioned (Prot2Text), the original authors adopted both cross-modal and uni-modal soft-label alignment. The uni-modal alignment was proposed to address the issue where, despite alignment between two modalities in a cross-modal setting, the feature distribution within the same modality could deviate. This deviation may cause non-corresponding samples from the same modality to be very close in the feature space, leading to incorrect retrieval. However, in our task, retrieval is not a concern, and incorporating an additional uni-modal alignment loss could lead to suboptimal performance. To evaluate this, we conducted a comparison experiment:

As shown in the table, adding uni-modal soft-label alignment resulted in decreased performance across all metrics. Therefore, we opted not to include this additional alignment in our final approach.

Re - Questions 3

Thank you for your insightful question! We built the ProTAD dataset based on UniProt, and it contains more than 570,000 pairs of protein sequences and textual descriptions. To ensure diversity and avoid redundancy, we used MMSeqs2 to cluster these sequences based on a sequence identity threshold. We then randomly selected one sequence from each cluster to construct the final dataset (as illustrated in Appendix A.2). Therefore, the size of the training set is determined by the number of clusters, which in turn depends on the clustering threshold.

Thank you again for your time and consideration.

Best regards,

Authors

Reference

[1] Zhang K, Yu J, Yan Z, et al. BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks. arXiv, 2023.

Dear Reviewer,

Thank you for your valuable feedback and constructive suggestions. Below is our point-by-point response to your comments.

Re - Weaknesses Evaluation 1

Regarding # of trainable params, the difference between MMSite and baselines is not very large, as the former has 9 trainable Transformer layers for sequence and the latter have 4. We are running experiments with 9 layers for baselines for fair comparison and will report results once they finish.

Regarding comparison with other baselines, we conducted experiments with SaProt, Random Forest (RF), Support Vector Machine (SVM), and DISCERN (a stat-based method designed for active site prediction). The results are shown below, with MMSite significantly outperforming them.

For DeepSurf, it is specifically designed for predicting binding sites with molecules, which is not applicable to our task. Besides, we have already compared methods that use structure (e.g. MIF and PST).

Re - Weaknesses Evaluation 2

We believe ProTAD is diverse enough, as it is derived from Swiss-Prot, referred to as "a reference resource of protein sequences and functional annotation that covers proteins from all branches of the tree of life" [1]. All Swiss-Prot data with both active site and function annotations are included without cherry-picking. The split strategy is strict (10% sequence similarity), which challenges the generalization ability.

Re - Weaknesses Evaluation 3

MMSite depends on the base model, but when the split strategy is harsh (e.g. 10% sequence similarity), the base models couldn't generalize, whereas MMSite can find meaningful patterns from both sequence and text to achieve good performance. At 50% threshold, their performances are close because (1) the dataset is easy enough for base models to generalize, and (2) the performance is already very high, leaving little room to improve.

Re - Weaknesses Methods 1

We agree that the novelty of MMSite mainly lies in the design of the framework and its application. However, we believe this does not undermine the significance of our work, which is among the first to integrate protein sequences and biomedical texts for fine-grained, residue-level prediction. As there is abundant text available in the biomedical domain, we anticipate our work will inspire more impactful research and lead to breakthroughs in important but data-scarce tasks.

Re - Weaknesses Methods 2

In Appendix C.1, we evaluated MMSite across multiple sequence lengths (from 128 to 1024), and the results show its robust and consistent performance.

Re - Weaknesses Claims

Thanks for your suggestions. We will revise our claim to clearly reflect how MMSite builds, and provide more context to help readers better understand the underlying biological rationale in the future version.

Re - Weaknesses Minors

We have updated Figure 1 in the submitted PDF, and we will correct the words and notations in the future version. Besides, "mutual tasks" refers to the tasks of generating texts from sequence and retrieving sequences from text. We have revised it in the PDF.

Re - Questions 1

To evaluate the performance of Prot2Text, we built a dataset of 1k proteins randomly sampled from ProTAD, as shown below. The performance on ProTAD-1k is good and generally consistent with the reported performance, indicating that Prot2Text can readily serve as an agent to generate textual descriptions for proteins.

Re - Questions 2

The table below compares the performance of MMSite using human-annotated and Prot2Text-generated texts for training. The performance of two settings was similar across most metrics.

Re - Questions 3

For missing values, we use "unknown" as the textual annotation. (Appendix A.1)

Re - Questions 4

Suppose proteins $i$ and $j$ have similar texts but different sequences. The core of the alignment loss is two KL terms, which pushes the distributions $Q_i^{\rm s2t}$ close to $P_i^{\rm s2s}$ and $Q_i^{\rm t2s}$ close to $P_i^{\rm t2t}$. This creates both attractive and repulsive forces between the sequence embedding for $i$ and the text embeddings for $j$, striking a balance between full alignment and full separation. Our soft-label alignment strategy ensures that the model can effectively learn the relationship between sequences and texts, even when the two modalities sometimes give conflicting signals.

Re - Questions 5

The ablation study in Table 3 have evaluated this scenario. It shows that adding MACross improves the model's performance across five metrics by about 1.7% on average. The main purpose of MACross is to highlight the importance of the Function attribute, because it contains richer information than others, which is essential for our task. Otherwise, the model may struggle to prioritize the important attributes, leading to suboptimal performance.

Re - Limitations

We discussed the limitations and impacts in Appendix D, and restricted the license to mitigate potential misuse in the code. We will discuss more scenarios in the future version.

Thank you again for your time and consideration.

Best regards,

Authors

Reference

[1] Coudert E, Gehant S, De Castro E, et al. Annotation of biologically relevant ligands in UniProtKB using ChEBI. Bioinformatics, 2023.

Dear Reviewer,

Thank you for your valuable feedback and constructive suggestions. Below is our point-by-point response to your comments.

Re - Weaknesses 1

We agree that using text descriptions to enhance protein representations may seem similar to existing works. However, our innovation lies in the integration of protein sequences and biomedical texts for the token-level prediction task of active site identification. Experiments show that our method outperforms existing multi-modal methods for sequence-level prediction tasks. As there are abundant text available in the biomedical domain, we anticipate our work will inspire more impactful research in this direction and lead to breakthroughs in important but data-scarce tasks.

Re - Weaknesses 2

Regarding the motivation of MACross, it is based on our judgement that the Function attribute is the most relevant attribute for predicting active sites and contains the richest information. Therefore, in our MACross module, we prioritize the Function attribute and integrate the remaining annotations through cross-attention mechanisms. This ensures the rich information from the Function attribute is complemented by other annotations.

The additional attributes, although seemingly insignificant, also contribute to improve active site predictions. For example, subcellular location provides context about protein accessibility and potential interactions, critical for identifying active sites. Similarly, organism information offers evolutionary context, highlighting conserved or mutated motifs. The table below shows the performance comparison between using only the Function attribute and using all attributes, demonstrating that incorporating all attributes leads to better performance.

Regarding how protein-level annotations help, we apologize for not having explained this clearly in the paper. In the rich study (i.e. abundant text) of the function of proteins, scientists have paied special attention to the identification, structural analysis, engineering of active sites. These valuable information are stored not in protein sequences but in biomedical texts. We expect BLMs to retrieve this information given protein-level annotations, providing context for active site prediction.

Re - Weaknesses 3

We appreciate your suggestion for a temporal-based evaluation. To address this, we conducted an additional experiment simulating the discovery of new proteins. Our ProTAD dataset includes data up to March 11, 2024. We collected data from UniProt recorded after this date, representing newly discovered proteins (115 samples with active site labels), to evaluate the model's performance. The results show that MMSite maintains high performance even on newly discovered proteins, as shown in the table below.

Re - Weaknesses 4

We apologize for the shortcomings in our writing. Here we provide a clearer explanation of our methodologies in the Methods Section:

• In Section 3.1, we formulate our task of predicting active sites in proteins using both protein sequences and multi-attribute textual descriptions.
• In Section 3.2, we detail our methodology through four main steps:
  • Attribute description reconstruction with prompt: We use manual prompting to reconstruct multi-attribute text descriptions into a format processable by BLMs.
  • Modality feature extraction: We extract features from both protein sequences and textual descriptions using a PLM and a BLM, respectively. In the MACross module, we separately process the Function attribute and integrate it with the remaining attributes using cross-attention.
  • Cross-modal soft-label alignment: Our two-stage training strategy, "First Align, Then Fuse", uses soft-label alignment to align features from two modalities. This method differs from the traditional contrastive approaches that "pull close" positive pairs and "push away" negative pairs, which may not be suitable when there is significant similarity between different sequences or texts in a batch.
  • Multi-modal fusion and active site identification: We then fuse the two modalities through fusion attention and skip concatenation to predict active sites. In addition, since newly discovered proteins may lack corresponding text descriptions during inference, we use an agent model to generate the necessary texts, which are then used alongside the sequences.

Thank you for the opportunity to present our writing structure, and we will reorganize the content of this section to enhance the clarity and readability of our work in future versions.

Re - Questions

While ProtST also uses text descriptions, it aligns global protein-level sequence representations with textual descriptions during its training stage, causing the representations to focus on the global function of the protein. This training approach overlooks individual amino acid features, which reduces their effectiveness in residue-level tasks. In contrast, MMSite integrates text-empowered sequence representations directly at the token level. By using the MACross module and skip concatenation, we preserve the original features of amino acids while enriching them with relevant textual context. This approach ensures that amino acid representations maintain their intrinsic properties and benefits from the added information provided by the text descriptions.

Thank you again for your time and consideration.

Best regards,

Authors

Thank you for addressing some of my concerns. I've updated my score to reflect these clarifications. However, I still have two main concerns:

Novelty: The work's originality compared to previous research still feels marginal.

Dataset motivation: The rationale for collecting such a large dataset with 17 attributes is not fully justified, especially given the experimental results. The new experiment shows that training on the function attribute alone achieves similar performance to using all 17 attributes. This raises questions about the necessity and efficiency of the full dataset.