Improving Multimodal Protein Function Prediction Using Bidirectional Interaction and Dynamic Selection Mechanisms

Thank you very much for your comments. We respond to your concerns as follows. The modified parts in the revised paper are highlighted in blue.

W1. Thanks for your suggestions. In this work, we did not use the BiMamba Block as a component of the Encoder to model the protein sequence, but used the BiMamba Block as a component of the Encoder to model the spatial features of the protein (PPI, subcellular localization domain). To further verify the effectiveness of BiMamba Block in Encoder, we have added an experiment in Table 6 of Appendix Section 6.2. In this experiment, the pre-trained framework is the same as stage 1 in the paper. In pre-trained encoder, BiMamba Block can be replaced by Multihead Attention Block, and vice versa. Similarly, we take the pre-trained encoder and use it for feature extraction for the fine-tuning task. In fine tuning, we use only one multi-layer perceptron for classification. The experimental results can be seen as follows:

As shown in experimental results, we find that the approach using BiMamba Block as a component for modeling spatial features shows significant advantages in MFO and CCO. When modeling sequence features, the method utilizing Multihead Attention as a component demonstrates considerable advantages in BPO and CCO. Furthermore, since the two types of features exhibit complementarity in different aspects, we choose to use BiMamba Block for modeling spatial features and MultiheadAttention for modeling sequence features.

As for the BOW encoding, in this paper, we do not propose a new coding approach, but classify domain and subcellular location and PPI as spatial information.

W2. Thanks to reviewer for the critical observations of our benchmark settings. We acknowledge that protein language modeling has indeed become a powerful tool for protein function prediction, and *many protein function prediction methods also use protein language modeling as feature extractors, such as GAT-GO[2] using protein structural characteristics and SPROF-GO[4] based on protein sequence. However, we note that SPROF-GO, GAT-GO and other papers do not compare with methods using protein language models. At the same time, our focus is on the interaction of protein features and the effective capture of relevant features (BInM and DSM), so this aspect was not considered in the original paper.

In addition, we pay high attention to the comments of reviewer, and show through the above supplementary experiments (i.e., rows 6 and 7 of Table 2 in Section 4: Ablation Studies) that sequence features extracted from protein language models contribute to improving model performance, but are not a decisive factor. Because in the feature ablation experiments, we found that models with interaction between spatial and sequence features clearly outperform models that only use sequence features extracted by large models.

Q1. We sincerely thank the reviewers for their valuable questions. Our proposed model includes two key components: the BInM and DSM modules. We believe both modules contribute significantly to the performance improvements of the model. This can be observed in Table 2 of the Ablation Studies, where removing either the BInM or DSM module results in a substantial performance drop. Additionally, our pretraining phase also plays a critical role in enhancing the model's performance, as detailed in Appendix Section 6.5.

As for the unfair comparison. We also appreciate the reviewers’ concerns regarding this issue. As clarified in the response to Weakness 2, we would like to address the following points: 1) Protein language models used as a feature extraction pipeline do not exhibit overwhelming superiority in performance when compared with spatial features. Therefore, we have not intentionally biased the benchmark comparisons. 2) Employing protein language models for characterizing protein sequences is a common practice. The approaches like SPROF-GO[4] and GAT-GO[2] also utilize protein language models without necessarily comparing them against methods that do not use these models.

We sincerely thank the reviewer for the detailed comments. The suggestions of reviewer 2RUc make our work more perfect and help us a lot. Thank you!

W1. We use an encoder with a BiMamba to process spatial structure information (i.e. protein-protein interaction network, subcellular location, and protein domains). The advantages of Mamba in processing spatial structure information are mainly reflected in its strong global modeling ability and linear complexity. Mamba, through its state-space model, can realize the perception and modeling of global information, so as to perform well in processing spatial structure information. Compared to the Transformer model, Mamba has lower computational complexity when dealing with long sequences. Mamba also performs well in processing multi-modal information. For example, in FusionMamba [1], the Mamba model is used in multi-modal image fusion tasks, which proves the effectiveness of Mamba in processing complex spatial structure information. Furthermore, in order to capture complex protein features more comprehensively in the model, we adopted a bidirectional scanning mechanism [2]. The BiMamba blocks we designed traverse features in both forward and reverse directions, enabling more accurate modeling of dependencies between proteins and between protein structures.

About BOW, this is a way of encoding protein structural features. We use BOW to encode protein subcellular localization and protein domain information, and the resulting features retain protein structural information. Just like when processing video signals, we need to convert video signals to digital signals through video encoding, the converted signal still has the main characteristics of the original video. Similarly, protein structure information is encoded so that the information is transformed into features that the model can learn, without affecting the main properties of the information.

In addition, the BiMamba Block is not the core innovation of this work, even if replacing it with another block such as the Multihead Attention Block does not affect our main model structure. We further demonstrated the effectiveness of BiMamba Block in the Protein Spatial Structure Information (PSSI) encoder by performing ablation experiments on key components of the encoder in Appendix 6.2. According to the table below, we can see that BiMamba Block is significantly more efficient in processing spatial structure information, costing much less time (-5.122ms) to predict a protein's function compared to Multihead Attention. We also find that the encoder using BiMamba Block as a component for modeling spatial structure features shows significant advantages in MFO and CCO. Considering both inference efficiency and performance, we decided to use an encoder with BiMamba Block as the component for spatial structure features modeling.

[1] Xie, Xinyu, et al. "Fusionmamba: Dynamic feature enhancement for multimodal image fusion with mamba", IEEE Conference on Computer Vision and Pattern Recognition, 2024.

[2]Zhu, Lianghui, et al. "Vision mamba: Efficient visual representation learning with bidirectional state space model", arXiv preprint arXiv:2401.09417, 2024.

W2. Thank you for your valuable reply. According to the reviewer's suggestion, we added more multimodal methods as comparison methods in Table 1 of the revised paper. For example, the Graph2GO [3] method uses the same multimodal features as in this work (i.e. protein-protein interaction network, subcellular location, and protein domains). The experimental results show that our proposed BDGO model achieves superior performance, proving that multi-modal features play an important role in protein function prediction tasks. Furthermore, our BDGO model is superior to other multimodal methods, which proves the effectiveness of BDGO using bidirectional interaction and dynamic selection mechanisms.

The experimental results are shown as follows:

[3] Kunjie Fan, et al. “Graph2go: a multi-modal attributed network embedding method for inferring protein functions”, GigaScience, 9(8):giaa081, 2020.

Q2 -> 1). Thank you for allowing us to clarify this issue. According to rows 6 and 7 of Table 2 in the Ablation Studies Section, we can see that, in our method, the performance of the sub-model using only sequence features is inferior to that of the sub-model using only spatial structure features. The experimental results are as follows. Here, w/o SP-F refers to removing spatial structure features from the input, and w/o SE-F indicates removing sequence features.

In order to make the comparison experiment more comprehensive, we have added the protein function prediction method (i.e., PredGO [4]) using the protein language model (PLM) as the comparison method in Appendix 6.6. According to the experimental results, it is evident that the BDGO model presented in this work outperforms other PLM-based methods. The experimental results are as follows:

[4] Zheng, Rongtao, Zhijian Huang, and Lei Deng. "Large-scale predicting protein functions through heterogeneous feature fusion", Briefings in Bioinformatics, 24(4): bbad243, 2023.

Thank you very much for your review. We respond to your concerns as follows. The modified parts in the revised paper are highlighted in blue.

W1/Q1. We attach great importance to the comments of reviewers, reintroducing structure-based protein function prediction methods in Section Introduction, such as DeepFRI and SaProt. At the same time, the important contribution of DeepFRI and other structure-based methods in this field is discussed. In addition, we believe that PPI data can provide information about protein interactions, and there are many papers on protein function prediction based on PPI networks, such as GeneMANIA[1], Mashup [2], DeepNF[3], NetQuilt[4], DeepGO[5] and etc. These methods do not compare their results against structure-based approaches, as they focus solely on PPI data. Similarly, our method also incorporates PPI data and adheres to the same principle by not comparing against structure-based methods[6].

W2/Q2. Thanks to the reviewer's comments, we are also curious about whether the model can make reliable predictions on unverified proteins. So we downloaded 272 unverified human protein data from the uniprot database. Among them, we removed proteins without PPI data. Finally, 136 proteins of BPO, MFO and CCO were obtained. The results of the test using our model are shown below. The experiment is supplemented in Appendix Table 8 of Section 6.4. At this time, this model has the highest results only in accuracy. This is because most of the 136 protein labels are not in our set 45 (BPO), 38 (MFO) and 35 (CCO), so they are mostly labeled 0. The results can be seen as follows:

W3/Q3. Thank you for giving us the opportunity to clarify the setting of ProtT5. We did not pre-train ProtT5. In our model, the ProtT5 model is frozen as part of our sequence encoder, and we pre-train the sequence encoder.

We are greatly encouraged by your insightful comments, e.g. an innovative method, valuable tool, and extensive experiment. In response to your concerns, we offer the following explanation. The modified parts in the revised paper are highlighted in blue.

W1. Thanks for the reminder of the reviewer, we have added the specific settings of parameters in Implementation Details in Section 3.1 of the revised paper.

W2. Thanks to the reviewers' suggestions, we will continue to explore protein function prediction methods in other species, such as mice, in future studies.

Q1. According to the author's suggestion, we have added the discussion of parameter setting in Implementation Details of Section 3.1. During fine-tuning, different hyperparameters are used for each aspect, with learning rates set to 3.6e-4, 8.6e-2, and 1.4e-4 for BPO, MFO, and CCO, respectively. The same pre-trained model serves as the feature extractor, and AdamW is used as the optimizer across all aspects. For BInM module, the cross-attention mechanism is configured with 8 heads, while other settings follow the default parameters in torch.nn. In DSM module, the temperature parameter $\tau$ is set to 1.

Q2. Thanks for the comments. The pre-training method can effectively enhance the expression of protein features, which is more prominent when combined with multi-modal information. Our study uses an encoder-decoder architecture that combines PPI, subcellular localization, protein domain, and sequence information to improve feature representation through the injection of multimodal knowledge, thereby alleviating overfitting problems in low-sample domains. In addition, large protein models (such as ESM, ProtTrans, etc.), which are also utilized in our study, extract deep features from massive sequences through self-supervised learning. These models perform better in data scarcity scenarios than traditional feature designs.

Q3. Thanks to the reviewer for inspiring us. When working with very large protein function data sets, there may be potential benefits to adding some extra layers to the DSM. As far as we know, the directions mentioned by the reviewers are highly relevant to some research work in recent years in multiple expert fields [1, 2, 3]. Therefore, dynamic allocation of expert modules is an effective strategy worth exploring. We believe that further improving the expert module on the basis of the existing DSM model may be a research direction worth exploring. In the future, we will actively attempt to implement these strategies on larger data sets and promptly begin related research to further validate their effects. Once again, we sincerely thank the reviewer for their innovative and valuable suggestions.

[1] Xue, Zihui, et al. "Dynamic multimodal fusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[2] Zhang, Qingyang, et al. "Provable dynamic fusion for low-quality multimodal data." International conference on machine learning. PMLR, 2023.

[3] Han, Xing, et al. "Fusemoe: Mixture-of-experts transformers for fleximodal fusion." arXiv preprint arXiv:2402.03226 (2024).

Q1. Thanks for the comments. In this work, we designed several modules for multimodal feature interaction and feature selection to improve the protein function prediction ability of the model.

As for BiMamba Block: Inspired by the bidirectional scanning mechanism proposed in the Vision Mamba [1] method from the field of computer vision, our proposed BiMamba Block innovatively applies the bidirectional scanning mechanism to the learning of spatial features of proteins. The modified parts in BiMamba Block of Section 2.1.1 are highlighted in blue.

As for Bidirectional Interaction Module (BInM): We are the first to propose a BInM for multimodal protein feature interactions. Multimodal BInM enhances the understanding and learning ability of complex patterns by combining spatial information and sequence information of different modes. BInM consists of two-branch cross-attention blocks, and mining the interrelation of multimodal data through two-way information interaction. The modifications made to BInM in Section 2.2.1 are indicated in blue.

As for Dynamic Selection Module (DSM): Where our DSM differs from traditional MoE is that our experts only need to acquire one feature, whereas traditional MoE uses a combination of features. This is because the input features of our DSM are multi-channel features that have been interactively integrated. In order to ensure the best interaction effect, we only learn one channel feature for each expert, and select the best expert through DSM. According to experimental results, we found that the effect of experts who selected only one feature was better than that of experts who selected multiple features. The experimental results are supplemented in Appendix Table 5 in Section 6.1.2. Additionally, we have made modifications to the description of DSM in Section 2.2.2 to provide further clarity on its design and functionality.

Q2.

1) To demonstrate that we capture the complementarity between sequence and spatial features, we supplement a feature ablation experiment in which we retain only spatial or sequence features. It can be found that features cannot interact when only spatial or sequential features are used, and the model performs poorly. The complementarity of the characteristics of various modes is proved. The experimental results are shown in row 6 and 7 of Table 2.

As for the artificial case mentioned by reviewer, we have compared our method (BDGO) with one that relies solely on information fusion (MSL-Branch) in ablation experiments. With the addition of MIL-Branch for feature information interaction, BDGO gets an overall performance boost. The experimental results are shown in row 1, 2, and 3 of Table 2. In addition, we have restated this comparison in ABLATION STUDIES for further clarification.

2) Thank the reviewers for their comments. We designed an ablation experiment in which we changed the feature interactions in the BInM module. As shown in Table 3 in Appendix Section 6.1.1, BDGO-BInM-0 and BDGO-BInM-1 represent removing the bottom and top cross-attention module from BInM. BDGO-BInM-0 is calculated as : $F_c^{(1)} = softmax(\mathcal{Q}^{(1)}(F_b^{(1)}) \otimes \mathcal{K}^{(2)}(F_b^{(2)})^T) \otimes \mathcal{V}^{(2)}(F_b^{(2)})$, $F_c^{(2)} = F_b^{(2)}$. BDGO-BInM-1 is calculated as : $F_c^{(1)} = F_b^{(2)}$, $F_c^{(2)} = softmax(\mathcal{Q}^{(2)}(F_b^{(2)}) \otimes \mathcal{K}^{(1)}(F_b^{(1)})^T) \otimes \mathcal{V}^{(1)}(F_b^{(1)})$. As can be seen in Table 3, the performance of BDGO decreased after changing the interaction mechanisms. The advantages of our BInM feature-interaction design are proved.

3) The experts in our DSM only need to acquire one feature, and instead adopt a combination of features. We supplement an experiment in the DSM in which experts select multiple features, in which we find that experts who select multiple features perform less well than experts who select only a single feature. The experimental results are shown in Table 5 in Appendix Section 6.1.2.

Q3. Thanks to the reviewer's suggestions, we have made corresponding modifications in the article.

a) In Section INTRODUCTION, we have added the abbreviations for "Gene Ontology" (GO), "Biological Process Ontology" (BPO), "Molecular Function Ontology" (MFO), and "Cellular Component Ontology" (CCO).

b) In Appendix 6.8 CRITICAL DIFFERENCE DIAGRAMS FOR STATISTICAL COMPARISON, we have added the critical difference diagrams for the comparison with other methods and the ablation experiments. Through critical difference diagrams, we can see that BDGO is significantly better than other methods, which proves the superiority of our method design.

[1] Zhu, Lianghui, et al. "Vision mamba: Efficient visual representation learning with bidirectional state space model." arXiv preprint arXiv:2401.09417 (2024).

We sincerely thank the reviewer for the comments and suggestions to our manuscript. We have carefully studied these comments and revised our manuscript based on them. Please find our responses to the comments below. The modified parts in the revised paper are highlighted in blue.

W1. Thank you for giving us the opportunity to clarify the purpose and relationship of each component.

Our proposed method efficiently captures multimodal information of proteins through a two-step training strategy . In the pre-training stage, we use the encoder-decoder model to learn and inject multimodal knowledge. For spatial features including PPI, subcellular location, and protein domains, a spatial encoder-decoder model using the BiMamba blocks is introduced in this stage. To mine sequence features including protein sequences, we design a sequence encoder-decoder model based on the Transformer blocks for pre-training. Then, during our BDGO model training phase, we integrate and learn features from multimodal information. The proposed model is primarily divided into two major branches: one is the multimodal shared learning branch, and the other is the multimodal interactive learning branch. Protein data are processed through these multiple branches to generate several sets of features, which serve as inputs for our well-designed hard gating network. Finally, the model dynamically selects the optimal features for each protein to enhance performance in protein function prediction.

Following your suggestions, we have added detailed explanations in Section METHODOLOGY and revised the descriptions in Section 2.2 to enhance clarity and coherence.

W2. Thank you for your comments. Regarding formal definitions, we have thoroughly explained and defined each component in detail in the METHODOLOGY section. As for efficacy measurement, we have analyzed the effect of each component in the Ablation Studies section. The corresponding experimental results are presented in Table 2.

W3. Thanks for the suggestions. In order to increase the statistical significance of the experimental results, we conducted five tests for each model and calculated the variation range of the test results. The supplementary experimental results of BDGO have been added to Table 1, which can be seen as follows: