ProteinAdapter: Adapting Pre-trained Large Protein Models for Efficient Protein Representation Learning

1. From the results across multiple tasks in Table 4, I did not observe a substantial improvement of ProteinAdapter over ESM-1b, which does not convince me of the effectiveness of the proposed approach.

2. The two core contributions of the paper are (1) multi-level complementarity (sequence-structure co-modeling) and (2) multi-level modeling, which are not substantial innovations to the field of protein engineering and have already been commonly used and mentioned in many previous works, including ESM-GearNet [1], SaProt-PDB [2], and PromptProtein [3]. I encourage the authors to clarify the essential differences with these efforts.

3. ESM3 is a state-of-the-art generative multimodal protein model, how does the performance of your proposed method compare with it on various tasks?

4. The paper focuses on two specific LPMs (ESM-1b and ESM-IF1). It would be beneficial to explore the performance of the method with other LPMs or ESM-2 with different parameter amounts to further demonstrate its generality.

5. From my point of view, the main contribution of this paper is a Mamba-based embedding merge module. Despite its effectiveness, is Manba necessary here? Is the merging of embeddings really working or is it a proposed merging architecture?

6. The performance of ProteinAdapter highly depends on the quality and suitability of the pre-trained LPMs. If the pre-trained models have biases or limitations in their learned representations, it may affect the performance of ProteinAdapter on certain tasks or protein datasets. Have the authors done more exploration in this issues?

[1] Zuobai Zhang, Minghao Xu, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Enhancing protein language models with structure-based encoder and pre-training. In International Conference on Learning Representations Machine Learning for Drug Discovery Workshop, 2023.

[2] Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure-aware vocabulary. In The Twelfth International Conference on Learning Representations, 2024.

[3] Wang Z, Zhang Q, Shuang-Wei H U, et al. Multi-level protein structure pre-training via prompt learning[C]//The Eleventh International Conference on Learning Representations. 2022.

• The experimental setup of the model contains significant omissions. According to 4.2, if the structure input is replaced with the output from layer of sequence embedding, the resulting data fails to demonstrate the effectiveness of the fusion module in integrating real structural and sequence embeddings, rendering the experiment invalid. It would be essential to align the data with a 3D dataset such as the Protein Data Bank (PDB), or generating pseudo structure using pre-trained structure prediction model as written in 4.2, and re-evaluate the benchmark for "ProteinAdapter (w/o structure)" to enhance credibility.
• If the original configuration, which includes both sequence and structural inputs, is maintained, the auxiliary tasks in the multi-task learning framework and the Fold Classification task in Table 2 are structure-related tasks, raising concerns about data leakage. This is evident in Table 5, where the performance gains of the two tasks are not on the same scale.
• The contributions and innovations of this work are not particularly substantial, as the patterns of integrating 3D and 1D data are not breakthroughing (as several 3D + 1D models compared within the text).
• This paper contains several writing issues, such as "bold" being misspelled as "blod" in all figure captions, and two arrows missing in the merge section of Figure 3. Additionally, the description of the dataset is quite vague, such as it does not clearly state the statistical information of the dataset and not fully listing its sources.

• The manuscript lacks sufficient detail regarding the baseline comparisons. Several key questions remain unanswered:

1. What method was used to obtain baseline figures? Were they cited from existing literature or reproduced through new experiments?
2. How were the models applied to downstream tasks? Was fine-tuning or linear projection employed?
3. Were the entire models fine-tuned, or did the authors experiment with fine-tuning only the final few layers?

• The ablation studies focus solely on GO and EC tasks. The rationale behind selecting these specific tasks is unclear. Are these purely sequence-based tasks or do they involve both sequence and structure? If they are sequence-only, can they be considered truly representative for comprehensive ablation studies?
• Despite the inclusion of some ablation studies, the specific contribution of the proposed ProteinAdapter to performance improvements remains ambiguous. Critical baseline comparisons are absent:

1. Fine-tuning the last few layers of ESM-IF1 for 3D and (3+1)D tasks. This baseline is crucial to differentiate between improvements due to the adapter versus those from ESM-IF1 representations alone.
2. Substituting Mamba with a cross-attention block integration. This would more effectively demonstrate Mamba's advantages over MLPs than the current ablation studies.

• The authors' claim of efficiency compared to ESM-1b at the end of section 4 is only valid if they fine-tuned the entire ESM-1b model. The comparison raises concerns about whether other parameter-efficient fine-tuning techniques or partial model fine-tuning were considered.
• The authors' use of the term "multi-task scenario" could be misleading. While they train models for individual tasks with an auxiliary secondary structure prediction loss, readers might misinterpret this as suggesting that a single fine-tuned model can perform multiple diverse tasks.
• For the sake of completeness, the experimental settings should be explicitly stated in the supplementary materials, rather than merely referencing another paper.

• The methodology feels like glueing together pLMs and inverse folding models like Lego blocks. I think there have been several works which have done similar things in the past without using Mamba (eg. https://icml-compbio.github.io/2023/papers/WCBICML2023_paper53.pdf). I think the overall idea is sensible, but I don't think this is a significant contribution in terms of methodology.

• There is no real discussion on new insights or limitations of the method. My main takeaway from the results seems to be that (a) yes, the method is performing empirically well, but (b) I don't really have any new insights here besides that they got the idea to work as well or better than what currently exists for protein property prediction.

In the introduction, the authors highlight two challenges. However, the explanation could benefit from more detail about how these issues manifest, along with supporting citations to strengthen the argument. Given the current prevalence of joint sequence and structure pretraining in large protein models, it would be helpful to clarify whether these challenges still persist in this context. In the related work section, the authors briefly outline the advantages of Mamba. However, there is no discussion in the introduction or methods sections regarding why Mamba was chosen and how it effectively supports the proposed work. Instead, the paper directly incorporates multiple Mamba modules without further explanation. The experimental design could be more transparent. In Tables 1-6, there is no clear explanation of how the baseline methods were implemented, nor is it specified whether they were fine-tuned on the same task datasets as the proposed method. This lack of detail makes it difficult to assess the fairness of the comparisons. In Table 1, the performance of ProteinAdapter is quite similar to that of ESM-1b, making it challenging to demonstrate a clear improvement. If the gains are not substantial, it might be more practical to use the features from large protein models directly, avoiding the complexity of extensive downstream task training. In Table 4, the authors state that the multi-task experiment involves adding an auxiliary task during training but testing only on the main task. However, multi-task experiments typically evaluate performance across all tasks to understand the overall impact of the auxiliary task. Additionally, since the results of the proposed method show limited improvement over ESM-1b, it is difficult to claim a significant differentiation of ProteinAdapter from LPMs. In Table 5’s (3+1)D experiment, the comparison with the SaProt method from Table 2 is omitted, but the reason for this exclusion is not clarified in the paper. In Table 6, the ablation study could be more comprehensive. The multi-scale prediction head should first be compared with a transformer in the same framework, followed by a comparison with an MLP to assess relative advantages. Additionally, comparisons with other representations like pooling would help demonstrate the effectiveness of the multi-scale approach. Some writing problems In the introduction section (lines 30-33), the background discussion lacks citations, which may make it challenging for general readers to understand the context and significance of the work. Mamba is mentioned multiple times without citation, which could be confusing for readers. The first citation appears in line 120 of the main text, so it might be helpful to introduce it earlier in the paper. In the experimental section, SaProt appears directly in the comparison tables, but there is no description of the method in the main text. Providing an introduction to SaProt would enhance clarity and context for the comparisons.