MULAN: Multimodal Protein Language Model for Sequence and Structure Encoding

Although they did an ablation study showing that further pre-training ESM-2 on Swiss-Prot resulted in worse performance than incorporating MULAN, I would suspect whether it works for structure-based PLM, such as SaProt. MULAN didn't get outstanding performance compared to other structure-based Models, and the features it used were very simple, even less than Foldseek which uses both torsion angles and spatial distances. Could the authors try pre-training Saprot without MULAN on their training set and see how the downstream performance would be?

We did our ablation studies based on the sequence-only model, particularly ESM2 for two reasons:

• ESM2 has a small computationally efficient model, and one can experiment a lot with this base model;
• MULAN is effective for sequence-only PLMs.

Following your suggestion, we additionally fine-tuned the SaProt 35M model on our dataset without the Structure Adapter. We got a similar performance with and without MULAN, our reason is that SaProt already uses the structure (Foldseek instead of torsion angles). Still, the main focus of MULAN is enhancing sequence-only models with structural information, as we already highlighted in the paper (lines 266-268, “Furthermore, we show that MULAN-ESM generally offers higher performance gains compared to MULAN-SaProt. We suppose that it is because SaProt is already a structure-aware model, while ESM2 is sequence-only.”).

Paper writing. In part 2.1 of MULAN Architecture, the authors introduced the structure features they used. However, I think they should include some basic introductions about what are torsion angles ϕ and ψ. Only including the name of angles is confusing and may result in misunderstanding about used features.

Thank you for suggesting the revision, we have extended Section 2.1 with additional details.

Dear Reviewer EUj2,

Thank you for your suggestions and questions, we address them below.

Motivation. The authors mentioned that existing sequence-structure models tend to be large that may hinders their applicability in downstream tasks. Although MULAN only introduces a lightweight structural module, it still may not be easy to use depend on the size of backbone PLM, e.g. ESM-2 650M or 3B. The proposed method seems not to be advantageous from the prospective of the motivation.

Our motivation is to have a model that

• requires minimal changes to the base architecture and can directly leverage pre-trained PLM checkpoints;
• makes a lightweight add-on -- the Structure Adapter -- to enable structural inputs, which leads to the same inference time and memory requirements;
• works on par with existing structure-aware PLMs that are larger and slower or require training from scratch.

To strengthen our point we have measured the inference time and GPU memory requirements of large PLMs needed for the same protein (see Appendix E and Table 11). In short, MULAN offers the best trade-off between quality and efficiency:

• Having the same computational costs as S-PLM, MULAN gives significantly better downstream results;
• MULAN requires 2 times less time and memory compared to PST, having similar downstream quality;
• MULAN requires only finetuning without training from scratch as in SaProt, ProstT5, and ESM3, having similar downstream quality.

Lack of novelty. There are many related works regarding protein sequence and structure co-modeling, as listed by the authors. The way of adding structural bias into per-residue embeddings has been introduced many times, such as ESM-3 and DeProt. The structural features are only torsion angles which are commonly used such as Foldseek. Also, the pre-training data derives from the Swiss-Prot database, which has been widely used and I suspect its diversity for both sequence and structure modality.

The novelty of our work is in line with our motivation written above. We show that MULAN has simple architecture and is effective both in terms of downstream quality and computational resources needed to run the model.

MULAN didn't get outstanding performance compared to other structure-based Models, and the features it used were very simple, even less than Foldseek which uses both torsion angles and spatial distances.

Indeed, we use very basic structural features, namely, the torsion angles. However, our investigation shows that torsion angles are enough to restore the protein structure with a small error, while Foldseek is not enough to decode the structure. Thus, torsion angles can be more expressive than Foldseek in our setup.

Ablation experiments with the use of Foldseek inputs, Foldseek inputs together with torsion angles, and residue-residue distance matrices as a prediction objective were outlined in our paper, Appendix C.1 and Table 7. These experiments have shown no clear benefits so we decided to keep the architecture simple yet effective. Also, we have done additional ablation experiment with passing 3D coordinates of residue C-alpha atoms as an input to MULAN. These coordinates are encoded similarly to torsion angle vectors with another Structure Adapter, that projects 3 coordinates to an embedding for the model. We made a sum of ESM, angle, and coordinate embeddings. We added this experiment and its results to Appendix C.1. The results show no consistent improvement over the base MULAN version (see Table 7). Overall, residue torsion angles are expressive enough to encode the protein structure.

Have you considered that some of the downstream task proteins are present in the set that you use to Swiss-Prot set to fine-tune for MULAN? If there is overlap I think the fraction of proteins in each task present in the training set should be reported (could be in Table 5). It is also worth reporting if any of the CASP12, TS115, or CB513 structures are present in the training set. It could be that they are mutually exclusive, but this should be made explicit none the less.

The finetuning of MULAN is done in an unsupervised manner: we do not use any of the labels from the downstream tasks. Thus, there is no data leakage when using the same proteins during the finetuning of MULAN and further evaluation of the downstream task performance.

In general it is a common practice to use all available data for unsupervised training: modern LLMs are pre-trained on the whole internet, and modern PLMs are pre-trained on all available protein sequences. For example, ESM2 is trained on 250M protein sequences from the UniParc database, which contains proteins from the downstream datasets.

Regarding the secondary structure prediction, the above answer is still valid. There is no data leakage in the secondary structure labels, because MULAN takes angle vectors explicitly as inputs.

Dear Reviewer tgHX,

Thank you for the detailed review. We address your questions below.

If the ultimate utility is in quicker training and inference, the authors should provide quantitative comparison and better motivate why this use-case is significant.

We added the measurements of the amount of GPU memory and time required for the inference of different PLMs on the same protein (see Appendix E and Table 11). In short, MULAN offers the best trade-off between quality and efficiency.

• Having the same computational costs as S-PLM, MULAN gives significantly better downstream results.
• MULAN requires 2 times less time and memory compared to PST, having similar downstream quality.
• MULAN requires only finetuning without training from scratch as in SaProt, ProstT5, and ESM3, having similar downstream quality.

The authors mention ProstT5 and ESM3 models in Section 6.2, which both include structural information in training. I think it is necessary to compare MULAN with these models to evaluate how well their approach compares to these models. That way users can compare the performance and cost tradeoffs between the two approaches.

Comparisons with ProstT5 and ESM3 were presented in our paper in Table 2. There is also a comparison to other relevant structure-aware PLMs: S-PLM, PST, and SaProt (Table 2). Moreover, there are results measuring the structural awareness of these models in Appendix D (Table 10).

What fraction of residues are unmasked once the pLDDT threshold is applied?

In our paper, residues are not unmasked according to the pLDDT threshold. We have the following algorithm for masking:

• Mask 15% of residues for further MLM decoding. This is done together for sequence and the corresponding angle inputs;
• Additionally, mask only residue angle vectors that have low pLDDT, while keeping the corresponding sequence letters unmasked.

As a result, we still have 15% of sequences masked and used in the training MLM objective. And we have slightly more structure inputs masked. In the dataset there are 15.4% of the residues with pLDDT lower than 70, which results in 28.1% of structure inputs masked on average during training and 15.4% during inference (when MLM is not used).

Why is training limited to the Swiss-Prot fraction of Alpha Fold DB? There are millions of predicted structures available, which might boost performance of MULAN?

It is enough to have a subset of the Swiss-Prot dataset for MULAN model. However, we experimented with larger datasets. For example, we took the dataset with 17M protein structures that were used in the ProstT5 paper and obtained lower scores compared to training on Swiss-Prot. We concluded that the Swiss-Prot subset contains protein structures of higher quality. Additionally, MULAN requires only a short finetuning stage, so there is no need for the large training dataset.

Can you explain some more about the tasks chosen for evaluation? Why these tasks?

Most of the datasets in our paper were taken from the SaProt evaluation setup. Also, we extended the downstream tasks with 2 tasks: Fluorescence prediction task, which dataset contains mutants of green fluorescent protein. The Fluorescence dataset can be used to measure sensitivity to the mutations of the protein; Secondary structure prediction task to check the structural awareness of MULAN.

Why does MULAN perform worse with more parameters (Table 1)? [I found the presentation of this table with bold vs. not bold to be confusing.]

Thank you, we have clarified the table by adding minus signs explicitly. Concerning the scale of our model, MULAN performs better with more parameters. In the paper, we reported the performance gain to the base model X, when MULAN is added to the X (row MULAN-X). Positive numbers (bold in Table 1) indicate that adding MULAN to X is beneficial, while negative numbers (not bold in Table 1) show that it is not. Table 1 shows that MULAN improves over base models X in most downstream tasks. So, larger MULAN performs better than medium MULAN, which in turn performs better than small MULAN (Table 2). However, the effect of adding MULAN is higher for small models than for large ones.

Dear Reviewer tgHX,

As the end of the discussion period is approaching, we would greatly appreciate your feedback on our rebuttals. Please let us know if you have some additional questions we can address to improve our paper and your score.

Best regards

Dear Reviewer 5GUe,

We appreciate your suggestions and address your questions below.

The paper does not include comparisons with the latest structure-aware protein language models, such as ESM-3, which incorporate advanced structural representations. Comparing MULAN with these models would provide a clearer perspective on its relative advantages and limitations within the current landscape of structure-aware PLMs.

Thank you, we would like to point out that all necessary comparisons were done in our paper. ESM3 results, as well as S-PLM, PST, SaProt, and ProstT5 results, are shown in Table 2. There are also results measuring the structural awareness of these models in Appendix D (Table 10).

Have the authors considered adding other structural features, like inter-residue distances, atomic coordinates, or functional information? Adding these could potentially improve the model’s structural understanding and task performance.

Yes, we have considered and used the following structural features.

1. Foldseek sequences were used both together with the angle information and separately.
2. Inter-residue distances were used as a prediction objective (Contact Head), where we predict binarized distances between residues.
3. Masked angle restoration objective (Angle Head) is used as an additional loss function.
4. XYZ Coordinates of C-alpha atoms were also added as another structure input using the Structure Adapter (this new experiment is added in Appendix C.1).

However, there is no significant improvement from these experiments compared to using only torsion angles, so we decided to keep the simple architecture. These experiments are presented in Appendix C.1 and Table 7.

How were structural features combined with ESM’s sequence embeddings? Were any experiments conducted to compare these methods?

The structural embeddings produced by the Structure Adapter are summed up to the sequence embeddings produced by the ESM embedding layer. This embedding sum is then passed to the ESM2 Transformer layers. We detail it in Section 2.1.

Has the model been tested for multi-task learning, handling tasks like stability prediction, function annotation, and interaction prediction simultaneously? Given the modular Structure Adapter, it would be interesting to see if MULAN can perform multiple tasks at once and how it compares to single-task models.

MULAN, as most PLMs, eg. ESM2 and SaProt, is trained in an unsupervised manner with a masked language modeling objective. Downstream task labels are not used as a training objective. Protein embeddings from frozen PLM can be extracted after the unsupervised pre-training is done. Then a small downstream model can be trained for a particular task. Thus, multitask learning is generally not the purpose of training PLMs and is out of the scope of our paper.

Dear Reviewer 3ooa,

Thank you for the valuable feedback and for pointing out our strengths! We will address your questions below.

More analysis should be conducted to analysis the advantages of employing the transformer model for encoding structure information rather than geometric-aware graph neural network.

We focus on Transformer-based models, because they are currently state-of-the-art model in a lot of tasks, while Graph DL models are worse at scale than Transformer models due to the over-smoothing problem of Graph Neural Networks [1].

[1] Chen, D., Lin, Y., Li, W., Li, P., Zhou, J., & Sun, X. (2020, April). Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 04, pp. 3438-3445).

The expression of line 480 is a little confusing.

Thank you for noticing, we have checked and rewritten this sentence (line 485).

In current framework of MULAN, the PLM models are full finetuned. What about applying PEFT (Parameter-Efficient Fine-Tuning) methods to PLMs?

Yes, we have conducted the experiments with fine-tuning ESM-2 650M with LoRA. However, this approach gave lower quality in the downstream tasks, so we decided to keep the full finetuning.