Elucidating the Design Space of Multimodal Protein Language Models

Many thanks for your comments that have greatly improved our manuscript! We address your concerns as below. We sincerely thank you once again and welcome any further feedback!

Q1: Can the authors comment on any simple baselines or alternative approaches they pursued to resolve the identified problems (O1, O2, and O3)?

We recap our approaches to address O1-O3 and discuss potential alternative approaches below.

O1: Structure tokenization results in information loss

Our approach: We train an additional continuous diffusion module ResDiff to recover the lost residuals of discrete tokens.

Alternatives: Training DPLM2 on continuous tokens. However, it's unclear whether this approach would work for joint modeling of structures and sequence, which is inherently discrete.

O2: High reconstruction accuracy of structure tokenizer does not ensure better structural generative results.

Our approach: We primarily improve the designs of latent structure modeling, including structure-aware generative modeling, architectural designs, and representation alignments.

Alternatives: Adopt direct modeling in the data space following a similar manner as Proteina [1]. Similarly, it's unclear if direct modeling is robust for joint modeling of discrete sequences and continuous 3D structures.

O3: Multimodal PLM gets index-based structure token prediction miserably wrong. Small bit-wise changes could result in a dramatically different index label. This challenge intensifies as codebook size grows, making direct index prediction even more difficult.

Our approach: Finer-grained token prediction might resolve such challenge. We achieve a finer-grain prediction on the "dimension-level" using bit-based labels in contrast to index-based labels.

Alternatives: Another potential direction is to hierarchically tokenize the structures to achieve fine-grained tokens at the "resolution" level following methods like RQ-VQVAE [2] and VAR [3]. However, it remains elusive how to well define "resolution" of proteins as a natural choice.

Q2: The paper discusses SaProt but does not explicitly compare to it or other related "structure-aware" pLMs beyond ESM3. Can the authors comment on their reasoning for including/not including specific comparisons?

Thanks for your question. In our submitted version, we focus on the generative capabilities of protein structure (e.g., folding). However, SaProt focuses on the protein representation learning ability, which utilizes structure tokens as the representation for structure input and is not capable of generating structure.

In the table below, we also provide the results of structure-aware protein representation learning, following the setting of SaProt. Due to the limited time, we only evaluate on the HumanPPI and DeepLoc Subcellular tasks, and we will provide results of more protein predictive tasks in the final version. As seen, the proposed bit-based modeling is able to further improve the representation learning performance of DPLM-2. We hypothesize that the bit-based modeling, which is finer-grained, is more suitable to learn and enables the multimodal PLM to capture the structural pattern in latent space more effectively, thus enhancing structure-aware representation learning.

[1] Geffner et al. Proteina. ICLR 2025.

[2] Lee et al. RQ-VAE. CVPR 2022.

[3] Tian et al. VAR. NeurIPS 2024.

Essential References Not Discussed: Lu, Amy X., et al. "Compressing the Latent Space of Single-Sequence Protein Predictors for Multimodal Generation."

Thanks for pointing this out to us, and we will elaborate on our discussion below in the final version.

This work compresses the latent space of ESMFold to form a joint latent space of protein sequence and structure, and learn a continous generative model to sample compressed latent embeddings, which is then decoded by the ESMFold structure head and a learned sequence head. On the other hand, the multimodal protein language models like DPLM-2, use discrete diffusion generative framework to generate the discrete structure token, then decode using the learned structure detokenizer.

The CHEAP embedding introduced by this work is an effective compression of ESMFold latent space. Integrating the CHEAP embedding with multimodal protein language models holds great potential for effective structural modeling. This is a very exciting direction and we will leave it for future work.

Many thanks for your comments that have greatly improved our manuscript! We address your concerns as below. We sincerely thank you once again and welcome any further feedback!

W1: The current manuscript does not address structure-conditioned protein generation or structure-aware protein predictive tasks.

Thanks for your suggestion. We provide additional results on structure-conditioned protein generation and protein predictive tasks below.

(1) Inverse Folding (structure-conditioned sequence generation).

We report the amino acid recovery (AAR) and self-consistency TMscore on the Cameo dataset. Our results show that the 650M-parameter DPLM-2 variant, with bit-based modeling, outperforms the 3B-parameter baseline. Incorporating geometric architectural designs and representation alignments leads to further improvements. These findings indicate that these design methods contribute positively beyond structure generation.

(2) Structure-aware protein representation learning.

In the table below, we provide the results of structure-aware protein representation learning, following the setting of SaProt. Due to the limited time, we only evaluate on the HumanPPI and DeepLoc Subcellular tasks, and we will provide results of more protein predictive tasks in the final version.

As seen, the proposed bit-based modeling is able to further improve the representation learning performance of DPLM-2. We hypothesize that the bit-based modeling, which is finer-grained, is more suitable to learn and enables the multimodal PLM to capture the structural pattern in latent space more effectively, thus enhancing structure-aware representation learning.

W2: The model fine-tuned without PDB multimer data achieved a higher TM-score but a lower RMSD. A more in-depth analysis would provide valuable insights.

Thanks for bringing up this discussion. RMSD captures local atom deviations, while TM-score measures global structural similarity. In multimers, chains are typically spaced farther apart than individual connecting residues, leading to a higher RMSD for models trained on only monomers. This highlights the importance of finetuning with multimer data to reduce RMSD. Meanwhile, fine-tuning on Swissprot (200K) helps learn the global protein structures better due to its larger dataset compared to PDB-Multimer (3.5K). As shown in Table 10 of our paper, incorporating Swissprot consistently achieves higher TM-score on both PDB-Multimer and Cameo.

Q1: Does the hybrid generative approach with flow matching significantly increase training time?

We report the training time of the following DPLM-2 variants on 16 H100s for 300k training steps. The increased training time of flow matching (FM) primarily comes from the on-the-fly computation of noisy structure $\mathbf{x}_t$ using structure encoder, as we can precompute the discrete structure tokens when FM is not applied. FM remains highly effective when inference efficiency is a priority, as it accelerates the sampling process by 10x by requiring fewer sampling steps.

Many thanks for your comments that have greatly improved our manuscript! We address your concerns as below. We sincerely thank you once again and welcome any further feedback!

The bit-level and index-level evaluation is really stark. Is there any broader intuition behind this that you could check experimentally?

Index-level prediction is highly challenging because small changes at the bit level can result in drastically different indices, as shown in the example below. This issue becomes even more problematic as the codebook size increases, further exacerbating the difficulty of direct index prediction.

W1: Some design choices feel somewhat ad-hoc due to the lack of discussions on motivation. It could benefit from greater clarity with a summary table linking the whole design choices.

Thank you for this comment. To make our presentation more structured, we provide a summary TABLE with greater discussion on all the design choices, traditional methods, motivations, and findings.

W2: It would benefit to have more uncertainty estimates in the tables, especially because some of the values highlighted seem to have small gains.

Thank you for your comment. We are actively launching three more independent runs of the following selected models to provide uncertainty estimates using the mean and the 95% confidence interval (CI). We will include the uncertainty estimates of other variants in the final version.

W3: It would be beneficial to provide a summary table with greater discussion about the novelty of the design space.

Thank you for your suggestion. We provide the summary table above in our response to W1.

The overall analysis is done on relatively small PLMs (up to 3B). I don't see this as an issue and I understand the constraints of computation resources, but it would be great to acknowledge more explicitly.

Thanks! We acknowledge this statement and will make it explicit in our final version as you suggested. While scaling is effective as shown in DPLM-2, we'd like to reassert that correct design choices are critical. In our work, we show that our 650M multimodal PLM could achieve on par results with the 3B specialized ESMFold on structure folding, showing high parameter efficiency.

Some missing benchmarks (RoseTTAFold, OmegaFold, and AlphaFold3) on structure prediction tasks could be useful, even though I'm happy for the authors to claim that the focus is on protein language models, which differ from the nature of these benchmarks.

Thanks for your suggestions. We will make a clear claim on our focus on PLMs and will add these benchmarks to the table in the final version.

Many thanks for your comments that have greatly improved our manuscript! We address your concerns as below. We sincerely thank you once again and welcome any further feedback!

W1: the paper would benefit from a more structured presentation of its contributions (e.g. a taxonomy)

Thank you for your suggestion. We present a comprehensive taxonomy of design choices in a table and discuss their orthogonality and synthesis, as compiled in this TABLE.

Q1: What is the final recommended design choice? (e.g., combining GeoDPLM + FM + RESDiff + folding SFT + REPA altogether.)

We structured the paper to clearly present the impacts of each design. To provide a complete picture, we now include new results that combine all design methods.

SFT: Folding SFT improves the structure folding but sacrifices the model's ability for multimodal co-generation, as it is fine-tuned specifically for the folding task. For models finetuned with the folding SFT objective, we skipped the evaluation of unconditional co-generation.

The recommended setting — Geo + Bitwise. Geometric architectures are compatible with bitwise modeling and their combinations achieve comparable results with models finetuned with folding SFT on structure folding, and further obtains an effective improvement on unconditional generation quality & diversity. This setting is also effective in terms of training efficiency as it avoids additional computational overhead from other methods like FM and REPA.

REPA and bit-based modeling. Both REPA and bit-based modeling enhance structure folding and generation diversity. Meanwhile, as shown in the table, their combinations do not lead to further improvements. We suggest that this is because REPA and bit-based modeling both help through enabling smooth and high-dimensional learning signals compared to index-based discrete tokens, hence their non-orthogonal effects.

Hybrid modeling (w/ FM) and geo module. FM effectively improves folding, but the benefits diminish with Geo modules, and can reduce generation diversity due to its ODE nature. However, benefitting from the same nature, FM accelerates the sampling process by requiring 10x fewer sampling steps.

ResDiff. Similar to the results in the paper, ResDiff does not bring a significant boost to folding metrics. The major benefit of ResDiff is to provide a finer local structure as discussed in the Fig. 7 of our Appendix.

Q2: Clarifications on experimental configurations in Table 4 (+ResDiff, +FM, and w/SFT).

We elaborate on our experiment settings, as compiled in this TABLE

Q3: Hybrid structure modeling method seems to rescue the skewed structure modeling ability of DPLM-2, which adopts the pre-trained sequence model DPLM and may be suboptimal compared to training from scratch.

Thanks for raising this insightful discussion. We conducted experiments as you suggested

As shown, DPLM-2 initialization outperforms training from scratch (see table), likely because: (1) sequence pretraining improves structural modeling (cf. ESMFold); (2) Sequence data vastly outnumbers experimental structures, making pretraining more effective.

Q4: On REPA boosting diversity and accuracy? Is it related to the unbalanced training set for the original DPLM?

We agree that the unbalanced DPLM training set may contribute to REPA's benefits. REPA further aids structure generation by:

• ​Overcoming discrete token limitations: Quantization loses finer details (Table 1), while representation alignment helps preserve structural nuances via smooth, informative and high-dimensional learning signals.

• ​On learning diffusion model: REPA paper points that high-quality representations are key to diffusion models; we bridge this gap by transferring structural features from folding models.