Protein Design with Dynamic Protein Vocabulary

Dear Reviewer Vivg,

First and foremost, we extend our gratitude for your thorough review. Here are our responses to the key points you've raised:

Q1: More ablation studies.

A1: Thanks for your valuable feedback.

What is the effect of removing the dynamic vocabulary mechanism while still allowing the model to access top-k fragment candidates selected using the text encoder from a static fragment pool?

As shown in Figure 3, ProDVa relies on both the static (tokens) and dynamic (fragments) protein vocabulary to determine: 1) whether to output a token or fragment next, and 2) which token or fragment to select. Therefore, without the dynamic vocabulary mechanism, these two mechanisms cannot operate as intended. In other words, the static fragment pool cannot be utilized by ProDVa without the dynamic vocabulary mechanism.

What is the effect of removing the fragment retrieval mechanism entirely (i.e., without any top-k fragment candidates)?

The table below compares two scenarios (i.e., the vanilla multimodal approach and the Top $K=0$ setting) that removing the fragment retrieval mechanism entirely. For further details on the vanilla multimodal approach, please refer to Section 4.4 in our manuscript.

The results indicate that, compared to the vanilla multimodal approach, ProDVa without the use of any Top $K$ fragments performs worse across most metrics. However, when utilizing the Top 1 fragments, ProDVa improves its performance by 11.17% in pLDDT and 11.67% in the ProTrek Score. This highlights the significance of incorporating Top $K$ fragments. The relatively poor performance without any Top $K$ fragments can be attributed to the fact that, during training, most sequences are split into tokens and fragments. Thus, relying solely on tokens (without incorporating any fragments) may lead to failure.

How does varying the size of the Fragment Candidates database affect model performance?

In our manuscript, the size of the supporting documents is identical to that of the training dataset for a fair comparison (see Lines 140–141). To investigate the effect of varying the size of the supporting documents (fragment database), we retain 20%, 40%, 60%, and 80% of the training dataset as supporting documents. Additionally, we incorporate the validation set to further increase the size of the supporting documents. For the statistics of the training and validation sets, please refer to Appendix C.2.

The table below presents the experimental results obtained by decreasing or increasing the size of the supporting documents. "T" denotes the training set (by default, "100% T" is used in our manuscript, see Table 2), and "V" denotes the validation set. We summarize the key observations as follows:

• The size of the supporting documents has minimal impact on PPL and foldability, with no significant differences observed as the size increases.
• Regarding the language alignment metrics, ProDVa generally performs better as the size of the supporting documents increases. A possible explanation is that a larger set of supporting documents enhances the likelihood of retrieving more relevant descriptions and corresponding sequences, thereby enabling the construction of more relevant fragment candidates.

---

Q2: Clarifications on function-based protein design problems (i.e., conditional generation).

A2: Sorry for misunderstanding. In our manuscript, we state that the problem addressed by ProDVa is function-based protein design (i.e., conditional generation). This problem can be formalized as follows (see Section 2, Lines 66–69):

$$p(P \mid t) = p((x_1, x_2, \cdots, x_k) \mid t, \forall i, x_i \in A \cup S)$$

where $t$ denotes the input textual description (i.e., the condition).

Here, we provide a more detailed clarification as to why ProDVa is specifically designed for function-based protein design problems, as opposed to unconditional protein generation.

• First, ProDVa leverages a Text Language Model in three key ways: (1) to interpret the input textual description, (2) to represent protein fragment descriptions during training, and (3) to retrieve relevant proteins based on functional descriptions during inference. In the context of unconditional protein generation, none of these functionalities are applicable.
• Second, the backbone Protein Language Model (ProtGPT2) employed in ProDVa is specifically designed for unconditional generation. Therefore, ProDVa inherently possesses the capability for unconditional protein generation.

Despite the above clarifications, we demonstrate how ProDVa can be adapted for unconditional protein generation. By simply fixing the input instruction to "Design a novel protein sequence," the problem addressed by ProDVa effectively reduces to an unconditional protein generation task. To further ensure that the text input as a no-op, we replace the retrieval method with the random selection of fragments. The table below presents the results on unconditional generation. Note that the language alignment metrics are not considered under unconditional settings.

ProDVa outperforms all baseline models on the unconditional protein generation task. Compared to the state-of-the-art baseline, Pinal, ProDVa generates 22.76% and 11.81% more well-folded proteins in terms of pLDDT and PAE, respectively. Compared to other fine-tuned models, ProDVa outperforms them by a large margin. This demonstrates the effectiveness of our method in both conditional and unconditional protein generation settings.

---

Q3: Explanations of applying ProDVa to Diffusion-based Language Models.

A3: Thanks for your question. ProDVa is specifically designed for autoregressive Transformer architectures. It employs a decoder-only backbone (see Lines 46 and 77) and is trained using curated learning objectives rather than simple next-token prediction (see Lines 92–102). Applying ProDVa to recent Diffusion-based language models remains a non-trivial and challenging task. Nonetheless, we believe this work establishes a foundation for future research on integrating ProDVa with diffusion models.

---

Q4: Explanations of the generalization under low-resource or novel protein design scenarios where related fragment annotations are unavailable or sparse.

A4: Thanks for your question. In fact, during inference, ProDVa does not rely on any fragment annotations (see Lines 121–123). The training paradigm involves learning functional annotations that are not required during inference. ProDVa learns to select the appropriate token or fragment at each step of decoding. Therefore, under low-resource settings, there is minimal impact on ProDVa's training paradigm. During inference, if no related fragments are provided, ProDVa generates tokens to design a novel protein sequence, similar to ProtGPT2.

Dear Reviewer VmDi,

First and foremost, we extend our gratitude for your thorough review. Here are our responses to the key points you've raised:

Q1: Explanations of using significantly less training data compared to Pinal.

A1: Thanks for your questions. For the first question:

Why is the training data used for ProDVa smaller than that used for Pinal?

During training, ProDVa utilizes approximately 712K text-protein pairs, whereas Pinal employs 1.76B pairs. Despite relying solely on annotated datasets (e.g., SwissProt), Pinal achieves this larger dataset through various data augmentation methods, including the generation (automatic annotation) of textual descriptions for proteins. For example, it uses models such as GPT-4 and GLM-4-Flash to convert records from SwissProt into complete sentences and domain-specific question-answer pairs that serve as descriptive annotations for different proteins. Additionally, Pinal leverages ProTrek to annotate proteins from UniProtKB and UniRef50, resulting in a large-scale synthetic dataset. In contrast, our approach primarily focuses on SwissProt and its corresponding annotations, which are manually curated and reviewed by experts. Our experiments yield the following observations:

• High-quality data (reviewed and manually annotated) is sufficient to train ProDVa to achieve performance comparable to Pinal, which is trained on a large-scale synthetic dataset (model-predicted, unreviewed annotations).
• ProDVa demonstrates data efficiency, utilizing only 0.02%–0.04% of the data used by Pinal, as it does not rely on synthetic annotations.
• The dataset used for ProDVa is entirely open-source, and its acquisition is straightforward and time-efficient, in contrast to Pinal, which depends on LLMs or other oracle models for data generation.

For the second question:

Whether increasing the training data might lead to better results than Pinal?

We have discussed the question in our manuscript (Lines 229-241). Here is a summary:

• Unfortunately, the large-scale datasets used by Pinal have not yet been open-sourced. Reproducing the synthetic annotations described above is challenging for us, as the process is both costly (e.g., requiring the use of LLM APIs and deployment of oracle models) and time-consuming (e.g., UniRef50 alone has over 41M proteins, yet UniProtKB includes more than 173M).
• In Table 2, we demonstrate that ProteinDT and PAAG perform significantly better after fine-tuning with additional data (the same dataset used by ProDVa), particularly with respect to Language Alignment metrics. Therefore, we carefully use the phrase "may potentially" to conclude that incorporating more data may improve performance.

---

Q2: Clarifications on choosing the tokenizer (tokenization method).

A2: Thanks for your question. Tokenization in our work involves two aspects: tokenizing amino acids and tokenizing fragments.

First, for amino acid tokenization, since ProDVa utilizes ProtGPT2 as the PLM backbone, we employ the ProtGPT2 tokenizer (a BPE tokenizer) for both tokenizing amino acids and constructing the static protein vocabulary. Alternative tokenization methods (e.g., per-amino-acid tokenization) are not suitable for our purposes.

Second, for fragment tokenization, we introduce our methods in Section 2, based on the intuition and empirical study presented in Section 1 (see Figure 1). Additional experiments in Appendix E.5 demonstrate that our tokenization method exhibits a degree of robustness to noise when encountering irrelevant fragments or random N-gram subsequences.

---

Q3: Evaluations on Sequence Diversity.

A3: Thanks for your valuable feedback. Using fragments to represent proteins has minimal impact on the diversity of designed sequences. In contrast, our training paradigm enables ProDVa to design user-specified proteins while maintaining diversity in the generated sequences.

We have included experiments and analyses on Sequence Diversity in our response to Reviewer Sz6U in Q1. Please refer to that response for more details. A summary of the key observations is provided below:

• ProDVa exhibits comparable sequence diversity to all other baseline models. This demonstrates that ProDVa is capable of designing diverse sequences that satisfy the same input descriptions.
• Under the same training settings, ProDVa maintains high sequence diversity, outperforming these two fine-tuned baselines by 14.14% and 7.92%, respectively. This underscores ProDVa’s ability to design protein sequences that satisfy user-specified requirements while preserving diversity in the generated sequences.

Dear Reviewer Dwmc,

First and foremost, we extend our gratitude for your thorough review. Here are our responses to the key points you've raised:

Q1: Expanding the figure captions to provide more descriptive information.

A1: Thanks for your valuable feedback. We will expand the figure captions in the final version as follows:

• Figure 1(b): Performance on pLDDT (higher is better). Our method significantly improves the pLDDT of proteins generated by Random+ by a large margin of 12%, with the average pLDDT exceeding the well-folded threshold.
• Figure 1(c): Performance on PAE (lower is better). Compared to Random+, our method achieves a 9% improvement in PAE and is the only model to surpass the well-folded threshold. Notably, our method outperforms the state-of-the-art baseline model Pinal in both pLDDT and PAE.

---

Q2: Evaluations on designing proteins under out-of-distribution settings (e.g., evaluating on novel synthetic functions).

A2: Thanks for your valuable feedback. To evaluate performance in designing novel proteins under out-of-distribution (OOD) settings, we first construct a test set comprising 1,000 randomly synthesized novel functions. The Mol-Instructions test set is utilized, and all unique functions are extracted. For each synthetic function, 2–5 different functions are randomly selected and concatenated into a single composite function. Due to the random selection, combination, and arrangement of functions, the resulting synthetic functions are guaranteed to be novel and under OOD settings.

Some key findings can be summarized as follows:

• ProDVa outperforms other baseline models under OOD settings, including the state-of-the-art baseline Pinal, across both foldability and language alignment metrics.
• In terms of foldability, both ProDVa and Pinal demonstrate promising performance, indicating that OOD settings do not affect a model’s ability to design structurally plausible proteins.
• For language alignment, all models perform worse than on the Mol-Instructions test set (see Section). This may be attributed to two main factors. First, the OOD settings pose significant challenges for current models. Understanding synthetic functions that may not exist in the real world likely requires further investigation in future work. Second, the OOD settings may not be properly validated, as verifying the plausibility of these synthetic functions is inherently difficult (e.g., may require further wet-lab experiments). Moreover, natural proteins with such properties may not exist, potentially rendering the design of proteins with novel synthetic functions unfeasible.

---

Q3: Explanations of weighting fragment types.

A3: Thanks for your question. As mentioned in our manuscript (Lines 105–110), the fragment types in our training set (CAMEO subset and Mol-Instructions) are inherently unbalanced. Enforcing a balanced distribution of fragment types would require removing up to 80% of the training data, potentially resulting in insufficient data for training. To address this issue, we introduce a weighted type loss $\mathcal{L}_{\text{TYPE}}$, as described in Equation (4) (see Lines 105–110).

Dear Reviewer Q2yf,

First and foremost, we extend our gratitude for your thorough review. Here are our responses to the key points you've raised:

Q1: Explanations of designing proteins with novel fragments.

A1: Thanks for your question. The novel protein sequences designed by ProDVa comprise two components: sets of tokens (amino acids) and sets of fragments. The novelty of these sequences is determined by both factors. In terms of tokens, ProDVa possesses capabilities comparable to those of ProtGPT2. In terms of fragments, fragment candidates are dynamically retrieved from a database (supporting documents) during inference. In our experimental settings, the retrieval database contains over 1 million unique fragments (1.16M for the function keyword setting and 1.43M for the textual description setting). Consequently, ProDVa is capable of designing novel protein sequences.

Additionally, we will explain our choice of using a Fragment Encoder instead of a Fragment Decoder, which might otherwise generate novel fragments. We randomly mutate the amino acids in fragments of the protein sequences designed by ProDVa with a 40% probability, replacing each amino acid with a different one. The table below presents a comparison between ProDVa and simple mutation on fragments. Both foldability and language alignment perform significantly worse under random mutation. These results highlight the challenges of incorporating novel fragments and indicate that generating novel fragments may not be suitable for designing user-specified protein sequences.

---

Q2: Clarifications on how to interpret the results and metrics in Table 1.

A2: Thanks for your valuable feedback. The metrics presented in Tables 1 and 2 are used to evaluate the quality of the designed proteins and to compare them with various baseline models. Admittedly, interpreting these metrics solely based on whether higher or lower values are better may not always be appropriate. To more intuitively illustrate performance differences among models, we report the absolute values of each metric rather than their relative differences (i.e., delta values) from natural proteins. This approach is consistent with prior studies (e.g., ProteinDT, Pinal), which also present absolute scores. Due to space constraints and to maintain the simplicity of our table presentations, delta values were not included in the current version. We will consider incorporating them in the final version. In this way, we believe that comparisons with other models using absolute values, as well as comparisons with natural protein distributions using delta values, are both intuitive.

---

Q3: Discussions and evaluations on computing the PPL metric with the alternative Protein Language Model.

A3: Thanks for your valuable feedback. To evaluate the impact of different Protein Language Models on the computation of the PPL metric, we selected three additional models: ProGen2 [1], RITA [2], and ProteinGLM [3]. The table below presents supplementary results for the PPL metric (i.e., PPL (ProtGPT2)) as referenced in Table 2.

The results are consistent with the PPL values computed by ProtGPT2, despite differences in absolute scores. ProDVa consistently remains within a low PPL range and is closer to natural proteins.

---

Q4: Explanations of fragment types.

A4: Thanks for your question. In our manuscript, we define fragments of a protein as continuous subsequences (see Lines 58–59), implying that any arbitrary subsequence can be considered a fragment. However, fragments lacking functional annotations are not significant for designing proteins with user-specified requirements. Therefore, identifying usable fragments is a central challenge in our work. In Section 2 and Appendix B, we describe how InterPro is employed to extract fragments with functional annotations. Moreover, a single protein may contain multiple fragments, each potentially associated with different properties (or functions). Thus, we consider "fragment" to be the most appropriate term, as the fragments used in our work possess diverse functional annotations.

---

Q5: Further discussions on the results of Retrieval Accuracy metric.

A5: Thanks for your question. We argue that ProDVa’s high Retrieval Accuracy is unrelated to retrieving fragments from existing protein datasets. The metric used (see Lines 525–534 for details) evaluates whether the designed protein shows the highest similarity to the textual description among a set of randomly selected candidates. Similarity is computed using the ProTrek model, which is not involved in the training or inference of ProDVa, thereby avoiding evaluation bias. Moreover, whether the sequences are designed from scratch or generated by ProDVa, they more or less exhibit features of natural proteins (e.g., evolutionary information). Importantly, the candidate pool for Retrieval Accuracy consists entirely of natural proteins, ensuring that the metric provides an unbiased comparison of protein sequences generated by different models.

References:

[1] Nijkamp, Erik, et al. "Progen2: exploring the boundaries of protein language models." Cell systems 14.11 (2023): 968-978.

[2] Hesslow, Daniel, et al. "Rita: a study on scaling up generative protein sequence models." arXiv preprint arXiv:2205.05789(2022).

[3] Chen, Bo, et al. "xTrimoPGLM: unified 100B-scale pre-trained transformer for deciphering the language of protein." arXiv preprint arXiv:2401.06199 (2024).

Dear Reviewer Sz6U,

First and foremost, we extend our gratitude for your thorough review. Here are our responses to the key points you've raised:

Q1: Evaluations on Sequence Diversity and structure-based metrics in terms of clutsters of sequences.

A1: Thanks for your valuable feedback.

Sequence diversity should be reported alongside other metrics.

We first present the experimental results on Sequence Diversity. MMseqs2 [1] is used to compute the similarity $\mathrm{sim}$ between each pair of sequences in a batch of size $N$, where all sequences in the batch are designed based on the same textual description (i.e., with different random seeds). Sequence Diversity is formalized as follows:

$$\text{Sequence Diversity} = \frac{\sum_i (1 - \mathrm{sim}_i)}{N(N - 1)}$$

The results (in %) are presented in the table below. Key observations are summarized as follows:

• ProDVa exhibits comparable sequence diversity to all other baseline models. Compared to the state-of-the-art baseline model Pinal, our method achieves similar performance on the Mol-Instructions dataset and outperforms it by over 15% on the CAMEO subset. This demonstrates that ProDVa is capable of designing diverse sequences that satisfy the same input descriptions.
• On the Mol-Instructions dataset, we observe that both ProteinDT and PAAG exhibit significantly reduced sequence diversity after fine-tuning, with decreases of 19.36% and 13.06%, respectively. In contrast, under the same training settings, ProDVa maintains high sequence diversity, outperforming these two fine-tuned baselines by 14.14% and 7.92%, respectively. This underscores ProDVa’s ability to design protein sequences that satisfy user-specified requirements while preserving diversity in the generated sequences.

Additionally, It would be interesting to report pAE (or other structural metrics) in terms of clusters of sequences.

To evaluate foldability in terms of clusters of sequences, we first use MMSeqs2 to cluster sequences at various identity thresholds (indicated in parentheses). The average pLDDT across clusters is defined as follows (similarly for PAE across clusters):

$$\text{pLDDT across clusters} = \frac{1}{k} \sum_{j=1}^{k} \left( \frac{1}{n_j} \sum_{i=1}^{n_j} \text{pLDDT}_{ij} \right)$$

where $k$ is the number of clusters, and $n_j$ denotes the number of proteins in the $j$-th cluster.

The results indicate that ProDVa consistently outperforms all baseline models in both overall foldability and foldability across clusters. At lower identity thresholds, the pLDDT decreases slightly compared to fine-tuned ProteinDT and PAAG, demonstrating ProDVa's robustness across diverse designed protein sequences.

We will include the results above in the final version.

---

Q2: Clarifications on the applicability of ProDVa.

A2: Thanks for your question. ProDVa is proposed for the design of single-chain proteins, consistent with previous work such as ProteinDT, PAAG, and Pinal.

---

Q3: Explanations of using ESM C as the embedding model in Figure 1.

A3: Thanks for your question. There are two main reasons why we prefer to use ESM C instead of Pinal.

First, Pinal is a text-to-protein model that takes natural language as input and is not specifically designed to compute protein embeddings. It comprises two main modules: a text-to-structure module and a SaProt [2] module, of which only the latter can be used as an embedding model.

Second, technically speaking, computing protein embeddings using ESM C and Pinal (the SaProt module) exhibits very similar behavior, including for those generated by Random(+) and ProDVa. This similarity arises because Pinal was trained on proteins from SwissProt, UniRef50, and UniProtKB. Consequently, both ESM C and Pinal are unlikely to have encountered the remaining proteins, whether random or novel.

Therefore, we believe that ESM C is a more suitable choice for embedding the landscape of random proteins and novel proteins designed by ProDVa, as it can offer more meaningful insights for comparison with natural proteins.

To validate this, we use Pinal (the SaProt module) as the embedding model to visualize the landscape, following the approach used in Figure 1(a). The visualization results are highly consistent with those obtained using ESM C. Proteins generated by Random are clustered, while those from Random+ exhibit a significantly more diverse distribution. As expected, proteins designed by ProDVa span the landscape of natural proteins. The visualization results using Pinal will be updated in the final version.

---

Q4: Clarifications on the colors chosen for legends used in Figure 1(a).

A4: Thanks for your valuable feedback. The legend colors in the upper right are consistent with AlphaFold's color scheme. We will adjust the legend color scheme for scatter points in the final version.

---

Q5: Clarifications on the Equations (1) and (2).

A5: Sorry for misunderstanding. There is nothing missing on the left side of the curly brackets in Equations (1) and (2). They denote a group of variables.

---

Q6: Further explanations of the representations on Line 86.

A6: Sorry for misunderstanding. The projection layer is optional. If the Fragment Encoder shares the same dimension as the PLM (i.e., $d_{\text{PLM}}$), the use of the adapter is determined by the user. Otherwise, the projection layer is employed to map the fragment embedding matrix (representing $m$ fragments) to the same embedding space as the PLM.

References:

[1] Steinegger, Martin, and Johannes Söding. "MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets." Nature biotechnology 35.11 (2017): 1026-1028.

[2] Su, Jin, et al. "Saprot: Protein language modeling with structure-aware vocabulary." BioRxiv (2023): 2023-10.