Improving Inverse Folding for Peptide Design with Diversity-Regularized Direct Preference Optimization

Dear Reviewer,

Thank you for your comments and suggestions! We hope our responses can clarify some of your concerns.

The verfication of the proposed method is limited. There's so many popular approaches on protein inverse folding, such as PiFold, ESM-IF, LM-Design, VFN, and InstructPLM [1-5]... Regarding recovery and self-consistency TM-Score, there are approaches performing better than or competitive than ProteinMPNN. Thus, a robust method with board application should include more experiments to verify the effectiveness of the proposed methods.

We agree with the reviewer’s assessment here: our point is to show that our DPO method is better than no finetuning, as well as the original DPO. Due to some data access issues, we were unfortunately unable to run any other benchmarks besides the ones in Tables 1 and 2. Unfortunately, there’s not much we can do here - we hope the reviewer can understand.

The proposed metric is not comprehensive enough, the self-consistency RMSD is also required to verifying the quality of sequence samples.

Thank you for this suggestion! We agree and apologize for this oversight. Unfortunately, we were not able to compute these during the review period.

The method is simple but the experiement results are not impressive enough to prove the effectiveness of DPO-based improvement. In Table1, the relative improvement over all metrics is smaller than or equal to 1% except TM-Score with 6%. The ablation results cannot indicate the power of proposed method enhancement.

We will address each metric separately to show that our DPO method does indeed significantly outperform standard DPO and the base method.

On recovery: the recovery metric is not a well-suited metric, as even if all generated sequences are high quality (high TM-score, or RMSD with target), native sequence recovery is by definition anticorrelated with the diversity of the sequence set. We report it only for consistency with prior work.

On TM-Score: as the reviewer noted, there are clear gains with DPO. These gains, importantly, are not diluted by applying diversity-regularized DPO, which matches the performance of DPO while also jointly optimizing for diversity.

On diversity: we note that applying DPO to ProteinMPNN results in a model with 14% less diverse generations but 8% better TM-Score. However, our DPO method is able to match the original diversity of ProteinMPNN while maintaining that 8% boost in TM-Score. Therefore, while DPO is unable to significantly beat vanilla ProteinMPNN’s diversity, it erases the diversity loss we get from finetuning. That is, with our method, we achieve a Pareto improvement along the diversity-TMScore frontier compared to standard DPO and ProteinMPNN.

Thus, our method achieves the diversity/TM-Score co-optimization objective originally set forth.

The length scale of main experiments are somehow not consistent, which makes the paper unclear to the readers. In Table1&2, the author states that they have had experiments on CATH4.3 (n=173) and Openfold (n=50), but 1) the filtering process is not clear in Openfold benchmark; 2) the filtering criteria of two benchmarks are inconsistent.

Apologies for the confusion. The OpenFold set was constructed by randomly sampling 128 proteins from the larger train set (from the ColabFold database with sequence length <= 50), then filtering sequences at 0.4 sequence identity threshold level. Since we ran OpenFold experiments first and did not want to modify the filtering criteria for CATH 4.3 since it is standard, we were unable to match the two criteria between benchmarks.

Why the diversity of ESM-IF on both datasets is zero?

Tables 1 and 2 report results at T=0. Since ESM is fixed decoding order, its sampling diversity is zero at T=0.

Once the DPO-based method is effective to inverse folding methods, why the experiments are only conducted to the general protein. It is because I believe that the challenges you stated also exist in the general proteins.

The algorithms we designed were selected for peptide design - for example, it is less likely that the diversity problem is as pronounced in general protein design, since two shorter sequences have a greater chance of being similar compared to two longer sequences. However, we agree with the reviewer that our algorithms can be applied to general sequence design for co-optimizing diversity and self-consistency score. We only run experiments on peptide inverse folding here, so we make no claims about other domains (general proteins) but suspect similar results may hold.

In the paragraph "Explaining diversity with differential entropy", can you please try to explain the interpretability via a mathematical proof to clearly state the connections between log-probabilities and diversity?

Thank you for the suggestion! We add a proof sketch in the appendix.

Thank you again for your patience with our responses! We hope this clarifies your concerns and look forward to additional discussion.

Dear Reviewer,

We respond to two remaining comments. Apologies for the lengthy response.

In Section 4.4, the paper introduces a new method using entropy instead of pairwise distance as a regularization term. However, this setting is not presented in the main results. Please include this in the main results.

Our apologies again for the confusion. The difference between the entropy optimization algorithm in the appendix and the main diversity-regularization algorithm in Section 3 is that the former is completely offline while the latter is online. In the language of our differential entropy theory, offline entropy optimization naively tries to upweight the discrete entropy of sampled sequences, while online diversity regularization implicitly upweights the differential entropy of log-probabilities themselves. As per the appendix, the offline method fails to meaningfully improve diversity. To clarify, the offline method is our work as well, a simple algorithm whose poor performance supports the differential entropy theory.

Thus, the entropy method is not part of our core results, but rather further justification for our choice in designing an online DPO method. Particularly, it provides empirical support for our theoretical argument about differential entropy optimization in online diversity-regularized DPO.

In line 267, the paper claims to use 4 samples and T=0, which is an unusual hyperparameter setting. T=0 implies deterministic sampling of ProteinMPNN (T=0.1 by default), and 4 samples are too few. I encourage using more samples (e.g., 64) and the original T setting.

Thank you for the suggestion! We have sampled with 64 instead of 4 samples per structure and at T=0.1, the results are in the appendix of the revised version. The ordering of the results does not change. On the AF benchmark, for example, our methods achieve around 13% diversity gains over regular DPO, and beat regular ProteinMPNN by 14%.

Dear Reviewer,

Thank you for your time and effort in your review! We appreciate your feedback and hope that our responses can clarify the points you have raised.

As discussed in Sections 3.1 and 3.2, incorporating diversity regularizations and scalar weighted control into DPO is not a novel technique; it has been extensively studied in prior work, which may limit the paper's novelty… The author should provide more comparisons to prior works, highlighting the differences, such as online optimization or ad-hoc scalar weighting. If possible, please include the original diversity regularization DPO and scalar weighting DPO in your baselines to study the improvements over these methods.

Our apologies for the confusion. To clarify, diversity regularization DPO and scalar weighted DPO are our original work.

To the author’s knowledge, we are the first to use DPO to improve sampling diversity - this is our first contribution. Our second contribution is deriving an online sampling term in the DPO algorithm, and providing justification for why our method works. Recent work [1, 2, 3] in the DPO literature has shown that online DPO, i.e. augmenting the train dataset with online samples, is helpful. However, previous work applying DPO to proteins [4, 5] only consider offline DPO. So, our work is novel in three ways: it considers online DPO for proteins, presents a principled derivation for the online sampling part, and is the first to consider DPO for diversity optimization at all.

Our last contribution is the scalar weighted DPO method, which has intuitive motivation presented in Section 3.2. This is similar to the “weighted DPO” method from ProteinDPO [4] in that both methods leverage continuous rewards instead of preferences as in standard DPO. However, in [4], weighted DPO does not significantly outperform standard DPO; we attempt to do better.

The presentation in Section 4 is somewhat unconventional; the theoretical analysis in Section 4.4 appears confusing and out of place.

We have updated the rebuttal revision, moving the theoretical analysis to Section 3 in methods.

The experimental setup is not comprehensive, particularly in terms of baseline details and sampling settings… The baseline results are limited and may not be convincing… Additionally, I recommend including more baselines, particularly fixed-backbone models specific to peptide design tasks (e.g., PepFlow by Li et al. 2024).

We were unfortunately unable to run the full set of benchmarks we wanted. Many newer popular methods are implemented in ProteinInvBench [6], but we were unable to get the benchmark harness to produce reasonable results with existing models, so we did not report the results here. We agree that the argument would be strengthened with additional baselines/experiments, and thank you for the specific direction towards PepFlow. Unfortunately, due to logistical constraints, it is no longer feasible for us to run any additional benchmarks. We hope you can understand in this regard.

Clarify how the baselines are used: are they pretrained or fine-tuned? Since baseline methods like GVP-GNN are not trained on the OpenFold benchmark, this comparison may not be fair. Please provide clarification.

The benchmark models are pre-trained. We would like to highlight that the most important comparison we are making is between vanilla ProteinMPNN, ProteinMPNN + standard DPO, and ProteinMPNN + our DPO methods. Ideally, we have similar DPO finetuned results for each base model. We unfortunately were not able to implement DPO for the other base models, but report their non-finetuned results for clarity.

However, our results suggest that our DPO methods do indeed outperform ProteinMPNN, as well as standard DPO. In Table 1, vanilla ProteinMPNN has 14% higher diversity than ProteinMPNN + DPO, but ProteinMPNN + DPO has 8% higher TM-Score. ProteinMPNN + our diversity-regularized DPO is able to maintain the diversity level of vanilla ProteinMPNN while also maintaining the 8% higher TM-Score that comes with fine tuning.

Thank you again for your patience with our responses! We hope this clarifies your concerns and look forward to additional discussion.

[1] Bootstrapping Language Models with DPO Implicit Rewards

[2] Direct Language Model Alignment from Online AI Feedback

[3] Understanding the performance gap between online and offline alignment algorithms

[4] Preference optimization of protein language models as a multi-objective binder design paradigm

[5] Aligning protein generative models with experimental fitness via Direct Preference Optimization

[6] https://github.com/A4Bio/ProteinInvBench

Dear Reviewer,

Thank you very much for your comments and insights. We are grateful for the opportunity to respond and clarify where possible. Thank you for your patience!

The use of the TM-score in training may lead to bias since the TM-score is obtained through a static structure prediction model. Although it may not be easy to analyze the reason, I think at least the authors can mention that and do some analysis (like using multiple structure prediction tools to calculate the TM-score to average the bias). It might be a little bit time-consuming, so it doesn’t have to be done in the rebuttal phase and it would have no effect on my rating. Just a concern.

We agree with the reviewer. Our initial plan was to use the 5 trained AF model weights to bootstrap the computation of TM-scores. However, due to infrastructure limitations, we were only able to use 1 such model for structure inference.

The approach itself is a bit target-specific, it did not work well on the co-optimization purpose as the author claims. From Table 1, the diversity and recovery do not excel from others, I am not blaming the results but curious about the reason. Agreed. We report it only for consistency with prior work. We would like to highlight that the recovery metric is not a well-suited metric, as even if all generated sequences are high quality (high TM-score, or RMSD with target), native sequence recovery is by definition anticorrelated with the diversity of the sequence set.

On the diversity metric, we highlight that our methods achieve a Pareto improvement over vanilla ProteinMPNN and ProteinMPNN + standard DPO. Applying standard DPO to ProteinMPNN produces a model with 14% worse diversity but 8% better TM-Score compared to the vanilla (base) model. However, applying our diversity-regularized DPO approach to ProteinMPNN produces a model with slightly better diversity and still 8% better TM-Score compared to the base model. Thus, our method does indeed achieve the diversity/TM-Score co-optimization objective originally set forth.

As the paper compared to GVP, ProteinMPNN, and ESMIF, why doesn’t the paper compare with some more recent approaches? Since the paper cited the ProteinInvBench, it provided a model zoom and colab demo to re-run the newer models easily if the input sequence is ready.

We were able to set up benchmarks with ProteinInvBench, but due to some unresolved issues in the benchmarking repository, the results we observed with these newer approaches were not sensible. We were ultimately unable to resolve these issues, and so could not in good faith put out comparisons to the models in ProteinInvBench. While ideally we could make these comparisons, we appreciate your understanding in this regard.

Could there be an appendix for analyzing the statistical features of the dataset used in training?

Thank you for the suggestion. We have added an appendix considering the length distribution, token distribution, and pLDDT score distribution of the train set.

Thank you again for your patience with our responses! We hope this clarifies your concerns and look forward to additional discussion.

Dear Reviewer,

Thank you very much for your thoughtful suggestions and comments! We appreciate your patience, and thank you in advance for considering our responses.

“ProteinDPO [1] has applied DPO to fine-tune ESM-IF to design more stable proteins. More comparisons are desired.”

While ProteinDPO is standard DPO [1] applied to maximize the stability of protein sequences, it does not target the diversity aspect of this design problem. In contrast, our paper proposes a new method specifically designed to jointly maximize sequence quality (TM-Score) and diversity.

Additionally, our paper applies our DPO methods to ProteinMPNN via an OpenFold dataset, whereas ProteinDPO applies DPO to ESM-IF via a stability dataset. In our experiments, ESM-IF is generally a stronger base model, making the comparison unfair. Furthermore, the core algorithm presented in the ProteinDPO paper is standard DPO, plus a weighted variant that performs similarly. We report standard DPO results in our experiments, showing our methods outperform it.

The modified DPO algorithm is mostly based on vanilla DPO and is similar to ProteinDPO.

We present two DPO modifications: diversity-regularized DPO (Section 3.1), and reward-weighted DPO (Section 3.2). We consider diversity-regularized DPO to be the more novel contribution. While it is based on standard DPO, we argue that it is a significant contribution due to 1) the novel application of post-training alignment to diversity optimization, and 2) the principled derivation/explanation of the online DPO algorithm.

Firstly, we note that DPO-tuned ProteinMPNN has 7-14% worse diversity compared to vanilla ProteinMPNN in our benchmarks (Tables 1-2). This motivated us to create new DPO methods to directly handle diversity. To the author’s knowledge, we are the first work to propose directly optimizing for diversity via DPO or any post-training alignment methods.

Secondly, recent research in the DPO alignment space ([2], [3], [4]) has begun to outline the effectiveness of online sampling in DPO for increasing downstream rewards. To the author’s knowledge, ours is the first work to produce an online DPO method by a simple derivation from the KL-constrained RL objective. In the computational biology space, ours is also the first work to even consider online DPO for protein work (ProteinDPO and [5] use strictly offline methods, i.e. the train dataset remains static throughout training). The comparison between online and offline DPO methods can be seen in Tables 1 and 2, where DPO with diversity regularization (non-zero alpha) matches the TM-score of offline DPO but with 11-18% improvement in diversity. Furthermore, we provide a principled derivation for the online term, whereas prior work in NLP generally applies online sampling ad-hoc, without formal justification [2], [3]. We also provide an empirically supported theory (see the differential entropy formulation) on why online DPO is so helpful compared to offline DPO.

In Table 1, the DPO fine-tuned ProteinMPNN does not achieve better diversity than vanilla ProteinMPNN significantly.

Compared to vanilla ProteinMPNN, standard DPO finetuning produces a model with 14% worse diversity, but also 8% higher TM-Score (Table 1). With our method (diversity-regularized DPO), we are able to achieve similar diversity as vanilla ProteinMPNN but also maintain that 8% increase in TM-Score. Thus, our DPO method allows us to reach a Pareto improvement over vanilla ProteinMPNN, achieving similar diversity but also better TM-Score. This is a testament to our original goal of jointly optimizing both diversity and TM-Score, rather than having to sacrifice one for the other as standard DPO requires.

In Table 2, even with fine-tuning, ProteinMPNN + DPO is worse than ESM-IF in TM-Score. Therefore, its interesting to fine-tune ESM-IF with the same algorithm.

We agree very much with the reviewer. Our goal is to show that our DPO method is ultimately base-model agnostic and can lift the performance of ESM-IF, PiFold, etc. in addition to ProteinMPNN. Unfortunately, due to compute constraints and severe logistical limitations with running new experiments, we were regretfully unable to finetune ESM-IF. We hope to see this in future research!

Thank you again for your patience with our responses! We hope this clarifies your concerns and look forward to additional discussion.

[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model

[2] Bootstrapping Language Models with DPO Implicit Rewards

[3] Direct Language Model Alignment from Online AI Feedback

[4] Understanding the performance gap between online and offline alignment algorithms

[5] Preference optimization of protein language models as a multi-objective binder design paradigm