Dual Flows with Contrastive Guidance for Generating Highly Designable Proteins

• The explanation regarding the statement “… avoid non-designable regions by gently steering it during sampling” could be clearer. It appears that the LQ model’s influence extends to all regions uniformly, rather than specifically targeting non-designable areas. It would be helpful if the authors could provide more detail on how the LQ model selectively identifies and influences non-designable regions during the sampling process.

• The LQ model is fine-tuned on synthetic data, which likely includes a significant portion of designable proteins. This raises a concern that the vector field of the LQ model may not always exert a negative influence. Additionally, Figure 3 suggests that the LQ model’s impact may not always be detrimental, which could warrant further discussion.

1. How useful is generating long monomer proteins?
2. Could you compare with Proteus [1]? It is published in same conference with MultiFlow; ICML 2024. From what I know, Proteus generates length-400 and 500 proteins much better than the proposed method. (~0.9 for length-400 and ~0.75 for length-500).
3. I am not persuaded that the performance increase entirely comes from contrastive guidance for two reasons:
   1. If you are using two models for your sampling, the fair comparison would be generating 2x more proteins and evaluate top half for baselines. How baselines perform in this setting?
   2. The tradeoff between designability and diversity is not clearly explained. For example, you can just boost up your designablity 2x by just generating similar proteins 2x.
      1. The proposed pairwise TM score doesn’t really give the sense of diversity. Can you explain the reasoning behind de-duplication and randomly selecting 50 proteins? It is really hard to interpret the numbers given all these procedures.
      2. I believe more common way of computing diversity is number of clusters. (as in MultiFlow [2], Proteus, Genie2 [3], and so on) Could you follow this evaluation scheme and provide the number of clusters?
4. Please explain the error bar and +- in the Figure and Tables.

[1] Wang, Chentong, et al. "Proteus: exploring protein structure generation for enhanced designability and efficiency." (2024)

[2] Campbell, Andrew, et al. "Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design." (2024)

[3] Lin, Yeqing, et al. "Out of Many, One: Designing and Scaffolding Proteins at the Scale of the Structural Universe with Genie 2." (2024)

1. Novelty: The idea to combine predictions of a high-quality and a low-quality model to improve predictions has been proposed in Karras et al. (2024) under the name of autoguidance. Except for the minor difference that the authors here use a flow matching loss instead of a score matching loss , the main idea is very similar (although the application is new). A discussion of the relation of autoguidance to the proposed method would benefit the paper. Especially the theoretical and experimental analysis as to why it works could be helpful since this part is not covered deeply in this submission. One interesting aspect in Karras et al is for example the notion that CFG eliminates outliers and samples more from the base of the respective distributions, whereas autoguidance does not drop any significant part of the data distribution. Comparing these considerations with the discussion about the effect of contrastive guidance here could strengthen the theoretical motivation of this paper.
2. Performance of baselines: The performance of the baseline methods in Table 1 and Table 2 is worse than expected from other publications. This is unexpected, especially since recent benchmarking efforts like ScaffoldLab and ProteinBench also show significantly better numbers for the baseline methods than suggested here in this paper. For example, in the setting of length 400/500 proteins and 8 ProteinMPNN samples, ProteinBench (whose benchmark is publicly reproducible) reports scRMSD scores of 2.12/xx for FrameFlow while this paper here reports 6.12. It reports them with an error margin of 4.94, which in theory includes that other measurement. The question is how strong the claim of improved performance can be made in that setting then, especially given that the pre-finetuned baseline Multiflow has a score of 2.71 with an error of 3.65. The authors are encouraged to check their numbers against these publicly available benchmarks and comment on the discrepancy and the validity of their conclusions based on these high error margins. Possible action items to strengthen the claims of the authors here could be
   • Provide a detailed comparison table showing their results side-by-side with the ProteinBench and ScaffoldLab results for the same methods and metrics.
   • Directly compare their evaluation protocol to that used in ProteinBench and ScaffoldLab, highlighting any differences.
   • If possible, re-run evaluations using the exact protocols from these public benchmarks.
   • Discuss how the large error margins impact the statistical significance of the performance claims.
3. Selection of evaluation lengths: The authors only show evaluations for lengths 400 and 500, not for shorter lengths. Since most of the baseline methods were only trained on proteins of length up to 256 or sometimes 384 residues, but the authors train their finetuning model exclusively on samples of length 400, it is not clear whether the claimed improvement comes from their contrastive guidance framework or just from the additional training on long sequences and structures. Additional results for shorter lengths would strengthen the claims made by the authors here.
4. Performance tradeoffs: The authors show that their model outperforms the previous state-of-the-art methods on designability (however with a high error bar), but also show that they are worse on diversity than other SOTA methods and even worse than their baseline model Multiflow. Since the designability-diversity trade-off of these methods can be tuned with factors like sampling temperature, it does not seem clear to me that this method actually improves the overall "Pareto frontier" of this designability-diversity trade-off. The authors are encouraged to present designability and diversity numbers at different temperatures for the different methods to strengthen their SOTA claims. If possible, include experiments or analyses that attempt to match the diversity of baseline methods while comparing designability and the other way around.

1. One key issue with Multiflow is that its sequence/structure co-design does not outperform a simpler approach: designing the backbone and then performing inverse folding. For any substantial improvement, it must be demonstrated that co-design offers advantages over this backbone-first approach (+ inverse folding). Currently, the manuscript lacks a comparison between co-design and PMPNN, which is essential. Additionally, a major advantage of Multiflow is its capacity to handle multiple modalities, such as inverse folding and forward folding (though performance in forward folding is limited). The manuscript does not adequately discuss these other modalities. I would consider raising the score if these additional results were presented.
2. Designability is a problematic metric to optimize (e.g., ESMFold has numerous false negatives, and its failure to refold a structure does not necessarily imply that the structure is incorrect). This reliance on the Flow-HQ dataset, which only includes structures foldable by ESMFold, may therefore be flawed. Designability also has a bias toward alpha helices (partially because folding algorithms handle them better). Once designability reaches a certain level, other metrics become more important. Predictably, the current method suffers from reduced diversity and novelty, which should be discussed further. Additionally, the metric of novelty in this paper is somewhat inadequate. It should measure similarity not only to PDB data but also to synthetic datasets, examining how closely generated samples resemble the training distribution. A secondary structure analysis should also be included, especially as Figure 4 primarily shows alpha helices.
3. Note that Multiflow already performed distillation, which substantially improved designability. I anticipate further improvements could be achieved with careful tuning of distillation procedures. Consequently, I do not find the empirical results particularly compelling; for instance, in Figure 3 (right), at epoch 3, simple fine-tuning on designable samples reaches almost the same performance as contrastive guidance.