Out of Many, One: Designing and Scaffolding Proteins at the Scale of the Structural Universe with Genie 2

Question 5 & 6: We now visualize the secondary structure distribution of the PDB (similarly to how we do this for AFDB, using 1,000 randomly selected structures) in Figure 17, which closely resembles that of our filtered AFDB dataset. Indeed, when comparing to AFDB, PDB contains a slightly higher content of stranded structures and this could potentially explain why Chroma, Proteus, and RFDiffusion generate slightly more beta strands since they are trained to capture the PDB data distribution while Genie 2 is trained to capture the AFDB data distribution.

We do note that the PDB does not necessarily reflect the natural distribution of proteins across life. The PDB is highly biased in its distribution towards experimentally-tractable structures, model organisms, and biomedically relevant proteins. We expect the fold distribution of the PDB to look quite different from overall protein structure space. The AFDB may indeed be more representative (given that it covers a large fraction of all sequenced organisms), and in particular, we use a clustered version of the AFDB which at least ensures that all protein folds are roughly equally represented.

Regarding the argument for underrepresented beta sheets and loop elements, we meant that existing models do not model structures with less than 50% secondary structure content (regardless of type). For example, RFDiffusion explicitly filters out structures with less than 50% secondary structure when generating the training dataset, possibly because these structures are more flexible and thus have less well-determined structures. This implies that the self-consistency pipeline might not be a good in silico evaluation pipeline for these flexible structures. This could also explain why Genie 2 performs poorly when normal sampling noise scale (refer to Appendix E.3 for more details). As shown in Figure 2B, Genie 2 is capable of generating structures with low secondary structure contents. However, the flexibility of these structures would render the existing self-consistency metrics ineffective, since there is no one correct solution for these flexible structures.

Question 8: The reason we did not include Twisted Diffusion Sampler as a baseline is due to it being a guidance-based approach, which does not take motif sequence and structure as input to the model but uses guidance to steer the generation process. In contrast, other methods like Genie 2, RFDiffusion, and FrameFlow-Amortized take motif sequence and structure information as model input, putting TDS at a comparative disadvantage. We do not expect TDS to outperform Genie 2 and FrameFlow-Amortized since FrameFlow-Amortized outperforms TDS on motif scaffolding and thus for our comparison, we select both RFDiffusion and FrameFlow-Amortized to provide a fair assessment of Genie 2 against existing SOTA methods.

Question 9: We now include comparisons on unconditional generation performance between Genie and Genie 2 in Appendix E.1.

Question 10: When curating the set of multi-motif scaffolding problems, we went through the literature in search for applications of multi-motif scaffolding; however, existing applications are limited and our dataset comprises effectively all known multi-motif scaffolding problems from the literature. The key reason that our benchmark problems (as well as recent motif scaffolding problems) focus on proteins with at most 200 residues is that for functional protein design, we generally prefer shorter proteins with the target functional motif, mainly because shorter proteins are easier to synthesize and test experimentally and have higher specificity. Practically all successful protein designs are in this length range. Hence, when assessing the performance of Genie 2, we only profile generation of longer proteins (more than 300 residues) on the unconditional generation task and not on motif scaffolding.

Question 11: We are actively developing support for multimer generation, including curation of a larger multimer dataset beyond existing PDB multimer structures and designing reliable and efficient pipelines for assessing multimer quality. We do not expect to be able to complete this work in time for this manuscript but we will integrate it into a future version of Genie.

We would like to thank the reviewer for the constructive and thoughtful feedback. Please see our responses below:

Weakness on Quality and Clarity: Please refer to our responses to questions below.

Weakness on Significance and Originality: We believe that our improved diversity results from the combined effect of training on AFDB and sampling at a lower noise scale. Training on AFDB allows us to leverage a much larger structural space than that of the PDB and results in a better protein diffusion model that can tackle new protein design tasks unaddressed by existing methods. Training a leading protein diffusion model on the whole of the clustered AFDB database, while apparently conceptually simple, in fact requires overcoming substantial engineering challenges as well as making critical design choices. This is supported by the fact that while AFDB has been available since 2021, Genie 2 is the first to learn to model the data distribution of AFDB structures. Some challenges of modeling the data distribution of AFDB structures include structural quality (since they are not experimentally validated), existence of multi-domain structures and the scale of AFDB (over 200 millions structures). Indeed, soon after Genie 2 was preprinted, FoldFlow-2 came out as a contemporary method that attempted to use AFDB, but found that training on AFDB actually hampered model performance. An innovation of this work is the use of low noise-scale sampling in combination with AFDB, which we believe contracts average protein radius and encourages the generation of more compact (and thus more designable) structures (refer to Appendix E.3 for more discussion on low sampling noise scale). By combining training on AFDB with low noise-scale sampling, Genie 2 is able to leverage the structural diversity of AFDB while retaining high quality generation capability, enabling successful utilization of AFDB.

Question 1: We agree that due to undertraining, the conclusion that higher conditioning task ratios lead to better model performance is speculative and we will update our manuscript to reflect this uncertainty. Our hypothesis is that a higher conditioning task ratio may help speed up training, but model performance may well be unaffected upon convergence since the underlying data distribution is the same.

Question 2: The primary metric that we are considering is diversity, which computes the percentage of unique AND designable clusters (given a fixed number of samples). This better reflects model performance than designability alone, which can be inflated when the model repeatedly generates the same designable structures (reflecting overfitting or mode collapse).

Question 3 & 7: We now include an additional ablation study on the effect of sampling noise scale in Appendix E.3. We also provide further details on how sampling noise scale is set for other methods in Appendix A.5. We find sampling noise scale to be a model-specific hyperparameter, and for fair comparisons with other methods, we use the pretrained weights and default hyperparameters provided in their GitHub repositories, which should provide the best generative performance. Upon revising the manuscript for the final version, we will include the default temperatures used for each model in the main text.

Question 4: Yes it is a reasonable suggestion to balance experimental and synthetic structures during training (this is done by AlphaFold3 for example). The primary reason that we only use AFDB filtered clusters is that these clusters are more structurally diverse and are expected to cover existing structures within the PDB. Training on both PDB and AFDB structures could introduce bias towards the intersection of two sets of structures and would introduce another hyperparameter to tune. It is something we will consider for future work however.

Regarding the bias on synthetic dataset, it is true that by training on filtered AFDB structures, the model is learning to model the data distribution of AFDB. However, from the perspective of structural diversity, AFDB is a super set of the PDB monomers and Genie 2’s expanded model capacity appears able to capture both (Appendix E.2). We believe that this is (indirectly) demonstrated by the fact that Genie 2 solves problems that methods trained on the PDB alone are unable to solve, suggesting that Genie 2 can tap into a larger structural space.

We would like to thank the reviewer for the constructive and thoughtful feedback. Please see our responses below:

Weakness 1: The decision to fix maximum sequence length to 256 residues during training was made based on available computational resources. The major computational bottleneck in the current Genie 2 architecture is the space complexity of the triangular multiplicative update layers, which require $O(N^3)$ memory (where N is the maximum sequence length). We are actively working on reducing the memory usage of Genie 2 through integration of deepspeed optimization strategies to support training on larger proteins.

Since the initial submission of the manuscript, we have also trained Genie 2 for additional epochs and observed substantially improved performance on the generation of longer proteins. The results and analysis are in Appendix E.4, which shows superior performance over all competing methods in generating diverse long proteins.

Given that Genie 2 is currently the SOTA method (with several additional and very recent baselines in the revised Appendix), we would welcome a clarification from the reviewer as to why the method only merits a rating of 3 (“reject, not good enough”), given the importance of the protein design problem and the substantially improved performance of Genie 2 (our SOTA gains are not merely incremental relative to existing methods.) An expanded explanation from the reviewer would help us address any outstanding issues now or in a future version of the manuscript, and would be much appreciated as it can lead us to actionable changes.

Weakness 2: When assessing multi-motif scaffolding performance, we aimed to pursue realistic design tasks known from the biological literature with proven utility, instead of artificial tasks that may inflate (or deflate) the performance of the evaluated methods. Because of the difficult and cutting-edge nature of multi-motif scaffolding, current known examples are limited to the 6 multi-motif scaffolding tasks curated in our evaluation dataset. This problem is too new to enable more systematic benchmarking, although we agree it would be preferred. We hope that future design tasks will enable retrospective benchmarking of Genie 2, but we do not believe that the nascence of the problem should preclude publication of Genie 2, given its demonstrated performance on the 6 tasks and on conventional motif scaffolding. This is further bolstered by the fact that other methods surveyed, for instance Chroma, fail on all 6 multi-motif scaffolding tasks, which suggests that these tasks are providing a signal regarding the performance of protein design methods.

We did consider the possibility of constructing artificial multi-motif scaffolding tasks using the PROSITE motif database [1] as a source of hypothetical motifs. However, PROSITE is a sequence database whose motifs are identified by scanning a predefined set of rules and patterns against sequences. This does not guarantee the formation of functional motifs due to various reasons; for example, multiple segments of a single motif can be structurally dispersed and unlikely to carry out an actual function. Hence, due to the lack of structural motif databases, we decide to curate a set of multi-motif scaffolding tasks based on recent literature.

Weakness 3: We now compare the performance of Genie 2 with RFDiffusion on multi-motif scaffolding. Since RFDiffusion does not support flexible motif positions and orientations, we randomly sample positions and orientations among motifs as suggested by the reviewer. We report the new results in Appendix G.4. While RFDiffusion is able to generate viable solutions for some multi-motif scaffolding problems, Genie 2 generally produces more diverse solutions. In addition, Genie 2 is much more sample efficient since it is able to reason over the relative positions and orientations between motifs, while RFDiffusion relies on randomness to obtain viable multi-motif configurations. This implies that as the number of motifs increases, RFDiffusion will be less likely to succeed since the search space for viable multi-motif configurations expands exponentially. This is supported by the observation that Genie 2 solves problem 1PRW_four, which involves four EF-hand $\text{Ca}^{2+}$ binding motifs, while RFDiffusion fails.

Reference

[1] Sigrist, Christian JA, Edouard De Castro, Lorenzo Cerutti, Béatrice A. Cuche, Nicolas Hulo, Alan Bridge, Lydie Bougueleret, and Ioannis Xenarios. "New and continuing developments at PROSITE." Nucleic acids research 41, no. D1 (2012): D344-D347.

We would like to thank the reviewer for the constructive and thoughtful feedback. Please see our responses below:

Weakness 1: We agree with the reviewer that functional protein design extends beyond motif scaffolding, although the design of proteins with multiple conformations remains extremely challenging and anecdotal. We also agree that the ultimate determination of successful protein design is through experimental validation. However, given our focus in this manuscript on the core computational methodology and given the machine learning focus of ICLR, we consider experimental validation to be out of scope, and this has been the standard for protein design papers at machine learning conferences. We do believe that our in silico evaluation pipeline, which follows the norms and standards of the field as a whole, can point to meaningful improvements in design capabilities, because of the use of independent protein structure prediction methods to assess the self-consistency of designs. Furthermore, even in an experimental setting, in silico evaluation can help reduce the number of potential targets and boost experimental success rates. This was recently demonstrated in binder design [1], where in silico evaluation considerably improved success rates.

Weakness 2: Thanks for bringing this oversight to our attention. We now provide a comparison between Genie and Genie 2 in Appendix E.1, demonstrating significant improvements from Genie to Genie 2. While we agree that it is unfair to compare Genie and Genie 2 due to differences in training dataset, model architecture, and model complexity, given limited computational resources, we decided to prioritize evaluating important hyperparameter choices in Genie 2 (e.g., number of diffusion steps and sampling noise scale) over retraining Genie in a controlled setting. This allows us to better understand various potential tradeoffs in the Genie 2 architecture, as well as explore the full capabilities of Genie 2. A full ablation study between Genie and Genie 2 would have consumed all our available computing resources and precluded our exploration of the hyperparameter space of Genie 2.

Weakness 3: We now provide a detailed analysis on the effect of different sampling noise scales in Appendix E.3.

Weakness 4 & 5: Thanks for this suggestion as it has led us to some new insights. We now provide further ablation studies and discussions on the self-consistency pipeline in Appendix D that we hope address all the reviewer’s concerns.

Question 1: We provide further results and analysis on model performance across different training checkpoints in Appendix E.4, where we discuss the tradeoff in checkpoint selection. In general for checkpoint selection, we assess the 30-epoch, 40-epoch, and 50-epoch checkpointed models on unconditional generation and motif scaffolding tasks and select the best performing checkpoint. It is worth noting that the performance differences between these three checkpoints are minimal.

Question 2: We now compare to Chroma on single-motif scaffolding in Appendix F.6. Genie 2 significantly outperforms Chroma on single-motif scaffolding, solving more problems while generating more diverse solutions at the same time. In addition, we attempt to use Chroma for multi-motif scaffolding by composing multiple substructure conditioners. However, Chroma fails to solve any multi-motif scaffolding problem.

Question 3: We now provide an additional ablation study for single-motif scaffolding when not using motif sequence information as input in Appendix F.7. Since the functionality of a motif generally relies on side chain conformations and thus is dependent on motif sequence, we maintain our motif scaffolding settings, which is to design a protein containing matching sequence and structure to the target motif. Hence, we omit only motif sequence from the inputs to Genie 2 but maintain the motif residue identities during inverse folding (when using ProteinMPNN). Under these settings, Genie 2 remains capable of providing viable solutions for the majority of motif scaffolding problems, but we do observe a drop in its performance, suggesting that Genie 2 is internalizing sequence information into its latent representations to improve protein design.

Reference

[1] Bennett, Nathaniel R., Brian Coventry, Inna Goreshnik, Buwei Huang, Aza Allen, Dionne Vafeados, Ying Po Peng et al. "Improving de novo protein binder design with deep learning." Nature Communications 14, no. 1 (2023): 2625.

We appreciate the time and effort you have put in reviewing our work. As the discussion period is coming to close (Dec. 2nd), we would like to ask if our replies and added results have helped to address your questions and concerns and if there is any further clarification we could provide.

Weakness 4: Thank you for your suggestions and we now provide further case studies on the two multi-motif scaffolding problems that Genie 2 fails on in Appendix G.3. In addition, we provide a controlled experiment on scaffolding an increasing number of EF-hand $Ca^{2+}$-binding motifs in Appendix G.3.3. As the number of motifs increases, multi-motif scaffolding performance drops since it is hard to find a viable solution that satisfies all motif constraints; however, as the number of motifs increases, the number of unique successes increases (bounded by the number of successes), possibly because the increase in sequence length provides a larger search space for solutions. Furthermore, we now compare Genie 2’s performance on multi-motif scaffolding with RFDiffusion in Appendix G.4, demonstrating that Genie 2 has an edge on multi-motif scaffolding.

Weakness 5: We now provide further ablation studies on the number of diffusion steps in Appendix B.2. As shown in Table 4, while using 1,000 diffusion steps does achieve the best performance on both unconditional generation and motif scaffolding, Genie 2’s performance with fewer diffusion steps is comparable. Genie 2 thus allows for fast sampling with minimal loss of generative capabilities.

Question 1: Using the set of Genie 2-generated structures (curated for analysis in Table 1), we have now visualized the distribution of scRMSD as a function of the percentage of helices (Appendix E.6). We observe that while scRMSD slightly decreases with higher helical fraction, the majority of the generated structures have relatively low scRMSD regardless of secondary structure composition. This, combined with the secondary structure distribution shown in Figure 2A, suggests that while Genie 2 has a tendency to generate more helical structures, its generation quality of either helical or stranded structures is similar.

Question 2: We believe the reviewer meant maximum sequence length instead of batch size for this question, but please correct us if we are mistaken. We set the maximum sequence length to 256 due to computational limitations. As Genie 2 utilizes triangular multiplicative update layers, it has $O(N^3)$ space complexity where N is the maximum sequence length. However, despite being trained only on proteins with up to 256 residues, Genie 2 is capable of generating proteins beyond 256 residues in length. As discussed in Weakness 1, compared to existing methods, Genie 2 is capable of generating more diverse structures, even for longer proteins. We would like to also note that for practical applications, the majority of protein design applications focus on proteins shorter than 200 residues.

We would like to thank the reviewer for the constructive and thoughtful feedback. Please see our responses below:

Weakness 1: As mentioned in Section 4.1, designability could be sometimes misleading since it does not account for structural diversity. Relating to Figure 3B, while Proteus achieves higher designability than Genie 2 on longer proteins, its diversity is similar or even lower than Genie 2 as shown in Figure 3C. This suggests that Proteus repeatedly generates the same or highly similar structures.

Since our initial submission, we trained Genie 2 for additional epochs, which has led to improved performance, particularly on longer proteins. We include further results and analysis in Appendix E.4. Genie 2 now achieves superior performance over all competing methods in generating diverse long proteins. Thus, we believe that Genie 2 outperforms Proteus in generative performance, even in the generation of long proteins.

Nonetheless, we will revise our wording in the manuscript to more precisely describe how Genie 2 outperforms existing methods.

Weakness 2: For unconditional generation, we now provide further comparisons with MultiFlow and CarbonNovo in Appendix E.7. The code for FoldFlow-2 is unavailable, precluding direct comparison with Genie 2. Compared to MultiFlow and CarbonNovo, Genie 2 achieves comparable or better designability while significantly outperforming them on diversity and novelty. For single-motif scaffolding, we additionally compare to Chroma and provide the results and analysis in Appendix F.6. Genie 2 significantly outperforms Chroma on motif scaffolding, solving more problems while generating more diverse solutions at the same time.

Weakness 3: We respectfully disagree on this point. While our methodological advances may seem incremental, in the sense that we use known components such as triangular multiplicative updates employed by other methods such as Proteus and FoldFlow2, our architecture is novel, i.e., how we use and combine those components is different. In particular, the Genie 2 architecture is the current state-of-the-art against all methods that have been suggested by the reviewers and that we previously compared against, on the key design metrics of designability, novelty, and diversity. We believe that achieving SOTA results merits consideration.

[We add the following additional point, copied from our response to Reviewer 4r8p, regarding the additional significance and originality of training on the AFDB database] We believe that our improved diversity results from the combined effect of training on AFDB and sampling at a lower noise scale. Training on AFDB allows us to leverage a much larger structural space than that of the PDB and results in a better protein diffusion model that can tackle new protein design tasks unaddressed by existing methods. Training a leading protein diffusion model on the whole of the clustered AFDB database, while apparently conceptually simple, in fact requires overcoming substantial engineering challenges as well as making critical design choices. This is supported by the fact that while AFDB has been available since 2021, Genie 2 is the first to learn to model the data distribution of AFDB structures. Some challenges of modeling the data distribution of AFDB structures include structural quality (since they are not experimentally validated), existence of multi-domain structures and the scale of AFDB (over 200 millions structures). Indeed, soon after Genie 2 was preprinted, FoldFlow-2 came out as a contemporary method that attempted to use AFDB, but found that training on AFDB actually hampered model performance. An innovation of this work is the use of low noise-scale sampling in combination with AFDB, which we believe contracts average protein radius and encourages the generation of more compact (and thus more designable) structures (refer to Appendix E.3 for more discussion on low sampling noise scale). By combining training on AFDB with low noise-scale sampling, Genie 2 is able to leverage the structural diversity of AFDB while retaining high quality generation capability, enabling successful utilization of AFDB.

Thank you for your further summary. However, my main concerns remain.

1. The method's novelty seems very incremental

   The proposed method appears to be a simple combination of existing components that have already been proven useful in backbone generation, and thus it does not offer significant novelty. Specifically:

   a. Model architecture: Genie2's denoiser network mainly includes an Evoformer-like architecture and an Invariant Point Attention (IPA) module. It shares a very similar architecture with CarbonNovo's structure module (see Figure 1 and Algorithm 1 in their paper). Proteus also incorporates an IPA and an Evoformer-like architecture called the Graph Triangle Block (Figure 2 in their paper). FoldFlow++ also leverages a similar architecture (Figure 1 in their paper). I also note that these previous works have been peer-reviewed in ICML 2024 or NeurIPS 2024, not just in their pre-print versions.

   b. Low noise scale sampling: Your clarification in your response is not accurate. Low noise scale sampling and low-temperature sampling essentially have the same meaning, both in motivation and technique. In both cases, the motivation is to trade diversity for quality. In your method, quality is measured in terms of designability.

   c. Utilizing the AlphaFold Database (AFDB): Training the model by combining existing techniques that have been proven useful for backbone generation on more data (from AFDB) seems like a very trivial idea to achieve SOTA.

2. Concern on overstatement remains

   The claim of "architectural innovations" in the abstract is very questionable, as your method shares a very similar architecture with AlphaFold2. Moreover, these architectural components have already been proven effective in backbone generation, as stated in my comment 1.a.