We thank reviewer S3eE for their time and review of our work. We appreciate the acknowledgement of the method’s quality and performance. We believe we have addressed all weaknesses and questions. If the reviewer agrees we would very much appreciate a score increase. If not, we are happy to continue discussing any remaining weaknesses and questions. Below we address the reviewer’s comments and concerns.

Violation of the anonymity policy…

We are unsure what violation the reviewer is referring to. We did not change the style file. It is common practice to highlight table numbers. There are no instructions or commands in the style guide regarding reference highlighting. Furthermore, we are unsure of how either of these would violate anonymity or result in unfair practices.

The paper proposes to sample the atomic coordinates. However, it is not clear how to guarantee that the sampled coordinates can form a possible structure on a higher level in practice. The constraint between the two levels is missing in the training process.

We are unsure what this question is asking and what “form a possible structure on a higher level in practice” means. We use FrameFlow to sample protein structures conditioned on latents from another latent diffusion model. FrameFlow is already known to sample real proteins. We show empirically by evaluating against prior works that our approach samples realistic, diverse, and novel protein structures.

Other important property such as SE3 is not discussed and addressed in the method.

We have added line 188 that addresses the equivariant properties. SE3 equivariance is addressed by using FrameFlow which was previously shown to already be SE3 equivariant. This is not a contribution of our work and thus we refer to [1] for more details. Latent diffusion does not require addressing equivariance since we encode structures with ProteinMPNN which is SE3 invariant.

Important theoretical proof is missing. It looks like the paper intuitively leverages the reverse process of the diffusion model on the sequence. The original diffusion models such as DDPM or DDIM conduct the forward and the reverse process on the same level, which makes solving the Markov process tracable. However, it is not proven it still works when applied to two different representations.

That is incorrect. Please see figure 1. We propose to use a latent diffusion model and a structure diffusion model to sample protein backbone structures. Latent diffusion follows [2] and structure diffusion follows FrameFlow [1] which are both theoretically justified. We do not perform diffusion on the sequence and diffusion is always performed on the save “level” (we are unsure about this terminology).

Quality of experiment. The paper only introduces only one dataset in the experiment. It is not conclusive. Please conduct the experiment on all benchmarks in [1]. [1] ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design

Our experiments follow previous works on protein backbone generation [1,3,4]. We use the same AFDB dataset as [3,4]. Using other datasets would make it unfair to compare our method to other baselines. The ProteinGym dataset is for sequence design and has no relevance to structure. Therefore, we are unsure why the reviewer is asking us to compare with ProteinGym.

Hierarchical Protein generation is already proposed by existing works, which may limit the contribution of this work. The author may want to integrate more baselines such as...

Out of the listed works, they are either not used for protein generation or are not hierarchical in the way the reviewer claims.

• Interaction-based Retrieval-augmented Diffusion Models for Protein-specific 3D Molecule Generation. This work is for small molecule generation. Our work is for protein generation.
• Multi-level Protein Structure Pre-training via Prompt Learning. This work is on pre-training and representation learning. Not for protein generation.
• Protein Conformation Generation via Force-Guided SE(3) Diffusion Models. This work focuses on a different task of conformation generation. They do not apply latent diffusion and do not do hierarchical generation.
• Towards Joint Sequence-Structure Generation of Nucleic Acid and Protein Complexes with SE(3)-Discrete Diffusion. This work does joint sequence and structure generation which is not hierarchical. Furthermore, they are focused on nucleic acid or complex generation. A different task than ours which would be unfair to compare against.

We have to the best of our knowledge listed the known hierarchical and/or latent diffusion models in our related works. Furthermore, many of the papers listed are from ICML 2024 which according to the ICLR guidelines are concurrent works. We ask the reviewer to only suggest possible related works that follow the guidelines.

Thanks for the rebuttal. Some of my concerns have not been addressed and new concerns have been observed.

W2: When you use FrameFlow to sample real protein structures. how do you guarantee the newly generated protein is still a real protein? If it is a real protein, what is the point of protein generation (i.e., why do we need to generate an existing real protein)?

W3: [1] discuss SE3 with a general diffusion model. While the author claims the proposed work is a hierarchical framework, SE3 on multiple levels should be discussed.

W4: I understand [3,4] use one dataset. But it does not mean [3,4] are high-quality papers. The experiment is not conclusive with one dataset. This can not changed no matter how many papers using AFDB.

W5: There is no paper that can directly be used as baselines without any changes. All these references can be easily introduced as the baselines. Note that the target of adding more baselines is used to improve the experiment quality, not SOTA a specific task.

We thank reviewer 6bhQ for their time and review of our work. We appreciate the acknowledgement of the method’s promise and novelty. We notice the review is relatively short with few concrete action items. We believe we have addressed all weaknesses and questions. If the reviewer agrees we would very much appreciate a score increase. If not, we are happy to continue discussing any remaining weaknesses and questions. Below we address the reviewer’s comments and concerns.

while the fact seems to be that latent diffusion does not work well on structural generation

We respectfully disagree with this statement. Our results show latent diffusion is competitive with state-of-the-art SDMs and improves in efficiency (GENIE2 taking 2.3 min while LSD taking 0.3 min to sample). We argue an approach does not need to beat state-of-the-art if it is novel enough and competitive. Structure diffusion models have been actively developed over the past 3 years [1] whereas latent diffusion models have not seen the same adoption. Yet, latent diffusion models are the de facto approach in image and video generation [2, 3]. This necessitates research to understand why and if there are benefits with latent diffusion for protein backbone generation.

Note the only other open sourced (continuous) latent diffusion approach is LatentDiff upon which we improve considerably. Our contribution is to present a latent diffusion approach that will broaden the diversity of machine learning techniques to protein structure generation.

I would reckon the authors to explore their methods on more specific applications to find better evidence for reviewers to support the submission, eg language-conditioned protein design (to excavate the advantage of LDMs), loop design for antibodies or motif-scaffolding for enzymes.

We believe these applications are a follow-up work. Unconditional generation is the first problem to address when developing a new generative model. For example, FrameFlow, FoldFlow, GENIE all started with unconditional backbone generation [4, 5, 6] before moving towards concrete applications [7, 8, 9]. We are planning to pursue all-atom generation for specific protein design tasks as a follow-up (as stated in our discussion). We aimed to focus this paper on method development; in particular, finding the right hyperparameters and architecture to work for latent diffusion. With our manuscript already at 20+ pages (including app.), more tasks would overload the content for a single conference submission.

The authors may avoid calling their method LSD, as it causes negative associations.

We may change the name for camera ready but not for discussion to avoid further confusion.

References

[1] Yim, Jason, et al. "Diffusion models in protein structure and docking." Wiley Interdisciplinary Reviews: Computational Molecular Science 14.2 (2024): e1711.

[2] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Brooks, Tim, et al. "Video generation models as world simulators." 2024-03-03]. https://openai. com/research/video-generation-modelsas-world-simulators (2024).

[4] Yim, Jason, et al. "Fast protein backbone generation with SE (3) flow matching." arXiv preprint arXiv:2310.05297 (2023).

[5] Bose, Avishek Joey, et al. "Se (3)-stochastic flow matching for protein backbone generation." arXiv preprint arXiv:2310.02391 (2023).

[6] Lin, Yeqing, and Mohammed AlQuraishi. "Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds." arXiv preprint arXiv:2301.12485 (2023).

[7] Yim, Jason, et al. "Improved motif-scaffolding with se (3) flow matching." arXiv preprint arXiv:2401.04082 (2024).

[8] Huguet, Guillaume, et al. "Sequence-Augmented SE (3)-Flow Matching For Conditional Protein Backbone Generation." arXiv preprint arXiv:2405.20313 (2024).

[9] Lin, Yeqing, et al. "Out of Many, One: Designing and Scaffolding Proteins at the Scale of the Structural Universe with Genie 2." arXiv preprint arXiv:2405.15489 (2024).

We thank reviewer Bnxy for their time and review of our work. We appreciate the acknowledgement of the paper’s clarity and SOTA results with a novel model. Below we address the reviewer’s comments and concerns. We believe we have addressed all weaknesses and questions. If the reviewer agrees we would very much appreciate a score increase. If not, we are happy to continue discussing any remaining weaknesses and questions.

About the model design and corresponding ablation study: The current ablation study is more about hyperparameter search, missing support for the method design, for example, why do we need the contact map. One ablation is a pure latent diffusion model without contact map (similar to GeoLDM).

In our initial submission, we performed a method design ablation in Table 6 of using DiT vs. DiT with RoPE vs. vanilla transformer as the latent diffusion architecture. As motivated in our introduction, the contact map is a coarse representation of protein structures that complements the structure diffusion in FrameFlow. Without a contact map, FrameFlow is unable to perform well on the AFDB dataset as shown in Table 2.

We performed more ablations of the method design as requested. Using our best hyperparameter settings, we trained the autoencoder without contact maps where only the latents were provided into the FrameFlow architecture. In addition, we investigated the need for two stage autoencoder and FrameFlow training. Each set-up was trained for an equal number of epochs then evaluated in how well FrameFlow could reconstruct encoded structures.

Specifically, for a structure $x$ in the validation set, we sampled latents $z \sim p_\phi(x)$ then sampled a structure using FrameFlow $\hat{x} \sim FrameFlow(z)$. We report the reconstruction RMSD as the average RMSD over the validation set $1/N \sum_{i=1}^N RMSD(x_i, \hat{x}_i)$ where $N$ is the validation set size. The results are in Table 7 and reproduced below where the reconstruction RMSD shows the two stage training with contact map loss leads to the lowest reconstruction RMSD. This shows both components are necessary for FrameFlow to learn how to decode the latents. We have included this discussion in Appendix C.1.

Lastly, the reviewer points out GeoLDM as another option. There are many ways to formulate latent diffusion and we emphasize that our approach is just one. Our approach is the first approach to be competitive with state-of-the-art SDMs.

About the task: this paper only focuses on backbone generation, can it be extended to all-atom generation?

Yes! In line 534 of the diffusion, we explicitly mention extending to all-atom generation. This introduces new challenges such as how to deal with ligands and side chains that we are interested to explore as a follow-up.

About baseline methods: AlphaFold3 also has a diffusion step. How to compare this method with AlphaFold3?

While AlphaFold3 is a diffusion model, it is not comparable since AlphaFold3 is a structure prediction method where a structure is generated conditioned on a sequence. Our approach is a sequence-agnostic structure generation method.

About the training: This method includes several training stages and is not trained end-to-end, which may limit the final performance. Can it be trained end-to-end? If so, what are the results?

Our previous comment already addresses joint autoencoder and FrameFlow training which we found to be suboptimal. We mention end-to-end autoencoder and latent diffusion training in our initial submission, line 187, “While end-to-end training of all models is possible, we found this to be unstable and difficult to optimize. We instead use a training procedure inspired by Rombach et al. (2022) where the autoencoder is frozen during latent diffusion training.” Our end-to-end training results were very poor. We found near 0 designability and the loss to be unstable. As mentioned, this is consistent with state-of-the-art latent diffusion models in images that are trained in multiple stages rather than end-to-end [1].

We thank reviewer NGTU for their time and review of our work. We appreciate the acknowledgment of the paper’s clarity and insightful evaluations. Below we address the reviewer’s comments and concerns. We believe we have addressed all weaknesses and questions. If the reviewer agrees we would very much appreciate a score increase. If not, we are happy to continue discussing any remaining weaknesses and questions.

The motivation is a bit unclear to me, as there are several approaches to get rid of equivariance issue or to develop SE(3)-equivariant models. For instance, FoldingDiff uses internal coordinates, and FrameFlow’s main architecture is SE(3)-equivariant. Intuitively, I don’t see a significant advantage to applying diffusion models on contact maps.

To clarify the problem and motivation, we have rewritten the introduction with this as the first sentences.

“A challenge across diffusion models for protein backbone generation has been scaling to large datasets: ideally benefiting from improved diversity and generalization, but this empirically results in unwanted biases from low quality protein structures (Huguet et al., 2024). In this work, we aim to develop a diffusion model that scales to the AlphaFold DataBase (AFDB) (Varadi et al., 2022) with the ability to control for desired properties.”

Removing equivariance is not our goal but rather to improve FrameFlow performance with contact maps. We empirically demonstrate the scaling issue in section 5.2 where a FrameFlow model trained on the large AFDB dataset performs very poorly (Figure 2). We improve upon FrameFlow by surpassing its performance in Table 2 (i.e. 23% compared to 94% with LSD).

FoldingDiff is another possible approach. However, based on their results in [1], we are unable to conclude that it can scale to large datasets. To reiterate, our approach is just one novel approach to overcome shortcomings in SE(3) equivariant models such as FrameFlow.

Finally, latent diffusion has not been adequately explored in protein generation as much as it has in other domains such as computer vision [2]. We emphasize that our approach is the first latent diffusion model to be competitive with structure diffusion models.

I'm unclear on certain aspects of the model design, such as why only node-wise features from ProteinMPNN are used for the latent representation

We aim to avoid the latents growing $O(L^2)$ where $L$ is the length of the protein. By only using the node features, we keep the latents $O(L)$. Furthermore, we found the ProteinMPNN autoencoder to work sufficiently well with 0.99 ROCAUC and 0.99 PRAUC in reconstructing contact maps when only the node-level latents were used.

And why FrameFlow was chosen for recovering atomic coordinates, given that there are other models for structure prediction based on contact maps.

Our method is not for structure prediction. Note structure prediction takes in a sequence and predicts the folded structure. Our method is for structure generation without a sequence. We chose to use FrameFlow since its architecture is essentially AlphaFold2’s structure module that naturally takes in the contact map as input. See FrameDiff’s architecture in Figure 2 [3] that FrameFlow uses. Other networks could be used as well as mentioned in line 188.

It would be better to apply evaluation metrics to natural samples in order to reveal any biases in the evaluation pipeline.

Our evaluation procedure follows a long line of existing works, i.e. [3,4,5,6], many of which we include as baselines and are published at machine learning conferences. By natural samples, we assume the reviewer is referring to the training dataset, AFDB in our case. Prior works [6] have found AFDB is highly diverse but contains low designable protein structures. We quantified this in Table 9 where AFDB has 12% designability with high diversity, 65% of examples belong in a unique cluster. As stated in our motivation, we wish to benefit from the increased diversity while keeping high designability – as done in prior works.

Additional analysis on how the models perform across proteins of varying sizes, classes, or folds would also be beneficial.

We are unsure what the reviewer is asking for since our method in its present form is structure generation model. It is not a prediction model that we can use to evaluate across different protein categories.

We have added figure 10 showing designability across length. We see that generally longer lengths lead to worse designability which has been reported in other works [3, 4, 5, 6]. We acknowledge that adding a capability for fold or class conditioning is an interesting future direction for LSD.