Q1. For the objective of unconditional 3D molecular generation, I noticed the absence of results from "Ours-GSSL". Including these would be crucial to validate the effectiveness of GCL. Could there be a more comprehensive validation of GCL's efficacy in Section 4?

In the context of conditional generation based on (invariant) quantum properties, it's essential to evaluate both ID and OOD. Yet, the "Ours-GSSL" performance doesn’t exhibit a noteworthy distinction compared to "ours", "Random", and similar methods. In some instances, it even underperforms. This might cast a shadow on the robustness of conclusions articulated in Section 4.2, especially point (v).

Answer.

• We provide additional results in Tables 15 & 16, on applying GSSL for unconditional generation and conditional generation with less training data.
• Based on our experiments, we observed that the benefit of GSSL is mainly on:
  • OOD scenarios when the GSSL prior is aligned with the condition semantics, as we stated in Section 4.2;
  • Unconditional or ID scenarios when training data are insufficient. This is consistent with the role of GSSL in discriminative modeling (https://arxiv.org/abs/2010.13902).
• For the OOD scenarios, though the improvement does not appear for all datasets, we know when it is improved: Specifically, we can measure the semantics-alignment by computing the homogeneity ratio as detailed in Appendix E.2. This indicates a future potential to design semantics-specific GSSL tasks for generative modeling.

Q2. Regarding the unconditional 3D molecule generation, the 'molsta' metric, which gauges molecular integrity, seems paramount. From the data presented, your findings exhibit significant variances when juxtaposed against the previously introduced 3D latent diffusion model, GeoLDM. Additionally, the AtomSta and MolSta metrics for the Drug dataset appear to be omitted.

Answer.

• Our method is different from GeoLDM. We summarize the difference across different methods in Table 10, mainly lying in two perspectives: (i) latent dimension and (ii) the modeling of topological and geometric features.
  • Latent dimension (LD). Assuming the feature dimension of data is $N \times (D+3)$. GeoLDM, diffusing at the latent space of nodes is of LD $N \times (D’+3)$. Guided by our theory in Section 3.2, we chase a lower dimensional latent space by compressing the node factor $N$, that our method at the other end of the spectrum of the minimum LD $D’$.
  • Modeling of topological and geometric features. EDM, GeoLDM and our one-shot model jointly model the topological and geometric features for all $N$ nodes. However, our one-shot model cannot manage to reconstruct the feature per node for its overly low LD, that the node factor $N$ is compressed. Guided by our theory in Section 3.2, we develop the cascaded model, achieving satisfactory reconstruction performance while maintaining symmetry and sufficiently low LD.
• The above aspects lead to different results: GeoLDM performs better in generating geometrically stable molecules, and our method performs better in generating topologically valid molecules. Both of them are important in real-world applications.
• The evaluation is standardized as in EDM and GeoLDM, and we do report AtomSta for Drugs. Following EDM and GeoLDM, the MolSta metric is less informative to be evaluated in Drugs, since the ground-truth molecules in Drugs have 86.5% atom-level and nearly 0% molecule-level stability, due to which contain larger and more complex structures, creating errors during bond type prediction based on pair-wise atom types and distances.

The author has addressed all my concerns. I will increase my score accordingly.

Q1. The novelty of the work is somewhat limited. It takes existing 3D molecule genration setup with diffusion models and transforms it into latent diffusion, which is a known concept in general and some latent 3D models do exist (e.g. GEOLDM), using mostly off the shelf architectures.

Answer.

• Our work is not a direct extension of latent diffusion into 3D graphs. We aim at generating 3D graphs of symmetry-group equivariance through diffusion. In other words, the quality of the generated graphs of topology and geometry is invariant / equivariant to a set of transformations (permutation group and Lie group E(3) of rotations and translations in our study). This aim has deep roots in 3D molecular generation as demonstrated in our numerical experiments for desired quantum properties (invariant to E(3) perturbation of the 3D molecules) or desired protein binding (equivariant to E(3) perturbation of the 3D molecules); and it has broader application scopes for 3D graphs of symmetry-group equivariance in general.
• Contribution (i). We provided theoretical analysis (Section 3.2) on what space should diffusion generative models operate on for generating 3D graphs of non-Euclidean structure and group-symmetry equivariance. We addressed group-symmetry equivariance while deriving model performance bounds. Specifically, Proposition 1 gave the upper bound of the diffusion model performances (as measured in total variation distance) operating in the original 3D graph domain ($D’$-dimensional); whereas Proposition 2 gave the upper bound in the latent space ($D’’$-dimensional). The latter led to a better upper bound of the diffusion model performances if reconstruction errors are low and the latent dimension is low, which motivates our approach of latent diffusion.
• Contribution (ii). In pursuing the latent space as guided by our theoretical, motivational analyses, we constructed a 3D graph autoencoder of the following properties: (1) permutation and E(3) invariant and (2) sufficiently low reconstruction errors despite low latent dimension (these two present nontrivial trade-offs), which is under-explored. We accomplished this by combining 2D graph autoencoder (for topology features) and point cloud autoencoder (for geometry features given topologies) in a cascade manner (Section 3.1).
• Contribution (iii). To improve the out-of-domain (OOD) generation, we further incorporate graph self-supervised learning into learning the 3D graph autoencoder and the resulting latent space for diffusion (Section 3.1). Our self-supervised latent diffusion has been largely unexplored for 3D graphs of group-symmetry equivariance. Our numerical experiments demonstrated its effectiveness in improving OOD performances, which is a critical barrier for designing molecules with much improved properties or new properties.
• Contribution (iv). We further extend our work into conditional generation of 3D graphs, which stems from practical applications and presents a new challenge. Namely the new challenge is to maintain conditional equivariance, that is, the generated 3D graphs should be E(3)-invariant to the property condition and E(3)-equivariant to the protein binder’s structure condition. We addressed this challenge by introducing conditional equivariant geometric autoencoder in our latent diffusion pipeline (Section 3.3).
• Difference from GeoLDM. We summarize the difference across different methods in Table 10, mainly lying in two perspectives: (i) latent dimension and (ii) the modeling of topological and geometric features.
  • Latent dimension (LD). Assuming the feature dimension of data is $N \times (D+3)$. At one end of the spectrum, EDM diffusing in the original space is of the maximum LD $N \times (D+3)$. The followup work, GeoLDM, diffusing in the latent space of nodes is of LD $N \times (D’+3)$. Guided by our theory in Section 3.2, we chase an even lower dimensional latent space by compressing the node factor $N$, that our method at the other end of the spectrum of the minimum LD $D’$.
  • Modeling of topological and geometric features. EDM, GeoLDM and our implemented one-shot model jointly model the topological and geometric features for all $N$ nodes. However, our one-shot model cannot manage to reconstruct the feature per node for its overly low latent dimension, that the node factor $N$ is compressed. Guided by our theory in Section 3.2, we develop the cascaded model, achieving satisfactory reconstruction performance while maintaining symmetry and sufficiently low LD.

Thank you for your answers. I will increase my score accordingly. Although I'm still lukewarm about the significance of the contribution.

Q1. What's the difference between a one-shot AE and a Cascaded AE? The paper claims that "a one-shot AE embeds and reconstructs molecule data, evaluating both structure topology and geometry simultaneously." How does a Cascaded AE differ? Are topology and geometry independent from each other in a Cascaded AE?

Answer.

• The reviewer understands correctly that the topological and geometric AEs encode and sequentially decode, independent from each other in the cascaded version, and in the one-shot version the two AEs jointly encode/decode. Please also refer to Appendices B.1 & B.2 for more information.

Q2. I would recommend adding an algorithm section to improve clarity. Based on my current understanding, a graph is initially encoded by an encoder in the latent space, then it undergoes diffusion in the latent space, and finally, it is transformed back to the graph space. Please correct me if I'm mistaken. Could you also clarify which diffusion model is used for the diffusion process in the latent space?

Answer.

• The reviewer understands our pipeline correctly.
• Following the suggestion, we add the detailed algorithm sections (Algorithms 1 & 2) to describe our latent 3D graph diffusion, beyond the overview Figures 2 & 3. They describe the processes of (conditional) in/equivariant latent encoding, latent diffusion, and sampling.
• The adopted diffusion model is DDPM (https://arxiv.org/abs/2006.11239).

Q3. The paper asserts that "a latent space should possess (i) low dimensionality, (ii) low reconstruction error, and (iii) preserve group symmetry." How do you ensure that the autoencoder can meet these objectives? Have you conducted any analysis in this regard?

Answer.

• The factors of (i) low dimensionality and (iii) preserving group symmetry are by design in the AE architecture, which is guaranteed.
• The factor of (ii) low reconstruction error is achieved by building the 3D graph AE in a cascaded manner, as one of our contributions described in Section 3.1. The numerical comparison of different AEs is shown in Tables 1 & 11.

Q1. According to this paper, I did not clearly catch how the authors project the 3D molecules into the 2D latent space. The technical details are missing. It naively mentioned that the authors borrow the architectures from previous works and such simple comment is not self-contained, in my opinion.

Answer.

• We are sincerely grateful to the reviewer’s feedback. We are committed to improving our clarity, and also would like to point out some aspects missed by the reviewers.
• We add the detailed algorithm sections (Algorithms 1 & 2) to describe our latent 3D graph diffusion, beyond the overview Figures 2 & 3. They describe the processes of (conditional) in/equivariant latent encoding, latent diffusion, and sampling.
• More specifically, the details of invariant latent encoding are further presented in Section 3.1.
• The details of conditional in/equivariant latent encoding are presented in Section 3.3. Regarding latent encoding architectures, we supplement more detailed descriptions of architecture design in Appendix B.2 & C.1
• The details of diffusion model are presented Section 2.

Q2.1. However, let's assume that this issue is okay with me. Then, the following question is ... why the latent space is typically important to the graph generation? I know that the several recent studies utilize the latent space for the representation of the diffusion-based generative models. Nonetheless, I could not find any theoretical analysis behind the bridge between 'the necessity of the latent space' and 'the properties of the 3D graph representation, typically for the 3D molecules'. For me, this is naive extension of latent diffusion models into graph representation.

Answer.

• Our work is not a naive extension of latent diffusion into 3D graphs. We aim to generate 3D graphs of symmetry-group equivariance through diffusion. In other words, the quality of the generated graphs of topology and geometry is invariant / equivariant to a set of transformations (permutation group and Lie group E(3) of rotations and translations in our study). This aim has deep roots in 3D molecular generation as demonstrated in our numerical experiments for desired quantum properties (invariant to E(3) perturbation of the 3D molecules) or desired protein binding (equivariant to E(3) perturbation of the 3D molecules); and it has broader application scopes for 3D graphs of symmetry-group equivariance in general.
• Thus, directly related to the reviewer’s question, one contribution of our work is to demonstrate the necessity of building the 3D graph latent space to achieve a potential generation quality. We provided theoretical analysis (Section 3.2) on what space should diffusion generative models operate on for generating 3D graphs of non-Euclidean structure and group-symmetry equivariance. We addressed group-symmetry equivariance while deriving model performance bounds.
  • Specifically, Proposition 1 gave the upper bound of the diffusion model performances (as measured in total variation distance) operating in the original 3D graph domain ($D’$-dimensional);
  • whereas Proposition 2 gave the upper bound in the latent space ($D’’$-dimensional). The latter led to a better upper bound of the diffusion model performances if reconstruction errors are low and the latent dimension is low, which motivates our approach of latent diffusion.
• Another contribution is to illustrate how to construct such 3D graph latent space (autoencoder), guided by our theoretical, motivational analyses, of the following properties: (1) permutation and E(3) invariant and (2) sufficiently low reconstruction errors despite low latent dimension (these two present nontrivial trade-offs), which is under-explored. We accomplished this by combining 2D graph autoencoder (for topology features) and point cloud autoencoder (for geometry features given topologies) in a cascade manner (Section 3.1).

Q2.2. While the authors describe full of equations with comments, the concept itself is highly straightforward. For instance in the manuscript, the authors said that 'Proposition 1. Performance bound of 3D graph diffusion is related to feature dimensionality'. In my opinion, simply if we increase the network capacity by adding more layers or increasing the channel-length with lots of training data, the diffusion models surely have the performance gain. Honestly, I cannot catch the authors' intension from this propositions.

Answer.

• We are sorry for the confusion. The feature dimensionality in Proposition 1 refers to the feature dimensionality of data, which does not relate to the network capacity. We revised the comment for better clarity. Please also see our last answer for the intention behind Proposition 1.

Dear Reviewer kaPc,

We sincerely thank you again for your time and effort to review our paper and read our response. Here are our further clarifications to your questions:

• We respectfully disagree the (1) and (2) points of lacking technical novelty and lacking proof. We will illustrate this below.
• For (1) and (2): The inequation "3D Graph Diffusion Performance <= Latent Space Reconstruction Quality + Symmetry Preservation * Data Dimensionality" is the informal expression of proposition 2, which is proved in Appendix A.2. More specifically in proposition 2 (Eq. (3)):
  • 3D Graph Diffusion Performance denotes the left-hand-side of the inequation, measuring the statistical distance between learned distribution and data distribution;
  • Latent Space Reconstruction Quality denotes the first term of the right-hand-side of the inequation, measuring the reconstruction error of the forward and backward mappings (i.e. reconstruction error of AE);
  • Symmetry Preservation denotes the first multiplier of the second term of the right-hand-side of the inequation, measuring how AE preserves symmetry affecting the data-processing factor (see Appendix A.1 for details);
  • Data Dimensionality denotes the second multiplier of the second term of the right-hand-side of the inequation, proportional to the dimension of data.

• We respectfully disagree for the point (3) of lacking novelty. Our innovations lie in four aspects, which are detailed in previous responses and briefly summarized below,
  • Contribution (i) is exactly the technical rationale as we responded (Sec. 3.2);
  • Contribution (ii) is also guided by the analysis, of how to build a qualified 3D graph AE (Sec. 3.1);
  • Contribution (iii) is the extension of our latent diffusion pipeline to the in/equivariant conditional generation setting (Sec. 3.3). This is an entirely new exploration in the field;
  • Contribution (vi) is the exploration of graph self-supervised learning for graph generative models, to improve the generalizability of graph generative AI (Sec. 3.1).

We completely respect the dispute here, and also hope that our clarification will assist the reviewer in addressing the confusion more effectively.

I thank the authors for their endeavor in preparing the rebuttal. Through the rebuttal, I can further understand the exact meaning and insights that the authors proposed. However, I am not that satisfied with several issues: (1) lack of technical contributions, (2) overclaim in "The plain language equation"(Q3), and (3) naive extension of latent diffusion into a graph structure. For some issues, the authors provide their opinions. However, it looks trivial for me. Also, motivation itself is not that easy to understand.

Let me keep my score as "5: marginally below the acceptance threshold". Though I am not that positive with this paper, if the other reviewers think this is okay, let me follow their opinions. However, the manuscript needs lots of modification and needs to deal with the exact problem and novelty. Though the authors updated the algorithm 1, I could hardly find clear insight from this paper.

Q1. The authors trained the topological AE and the geometric AE separately with different constrains. I understand the difficulty in training them in one shot, but I'm wondering the influences from each of the AE, e.g., would it be possible information about the topology is lost while training using the geometric AE? If so, how much is lost/kept?

Answer.

• In the cascaded model, the geometric AE still preserves topology information, and the topological AE does not have access to geometry information.
• Specifically, the model (i) performs autoencoding on topological features, and then (ii) given the topological features as inputs, performs autoencoding on geometric features.
• The information loss in the topological AE is minor to our pipeline, as quantified with the reconstruction performance shown in Table 1 and Appendix B.2.

Q2. While GSSL improves the results on OOD, it seems it can worsen the results on ID. Can the authors provide more insights on this?

Answer.

• We provide additional results in Tables 15 & 16, on applying GSSL for unconditional generation and conditional generation with less training data.
• Based on our experiments, we observed that the benefit of GSSL is mainly on:
  • OOD scenarios when the GSSL prior is aligned with the condition semantics, as we stated in Section 4.2;
  • Unconditional or ID scenarios when training data are insufficient. This is consistent with the role of GSSL in discriminative modeling (https://arxiv.org/abs/2010.13902).
• For the OOD scenarios, though the improvement does not appear for all datasets, we know when it is improved: Specifically, we can measure the semantics-alignment by computing the homogeneity ratio as detailed in Appendix E.2. This indicates a future potential to design semantics-specific GSSL tasks for generative modeling.

Q3. Considering one utility of the model is for generating new drugs, model interpretability could be important. In the latent space, would it be possible for the authors to provide some visualization on the learned features and if available, colored by some topological and geometric features of the graph? If there are certain patterns there then it may be useful to help better understand what features are mostly kept in the latent space and enhance the interpretability of the model.

Answer.

• We provided the latent embedding visualization in Figures 11 & 12.
• Specifically, we perform t-SNE (https://jmlr.org/papers/v9/vandermaaten08a.html) on topological and geometric embeddings, and annotate them with four 2D and six 3D property values of molecules.
• Our visualization shows information of 2D properties is more preserved in topological embeddings, that data with similar properties tends to cluster. In addition, information of 3D properties is more preserved in geometric embeddings.
• The observation is also confirmed by quantitative evaluation, by computing the silhouette score of the latent embeddings w.r.t. properties.

Q1. Limited generalization capability in comparison. How to improve generalization capability?

Answer.

• We provide additional results in Tables 15 & 16, on applying GSSL for unconditional generation and conditional generation with less training data.
• Based on our experiments, we observed that the benefit of GSSL is mainly on:
  • OOD scenarios when the GSSL prior is aligned with the condition semantics, as we stated in Section 4.2;
  • Unconditional or ID scenarios when training data are insufficient. This is consistent with the role of GSSL in discriminative modeling (https://arxiv.org/abs/2010.13902).
• Our results indicate a future potential to design semantics-specific GSSL tasks for generative modeling.