Lift Your Molecules: Molecular Graph Generation in Latent Euclidean Space

We thank the reviewer for their valuable feedback. In the following, we address the reviewer's concerns and highlight the changes to the manuscript.

Geometric GNN instead of EDM

As suggested by the reviewer, we replace EDM by the Geometry-Complete Diffusion Model (GCDM) from [1] and combine it with our framework. We used the default hyperparameters from [1] and trained for 140 epochs so far. GCDM has 6.2M parameters, so we compare it with a smaller version of EDM than the one we used in the paper, with 5.7M parameters. We find that GCDM already yields better performance than EDM for a similar model size. This shows that our framework is robust to the change of the 3D generative model, and better 3D performance corresponds to improved 2D generation performance. We add these two models to Table 1 in the manuscript.

Conditional Generation - LogP

Regarding the conditional generation results for LogP in Table 5, the results presented in the paper were obtained by using a single regressor trained to predict LogP and MW of a molecule. We now trained a model to only predict LogP and found that it improved the results. We updated the values in Table 5.

Similarity-Constrained Optimization - Novelty

Regarding the diversity scores from Table 6, we discovered a bug in the way we computed these scores compared to the baselines. For the baseline methods, diversity was only computed for the subset of molecules that were successfully optimized (i.e. non-negative improvement for the LogP task and success for the QED and DRD2 tasks), while we have been computing the diversity over the whole set and assigning a diversity score of 0 for unsuccessful molecules, significantly underestimating the true score. We provide the previous and newly computed values and we update Table 6 in the manuscript accordingly. With this change, our model now achieves the best diversity score for 2 out 4 tasks, while being very close in another task. Similarly to the previous point, we expect that training different regressors for each property will further improve the optimization results and the diversity scores. We updated Table 6 accordingly.

Pros and Cons of RDKit-generated coordinates

To test the effect of the inductive bias introduced by using RDKit's coordinates, we train a new variant of EDM-SyCo on 3D coordinates produced by the Fruchterman-Reingold force-directed algorithm [2], which generates 3D positions solely based on the graph structure without any chemical inductive bias. We refer to it as EDM-SyCo-graph-layout. We find that the RDKit-based model significantly outperforms the new model, highlighting the effectiveness of this chemical inductive bias. This experiment is now in Appendix G.10. However, one drawback is that conformer generation methods would not work for non-molecular graphs such as social networks or knowledge graphs.

Questions:

• The error bars in Table 3 represent standard deviations across 3 different training runs of differnt models, unlike in Table 4 (in table 4 we used the same model due to computational constraints on the larger GuacaMol dataset). We clarify this in the updated manuscript.

References

1. Morehead, A., & Cheng, J. (2023). Geometry-complete diffusion for 3d molecule generation.
2. https://en.wikipedia.org/wiki/Force-directed_graph_drawing

We thank the reviewer for their valuable feedback. In the following, we address the reviewer's concerns and highlight the changes to the manuscript.

Scalability with respect to the molecule size

To check the scalability of our model with respect to the molecule size compared to previous diffusion models, we compare the average runtime required to generate a single molecule using DiGress [1] and our model. DiGress is a discrete diffusion model that operates on 2D graphs. For both models, we generate a batch of 100 molecules of size N, for an increasing N, and report the average runtime to generate one molecule. Results are shown in the following table. We added this experiment to Appendix G.13.

While the runtime for both models scales roughly quadratically with the molecule size, our model is more than one order of magnitude faster than DiGress across all tested sizes. This confirms that working with latent point clouds does not introduce additional computational or memory costs.

In addition, we highlight that the second dataset we used in our experiments, GuacaMol, contains larger molecules than ZINC, with up to 88 atoms per molecule. Results in Table 4 of the manuscript show that also in this setting, our model signifcantly outperforms DiGress (0.79 vs 0.68 FCD scores).

Formally, graph generative models learn a distribution of $\mathcal{O}(N^2)$ random variables due to the quadratic number of edges, where $N$ is the number of nodes in the graph. On the other hand, point cloud generative models need only to learn a distribution of $\mathcal{O}(N)$ random variables as they do not model edge distributions. In our experiments, we use GNNs to process point clouds and treat them as fully connected graphs, increasing the runtime to $\mathcal{O}(N^2)$. As we show in the previous table, this is computationally feasible for medium-sized molecules. For larger ones, the advantage of defining the diffusion process in the 3D latent space is that one could introduce distance cutoffs to define neighbors and limit the number of edges, as e.g. done in [2].

Finally, in Appendix G.9, we discuss a way to accelerate the denoising process up to 5 times, while only marginally decreasing performance, by running a strided sampling schedule as proposed by [3].

References

1. Vignac, C., Krawczuk, I., Siraudin, A., Wang, B., Cevher, V., & Frossard, P. (2022). Digress: Discrete denoising diffusion for graph generation.
2. Corso, G., Stärk, H., Jing, B., Barzilay, R., & Jaakkola, T. (2022). Diffdock: Diffusion steps, twists, and turns for molecular docking.
3. Nichol, A. Q., & Dhariwal, P. (2021, July). Improved denoising diffusion probabilistic models.

Dear Reviewer,

Thank you for raising the important concern about the scalability of our approach with respect to the molecule size. We hope to have addressed your concerns in our rebuttal and look forward to your feedback. We are happy to answer any remaining questions.

Best regards,

Authors

We thank the reviewer for their valuable feedback. In the following, we address the reviewer's concerns and questions and highlight the changes to the manuscript.

EGNN in the decoder

We implement the reviewer’s suggestion by training a new diffusion model on ETKDG-generated coordinates and moving the EGNN to the decoder. Training the diffusion model remains necessary because the used datasets lack public ground-truth coordinates and pretrained 3D diffusion models.

While both model variants achieve similar unconditional generation performance, an MLP-only decoder offers an advantage in scaffold-constrained generation. Given a scaffold that can be reconstructed by the autoencoder, we perform inpainting in latent 3D space to complete the scaffold. Using an EGNN decoder, bond predictions between scaffold atoms depend on added atoms, potentially altering the original 2D scaffold, unlike an MLP-only decoder. We validate this by generating 1000 valid completions for a 6-carbon ring (SMILES: C1CCCCC1, present in 7% of training molecules) and computing the Scaffolding Success Rate (how often the decoded molecules actually contain the scaffold), shown in the table below. We added this experiment to Appendix G.11.

GeoLDM with more bond prediction algorithms

We evaluate GeoLDM paired with the suggested bond-determining algorithms and find that the results are significantly worse than EDM-SyCo across all metrics. This suggests that having a trainable architecture such as our autoencoder is necessary to adapt 3D diffusion models to 2D molecule generation. We include this experiment in Appendix G.12.

Hyperparameter tuning on GuacaMol

We clarify this point in Section 6.2:

Note that due to computational constraints, we use the same hyperparameters from ZINC250K to train our model on GuacaMol

Questions

How does your model handle chirality, since the diffusion model is E(3)-equivariant?

Chirality is a property of the 3D configuration of a molecule, not of its graph structure. Therefore, as a 2D molecule generation method, our model must be invariant to chirality. To satisfy this invariance, we use E(3)-equivariant generative model. We clarify this point in Section 3.

how the normalization of the FCD and KL divergence metrics was performed

We added a new section in Appendix F.8 that provides more details on the evaluation metrics following [1]

[...] The KL divergence $D_{KL,i}$ is computed for each descriptor between the two sets of molecules and aggregated to a final normalized score via $\frac{1}{k}\sum_{i=1}^k \exp(-D_{KL,i})$.

[...] This distance is then normalized via $\exp(-0.2\cdot FCD)$ to lie between 0 and 1.

How is the novelty metric so high?

The high novelty metric arises from the larger molecular sizes in our datasets compared to QM9. The number of possible molecules grows exponentially with size, making it unlikely to sample duplicates or training molecules. As the reviewer pointed out, QM9 is the exact enumeration of organic molecules with up to 9 heavy atoms under predefined constraints [2], while molecules from ZINC250K have more than 23 atoms on average.

can you train only on smaller graphs but generate much larger ones?

To evaluate generalization, we generate molecules larger than every other molecule from the training set and report the validity rate. The better performance on GuacaMol compared to ZINC250K can be explained by the higher diversity of training molecule sizes in GuacaMol. We outline some ideas that can improve the generalization capability:

• instead of modeling point clouds as fully-connected graphs for the diffusion model's EGNN, use distance cutoffs to make the number of neighbors of an atom roughly independent of the total number of atoms, as e.g. done in [3].
• modify the forward diffusion process to avoid that all atoms cluster around the CoM, e.g. by increasing the variance of the transition kernels.

Thank you for the response! These experiments address some of my concerns, and I have raised my score accordingly.

We thank the reviewer for their valuable feedback. In the following, we address the reviewer's concerns and highlight the changes to the manuscript.

Definition of KL and FCD

We added a new section in Appendix F.8 to provide more details on the computation of the KL and FCD metrics following [1]:

KL: The following descriptors are computed for the generated and reference molecules: BertzCT, MolLogP, MolWt, TPSA, NumHAcceptors, NumHDonors, NumRotatableBonds, NumAliphaticRings, NumAromaticRings, and similarity to nearest neighbor with ECFP4 fingerprints. The KL divergence $D_{KL,i}$ is computed for each descriptor between the two sets of molecules and aggregated to a final normalized score via $\frac{1}{k}\sum_{i=1}^k \exp(-D_{KL,i})$.

FCD: The FCD is computed based on the hidden representations of molecules in ChemNet, trained for predicting biological activities, similarly to the FID usually applied to image generative models. Concretely, the means and covariances of the last hidden activations of ChemNet are computed for the reference and generated molecules and the Frechet distance between them is computed. This distance is then normalized via $\exp(-0.2\cdot FCD)$ to lie between 0 and 1.

De novo generation

While de novo generation is not a very useful task in practice, it is usually a prerequisite for more complex and more useful tasks such as similarity-constrained optimization, especially in the context of diffusion models where unconditional models are guided at inference time. We clarified this point in Section 6.2.

Inductive bias

We analyze the inductive bias of our architecture in two steps and perform a new ablation study to highlight it, which we included in Appendix G.10.

• Chemical bias of RDKit-generated coordinates: we train a new variant of EDM-SyCo on 3D coordinates produced by the Fruchterman-Reingold force-directed algorithm [2], which generates 3D positions solely based on the graph structure without any chemical inductive bias. We refer to it as EDM-SyCo-graph-layout.
• Representation bias of the 3D latent embedding: we highlight baselines that operate directly on the graphs, through discrete diffusion (DiGress [3]) or continuous diffusion by adding Gaussian noise to the adjacency matrix (GDSS [4]).

Results are presented in the following table. We see that our base model performs better than EDM-SyCo-graph-layout across all metrics, which in turn outperforms GDSS and Digress on FCD and KL. We are happy to provide any additional results the reviewer wants to see regarding the inductive bias of our architecture.

Similarity-Constrained Optimization

We refer the reviewer to Section F.7 as well as Figures 6 and 7 in the Appendix, where more details and visualizations on this task are provided, which were not presented in the main paper due to space limit. If the reviewer wants to see any additional concrete experiments in this context, we are happy to also include them.

References

1. Brown, N., Fiscato, M., Segler, M. H., & Vaucher, A. C. (2019). GuacaMol: benchmarking models for de novo molecular design.
2. https://en.wikipedia.org/wiki/Force-directed_graph_drawing
3. Vignac, C., Krawczuk, I., Siraudin, A., Wang, B., Cevher, V., & Frossard, P. (2022). Digress: Discrete denoising diffusion for graph generation.
4. Jo, J., Lee, S., & Hwang, S. J. (2022, June). Score-based generative modeling of graphs via the system of stochastic differential equations.