Forked Diffusion for Conditional Graph Generation

Dear reviewer, thanks for raising thoughtful questions and valid points that help us to write better work. We address your points as follows.

W1: The 3D information is incorporated into our experiment, as outlined in Section 4.2. Our architecture is derived from Jodo Huang et al. 2023 [4], specifically adapted to support the forking diffusion mechanism. Detailed information about the architecture is provided in Appendix B.3. We can expand this section if you believe this is necessary.

For the experiment detailed in Section 4.1, we follow the setup of GDSS [4] (Jo et al 2022), utilizing only 2D information. Further specifics on this setup can be found in Appendix B.2.

W2: We focus on property-conditional generation. We take into account all the best performing previous work considering the mentioned task that appeared into top conferences [1,2,3].

Our goal is demonstrating the effectiveness of the learning mechanism of forked diffusion for this specific task. As a result, we follow the setup of established works on property conditional generation of molecules [1,2,3] .

W3: Our method differs from GDSS as we are learning a joint dynamics between properties and structures. GDSS does not model properties. Differently from GDSS, our model is specifically designed for conditional tasks.

W4: Our modeling approach adopts the assumption of independence among child properties in the forking process, a choice made for effective formulation. Whether chemical properties of molecules are considered independent or dependent depends on the specific characteristics under examination. While certain chemical properties may exhibit relative independence, others manifest interdependence influenced by the underlying molecular structure. Leveraging this understanding as an inductive bias, we integrate it into our forked diffusion process. This incorporation provides a more nuanced control over the joint space of properties and molecules, allowing for a refined representation that captures both independent and interdependent relationships in the data.

W5: we will update W6: We have fixed this. W7: We define the dimension that are specific to each experiment in the Appendices B. The variable $Y_s$ represent adjacency matrix and node features, while $y_i$ is a scalar representing a graph property (QED, plogp, SAS) in case of molecules.

Q1: We added 3 baselines: U-bounds, L-bounds, #Atoms, following [1,2,3]. The upper bound ’U-bound’ baseline randomly shuffles the property labels of molecules and reports their MAE. The ‘#Atoms’ baseline predicts molecular properties based only on the number of atoms in molecules. Lower bound (L-bound) is the best performance bound from QM9.

Digress comparison is not appropriate as its conditional generation task, because

1. Digress does not compare to any previous works on conditional generation, such as EDM [1], EEGSDE [2], Jodo [3], all these methods provide comprehensive experimental details, and we compare with them.
2. Digress does custom dataset processing, differently from the other benchmarks [1,2,3].

In our first experiment (Section 4.1) we follow the setup of GDSS [4], in Section 4.2 we follow the setup from EDM [1], EEGSDE[3], Jodo[3].

In addition, Digress codes presents reproducibility issues for the conditional generation, as mentioned by the authors on github: "Warning: The paper experiments were run with an old version of the code. We have incorporated the changes into the public version to create this branch, but we have not tested thoroughly yet. Please tell us if you find any bugs." The problem is confirmed by various issues raised on github https://github.com/cvignac/DiGress/issues/68.

This said, we can still try if the reviewer think necessary, and we added digress to related work.

Q2: Could you elaborate please? According to our understanding, Figure 2 presents valid molecular structures.

Please let us know whether we answered your points fairly, please consider to raise the score, in case we have, or let us know the issues that need to be fixed to improve the score and to improve the work.

Best regards.

References

[1] Hoogeboom et al, "Equivariant diffusion for molecule generation in 3d", ICML 2022.

[2] Bao et al, "Equivariant energy-guided sde for inverse molecular design", ICLR 2023.

[3] H Huang "Learning Joint 2D & 3D Diffusion Models for Complete Molecule Generation" 2023

[4] Jo et al, "Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations" ICML 2022

Dear reviewer, thanks for helping us improving our work. We address your points as follows.

W1.1: The parent and child variables in each experiment vary, typically with the structure variable representing the graph and the child variables representing associated properties. Appendices B2, B3, and B4 provide detailed explanations for these variables.

W1.2: $S_{\phi_i, t}$ serves as an approximation for the coupling between the $i^{th}$ child variable (or property) and the parent variable (graph). This influence on the reverse Stochastic Differential Equation (SDE) process enhances control over the conditional generation, allowing for a more nuanced and precise generation process.

W1.3: The training objective is based on score-matching but incorporates the forked diffusion settings given by propositions 1 and 2, which are novel results that we obtained via formal derivations in Appendix A.1 and A.2.

W2.1: A singular parent diffusion process is linked to a primary variable, such as the graph structure, while multiple child diffusion processes are dedicated to dependent variables, such as graph properties or labels. The parent process orchestrates the co-evolution of its child processes, as mathematically depicted by the system of equations governing the forward Stochastic Differential Equation (SDE) process in Eq. 2. Here, $y_{s}$ represents the parent variable (graph structure), and $y_{i}$ represents the dependent variable, i.e., properties. This influence persists in the reverse SDE, as illustrated in Eq. 4.

W2.2: We adopt the assumption of independence among child properties as a modeling choice to formulate the forking process effectively. The consideration of chemical properties of molecules as either independent or dependent is contingent on the specific characteristics being examined. While certain chemical properties can be relatively independent, others exhibit interdependence, influenced by the underlying molecular structure. Leveraging this inductive bias, we incorporate this into our forked diffusion process to achieve a more nuanced control over the joint space of properties and molecules.

W2.3: $S_{\phi_i, t}$ serves as an approximation for the coupling between the $i^{th}$ child variable (or property) and the parent variable (graph). See the answer at point W1.2.

W4: Could you let us know what kind of ablation study you require ? Is it like using all (n out of n) properties as children vs 1 out of n properties etc?

W5: We agree, will move such important details in the main paper.

W6: We agree, will move the unconditional experiment into appendix, as our method is mostly focused on conditional generation.

W7: Thanks for pointing out those relevant works. We will add into related work section.

Q1: We generate unique noise samples for each child process to ensure the validity of the conditional assumption. The conditional information for each child process is derived from the diffusion function, which, in turn, is conditioned on the specified structure.

Q2: Thanks for the question. In fact, our implementations for are using transformers. With a careful read of the GDSS work [1] one can see that they are employing attention mechanisms. In fact, the architecture for processing graphs in GDSS is the graph multi-head attention (Baek et al.,2021) [2] . Our method in experiment 4.1 is built using similar architectures. For the experiment in 4.2 we compare to Jodo [3], which also uses Transformers, and we inherit a similar architecture for the our forked implementations for Section 4.2.

If the concerns you raised have been satisfactorily resolved, we kindly request you to consider raising the score. Your feedback is valuable to us, and we want to ensure your satisfaction. If there are any remaining issues or if you require further clarification, please do not hesitate to let us know. Best regards.

References

[1] Jo et al, "Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations" ICML 2022

[2] Baek et al, "Accurate learning of graph representations with graph multiset pooling" ICLR 2021

[3] H Huang "Learning Joint 2D & 3D Diffusion Models for Complete Molecule Generation" 2023

Thanks for your useful and detailed review that improves our work. We address your points as follows:

W1: We introduce an innovative method for refining guidance within diffusion models to achieve superior controllable generation. Our approach adopts a unique perspective by concurrently modeling the variable and its properties, leading to an aligned latent space. Empirically, our method outperforms naive conditional diffusion models guided by either classifier-free or classifier-based methods [5]. Furthermore, we establish a robust mathematical framework using Stochastic Differential Equations (SDEs) to derive both the forward and backward forking diffusion processes.

W2: Thank you for the reference. We have added Digress in Appendix C (extended related works), and will further refine, to bring it in related works. We put this reference appropriately in our paper.

W3: Thanks, we are in the process of developing the large scale experiment on Guacamol. However, we note that our work focuses on the task of property conditional generation, that has been published into ICML [4] and ICLR [5]. We compare to such works, following their dataset setup.

W4: Thank you for providing the pertinent references for our research. We have incorporated these citations in appendix C. It's important to highlight that MolHF [3] does not possess a property conditional generation feature, thereby hindering a direct comparison with models that exhibit this capability.

While the MiCAM work [2] is related (included in the Expanded Related Works section in appendix C), it does not evaluate on Mean Absolute Error (MAE) like the literature we are following for property-conditional generation (e.g., [4,5]). In MiCAM [2], the task is goal-directed generation, making it not directly comparable with any diffusion approach considered in [4,5].

Q1: Thanks for pointing our this issue. We are running the ZINC experiment and will complete that evaluation.

Q2: The Uniqueness results were equal to 1 for all methods, we will complete the table. Thanks for the help in the details.

Final remark: We thank the reviewer for the pointers on improving the paper. If we have addressed the questions, please consider increasing the score, or let us know what other points are still missing/ unclear/ necessary for improving this work!

Best regards.

References

[1] Vignac, Clement, et al., "DiGress: Discrete Denoising Diffusion for Graph Generation." The Eleventh International Conference on Learning Representations, 2023.

[2] Geng, Zijie, et al., "De Novo Molecular Generation via Connection-Aware Motif Mining." The Eleventh International Conference on Learning Representations, 2023.

[3] Zhu, Yiheng, et al., "MolHF: A Hierarchical Normalizing Flow for Molecular Graph Generation." International Joint Conference on Artificial Intelligence, 2023.

[4] Hoogeboom et al, "Equivariant diffusion for molecule generation in 3d", ICML 2022.

[5] Bao et al, "Equivariant energy-guided sde for inverse molecular design", ICLR 2023.

Thanks for your thoughtful review which is helping to improve our paper, please see our replies below:

W1:

The novelty of our approach is given by the following point: We present an unexplored approach to characterize the propagation of conditional information, or guidance, within diffusion models, aiming for enhanced controllable generation. Our method introduces a novel perspective by jointly modeling the variable and its properties, resulting in an aligned latent space. Empirically, our approach outperforms naive conditional diffusion models, guided by classifier-free or classifier-based methods. Additionally, we establish a robust mathematical framework utilizing Stochastic Differential Equations (SDEs) to derive both the forward and backward forking diffusion processes.

W2:

Our objective is to capture the joint dynamics between the graph (parent variable) and its associated properties (child variables) through our forked diffusion processes, both in the forward and reverse directions. In the inference phase, the reverse Stochastic Differential Equation (SDE) responsible for graph generation is influenced by the properties and their reverse process dynamics (refer to Eq. 5). This nuanced interplay allows for superior control over the generation process, showcasing exceptional performance compared to conventional methods relying on classifier-free or classifier-based guidance. It's worth noting that, like many molecule generation and property prediction methods in the Deep Learning literature [1,2], usually population-level MAE is used as a metric to quantify the performance of the generated molecules via the model which may result in some discrepancy towards the alignment. We follow the setups from such previous works [1,2], which have been published in top conferences (ICML,ICLR) and highly cited.

W3: In the context of generic graph generation, we extracted graph properties from the data using the NetworkX library and benchmarked our approach against GDSS [3], one of the most recent state-of-the-art methods in this domain. Notably, our method demonstrates superior performance compared to existing alternatives. While these properties may not be as inherently intuitive as molecular ones, they serve as a testament to the efficacy of our approach, particularly on synthetic datasets.

W4: We agree that the unconditional tasks for our model did not show a strong result. Our method is mostly suitable for controlled generation, and will move this experiment into the appendix.

W5: Yes, we will fix the typo.

Q1: In Appendix A.2, we elucidate the reverse Stochastic Differential Equation (SDE) system governing our forked diffusion process. The derivation relies on the theoretical application of both forward and backward Kolmogorov equations to establish the reverse SDE for the joint system encompassing the graph and its associated properties.

We are confident that our response effectively addresses your concerns. If we have successfully addressed your questions, we kindly ask you to consider raising your score. Alternatively, if there are areas where we can further improve to better meet your expectations, please let us know. Your feedback is highly valued, and we appreciate your cooperation. Thank you.

References:

[1] Hoogeboom et al, "Equivariant diffusion for molecule generation in 3d", ICML 2022.

[2] Bao et al, "Equivariant energy-guided sde for inverse molecular design", ICLR 2023.

[3] Jo et al, "Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations" ICML 2022