We sincerely appreciate your valuable feedback. We are pleased that you found it logical to apply a beta distribution-based diffusion method to graph modeling and recognized the novelty of our concentration modulation technique! In response to the concerns you raised, we provide detailed answers below:

1. The authors give no clear evidence as to why beta diffusion is effective at modeling the graph structure. The discrete distribution can take many forms. Or why not use BFN to model the distribution of a graph.

Regarding the motivation behind choosing the beta diffusion model over other discrete models, please refer to the second part of our discussion in the “global response”. We are not aware of the application of BFN to graph generations in existing literature and appreciate the reviewer for providing more guidance on this point.

2. In terms of experimental results, if concentration modulation is not included, the effect of beta diffusion is not better than Digress and GruM.

We would like to clarify that, as demonstrated by the results in Tables 2, 3, and 5, our model has already achieved clear improvement in most cases compared to DiGres, even without concentration modulation, particularly in molecule generation. This highlights the power and potential of modeling graph data using our proposed Graph Beta Diffusion (GBD) model.

To further adapt the proposed model to graph generation tasks, we developed a concentration modulation technique that incorporates inductive biases based on the statistical properties of graphs. This novel technique further boosts the performance of GBD across various graph generation tasks. These results not only validate the effectiveness of our concentration modulation technique but also inspire future exploration of Beta diffusion modeling in broader scenarios.

3. Can the ordinary diffusion model also use this idea to control the noise process?

This is an excellent question, and addressing it will help clarify the unique advantages of beta diffusion in graph modeling. To begin, please refer to the first point in our Global Response. Below, we provide additional discussion.

From the perspective of our motivation, the concentration modulation technique in our model plays a crucial role in controlling the mixing process, as described in Section 2.3. Since the initial state of the reverse beta diffusion process is set to 0, important graph structures emerge first (state > 0) under an appropriately chosen concentration parameter. This allows us to predict the topology more accurately and model the graph more efficiently. In contrast, with an uncertain initial state (usually sampled from $N(0, I)$), it remains an open question how Gaussian diffusion processes could effectively describe the concept of "first appearance of important structures." This gap motivates us to continue exploring this technique in future research.

While the concentration modulation technique enhances performance, we emphasize that the successful adaptation of beta diffusion to graph modeling is primarily due to the synergy between the inherent flexibility of beta diffusion and the meticulous design of our GBD framework.

We sincerely appreciate your valuable feedback and thank you for highlighting the contribution of our work!

In response to the concerns you raised, we provide detailed answers below:

1. Details about complexity.

Following the design of Graph Transformer in GruM and DiGree, the overall memory and time complexity of our network is O(n^2) each layer. In addition, we set up the same network configuration in the specific experiments as in DiGress.

Regarding the detailed elucidation about concentration modulation during the sampling stage, please refer to the first part of our discussion in the "global response" and we note that the cost of applying the summary statistics obtained from the training data during sampling is negligible.

2. Lack of detection for networks with other topological properties. BA networks, for example, follow a power-law distribution.

Generally, the BA model primarily focuses on power-law distribution, like degree distribution, and may not capture other important features or attributes. It is worth noting that graph data elements exhibit diverse probabilistic characteristics and it is the most important motivation to adapt beta diffusion to graph modeling, as detailed in the second part of our discussion in the “global response”. Therefore, we model these complex data with advanced diffusion models, providing powerful ability to model the joint distribution of various types of elements within graph data.

3. Details about the sample rate analysis and scalability of this method.

Follow your suggestion, we add the ablation of different sampling rates on Planar as below. From the results in this Table, We can find that the performance of all methods decreases when the number of sampling steps is reduced to 300, but our model is still able to generate high-quality graphs with 75% V.U.N. and have the best V.U.N. when the number of sample steps is greater than 500.

V.U.N (%) on Planar

4. Clarification about graph conditional generation.

Thanks for your insightful question about graph conditional generation! For the graph conditional generation task, we note that the neural network and generative paradigm are the same as in DiGress, so our model can also have the same ability to generate graphs with conditional properties like DiGress does - incorporating the label or properties into the network as a global feature directly.

However, since the focus of this paper is on unconditional graph generation and providing a comprehensive comparison with existing methods, this topic lies beyond the scope of our current work. Moreover, effective conditional graph generation often requires more than simply injecting label conditions; it typically involves various preference optimization techniques, which are distinct research areas worth exploring in their own right.

Thanks for your replies and valuable suggestion about experiments on the BA networks!

Follow your suggestion, we conduct additional experiments and show the effects of our proposed concentration modulation on BA networks. Specifically, we construct two groups of BA-networks with where $n=[20, 100]$ and corresponding $m=[2, 3]$, respectively. $n$ denotes the total number of node per graphs and $m$ denotes the number of edges that each new node will attach to existing nodes. In the tables below, we compare the results of our proposed model and its vanilla vision which denoted as $GBD$ and $GBD_{w/o -cm}$, respectively. The results from these tables demonstrate that our proposed node concentration modulation can provide positive effects on BA networks where the node degrees follow distinct power-law distributions. We will provide details in the revision.

Additionally, we have provided the corresponding visualization of BA-networks ($n=20, m=2$) generated with our model in the revision, please refer to Appendix F.4.

Results of (n=20, m=2)

Results of (n=100, m=3)

Thank you for your thoughtful review and for recognizing the strengths of our work. We are glad to hear that you valued our adaptation of the beta diffusion process to graph-structured data, the diverse evaluation on both synthetic and real-world datasets (including molecular benchmarks), and the clear organization of the paper.

In response to the concerns you raised, we provide detailed answers below:

1. Regarding the unique contribution for our proposed model:

Beyond the adaptation of beta diffusion to graph modeling, we note that several designs have been proposed to improve the performance of modeling graph data in Section 2.3, especially the concentration modulation technique we further elucidated in the first part of our discussion of “global response”. With this novel technique, we are able to improve the performance of GBD in a variety of graph generation tasks, which not only shows the effectiveness of our concentration modulation technique, but also brings inspiration to utilize beta diffusion model in more scenarios in the future.

2. Insufficient background of beta diffusion

Thanks for your valuable suggestion. Following your advice, we will add more technical details about Zhou et al. (2023) in the revision.

3. Regarding the questions about results of experiments, we clarify as follows:

We sincerely appreciate your valuable feedback and careful reading!

For the missing baselines, we note that some state-of-art baselines do not include all the main experiments we have done in their own paper. To make the comparison as comprehensive and fair as possible, we used a wider range of baselines and did more experiments in our submission, demonstrating the strong modeling ability of our model in graph generation tasks.

For the missing metrics of Spec. and V.U.N., we note that measuring the performance of a model with these metrics on simple and small graphs is not an adequate reflection of the models’ modeling capabilities Therefore, to better measure the ability of modeling real-data for graph generative models, it is necessary to evaluate these metrics on graphs with complex structure and large scale.

With these metrics reported in Table 2 for complex graphs, as well as the valid, unique, and novelty for molecule graphs in Appendix C.2, we believe our proposed model is capable of modeling complex graphs and achieves great performance on various datasets.

For the miss standard deviation, we have reported in the following tables and will update Table 2 and Table 3 in the revision.

Planar

SBM

QM9

ZINC250k

4. Regarding the marginal improvements over baselines in some cases:

Thanks for your careful reading! We note that our proposed GBD has significant improvement on all experiments compared to either continuous or discrete diffusion models (GDSS+TF, ConGress and DiGress), demonstrating the GBD’s comprehensive ability of graph modeling. Compared to the state-of-art baselines (GraphARM, SwinGNN, EDGE and GruM), our GBD outperforms these baselines in most cases, and achieves competitive performance compared to GruM. Additionally, we also provide rich visualizations in Appendix F, showing that our GBD can generate high-quality valid graphs both on complex general and molecular graphs.

5. Regarding the connectivity guarantee while modeling graphs.

Our model does not provide the explicit guarantees for modeling the connectivity of graphs, so do other baselines.

Following your advice, we report the connectivity metric evaluated on Planar and SBM as below. From the results in the Table below, we can find that our model can always generate connected graphs for both two complex graphs, demonstrating that our model has the ability to implicitly constrain connectivity very well.

Connectivity

6. Additional comments.

Thanks for your suggestive comments! We will provide more additional background about Zhou et al. (2023) in the revision and improve the visualization, including improving the quality of Figure 2 and adding the additional adjacency matrix of SBM in the revision.

Could please acknowledge and respond to the rebuttal.

We sincerely thank you for your thoughtful and detailed feedback. We are pleased that you found the application of a beta distribution-based diffusion method to graph generation logical, recognized the relevance of our research to current trends, and appreciated both the clarity of our writing and the comprehensive evaluation, including ablations.

1. Regarding your question about alternatives to beta diffusion and what makes beta diffusion more suitable for graph generation:

Thank you for highlighting Dirichlet Diffusion Score Model (Avdeyev et al., 2023) and Dirichlet Flow Matching (Stark et al., 2024). We note that these models primarily focus on generative modeling of categorical data and their applications on biological sequence design. While related, our research aims extend beyond categorical variables. We are developing a diffusion model that captures the joint distribution of both categorical and numerical elements within graph data, which is not the focus of the aforementioned models.

In pursuit of a thorough comparative analysis, we reviewed all papers referencing these models and found only one closely related to our work:

• Floor Eijkelboom, Grigory Bartosh, Christian Andersson Naesseth, Max Welling, and Jan-Willem van de Meent. "Variational Flow Matching for Graph Generation." arXiv preprint arXiv:2406.04843 (2024).

In their study concurrent to ours, Eijkelboom et al. (2024) introduced CatFlow, a flow matching method tailored for categorical data, and evaluated it across two abstract graph generation tasks and two molecular generation tasks. They noted: "Dirichlet Flows have not been evaluated on graph generation tasks. Our preliminary experiments, conducted using the released source code, did not yield satisfactory performance out of the box, and we decided against investing substantial time in optimizing architecture and hyperparameters."

Interestingly, while our study encompasses a broader array of datasets, Eijkelboom et al. (2024) evaluated their method on four datasets that we also used. The results we achieved on these shared datasets, as detailed in the following tables, demonstrate superior performance, effectively underscoring the efficacy of our approach in graph generation tasks.

2. We appreciate the opportunity to provide further insight into the beta diffusion approach, particularly as it represents a recent advancement in the field.

Traditional diffusion and flow-matching methods are generally designed to handle either continuous or discrete data types exclusively. In contrast, beta diffusion utilizes scaled and shifted beta distributions governed by multiplicative diffusion processes. This approach is adept at modeling a broad spectrum of continuous distributions and can effectively approximate discrete distributions as well. By leveraging these unique properties, beta diffusion establishes a robust and adaptable framework that is ideally suited for the diverse data types encountered in graph generation.

3. The reviewer is hoping to see more elucidation of what is gained from using beta diffusion vs standard diffusion for graphs.

Thank you for this insightful question. The advantage of using beta diffusion in the graph context lies in its ability to model both discrete and continuous components.

Unlike standard diffusion models based on Gaussian or categorical distributions that specialize in either continuous or discrete domains, beta diffusion bridges this gap with its flexibility in shape and concentration and its multiplicative diffusion processes, allowing it to better approximate the true data distributions of various types.

While our GBD model does not explicitly regularize motif similarity, its ability to capture the statistical characteristics of graph elements enhances the topological similarity between real and generated graphs, including motifs. This is reflected in high V.U.N. and Scaf scores observed on Planar, SBM and molecular datasets, which are metrics directly related to motif similarity. Additionally, beta diffusion exhibits the "destiny driven" property identified in recent work (GruM), which has been linked to improved graph quality and faster convergence, further demonstrating its advantages in graph generation.

4. Regarding the choice of $\omega$:

We’d like to clarify that we used the default value of $\omega = 0.01$, as it had already yielded state-of-the-art results. However, following your suggestion, we have conducted an ablation study on Planar datasets. Our findings indicate that while performance remains robust across a wide range of $\omega$ values, there is potential for improvement by fine-tuning this parameter. Specifically, as shown in the table below, performance improves from 92.5% to 95% when adjusting $\omega$ from 0.01 to 0.1. Therefore, we recommend tuning $\omega$ if computational resources permit. If resources are limited, maintaining $\omega$ at 0.01 while focusing on other hyperparameters—such as concentration parameters—that are more closely related to the specific characteristics of the graph data, may be advantageous.

V.U.N.(%) on Planar

5. Regarding predicting $E[G_0|G_t]$ in diffusion models:

We clarify that predicting $E[G_0|G_t]$ is standard within the EDM framework. In Gaussian diffusion, commonly used in the DDPM framework, it is also typical to predict the noise, $\epsilon$, rather than $x_0$. As demonstrated in EDM (Karras et al., Elucidating the design space of diffusion based generative models. NeurIPS, 2022), these approaches under Gaussian diffusion are essentially equivalent, with the primary distinction being the method of network preconditioning. Both $x_0$ prediction and $\epsilon$ prediction are widely employed strategies in Gaussian diffusion. However, $x_0$ prediction is more suitable under beta diffusion. This is because the beta distribution is not reparameterizable, and there is no direct analogy for added noise as there is in Gaussian diffusion.

6. Details about concentration modulation.

We choose $\eta$ based on training data statistics during training and use the same prior statistics to define the $\eta$ for sampling. Please refer to the global response for more details. We expect the nodes of the generated graph to be stochastically ordered based on their corresponding order of $\eta$. Since our model is permutation equivariant and takes only the indirect guidance from $\eta$, the predefined order does not affect the final diversity and quality during sampling. Empirically, our model can generate more high-quality and valid graphs with this technique as shown on various experiments and visualization.

Could please acknowledge and respond to the rebuttal.

(2) The motivation of using beta diffusion

• Unlike standard diffusion models that rely on Gaussian or categorical distributions, which specialize in either continuous or discrete domains, beta diffusion bridges this gap with its flexibility in shape and concentration, coupled with its multiplicative diffusion processes. This allows it to model a broad spectrum of continuous distributions and effectively approximate discrete distributions as well. By leveraging these unique properties, beta diffusion establishes a robust and adaptable framework ideally suited for the diverse data types encountered in graph generation.

• Graph data elements exhibit diverse probabilistic characteristics: edges in the adjacency matrix can be binary or discrete, while node attributes can be either discrete or continuous, depending on their specific definitions. Existing diffusion-based graph generative models built upon Gaussian or categorical distributions, each of which has limitations in handling data with both continuous and discrete elements. Gaussian-based diffusion is well-suited for continuous data but lacks efficiency for discrete structures, while categorical-based diffusion specializes in discrete data but struggles with continuous components.

• Beta diffusion bridges this gap with its flexibility in mean and concentration, coupled with its multiplicative diffusion processes. This allows it to model a broad spectrum of continuous distributions and effectively approximate discrete distributions as well. By leveraging these unique properties, beta diffusion establishes a robust and adaptable framework ideally suited for the diverse data types encountered in graph generation.