ClavaDDPM: Multi-relational Data Synthesis with Cluster-guided Diffusion Models

A-W1 Metric Details:

Thank you for your suggestion. We agree that expanding on the details about long-range dependencies in Section 4.2 and mapping them to the example hierarchy from Figure 1 would greatly enhance the clarity. We will make this change in the revised version.

$\quad$

A-W2 Limitations and Future Works:

Thank you for identifying this issue, and we apologize for the confusion.To respect NeurIPS page limit, we submitted a shortened version of our paper. However, after the submission deadline, we realized that the limitations and future work that we referred to were unfortunately part of what was cut . Here is the original second paragraph of the conclusion, we will make sure to add this to the main paper in the revised version:

"We focused on foreign key constraints in this work, and made the assumption that child rows are conditionally independent given corresponding parent rows. This brings three natural follow-up research directions: i) extension to the scenarios where this prior information is not available and these relationships need to be discovered first[3], ii) further relaxing the assumptions, and iii) inspecting multi-relational data synthesis with other integrity constraints (e.g, denial constraints[4], general assertions for business rules).

Furthermore, we evaluated ClavaDDPM's privacy with the common (in tabular data literature) DCR metric. Nonetheless, we think it is worthwhile to: i) evaluate the resiliency of ClavaDDPM against stronger privacy attacks[5], and ii) investigate the efficacy of boosting ClavaDDPM with privacy guarantees such as differential privacy. Similarly, the impacts of our design on fairness and bias removal, as another motivating pillar in synthetic data generation, is well worth exploring as future work. We believe the thorough multi-relational modeling formulation we presented in this work, can serve as a strong foundation to build private and fair solutions upon. "

$\quad$

A-Q1 Real-world Metrics:

Yes, there are other metrics being used for evaluation. We conducted single-table machine learning efficacy (MLE) experiments using the same settings as in TabSyn, which employs an XGBoost model to predict one column given the remaining columns. The results of these experiments are presented in the appendix (section D.2). These results demonstrate that, in terms of high-order or real-world metrics, ClavaDDPM achieves state-of-the-art performance, even though it was designed for multi-table synthesis. In the revised version of our paper, we will include comments about the MLE results in the main paper.

Additionally, our primary focus in this work is on multi-table quality. As shown in Table 1, ClavaDDPM exhibits significant advantages over other baselines regarding long-range dependencies. This advantage becomes more pronounced as the number of hops increases, making the quality of ClavaDDPM more distinguishable.

For multi-table MLE, this remains an unexplored area, and we plan to address it in our future work.

$\quad$

Reply to comments:

Thank you for pointing out the typo, and suggestions about captions. We will make these changes in the revised version.

A1: Presentation improvements

Thank you for your valuable feedback. We will take steps to improve the writing and overall presentation of the paper, and have fixed the table names issue, and will have a better written revised version.

To further enhance our manuscript, we would greatly appreciate it if you could point out any specific issues with the mathematical formulations or inconsistency in definitions, so we can address them in detail.

$\quad$

A2: Justification of assumptions

Thank you for highlighting the need for more rigorous justification of our assumptions. We appreciate your feedback and would like to provide a detailed explanation: We understand the importance of validating assumptions in tabular data synthesis. When reviewing the literature, such as TabSyn and TabDDPM, we observed that they implicitly rely on the naive row-wise i.i.d. assumptions, which lack validity when extended to multi-table scenarios. To address this, we reexamined the problem and adopted a weaker and more realistic assumption: that child rows are conditioned on foreign key constraints. We provided a mathematical deduction to strengthen the theoretical foundation of our approach, aiming to address tabular data synthesis more rigorously than simply applying generative models as done in previous works. While it is challenging to justify these assumptions perfectly in alignment with real-world scenarios, we designed experiments to empirically validate them. In section 5.1, Table 1, we compared ST-ClavaDDPM with ClavaDDPM. ST-ClavaDDPM utilizes the same model backbone as ClavaDDPM but makes the i.i.d. assumption on all child table rows. In contrast, ClavaDDPM assumes that child rows are conditioned on parent rows. The experimental results in Table 1 indicate that ClavaDDPM significantly outperforms ST-ClavaDDPM, empirically demonstrating that our assumption is more effective on real-world datasets compared to the naive row-wise i.i.d. assumption. Although our assumptions may not be perfect, our empirical results show that the theoretical analysis in our paper is robust and surpasses the assumptions made in previous works. Additionally, we consider exploring even weaker and more realistic assumptions as a direction for future work.

$\quad$

A3: Complexity analysis

Thank you for your suggestion. We agree that a complexity analysis is essential for understanding the scalability and efficiency of our model, especially for large-scale databases. Here is a preliminary analysis, which we will expand upon with detailed results in the revised version.

Given a multi-relational database $G = (R, E)$ with $m$ tables, $n$ foreign key constraints, and $p$ rows per table. Given a p-row table, we denote the time complexity of performing GMM clustering $c_{GMM}(p)$, training a diffusion model as $c_{diff}(p)$, training a classifier as $c_{class}(p)$, synthesizing as $c_{syn}(p)$, ANN searching complexity as $c_{ann}(p)$.

Phase 1: Latent Learning and Table Augmentation:

• Runtime: $n \cdot c_{GMM}(p)$.

Phase 2: Training:

• Runtime: $n \cdot c_{class}(p) + m \cdot c_{diff}(p)$.
• This phase is dominated by diffusion training, primarily influenced by $m$.

Phase 3: Synthesis:

• Runtime: $n \cdot c_{syn}(p)$.

Additional Step: Matching:

• Runtime: $n \cdot c_{ann}(p)$.
• Negligible runtime with FAISS implementation.

Summary:

• Dominated by Phase 2 (training) and Phase 3 (synthesis).
• Critical factors: $m, n$, and $p$.
• Robust against the number of clusters $k$ in Phase 1 due to the dominance of later phases. Our model shows significant scalability and practicality compared to existing methods like SDV, which is limited to synthesizing at most 5 tables with a depth of 2 (section 5.2). We will provide empirical runtime measurements and baseline comparisons in the revised version.

$\quad$

A4: Fine-grained investigation of $k$

Thank you for your valuable suggestion. We agree that a more fine-grained analysis of the cluster number $k$ is essential for understanding its impact on the model's performance and identifying optimal settings for different scenarios. Our preliminary results show an interesting trend: the model quality initially performs lower when $k$ is small, peaks at $k=25$, and then decreases. When $k$ is very large, the model performance converges to a local optimum. This suggests that a technique similar to binary search could be applied empirically to find the optimal $k$.

$\quad$

$**Table 1: Model Performance for Different k**$

The results of these experiments will be included in the revised paper. This will provide a clearer understanding of the model's behavior and its adaptability to various contexts.

$\quad$

A5: Relocation of research problem

Thank you for pointing this out. We agree that moving the section on the multi-relational synthesis problem to the main body of the paper would improve the flow and make it easier for readers to follow the synthesis process and understand its relevance to the overall work. However, due to the NeurIPS page limit, we had to place the definition in the appendix. We will integrate this section into the main text in the revised version to enhance the coherence and comprehensiveness of the paper.

A1: Known relationships

We fully agree on the importance of realistic assumptions. This work tackles a prevalent challenge in the finance industry [in collaboration with them] on tabular synthesis. Synthesizing data for multiple interconnected tables, even with known foreign keys, has been entirely overlooked in the literature, with only two notable exceptions offering inefficient solutions [1,2]. Our work presents a practical approach that surpasses the state of the art (SOTA) with known foreign keys. We will discuss addressing unknown constraints in future work in the revision.

$\quad$

A2: I.i.d assumptions

We appreciate your perspective and agree that the assumption in Sec 4.1 is strong. While previous works TabSyn and TabDDPM [6,7] made even stronger assumptions (each row is i.i.d), we introduced a weaker assumption: in a parent-child two-table scenario (in Sec 4.1), each parent row is i.i.d., while the child rows are dependent on its corresponding parent row. In fact, we are making a Bayesian modeling assumption that each child row, although not independent by itself, is conditionally independent of other child rows given its parent.

Our evaluation in Sec 5.1 demonstrates the effectiveness of our weaker assumption (ClavaDDPM) on real-world datasets compared to that of prior works (ST-ClavaDDM with strong i.i.d assumptions for child rows). While our assumption may not be perfect (modeling all dependencies with perfect accuracy is hard), our empirical results support our theoretical analysis and outperform previous works' assumptions. We will clarify these and discuss weaker and more realistic assumptions as future work.

$\quad$

A3: Choice of baselines (CTGAN and TabDDPM over CTABGAN, TabSyn, and CoDi)

Our work focuses on a multi-table synthesis with a new paradigm. Thus, the strength of single-table baseline models was not our only consideration. Our goal was to pick one diffusion model and one GAN-based model, each being SOTA and representative in their domain for a diverse comparison. Firstly, we excluded CTABGAN(+) or CoDi because they were shown inferior to TabDDPM and TabSyn [6,7]. Since both TabDDPM and TabSyn are SOTA, we chose between them. Given our ClavaDDPM employs a similar model to TabDDPM, comparing against baselines using TabDDPM offers a fair evaluation, highlighting the effect of our Clava framework while disregarding the inherent advantages of TabSyn over TabDDPM. Additionally, our preliminary results (with slightly different metrics) at Clava’s early development indicated TabDDPM’s superior performance over TabSyn as the backbone on real-world datasets. (Table 2 in comments)

Thus, we chose TabDDPM as the backbone for both our model and the two baselines, ST- and Dnorm-. Even with other baselines, ClavaDDPM's advantage would remain evident.

Though CTGAN is weaker than TabDDPM, we included it because it is a representative GAN-based tabular synthesis model, aligning with the TabSyn paper’s inclusion of CTGAN over CTABGAN. We will add the clarification and include experiments on other baseline models, as you suggested in the revised version.

$\quad$

A4: Clarification on agree rate

Thanks for the suggestion. We will detail the agree rate formulation in the appendix. The agree rate is only influenced by k and lambda, but not eta or “no matching” (the last two columns that are not involved in the clustering process). We will include the agree rates in those columns for completeness and clarity.

$\quad$

A5: Clarification on convergence

We will add a complexity analysis in the revised paper. Convergence depends on both the data size and the domain size. Even though TabDDPM can handle all datasets on a 2080Ti GPU (A6000 is overkill), its multinomial diffusion for categorical features is a bottleneck with large domain sizes, as exemplified by Instacart05 and Movie Lens datasets. For instance, Instacart05 has a categorical column with 49,688 values, requiring one-hot encoding and expensive multinomial diffusion. In TabDDPM's single-table datasets, categorical columns have relatively few values(e.g., the largest category count in the adult dataset is 42). The Berka dataset, despite having more tables, has the largest categorical domain size of 77, allowing faster convergence for TabDDPM.

Real-world data often contain categorical columns with numerous distinct values. ClavaDDPM accelerates training by applying a unified Gaussian diffusion to both categorical and numerical columns, leading to faster convergence.

Additionally, for fair comparison, we aligned TabDDPM's hyperparameters with ClavaDDPM's, which are larger than the original TabDDPM settings, and thus will be slower than the original TabDDPM (e.g., diffusion timesteps = 2000 vs. 1000).

Our TabDDPM experiments used the TabDDPM library within the Synthcity framework. We will release the baseline code for reproducibility.

$\quad$

A6: The choice of evaluation metrics

We used single-table data quality metrics from TabSyn and reported KS/TV metrics in the main paper, and the rest (alpha precision, beta recall, classifier detection, machine learning efficacy) in the appendix (section D.2) due to space constraints.

We did not use the multi-table metric like relative error from PrivLava for

• It relies on human-designed queries, not applicable to all datasets.
• No open-source evaluation code, hard to replicate.
• No theoretical validation for these hand-designed queries.

These limitations highlighted a gap in multi-table synthesis evaluation. Therefore, we proposed the long-range dependency metric, a statistical measure applicable to all multi-table datasets without needing hand-designed queries.

$\quad$

A 7,8,9: Formatting issues

Thank you for your feedback. We will address them in our revision.

$\quad$

A10: Conclusion

We apologize for the omission of limitations and future work due to page constraints. These sections will be included in the revision.

[1] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.

[2] A. Kotelnikov, D. Baranchuk, I. Rubachev, and A. Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pages 17564–17579. PMLR, 2023.

[3] H. Zhang, J. Zhang, B. Srinivasan, Z. Shen, X. Qin, C. Faloutsos, H. Rangwala, and G. Karypis. Mixed-type tabular data synthesis with score-based diffusion in latent space. arXiv preprint arXiv:2310.09656, 2023.

[4] K. Cai, X. Xiao, and G. Cormode. Privlava: synthesizing relational data with foreign keys under differential privacy. Proceedings of the ACM on Management of Data, 1(2):1–25, 2023.

[5] Razavi, Ali, Aaron Van den Oord, and Oriol Vinyals. "Generating diverse high-fidelity images with vq-vae-2." Advances in neural information processing systems 32 (2019).

Although this work attempts to elaborate on its theoretical framework with substantial content, the notation system and descriptive symbols used do not conform to standard academic mathematics conventions (see sections 3. Background and 4.1 Notations).

R: Non-standard notations

Thank you for pointing this out. We acknowledge that our notation differs slightly from the standard. In probability theory, capital letters typically denote random variables, while lower case letters represent their actual values. We modified this convention to distinguish between row data and table data. For example, we use $x$ to denote a row of a child table and $X$ to denote the entire child table. To avoid conflicts, we use bold letters $\mathbf{x}$ and $\mathbf{X}$ to represent the corresponding random variables. To clarify this further, we will describe our design choices in detail and ensure consistency throughout the manuscript in the revised version.

$\quad$

The marks in equation are not uniform (Eq.2)

R: Non-uniform marks

We obtained Equation 2 from the Background section of TabDDPM[2]. We would greatly appreciate it if you could point out the specific issues with the notation or formality, and we are more than happy to make any necessary improvements.

$\quad$

Furthermore, several key formulas lack thorough proofs (for example, Formula 8). The proof of this key formula would be best placed in the appendix.

R: Proofs for formula

We agree with your observation that this is a key formula in our work. The derivation is based on Equation 7. Due to space constraints, we provided only a brief one-line expansion in the main paper. We appreciate your suggestion and will include a detailed proof in the appendix.

$\quad$

Formula 3 from a previous study lacks supporting evidence; a brief explanation of its proof process or an indication of its rationale is necessary.

R: Formula 3

Formula 3 is directly adopted from previous work [1], which was derived using Bayes' theorem. Due to space limitations, we were unable to provide a more detailed explanation in the main paper and included only the result. However, we appreciate your point and will add a more detailed background section in the appendix of the revised version. Additionally, we will briefly mention the Bayesian modeling rationale in the main paper.

$\quad$

The validity of the method “we introduce latent random variables c such that g is independent of y conditioned on c” also requires theoretical verification.

R: Validity of assumption

This is a valid concern. In fact, ClavaDDPM adopts a Bayesian modeling paradigm, where we make certain Bayesian assumptions and let the trained models enforce those assumptions. Therefore, we chose GMM with diagonal covariance, which by design learns latent variables $c$ that enforce conditional independence between $g$ and $y$ (with covariance being zero). We will include a more detailed demonstration of this aspect in the appendix to better address the theoretical foundations.

A1: Terminologies of table

Yes, a relation is a table. In this paper, we primarily adhere to the terminology used in relational database literature, where a “table” is formally referred to as a “relation”, and multi-relational synthesis corresponds to multi-table synthesis. Consequently, our notation slightly differs from previous works that only consider single tables, such as TabDDPM[2] and TabSyn[3]. To be clear, we used “tables” and “relations” interchangeably in the paper. Specifically, we model “relations” or “tables” as nodes and “foreign key constraints” as edges, consistent with relational database terminology. We apologize for any confusion and will clarify this further in the revised version.

$\quad$

A2:I.i.d. assumptions

We fully understand your concern. In our work, the term “unique identifier” refers to a unique ID for a row and is not related to the data distribution itself. Below is an example table:

In this table, the primary key (unique identifier) is simply an ID, while the actual data distribution involves attributes like (gender, country). By i.i.d., we mean that the distribution of $(gender, country)$ is independent. When modeling, the primary key is not included as part of the data but is used solely for enforcing foreign key constraints. For example, if a child table row has a foreign key value of $3$, it refers to the row $(F, UK)$ in the parent table. The value $3$ is not passed into the diffusion model and is assigned additionally.

Similarly, previous works in tabular data synthesis, such as SDV, PrivLava, and TabDDPM, treat keys as identifiers rather than data, with key values not being passed into the model. We will clarify this in our revised version to prevent further confusion.

$\quad$

A3: Latent variable learning

In the context of tabular data synthesis, PrivLava[4] is the only prior work that has learned latent variables (excluding the works that directly adopt latent generative models, which do not explicitly learn latent variables). Compared to PrivLava, our approach is agnostic to data domain size and has faster convergence. PrivLava's latent learning is marginal-based, making it applicable only to predefined fixed domains (they primarily support integers). Additionally, their use of the EM algorithm is slow and prone to convergence issues in practice. Our experiments in Section 5, Table 1, demonstrate that their method fails to converge on many real-world datasets.

Importantly, our method (diagonal covariance GMM) enforces our independence assumption and aligns with our theoretical analysis. In Section 4.1, Equation 7, we assume that child table groups $g$ are independent of parent row $y$, conditioned on latent variable $c$. Thus, we learn latent variable $c$ such that its distribution supports this assumption. As mentioned in Section 4.3.1, we use a GMM with diagonal covariances in the joint space of $(X, Y)$. This ensures that, when the GMM is properly trained, the covariance between $X$ and $Y$ is zero in each Gaussian cluster with centroid $c$. Consequently, conditioned on $c$, $g$ and $y$ will tend to be independent, as $g$ is a collection of $X$.

While more advanced latent learning methods, such as VQVAE[5], could perform latent learning and quantization simultaneously, we considered them during development but identified potential trade-offs in terms of runtime. Currently, the runtime of the latent learning phase is negligible compared to other phases, and VQVAE lacks synergy with our theoretical assumptions. Nevertheless, as noted at the end of Section 4.3.1, we are open to exploring the impact of other latent learning methods in future work.