How Realistic Is Your Synthetic Data? Constraining Deep Generative Models for Tabular Data

We thank the reviewer for their feedback, which allowed us to study in more detail the behaviour of our constraint layer.

Question: How does the CL training procedure work? And how does CL affect the training?

Answer: The basic intuition behind our layer is very simple: given a set of constraints expressed as linear inequalities, we compile our constraints into a differentiable layer (CL), which then gets added to the topology of the network itself right after the DGM layer that generates the samples. At training time, the network is trained with CL injected in the topology and the gradients backpropagate seamlessly through it. We have added a paragraph to explicitly state that the gradients can backpropagate through CL at the beginning of Section 3.

Question: Can CL be applied to other generative models?

Answer: Yes, we applied it to TVAE [1] and GOGGLE [2], and we added the results in Table 2. As can be seen from the results, adding CL at training time results is an improvement in all metrics but one for TVAE and in all metrics for GOGGLE. As we stated in the general answer, note that the results on GOGGLE are computed on 5 out of 6 datasets. We will update the results with the final values as soon as possible.

I thank the authors for answering my questions and adding extensive experiments and results to the paper. While I believe that the quality and clarity of the paper have already improved, there are still a couple of points that remain unclear to me.

• From the Wasserstein distance results, it seems like the C-DGMs can sometimes have higher distances compared to non-constrained models, which would suggest that the learned distribution becomes worse. In addition, sometimes the distance is also greater than the one obtained with the P-DGMs, which to me is counterintuitive given the considerations made in your answer that when the CL is added at training time, "the DGMs are aware of the presence of the layer, teaching the model to distance its outputs from the invalid regions". Do you have some insight on why in some cases the Wasserstine distance increases for C-DGMs?
• From answer 3, do I understand correctly that CL cannot be applied to categorical features? Could you please clarify this point, and explain the differences in the Jensen-Shannon divergence between categorical features in Table 12?

First of all, we would like to thank you for answering our comments and appreciating the efforts we have made to extend the experimental analysis and improve the clarity of the paper.

• Regarding the analysis of the results obtained using the Wasserstein distance (WD), some considerations are in order. Indeed, in the experiment conducted to study the overpopulation problem, we computed the Wasserstein distance following the approach of Zhao et al. 2021 [1] (then replicated by Kotelnikov et al. [2]), which consists in considering every feature separately, computing the Wasserstein distance for that feature, and finally averaging over the values obtained for all features. We then saw that the results were inconclusive (as we were doing better/worse $\sim 50$% of the times) and thus focussed on how to show that our models do not suffer from the overpopulation problem. Upon a closer look though, we realised that while this approach might have been sensible for the data considered in [1,2], it is not in our case. Indeed, our datasets contain features ranging from $0$ to $1$, as well as features spanning from $10^3$ to $10^7$. This entails that our results were solely driven by the features which have a high magnitude (as an example, notice that $WD([0, 1, 3], [5, 6, 8]) = 5$, while $WD([0, 1, 3]*10, [5, 6, 8]*10)=50$). We thus thank the reviewer for asking about this, as it allowed us to look deeper into the metric computation. To overcome this issue, we decided to first apply mix-max scaling to all the features, and then repeat the procedure outlined above. The newest results are in Table 11, where we can see that the differences are very small with only one big gap equal to $0.17$, as C-GOGGLE (resp. GOGGLE) obtains Wasserstein distance equal to $0.05$ (resp. $0.22$) on the WiDS dataset. Additionally, we can see that the biggest "negative" gap (meaning that a DGM obtains a lower distance than its corresponding C-DGM) is equal to $0.05$ and is obtained by GOGGLE on the News dataset. While this might seem to suggest that the distribution produced by C-GOGGLE is worse, we can clearly see that C-GOGGLE does better in terms of utility (according to both XV and MAE). This shows that, while the Wasserstein distance is an indicator of the quality of the captured distribution, it is not by any means an exhaustive metric. This makes sense, as in this metric each feature is considered separately and, thus, it cannot capture the correlations existing among features.

• Regarding the Jensen-Shannon divergence, we updated the results in Table 12. Notice that the C-DGM model is a different model from the DGM one, and thus can learn a different distribution from the initial DGM even for the categorical features. On the other hand, the P-DGM is exactly the same model as the DGM with the constraint layer added at inference time, and indeed we get exactly the same values as the DGMs (in our dataset we do not have any constraint over the categorical features).

• Regarding the ability of writing constraints over categorical features, we can only constrain the continuous values outputted by the neural networks. For example, given two DGM outputs $x_1, x_2 \in [0,1]$, we can write the constraint $x_1 > x_2$ expressing that if $x_2$ is positively predicted w.r.t. some given threshold, then $x_1$ should also be positively predicted. How to integrate constraints expressed in, e.g. propositional logic, surely represents interesting future work.

References

[1] Zhao, Z., Kunar, A., Birke, R., and Chen, L. Y. Ctab-gan: Effective table data synthesizing. In Asian Conference on Machine Learning, pp. 97–112. PMLR, 2021.

[2] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. TabDDPM: Modelling Tabular Data with Diffusion Models. In Proceedings of International Conference on Machine Learning, 2023.

We thank the reviewer for their feedback. Below are detailed answers to all their comments.

Question: The paper mentions the choice of $\lambda(i) = i$ is for ease of notation, while different orderings can drastically change the performance of a C-DGM. [...] In fact, the notion of "optimality" is not well defined, as each ordering can produce a different optimal with none of them being comparable for defining an optimum. Moreover, the notion of optimality should also consider the discrepancy between the $CL(\cdot)$ outputs and the generative samples to be minimal.

Answer: We articulate our answer in three points:

• In a tabular dataset, the features are not ordered (contrarily to e.g., natural language datasets), and thus we can associate e.g., the variable $x_0$ to any feature. Thus, our choice of notation does not affect our results in any way.
• The optimality is well-defined and takes into account the discrepancy between the final CL outputs and the generative samples. Let us consider the example proposed by the reviewer. As it can be seen, both the final outputs $\langle 10, 10, 10 \rangle$ and $\langle 5,5,5 \rangle$ are optimal according to our definition. Indeed, for both the final outputs there does not exist another sample such that (i) it satisfies the constraints and (ii) for each $i=1,2,3$ it associates a value closer to the original output than CL. The very fair question the reviewer asks is essentially why are both final outputs considered optimal. To answer this question, suppose that we are given a DGM that is not able to capture the distribution of the features $x_2$ and $x_3$, while it is particularly good at capturing the distribution of $x_1$. In this scenario, the output $\langle 10, 10, 10 \rangle$ is much more desirable than $\langle 5,5,5 \rangle$. Thus, ruling out a priori the solution $\langle 10, 10, 10 \rangle$ as ``non-optimal'' would be a mistake. The fact that closeness to the original output is not the only factor to take into account when correcting a deep learning model has already been shown in the neuro-symbolic community (see e.g., [1]), where it has been shown that selecting the closest solution leads to worse performance than selecting the solution taking into account the confidence in the predictions. We understand though that this might be counter-intutive, and for this reason we have added a paragraph in Section 3 (right above Theorem 3.5).
• As a consequence of the point above, in Appendix C.3 a study on the impact of the variable orderings can be found, where we also propose two heuristics to decide favourable variable orderings. We could not include such a discussion in the main paper due to the lack of space. A clear reference to this section has been added to the above-mentioned paragraph that has been included in the main body of the paper.

To conclude, we thank the reviewer for these comments which will surely further improve our paper. The insightful discussion we had in the past days highlights the importance of the problem handled and the many directions this formulation of the problem of tabular data generation (where the methods not only have to approximate a distribution but also be compliant with a set of constraints) opens up. We see this paper as the first stepping stone in this direction, and we believe that many people will take up the challenge and build upon it.

References:

[1] Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.

[2] Thibault Simonetto, Salijona Dyrmishi, Salah Ghamizi, Maxime Cordy and Yves Le Traon. A Unified Framework for Adversarial Attack and Defense in Constrained Feature Space. Proceedings of IJCAI, 2022.

We thank the reviewer for the response and for the discussion, which has sparked many ideas about possible directions for future work.

• Regarding the variable ordering: We absolutely agree that the ordering is critical. We also thank the reviewer for the suggestion of using the causal information to decide the ordering, which would allow us to compute a favourable ordering without the need for first training an unconstrained model, thus making our method even more convenient and usable for a end-user. However, our intuition is that, if it is possible to first train an unconstrained DGM, then an ordering based on the ability of the model itself to learn the distribution of each feature would probably lead to better results. As an example, consider a DGM that has to learn the distribution of some data collected during a clinical trial which includes the features "High Blood Pressure" and "Heart Attack". In this scenario, we know that there exists a causal relation between these two features and thus a causality-inspired ordering would first consider the "High Blood Pressure" feature and then the "Heart Attack" feature. Now, suppose that the DGM approximates very well the distribution for the feature "Heart Attack" but poorly the feature "High Blood Pressure", then in this scenario it would probably lead to better results to consider the "Heart Attack" feature first. Given this example, an interesting question that arises is whether there is any relation between the causal relationships existing among features and the difficulty of learning them with a DGM. As the above conjectures make very interesting problems to be studied, we agree with the reviewer that a more systematic study on the variable orderings is necessary. For this reason, we will do the following:

  • We will consider an ordering based on the causal relationships among the features. Unfortunately, none of our datasets comes with such information. However, we can use some of the existing algorithms for causal relation extraction (e.g., the PC algorithm and the package pcalg: https://cran.r-project.org/web/packages/pcalg/index.html) to first extract the causal relations and then use this information to calculate the ordering.
  • We will consider an ordering based on the Wasserstein distance computed for each feature between the real data distribution and the distribution of the samples obtained with the unconstrained DGMs.
  • We will better explain the orderings already proposed and put them in context with the newly considered orderings.

  Unfortunately, in the little time remaining before the end of the discussion period, we cannot update the paper with the results above. We will do so in the final version.

• Regarding the exponential blow-up: We are very happy that we agree on the issue of exponential blow-up. We also agree on the fact that there are many interesting ways of formulating the problem we studied and we cannot wait to see how the field will develop. For example, in [2] the authors study how to constrain the generation of adversarial attacks by embedding the constraints in the loss function. This formulation is interesting because it does not suffer from the issue of the exponential blow-up. However, notice that by formulating the problem in this way, the authors lose the ability of guaranteeing the satisfaction of the constraints, which is one of the strength of our proposed method. On the other hand, we are not aware of any work where the authors managed to linearize the objective that maximizes when the data is "realistic" while having some linear constraints to satisfy. We again agree with the reviewer that this could be an interesting direction to explore.

• Regarding the linear constraints: Certainly there are cases in which there are interesting constraints that are not linear and are polynomial or even transcendental. Even though in some of such cases the complexity of deciding the satisfiability of the constraints may greatly jump up (up to become undecidable), we acknowledge that there are cases in which a higher complexity might not be a problem, e.g., because there are no stringent requirements on the time necessary to produce the output. In any case, the suggested idea of using invertible neural networks seems applicable to a broad range of scenarios and interesting for future work. In our current settings, non-linear constraints did not arise and the time to produce the output is indeed considered an important metric according to which the DGMs are evaluated.

Thank you for the comprehensive response and the additional analysis provided. The inclusion of the analysis on TVAE and GOGGLE is particularly appreciated and it coupled with the addition of the Botnet experiments has positively influenced my evaluation. While I am inclined to adjust my score upwards, I still have some reservations. Addressing these issues could significantly enhance the paper's robustness, at least from a theoretical point of view. I encourage a deeper exploration of these aspects in your future revisions.

• About the ordering: I concur that tabular data inherently lack an ordered structure, yet they often exhibit an underlying causal sequence reflective of their data generation process. For example, selecting an order precisely opposite the causal direction would presumably yield non-realistic results. The distinction between correlation and causation is crucial here; choosing an order based solely on correlation seems impractical (the first heuristic), and I foresee scenarios where this method could fail. On the other hand, the KDE-based technique, as it stands, appears somewhat arbitrary. Although Table 13 shows marginal improvements, a more systematic method for determining this ordering would be preferable. Furthermore, I appreciate the additional material that clarifies the limitations of using 'closeness' to model-generated samples as a measure of output realism. While I agree that proximity in terms of some distance in a normed vector space to the model's output does not guarantee realism, the impact of different orderings on the output cannot be overlooked. There's a clear need for a more systematic approach to determine this ordering.

• About exponential blow-up: Based on the explanation provided, I am now inclined to believe that the issue of exponential blow-up, as described, is unlikely to occur in practical scenarios. The evidence presented in the current literature, especially the insights from the botnet experiment, has effectively addressed my initial concerns in this regard. However, from a theoretical perspective, I still find the idea of exploring more efficient sampling mechanisms from the polyhedra induced by linear constraints to be a compelling avenue for research. Is there any existing literature on writing the constraints as an optimization problem? For example, an objective that maximizes when the data is "realistic" and some linear constraints showing the constraints that should be satisfied from our knowledge base? If one can linearize the objective, then this would be an LP and solving it would not take exponential time.

• About linear constraints: The revised manuscript primarily highlights the advantages of linear constraints, emphasizing their ease of management. I appreciate the authors' inclusion of an example demonstrating how polynomial constraints are not necessarily convex, which adds valuable context to the discussion. While I acknowledge the utility of Farkas' lemma in the world of linear constraints, the reality is that not all constraints we encounter are linear. A potentially interesting approach to circumvent the linear assumption might be to employ an invertible neural network. This network could project data into a space where non-linear constraints are linear, apply the post-hoc method, and subsequently map the data back to its original space. However, the current argument presented by the authors seems to narrowly focus on the convenience of linear constraints without adequately addressing the occurrence and implications of non-linear constraints. This gap in the argumentation leaves room for further exploration and justification regarding the choice of constraints in various scenarios.

We thank the reviewer for their appreciation of our work and for their useful suggestions.

Question: Tabular data synthesis methods used for the experiments are outdated. Please consider [1] GOGGLE, [2] GReaT, [3] STaSy, [4] CoDi, and [5] TabDDPM if possible. [...] How was the performance improvement of the state-of-the-art methods after applying the proposed method?

Answer: We included two new models in our experimental results: the suggested model GOGGLE [1] and TVAE [2] (which was requested by both reviewers ikQ7 and 72Ff). The results for both models can be found in Table 2.

Regarding GOGGLE, we reported the results for 5 out of 6 datasets in green, and we can see that our method gives an overall improvement in both utility and detection metrics, when using the constraint layer during training. We are currently running the last experiments on the remaining dataset, along with the post-processing experimental analysis.

Regarding TVAE, we can see that adding our layer at training time improves the utility according to all metrics and the detection according to 2 out of 3 metrics.

References:

[1] Tennison Liu, Zhaozhi Qian, Jeroen Berrevoets, and Mihaela van der Schaar. GOGGLE: Generative modelling for tabular data by learning relational structure. In Proceedings of International Conference on Learning Representations, 2022.

[2] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. In Proceedings of Neural Information Processing Systems, 2019.

We thank the reviewer for recognizing the clarity of the paper, its correctness, and the importance of the problem solved.

Question: The issue with the paper is the proofs can be difficult to follow. It is possible that the authors can spend more time rewording it to make it easier to flow. This also makes it a bit hard to flow and check.

Answer: We have reworded all the proofs to make them more readable.

Question: I think the authors can potentially frame the use of linear constraints in a better light to highlight how its logical completeness can be useful when specifying background knowledge.

Answer: Thanks for pointing out the logical completeness of linear inequalities, which further emphasizes the advantage of utilizing linear constraints in our work. We have added a paragraph in Section 2 (below Example 2.1) highlighting all the positive properties of linear inequalities that we exploit in our paper.

As the rebuttal period draws to a close, we express our appreciation for the constructive dialogue with the reviewers. Their valuable feedback has significantly shaped the evolution of our paper, resulting in what we believe to be a more comprehensive and improved version. All recent modifications, including those initially highlighted in green, are now marked in blue. Notably, we have completed the experiments for the GOGGLE model. The thoughtful comments provided by the reviewers had not only motivated us to refine our current work, but have also sparked ideas for potential future improvements, which we plan to explore in the post-rebuttal period. As a closing remark, we would like to emphasize that our paper marks an initial step towards generating synthetic tabular data that guarantees the satisfaction of background knowledge expressed as linear constraints, a crucial aspect in diverse applications. We envision that such practices will become common in future research within this domain. Furthermore, we anticipate that in the forthcoming works we will broaden the range of background knowledge types supported by our constraint layer integrated into deep generative models.