Programmable Synthetic Data Generation

First of all, we would like to thank the reviewer for their detailed assessment of our paper, and for their constructive feedback. We are also pleased to read the reviewer’s recognition of the timeliness and importance of the problem setting, the quality of presentation, and the strong empirical results. Below, we address the reviewers comments and questions:

Q1: Can ProgSyn support fairness notions beyond demographic parity?

Yes, there is no technical restriction in ProgSyn that would only allow for demographic parity, and the modularity of ProgSyn allowed us to easily implement and test two more fairness measures: equality of opportunity (EoO), and equalized odds (EO). We repeat the fairness experiment on Adult on these two new measures in all settings and report our results in the tables below:

Equalized Odds (non-private) – True data acc.: 85.4% EO: 0.08

Equalized Odds (differentially private):

Equality of Opportunity (non-private) – True data acc.: 85.4% EoO: 0.09

Equality of Opportunity (differentially private):

As we can observe from the above tables, ProgSyn still prevails as the method providing the best accuracy at a low bias level. In the non-private setting ProgSyn achieves both the best fairness and accuracy from all methods, while in the differentially private setting, it achieves comparably low bias to PreFair while having >3% higher accuracy.

In general, any fairness measure (as well as any other specification) that can be expressed as a continuous function or a good enough relaxation is naturally supported by ProgSyn. Therefore, there is no fundamental reason why individual fairness notions could not be integrated into ProgSyn either, however, we do not consider it here as (i) there are conflicting notions of individual fairness, (ii) the design of individual fairness pipelines involve challenges that are orthogonal to the objective of our work, and (iii) the fair synthetic tabular data generation methods we have surveyed all concern group fairness, which is also where we position our work. We definitely believe that developing a flexible individually fair synthetic data generation pipeline is an interesting research challenge that could lead to highly valuable contributions for the community.

Further, regarding causal fairness, we did not consider this fairness notion as it requires knowledge of the causal structure of the dataset, introducing a tight, second information bottleneck in the pipeline, and additional challenges worth their own contributions, such as the question of how can the elucidation of the causal structure be united with differential privacy. The requirement for the causal graph is a serious limitation of causal fairness methods, as also exemplified by DECAF, one of the fair synthetic data generation methods we compare ProgSyn against, where we were only able to conduct comparisons on Adult, and not on the other three datasets, as the method’s implementation included only the causal graph of Adult.

Q5: “Could you provide some information on the formulation process for statistical control?”

Yes, let us illustrate the technique using the following example:

ENFORCE: STATISTICAL:
	E[2 * age - capital_gain | education == Doctorate] >= 500;

The above specification consists of a conditional expectation in a comparison term. Our method first calculates the statistical operation and then combines them in the formula using t-norms and DL2 [11] primitives.

Proceedingly, let us first calculate the term E[2 * age - capital_gain | education == Doctorate]: here, the two involved features are age and capital_gain, therefore $\mathcal{S} = \\{\`age`, \`capitalgain`\\}$ (following the notation in the paper). The operation is the expectation $E$ , the features are involved in the function $f(\mathcal{S}: x, y) = 2 * x - y$ , and the condition $\phi$ is education == Doctorate. First, we calculate the binary mask belonging to the condition following the technique described for logical constraints, obtaining $b_{\phi}(\hat{X})$. Then, we apply the mask to the sample $\hat{X}\_{\phi} = b_{\phi}(\hat{X}) \odot \hat{X}$, zeroing out all the rows where the condition does not apply. Using $\hat{X}\_{\phi}$, we calculate the joint marginal of the age and capital_gain features, obtaining: $\bar{\mu}(\`age`,\`capitalgain`, \hat{X}\_{\phi})$, which is the empirical estimate of the distribution $p_{\theta}(\`age`, \`capitalgain`|\phi)$. Now, we can insert the obtained distribution into the mathematical definition of the expectation operation and obtain the value for our first term:

$t\_1 = \sum_{\`age`}\sum_{\`captialgain`} (2 * `age` + \`captialgain`)\, \bar{\mu}(\`age`,\`capitalgain`, \hat{X}\_{\phi})$ Notice that due to the careful steps taken, this process is fully differentiable, even though it involves conditioning and arithmetic expressions in discrete features.

Next, we are left with the expression $t_1 > 500$ which is a logical expression of reals, for which we have to construct a differentiable loss term. Here we use the primitives introduced in [11] to construct a differentiable loss term:

$\mathcal{L}\_{\`stat`}(\hat{X} \leftarrow g\_{\theta}(z)) = max(0, 500 - t\_1).$ Notice that the term is positive, and therefore adds a loss penalty only if $t_1$ is less than $500$, i.e., the desired constraint is not satisfied. Otherwise, when the constraint is met, the loss is zero.

Q6: Please clarify the organization of the introduction section, and address other comments regarding the formatting of certain elements in the paper.

We believe that a good presentation in a paper is extremely important, therefore, we appreciate any feedback and comments on it, and are eager to incorporate them, further improving the paper.

Introduction: We believe that the introduction section of our paper follows the common pattern of how introductions are and have to be structured, at least what is commonly adopted in the machine learning literature. We first start by setting a general scene to our problem (first paragraph), then we introduce the background and most related work, highlighting the research gap our work will target (the two paragraphs connected to “Synthetic data”), finally, we briefly introduce our method, structured in a short technical introduction, a working example, and a summary of our results. For some examples on this structure, see [4], [7], [12], [13], and [14], all published papers at prestigious venues. Nevertheless, we are always looking to improve our paper, therefore, if the reviewer has any concrete suggestions for improving the introduction, we would be highly appreciative of that.

Lengthy formulas: Although due to the space restrictions of the paper format it will be unavoidable to sometimes have inline math formulas, we thank the reviewer for raising our awareness on readability concerns, and have uploaded a revised version of the paper.

Pseudocode: In general, we believe that including excerpts of code in the paper, especially if it is core to its message, is not non-standard writing. Additionally, the referenced code snippet is not just mere pseudocode, but an actual example of how specifications can be defined and passed to ProgSyn, written in a domain specific language constructed by us for this task. If this has been unclear from the presentation in the paper, we are eager to improve it, and would be very grateful if the reviewer could share their concrete suggestions with us.

First of all we would like to thank the reviewer for their thorough investigation of our work, including the supplementary materials, and for their far-reaching and constructive feedback. We also appreciate the reviewer’s recognition of the novelty of our work, its strong empirical performance, and the quality of the provided artifacts. Below we address your questions and points raised:

Q1: Does the empirical evaluation in the paper provide a compelling argument?

Yes, we believe so. The evaluation method used in the paper to obtain our results follows the practices established in the literature on synthetic tabular data generation (e.g., [1, 2, 3, 4]). Although we also believe that providing a robust and principled statistical hypothesis test for validating the synthetic data would be ideal, this is currently not the method of evaluation in this field, due to it being fundamentally hard. Due to the high complexity and dimensionality of the involved distributions, there is currently no gold-standard method that is tractable and reliable to a degree that would allow it to be adopted by the field of synthetic tabular data research for evaluation. The closest to this comes the comparison of low dimensional marginals, which is the basic building block of our method already. All in all, we believe that strong contributions can be made here that would benefit the community, but addressing this problem is beyond the scope of our work.

In more detail, we evaluate mainly using the downstream accuracy of a state-of-the-art classifier, and, as shown in the experiments in Appendix C, we also measure the TV-distance between the learned and the true k-way marginals, effectively calculating a low-dimensional approximation of the divergence between the true and the modeled distribution. Note that we measured both metrics for all conducted experiments, merely, for presentational brevity, we opted to include the accuracy metric in the main paper, due to its easy understanding, and higher interest for practitioners. Furthermore, a strong classifier is also a meaningful proxy summarizing how well basic to very high order correlations have been preserved in the synthetic sample with respect to the original data.

As for assessing the alignment of the synthetic data with the custom specifications, we directly measure their satisfaction rates/degrees on the generated synthetic samples, providing the closest metric of interest.

Q2: Would using different models to evaluate the quality of the data change the overall picture of ProgSyn’s strong performance compared to other methods?

No, and for the sake of demonstration, we have re-run the fairness experiment on Adult adding more classifiers. In the table below we compare the performance of ProgSyn to the next best non-private method TabFairGAN on the classifiers Logistic Regression (LogReg, linear model), SVM, Random Forest (RF, tree ensemble), XGBoost (XGB, gradient boosted tree-based, used in the paper), and CatBoost (CB, gradient boosted tree-based):

As we can see, the overall picture is unchanged: ProgSyn prevails as the best method in terms of fairness-accuracy trade-off. Note that for LogReg and SVM, TabFairGAN is unable to produce data that is suitable for training and therefore results in classifiers that largely only predict the majority class for every instance, irrespective of any feature (including the protected feature, therefore such classifiers appear as fair, however are not useful as they are only able to assign every instance to the same class).

Further, note that [5] asserts that for evaluating synthetic tabular data, one should use a state-of-the-art classifier, as weaker models are unable to exploit the edge in distributional detail that better synthesization methods provide, painting a false picture.

Finally, we do not test on recommendation models, as, again in line with prior work, the datasets used in this paper are classification datasets. Note that this is representative, especially in the context of our work, as none of the referenced methods in which ProgSyn is positioned examine recommendation datasets, but focus mainly on classification datasets (e.g., [1, 2, 3, 4, 5, 6, 7, 8, 9]). Also, as table 10 in [10] highlights, albeit in the context of federated learning, only 2 out of 61 tabular data applications concern recommendation systems.

We would like to thank the reviewer for their acknowledgment of our timely and strong empirical contribution and quality of writing, their informative feedback, the informative resource provided, and their favorable overall assessment of the paper. Below we address their questions and points raised:

Q1: What novel technical and conceptual contributions does ProgSyn make?

Additionally to ProgSyn’s empirically demonstrated strong performance and high level of versatility, we firmly believe that ProgSyn brings both significant technical and conceptual novel contributions to the field of synthetic tabular data generation.

Conceptually, it is the first work to facilitate extended customizability of the generated data, allowing for row-wise, distributional, and downstream-dependent specifications. Our framework cannot only be used to improve the synthetic data quality, but also to hand-tailor its distribution and other properties to specific, custom use-cases defined by the user. At the same time, ProgSyn effectively captures the capabilities of prior works across different aspects of tabular data generation.

Technically, ProgSyn introduces several novel elements in the differentiable computation of the specification loss-terms, tackling several difficult challenges on the way. The common key challenge of each specification type arises from the fundamentally non-differentiable nature of categorical features in the produced tables. Once the conversion from complex constraints involving discrete features is achieved, using them as a regularizer in a two-staged optimization pipeline is indeed a natural approach for manipulating the modeled distribution to our liking. Nevertheless, note that ProgSyn makes its key technical contributions addressing the challenge of arriving at these differentiable loss terms (something that is for example in [1] in the domain of time series data is treated as a given). For instance, for logical constraints and conditions, we address this through our masking scheme introduced in Section 4. Further, we are the first to offer a technical solution to incorporate complex formulas involving conditional statistical expressions acting directly on the distribution of the data, going beyond row-wise constraints. We are also first to provide a general framework of incorporating downstream classifiers in the synthetic data generation training process; tackling the difficult additional challenges of training-in-the-loop, and uniting it with the rigorous requirements of differential privacy. Notably, this allowed us to set a new state-of-the-art on fair synthetic data generation, both in the non-private and the differentially private setting, while this objective not being the main focus of our work.