Debiasing Synthetic Data Generated by Deep Generative Models

We would like to thank Reviewer ZgH6 for their careful reading of the manuscript and constructive review.

It would be good to put the work in contrast with related works <...>.

We noted that other reviewers raised similar questions and have therefore included an extended comparison with related work in the global rebuttal, which will also be integrated into the manuscript.

Could the authors provide more details on how their approach scales with larger datasets and more complex models?

We elaborate on this in our global response as well, but agree that future work should additionally investigate high-dimensional settings. Complex joint distributions of the observed data may impede the baseline convergence of the DGM, which in turn could diminish the utility gain provided by our debiasing approach. This is a limited weakness of our proposal as it may be the unavoidable consequence of working with high-dimensional data (e.g., debiased machine learning strategies likewise provide no guarantees in high-dimensional complex settings). Nevertheless, we expect that our approach will still improve inference, as is likewise observed for debiased estimators based on machine learning predictions.

Concerning an extension to more complex estimators, we would like to highlight that the proposed debiasing strategy is already applicable to all pathwise differentiable parameters. This includes essentially all finite-dimensional parameters used in routine practice, such as variances, regression coefficients,... An empirical study for other parameters than the population mean and linear regression coefficient is of interest, but beyond the scope of this work, whose aim is to show, for the first time, that valid inference on DGM-based synthetic data is feasible, but requires proper debiasing. Lastly, we would like to stress that our proposed strategy is generator-agnostic and can therefore be extended to more complex DGMs.

How does the proposed method perform in scenarios with multiple sources of bias or more intricate data dependencies?

Regarding the first part of the question concerning the multiple sources of bias, we wonder whether there is a misunderstanding regarding bias in the context of fairness. In view of this, we clarified the meaning of bias in our manuscript in the global rebuttal. In particular, we focus on the estimation bias that arises in estimators based on synthetic data generated by DGMs. In our paper, we target each estimator separately (e.g. the mean of age vs. the effect of therapy on blood pressure), but future work should focus on targeting all these sources of bias at once. This entails that the person generating the synthetic data is aware of all the analyses that will be run on those data, and has access to the corresponding EICs. Whether targeting multiple estimators at once affects the efficacy of the debiasing approach remains to be studied. The effect of more intricate data dependencies on the performance of our debiased strategy is an interesting yet open research question at this point. Our approach relies on fast baseline convergence of the DGM, which is influenced by the dimension of the data and the complexity of the true data-generating model.

Can the authors elaborate on any potential trade-offs between debiasing effectiveness and computational efficiency?

There are indeed potential trade-offs that should be taken into consideration. As alluded to in the paper, the current implementation of our strategy does not use sample splitting to estimate the bias term and therefore does not require the generative model to be retrained, making our strategy computationally efficient. Future extensions that necessitate a sample splitting procedure during which the generative model is retrained on different subsets of the data (and as such, computation time will depend on the dimensionality of these subsets) should focus on both computational and statistical efficiency. Our theory currently discards sample splitting due to preliminary results suggesting that the bias reduction did not outweigh the increase in finite-sample bias that may result from training the DGM on smaller sample sizes. This suggests a trade-off between debiasing effectiveness and statistical properties such as finite-sample bias and increased variability when using sample splitting.

How is “a very large sample” defined? <...> Is the necessary synthetic data sample size generalizable <...>?

In the set-up of our simulation study (which Section A.7.2 relates to), we consider different sample sizes of observed and synthetic data: $n \in$ {50, 160, 500, 1600, 5000}. These sample sizes were chosen to reflect a range that is common in real medical settings. Next, in the theory of our debiasing strategy for a linear regression coefficient we need the terms $E_{\hat{P_n}}(A|X_i)$ and $E_{\hat{P_n}}(Y|X_i)$. Given that these reflect unknown conditional population means, we need a way to approximate these in practice. In this case, one often creates an artificial “population” by generating a very large sample of e.g. one million observations, to then approximate the population mean by calculating the sample mean of this Monte Carlo sample. The purpose of this very large sample is merely to recreate a population of samples generated by the trained DGM and is thus not comparable to the magnitude of sample sizes of the observed data. We believe one million observations will suffice to capture the distribution of samples generated by the DGM, regardless of the specific type of DGM.

Chernozhukov, V., Chetverikov, D. et al. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1–C68.

Decruyenaere, A., Dehaene, H. et al. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In The 40th Conference on UAI.

van der Laan, M. J. and Rose, S. (2011). Targeted Learning. Springer Series in Statistics.

We would like to thank Reviewer 4cJP for their careful reading of the manuscript and constructive review.

How well does the debiasing strategy perform in high-dimensional settings <...>?

We elaborate on this in our global response as well, but agree that future work should additionally investigate high-dimensional settings. Complex joint distributions of the observed data may impede the baseline convergence of the DGM, which in turn could diminish the utility gain provided by our debiasing approach. This is a limited weakness of our proposal as it may be the unavoidable consequence of working with high-dimensional data (e.g., debiased machine learning strategies likewise provide no guarantees in high-dimensional complex settings). Nevertheless, we expect that our approach will still improve inference, as is likewise observed for debiased estimators based on machine learning predictions.

Can the debiasing strategy be extended to more complex estimators <...>?

We elaborate on this in our global response, but note that the proposed debiasing strategy is already applicable to all pathwise differentiable parameters. This includes essentially all finite-dimensional parameters used in routine practice, such as variances, regression coefficients,... An empirical study for other parameters than the population mean and linear regression coefficient is of interest, but beyond the scope of this work, whose aim is to show, for the first time, that valid inference on DGM-based synthetic data is feasible, but requires proper debiasing.

Are there alternative debiasing approaches that could be explored to address the limitations of the proposed method <...>?

We refer the reviewer to our global response on related work and alternative debiasing approaches. Regarding their specific questions on the dependence on EICs and conditional sampling, future work could:

• integrate insights from the literature on automatic debiased machine learning (Chernozhukov et al. (2022)) that allows to estimate the EIC automatically from data.
• combine the findings from our work and work of Ghalebikesabi et al. (2022) on importance weighting, where the weights could be targeted to eliminate the impact of the data-adaptive estimation of the weights. This could relax the fast baseline convergence assumption, and enable the same ‘debiased’ synthetic data to be used for multiple analyses. The latter might also be possible by undersmoothing the DGM, by extending the recent work on kernel debiased plug-in estimation to the synthetic data context.
• incorporate strategies for conditional sampling from DGMs (e.g. the work of Zhou et al. (2023)). Note that in case of a limited number of discrete covariates, conditional sampling can be approximated by (unconditionally) generating a large synthetic sample and taking a conditional subset, as done in our simulation study and second case study.

How can the debiasing strategy be balanced with privacy concerns <...>?

The privacy-utility trade-off is well-known in synthetic data literature and relates to the sample size of synthetic datasets: as m → +∞, the synthetic dataset has higher probability to contain synthetic records that are close to the original, resulting in higher disclosure risk (Reiter, J. and Drechsler, J. (2010)), while the inferential utility improves in terms of more precise estimates (as can be seen from the expressions for the standard error). Although formal privacy assessment is beyond the scope of our paper, this trade-off is relevant to our work. Default synthetic data created by differentially private (DP) generative models will by definition not have worse privacy with increasing sample size, but will only have better inferential utility when the estimators remain roughly √n-consistent. With DP DGMs, this is not the case (see Decruyenaere et al. (2024)), but it could possibly be attained by extending our debiasing approach, without compromising any privacy guarantees due to the post-processing immunity of DP. This extension should incorporate the additional DP-constraints into the derivation of the difference between $\theta(\tilde{P}_m)$ and $\theta(P)$. It is not clear yet what this would look like, given that different DP generators add noise in different ways (e.g. DPGAN adds noise to the gradients, while PATE-GAN adds noise to the majority vote in the training procedure). We will discuss this nuance more clearly in our revised manuscript.

Are there specific scenarios where the assumptions required for the debiasing method might not hold <...>?

Our approach relies on fast baseline convergence of the DGM: faster than n-to-the-quarter convergence rates for the unknown functionals of $\tilde{P}_m$ and $\hat{P}_n$ that appear in the EIC are required. Their attainability depends on the number of parameters in the DGM itself, the dimension of the data and the complexity of the observed data distribution. If these rates are not attained, then we expect that our approach will still improve coverage, but that no root-n consistent estimators will be obtained based on the synthetic data, resulting in increasingly anti-conservative coverage with larger sample size n.

Chernozhukov, V., Newey, W., and Singh, R. (2022). Automatic debiased machine learning of causal and structural effects. Econometrica, 90(3), 967-1027.

Decruyenaere, A., Dehaene, H. et al. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In The 40th Conference on UAI.

Ghalebikesabi, S., Wilde, H. et al. (2022). Mitigating statistical bias within differentially private synthetic data. In The 38th Conference on UAI.

Reiter, J. and Drechsler, J. (2010). Releasing multiply-imputed synthetic data generated in two stages to protect confidentiality. Statistica Sinica, 20, 405–421.

Zhou, X., Jiao, Y. et al. (2023). A deep generative approach to conditional sampling. Journal of the American Statistical Association, 118(543), 1837-1848.

We would like to thank Reviewer LYaP for their careful reading of the manuscript and constructive review.

I do not know if this comes from my lack of knowledge in the field, but I found the paper very hard to understand.

We are aware that the proposed strategy is not trivial and combines different concepts in order to eliminate the deviating behaviour of estimators seen in synthetic data (Decruyenaere et al., 2024). However, we hope that the proposed guidance below for Section 3 of a revised manuscript improves readability. We will also rewrite the introduction to stress the difference with other related work and emphasize our contribution in the field. Therefore, we have included an extended comparison with related work in the global rebuttal. We summarize our contributions as follows. Inferential utility of synthetic data is compromised when data are generated with deep generative models (DGMs), even when using earlier proposed correction factors (Decruyenaere et al., 2024). In other related work, procedures were developed relying on multiple synthetic datasets or focussing on only statistical generators (and hence implicitly rely on root-n consistent estimators). In this manuscript, we propose a debiasing strategy that allows to create synthetic data via DGMs while maintaining valid inference for a population parameter.

Line 90, how exactly $S_i$ is sampled? Are you sampling first $A_i$ and $X_i$, and then $Y_i | A_i, X_i$?

In this work, we focus on DGMs but within this setup, the proposed strategy is generator-agnostic. Therefore, the specific sampling procedure depends on the generator that is used in specific cases. In our simulation and case studies, we apply TVAE and CTGAN (Xu et al., 2019) to generate synthetic samples. In these algorithms, the trained decoder (i.e. TVAE) or generator (i.e. CTGAN) are used to generate synthetic samples where all features are sampled jointly as $Y_i, A_i, X_i$ (and thus not conditional as $Y_i | A_i, X_i$).

In our proposed method, we perform a post-processing of the obtained synthetic samples to eliminate the bias that would otherwise occur when the parameter of interest is estimated based on synthetic samples. The exact procedure for this post-processing is explained in Section 3.1 and 3.2. In Section 3.2 it can be seen that we need to estimate e.g. $E_{\widehat{P}_n}(Y|X_i)$ and this requires the generation of synthetic values of $Y$ conditional on a given observed level $X_i$. This can be circumvented by generating, based on the DGM, a very large synthetic sample, taking the subset where $X$ equals $X_i$, and estimating the mean within this subset. If $X$ were continuous, then the DGM would necessitate a built-in conditional sampling module, which is not a given in any type of DGM, and lies beyond the scope of our simulation study.

What is your main theoretical result? Is it Equation 2?

Our main theoretical result can be found in Sections 3.1 and 3.2, where we propose a debiasing strategy for two estimators: the population mean and the linear regression coefficient. In short, we propose to shift the DGM-based synthetic data. The remainder of Section 3 provides the theoretical background for why this debiasing approach works. We understand the confusion given the detailed yet dense explanation within the page limit. In order to prevent future confusion, we will slightly rewrite this part to ensure that the setup of Section 3 is clear. In the remainder, we provide an outline of how we would guide the reader through Section 3.

We first aim at establishing an understanding of the deviation when the parameter of interest is estimated on synthetic versus observed data (i.e. $\theta(\widetilde{P}_m)-\theta(P)$). By doing so, the equation provides us with eight elements that we then further explore. We start with the two remainder terms and the two empirical process terms and elaborate on why we foresee that these terms are negligible under certain conditions. In the next paragraph, we state that we foresee no problems with terms (3) and (4), but that terms (5) and (6) however will cause bias if not taken into account. We then proceed by suggesting a new approach to tackle those two latter terms. For term (5) we rely on theory proposed by van der Laan and Rose (2011) and Chernozhukov et al. (2018) by analyzing the data with debiased estimators.

Our contribution lies in the way we handle term (6). Finding a solution for this bias term relies on the efficient influence curve and therefore on the parameter of interest. In the next two subsections, we specifically worked out the proposed strategy for the population mean and a linear regression coefficient and we pinpoint how bias term (5) and (6) can be resolved. We finalize the Methodology section with stating the properties of the resulting estimator (i.e. the debiased estimator based on synthetic data).

What is $\phi$?

$\phi(\cdot)$ is defined as the efficient influence curve on line 75, but it could enhance readability to repeat this on line 98-99.

How is the deep generative model learned in the first place?

We are not sure that we fully understand this question as we rely on commonly used generators such as TVAE and CTGAN (Xu et al., 2019) to generate synthetic data. They are trained on original data and our proposed strategy does not alter this standard training phase.

Chernozhukov, V., Chetverikov, D. et al. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1–C68.

Decruyenaere, A., Dehaene, H. et al. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In The 40th Conference on UAI.

van der Laan, M. J. and Rose, S. (2011). Targeted Learning. Springer Series in Statistics. Springer New York, New York, NY.

Xu, L., Skoularidou, M. et al. (2019). Modeling tabular data using conditional GAN. Advances in neural information processing systems, 32.

We would like to thank Reviewer eYfd for their careful reading of the manuscript and constructive review.

What would be needed to handle more interesting estimators for downstream applications? It is unclear how one can generalize these to settings where the estimators are much more complex/ubiquitous such as the covariance matrix or group fairness.

Given the importance of this concern, we expand on this topic in the global response to all reviewers. We share the concern that theory should be aligned with practical real-world settings and we advocate that future research should focus on extending our results to broader settings. However, Decruyenaere et al. (2024) have shown that inferential utility is compromised when statistical analysis is performed on synthetic data created by deep generative models (DGMs), even in low-dimensional settings. We see addressing these low-dimensional settings as a necessary first step, before tackling more complex settings where valid inference is already challenging in observed data, let alone synthetic data.

The references are a bit sparse on the synthetic data literature and would it be possible to enhance it with appropriate hooks? There are quite a few surveys available now.

Indeed, it would be useful to contrast our work more extensively with additional recent research on synthetic data in the context of inferential utility. Other reviewers raised the same question and therefore we elaborate on this in the global rebuttal. There, we provide a comparison of our work with related literature, which we will add to the revised manuscript. We are convinced that this addition will improve our manuscript and will better capture how our work stands out w.r.t. other contributions.

While the results are interesting, it is unclear how much is innovative in this work as opposed to building on the results from Chernozhukov, Decruyenaere and other cited works. Can you elucidate that clearly?

For a global overview of related work and our specific contribution, we would like to refer to the global rebuttal. In addition, we will address the specific question related to the work of Chernozhukov et al. (2018) and Decruyenaere et al. (2024).

Our debiasing strategy originates from the observation in Decruyenaere et al. (2024) that inferential utility is compromised when naive statistical analyses are performed on synthetic data created by deep generative models (DGMs), even when applying a previously proposed correction factor to the standard error. They claim that this can be attributed in great extent to the additional layer of uncertainty resulting from the synthetic data generation process and the deviating convergence rate of parameters when estimated in synthetic data, leading to unreliable confidence intervals and overly confident conclusions made from these synthetic data. In our work, we first aim to establish an understanding of this deviation when the parameter of interest is estimated based on a synthetic versus observed sample (i.e. $\theta(\widetilde{P}_m)-\theta(P)$). By doing so, we are able to identify two bias terms which can be targeted in order to decrease this deviation between synthetic and original data.

For the first bias term (referred to as bias term (5) in the manuscript), we rely on theory proposed by van der Laan and Rose (2011) and Chernozhukov et al. (2018), allowing us to analyze the data with debiased estimators. Our contribution lies in the way we handle the second bias term (i.e. bias term (6) in the manuscript). As stated in the manuscript, finding a solution for this bias term relies on the efficient influence curve and therefore on the target parameter of interest. In the first two subsections of the Methodology section, we specifically work out a strategy for the population mean and the linear regression coefficient, showing how our debiasing strategy combines a solution for both bias term (5) and (6). We finalize the Methodology section with stating the properties of the resulting estimator (i.e. the debiased estimator based on the synthetic data).

To summarize, we show how the problems concerning inferential utility can be removed, by targeting synthetic data generated by DGMs to perform optimally for the specific data analysis that is envisaged. This builds on ideas from debiased or targeted machine learning (van der Laan and Rose (2011), Chernozhukov et al. (2018)), but is nonetheless a non-trivial extension as (a) those works do not consider synthetic data; and (b) it required us to work out how the estimation errors in the DGM propagate into the estimator calculated on synthetic data, and based on this, how those data must be adapted to dampen propagation of those errors. As far as we are aware, our approach is the only one that provides formal guarantees for (more) reliable inference from synthetic data created by deep generative models.

Chernozhukov, V., Chetverikov, D. et al. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1–C68.

Decruyenaere, A., Dehaene, H. et al. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In The 40th Conference on Uncertainty in Artificial Intelligence.

van der Laan, M. J. and Rose, S. (2011). Targeted Learning. Springer Series in Statistics. Springer New York, New York, NY.