We thank the reviewer for the many detailed comments that help improve the paper. In particular, we thank the reviewer for recognizing the novelty of the PCC-based metric and encouraging us to elaborate on it.

Elaboration of the PCC-based metric: We agree that it is helpful to have a deeper elaboration on the distinction between model-free and the PCC-based metric, as well as an enriched explanation on why one would like to have the model-based metrics. To achieve that, we will add a summary of our responses below to the Introduction and Section 3. The response below is also our point by point response to the specific questions raised by the reviewers.

Whether the evaluation is holistic or not depends on whether the metrics provide coverage of the full joint distribution. This means that evaluating only marginal and pairwise relationships would not provide a holistic perspective, because higher order statistics such as 3-way interactions would not be captured by the evaluation. Using metrics that target the full joint distributions (i.e., LOO or full-joint) could in principle provide a holistic picture. This is true regardless of whether the metrics are model-free, model-based, or a combination of the two. The issue with the model-free metrics is that they are often noisy, especially towards the RHS of the spectrum for the LOO and full-joint metrics, as shown by the large error bars and non-monotonicity in panels A and C of Figures S1–S3. The large error bars and non-monotonicity originate from (1) the use of multiple ML models with different expressiveness, as well as (2) the different types of estimators and scores used for different distributional properties and data types. For example, the full-joint metrics were evaluated indirectly via a discriminator, which also included a linear version (i.e., logistic regression) and a simple nonlinear one (i.e., SVC). In other words, the model-free metrics are incoherent in terms of the estimators (e.g., the underlying ML models used) and scores. In contrast, the model-based metrics neither have the incoherence problem, nor exhibit the noisy and non-monotonicity, as evidenced in panels B and D of Figures S1–S3. Thus, the inherent advantage of the model-based technique is the coherence in methodology, that is, all the metrics can be derived from the same type of estimator and score.

The reviewer is correct that the model-based technique puts the evaluation on another synthetic standard. The worry is then whether this synthetic standard would bias the evaluation. The degree of bias depends on how much the surrogate model would mask statistical properties because of a lack of expressiveness. To illustrate, if the surrogate model captures only the marginal distributions but not the pairwise and higher-order dependencies, it would assign similar likelihoods to the PERM baseline and the real dataset. We have two pieces of evidence that PCC does not suffer from such lack of expressiveness. First, PCC is consistently a top data synthesizer as evaluated by the model-free metrics (panel A in Figure S1–S3). Second, the model-based evaluation derived from PCC does not always put itself as the highest performing model but can capture superiorities of other techniques (panel B in Figure S1–S3). In fact, the ordering of the performance of the different data synthesizers is consistent between the PCC-based and model-free evaluations.

We thank the reviewer for noting that our approach is systematic and our evaluation is comparative, as well as for all the comments that help improve the paper.

To improve the readability of the paper, we now make a cleaner separation between plain English and math notations throughout Section 3 and Appendix A. We hope that this would help lessen the cognitive load and make the reading feel less dense.

We thank the reviewer for asking the question about practical implications and impact on real-world applications. We provide the following main points:

• Synthesizer selection: The structured evaluation provides a complete and coherent picture of the performance of different synthesizers on a given dataset. This allows a practitioner to select the best data synthesizer with confidence, knowing the degree and extent to which the full joint is mimicked. The HALF baseline also provides a rough estimate on how far the data synthesizer is from being very good.
• Synthesizer improvement: The structured evaluation with the baselines show where along the structural spectrum a data synthesizer falls short. This could expose particular failure modes and facilitate better design of the data synthesizer.
• Evaluation improvement: The structured spectrum suggests that one could design many new metrics, such as 3-way interactions and leave-n-out (as suggested by Reviewer fbzR), and know where to position them relative to the other metrics. Furthermore, one could investigate the error bars and non-monotonicity of the ordered metric groups to identify problematic metrics. Finally, thinking about the chain of sufficiency could help the selection of estimators to use (e.g., which ML models to use in ML efficacy).
• Human-centered debugging: The structured framework provides a nice, ordered visualization for people to spot strange behaviors more easily. An example is given at the end of this response related to the comments on “qualitative insights or user-based evaluations.” We will add a summary of the above points towards the end of the paper.

We echo the reviewer’s sentiment on testing our approach on large-scale datasets. This is of course difficult to realize because computational resources are real constraints. We would also like to point out that the census dataset we analyzed is considered a sizable tabular dataset. Many seemingly larger datasets are larger because the categorical columns have been preprocessed to use the one-hot encoding. Nevertheless, by looking at the three datasets of different sizes analyzed in the paper (Figure 2), we can already make some qualitative inferences about the performance of the methods:

• Deep-learning methods generally become better as the dataset size increases.
• Methods that are not designed to capture the full-joint distribution (i.e., GaussianCopula) generally become worse as the dataset size increases.
• The performance of structurally based methods (synthpop and PCC) seems to be less dependent on dataset size. Some of these were mentioned in the paper on page 8, but we will make sure all the above points are covered.

We definitely agree with the reviewer that “qualitative insights or user-based evaluations might provide a more holistic view of synthesizer effectiveness.” We think that the structured evaluation would aid user-based evaluations by helping the users to more quickly drill down to where the problems might be. For example, in the updated Figure 2 first row, second column, GReaT has a suspiciously low performance in the Missingness group. One can then drill down to see that the cause is due to GReaT generating missing values in columns that do not have any missing values in the original data (mentioned in page 8, second last paragraph of the original manuscript).

We thank the reviewers for the helpful comments to make the paper more accessible and useful.

On rewriting and implementation feasibility: To help with readability, we will revise Section 3 such that the intuitive description in plain English is well separated from complex mathematical notation. We now also make sure the main text mentions all the metrics in Table 1 with references to the Appendix. Our intention in presenting Table 1 was not to provide the recipe for constructing a structured evaluation but to provide a summary of the structured framework and what it covers. Upon acceptance, we will open source our implementation, as mentioned in the list of contributions, so that any reader can use and build on it, and to ensure feasibility. To emphasize this point, we add in the caption to Table 1 that the code is at <url TBA>. Given this readily available codebase, we believe it reasonable and important to use ALL the metric groups instead of a subset of them. Section 5 and Figure 2 show precisely how all the metrics can work together to form a coherent and complete picture of the evaluation via the naturally ordering induced by the structured framework.

On ML efficacy and LOO: We thank the reviewer for bringing up these questions. If the target includes multiple columns, we can simply generalize the leave-one-out (LOO) to leave-n-out (LNO). The implication on how LNO is related to the objective of Q = P is the same as LOO, because all the dependencies described by the chain rule factorization would still be covered. Its natural ordering would be after the LOO and before the full joint when added to Figure 2. The reviewer is correct that if ML efficacy is tested on a single column, it would not be sufficient. This is why in our implementation we loop through all the columns. We now make a note that this is a slight generalization from how people usually use ML efficacy in the Leave-one-out subsection. Lastly, the binary problem is a special case of the classification problem with 2 classes, so the argmax would cover the binary case. Appendix A.3 lists the ML models used for both the binary and categorical classification. The reviewer is correct that the argmax would not be used for a regression problem. We did not include ML regressors because their measure of accuracy (i.e., RMSE) does not fall inside the range of [0,1], and hence, not in the right scale to be averaged with the other metrics. To fill this lack, the model-based LOO metrics is agnostic to column types and do cover all the columns. This is a strength of the model-based metrics that we now highlight explicitly in various parts of the paper.

On sufficiency: We thank the reviewer for pointing out the vagueness about the sufficiency related to the joint-distribution metric. Because we cannot show sufficiency, we now say it is likely not sufficient because there may be multiple ways that the implicit $f_q$ and $f_p$ can produce a discriminator to achieve perfect ROC AUC. The condition is necessary for Q = P, which we mention in the main text just before the text quoted. The approach we show necessity and/or sufficiency was outlined in the “Sketch of analysis” subsection in the beginning of Section 3. We now add a few sentences in that sketch to describe in more detail our approach to show necessity and/or sufficiency.

On the HALF baseline: The idea of the HALF baseline is to split the real dataset into two, and then run the evaluation to compare the two datasets. It is useful because this ensures that both datasets are from the real data generating distribution, yet not direct copies of each other. In other words, the HALF baseline is the closest thing to meeting the proposed objective of Q =P and S $\neq$ X. Thus, it should provide the desirable target to the metrics. We agree that “upper bound” is probably not the right word for the HALF baseline, because the SELF baseline beats it. We now change the “upper bound” associated with the HALF baseline to the “target value” in the text. For the SELF baseline, the upper bound on fidelity metrics follows from the fact that the synthetic and real distributions are equal (Q=P), and the lower bound on the privacy metrics follows from the fact that the synthetic dataset is a direct copy of the real dataset (S=X). How the metrics relate to Q=P is summarized in the Implementation column in Table 1. Also, the second row of Figure 2 shows that empirically, the SELF baseline is an upper bound, and the HALF baseline is also higher or equal to the methods.

On pp-plot: A pp-plot is a plot that plots two cumulative distributions against each other. It is used to detect how different the two distributions are. If the two distributions are the same, the plot will follow the x=y line. We will revise Section 3.3 to clarify this.

We have just updated the PDF. The updated version now includes all the changes we mentioned in the responses. Updated text are colored in blue (except for the content in Table 1).