Does Training with Synthetic Data Truly Protect Privacy?

Dear Reviewer YLFf:

We sincerely appreciate your recognition of our systematic evaluation and find our analysis of visual similarity and logits similarity interesting. We now address your concerns below.

---

Q1: The technical contribution can be somewhat limited.

We would like to kindly clarify that we are not claiming our contribution as proposing a new or stronger MIA method. Our aim is to show that even with existing MIA methods, it becomes evident that there are numerous misleading claims regarding the use of synthetic data in training models.

Breaking the four defenses we analyzed was not particularly challenging. This highlights that before researchers claim their empirical defenses to be privacy-preserving, they should take one additional step - conducting rigorous evaluations, as demonstrated in our work. Such evaluations are not difficult to perform. We hope this inspires future researchers to approach this issue more carefully, which is our primary goal.

Q2: Why synthetic data with similar teacher logits as the original private data harms the privacy of private data and leads to higher MIA metrics.

That’s a great question! We’ll answer it from the following two perspectives:

• Why previous studies suggested that DFKD can protect privacy: Intuitively, since the student model is never trained on private data, it is assumed that releasing the student model should safeguard the privacy of private data.

• Why DFKD actually fails to protect privacy: For the most vulnerable data point in the dataset (denoted as X), the teacher model memorizes this sample strongly. During the distillation process, a synthetic data point X’ (which is visually completely dissimilar to X) may inadvertently trigger the teacher model's memory of X. In the MIA process, if the logits similarity between X and X’ is very high, it implies that during the MI attack, even though the student model never directly saw private data X, it still indirectly learned about X by seeing synthetic data X’. As a result, the membership privacy of X is completely leaked.

Therefore, even if a model has never directly seen private data, it can still potentially leak privacy. We should not rely on data that is visually completely dissimilar to private data as a guarantee of privacy protection. More rigorous evaluations are essential.

Q3: Figure 7 shows that private data that has high confidence in original label and Figure 9 shows the private data that has higher confidence in the mislabel label. Would this show the different canary property for different kind of methods.

That's a great observation! Indeed, canaries should be carefully tailored to different defenses and datasets. For instance, mislabeled data would not be effective canaries for defenses based on self-supervised learning, which does not rely on labels.

In our work, we did not use the most optimized canaries for each defense we evaluated. This means that with better-designed canaries, it's likely that even higher MIA success rates could be achieved. Despite this, our approach of using mislabeled data as canaries already yielded strong results, clearly demonstrating that these defenses provide a false sense of privacy protection.

Thank you for your detailed response.

Q2. I am still somewhat confused about privacy leakage about the logits similarity between teacher's prediction on private image, teacher's prediction on synthetic image, and student's prediction on synthetic image.

Figure 9 shows that the student's prediction on the private image is different than the other three and therefore it is not clear to me what there is a privacy concern for the private image regards to the student model.

Q4 &Q5. thanks for the clarification.

Q6. thanks for your efforts.

For Q1, along with Q2, Q3, Q6, given that there are already prior works showing that evaluations of privacy preserving machine learning methods should consider rigorous evaluation, and the dataset distillation method studied in this work, its privacy leakage is also studied in [2], I wonder if this work could provide some outlines and suggestions how to to design the rigorous evaluations for future research.

[1] Aerni et al. Evaluations of Machine Learning Privacy Defenses are Misleading. CCS 2024.

[2] Carlini et al. No free lunch in” privacy for free: How does dataset condensation help privacy. arxiv 2022

Dear Reviewer ybiF,

Thank you so much for your thoughtful feedback and for taking the time to review our work—we truly appreciate it!

We understand your concern about our experiment being conducted on a single dataset and the lack of theoretical support (please refer to our detailed response below). That said, we’d like to emphasize that many studies in privacy evaluation often present misleading conclusions, and our experiment serves to highlight this critical issue. Through our work, we aim to encourage researchers to adopt more rigorous evaluation practices before claiming their methods are privacy-preserving—a step we believe is incredibly important!

Tackling systemic issues in research methodologies, especially in evaluations, is central to advancing the field. We respectfully ask you to reconsider the potential impact of our contribution, as it has the ability to positively shape future research directions.

Once again, thank you for your valuable feedback. We address your concerns in detail below.

---

Q1: Experiments on more datasets and more network architectures.

Thank you for suggesting reporting results on more datasets and network architectures. While we agree that using more datasets would nevertheless have been interesting, we decided against this:

(1) To do fair comparisons, we needed all studied methods to achieve reasonable test performance. Some methods, such as Dataset Distillation, perform poorly on more complex datasets like CIFAR-100 or ImageNet, which would make comparisons less meaningful.

(2) Similar datasets (e.g., CIFAR-100) Figure 1 alone requires more than 5800 GPU hours (without hyperparameters tuning and additional experiments). We decided that the auxiliary insights were not worth the cost.

(3) Our approach aligns with previous studies like [1], which also conducted comprehensive evaluations of defense methods primarily on CIFAR-10. This consistency facilitates a direct comparison between our results and existing literature.

As our goal is to reveal the false sense of privacy protection of existing methods that train ML models with synthetic data, we believe that it is sufficient to focus on the most standard dataset used in empirical evaluations of privacy defense.

Q2: DP-SGD is resource-intensive. Discussion about computational cost and scalability in real-world scenarios is required.

Our main argument is that DP-SGD can serve as a strong heuristic defense, providing both good utility and empirical privacy protection. In fact, when using a small noise level and setting the privacy budget $\epsilon \gg 1000$, DP-SGD can be more computationally efficient than many other defenses.

The key insight is that, by using small noise, we forgo DP-SGD’s provable privacy guarantee in exchange for significantly higher utility, while also improving efficiency. For training a single model with high-utility DP-SGD in our experiments, it only took 78 minutes. In this way, it is not particularly resource-intensive.

Q3: An in-depth theoretical framework to explain why certain synthetic data techniques lead to privacy leakage.

Thank you for the valuable suggestion. We agree that a theoretical framework is indeed helpful and can offer predictive insights. However, since our focus is primarily on different types of heuristic defenses, which are highly heterogeneous, we believe it is both extremely difficult and not very meaningful to provide a generic theoretical framework for these defenses.

Additionally, none of these methods come with a theoretical foundation themselves, as they are heuristic defenses rather than theoretical defenses.

Q4: Why is the data from coreset selection considered as synthetic data?

Thank you for this question! Our initial idea is that, although coreset selection does not involve a data synthesis process, it shares a common goal with the other synthetic data methods: obtaining an informative proxy training set. The difference lies in the approach—data synthesis methods generate samples directly, while coreset selection automatically selects an informative subset from the original dataset. Therefore, we gave a footnote on page 1 to explain that, for simplicity, we use the term “synthetic data” (also for coreset) in the rest of the paper.

Moreover, we would like to note that coreset selection just serves as a starting point—a very simple method—to demonstrate that average-case evaluations can be misleading. Additionally, through studying coreset selection, there is an interesting finding that the selection or unlearning of specific samples could introduce further privacy leakage—some selected samples exhibit a greater degree of privacy leakage compared to when they are part of the entire training set.

We will also consider moving this discussion to the appendix in the revised manuscript.

[1] "Evaluations of Machine Learning Privacy Defenses are Misleading." CCS 2024.

Dear Reviewer cuks:

Thank you for your insightful comments on our paper. We sincerely appreciate your recognition of how our work advances prior analyses of empirical defenses using synthetic training data. We now address your concerns below:

---

Q1: Comparison with DP-SGD baselines could be misleading as they do not satisfy differential privacy. Could add a baseline that satisfies DP and a non-private baseline.

Thanks for the valuable suggestions and we are sorry for anywhere that is unclear.

It is true that we give up meaningful provable privacy guarantees and view DP-SGD as a purely heuristic defense. This is exactly our point because we need to tune the hyper-parameters of these DP-SGD baselines for higher utilities to ensure fair comparisons with other empirical defenses. Results prove that none of the empirical defenses based on synthetic training data outperforms heuristic DP-SGD baselines in the context of privacy-utility-efficiency tradeoff under similar utility levels. We will improve the writing to clarify and avoid being misleading.

For the non-private baseline, it is reported in Figure 1 and Table 4 named “Undefended”. It represents a ResNet trained with a standard training routine using SGD. TPR@0.1%FPR of the non-private baseline is 100% and the test accuracy is 94.78%. More details can be found in Appendix A.1. We will also improve the writing to introduce it in the main text.

Q2: The paper would benefit from a discussion of how the formal guarantee of differential privacy differs from heuristic privacy - particularly, with respect to DP applying to adding or removing any possible record within the data domain; whereas, the MI attacks in this paper are only considered with respect to the training data.

Thank you for your valuable suggestions on this! We will incorporate more detailed discussions based on your input.

One important point to clarify: we aim to emphasize that DPSGD can serve as a very strong heuristic defense. Unlike canonical DPSGD, where random noise is added to make the privacy budget meaningful (e.g., setting $\epsilon = 1$), our focus is on the fact that even with a privacy budget much larger than 1000, DPSGD remains one of the most effective defenses among the empirical methods we've examined.

Q3: Why are the gridlines in Figure 1 not uniform? Consider changing this or explaining in figure caption.

Thank you for pointing out this! We will address this by updating Figure 1 in the revised manuscript to make gridlines uniform.

Q4: As with [0], are the DP-SGD models reported in Table 4 only non-private due to hyperparameter tuning? If so, what privacy parameters were these models trained with?

Thanks for the question. We indeed tune the hyper-parameters to achieve higher utility (for a fair comparison with other defenses) and forgo provable privacy to construct heuristic DP-SGD defenses. The privacy budgets are $\epsilon \approx 1.8\times 10^8$ for high-utility DP-SGD and $\epsilon \approx 4.4\times 10^7$ for medium-utility DP-SGD. Their hyperparameters are provided in Appendix A.6. We apologize for not mentioning the table of privacy parameters in the appendix within the main text. We will include this in the revised manuscript to be clearer.

Q5: Additional stylistic notes and suggestions.

We appreciate your careful attention to stylistic details and suggestions for improvement. For the typos and odd phrasings, we will correct them in the revised version. Regarding the intuition for why initialization on public data provides privacy guarantees, we will incorporate the suggested citations into our analysis and discussion to strengthen the argument.

Dear Reviewer qB2Z:

We really appreciate your acknowledgment of the solidness of our experiments and recognition of our work as a potentially valuable reference in the topic of privacy protection for synthetic data. In the remainder of this note, we have tried to address your comments and questions in as much detail as possible. Thank you once again for your kind consideration and time. We welcome any further questions or suggestions that could improve the clarity and impact of our paper.

---

Q1: Novelty of our work. Privacy audit on worst-case samples has been extensively studied.

Thanks for providing the references and we will include a discussion of them in the related work section.

We would like to kindly clarify that we do NOT claim privacy auditing on worst-case data to be our main contribution. Our aim is to show that even with existing auditing methods, it becomes evident that there are numerous misleading claims regarding the use of synthetic data in training models.

Breaking the four defenses we analyzed was not particularly challenging. This highlights that before researchers claim their empirical defenses to be privacy-preserving, they should take one additional step - conducting rigorous evaluations, as demonstrated in our work. Such evaluations are not difficult to perform. We hope this inspires future researchers to approach this issue more carefully, which is our primary goal.

We hope that this work can bring these important concerns to the forefront of the community's attention and foster further discussions on avoiding misleading results in privacy.

Q2: Vulnerability of synthetic data against membership inference attacks has been considered in the literature.

Thanks for the questions. We have reviewed the suggested work [1]. It primarily focuses on applying MIA to generative models, with the attack target being to determine whether a specific sample was used to train a GAN. This is quite different from our approach, where we use synthetic data generated from the original private dataset to train classification models. Our goal is to determine whether a classification model trained solely on synthetic data can protect privacy.

Q3: Results on more datasets could be provided, e.g., datasets from other domains.

Thank you for suggesting reporting results on more datasets. While we agree that using more datasets would nevertheless have been interesting, we decided against this:

(1) To do fair comparisons, we needed all studied methods to achieve reasonable test performance. Some methods, such as Dataset Distillation, perform poorly on more complex datasets like CIFAR-100 or ImageNet, which would make comparisons less meaningful.

(2) Similar datasets (e.g., CIFAR-100) Figure 1 alone requires over 5800 GPU hours (without hyperparameters tuning and additional experiments). We decided that the auxiliary insights were not worth the cost.

(3) Our approach aligns with previous studies like [a], which also conducted comprehensive evaluations of defense methods primarily on CIFAR-10. This consistency facilitates a direct comparison between our results and existing literature.

[a] Evaluations of Machine Learning Privacy Defenses are Misleading. CCS 2024.

Q4: Do you think similar comparisons could be easily carried out in different image datasets or in other domains (e.g., tabular data) ?

That's a good point! It's worth mentioning TabDDPM[b] as an example. In this approach, the researchers first train a diffusion model using private training data, then generate synthetic tabular data, and finally train a high-performing classifier using only the synthetic data. While the authors claim this method protects privacy, it hasn't been evaluated under rigorous privacy testing protocols.

I believe this approach requires a more thorough privacy evaluation to validate these claims. We could also consider adding this experiment to the appendix based on your suggestions!

[b] TabDDPM: Modelling Tabular Data with Diffusion Models. ICML 2023.