PCEvolve: Private Contrastive Evolution for Synthetic Dataset Generation via Few-Shot Private Data and Generative APIs

We sincerely appreciate your time and effort in reviewing our work! Below, we provide responses to address your concerns and suggestions, with Lines xxx and Section x referring to specific parts of our paper. We hope these clarifications effectively resolve your concerns, and we also thank you for your constructive feedback!

Concern 1: Could the pre-trained dataset of a generative model overlap with sensitive datasets?

We verified that the following pre-trained datasets of our generative models do not overlap with our private few-shot data by cross-referencing unique identifiers and metadata, and using hash comparisons. Specifically:

• Stable Diffusion (SD) is pre-trained on the public multi-modal dataset LAION-2B [1].
• SD+IPA is pre-trained on the same LAION-2B dataset, supplemented with the public multi-modal dataset COYO-700M.
• OpenJourney is pre-trained on LAION-2B and an additional set of over 100K images from Midjourney v4 (a commercial image generative model API).

Reasonably, the images we used to construct the few-shot specialized domains (medical and industrial images) are originally images without corresponding textual data and, therefore, cannot be added to these multi-modal datasets.

[1] Schuhmann, Christoph, et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." NeurIPS 2022.

Concern 2: Is it correct that we can generate as many images as we want after the final iteration of PCE? If so, does generating more synthetic images enhance downstream classification performance?

• The answer to the first question is YES. We can generate as many images as needed using PCE.

• The answer to the second question is NO. We have demonstrated this in Section CAS w.r.t. N-Shot Synthetic Data and Figure 4, where we show that the utility of the synthetic dataset decreases when generating more than 150 images per class, given noisy data generation APIs, a fixed privacy budget ϵ*, and cost in few-shot private data scenarios. Specifically:

  1. Noise: As the number of synthetic images increases, the valuable information plateaus due to the limited few-shot private data, while noise continues to increase. Since current generative models cannot produce content free of noise, finding a balance is essential to optimize the utility of the synthetic dataset.

  2. Privacy: PCE's privacy protection relies on differential privacy, constrained by a given privacy budget ϵ*. Each PCE iteration consumes a portion of the privacy budget, so PCE will stop once the total privacy budget exceeds ϵ*, meaning that PCE cannot run indefinitely. Thus, the total number of synthetic images is finite when the number of synthetic images is fixed in each PCE iteration.

  3. Cost: Each image generation requires a request to the image generation API service, which incurs a corresponding cost. If cost is not an issue, it is possible to generate as many images as needed in each PCE iteration.

Suggestion 1: Performance on a Private Dataset with More Classes but Fewer Images per Class to Further Show the Advantage of the Contrastive Filter

We thank you for the suggestion to consider a private dataset with more classes to highlight the advantage of our contrastive filter in our PCE.

• Private datasets in specialized domains typically have only a few classes, such as positive vs. negative, which is common in fields like medicine. The few-shot issue is particularly prevalent in these specialized domains. After a thorough survey of the literature, we selected four representative specialized datasets used in our paper, each of which has a limited number of classes.
• We have included the Cifar100 dataset in R-Table 1 with other settings fixed, where our PCE outperforms PE by 10.46% in accuracy. This improvement is greater than that observed with existing datasets in Table 1, demonstrating the advantage and adaptability of our contrastive filter, even though Cifar100 is not from a specialized domain.

R-Table 1: Top-1 accuracy (%) on Cifar100 (10 private samples per class)

Suggestion 2: Consider Using Approx-DP Instead of Pure-DP for Stronger Composition and Improved Results

We thank you for the suggestion of considering approx-DP to further improve our PCE.

While stronger privacy budget compositions for approx-DP may exist, we chose to use the same composition method as the baseline PE for a fair comparison when calculating the total privacy budget. Using pure-DP in PCE ensures that utility improvements are attributable solely to our methodological innovations rather than differences in privacy accounting. Additionally, pure-DP can provide stronger differential privacy guarantees than approx-DP under the same $\epsilon$. Improving PCE with approx-DP could be explored in the future.

We sincerely appreciate your time and effort in reviewing our work! Below, we provide responses to address your concerns, with Lines xxx and Section x referring to specific parts of our paper. We hope these clarifications effectively resolve your concerns.

Concern 1: Generalization to Broader Domains

• As mentioned in Section Introduction, we focus on the few-shot issue, which is a critical challenge especially in specialized domains such as medical fields. Therefore, we prioritize these domains to showcase the value of our PCE. Furthermore, publicly available specialized pre-trained large models are scarce, making PCE— which relies solely on general pre-trained large model APIs— even more valuable. Our PCE can be applied to any domain to address the few-shot issue, as the underlying concept of using additional inter-class contrastive information is general and adaptable.
• We include a dataset from the natural domain with a larger number of classes while keeping other settings fixed and demonstrate that our PCE can also generalize to the natural domain in R-Table 1.
• Large-scale datasets are out of scope, as our paper focuses on few-shot scenarios.

R-Table 1: Top-1 accuracy (%) on Cifar100 (10 private samples per class)

Concern 2: A Larger ϵ* Range Such as ϵ* ≤ 1

• We follow PE in using ϵ* near 10 (see Section 5.1.2 in PE's paper) for specialized domains like medicine to ensure a fair comparison.
• We have provided results for different ϵ* values ranging from 4 to 20 in Appendix B, showed the robustness of our PCE, and analyze their impact on the privacy-utility trade-off. As shown in Table 6, ϵ*=8/10 emerges as the optimal choice for balancing privacy and utility, where the accuracy decreases greatly when ϵ*<8 but increases slightly when ϵ*>10, since the private information is limited for few-shot scenarios.
• We include results with a wider range of ϵ* in R-Table 2, demonstrating that our PCE consistently outperforms PE.

R-Table 2: Top-1 accuracy (%) with more values of ϵ on Camelyon17*

Concern 3: Empirical Attack Results

• First of all, Differential Privacy (DP) techniques, such as the Exponential Mechanism (EM), have been widely proven to resist empirical attacks [1]. Since the privacy protection ability of our PCE is supported by DP, implemented via the EM in PCE, our approach is guaranteed to resist empirical attacks. Our primary focus is on addressing the few-shot issue when applying DP techniques.

• We incorporate a membership inference attack (MIA) for COVIDx, where the attack model achieves 50.86% success with DP and 65.27% without, showing PCE's protection ability to such attacks. Specifically, we select shadow data from the public covid-chestxray-dataset, which is similar to COVIDx, and train an attack model until convergence using the ResNet-18 [2] architecture with randomly initialized parameters and default settings of SGD.

[1] Nasr, Milad, Reza Shokri, and Amir Houmansadr. "Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning." IEEE symposium on security and privacy (SP), 2019.

[2] He, Kaiming, et al. "Deep residual learning for image recognition." CVPR 2016.

Concern 4: Computational Cost on Clients

Our PCE can be applied to real scenarios across resource-constrained clients (Line 258), as the client-side computational cost is negligible, limited to feature extraction (with a small encoder and few-shot private data) and minimal distance calculations. R-Table 4 further demonstrates this. The API service requires no local computational resources.

R-Table 3: Total time cost (seconds) under 20 iterations.

We sincerely appreciate your time and effort in reviewing our work! Below, we provide responses to address your concerns and suggestions for extensions, with Lines xxx and Section x referring to specific parts of our paper. We hope these clarifications effectively resolve your concerns. We also thank you for your creative suggestions for extensions!

Concern 1: Additional Privacy Budgets for ϵ*

• We follow PE in using ϵ* near 10 (see Section 5.1.2 in PE's paper) for specialized domains like medicine to ensure a fair comparison.
• We have provided results for different ϵ* values ranging from 4 to 20 in Appendix B, showed the robustness of our PCE, and analyze their impact on the privacy-utility trade-off. As shown in Table 6, ϵ*=8/10 emerges as the optimal choice for balancing privacy and utility, where the accuracy decreases greatly when ϵ*<8 but increases slightly when ϵ*>10, since the private information is limited for few-shot scenarios.
• We include results with a wider range of ϵ* in R-Table 1, demonstrating that our PCE consistently outperforms PE.

R-Table 1: Top-1 accuracy (%) with more values of ϵ on Camelyon17*

Concern 2: At what dataset size does PE overtake PCE?

We analyzed the impact of the few-shot dataset size ($K$) in Section CAS w.r.t. K-Shot Private Data and Figure 3, finding that PCE maintains superiority over PE when $K \leq 100$. In R-Table 2, we extend our analysis to a larger $K$ and observe that PE outperforms PCE slightly (0.51%) when $K \ge 1000$. However, $K \ge 1000$ does not represent a few-shot setting, while PCE is best suited for few-shot scenarios.

R-Table 2: Top-1 accuracy (%) with more private data (large $K$) on KVASIR-f

Suggestion for Extension 1: Parallel Composing for DP

We really appreciate your suggestions on strengthen our method with parallel composition.

However, since our PCE addresses the few-shot issue by leveraging inter-class contrastive information through the contrastive filter ($g$), it requires access to private data across all classes when evaluating each synthetic sample, making parallel composition challenging. Therefore, we believe that parallel composition deserves a separate paper in the future.

Suggestion for Extension 2: Combining PE and PCE for Class Imbalance Scenarios

We thank you again for your suggestions on combining PE and PCE for class imbalanced data.

Following your suggestion, for class imbalanced scenarios, we combine PE and PCE, terming it PE+PCE. Specifically, in PE+PCE, PE is applied to the majority private classes, while PCE is used for minority (few-shot) classes. We compare PE, PCE, and PE+PCE in a class imbalanced experiment as follows. Specifically, we design an imbalanced dataset with 1,000 samples from class 0 (normal) and 10 samples from class 1 (breast cancer with tumor tissue) based on the Camelyon17 dataset, while keeping all other settings unchanged (still generating 100 samples per class). The results can be found in R-Table 3.

In this class-imbalanced experiment, both PE and PCE perform better with more private data, but PCE still outperforms PE. Since one class remains few-shot, PE suffers from noise overwhelming the actual votes, leading to nearly random similarity voting and selection for the synthetic data in this minority class. In contrast, PCE alleviates this few-shot issue, providing more informative scores to evaluate the synthetic data. By leveraging PE's advantage in majority private classes and using PCE to address the issues in minority (few-shot) classes, PE+PCE performs the best.

However, simply replacing PCE with PE for majority private classes limits improvement by losing inter-class contrastive information. Future work needs a more creative combination of PE and PCE to fully leverage their advantages.

R-Table 3: Top-1 accuracy (%) on class-imbalanced Camelyon17

Suggestion for Typos

Thank you for pointing out the two typos. We will correct them in the revised version.

We sincerely appreciate your time and effort in reviewing our work! Below, we provide responses to address your concerns, with Lines xxx and Section x referring to specific parts of our paper. We hope these clarifications effectively resolve your concerns.

Concern 1: An Ablative Analysis

We have only two components: the contrastive filter ($g$) and the similarity calibrator ($h$). In Section Ablation Study (page 8), we have provided an ablative analysis of these two components. We remove each component individually to evaluate their effects. As shown in Table 3, eliminating either $g$ or $h$ results in a significant performance drop. $g$ and $h$ must function together to address the few-shot issue; neither can be used in isolation. While $g$ leverages inter-class contrastive information, it introduces a trade-off by sacrificing similarity measurement, which is then compensated by $h$.

Concern 2: Other APIs for More Comprehensive Results

• The reason for choosing Stable Diffusion, SD+IPA, and OJ is that diffusion models dominate image generative models, and these three are among the most widely used and recognized approaches [1].

• We add a structurally distinct Transformer-based API based on FLUX.1-dev, along with the closed commercial API GPT-4o, and show our PCE's generality to these new distinct APIs in R-Table 1 and R-Table 2.

R-Table 1: Top-1 accuracy (%) with FLUX.1 API

R-Table 2: Top-1 accuracy (%) with GPT-4o API

[1] Croitoru, Florinel-Alin, et al. "Diffusion models in vision: A survey." IEEE Transactions on Pattern Analysis and Machine Intelligence 45.9 (2023).

Concern 3: Impact of Privacy Parameters ϵ*

• We follow PE in using ϵ* near 10 (see Section 5.1.2 in PE's paper) for specialized domains like medicine to ensure a fair comparison.
• We have provided results for different ϵ* values ranging from 4 to 20 in Appendix B, showed the robustness of our PCE, and analyze their impact on the privacy-utility trade-off. As shown in Table 6, ϵ*=8/10 emerges as the optimal choice for balancing privacy and utility, where the accuracy decreases greatly when ϵ*<8 but increases slightly when ϵ*>10, since the private information is limited for few-shot scenarios.
• We include results with a wider range of ϵ* in R-Table 3, demonstrating that our PCE consistently outperforms PE.

R-Table 3: Top-1 accuracy (%) with more values of ϵ on Camelyon17*

Concern 4: Qualitative Metrics

• Using CAS (top-1 accuracy) aligns with our goal. Our objective is to achieve high quantitative accuracy in downstream models, which directly reflects the value of synthetic data for these tasks. As mentioned in our paper (Lines 254-256), the CAS metric (top-1 accuracy) is widely used for assessing the quality of synthetic datasets in downstream tasks. This is also supported by prior influential work [2], which states that "traditional GAN metrics such as Inception Score [3] and FID are neither predictive of CAS nor useful when evaluating non-GAN models."
• We have provided visual analyses with high perceptual quality in Section Synthetic Images (page 7). From Figure 6, our PCE can generate visually high-quality images that align with downstream tasks. The FID (↓) value on MVAD-l is 8.47 for PCE but 58.14 for PE.
• Metrics for perceptual quality pose risks. While visually high-quality synthetic data may appear realistic, it can exhibit high similarity (low diversity) to private data, reducing its informativeness and utility for downstream tasks.
• Our PCE is also effective in diversity metrics.
  • Our PCE is designed to enhance diversity. The core of PCE lies in its contrastive filter ($g$), which explicitly improves class discrimination.
  • On MVAD-l, PCE achieves 12.51, whereas PE only reaches 82.17 in the Inception Score (↓).

[2] Ravuri, Suman, and Oriol Vinyals. "Classification accuracy score for conditional generative models." NeurIPS (2019).

[3] Chong, Min Jin, and David Forsyth. "Effectively unbiased fid and inception score and where to find them." CVPR 2020.

Concern 5: Computational Cost

Our PCE can be applied to real scenarios across resource-constrained clients (Line 258), as the client-side computational cost is negligible, limited to feature extraction (with a small encoder and few-shot private data) and minimal distance calculations. R-Table 4 further demonstrates this. The API service requires no local computational resources.

R-Table 4: Total time cost (seconds) under 20 iterations.