We appreciate the reviewer’s insightful feedback. Below, we provide detailed responses to each of the raised points.

Response to Weakness 1: more verification of information loss via captioning.

Thank you for the question. We have conducted additional experiments on downstream tasks and evaluated the proposed CMMD metric to further validate that our approach preserves semantic fidelity. The results are presented below in Question 5.

Response to Weakness 2: Missing results for DP-finetuning.

We provide evaluation of DP-finetuning below in response to Question 3.

Response to Weakness 3: Choice of similarity metric.

We provide our argument below in response to Question 4.

Question 1

We're sorry for not implementing downstream tasks, we will provide more systematic experiments in the future. We prepare to use CelebA dataset and conduct a series of 2-class classification tasks according to its features. We're currently running experiments on Wearing_Lipstick feature, and acquired good results. We will update this series of experiments in the future version of our paper.

Question 2

We test our method on various datasets to confirm that our pipeline works well in different circumstances, and experiments did show good results. However, we admit that there are cases where text description doesn't align well with the images. This usually happens when images have such intensive information that VLMs can hardly capture all of them, or limits on text length prohibits VLMs to fully describe all the scenes. Nevertheless, we believe the performance of our method will improve as state-of-the-art VLMs and diffusion models reach better results.

Question 3

Our method focuses on conditions where private data may not be directly accessed. In our method, we refer to private data through it representations. In contrast, DP-finetune has direct access to private data, which can make full use of it. This is the key reason why we focus on comparison between SPTI and PE image, since our method is an improvement on PE image method.

Nevertheless, SPTI performs better than DP-finetune in many circumstances, and we also conducted more experiments after paper submission, and we provide experiment results below.

Question 4

We adopt the same similarity metric used in the Private Evolution (PE) work. Since this component is not the focus of our technical contribution, we retain it unchanged to ensure a fair and consistent comparison with PE.

Question 5

We're sorry for not considering more evaluation metrics before, so we added more evaluation and re-run some of our previous experiments for results. Here we provide experiment results with both FID and CMMD evaluation. The consistency in these two metrics show that our method indeed improves PE image method.

The table below shows CMMD value for SPTI and PE method. In this table, we found our method performs the best except for the case in LSUN Bedroom. We suppose it's because LSUN Bedroom has such a large quantity of image data that improves the performance of fine-tuning, but this needs further experiments. Other conditions show the excellence of our method.

Question 6

We propose that PE image works better than SPTI when the diversity of the private data is too low. For example, SPTI performs poorer than PE image because Cat dataset only involves images of 2 kinds of cat, Maine Coon and Siamese Cat. To validate this idea, we conducted a simple experiment. We build four datasets using images from ImageNet100 dataset, and use exactly the same setting to run our experiment (with $\epsilon=1.0$, num_samples=200, same text prompt and FID metric). We provide its name, data composition and results below.

TwoSpecies_Bird_200 dataset shows the same results as Cat dataset, so TwoSpecies_Bird_200 is capable to be an alternative of Cat dataset. In this table, PE image performs better than SPTI when there's only 2 kinds of bird, no matter how many images are there. In contrast, SPTI performs better than PE image when there are more kinds of bird, no matter the total number of images. This validate our hypothesis, SPTI is suitable for diverse data while PE image suits for image that are less diverse.

Response to Comment: Fair comparison with image-only baselines and missing related work on image-only methods

Thank you very much for pointing out these related literature. We appreciate the reviewer’s thoughtful comment on fairly comparing against image-only baselines, especially the references [1,2,3]. These methods showcase how differentially private generative models can be trained to produce synthetic image datasets le. Specifically:

• [1] and [2] advocate for using public datasets or pre-trained public feature extractors to enhance DP training and achieve improved image quality.
• [3] introduces Sinkhorn divergence as a robust objective for training DP generative models, with promising results on image-only datasets.

Our work shares a similar high-level motivation—leveraging public or existing models to improve private data generation—but advances the field in several notable ways:

1. Methodologically: Our framework can directly utilize state-of-the-art foundation models via API access, without requiring any training or fine-tuning.
2. Conceptually: We bridge modalities by introducing a text-based intermediary representation, which allows us to exploit the strong alignment between vision and language models. This cross-modal design expands the range of generative capabilities and circumvents limitations of purely image-based pipelines.
3. Empirically: To the best of our knowledge, our method is the first to demonstrate the ability to generate high-resolution image datasets with differential privacy guarantees, going beyond the typical CIFAR-10 resolution benchmark.

We will integrate this comparison and discussion into the related work section to clarify how our approach builds upon and differs from these prior contributions.

To Reviewer

We appreciate the reviewer’s insightful feedback. Below, we provide detailed responses to each of the raised points.

Reply to Weaknesses

Weakness 1

w1: There is not a formal proof in the main text or appendix that the proposed mechanism satisfies differential privacy. Moreover, the privacy analysis provided is insufficient. Most results such as the post-processing property are not clearly stated and are missing proper citations. Though stated on Line 188, the Advanced Composition Theorem (for Approx. DP) is not used for privacy accounting since the authors state that the privacy analysis uses RDP rather than Approx. DP.

WA1: We provide a detailed privacy analysis below. This analysis provides a tighter bound for DP guarantee, and is updated to the latest version of our paper.

Privacy Analysis

We analyze the privacy guarantees of our proposed SPTI framework, which synthesizes image data under differential privacy (DP) by operating in the text space. Our goal is to ensure that the final synthetic images $\mathcal{D}'$ are generated through a process that satisfies $(\varepsilon, \delta)$-DP with respect to the private dataset $\mathcal{D}$.

Privacy-Critical Step. The only step in the SPTI pipeline that accesses the private data is the Private Evolution module, specifically through the similarity-based voting mechanism in algorithm Aug_PE_Image_Voting. This module operates on embeddings of the private image dataset $\mathcal{D}$ and is used to guide the sampling of synthetic text candidates. Thus, our privacy analysis focuses on this step.

Voting Mechanism. For each private image embedding, we identify its nearest neighbor among the generated image embeddings , and increment a vote count histogram $H$ indexed by the candidate text that produced the nearest image. This step corresponds to a histogram query over the private data.

To ensure differential privacy, we add Gaussian noise $\mathcal{N}(0, \sigma^2 \mathbf{I})$ to the vote histogram $H$ , which is equivalent to applying the Gaussian mechanism to a function with bounded sensitivity.

Sensitivity Analysis. The histogram query has $L_2$ sensitivity at most $1$, since each private image contributes a vote to only one candidate (i.e., changing a single image can affect the histogram by at most 1 in one coordinate). This satisfies the precondition for the Gaussian mechanism.

According to the Gaussian mechanism \cite{balle2018improving}, adding noise of variance $\sigma^2$ per coordinate to a function with $L_2$ sensitivity $1$ ensures $(\varepsilon, \delta)$-DP, provided:

Privacy Composition. According to the adaptive composition theorem of Gaussian mechanisms \cite{dong2019gaussian}, applying the above Gaussian mechanism across $G$ iterations satisfy $(\varepsilon, \delta)$-DP, provided:

Post-processing Immunity. All downstream steps in SPTI—including text mutation and image generation via a fixed diffusion model—depend solely on the privatized output of the voting mechanism. By the post-processing property of DP, these steps incur no additional privacy cost.

Overall Guarantee. Hence, SPTI satisfies $(\varepsilon, \delta)$-differential privacy with respect to the private image dataset $\mathcal{D}$, provided that the noise scale $\sigma$ is properly chosen to satisfy equation above.

Weakness 2

w2: In the limitations section, the authors mention that the proposed method requires significantly more resources than baseline Private Evolution since the former requires several calls to muti-modal models; yet, there is no comparison of running time or token usage between the models.

WA2: Thank you for raising this point. We have conducted a systematic evaluation of the computational cost of our pipeline and will include the results in the next version of the paper.

Our full data generation process involves two main components: an 8B-parameter large language model (LLM) and a 3.5B-parameter diffusion model. In practice, a complete generation run takes approximately 19 hours on a single NVIDIA A800, with a peak memory footprint of around 70 GB. In comparison, the Private Evolution baseline uses a 3.5B-parameter diffusion model, does not involve an LLM, and completes in about 7 hours on a single NVIDIA A800, with a peak memory usage of approximately 35 GB.

While our method introduces a modest increase in computational cost—roughly doubling GPU memory and requiring ~2.7× runtime—this is a marginal trade-off given the qualitative generation gains. By leveraging multi-modal language-guided synthesis, our approach significantly improves semantic richness, data diversity, and downstream utility, justifying the additional resources.

Weakness 3

w3: There are several grammar mistakes throughout the text.

WA3: We went through our paper again and fixed most grammar mistakes.

Reply to Questions

Question 1

We ran more experiments on DP-finetune, and provide comparison between SPTI, PE Image and DP-finetune below. We fix the experiment settings as the same, with $\epsilon=1.0$, num_samples=2000. It's noticeable that we updated chat template for LLM, so we re-run all the previous experiments. Still, new results show that our method performs better than PE image and DP-finetune.

Question 2

PE image works better than SPTI when the diversity of the private data is too low. For example, SPTI performs poorer than PE image because Cat dataset only involves images of 2 kinds of cat, Maine Coon and Siamese Cat. To validate this idea, we conducted a simple experiment. We build four datasets using images from ImageNet100 dataset, and use exactly the same setting to run our experiment (with $\epsilon=1.0$, num_samples=200, same text prompt and FID metric). We provide its name, data composition and results below.

TwoSpecies_Bird_200 dataset shows the same results as Cat dataset, so TwoSpecies_Bird_200 is capable to be an alternative of Cat dataset. In this table, PE image performs better than SPTI when there's only 2 kinds of bird, no matter how many images are there. In contrast, SPTI performs better than PE image when there are more kinds of bird, no matter the total number of images. This validate our hypothesis, SPTI is suitable for diverse data while PE image suits for image that are less diverse.

To Reviewer

We appreciate the reviewer’s insightful feedback. Below, we provide detailed responses to each of the raised points.

Reply to Weaknesses

Weakness 1

W1: The interdependencies among these components make it challenging to assess their impact on overall performance. In Section 4.3, the ablation study shows that the method is sensitive to changes in the image generation models.

WA1: This statement is inaccurate. In addition to analyzing the image generation models, we also conducted a sensitivity analysis on the image captioning component. Overall, the proposed method is not particularly sensitive to the choice of these base models, provided that each performs its respective task effectively.

Weakness 2

W2: Using image-to-text as an intermediate representation has been explored in some existing works [1]. Although the application scenarios differ, the underlying idea is similar.

WA2: Thank you for referring to paper [1], which proposes a label-free evaluation framework for image captioning models. The core idea is that a well-written caption should enable a text-to-image diffusion model to reconstruct an image visually similar to the original.

While the use of an image-to-text-to-image pipeline is conceptually related and increasingly common, its application to the task of private dataset synthesis is nontrivial and remains unclear. Our work differs substantially in terms of motivation, privacy guarantees, and evaluation methodology:

Objective: Paper [1] aims to assess the quality of image captioning models. In contrast, our work is centered on synthesizing datasets that preserve data utility while offering rigorous differential privacy (DP) guarantees.

Evaluation Metric: The prior work evaluates the similarity between individual original and reconstructed images. We, however, focus on measuring the similarity between the distributions of the original private dataset and the generated dataset.

We will incorporate this paper, along with other relevant literature, into the related work section and clearly articulate how our approach differs.

Weakness 3

W3: It is unclear whether text-based representations are always sufficient to preserve fine-grained image details.

WA3: We acknowledge that, for static evaluation, we cannot provide a definitive answer to whether text-based representations are always sufficient to preserve fine-grained image details. However, our approach demonstrates a clear improvement in preserving such details compared to existing methods. Furthermore, from a longer-term perspective, we believe this challenge will become increasingly addressable as foundation models continue to evolve. Our method is designed to seamlessly benefit from these advancements, allowing it to leverage future improvements in underlying models with minimal effort.

Weakness 4

W4: The baseline comparisons are incomplete.

WA4: Thanks for the suggestion, we add the baseline as follows, and provide comparison between SPTI, PE Image and DP-finetune below. We fix the experiment settings as the same, with $\epsilon=1.0$, num_samples=2000. It's noticeable that we updated chat template for LLM, so we re-run all the previous experiments. Still, new results show that our method performs better than PE image and DP-finetune.

Also, we will run more experiments with different settings to fully establish the excellence in our method.

Weakness 5

W5: The DP guarantee is limited to the Private Evolution voting step, while other processes, particularly image-to-text captioning, access private data without DP protection. Although the paper cites the post-processing property of DP as justification, this design choice warrants further examination.

WA5: PE process refers to the private data, and synthesize data that satisfies DP constraints. In our pipeline, we refers to captions, and generates image data that satisfies DP constraints, so it's fine for image-to-text captioning to access private data.

Weakness 6

W6: Computational costs are not fully discussed. WA6: We provide details of computational costs below in Question 4.

Weakness 7

W7: The paper does not fully analyze failure modes or limitations in the semantic fidelity of the generated images, especially when the text descriptions do not align well with the images in complex scene data.

WA7: We test our method on various datasets to confirm that our pipeline works well in different circumstances, and experiments did show good results. However, we admit that there are cases where text description doesn't align well with the images. This usually happens when images have such intensive information that VLMs can hardly capture all of them, or limits on text length prohibits VLMs to fully describe all the scenes. Nevertheless, we believe the performance of our method will improve as state-of-the-art VLMs and diffusion models reach better results.

Reply to Questions

Question 1

When using text-voting in our pipeline, the quality of generated caption determines the upper bound of DP generated data, since text samples are iteratively getting closer to private caption in text-voting. However, when using image-voting in our pipeline, text samples are directly referring to representation of private images, which actually doesn't need caption text for reference, so the overall performance is not affected by caption models.

Notably, we uploaded a new chat template for the LLM we use and run experiments after paper submission, and found image-voting still performs better than text-voting even in Cat dataset, which argues that image-voting performs generally better than text-voting.

However, as text-voting may be necessary in some condition, we run systematic experiments with text-voting with fixed settings. The following table shows the results. We will provide more comparison in our newest version of paper.

Question 2

It would work fine. Actually, many of the experiments conducted are using prompts like 'An image of ...', and received good results.

As for privacy concerns, we assert that all the text prompts used in our experiments can be regarded as public information, which doesn't violate privacy. For example, if we consider LSUN Bedroom is a private dataset collected from sensitive user data, then 'this dataset is about bedroom' should be considered as public information, since the real sensitive part is the detail.

Question 3

Thanks for pointing out the deficiency in our work. We'll use UMAP for visualizing our results, and CMMD as an alternative metric other than FID. CMMD also show results better than PE Image method.

UMAP shows our results align well with private data, and the visualization is uploaded in our newest version of paper. However, we can only provide our results in text, so only more results in CMMD is provided. In this table, we found our method performs the best except for the case in LSUN Bedroom. We suppose it's because LSUN Bedroom has such a large quantity of image data that improves the performance of fine-tuning, but this needs further experiments. Other conditions show the excellence of our method.

Question 4

Our full data generation pipeline involves two major components: an 8B-parameter large language model (LLM) and a 3.5B-parameter diffusion model. Under our current implementation, a complete data generation run takes approximately 19 hours on a single NVIDIA A800, or 15.5 hours on 2×NVIDIA A100 GPUs, with a peak memory requirement of about 70 GB.

In comparison, the baseline method uses only a single 3.5B model, requires no LLM, and completes in about 7 hours on a single NVIDIA A100 GPU, with a peak memory requirement of about 35 GB. While our method has higher compute demands, it introduces a semantically enriched generation process through language-guided synthesis, which we show leads to superior data diversity and downstream performance.

We believe the additional cost is justified by the significant gains in quality and generalization, and we note that our pipeline can be modularly optimized (e.g., via model distillation or prompt caching) in future work to reduce the runtime.

As for API usage, caption process will take 250M tokens for 10,000 images. During data generation, LLM will process 7.3M tokens during each iteration, and diffusion model will process 5M tokens in each iteration. A total of 10 iterations will count for 123M tokens.