When Model Knowledge meets Diffusion Model: Diffusion-assisted Data-free Image Synthesis with Alignment of Domain and Class

Thank you for providing meaningful feedback. We appreciate your thoughtful consideration. We have tried our best to provide responses to address your comments.

Claims and Evidence

Please refer to our response W1) to Reviewer #83BW for a detailed explanation. As a result of supporting the above limitation of previous works, prior methods fail to approximate the original data distribution, resulting in low-quality synthesized images shown in Figure 2-5 in the main paper. This mismatch also negatively impacts performance in downstream applications, shown in Table 2 and Figure 6 in the main paper, further highlighting the practical limitations of existing approaches.

Methods and Evaluation Criteria

Thank you for providing valuable discussion points regarding related works. We have carefully reviewed all the papers you mentioned and have identified the following key differences between our work and theirs.

1)Null-Text Inversion: We would like to first clarify the differences between Null-Text-Inversion(NTI) and the problem setting of our research(DAG). NTI is primarily focused on effective image editing by performing DDIM inversion to obtain z_T​ from a given input image. In contrast, our goal is to synthesize data that approximates the distribution of the training dataset learned by a classifier—without requiring any input image. We start from random noise and leverage the classifier’s knowledge(BatchNorm statistics) as guidance. In other words, while NTI requires a source image as input and produces an edited version of that image as output, our method only requires an arbitrary classifier. To ensure clarity, we will explicitly detail these differences in the revised version of the paper.

2)Textual Inversion (TI): TI and DDIS both optimize a pseudo-word token embedding, but with different goals. TI personalizes text-to-image models by learning an embedding from a few reference images to capture a visual concept, guiding image generation accordingly. In contrast, DDIS generates data approximating a classifier’s training distribution without reference images, optimizing the embedding via cross-entropy loss between classifier predictions and target labels to steer generation toward the target class. While TI enables controlled generation, our method leverages a fixed diffusion model as a strong prior to constrain the image search space in data-free synthesis, yielding realistic, class-aligned samples.

We also appreciate your feedback regarding the ethical review. We will provide the following discussion and incorporate it into the revised version of the paper.

Ethical Review

This work aims to enhance the utility of pre-trained models by generating synthetic data in scenarios where access to the original data is unavailable. We acknowledge that data-free image synthesis approaches, designed to recover data following the training dataset’s distribution, can raise concerns regarding privacy leakage and other ethical issues. However, our objective is not to reconstruct individual instances or facilitate unauthorized data access but rather to synthesize a surrogate dataset that can be effectively utilized for data-free applications such as data-free knowledge distillation or data-free pruning. Specifically, our method guides the data generation process by leveraging the running statistics from classifiers, which represent the averaged information of the entire dataset. This design inherently prevents the inclusion of individual instance details in the generated images, making it extremely difficult to recover any specific sample (see Figs. 2–5 in the main paper). We hope that this work contributes to the responsible development of data-free techniques for scenarios in which data sharing is limited or infeasible.

Thank you for your detailed comments. We sincerely appreciate your insights and hope to address your concerns.

W1) A well-trained generator captures rich knowledge about the training data, enabling it to produce desired images through appropriate combinations of learned features. As a result, many recent studies have explored aligning models to specific tasks using only prompt tuning. The better the pre-trained model, the more effectively it can generalize to unseen combinations within the embedding space. Therefore, optimization within the embedding space alone is often sufficient. In our setting, we leverage the strong image prior provided by Stable Diffusion. This allows even a simple prompt such as “a photo of [class]” to effectively generate class-representative images (see “SD 2.1” in Fig. 2–5). However, the reason we utilize the CAT token is that the generated class is not perfectly aligned with the classifier’s learned class distribution (see Fig. 5, third column). For example, in ambiguous cases such as “kite,” which can refer to both a raptor and a toy, or for classes that require fine-grained details like “tiger cat,” the use of a Class token alone is sufficient to achieve closer alignment with the original dataset’s learned class distribution (see Fig. 5, fourth column).

W2-1) Validation of DAG: To align BN statistics, we design a guidance mechanism within the diffusion sampling process. To address this concern of our DAG design, we conduct experiments using alternative design choices for BN statistics alignment. Specifically, instead of applying guidance during the sampling process, we optimize the CAT embedding directly using Equation (9) with the Cross-Entropy loss as the objective function. The results shown in Figure 2 of the attached link demonstrate the outcome when CAT token optimization is performed without DAG, where both cross-entropy loss and BN loss are optimized during training. In this case, although each PACS-art sample captures certain domain characteristics, the influence toward the target class becomes weaker. As a result, the generated images tend to lose critical target class attributes, leading to degraded class-specific representations. Furthermore, BN loss encourages the synthetic images to follow the averaged running statistics over the entire training set, which can lead to the generation of images that contain mixed information from multiple classes. In contrast, our proposed DAG performs BN statistic alignment only during the diffusion sampling process, while the CAT embedding is optimized solely using cross-entropy loss. This separation allows the CAT embedding to focus on capturing class semantics while the sampling process handles domain characteristics. As a result, our method effectively approximates both the class and domain properties of the original training data.

W2-2) Zero-shot image synthesis We conduct a zero-shot image synthesis experiment to evaluate whether Stable Diffusion can effectively generate images for unseen classes. First, using Stable Diffusion V3, we select 10 classes that the Stable Diffusion 2.1 model used in our study has never encountered and generate images for those classes with SD V3. Then, we train a ResNet-50 model using the dataset of these 10 classes. Finally, we utilize Stable Diffusion 2.1 to synthesize the images used for training ResNet-50 with the our proposed DDIS. Surprisingly, as shown in Figure 4 in the link, although Stable Diffusion 2.1 struggles with these classes, the CAT embedding can capture the class attributes from the training set and generate images with a distribution similar to that of the original dataset. (Please see this link: https://drive.google.com/file/d/1S2Qt0XSgMH2pOzdzsNekhlyP-bLBY5kt/view?usp=sharing)

W3) Please refer to our response to Reviewer #e4Ls’s W1) for a detailed explanation.

W4) To assess whether our generated images are distinguishable by standard classifiers, we conduct an additional experiment where we pass 100,000 synthetic images through a ResNet-34 pretrained on ImageNet-1k and report the average top-1 classification confidence for each method. As shown in the above Table, our method achieves the highest average confidence across 1000 classes, indicating that our synthetic samples are more consistent with the model's learned distribution. These results further support the effectiveness of CAT embedding in generating class-consistent and classifier-recognizable images. We will include this analysis in the future to reinforce the practical impact of our method.

We would like to express our gratitude for your thoughtful comments and valuable feedback. We understand your concerns, and hope the following response addresses them.

W1) Compared to existing DFIS methods, our DDIS offers significant improvements in efficiency and cost-effectiveness for large-scale dataset generation on ImageNet-1k while maintaining high image quality. To illustrate, we compare the number of optimization iterations required to synthesize 100,000 ImageNet-1k images. All experiments were conducted using a single RTX 4090 GPU:

• DeepInversion (DI): DI requires iterative optimization for each batch of images. With an optimal batch size of 250 on a single RTX 3090, approximately 20,000 iterations are needed per batch. To generate 100,000 images, DI must repeat this process 400 times, resulting in a total of 800,000K iterations.

• PlugInInversion (PII): PII similarly optimizes a randomly initialized input but across 7 progressive upsampling stages (from 7×7 to 224×224), with 400 iterations per stage. Thus, generating 100,000 images with a batch size of 250 results in around 1,120K total iterations.

• DDIS (Ours): DDIS optimizes only class-wise CAT embedding vectors in a relatively low-dimensional space (1×784). For ImageNet-1k with 1000 classes, we perform just 30 iterations per class, totaling only 3,000 iterations. Once a CAT embedding is found, we can synthesize 100,000 images simply by sampling latent vectors—without additional training or optimization. CAT optimization takes slightly longer per iteration than prior works, but the total required iterations are significantly lower, making the overall process more efficient.

W2) As you said, comparing data distributions is inherently challenging. Generally, in the image synthesis tasks, Is, FID, Precision, and Recall are widely used to quantify fidelity and diversity, respectively, providing a comprehensive view of how closely the synthetic data approximates the real training distribution. The metric results presented in Table 1 reflect this distributional gap between the real and synthetic data. Our proposed method, DDIS, consistently outperforms prior approaches across all metrics (IS, FID, precision, and recall), indicating strong alignment with the true data distribution. This is further supported by the results in Table 2 and Figure 6, where using our synthetic data in place of the original training set yields the best performance in applications such as DFKD and pruning. We would also like to say that Reviewer #UzMa, in Claims and Evidence #1, acknowledged these results as strong evidence supporting the effectiveness of our approach.

W3) In our view, the proposed method, DAG, is conceptually more aligned with Classifier Guidance than with Classifier-Free Guidance. DAG guides the diffusion sampling process by leveraging the running statistics stored in a classifier, which closely resembles the mechanism of Classifier Guidance. In traditional Classifier Guidance, It requires $t$ additional classifiers trained on noisy latents $z_t$ at different time step t, making it incompatible with a standard pretrained classifier. In contrast, DAG makes use of batch normalization statistics instead of classifier outputs, allowing it to work with general classifiers that are trained on $z_0​. Furthermore, A key distinction lies in the role each method plays: DAG provides guidance based on the training set domain, encouraging the generated samples to follow the underlying distribution of the original data. On the other hand, CG is typically used to guide an unconditioned diffusion model to generate images belonging to a specific target class.

Classifier-Free Guidance (CFG), by contrast, does not rely on a separate classifier and instead offers text-conditioned guidance to ensure prompt fidelity. Importantly, DAG and CFG can be used together, as their mechanisms are orthogonal and complementary.

In summary, DAG shares the classifier-based foundation of CG but distinguishes itself by using the gradient of the BN loss and focusing on aligning samples with the overall training distribution, rather than generating class-specific outputs.

We would like to express our gratitude for your thoughtful comments and valuable feedback. Our response to the weakness is as follows.

W1) Previous DFIS methods like DeepInversion lack strong image priors and rely on weak constraints (e.g., total variation), leading to unrealistic artifacts or samples far from the true data manifold (see Figs. 2–5). They also optimize in high-dimensional pixel space (e.g., B×3×256×256) from random noise, making the process unstable and inefficient. In contrast, our method is the first to leverage strong priors from text-to-image (T2I) diffusion models, guiding synthesis toward realistic images and constraining the search space. Moreover, we operate in a compact embedding space (e.g., 1×784), which simplifies optimization. These improvements allow our method to more accurately match the original training distribution and address key limitations of prior work.

W2,W6)(Please see this link: https://drive.google.com/file/d/1S2Qt0XSgMH2pOzdzsNekhlyP-bLBY5kt/view?usp=sharing) We compare DDIS with methods [1–3] on synthetic data quality and KD performance using CIFAR-10/100, as [1–3] don’t support large-scale, multi-domain synthesis. All methods use ResNet-34 to generate 50K images per dataset. We evaluate IS, FID, precision, and recall (see Fig. 1–2), and Table 1 shows DDIS outperforms all baselines with more realistic images. For data-free knowledge distillation (DFKD), we use ResNet-34 as teacher and ResNet-18 as student. While student accuracy is slightly lower than KD-focused methods using special losses, our goal is to match real data distribution, not optimize for KD.

To show generality, we apply DDIS to data-free pruning. Table 2 reports top-1 accuracy of pruned ResNet-34 (ratios 0.5–0.9) fine-tuned on our synthetic data. DDIS consistently achieves the best results, proving its broad applicability.

W3) NaturalInversion (NI) focuses on small-scale datasets like CIFAR-10/100 and faces scalability issues due to quality and compute limits. It shifts optimization to parameter space using both a main and sub-generator (FTP), increasing trainable parameters. Scaling to larger datasets requires bigger generators, expanding the search space and computational cost, making large-scale synthesis infeasible. For fair comparison, we apply our method to CIFAR-10/100 with pretrained ResNet-34, generating 50k samples. As shown in Table 1, our method outperforms NI in IS, FID, precision, and recall, better aligning with the original data distribution. These results demonstrate our method’s scalability and effectiveness, and we will include them in the revised paper to strengthen our evaluation.

W4) Algorithm 1 outlines Domain Alignment Guidance (DAG) used during diffusion sampling. The noisy latent $z_t$​ is decoded by $D$ into $x_t$​, which is input to classifier $f$ to compute BN loss. The gradient of this loss guides $z_t$​, yielding z_t^{\tilde} (line 6). This guided latent is then used for classifier-free guidance (line 7) and for predicting the next-step latent (line 8), following standard diffusion sampling. This process repeats until z_0^{\tilde} is obtained. The full pipeline with DAG is detailed in Algorithm 2.

W5) As detailed in W1) to Reviewer #e4Ls and Appendix A.3–A.4.