Real-Fake: Effective Training Data Synthesis Through Distribution Matching

Dear Reviewer wBAQ,

Thank you for your valuable feedback and constructive comments. We appreciate you recognizing our work is easy to follow and results appear promising. We now address your concerns and questions below.

Q1: Maximum Mean Discrepancy (MMD) for learning data distribution is not novel, which has been explored before.

A1: We kindly note the main idea of our work is not leveraging MMD loss as a standalone technique for learning data distribution but to formulate the training data synthesis as a distribution matching problem (Eq. 3 and Eq. 4). Our specific implementation of this framework on latent diffusion model involves using distribution matching loss (MMD) along with latent prior initialization and conditional visual guidance, integrated in Stable Diffusion for synthesizing informative training data. The significant performance improvements brought by our method empirically indicate the effectiveness of the proposed framework. We have also validated the importance of all three components of our method. We have revised our manuscript to emphasize and clarify our main idea.

Q2: The derived MMD loss based on diffusion model (Eq. (7)) is the same as the simplified diffusion model loss (Ho et al. 2020).

A2: We would like to clarify that our derived MMD loss is not identical to the diffusion model loss presented by Ho et al. (2020). Their simplified diffusion model loss is a mean square error calculated on top of the predicted noise and added noise whereas the MMD loss, Eq. (7), is the L2-norm of the averaged difference between the predicted noise and added noise.

$$L_{DM} := ||\frac{1}{|N|} \sum_{i=1}^{|N|} (\epsilon - \epsilon_{\theta}(x_t, t))||^2 \leq \frac{1}{|N|} \sum_{i=1}^{|N|} ||(\epsilon - \epsilon_{\theta}(x_t, t))||^2 = L_{Diffusion}.$$

The diffusion model loss is an upper bound of the MMD loss by Jensen’s inequality. The derived MMD loss is thus a tighter bound on the distribution discrepancy between real and synthetic data under the MMD measure. We have revised the relevant sections to better clarify this distinction.

[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems 33:6840–6851, 2020.

Q3: From Table 2, the performance gain brought by MMD is not significant compared to finetuning only.

A3: As mentioned above, we show that the diffusion loss (Ho et al. 2020) is an upper bound of the MMD loss. Therefore, finetuning with the diffusion model loss still aligns with our distribution matching framework for training data synthesis. Our motivation (Section 3.1) to further include MMD loss rather than simply diffusion model loss is due to the known looseness problem of diffusion loss (Kingma et al. 2021). The results in Table 2 also show that the extra MMD loss can bring notable improvement in some settings, which are selected in the following table for illustration. MMD loss leads to consistent improvements of 0.6/1.1%, and 1.0/0.7% on ImageNette/ImageNet100, on top of vanilla diffusion loss, accordingly. We revised our manuscript to clarify it and provided further analysis in Appendix G.5.

[2] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696–21707, 2021.

1. I think it is trivial to formulate training data synthesis as a problem of learning data distribution, in particular a joint data distribution $q(x, y)$. This type of joint data distribution learning has been well established in the literature, e.g., [1]. The author emphasizes the significance of this contribution, with which I do not agree. Implementing the joint distribution learning through feature distribution alignment and conditional distribution alignment is a standard procedure, not novel.

2. From the description in Section 3.3, the component of latent prior initialization is not derived from the joint distribution learning framework but rather from the literature on the diffusion model. Therefore, I am not convinced by the claim in A4 that the proposed approach “integrates latent prior within a theoretical framework, offering a novel perspective in this field”.

3. I was misled by Eq. (21) before, erroneously thinking that the work employed the upper bound of $L_{DM}$ as the objective for DM. Thanks for the clarification in A2. It is clear to me now. I suggest polishing Eq. (21) to avoid confusion.

4. The superiority of MMD loss over original diffusion model loss is not well verified. The authors claim that "Our motivation (Section 3.1) to further include MMD loss rather than simply diffusion model loss is due to the known looseness problem of diffusion loss (Kingma et al. 2021)." Then a comparison of the MMD loss solely with the simplied diffusion model loss should be included.

5. Given the responses, now I believe deriving MMD loss under the diffusion model introduces some novelty. The authors may consider re-organize the work for delivering the contributions of this work. I think the current emphasis on a principled theoretical framework from the joint distribution matching perspective overstate the contributions of the work.

[1] Li, C., Xu, T., Zhu, J., & Zhang, B. (2017). Triple generative adversarial nets. Advances in neural information processing systems, 30.

Dear Reviewer BpUK,

Thank you for your insightful comments on our paper. We appreciate that you consider our work as a well-motivated theoretical framework, and recognize the comprehensive experiments with gains over the state of the art and promising advantages for OOD generalization and privacy preservation. We now address your concerns below.

Q1: Improvement over Imagen is limited. Only the comparison on ImageNet-1K is provided.

A1: Thank you for raising this point regarding the comparison to Imagen results (Azizi et al. 2023). We would like to kindly highlight several key aspects to consider Imagen (Saharia et al. 2022) in this context.

Firstly, Imagen is a closed-source model trained on private data, which makes it impossible for us to obtain more experiment results of Imagen to compare except their reported ones on ImageNet-1K dataset in (Azizi et al. 2023).

Secondly, the Imagen, trained on private data, involves a more powerful generative model than Stable Diffusion and is fine-tuned with full parameters on ImageNet-1K, which is computationally unaffordable by general researchers. In contrast, our approach leverages the open-source Stable Diffusion model coupled with a low-budget tuning (LoRA) and synthesis framework.

Thirdly, despite the resource constraints, our method still achieves notable performance improvements (1.7%) on the very challenging large-scale dataset ImageNet-1K, underlining the effectiveness of our techniques.

Last but not least, with a fair comparison with other baselines using the same generative model backbone, Stable Diffusion, our synthesis framework brings significant (10% ~ 35%) performance improvements on other datasets.

[1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022

[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023

Q2: Are there any other comparative methods that also use the same "[Image Caption]"? Whether the improvement are mainly from "[Image Caption]" instead of "[Intra-class Visual Guidance]"?

A2: Thanks for the question. Actually, the CiP (Lei et al. 2023) method generates images based on the "[Image Caption]". Specifically, they use the template “A photo of {class name}, {image caption}” to generate prompts. We provided a comparison to their results in Table 1 in the submission. Our method outperforms the baseline by 10%, 18%, and 17% on ImageNette, ImageNet-100 and ImageNet-1k datasets respectively. Furthermore, our ablation study in Table 2 has proven the importance of "[Intra-class Visual Guidance]" component. We further clarify it by showing the following selected results from Table 2. As shown, "[Intra-class Visual Guidance]" brings 1.5% and 0.7% performance improvements on ImageNette and ImageNet100 respectively, when cooperating with “Distribution Matching” components. To clarify it, we revised our manuscript and provided further analysis in Appendix G.5.

[3] Shiye Lei, Hao Chen, Sen Zhang, Bo Zhao, and Dacheng Tao. Image captions are natural prompts for text-to-image models. arXiv preprint arXiv:2307.08526, 2023.

Dear Reviewer 8NDL,

Thank you for your insightful feedback and constructive comments on our manuscript. We appreciate that you find the study problem interesting, the proposed framework sound, and the experiments extensive. We now address your concern below:

Q1: More rounds of experiments

A2: Thank you for reminding us of this problem. We have included additional multi-runs for ImageNet (IN-10, 1N-100, 1N-1K) experiments as shown in the table below, where we find the standard deviation of the three-seed experiment on all scales stays within a small range. This validates the stability of our method. We have added these results in our revised manuscript in Appendix G.6.

Multi-Seed Experiment Results: Top-1 Classification Accuracy across various datasets (IN-10, IN-100, IN-1k) over multiple runs. Average shows the mean and standard deviation of multiple run results.

Q1: Application to more complex task, for example tasks on human face

A2: Thank you for suggesting testing our framework on more complex tasks. We conducted additional experiments with a subset of the human face dataset CelebA (Liu et al. 2015) that included a general facial attribution classification task. As shown in the table below, the training data synthesized by our framework adapts well to various facial attributes, with close performance to the real data and notable improvement against the FakeIt baseline. We leave more complex face recognition tasks as future work. We have added the experiment settings, results, and discussion in our revised manuscript in Appendix G. 7.

CelebA Facial Attribute Classification Results on Three Attributes

[1] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.

Q3: Some figures can be improved

A3: We appreciate your helpful suggestions on the figures, especially Fig. 3. We have polished Figure 3 to ensure that all elements are clearly visible and the information is conveyed effectively. We also polished other figures in the manuscript, e.g., adjusting the colors in Figure 1.

Dear Reviewer WEFi,

Thank you for your insightful review and the affirmative evaluation of our work. We appreciate that you recognize the good clarity, originality, quality, and significance of our paper. We now address your concern below:

Q1: Details of privacy analysis are missing.

A1: Thank you for pointing out the insufficiency of details. We addressed this by updating a comprehensive privacy analysis details in Appendix G.4 in the revised manuscript.

Q2: Missing Entry in Tab1

A2: We would like to kindly note a few reasons for the empty entries in Table 1. Firstly, direct evaluations against the Imagen model on diverse datasets are challenging due to its closed-source nature. Secondly, relatively older baselines in Table 1 have been surpassed by more advanced methods, making them less competitive for current benchmarks. Therefore, we focused our evaluations on the large-scale ImageNet-1K dataset, considering it the most representative and classical benchmark for evaluation in this domain. Also, regarding the benchmarking on other datasets, we compare with the current state-of-the-art FakeIt. It delivers a fair comparison given the same generative model backbone and demonstrates significant performance improvement.

Dear Reviewers and ACs:

Thank you so much for your valuable feedback and suggestions. Hope our rebuttal has addressed your concerns. We're glad to see that our feedback has resolved some of the concerns from Reviewer wBAQ and we are willing to actively address the concerns raised by any reviewers.

We have revised our manuscript based on the current feedback. We have specified the changes in our revision in the previous General Response. We hope to receive more constructive comments from all reviewers to further improve our paper.

Thank you again for your time and efforts in assessing our paper.

Best Regards, All Authors.