Mousterian: exploring the equivalence of generative and real data augmentation in classification

Many thanks to Reviewer TBMi for providing detailed suggestions.

W1: "include experiments on additional datasets with varying complexity to validate the proposed approach across different domains."

We consider that ImageNet and CIFAR-10 are widely recognized as representative datasets in image classification, making conclusions drawn from them broadly applicable and indicative of general trends.

W2 & Q1: "theoretical insights or justification for the empirical equivalence between synthetic and real data augmentation, and the generalization of this relationship across different types of datasets and generative models."

Our formula is not derived theoretically but rather an empirical fit to the data points. While it may not be highly precise, it effectively reflects our conclusions by illustrating the quantitative equivalence between real and generative data augmentation.

For the generalization of this relationship, although the relationship is derived on CIFAR-10+Vanilla-DDPM and ImageNet-100+Stable Diffusion 3 settings, we assume this form of relationship will generalize to other image classification scenarios. Note that the coefficients in the formula may change, but we expect the form to hold. We leave validating on more image classification datasets for future work.

W3:"a more detailed analysis of the characteristics of synthetic data (e.g., diversity, fidelity) is needed to explain the diminishing returns observed in these figures"

Previous work [1] has explored this issue and found that both recognizability and diversity are essential for synthetic data to enhance image classification effectively. Our focus, however, is not on improving synthetic data generation but on understanding the general trend of its effectiveness, which does not change with varing data quality.

W4 & Q2: "examine whether combining internal and external generative models could lead to better data augmentation strategies,"

This is a valuable suggestion which can be explored in future work. However, in this paper, we mainly focus on the internal and external settings, which are more common in most synthetic data application scenarios. We will explore this in the future.

W5 & Q3: "consider the computational cost of generating synthetic data. discuss on the associated computational costs to assess the feasibility of the proposed approach in real-world scenarios."

We have conducted experiments to record the computational cost of generating synthetic data, as shown below:

1. CIFAR-10 Experiments (A4500 GPU)

• EDM: 18 sampling steps, batch size of 100

  • Memory usage: 4000MB
  • Time per image: ~0.06 seconds

• Vanilla-DDPM: 500 sampling steps, batch size of 100

  • Memory usage: 3300MB
  • Time per image: ~0.95 seconds

We used 4 A4500 GPU, and it took 16 hours to generate Vanilla-DDPM data at 5 times the scale of CIFAR-10.

2. ImageNet Experiments (A6000 GPU)

• Stable Diffusion 2: 50 sampling steps

  • Memory usage: ~3900MB
  • Time per image: ~1 second

• Stable Diffusion 3: 28 sampling steps

  • Memory usage: ~21800MB
  • Time per image: ~7 seconds

We used 4 A6000 GPU, and it took 2~3 days to generate SD-3 synthetic data at the same scale as ImageNet-100.

Both computational costs are affordable in our settings.

W6 & Q4: "incorporating additional metrics, such as robustness to adversarial attacks or generalization to out-of-distribution samples."

This is a helpful suggestion. Previous works [1], [2] have explored this issue. Our focus, however, is on understanding the effectiveness of generative data augmentation for image classification within a given dataset. Thus, accuracy remains our primary concern, while robustness and adversarial attack resistance are viewed as potential extensions to the core problem.

Q5: "Is there an optimal ratio of synthetic to real data that consistently yields the best performance"

How is the optimal ratio of synthetic to real data defined? Are you referring to a point where synthetic data provides maximum benefit, beyond which adding more may actually reduce accuracy?

[1] Fan, Lijie, et al. "Scaling laws of synthetic images for model training... for now." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Singh, Krishnakant, et al. "Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

We would like to sincerely thank Reviewer aYQx for providing a detailed review and insightful questions.

W1： "Most of the findings are generally known and have been reported from previous works on more datasets".

We admit that conclusion 1,2,3 are somewhat similar to [1], but the problem we investigate is much different than theirs. The differences can be summarized as follows:

1. Their work only focuses on few-shot learning, assuming that "the positive impact of synthetic data will gradually diminish as the amount of real data increases." However, there are two main issues with this approach. First, their conclusion relies simply only on observing a converging trend of accuracy curves with and without synthetic data when adding more real data. However, when a large amount of real data is available and the accuracy is already high, even if we add additional real data instead of synthetic, further accuracy improvements still become increasingly difficult. In other words, the positive impact of additional real data will also gradually diminish as the amount of given real data increases. Only by studying the quantitative equivalence of real and generative data augmentation can we reach a solid conclusion. Second, even if the positive impact of synthetic data does gradually diminish with more real data, it is still valuable to explore the improvements of synthetic data under this condition. Even a 0.x% accuracy improvement when accuracy is already high (e.g., around 90%) can be meaningful, as it could push results to a new SOTA level.

2. They consider only generative models trained on large-scale external datasets. However, in specialized scenarios—such as medical imaging—pre-trained generative models may perform poorly due to a significant distribution gap from common images. In these cases, internal data augmentation becomes essential to address the issue. Thus, the problem we investigate is more foundational than theirs.

Q1: "Guidelines for selecting the optimal syn-real ratio for any new datasets"

This is a good question, but could you provide a more concrete definition of the optimal syn-real ratio? Is it a point where synthetic data provides maximum benefit, beyond which adding more may actually reduce accuracy?

Q2: "The current manuscript is more like a technical report."

We respectively disagree with this comment. As mentioned above, our work on the quantitative equivalence of real v.s. synthetic data and internal v.s. external augmentation are foundational and helpful to the machine-learning community. We provide insightful guidance for the future research on synthetic data. Please also see our reply to the common questions at the beginning.

[1] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip H. S. Torr, Song Bai, & Xiaojuan Qi (2023). Is Synthetic Data from Generative Models Ready for Image Recognition?. In The Eleventh International Conference on Learning Representations, ICLR 2023.

We would like to sincerely thank Reviewer AgTc for providing a detailed review and insightful questions.

W1:"Numerous papers cover synthetic data for image classification, including studies like arXiv:2304.08821 and arXiv:2204.08610"

arXiv:2304.08821 proposes a new method for data augmentation. Although Figures 2 and 3 in their paper show accuracy curves for different synthetic image ratios under varying amounts of real images, their main focus is to validate the effectiveness of their method, rather than studying the effectiveness of generative data augmentation for image classification on a given dataset. In contrast, we provide a more comprehensive analysis, including a quantitative comparison between real and generative data augmentation, along with a diverse set of conclusions in Sections 3.1, 3.2, and 3.3. arXiv:2204.08610 includes generative data augmentation as one of the 10 augmentation techniques with results reported just on the mixed augmentation; it does not examine the effectiveness of generative data augmentation in particular. Please see also our reply to the common question.

W2:"Results in image classification does not translate well to other domains with higher current relevance, such as large language models (LLMs) or language-related synthetic data"

Thank you for highlighting the area of LLMs. However, we feel this may not be highly relevant here, as pixels differ significantly from words. Focusing on conclusions that benefit the field of computer vision alone would already be a valuable contribution.

W3:"Factors like model size and architecture can significantly impact outcomes. Investigating performance variability across different model sizes or architectures, and including training variance across runs, would strengthen the findings."

Thank you for the suggestions. Different classification models indeed affect performance, and we will work to include results from additional classifiers, though we believe they will have the similar trend. Including training variances is also a valuable suggestion. We did not report the variances, but we conducted the same experiments over three trials, and the results are robust.

W4:"Simply focusing on accuracy is not as meaningful for many research today as we already know "more high quality" data always leads to somewhat better results from all previous studies."

Although "more high quality" leads to better quality seems to be trivial, we still consider studying the effectiveness of synthetic data worthwhile, because we need to take the computational and time cost into account, and choose whether to gather more real data or utilize generative models. The conclusions regarding the effectiveness of generative data augmentation provide valuable guidance to people on when and how to use synthetic data.

Q1:"For quality assessment, FID and IS are very bad scores. There are many new quality assessment metrics, such as Clip-MMD."

Thanks for mentioning. Currently we have not used other quality assessment metrics, and we will consider them.

We would like to sincerely thank Reviewer gJjx for providing a detailed review and insightful questions.

W1: "There are shortcomings in the ablation studies, as they do not adequately analyze the specific impact of different generative models or the effect of varying dataset sizes."

In fact, we systematically explored the effects of different generative models and dataset sizes across CIFAR-10, ImageNet-10, ImageNet-100 datasets in our experiments.

For CIFAR-10, we utilized two generative models trained on internal dataset: Vanilla-DDPM and EDM. Additionally, we used the CIFAKE dataset generated by Stable-Diffusion-1.4 and resized to 32x32. To examine the impact of dataset size, we tested with three scales of real data (500, 2500, and 5000 samples per class) along with varying proportions of synthetic data.

On ImageNet-10 and ImageNet-100, we experimented with Stable Diffusion 2 and Stable Diffusion 3 using diverse prompts to generate synthetic data. Real dataset sizes included 65, 260, and 1300 samples per class, again with varying levels of synthetic data integration.

W2: "The paper does not discuss the significant computational resources required for generating high-quality synthetic data."

Thanks for mentioning the aspect of computational resources. Here, we provide the memory usage and GPU hours for the generative models used in our experiments:

1. CIFAR-10 Experiments (A4500 GPU)

• EDM: 18 sampling steps, batch size of 100

  • Memory usage: 4000MB
  • Time per image: 0.06 seconds

• Vanilla-DDPM: 500 sampling steps, batch size of 100

  • Memory usage: 3300MB
  • Time per image: 0.95 seconds

We used 4 A4500 GPU, and it took 16 hours to generate Vanilla-DDPM data at 5 times the scale of CIFAR-10.

2. ImageNet Experiments (A6000 GPU)

• Stable Diffusion 2: 50 sampling steps

  • Memory usage: 3880MB
  • Time per image: ~1 second

• Stable Diffusion 3: 28 sampling steps

  • Memory usage: 21808MB
  • Time per image: ~7 seconds

We used 4 A6000 GPUs, and it took 2~3 days to generate SD-3 synthetic data at the same scale as ImageNet-100.

W3: Real-world scenarios with variable conditions, such as those involving autonomous vehicles, may not have adequate validation

We agree that the impact of synthetic data can be further studied on more specific real-world tasks. However, this paper focuses on identifying general trends, and in future work, we plan to validate our findings on tasks like autonomous driving or medical imaging.

Q1: Why were generative models like Vanilla-DDPM and Stable Diffusion 2 chosen for image generation? How might the choice of these specific models impact the generalizability of the results? Are there alternative models that could also be explored in future work?

We selected Vanilla-DDPM and Stable Diffusion 2 due to their widespread use for generating synthetic data. We acknowledge that various generative models could affect classification outcomes, yet we believe the core conclusions of our paper remain robust. Our findings with EDM and Stable Diffusion 3 support this view, and we leave the exploration of additional generative models to future work. There are many alternatives to explore, like GLIDE, Imagen, Muse, and DALL-E for diffusion-based models. For further details, please refer to the "Text-to-Image Diffusion Models" section in "4 Related Work".

Q2: The rationale behind selecting different image encoders at various stages of the generation process, and the influence to the final results.

For external generative data augmentation, we use the original Stable Diffusion configurations. They use CLIP as the text encoder because CLIP acts as a "translator" between language and visuals, allowing Stable Diffusion to create images that match the text prompt closely. DINOv2 is not used in the whole process. For internal generative data augmentation, we use ordinary conditional data generation without text prompts.

We would like to sincerely thank Reviewer gL25 for providing a detailed review and insightful questions.

W1: "cautious about such strong statement as 'synthetic data is less sample efficient than real data'"

We sincerely thank Reviewer gL25 for the thoughtful and careful statement. We will adjust the presentations in our paper.

W2:"'synthetic data is less sample efficient than real data' might just because the generated samples lack diversity."

Synthetic data is less sample-efficient than real data, not only due to limited diversity but also because of fundamental differences in their underlying manifolds.

The diversity of synthetic data indeed plays a critical role in classifier accuracy. As demonstrated in Figure 4, Sable Diffusion 3 images, with greater diversity than Stable Diffusion 2, lead to a higher accuracy boost. However, as both the paper and reviewer gL25 highlight, a substantial domain gap persists between generated and real images, preventing synthetic data from fully matching the distribution of natural images. Regardless of generation methods, synthetic data consistently diverges from real data. Even in internal augmentation settings without external data introducing a data domain gap, synthetic data generated through different models and techniques still fails to outperform real data.

Q1: "a more fine-grained analysis such as analyzing whether the ratio between real/synthetic data has the same impact across all classes or if some classes benefit more than others."

This is a great question and worthy to be studied in the follow-up work. However, it is beyond the scope of this work. We will consider this in the future work.

Q2: "any suggestions on how to improve the "efficiency" of the synthetic data?"

We tried multiple ways to improve the efficiency of the synthetic data.

1. We first developed an edge decoder to extract edge information from mid-layer features of real or synthetic data, as we found that distinguishing edges between real and synthetic images was more challenging than separating the original images themselves.
2. Treating real and synthetic data as separate domains, we tried using domain adversarial training techniques to close the gap between real and synthetic features.
3. We also tried extracting edges from real images and generating synthetic data using a ControlNet with the extracted edges as the control condition.

However, methods 1 and 2 do not yield better results than simply mixing real and synthetic data, while method 3 struggles to generate high-fidelity images, as the edges of real images in ImageNet are typically too noisy to provide a reliable generation condition.