Diffusion Curriculum: Synthetic-to-Real Data Curriculum via Image-Guided Diffusion

Thank you for your feedback and suggestions! We provide responses to your questions below.

Detailed information for Section 4.

Thanks for your advice! The details of DoCL and its adaptive curriculum are discussed in Appendix A.3.2. We will try our best to move back more details from the Appendix to the main paper.

Related works about using generative models for data augmentation.

Thank you for pointing out these related works! We would like to clarify key distinctions between our approach and these methods.

Our DisCL framework is built on two primary components: the generation of a spectrum of sync-to-real synthetic data and a curriculum learning paradigm designed to effectively leverage this data. Unlike previous works, which focus primarily on synthetic data generation for downstream tasks without integrating curriculum learning, DisCL strategically bridges the distributional gap between synthetic and real data. For example, Cap2Aug (Roy et al., 2022) generates data with fine-grained modifications, while Boomerang (Luzi et al., 2022) synthesizes data through local sampling to closely resemble the original inputs. Both methods focus on localized changes and thus lack the diversity needed for long-tail tasks and easier or typical samples needed for low-quality tasks. GeNIe (Koohpayegani et al., 2023) generates hard negative samples to improve the classifier’s ability to distinguish between positive and negative samples. However, for low-quality tasks like iWildCam, the primary challenge is aligning visual features of hard positive samples with pre-trained knowledge—an issue that GeNIe does not fully address. DA-Fusion (Trabucco et al., 2023) increases synthetic data diversity, yet it lacks either the spectrum of varied synthesis or the curriculum-driven approach that DisCL provides. DisCL’s curriculum-based framework constructs a smooth transition from synthetic data (representing prototypical or diverse features) to real data (which contains task-specific but often limited features). This smooth spectrum aids in adapting pre-trained models more effectively to low-quality or long-tail tasks. We plan to explore integrating image editing approaches from these works with DisCL’s spectrum generation and curriculum paradigm to further enhance our results.

Question related to ALIA results.

The results of ALIA are shown below. We will update the result in Table 5.

Question related to curriculum strategies.

The 'diverse to specific' and 'easy to hard' curriculum strategies both refer to 'synthetic (lower image guidance) to real (higher image guidance)' curriculum strategies (as mentioned in line 360 and 407). However, for different tasks, synthetic data has different properties. For long-tail task, synthetic data with lower image guidance provides more diversified visual features, while providing easier and more proto-typical features for low-quality tasks. To illustrate the property of synthetic data under different tasks, we use these two names.

Questions related to Cross Entropy and Balanced Softmax.

Due to the population bias among classes, the model struggles to sufficiently learn visual features from underrepresented “Few” classes while using Cross Entropy loss function. As a result, the Cross-Entropy (CE) baseline performs significantly worse compared to the Balanced Softmax (BS) loss, which explicitly addresses class imbalance by reweighting softmax probabilities based on class frequency.

When applying DisCL, our method introduces more diverse features for underrepresented classes, partially mitigating the effects of population bias and improving performance under CE. However, for Balanced Softmax, which already compensates for class imbalance directly, the additional impact of DisCL’s diverse features and samples is less pronounced.

• Meng, Chenlin, et al. "Sdedit: Guided image synthesis and editing with stochastic differential equations." arXiv preprint arXiv:2108.01073 (2021).
• Roy, Aniket, et al. "Cap2aug: Caption guided image to image data augmentation." arXiv preprint arXiv:2212.05404 (2022).
• Luzi, Lorenzo, et al. "Boomerang: Local sampling on image manifolds using diffusion models." arXiv preprint arXiv:2210.12100 (2022).
• Koohpayegani, Soroush Abbasi, et al. "GeNIe: Generative Hard Negative Images Through Diffusion." arXiv preprint arXiv:2312.02548 (2023).
• Trabucco, Brandon, et al. "Effective data augmentation with diffusion models." arXiv preprint arXiv:2302.07944 (2023).

Thank you for your valuable feedback! Here we address you comments.

Tradeoff between time and synthetic dataset.

DisCL generates synthetic data exclusively for hard samples in the original datasets, covering approximately only 10% of total samples. Additionally, as noted in Appendix A.5, generating a complete spectrum of data across six image guidance levels (at a resolution of 480×270) takes only 10 seconds. This generation time is efficient and manageable within these parameters.

Failure cases for low-resolution datasets.

For low-resolution datasets, image editing with diffusion models can be challenging. To address this, we start by using a super-resolved version of CIFAR100 images (resized with CAI super resolution) as the base for synthetic data generation. The generated images are then downsampled to 32×32 for training. Additionally, we observed that lower image guidance levels can produce significantly altered synthetic images, creating a large discrepancy between the synthetic and original images. To improve the quality of synthetic data for low-resolution datasets, we adjust the image guidance level to a range of 0.5 to 0.9, ensuring closer alignment with the original images.

• 128*128 resized CIFAR100 dataset: https://www.kaggle.com/datasets/joaopauloschuler/cifar100-128x128-resized-via-cai-super-resolution

Application to non-realistic domains.

For datasets in non-realistic domains such as Comics or Drawings, we can first evaluate the generation quality of the pre-trained diffusion model on the target dataset. If the quality is insufficient, we can switch to pre-trained models specifically trained on the corresponding or similar domains.

In the future, we plan to integrate advanced diffusion training techniques, such as ControlNet and DreamBooth, to enhance the compatibility of diffusion models with the original dataset, ensuring higher-quality generation aligned with the domain’s characteristics.

Questions related to CLIPScore filtering process.

We assess the effect of CLIPScore filtering with human evaluation. To ensure the quality of synthetic data, we choose a relatively high threshold of CLIPScore for data filtering, which results in a high ratio of abandoned data. For percentage of filtered images is shown as below, which is shown in Table 6 in Appendix.

Thank you for your valuable feedback! We have carefully reviewed your comments and address them as follows.

Existing works related to image guidance and contribution.

One of our contributions is generating a complete and smooth spectrum of synthetic-to-real data and using this spectrum in training simultaneously. Different data are selected to be used at each epoch. Previous work, such as SyntheticData [1], introduces image guidance to control the similarity between generated images and the original input. However, although they incorporate image guidance in the data generation process (Real Guidance (RG) Strategy in [1]), each training run only utilizes a single image guidance level to produce synthetic images, rather than leveraging multiple guidance levels across training. In [1], specific image guidance levels are chosen for different few-shot settings, and the corresponding synthetic data is used exclusively for each setting. In contrast, our approach applies the entire spectrum of image guidance levels in a unified training process, offering a richer and more continuous range of synthetic-to-real features for curriculum designs and model adaptation.

• [1] He, Ruifei, et al. "Is synthetic data from generative models ready for image recognition?." arXiv preprint arXiv:2210.07574 (2022).

Questions related to current results.

DisCL primarily targets hard samples within the original dataset, focusing on generating synthetic data specifically for underrepresented classes, labeled as “Few” classes in Table 1. As shown in Table 1, the accuracy for “Few” classes improves from 17.90% (Text-only) and 19.17% (All-Level) to 23.64% (DisCL), highlighting the effectiveness of the curriculum paradigm applied in our method.

However, given that only 136 out of 1,000 classes (1,643 out of 115,846 samples) fall into the “Few” category, the overall improvement in accuracy appears less pronounced, as the impact is more localized to these underrepresented classes.

Ablation studies on CIFAR100-LT and iNaturalist2018.

We use ImageNet-LT as the primary dataset for our long-tail classification tasks and conduct ablation studies on this dataset. We plan to run additional experiments on CIFAR100-LT and iNaturalist2018, to further validate our approach and strengthen the robustness of our findings.

Thank you for your detailed feedback! We carefully address your comments as below.

Relevant related works regarding data augmentation.

Thank you for pointing out these related works! We would like to emphasize key distinctions between our approach and these methods. Our DisCL framework consists of two key components: generation of spectrum of sync-to-real data and the curriculum learning paradigm designed to effectively leverage this synthetic data. In contrast, SynCLR (Tian et al., 2024a; Tian et al., 2024b), DAIC-GAN (Astolfi et al., 2023), and Feedback-guided synthesis (Hemmat-Askari et al., 2023) primarily focus on generating data for downstream tasks without integrating curriculum learning. Specifically, feedback-guided data synthesis proposes to use the Feedback Criteria (Loss / Entropy / Hardness) from the pre-trained model to generate the synthetic data. However, once the data is generated, all synthetic samples are used indiscriminately in training without any selection or progressive filtering.

In DisCL, beyond data generation with various image guidance, we further implement a curriculum learning paradigm to select data at each training epoch. This iterative selection helps the classifier incrementally bridge the gap between synthetic and real data distributions, enhancing model robustness and generalization.

The ALIA result is shown as below. we will update Table 5 in our paper with these results.

For experiments with larger models (ResNet32 and ResNet50), we expect our result will still hold. We will run them and use the results to strengthen our paper.

Questions related to training settings.

For question related to training epochs, in our experiments, we always keep the total training epochs of DisCL less or equal to that of the baselines.

For question about balanced ratio, DisCL is designed to improve performance on hard samples in the original dataset, specifically targeting “Few” classes in long-tail classification scenarios. In the data generation process, we focus exclusively on generating synthetic data for underrepresented classes, excluding “Medium” classes. Furthermore, generating synthetic data to achieve a fully balanced dataset would be both time- and resource-intensive. Thus, we maintain the original imbalance ratio in the generated dataset, allowing us to evaluate DisCL’s performance without changing population bias due to class distribution changes. To further validate our approach, we plan to conduct additional experiments using balanced synthetic datasets, aiming to strengthen our findings.

Questions related to current results compared with Hemmat-Askari et al 2023.

For the difference between Feedback-guided synthesis (Hemmat-Askari et al., 2023) and DisCL, one key difference is the scale of the synthetic dataset used (1.3M samples vs. 25k in DisCL). Additionally, Feedback-guided synthesis generates synthetic data to augment the entire real dataset, whereas DisCL focuses on enhancing model performance specifically on challenging samples within the original dataset. For long-tail classification tasks, DisCL selectively generates synthetic data only for underrepresented classes (few classes in the original dataset). To further investigate this distinction, we plan to conduct additional experiments that control for data scale and class distribution.

Questions related to Table 2 results.

Thank you for your questions and pointing out! We will correct our Table 2 as follows.

Since the Balanced Softmax (BS) function helps mitigate bias between dominant and underrepresented classes, BS+DisCL still achieves better performance than CE+DisCL, as DisCL continues to work with an imbalanced training dataset.