GenDataAgent: On-the-fly Dataset Augmentation with Synthetic Data

Q-8: Line 474, the statement that accuracy is highly correlated with the synthetic data volume seems to be contradictory with a concurrent ICLR25 submission (Mousterian: exploring the equivalence of generative and real data augmentation in classification). In that work, they state that synthetic data is more valuable when the quantity is small, yet its value diminishes quickly rather than staying correlated with accuracy.

A-8:

1. These are two conclusions derived under different conditions.

• Mousterian claimed that the accuracy improvement from incorporating synthetic data depends on the volume of real data. As stated in Line 350, “Integrating synthetic data greatly enhances classification performance when real data is scarce, but the benefit decreases as real data becomes more plentiful.”
• In contrast, we argued that, given the same amount of real data, the performance gain for each category is correlated with the volume of synthetic data. However, this correlation does not imply proportionality.

2. That said, we share part of the conclusion that the improvement is not proportional to the volume of synthetic data (as shown in Figure 4). However, within a certain threshold, an increase in synthetic data volume is correlated with accuracy improvement (Figure 5).

By the way, they have withdrawn their paper for some reasons.

Q-9: The discussion on overfitting in line 483 requires more clarity and experimental evidence to support the strong statement made. The current evidence on Pets dataset appears quite weak and insufficient.

A-9: Thank you for your suggestion. Below, we present the Train Accuracy, Validation Accuracy, and Accuracy Gap after convergence for all datasets. Notably, the train-validation accuracy gap is reduced on GenDataAgent, which can be seen as a form of mitigation for overfitting. The reason might be the synthetic data covers some cases that the real data ignore (the diversity introduced by LLaMA caption perturbation), thus enhancing the model's robustness and generalization capabilities.

Q-5: Can you also include robustness metrics, such as accuracy under common corruptions and perturbations as well as adversarial settings? Is the proposed method still better than the baselines?

A-5: To evaluate robustness, we introduced common corruptions such as Gaussian Blur and Speckle Noise, following the methodology outlined in [1]. All methods were trained on clean images and tested on corrupted images to assess their robustness.

Below is a comparison of our GenDataAgent with the SOTA baseline Real-Fake across six datasets:

Our method consistently outperforms Real-Fake across all datasets and corruptions. We will include these results in the final version of the manuscript.

[1] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR, 2019.

Q-6: Would the proposed method generalize across different Text2Image generation methods, such as SD3 and Kandinsky (does not have to be the two)?

A-6: The proposed GenDataAgent framework is indeed extendable to other TTI generation methods. However, the choice of SD1.5 in our experiments was motivated by its status as a common baseline in the literature [2,3], ensuring a fair and consistent comparison with existing methods.

It is important to note that SD3 and Kandinsky employ different architectures and mechanisms, which could affect the application of certain techniques used in GenDataAgent:

• Fine-Tuning Challenges: Both SD3 and Kandinsky lack official support for LoRA fine-tuning with CLIP-based image features, making it non-trivial to directly adapt these models to the target dataset.
• Framework Adaptability: Despite these challenges, the principles of GenDataAgent—such as marginal sample selection, VoG filtering, and LLM-based caption perturbation—are general and can be adapted to other Text-to-Image models with suitable modifications. We plan to explore these adaptations in future work and will include a detailed discussion on the potential challenges and solutions for extending GenDataAgent to different TTI models in the final version.

[2] Image Captions are Natural Prompts for Text-to-Image Models, arxiv, 2023.

[3] REAL-FAKE: EFFECTIVE TRAINING DATA SYNTHESIS THROUGH DISTRIBUTION MATCHING, ICLR, 2024.

Q-7: Would the proposed method generalize to other discriminative CV tasks such as image retrieval or object detection?

A-7: Our GenDataAgent can be extended to other computer vision tasks involving paired data, such as object detection and image segmentation. For instance, in object detection tasks, given bounding boxes and categories, generative models like GLIGEN [4] and LayoutDM [5] can be employed to synthesize corresponding images. Similarly, given a segmentation map, UniControl [6] can generate the corresponding images. In other words, as long as paired data exists, GenDataAgent can be adapted to these tasks by substituting the general Stable Diffusion model with task-specific controllable generative models.

[4] GLIGEN: Open-Set Grounded Text-to-Image Generation, CVPR, 2023.

[5] LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation, CVPR, 2023.

[6] UniControl: A Unified Diffusion Model for Controllable Visual Generation in the Wild, NeurIPS, 2023.

Q-1: The paper's critical weakness lies in its lack of novelty, as most of the techniques employed, including the VoG metric, are derived from previous works. This aspect positions the paper more as an engineering-heavy work rather than an original contribution.

A-1: Leveraging large volumes of synthetic data for downstream tasks remains a significant challenge, with no comprehensive pipeline in existing literature. GenDataAgent addresses this gap by introducing a unified framework that ensures synthetic data is diverse, in-distribution, and task-specific through:

• LLM-based Caption Perturbation: Repurposes LLMs, traditionally used for text generation, to diversify image captions for enhanced synthetic data variation.
• VoG Filtering: Adapts the VoG metric from real datasets to detect outliers in synthetic data, aligning it with the target distribution.
• Marginal Sample Utilization: Instead of applying weighted losses, uses challenging samples to guide synthetic data generation, dynamically aligning with model requirements.

Though GenDataAgent involves engineering aspects, its principled framework lays a foundation for future advancements by integrating and repurposing existing techniques into a cohesive pipeline.

Q-2: Specifically, the extensively discussed VoG metric appears somewhat subjective, as the choice of using gradients from epochs 10, 20, and 30 to compute the metric lacks proper justification or guidelines for other benchmarks or tasks.

A-2: The choice of epochs 10, 20, and 30 for computing the VoG metric corresponds to the 1st, 2nd, and 3rd iterations of on-the-fly fine-tuning, balancing model refinement with resource efficiency. Our ablation study (Table 5) shows that increasing the number of checkpoints does not significantly improve performance. Therefore, we use the minimum of 3 checkpoints to reduce computational overhead while maintaining effectiveness.

Q-3: Moreover, while the proposed method demonstrates significant improvement over using only real data, it performs only marginally better than other synthetic augmentation methods, such as Real-Fake and Internet Explorer, while also taking much longer GPU hours.

A-3:

1. Our GenDataAgent is developed upon the SOTA method Real-Fake, which serves as a strong baseline with saturate accuracy in the "synthetic data augmentation" setup. Furthermore, the proposed techniques demonstrate greater improvements in the "synthetic data only" setting, as illustrated in Table 1. Additionally, Internet Explorer uses part of our components (SD adaptation, LLaMA caption perturbation) so that it also performs well.
2. In our ablation study, we also introduce two lightweight versions: GenDataAgent (1:0.1) and GenDataAgent (1:0.5). As illustrated in Figure 4, the training time for GenDataAgent (1:0.1) is close to that of Real-Fake (1:1), while delivering comparable or superior performance. GenDataAgent (1:0.5) already exceeds the SOTA Real-Fake across all datasets. Additionally, Figure 4 shows a breakdown of training time, indicating that further acceleration can be achieved by using more efficient SD and LLM models.

Q-4: In fact, synthetic data augmentation has been extensively studied in the context of image classification, and the observed improvement may merely be a result of overfitting to those small-scale benchmarks and may not generalize to real-world or in-the-wild settings.

A-4: Our focus on small-scale benchmarks is intended to demonstrate that synthetic data can address gaps in training data by covering cases that might otherwise be ignored. Importantly, the results suggest that synthetic data augmentation is unlikely to lead to overfitting on these benchmarks:

• If synthetic data augmentation caused overfitting to small-scale benchmarks, increasing the volume of synthetic data would likely continue to improve performance. However, as shown in Figure 4, performance drops when using 10x synthetic data (Real-Fake 1:10). This indicates that overfitting is not a significant concern in this context.
• The GenDataAgent pipeline provides flexibility by allowing the user to adjust the VoG filtering threshold. This balances diversity with in-distribution alignment, ensuring that the synthetic data contributes meaningfully without overwhelming the training process. This adaptability makes GenDataAgent more robust than static augmentation methods like Real-Fake, which lack dynamic controls.

Q-5: This paper would benefit from a more comprehensive literature review. Specifically, it overlooks some closely related works, such as [a]. I recommend that the authors include a discussion of this work and its connections to their research. This will help to contextualize their contributions within the existing body of knowledge. [a] A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptation. CVPR, 2023.

A-5: Thank you for the valuable suggestion. Incorporating [a] will strengthen the contextualization of our work.

1. The domain adaptation (DA) techniques in [a] effectively reduce distribution shifts between real and synthetic data, offering insights that complement our approach.

2. GenDataAgent also aligns distributions through SD fine-tuning, it adds:

• Marginal Sample Selection: Generates challenging samples to address model weaknesses.
• LLM Caption Perturbation: Increases prompt diversity for richer synthetic data.
• VoG Filtering: Ensures synthetic data remains in-distribution and relevant.

These techniques are designed by general principle and can be combined with other techniques like DA in [a]. We have incorporated [a] into the related work section, highlighted in red, and will provide a more detailed discussion in the final version.

Q-6: The authors claim that maintaining a consistent prompt format for both adapting stable diffusion and generating synthetic data ensures greater alignment between the distribution of the generated data and the target data. However, it is not entirely clear how a consistent prompt format alone guarantees this distribution alignment. I recommend that the authors provide a more detailed explanation of this relationship.

A-6: Thank you for the feedback. SD uses the CLIP [1] text encoder to generate embeddings for image generation. Consistent prompt formatting during LoRA fine-tuning and synthetic data generation ensures stable embeddings, reducing variance and improving alignment with the target dataset.

However, distribution alignment relies more on LoRA fine-tuning, which adapts SD to the target dataset by refining its generative capabilities. Consistent prompts enhance embedding coherence, but fine-tuning ensures alignment by tailoring the model to the domain.

We will clarify these complementary roles in the final version.

[1] Contrastive Language-Image Pre-Training, ICML, 2021.

Q-7: Additionally, it would be valuable to discuss whether incorporating domain adaptation (DA) techniques, as described in [a], during the model fine-tuning process could lead to further performance improvements. Exploring this potential synergy could enhance the robustness of the findings presented in the manuscript.

A-7: Thank you for the insightful suggestion. Incorporating domain adaptation techniques from [a] during the model fine-tuning process could indeed further enhance the alignment between the generated synthetic data and the target dataset distribution. This alignment has the potential to improve the quality of synthetic data, addressing one of the key bottlenecks in synthetic data augmentation. We will explore this integration and discuss its potential impact in the final version of the manuscript.

Q-1: There exists a conflict: tailoring synthetic data generation for specific downstream tasks limits its generality, while using it for general purposes diminishes its specialty. The authors could provide further insights to clarify this issue.

A-1: Adapting SD to a target dataset via LoRA fine-tuning tailors synthetic data generation to specific tasks, potentially reducing generalizability to other datasets. However, this does not imply overfitting, as SD’s inherent diversity ensures coverage of cases missing in limited real data.

GenDataAgent offers flexibility to balance this trade-off:

• For Robustness: Raising the VoG threshold accepts more outliers, increasing diversity and enhancing generalization.
• For Specificity: Lowering the VoG threshold focuses on a specific region, optimizing task-specific performance. This balance between generality and specificity is a design choice, and GenDataAgent provides tools to adapt as needed.

Q-2: Is it necessary for synthetic data to be in-distribution? While aligning the distribution of synthetic data with the training data is important, it does not align with the distribution of the test data, particularly in out-of-distribution (OOD) scenarios. This raises the question of whether this paper focuses solely on generalization within in-distribution contexts, potentially limiting its applicability to real-world situations where test data may vary significantly. The authors may evaluate the proposed agent's capability for OOD generalization.

A-2: Thank you for the observation. While keeping synthetic data in-distribution is important, including outliers can improve generalization, especially in OOD scenarios.

• As shown in Table 4 (Appendix D), higher VoG filtering ratios (fewer outliers) don’t always yield the best results, suggesting that outliers can expose the model to valuable variations.
• For OOD scenarios, we recommend lowering the VoG filtering threshold to include more diverse and potentially OOD samples, enhancing robustness to unseen data.

We will expand on OOD generalization in the final version.

Q-3: Though the on-the-fly generation of relevant data saves the storage cost and improves relevance to the task at hand, how reusable and sustainable is synthetic data?

A-3:

1. We also provide an offline version of GenDataAgent, where the synthetic data generation is done before training and on-the-fly generation becomes on-the-fly selection. In this way, all synthetic data are saved and thus reusable. That’s a trade-off between storage efficiency and synthetic data reusable.
2. Moreover, an ablation study for threshold selection (for instance, adjusting VoG filtering threshold for different scenarios) can be realized as all synthetic data are saved.

Q-4: For marginal training samples, synthetic data that exhibit higher variance in gradient updates are specifically chosen. Does this imply that these synthetic samples are those that the current model struggles to learn effectively? If so, it suggests that these samples may be more challenging or informative, potentially indicating areas where the model's performance is weak. This focus on high-variance samples could enhance the model's robustness by addressing its limitations and improving its ability to generalize to diverse data distributions. Further clarification on this selection strategy and its implications for model training would be beneficial, e.g., discussing it with the selection criteria in active learning.

A-4: Yes, the marginal sample selection strategy targets synthetic samples with high variance in gradient updates, indicating areas where the model struggles. This approach enhances robustness and generalization by focusing on challenging regions.

Active Learning:

• Objective: Selects unlabeled samples for labeling, focusing on informativeness without prior label knowledge.
• Dynamic Behavior: Relies on static difficulty measurements without feedback from the downstream task.

Marginal Sample Selection:

• Objective: Leverages synthetic sample labels to compute marginal scores, estimating difficulty relative to the downstream model.
• Dynamic Behavior: Dynamically reassesses sample difficulty in an on-the-fly framework as the downstream model evolves, enabling adaptive selection.
• Focus: Marginal sample selection specifically enhances synthetic data for training, whereas active learning targets real-world unlabeled data for annotation.

We will include this discussion in the final paper to clarify these distinctions and their implications.

I have reviewed the authors' rebuttal along with the comments from the other reviewers. I appreciate the thorough discussions and analyses provided, which effectively address many of my concerns.

The revised paper adequately incorporates the suggested works for discussion, which is commendable. For the final version, I expect the authors to include the valuable analyses and discussions from the rebuttal in the main paper or supplemental material.

Given these improvements, I will raise my current rating.

Q-1: The proposed method is slow --- it can be up to 350 times slower than using only real data and 2 times slower than the comparing method (Real-Fake). Although the accuracy increases, this can be a major bottleneck.

A-1: While the training overhead of GenDataAgent (1:1) is higher compared to using only real data or the Real-Fake method, it is important to note that inference time remains unaffected across all methods. This ensures that real-world application performance, where inference time is critical, is not impacted by the additional training cost.

To address the training overhead, we provide two lightweight versions in the ablation study:

• GenDataAgent (1:0.1): Achieves comparable training time to Real-Fake (1:1) while delivering performance that is on par or better.
• GenDataAgent (1:0.5): Surpasses the performance of Real-Fake across all datasets with a moderate increase in training cost.

Furthermore, as shown in Figure 4, the training time can be further optimized by decoupling specific components and employing more efficient versions of SD and LLMs. This demonstrates the scalability of GenDataAgent and its potential for integration with future advancements in generative model efficiency.

Q-2: Related to the speed issue, I assume the reason is because it's the on-the-fly method. Can the method be adapted/switched between online and offline modes based on its need? Are there other ways to improve the speed?

A-2: Yes, GenDataAgent can be adapted to include an offline mode for situations where speed is critical. Specifically:

• Offline Mode: Key processes such as SD adaptation, LLaMA caption perturbation, and synthetic data generation can be performed offline. The on-the-fly mechanism then shifts to selecting relevant samples from a pre-generated synthetic data pool, significantly reducing online training time. As shown in Figure 4, this reduces the training phase to primarily classifier fine-tuning, which is comparatively fast.

Additionally, there are potential ways to further accelerate the process:

• Asynchronous Generation: Synthetic data generation and classifier fine-tuning could be parallelized. This approach minimizes waiting times by overlapping the generation and training processes, reducing the overall duration. This can be explored in future work.

Q-1: Lack of technical novelty. Though the motivation of the idea makes sense, my biggest concern is on the technical novelty of the proposed algorithm. The proposed design components on ensuring sample diversity, generation distribution alignment with real data distribution, and the marginal sample selection are all coming from existing literature or prior works.

A-1: The challenge of effectively leveraging large-scale synthetic data to enhance downstream tasks remains a significant open problem, and a general solution or pipeline for addressing this issue is still underexplored. The proposed GenDataAgent offers such a pipeline, addressing key challenges like ensuring diversity, maintaining alignment with real data distributions, and targeting marginal samples as generator feedback.

Although the individual components (e.g., LLM caption perturbation, VoG filtering, and marginal sample targeting) are grounded in existing literature, their application in GenDataAgent is novel in both purpose and integration. For example:

• LLMs are commonly used for generating human-like conversational responses, but we repurpose LLMs to perturb image captions, thereby enhancing the diversity of prompts and, in turn, the synthetic data distribution.
• In prior works, marginal samples—those hard to classify—are typically handled with weighted loss adjustments. In contrast, we utilize these samples as direct feedback to improve the generation process, enabling better alignment between synthetic and real data distributions.

Q-2: Lack of sufficient experiment results to validate the effectiveness of the proposed methods and the claims in the contributions. The contribution claims the effectiveness of the proposed method from the three aspects of marginal sample selection, sample diversity and the distribution alignment to real data distribution. However, from the ablation study, it appears that even without these proposed improvements, the classification model trained with synthetic data also show good improvements. And the improvements from the proposed claims are relatively smaller compared to the gains from directly using synthetic samples from SD without these proposed changes, as in Table 3. It then raises the question on the effectiveness of the propose method and the claimed contributions.

A-2:

1. GenDataAgent builds on the SOTA Real-Fake method, which already achieves saturated accuracy in the “synthetic data augmentation” setting. This makes further improvements challenging. However, our techniques demonstrate larger performance gains in the “synthetic data only” setting, as evidenced in Table 1, where the baseline's reliance on real data is reduced, and the proposed methods prove their effectiveness.

2. To further address the concerns about scalability and robustness, we conducted a scaling ablation study, varying the ratio of synthetic to real data (#synthetic data = 10x #real data and other ratios), as shown in Figure 4. Notably, Real-Fake's performance deteriorates when the volume of synthetic data increases across most datasets. In contrast, GenDataAgent remains stable, indicating its ability to handle varying synthetic data ratios effectively. This highlights a critical limitation of existing methods like Real-Fake: their inability to maintain performance when scaling synthetic data. Our proposed techniques mitigate this limitation, ensuring consistent performance improvements even as synthetic data scales.

Q-3: Introduction of additional information from SD. In Table 1 and 2, it shows that the classification performance does increase by adding synthetic samples following the proposed method. However, it is unclear that if the gain is indeed coming from these synthetic augmentation or just distilled information from the diverse datasets which the image generator Stable Diffusion is trained over.

A-3:

1. We think the gain comes from both aspects, and they cannot be disentangled when using synthetic data augmentation. For one thing, the synthetic data augments the real data and thus enlarges the training data volume. For another, the synthetic data are not supposed to provide totally the same information as real ones. To make the augmentation meaningful, the synthetic data needs to contain some attributes that the real one doesn’t have.
2. In the SD scenario, we adapt SD to target data distribution by LoRA fine-tuning, yet the generated images still benefit from the generating ability of SD, i.e., the diverse datasets which SD is trained over. Even though the performance gain mostly comes from distilling information from SD, this approach is also effective as it avoids injecting massive unrelated data into the downstream task.

Q-4: Can you explain the difference of the proposed marginal sample selection algorithm to Focal loss which also adjusts the sample weights based on the difficulty of the sample indicated by the raw softmax score of the classifier being learnt.

A-4:

1. Focal loss assigns a weighted loss based on the difficulty of the sample by introducing a modulating factor $(1-p_t)^{\gamma}$, which aims to down-weight the loss of "easy" examples. It focuses on a very im-balance dataset and directly impacts the training process.
2. First, GenDataAgent does not require any assumption of the target dataset. Second, the marginal sample selection picks challenging examples and uses them to guide the generator. After generating synthetic data, real and synthetic data are treated equally. The data distribution is transparent, so we do not assign different weights to the data. There, the difficulty level used in marginal sample selection does not directly affect the training.

Q-5: Line 13 of algorithm1, S_f is not defined in section 3.4, pls explain.

A-5: Thank you for pointing it out. $S_{f,j}$ is defined in L267, which is a subset of synthetic dataset $S_c$ with minimum sum of VoG scores. We refine this formula by adding the constraint to the number of filtering samples, highlighted in red color.

Dear Reviewer Ln52,

We sincerely appreciate your time and effort in reviewing our manuscript. We have carefully reviewed and responded to the questions and concerns raised in your initial review:

1. We clarify that the novelty of our method lies in proposing a general pipeline for focused, diverse, and in-distribution synthetic data augmentation, which is underexplored in previous works [1,2,3,4].
2. Regarding the experiments, we demonstrated a significant performance gain in the "synthetic data only" setting (Table 1), as well as in the scaling ablation of synthetic data volume (Figure 4).
3. Additionally, we provide clarification on the role of SD, the difference between our marginal sample selection and Focal loss, and have refined the notation in the manuscript to improve clarity.

As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have adequately addressed your concerns. We are pleased to report that all other reviewers (NMJi, rGcL, and rpaL) indicated that their concerns have been satisfactorily resolved. If you still have any remaining questions or if any parts of our work need further clarification, please do not hesitate to let us know.

Thank you in advance for your help.

Sincerely,

Authors

[1] Fake it till you make it: Learning transferable representations from synthetic imagenet clones. CVPR, 2023.

[2] Image captions are natural prompts for text-to-image models. arxiv, 2023.

[3] Is synthetic data from generative models ready for image recognition? ICLR, 2023.

[4] Real-fake: Effective training data synthesis through distribution matching. ICLR, 2024.