UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation

Dear Reviewer,

Thank you for your thoughtful feedback and insightful questions. Your comments have helped us to identify areas where we can improve the clarity and rigor of our paper. We have carefully considered your suggestions and provide our detailed responses below.

Q1. The proposed framework is complex due to the integration of multiple modules.(Weakness 1)

We thank the reviewer for this thoughtful concern. Each module in our framework is designed with clear modularity, allowing independent development and evaluation. This facilitates practical adoption while enabling the components to work synergistically for the substantial performance gains observed in our experiments.

Q2. Efficiency concerns regarding the overall computational cost of the framework.(Weakness 2)

We thank the reviewer for highlighting this important aspect. While UtilGen integrates several components to improve sample quality and task alignment, we would like to clarify that the overall framework remains computationally reasonable. As detailed in Appendix D, UtilGen exhibits favorable GPU memory usage and runtime compared to existing baselines. We believe these costs are justified by the substantial accuracy improvements, and we plan to further explore efficiency optimizations in future work.

Table: Computational cost (in hours) comparison on ImageNet-1K for generating 1000 images per class

Table: GPU memory usage (peak usage in GB) per stage

Q3. The sensitivity of overall method on choice of different classifier such as CNNs vs pretrained transformer models.(Weakness 3)

We thank the reviewer for this valuable point. To investigate the sensitivity of UtilGen to the choice of classifier, we conducted experiments on ImageNet-100 using various architectures, including CNNs (ResNeXt, WideResNet, MobileNet) and pretrained transformer models (CLIP). As shown in the table below, UtilGen consistently improves downstream performance across different classifiers, demonstrating robustness to the classifier choice. This validates that our weight network can effectively leverage diverse model architectures without being overly dependent on a specific backbone.

Q4. Compare the approach against simple augmentation or self-supervised pretrained models.(Weakness 4)

We thank the reviewer for the helpful suggestion. We compare UtilGen with Cutout[4] and CLIP on ImageNet-100 using a 10× enlarged training set that combines both real and synthetic data. For fair comparison, Cutout[4] and UtilGen are trained using ResNet-50. UtilGen significantly outperforms Cutout[4] and even slightly surpasses CLIP in the zero-shot setting, while fine-tuning CLIP on the same augmented dataset yields the best performance. These results highlight the effectiveness of our approach compared to simple augmentation and self-supervised pretrained models.

Q5. Sensitivity to the choice of initial classifier architecture and train/val data selection.(Question 1)

As noted in Q3, UtilGen is not sensitive to the initial classifier architecture, showing consistent gains across different backbones. It is also robust to variations in training/validation data selection: for training the weight network, we randomly sampled a few real images per class, and observed consistent improvements across different random splits. This stability indicates that UtilGen is not sensitive to the choice of train/val data.

Q6. Potential scenarios or dataset types where UtilGen may not perform well.(Question 2)

Since UtilGen relies on a few real images for guidance, very low-quality inputs (e.g., noisy or low-resolution) may reduce generation quality by affecting textual inversion. Nonetheless, this has minimal impact on overall performance and does not compromise our main contributions. This also suggests a valuable direction for future work—improving robustness to low-quality real data.

Q7. Sensitivity to hyperparameter choices and guidelines for tuning.(Question 3)

We appreciate the reviewer’s question. The implementation details and hyperparameter configurations for each component of our framework are provided in Appendix B.

Among all components, the most sensitive hyperparameter is the guidance scale used in the diffusion model for image generation. We conducted experiments to study its impact:

Q8. Performance when using pretrained vision transformer models as the classifier.(Question 4)

We evaluated UtilGen using CLIP as the downstream classifier and compared it with DataDream under the same setting on the ImageNet-100 dataset. Both methods use only 5× synthetic data for training. UtilGen consistently achieves better performance:

This shows that UtilGen remains effective even when using pretrained transformer-based classifiers.

References

[1] Controlled Training Data Generation with Diffusion Models

[2] Expanding Small-Scale Datasets with Guided Imagination

[3] DataDream: Few-shot Guided Dataset Generation

[4] Improved regularization of convolutional neural networks with cutout.

Thank you once again for your valuable feedback, which has helped us to significantly improve the scientific rigor of our paper. We will incorporate the corresponding revisions and clarifications into the updated manuscript. If you have any further questions or concerns, we would be glad to address them.

Sincerely,

The Authors

Dear Reviewer,

Thank you for your thoughtful feedback and insightful questions. Your comments have helped us to identify areas where we can improve the clarity and rigor of our paper. We have carefully considered your suggestions and provide our detailed responses below.

Q1. Concern regarding the claim of not requiring a classifier-based method to evaluate data utility, and the potential reliance on training and validation data for utility learning (Weakness 2)

We appreciate these insightful questions and would like to provide further clarification.

- Evaluation without exhaustive classifier-based re-training: We appreciate the reviewer’s comment and would like to clarify that our claim of not requiring classifier-based methods refers to avoiding exhaustive re-training of downstream classifiers. Instead, it uses a weight network as a surrogate utility estimator, which enables efficient evaluation of synthetic samples based on their estimated contribution to downstream performance. This avoids the computational burden associated with repeatedly training classifiers on different subsets of generated data.

- Training data requirements of the weight network: We thank the reviewer for raising this important point. While the weight network is indeed trained on a large number of samples, we would like to clarify that this does not require a large-scale real labeled dataset. In fact, UtilGen naturally supports few-shot learning, requiring only minimal labeled data while maintaining competitive performance. The remaining training data for the weight network consists entirely of synthetic samples. This capability is empirically validated in our comparative experiments with DataDream[4] on ImageNet-100, where UtilGen demonstrates superior effectiveness in data-scarce regimes.

Experimental Setup:

• Half of the images are simultaneously used for:
  • The training set for Weight-Net training
  • Textual inversion optimization
• The remaining half serves as the validation set for Weight-Net training.

Q2. The Clarity of Section 3.1 and Equations (Weakness 1 & Question 1)

We thank the reviewer for the helpful suggestion, and we will provide a more transparent and detailed description of the equations in Section 3.1 to improve clarity and understanding. Below we provide a step-by-step clarification of the interaction between the classifier parameters $\theta$, the weight network parameters $\phi$, and the computed weights $\omega_i$.

---

Bi-Level Optimization Procedure

Our method employs a bi-level optimization framework with two alternating stages. The goal is to meta-learn a weight network $\mathcal{W}_\phi$ that assigns utility scores $\omega_i \in [0, 1]$ to data points, such that samples with higher utility contribute more effectively to downstream generalization.

Stage 1: Inner Optimization — Classifier Training

Given a current weight network $\mathcal{W}_\phi$, we first compute the weight $\omega_i$ for each training example using the weight network.

The classifier parameters $\theta$ are then optimized on the merged training set using a weighted loss:

$$\theta^*(\phi) = \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} \omega_i \cdot \mathcal{L}(f(x_i; \theta), y_i) \quad \text{(Eq. 2)}$$

Note that $\theta^*$ is a function of $\phi$ because the weights $\omega_i$ used in training depend on $\mathcal{W}_\phi$. This shows that the optimal classifier $\theta^{\ast}(\phi)$ is implicitly determined by the weight network parameters $\phi$ via the computed weights.

Stage 2: Outer Optimization — Weight Network Update

Once the classifier has been trained with the current weights $\omega_i$, we evaluate its generalization on a validation set. The goal of the outer optimization is to update $\phi$ such that the classifier trained with weighted loss performs well on validation set:

$$\phi^{\ast}(\theta) = \arg\min_{\phi} \frac{1}{M} \sum_{j=1}^{M} \mathcal{L}(f(x_j; \theta^{\ast}(\phi)), y_j) \quad \text{(Eq. 3)}$$

This outer loop updates $\phi$ so that the next round of training with newly computed weights $\omega_i$ better guide the classifier training, leading to improved downstream generalization.

This process is repeated iteratively: the weight network is progressively refined to better estimate the utility of each sample, thereby guiding the classifier to train more effectively using weighted loss in the next iteration. Over time, the weight network progressively learns to assign higher weights to training samples that contribute more significantly to improved validation performance. In other words, the learned weights act as a surrogate for each sample's utility with respect to downstream generalization.

---

Intuition

In essence, we are meta-learning the weight network to simulate the effect of each training sample on validation performance:

• If increasing the weight of sample $x_i$ leads to a lower validation loss, $\mathcal{W}_\phi$ is encouraged to assign it a higher weight.
• Conversely, samples with low or negative impact on generalization are down-weighted.

This interaction forms a feedback loop between the classifier training (inner loop) and the weight assignment (outer loop), enabling the weight network to serve as a proxy utility estimator for guiding data generation in later stages.

We appreciate the reviewer’s reference to Fig. 2 — we will further revise Section 3.1 to explicitly describe these dependencies and clarify the semantics of the equations involved.

Q3. Typos and Wording Issues.(Question 2)

We thank the reviewer for pointing out the typos and wording issues. We have carefully revised the manuscript to correct these and other textual errors for improved clarity and accuracy.

References

[1] DataDream: Few-shot Guided Dataset Generation

Thank you once again for your valuable feedback, which has helped us to significantly improve the scientific rigor of our paper. We will incorporate the corresponding revisions and clarifications into the updated manuscript. If you have any further questions or concerns, we would be glad to address them.

Sincerely,

The Authors

Dear Reviewer,

We sincerely appreciate your thoughtful feedback and insightful questions, which have helped identify key areas for improving our paper's clarity and academic rigor. After careful consideration of your suggestions, we provide the following detailed responses.

Q1. How is the bilevel optimization solved?(Weakness 1.a)

We implement the bi-level optimization through one-step gradient unrolling, where the classifier performs a single gradient update step on the training data, and then the weight network is updated based on the validation loss. While gradient unrolling can be memory-intensive, in our setup it remains computationally tractable due to the lightweight MLP architecture of the weight network and the limited unrolling steps. This approach effectively avoids excessive computational and time costs, making the optimization practical and efficient. Below, we report the peak GPU memory usage during training with and without the weight network on ResNet-50 as the classifier:

As shown, incorporating the weight network introduces only a modest increase in memory consumption, demonstrating the practicality of our bilevel optimization implementation.

Q2. How is the prompt optimized during the denoising stage?(Weakness 1.b)

We thank the reviewer for raising this important point. In UtilGen, we optimize the prompt embeddings by backpropagating through the denoising process of the diffusion model. This involves computing gradients through denoising steps, which can indeed incur additional computational overhead. We acknowledge this as a limitation of our current design. However, in practice, the overhead remains manageable because we only optimize one prompt per class, rather than per instance. This significantly reduces the total optimization cost and makes the method practically feasible.

To evaluate the computational efficiency, we compare our method UtilGen with GAP[1] in terms of memory and runtime for prompt optimization. A broader comparison of UtilGen and all other baselines in terms of GPU memory usage and runtime is provided in Appendix D. In future work, we plan to explore more efficient prompt optimization strategies to further reduce the computational burden while preserving performance.

Q3. Diversity analysis of synthetic data.(Weakness 2)

We thank the reviewer for the valuable suggestion. UtilGen maintains data diversity during the optimization of data generation. To quantitatively evaluate the diversity of synthesized samples, we compare the mean intra-class diversity of synthetic data produced by Stable Diffusion v2.1, DataDream[3], and UtilGen on ImageNet-100.

The diversity metric is defined as the mean pairwise cosine distance between CLIP (ViT-L/14) features within each class, averaged across all classes. Higher values indicate greater intra-class diversity, which typically benefits model generalization.

These results demonstrate that UtilGen produces diverse samples within classes compared to the baselines, supporting the claim that our utility-guided generation process also does not harm or suppress diversity, which is beneficial for downstream tasks.

Q4. How does the performance gap between UtilGen and other baselines evolve under different synthesis budgets?(Question 1)

We thank the reviewer for the insightful question. We conducted a controlled comparison of UtilGen and four baselines (SD v2.1, GIF[2], GAP[1], and DataDream[3]) under varying synthesis budgets (i.e., the ratio of synthetic to real data). As shown in the table below, UtilGen consistently outperforms all baselines, and the performance gap becomes more pronounced as the synthesis budget increases.

This trend suggests that UtilGen better scales with larger amounts of generated data, owing to its utility-centric data augmentation.

Q5. Why not perform textual inversion before utility learning to reduce distribution shift?(Question 2)

We thank the reviewer for the insightful suggestion. In our initial design, we trained the utility MLP using synthetic samples directly generated from the pre-trained SD v2.1 model, without applying textual inversion. This was motivated by the observation that such raw generations often contain low-utility samples (e.g., off-distribution or task-irrelevant data), which we believed would help the utility MLP better distinguish varying levels of usefulness.

Following the reviewer’s suggestion, we conducted additional experiments using class-adapted samples generated via textual inversion to train the utility MLP. These adapted samples reduced the distribution shift encountered during downstream diffusion alignment and DPO labeling. Our experiments confirmed that using these samples indeed improves the effectiveness of the utility MLP.

Moreover, we also found that combining both types of data—raw samples from SD v2.1 and class-adapted samples from textual inversion yields the best overall performance. This hybrid training strategy not only mitigates distribution mismatch but also enhances the MLP’s ability to assess utility across a broader quality spectrum.

Q6. What is the relationship between sample weights and model uncertainty or loss?(Question 3)

In our analysis, we observe a nonlinear relationship between the sample weights predicted by the utility MLP and the model’s cross-entropy loss. While we cannot include plots here, the general trend roughly forms an “inverted U-shaped” curve, where the weights tend to increase with the loss when the loss is relatively small, but starts to decrease as the loss becomes large.

This pattern aligns well with intuition: samples with very low loss are often easy or redundant, thus contributing less utility to model performance; samples with very high loss are likely noisy or exhibit significant distribution shifts, and are therefore also assigned lower weights.

Regarding the evolution of this relationship across different optimization stages, our experiments indicate that the correlation between sample weights and cross-entropy loss gradually stabilizes to this nonlinear “inverted U-shaped” trend as training progresses.

Q7. Would iterative utility learning improve performance?(Question 4)

We thank the reviewer for this insightful suggestion. We have conducted experiments to evaluate the iterative training of the utility MLP as proposed. Specifically, we compared two settings:

1. Direct 5× augmentation: generating all synthetic samples at once using a single utility MLP trained initially.

2. Iterative augmentation: first generating 1× synthetic data, fine-tuning the utility MLP based on the augmented set, then generating an additional 1× data with the improved MLP, and repeating this process until reaching 5× augmentation.

Our results show that the iterative approach achieves better performance compared to direct generation, confirming that adapting the utility MLP at different training stages can bring additional gains. However, this improvement comes at the cost of increased computational overhead due to repeated MLP fine-tuning and data generation steps.

References

[1] Controlled Training Data Generation with Diffusion Models

[2] Expanding Small-Scale Datasets with Guided Imagination

[3] DataDream: Few-shot Guided Dataset Generation

Thank you once again for your valuable feedback, which has helped us to significantly improve the scientific rigor of our paper. We will incorporate the corresponding revisions and clarifications into the updated manuscript. If you have any further questions or concerns, we would be glad to address them.

Sincerely,

The Authors

Dear Reviewer,

We sincerely appreciate your thoughtful feedback and insightful questions, which have helped identify key areas for improving our paper's clarity and academic rigor. After careful consideration of your suggestions, we provide the following detailed responses.

Q1. Concern on the interpretability of the utility weight network and why certain samples are rated as high-utility.(Weakness 1)

Thank you for highlighting the need for further clarification regarding why certain samples are rated as high-utility. As mentioned in our paper, the weights learned by our network effectively reflect sample utility for downstream tasks. This behavior is theoretically grounded in the Meta-Weight-Net[1] framework, where the updating equation for Weight-Net parameters can be explained by that samples that better comply with the knowledge derived from validation data (i.e., those that help improve the classification model's performance on downstream tasks) will have their weights increased, while harmful samples are down-weighted. Although we cannot show the full derivation here due to space constraints, Meta-Weight-Net[1] provides rigorous proofs (Section 2.3 and Appendices B–C). Essentially, our weight allocation network inherits this property, where higher weights reliably indicate samples that enhance classification performance, thus guiding effective utility optimization. We will include further explanation in the revised version to improve clarity.

Q2. Suggest including DisCL[2] as a task-adaptive baseline for comparison.(Weakness 2)

Thank you for the valuable suggestion. We have now included a comparison between UtilGen and DisCL[2] on the ImageNet-100 dataset. In addition, our main paper also presents a comparison with GAP[3], a feedback-driven approach that focuses on generating hard samples for downstream tasks. This unified comparison allows us to evaluate three different feedback-driven generation strategies. The evaluation covers two settings: synthetic data only, and a mixed setting combining synthetic and real data. The results confirm the superiority of our utility-guided data generation method compared to other baselines that are also guided by downstream task feedback.

We will include additional experimental comparisons and discussion with DisCL[2] in the revised manuscript to better position UtilGen among task-adaptive generation methods.

Q3. Analysis of cost-benefit tradeoff compared to collecting more labeled real data.(Weakness 3)

We thank the reviewer for raising this important point regarding the cost-effectiveness of our approach. Compared to manually collecting and annotating real data, UtilGen significantly reduces both time and cost for dataset expansion. Specifically, manually annotating 10,000 images (e.g., collecting 100 images for each of 100 classes), according to Masterpiece Group, would typically take around 2 weeks and cost approximately $800. In contrast, generating the same amount of data with UtilGen takes only about 0.94 hours and costs roughly $20 for renting 8 V100 GPUs according to Google Cloud. Furthermore, on the ImageNet-100-Subset dataset (which contains 100 classes with 100 real images per class), we achieve better performance than the real-data baseline by using only 5× synthetic data. Generating this 5× synthetic dataset requires approximately 4.70 hours, costing around $100 for renting 8 V100 GPUs.

Q4. What enables UtilGen to achieve strong performance in the synthetic-only setting, unlike prior work?(Weakness 4)

We think this breakthrough in synthetic-only training stems from shifting the optimization objective of synthetic data toward enhancing utility for downstream tasks, thereby aligning the generative model’s optimization objective with the downstream task goals.

Previous methods primarily focus on enhancing the visual quality of synthetic images; however, there often exists a gap between generating visually high-quality data and producing data that effectively improves downstream model performance. In particular, many approaches struggle to directly optimize the utility of synthetic data for target tasks.

On the other hand, efforts that concentrate on optimizing intrinsic data attributes may introduce new limitations. For example, DataDream[4] aims to enhance image fidelity and better align synthetic samples with the real data distribution. While effective in certain respects, this process can inadvertently reduce the diversity of the generated data. To quantify this, we compare the intra-class diversity of synthetic data generated by Stable Diffusion v2.1, DataDream[4], and UtilGen on ImageNet-100.The diversity metric is defined as the mean pairwise cosine distance between CLIP features within each class, averaged across all classes. Higher values indicate greater intra-class diversity.

Q5. Limited reusability of synthetic data across tasks.(Weakness 5)

Thank you for raising this important point. While we evaluated UtilGen's generalization across network architectures, we acknowledge that the paper did not explicitly analyze data reusability across broader task settings. Below, we offer additional analysis and discussion regarding this aspect.

If “different tasks” refers to directly using the generated data out-of-the-box as training data for different downstream models (e.g., other classifiers), we clarify that UtilGen-generated data demonstrates strong cross-architecture transferability. As evidenced by the table below, even if the utility predictor(i.e., Weight-Net) is trained using ResNet-50, the resulting data still yields strong performance when used with MobileNet and CLIP—suggesting that high-utility samples identified by the weight network are not tightly coupled to a specific architecture.

However, when task differences involve changes in dataset or domain, UtilGen does require re-optimization to align with new objectives. We recognize this as a limitation, though it does not undermine our main contribution—demonstrating that utility-guided generation leads to highly effective synthetic data for the target task. We will dedicate our future work to making UtilGen more generalizable across tasks.

Q6. Can this framework be extended to few-shot settings? (Weakness 5)

We appreciate this insightful question regarding the framework's adaptability to low-resource scenarios. As designed, UtilGen naturally supports few-shot learning, requiring only minimal labeled data while maintaining competitive performance. This capability is empirically validated in our comparative experiments with DataDream[4] on the ImageNet-100, where UtilGen demonstrates superior effectiveness in data-scarce regimes. Thank you for raising this important point, and we will include additional analysis of few-shot scenarios in our revised manuscript to further strengthen this discussion.

Experimental Setup:

• Half of the images are simultaneously used for:
  • The training set for Weight-Net training
  • Textual inversion optimization
• The remaining half serves as the validation set for Weight-Net training.

References

[1] Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting

[2] Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning via Image-Guided Diffusion

[3] Controlled Training Data Generation with Diffusion Models

[4] DataDream: Few-shot Guided Dataset Generation

Thank you once again for your valuable feedback, which has helped us to significantly improve the scientific rigor of our paper. We will incorporate the corresponding revisions and clarifications into the updated manuscript. If you have any further questions or concerns, we would be glad to address them.

Sincerely,

The Authors