Distribution-Aware Data Expansion with Diffusion Models

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We value your feedback on our method, noting that the prototype technology is sound and interesting and that it is easy to follow. We also thank you for recognizing that our method achieves SOTA performance compared to various baselines.

Re Weakness #1:

We apologize for missing this comparison. We have now included the baseline results in Table 1 and will update the revised paper accordingly. Our method outperforms both baselines. DA-Fusion modifies images while respecting semantic attributes but lacks fine-grained control and incurs additional time costs (around 14 minutes per class on a 3090 GPU) . Our training-free method uses hierarchical guidance for better alignment with the real distribution. DiffuseMix, which uses bespoke prompts and fractal images for augmentation, treats all datasets equally and may not handle distribution shifts well. Our method shows superior performance compared to these approaches.

Table 1: Comparison of accuracy in expanding Caltech-101 by $5\times$.

Re Weakness #2:

We follow previous work, such as GIF-SD and DA-Fusion, in conducting experiments on small datasets, as data augmentation strategies are often necessary in data-scarcity scenarios. Taking your advice into account, we have included results on ImageNet with $0.2\times$ expanding ratios, resulting in approximately 256K generated samples. As shown in Table 2, our method demonstrates improvements on large-scale datasets.

Table 2: Comparison of accuracy in expanding ImageNet by $0.2\times$. We trained ResNet18 with a $224 × 224$ resolution for 90 epochs.

Re Question #1:

You might have some misunderstanding regarding our method's principles. The energy function guidance used in our paper does not dramatically increase the time cost. As illustrated in Lines 310-315 (More Optimization Steps) of the manuscript, our method only introduces two additional optimization steps, which means calculating the gradient 2 times rather than 50 times. Therefore, our method does not dramatically increase the time cost as you suggested.

Re Question #2:

When $K=1$, the group-level prototype becomes the class-level prototype, leading to results similar to those with class-level prototypes. In Table 6 of our manuscript, performance changes are not apparent, as the differences between $p_c$ and $p_g$ are minimal in the small class variance dataset Caltech-101. Our method shows more benefits with fine-grained datasets with greater class variance, as illustrated in Table 3 with StanfordCars. Additionally, having more groups is not always beneficial; an excessive number of groups may cause prototypes to fit noise points or outliers, which could degrade performance.

Table 3: Prototypes comparison of accuracy in expanding StanfordCars by $2\times$. We trained ResNet50 with a 448 × 448 resolution for 128 epochs.

Thank you for recognizing that "most of the concerns have been addressed" and for considering an increase in the score. We truly appreciate your positive feedback! 

1. Regarding the ImageNet experiment, we further applied the Stable Diffusion (SD) baseline to expand ImageNet by $0.2\times$ and conducted expriments. The baseline method achieved $69.66\%$ accuracy. Our method surpasses the SD baseline by $0.29\%$ and the accuracy of training on the original dataset by $0.65\%$. This confirms that our method has an advantage in data expansion compared to the original SD method. In addition, training accuracy typically increases with a larger expansion ratio, and the performance gap between our method and existing methods tends to grow. This phenomenon has been validated across multiple datasets, as shown in Figure 4 of the manuscript. We are adding more experiments with ImageNet expansion ratios and will include these in the final version.

2. Thank you also for your suggestion to choose $K$ adaptively. We have explored this idea using classical adaptive clustering strategies. However, this introduces another parameter to tune, such as the neighborhood radius or cluster distance, which can be more challenging to adjust than $K$. Additionally, this poses challenges for parallel computation since $K$ may vary within each batch. Nevertheless, this is a valuable idea, and we will consider your suggestion for further exploration and optimization of adaptive $K$ selection methods.

Thank you once again for your valuable feedback. If you have any further questions regarding our rebuttal, we would be happy to provide additional clarification.

We sincerely appreciate the time and effort you invested in reviewing our paper. We address the raised concerns as follows:

Re Weakness #1:

Thank you for your valuable feedback. We appreciate your suggestion to provide a clearer explanation of the hierarchical prototypes. We recognize that the manuscript would benefit from a more detailed description of the prototype generation process. In Lines 135-136 and Lines 139-141, we describe how prototypes are selected or generated from the dataset. Specifically, our method first extracts feature vectors using a pre-trained image feature extractor. Class-level prototypes are then obtained by averaging feature vectors within each class. To refine these prototypes further, we apply the agglomerative hierarchical clustering algorithm [1] to group samples from the same class into $K$ clusters. The group prototypes are computed by averaging feature vectors within each cluster.

In the final version of the manuscript, we will include a more comprehensive explanation of the prototype generation process and provide illustrative examples to clarify this methodology further.

Re Weakness #2:

The high efficiency of DistDiff stems from its direct guidance on the pre-trained model without the need to retrain the diffusion models. We provide computational time in Lines 334-339 and resource usage in Line 617. Compared to the stable diffusion baseline model, we only introduce an extra 0.48 seconds of inference time per sample, which is only 4% of the original inference time.

We included an efficiency analysis in Table 1. Our method incurs minimal additional time costs and generates samples directly, unlike LECF, which needs extra post-processing to filter low-confidence samples. This makes LECF slower in generating the same number of samples. We are incorporating this cost analysis into the paper.

Table 1: Inference Efficiency comparison with existing methods on Caltech-101 dataset. * denots that the actual time required of LECF to derive one sample after filter post-processing. Evaluation processes are conducted on a single GeForce RTX 3090 GPU.

Re Weakness #3 and Question #1:

As shown in Table 4, both scores significantly improve performance, and further improvements can be achieved by combining them. However, we did not provide a clear analysis of their relationship, and the claim that these two scores are "indispensable mutually" lacks sufficient evidence and may not be appropriate. This is primarily due to the small class variance in the Caltech-101 dataset, where the differences between $p_c$ and $p_g$ are minimal. In contrast, when we applied hierarchical prototypes to a dataset with larger class variance, such as StanfordCars, the differences between hierarchical prototypes were more pronounced. This led to a more significant combined improvement, as shown in Table 2.

Additionally, we selected $K=3$ as the optimal prototype since it shows the best quantitative performance, although changes in the $K$ value are not apparent in the Table 6 of the manuscript. The influence of $K$ is more pronounced for fine-grained StanfordCars dataset, highlighting the importance of $p_g$, as shown in Table 2.

Table 2: Prototypes comparison of accuracy in expanding StanfordCars 2$\times$. We trained ResNet50 with a 448 × 448 resolution for 128 epochs.

Re Question #2:

Thank you for the valuable suggestion. We will revise the captions to provide a more detailed explanation of the entire pipeline process and ensure that the terms used in the captions align clearly with those labeled in the figure. This will help improve comprehension and the overall effectiveness of the figure.

Re Question #3:

We appreciate your insightful recommendation. We will revise Figure 1 to include performance comparisons and highlight significant differences in the generated results. This will enhance the figure's utility and informative value, making the complex process of DistDiff clearer and more impactful.

[1] Hierarchical clustering. Introduction to HPC with MPI for Data Science 2016.

Dear Reviewer 4Bhc,

We appreciate your thoughtful evaluation and the opportunity to clarify and expand upon key aspects of our work. Based on the detailed responses and additional data provided, which directly address the concerns raised:

1. We have provided a detailed explanation of the generation process for our hierarchical prototypes.
2. We have included a comprehensive analysis of time complexity compared to existing methods, substantiating our claims of high efficiency.
3. We have supported the effectiveness of our hierarchical prototypes with additional ablation results on downstream tasks, confirming the efficacy of these prototypes.

We hope that our responses have clarified and addressed your questions satisfactorily. We will carefully revise the manuscript in accordance with the suggestions from all reviewers. If our explanations have resolved your concerns, we would be grateful if you could reconsider your rating. We are eager to make any further improvements necessary to meet the conference's standards.

Given the comprehensive nature of our response, we kindly request that you review our rebuttal and provide your feedback. If you have any further questions regarding our responses, we would be happy to provide additional clarification.

Thank you for your time and consideration.

We would like to express our sincere gratitude for the detailed and professional attention you have given to our work during the review process. We greatly appreciate your recognition of the very important problem our work addresses, as well as your acknowledgment of our detailed ablation studies and the considerable improvements made.

Below are our detailed responses to the weaknesses and questions:

Re: Weakness #1
In Section 3.3, we introduce our method starting with a sample $x$. However, in the algorithm presented in the appendix, we begin with the latent point $z_T$, which omits the image encoding process from $x$ to $z_T$ for simplicity. Thank you for pointing this out. We recognize that this may cause confusion and will revise it in the final version.

Re: Weakness #2
Our method effectively generates classification data and has the potential to be extended to detection and segmentation tasks. This extension requires incorporating more advanced foundational models, such as ControlNet [1], which can use layout maps or segmentation masks as conditions to control the spatial positioning of targets. An intuitive approach to extend our method might involve cropping features from relevant target regions, averaging them to obtain feature vectors, and then constructing prototypes to guide the generation process.

Re: Weakness #3
The residual multiplicative transformation applies channel-level transformation to the original latent point. Compared to directly optimizing the latent point, this transformation constrains the optimization space, preventing out-of-control guidance and making the optimization process easier.

Re: Question #1
We apologize for missing this detail. We follow previous data expansion methods by expanding each sample certain ratios. The sampling category distribution is consistent with the original dataset category distribution. We will add an description in the final version.

[1] Adding Conditional Control to Text-to-Image Diffusion Models, ICCV2023.

Dear Reviewer ghiL,

Thank you again for your valuable comments. We have tried our best to address your questions (see rebuttal above), and will carefully revise the manuscript by following suggestions from all reviewers. Please kindly let us know if you have any follow-up questions.

Your insights are crucial for enhancing the quality of our paper, and we would greatly appreciate your response to the issues we have discussed.

Thank you for your time and consideration.