AutoGenDA: Automated Generative Data Augmentation for Imbalanced Classifications

[W1] Fine-tuning. Thanks for your comment for training the generator with long-tailed data. We are afraid that the reviewer may miss some parts of our work. We would like to clarify that the generator of our method is finetuned using the textual inversion technique on the long-tailed datasets (in P.6, line 315).

[W2/W3] More baselines and datasets. Thanks for your suggestion. Under the limited rebuttal period, we apologize for not being able to include all datasets and baselines suggested by every reviewer. We add the baseline Feedback-Guided Data Synthesis (FGDS) suggested by Reviewer AgGB as it is a recent work that also focuses on imbalanced classification, which is more related to our work. We also follow the suggestion by Reviewer AgGB to add the CIFAR-10-LT and CIFAR-100-LT experiments. The new results are illustrated in Table 5 of Appendix E. Our method slightly leads the other baselines on the two new datasets.

[W1/W3/Q1] Evaluation of vanilla SD1.4. Thank you for the suggestion. It is true that SD1.4 is pre-trained on a large dataset. However, given a dataset containing less common objects or unseen objects in the pre-training stage, the vanilla SD1.4 may fail to generate good images. Comparison between vanilla Stable Diffusion versus fine-tuned Stable Diffusion model using textual inversion had been studied in the DAFusion work. Under the limited time for the rebuttal period, we did not replicate the comparison on our dataset. However, for the completeness of our paper, we would add this comparison in our final version.

[W2/W8] Computation overhead. We add Appendix F to compare the computation overhead of the baseline methods. The computation effort is mainly dominated by the image-generating process. The search effort for AutoGenDA only occupies a small portion of the total computation time. On the CIFAR-10-LT dataset, our method takes 9.6 GPU-hour on a single NVIDIA RTX3090 GPU card. The details can be found in Table 6 of Appendix F.

[W4] Class-specific vs. class-agnostic. This is a good question. For local caption images, the caption may contain both class-specific and class-agnostic parts; however, as the captions are extracted in the same class as the input image, the caption should be appropriate. For transfer caption images, the caption can contain class-specific parts unsuitable for the transfer. This highlights the importance of our automated search. In the search phase, the search gradually learns to sample fewer images if the transfer captions lead to unnatural images that harm the classification performance. In addition, we also introduce the filtering mechanism to use captions from more similar classes to mitigate the chance of generating unnatural images.

[W5/W7] More baselines and datasets. Thanks for your suggestion. Under the limited rebuttal period, we apologize for not being able to include all datasets and baselines suggested by every reviewer. We add the baseline Feedback-Guided Data Synthesis (FGDS) suggested by Reviewer AgGB as it is a recent work that also focuses on imbalanced classification, which is more related to our work. We also follow the suggestion by Reviewer AgGB to add the CIFAR-10-LT and CIFAR-100-LT experiments. The new results are illustrated in Table 5 of Appendix E. Our method slightly leads the other baselines on the two new datasets.

[W6] More illustrations. We add more examples of the generated images in Fig. 11 and Fig. 12 of Appendix H.

[Q1] Ablation study. In the ablation study, the results are the average of the 16 imbalanced settings of the four tested datasets. We observe that the reporting of the average may not clearly highlight the edge of our method for more imbalanced scenarios. Therefore, we report all the experimental results in Table 3 of Appendix D. Following the suggestion from Reviewer AgGB, we also conduct a statistical test in Table 4 of Appendix D to validate the effectiveness of our method.

[W1/W4] More baselines and datasets. Thanks for your suggestion. Under the limited rebuttal period, we apologize for not being able to include all the suggested datasets and baselines. We add the baseline Feedback-Guided Data Synthesis (FGDS) as suggested, as it is a recent work that also focuses on imbalanced classification, which is more related to our work. We also follow your suggestion to add the CIFAR-10-LT and CIFAR-100-LT experiments. The new results are illustrated in Table 5 of Appendix E. Our method slightly leads the other baselines on the two new datasets.

[W2/W3] Novelty and relevant work. We would like to highlight the difference between our work and existing work from two perspectives. (1) Unlike most existing works that study the generation of more diverse and realistic examples on balanced datasets, we focus more on the more challenging imbalanced settings, where some tail classes do not provide sufficient samples to generate diverse augmented images. (2) One large part of our method is an automated framework that automatically learns the effective ratio of different types of augmented samples. This framework can be deployed to any generative method, and we believe such automation is novel in generative augmentation research. We sincerely hope the reviewer can reconsider the contribution of our work by taking into account the two perspectives.

[W4/Q2/Q3] Ablation study. In the original ablation study, the results are the average of the 16 imbalanced settings of the four tested datasets. We observe that the reporting of the average may not clearly highlight the edge of our method for more imbalanced scenarios. Therefore, we report all the experimental results in Table 3 of Appendix D. Following your suggestion, we also conduct a statistical test in Table 4 of Appendix D to validate the effectiveness of our method.

[Q1] Background robustness. We agree that improving background robustness is one of the major benefits of using AutoGenDA. In addition, we also observe other changes from the generated images, such as the pose and shape of the objects. This can be observed in the newly added augmented samples in Appendix F. This suggests that the variations generated by our method are not limited to background changes.

[W1] Class-filtering. This is a good question. For local caption images, the captions are extracted in the same class as the input image; the caption should be appropriate. For transfer-caption images, the caption can contain class-specific parts unsuitable for the transfer. This highlights the importance of our automated search. In the search phase, the search gradually learns to sample fewer images if the transfer captions lead to unnatural images that harm the classification performance. Instead of adjusting the parameter m for each dataset, we tend to fix the parameter and rely on the search algorithm to find the compelling augmented images to be included.

[W2/Q3] Trend. We added the analysis on the trend of the learned sampling parameters in Appendix G and Fig. 10. In Fig. 10, we show the averaged augmentation probabilities over all classes for the PASCAL VOC, Caltech, COCO, and LVIS datasets with 2, 4, 8, and 16 shots of data. The learned probabilities for identity images show an increasing trend, while the learned probabilities for local-caption and transfer-caption images show a decreasing trend. This can be explained by the fact that with less data, the classifier easily learns variance within the dataset and demands more varied data to improve generalization performance. Conversely, with more data, the classifier can learn diverse variance information from the real data, thereby assigning a larger probability to sample an identity image.

[W3/Q2] Use of LLM. This is a good suggestion. In AutoGenDA, we use the captions contained within the datasets to create augmented samples. We believe the captions contain variations that are specific to the target dataset, thus leading to more customized augmented samples for the dataset. For common objects, we agree that Introducing LLM can create more diverse captions for generating good augmented samples.

[Q1] Ablation study. In the ablation study, the results are the average of the 16 imbalanced settings of the four tested datasets. We observe that the reporting of the average may not clearly highlight the edge of our method for more imbalanced scenarios. Therefore, we report all the experimental results in Table 3 of Appendix D. Following the suggestion from Reviewer AgGB, we also conduct a statistical test in Table 4 of Appendix D to validate the effectiveness of our method.

[Q4] Overhead. We add Appendix F to compare the computation overhead of the baseline methods. The computation effort is mainly dominated by the image-generating process. The search effort for AutoGenDA only occupies a small portion of the total computation time. On the CIFAR-10-LT dataset, our method takes 9.6 GPU-hour on a single NVIDIA RTX3090 GPU card. The details can be found in Table 6 of Appendix F.