Feedback-guided Data Synthesis for Imbalanced Classification

Thank you for your feedback on our paper. We note a mismatch between your positive evaluation and your recommendation. Below, we address all your concerns and believe that upon reviewing our responses, you might consider revising your score.

The font format of the article is not uniform. Do the words in italics want to express any special meaning? Make it difficult for readers to read.

We have followed ICLR 2024's official style guidelines, including the use of italics for introducing either new concepts (e.g., “usefulness”) or emphasizing key points (e.g. “close to the support of the real data distribution”). If any instances create readability issues please let us know, we will modify them. However, these formatting concerns could be easily rectified and do not reflect the quality of the research presented.

The charts are mixed up, for example, Figure 5. Is it a table or a graph? The sizes of some pictures also don’t match.

We have now made the size of the tables consistent across the paper and modified Figure 5 (now Table 1) to include only the table and moved the figure to appendix.

How about the time complexity of this method?

Thank you for raising this question. We have now included an additional Section (G1) in the Appendix which compares the time-complexity of feedback guidance versus regular sampling.

Are there more evaluation metrics to evaluate the performance of the proposed method versus the baseline method?

We used standard metrics used to report results on ImageNet-LT and NICO++ [1, 2] such as average accuracy, many-med-few accuracy and worst group accuracy. In addition to the benchmark’s metrics, in our submission we include standard generative model metrics to evaluate the effect of our criteria on the quality of the generated images such as FID, coverage and density – see Table 2. Thus, we argue that our evaluation metrics are in-line with both representation learning and generative model literatures. If the reviewer has suggestions on which metrics they would like to see included in the paper, please let us know and we would be happy to include them.

—---------------

[1] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Largescale long-tailed recognition in an open world. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[2] Xingxuan Zhang, Yue He, Renzhe Xu, Han Yu, Zheyan Shen, and Peng Cui. Nico++: Towards better benchmarking for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16036–16047, 2023.

We would like to thank the reviewer for their insightful review. We are glad to see that the reviewer finds our framework novel. We would like to highlight important differences between Active Learning (AL) and our method. The goal of AL is to interactively query a user (or a model) to label new data points from a given static dataset of unlabelled data points, thus the dataset is known but the labels are unknown. However, in our setup the labels are given and the dataset is unknown. The dataset has to be obtained by sampling from a parametric generative model – our framework describes how to class-conditionally sample from a generative model to obtain useful data points to train a classifier. We agree with the reviewer that our criteria bear some similarities with AL acquisition functions. However, we argue that our main contribution is the design of a queryable framework for generative models that “smartly” balances the training data for a classification model, and not the criteria itself. Note that we do not claim the criteria to be our contribution – see contributions list at the end of the introduction. Moreover, in our experiments we show that the criteria we use are effective enough to obtain state-of-the-art results on three challenging datasets (ImageNet-LT and NICO++ and Places-LT). Thus, we disagree with the main premise of the reviewer’s critique that our simple and effective criteria weakens our main contributions. Contrarily, we see this as a proof of strength for our framework as it can reach the state-of-the-art with simple and broadly used criteria. Thus, we kindly ask the reviewer to reconsider their initial recommendation. Below, we provide detailed responses to the reviewer's questions.

.. using the recent active learning criteria, such as BALD [1], VAAL [2], or MCDAL [3]

Thank you for suggesting the relevant literature. We have included them in the future work section (6) in our paper. In general, any differentiable criterion which is a function of the classifier can be applied as a feedback guidance in the sampling process, as long as it is computationally efficient. Among the mentioned methods, in theory, both BALD and MCDAL could be applied as a feedback criterion. However, compared to the entropy, they are computationally 10x and 3x more expensive, respectively, limiting their application in practice. The suggested VAAL method is not applicable in our framework since it is task-agnostic and its acquisition function does not depend on the classifier of the task.

...according to other recent Long-tailed recognition papers, they usually evaluate their methods on iNaturalist and Place-LT datasets to demonstrate the scalability. At least the authors should have evaluated their method on CIFAR datasets...”

Thank you for your feedback regarding our experimental approach. Our paper focused on ImageNet-LT, which has a substantial number of classes (1000), and NICO++, featuring 360 groups. To further provide evidence for our methodology's effectiveness on large-scale datasets, we have applied our framework for the Places-LT[1] dataset. This is a highly imbalanced dataset where the smallest class only has 5 examples and the largest class has 4980 examples with 365 classes overall. We upsample this dataset such that every class has 4980 samples resulting in a dataset of size 1.8 million examples. We compare against the Fill-up [2] work which also uses 1.8 million synthetic samples and other baselines that do not use synthetic data. We observe that feedback guidance with entropy achieves SOTA results on this dataset. Below is the summary of our results and see full details in Section G.4 of the paper.

Thank you for your valuable feedback. We are glad that you found our experimental results “comprehensive/stunning" and the writing “easy to follow”. Below we clarify your question and hope that you consider increasing your score.

ImageNet-LT is essentially a pseudo long-tail dataset, where the tail classes may not necessarily be the minority in the actual data distribution. Therefore, generative models can sample relatively well. However, for real-world long-tail distributions, is it also difficult for generative models to obtain sufficiently good samples?

Thanks for raising this point. As discussed in the limitations paragraphs in the conclusion section of our initial submission: “...our guidance mechanism can only explore the data manifold already captured by the generative model” and this limitation is imposed by the generative models.

Yet, we would like to add that state-of-the-art generative models are capable of generating very unlikely data points under the real world distribution of images - e.g. they are able to generate the rare concept of "an astronaut riding a horse" by combining the common concepts of “astronaut” and “horse”. This is mostly what we observed in our NICO++ experiments, where we saw that generative models can indeed generate new combinations of object and background which is useful to improve generalization.

Our findings on Imagenet-LT, NICO++, and Places-LT (recently added - refer to new results in reviewer’s tjGi answer) show that this inherent potential can be harnessed towards achieving tangible improvements in classification tasks.