DiffuLT: Diffusion for Long-tail Recognition Without External Knowledge

W1

The innovation is limited. Essentially, the authors' method designs a reconstruction loss to constrain the generative model to learn a distribution more consistent with real distributions, and many existing studies have proposed similar ideas.

The focus of our approach is not on training a generative model to learn a distribution that closely matches real-world data. Instead, we emphasize the relationship between generative models and discriminative models, addressing two questions that are yet to be fully explored: Can a generative model without external knowledge improve a classification model? And what type of generated data is most beneficial?

Previous research has primarily relied on pretrained diffusion models, which we believe limits practical applicability and may lead to data leakage. We would appreciate it if you could provide examples of existing studies with similar concepts. To the best of our knowledge, no such prior art exists.

W2

The authors emphasize that this method has generalizability in real-world applications; however, I hold a negative view on this point. I believe that the performance improvement observed in this paper is fundamentally due to training data and test data having identical distributions. Therefore, constraining the diffusion model to learn a distribution closer to training data benefits classifier learning. However, in real-world applications where limited training data may not share identical distributions with potentially infinite test data, using this method might lead to negative impacts.

As mentioned in previous publicity in CVPR [1], the long-tailed setting assumes that the distribution of the label $p(y)$ differs between the training and test sets, but the conditional distribution $p(x|y)$ remains unchanged. These default settings hold true in current long-tailed recognition settings, and also can serve as the potential theoretical foundation for many long-tailed literatures [2][3]. The scenario you mentioned, where the training and test samples do not share the same distribution, might be different from the definition of long-tailed learning.

To address concerns about the potential negative impact of distribution shifts, we conducted a firmly experiments: we trained a ResNet-50 backbone on both the ImageNet-LT dataset and ImageNet-LT augmented with data $\mathcal{D}_{gen}$ generated by our DiffuLT method. We then performed linear probing (the transfer learning settings) on the CIFAR100-LT with an imbalance ratio of 0.01.

Despite the distribution differences between ImageNet-LT and CIFAR100-LT, our method still showed improvement. Therefore, we believe our method does not negatively impact model performance when distributions change.

Regarding real-world applications, our paper demonstrates that our methods can benefit medical imaging, remote sensing, and general image classification under long-tailed settings. We recognize that the understanding of "real-world application" may vary, and we are open to further discussion if you could specify your interpretation of this term.

W3

On what principle was Equation 4 determined? Why define [df, 2df] as AID? What impact would modifying this equation? For instance, would defining [df, \sqrt(2)df] as AID affect performance?

The definition of AID samples in the paper can be seen as `somewhat arbitrary'. However, we prioritized efficiency, performance, and simplicity in arriving at this definition. An ablation study shows that this approach achieves the best results with the highest efficiency.

We opt not to use the first definition because the range should be as wide as possible to ensure efficient generation, while avoiding the production of too many unused samples. Although a more precise search might yield better results, we prioritize simplicity and avoid complex parameter tuning, leading us to set the region as defined.

W4

In Table 6, why does combining ID and AID result in worse performance than using only AID for CBDM?

Actually, this is one of the key ideas of our paper. In this setting, the number of generated samples is consistent, although the generation frequency varies due to the different distributions of data types. Our key observation, as shown in Table 3 of paper, is that AID samples contribute more to classification performance compared to ID samples. Since the "ID & AID" setting includes fewer AID samples than the "AID" setting, it results in a worse outcome.

W5

Authors should provide ablation studies for Nt.

We conducted an ablation experiment for $N_t$ under the CIFAR100-LT setting with an imbalance ratio of 0.01.

The performance shows only a slight improvement after $N_t > 500$ and begins to decrease after $N_t > 600$. We believe the main reason for this is that the class with the most samples has only 500 instances, so choosing $N_t > 500$ inevitably introduces noise. When $N_t$ is too large, the impact of this noise outweighs the benefits of the newly added information.

[1] Disentangling Label Distribution for Long-tailed Visual Recognition. CVPR 2021.

[2] Long-tail learning via logit adjustment. ICLR 2021

[3] Balanced Meta-Softmax for Long-Tailed Visual Recognition. NeurIPS 2020

Dear Reviewer,

We hope that our rebuttal has effectively clarified any confusion and that the additional experiments we provided have strengthened the validity of our approach. We eagerly await your feedback on whether our response has adequately resolved your concerns or if further clarification is needed.

Dear Reviewer,

Thank you for your feedback and patience. We would like to address your concerns regarding the novelty and methodological design of our work.

Novelty

The novelty of our approach lies in two main areas: primarily in the long-tailed learning domain and, to a lesser extent, in the generative modeling domain.

Long-Tailed Learning:

• Diffusion Model without External Knowledge: Our key contribution is the novel application of diffusion-based models to long-tailed learning without relying on external knowledge. While recent works like [1] and [2] utilize Stable Diffusion to address data scarcity in classification tasks, they often face criticism due to the extensive pretraining on external large datasets (much larger than those for classification tasks), which can overshadow the novelty of their approach. We specifically address the question of whether a generative model, trained only on the limited data available in classification tasks, can still be effective. Our work demonstrates that this is indeed possible for the first time, providing a significant insight, especially when the training data is not composed of natural images (as discussed in our response to W3 of reviewer #vaue). This contribution may open new research avenues within the long-tailed learning community, offering a fresh perspective on the integration of generative models for classification tasks.

• Understanding Diffusion Models: We also provide an explanation of why diffusion models, even without external knowledge, are effective in this context. Our work is the first to introduce the concept of AID samples and the role of generation model in blending information of each class to generate beneficial and novel samples.

Generative Modeling:

• A New Objective for Generative Models: Traditionally, models like VQ-GAN are trained with objectives such as reconstruction loss and perceptual loss to generate realistic images. Our work introduces a novel objective: optimizing the generative model to produce images that are beneficial for classification tasks. This represents a significant shift from the traditional goal of generating visually appealing images to a more practical application, where the generated images directly enhance discriminative tasks. This objective is inherently challenging to optimize, and our introduction of AID samples provides a novel pathway for achieving this goal.

Methodological Design

To address your concerns, we highlight the key designs of our method:

New Designs:

• Classification of Generated Images: We classify generated images based on their distribution in the feature space relative to real data.
• Novel Loss Function: We introduce a new loss function that uses classifiers to guide the training of the generative model. This simultaneously addresses the long-tailed generation problem and enhances the utility of the generative model for classification tasks.
• Diffusion Model without External Knowledge: As previously mentioned, our method employs a diffusion model trained without external knowledge to generate images that aid long-tailed learning.
• Weighted Synthetic Samples: We apply weights to synthetic samples during backpropagation to improve training efficacy.

Adapted Designs:

• Sample Filtering: We use a trained classifier to filter out harmful samples, similar to existing methods but without relying on models like CLIP.

We hope this addresses your concerns. Please feel free to reach out if further discussion or additional experiments are needed. We look forward to your feedback. Thank you again!

[1] Expanding Small-Scale Datasets with Guided Imagination. NeurIPS 2023.

[2] Feedback-guided Data Synthesis for Imbalanced Classification. NeurIPS Workshop 2023.

W1

Some typos: line 137 ‘am input’

Thank you for your feedback; we'll fix it in the next version.

W2

The baseline setup in Table 1,2,3 requires further explanation.

We are sorry for the confusion. Actually, the baseline setup adheres to the code and protocols from [1], with a 200-epoch training schedule. Training is conducted with a batch size of 128 using an SGD optimizer at a learning rate of 0.1. These detailed settings can also be found in our appendix.

W3

The paper mentions the time cost is four times longer than baseline, indicating that this method may be impractical for datasets with a significantly large imbalance ratio due to the extended generation and retraining times.

Thanks for your review. We argue that since DiffuLT doesn’t rely on external knowledge like pretrained models or additional data, it is highly practical in scenarios where data is limited and that acquiring more training data is costly or impossible. In such cases, training time is not the primary concern, while acquiring satisfactory accuracy is.

To make our arguments valid, we showcase two quantitative examples: medical imaging using the KVASIR dataset and remote sensing with the EuroSAT dataset. We train the classifier on long-tail versions of these datasets with an imbalance ratio of 0.01, where models typically struggle due to the skewed distributions.

As shown in the table, our method significantly mitigates the long-tail issue in these important scenarios, showing our great generalization ability for practical usage. We would provide more such discussions in the final version.

W4

Some baseline results in Tables 8 and 9 are indicated as ‘-’ and ‘ ’; an explanation for these forms of absence is missing.

Thanks for your careful review. Since the comparison results are based on the reported values in their respective papers, any results not shown in the original papers are indicated as “-”.

W5 & W6

This method focuses on generating data for tail classes, resulting in significant improvements for these classes compared to other baseline methods. A fairer comparison would be with the same pipeline using different generative models.

A long-tailed diffusion model baseline is missing.

Following your suggestions, we compare our method with different generation models using the same pipeline in Table 5 of the paper. While T2H, as you mentioned, shows slightly better performance than CBDM, it falls short of our proposed model because it isn't primarily designed for discriminative tasks.

Additionally, using pretrained diffusion models like Stable Diffusion could lead to data leakage, resulting in an unfair comparison, and they are not well-suited for data beyond natural images. We will add these comparisons in our final version.

W7

The experiment are mainly conducted on manually-selected long-tailed datasets, it would improve the robustness of results by including real-world long-tailed datasets such as iNaturalist.

You have raised an interesting point. We have tried our best to train a generation model with iNaturalist dataset, but we found that this dataset is too large that it consumes all of our GPU computing power. We are running DiffuLT pipeline with a manually sampled subset of iNaturalist, but due to time constraint, experiments are not finished at present. We promise to showcase analysis and experiment results in our final version.

In the meanwhile, since iNaturalist is similar to ImageNet-LT, both being natural image datasets, we believe the effectiveness of our model on this type of data is validated. Additionally, the results on KVASIR and EuroSAT, which are real-world long-tailed datasets that have entirely different distributions, further demonstrate the robustness of our model.

[1] Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. CVPR 2020.

[2] KVASIR: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection. MMsys 2017.

[3] Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2019.

Dear Reviewer,

We hope that our rebuttal has effectively clarified any confusion and that the additional experiments we provided have strengthened the validity of our approach. We eagerly await your feedback on whether our response has adequately resolved your concerns or if further clarification is needed.

W1

It would be better if you show how “centralized” figure 2 is, for example, add support data of the proportion of ID, AID and OOD data. This figure is not intuitive enough.

Thank you for your feedback. We agree that the centralization of data is not immediately apparent in Figure 2. It becomes clear through the statistics presented in Table 2. We believe that adding annotations to indicate each data type's region in the figure would help readers understand it more effectively.

W2

Citation suggestions. I believe your work would benefit from referencing some additional literature to provide a more comprehensive context for your study.

Thank you. We will include the first and last papers in our comparison results in the experiment, and we will incorporate the middle two works into our related works section in the future version.

W3

Minor typos. Line 137 “as an input”. Line 227 “because it doesn’t rely on any external dataset or model”.

Thanks for your patient review. We’ll address those issues accordingly.

Dear Reviewer,

Thank you for your prompt response and patience. We greatly appreciate your valuable advice and your support for our paper.

Due to an oversight, we did not provide a satisfactory answer to your question. Recognizing its importance, we would like to address it comprehensively here.

Q1

Could you explain how your method differs from other data augmentation methods?

We conducted a thorough survey of data augmentation methods for long-tail learning, categorizing them into two groups: feature augmentation and image augmentation.

Feature augmentation methods, such as RSF[1] and OFA[2], primarily focus on biasing the decision boundary towards tail classes. These methods are similar to re-weighting or re-sampling techniques and do not inherently increase data diversity. Moreover, because features are indirect and often difficult to map back to images, these methods differ fundamentally from image augmentation techniques.

Image augmentation methods share our motivation of increasing dataset richness. We specifically examined M2m[3], MiSLAS[4], Remix[5], CMO[6], and CSA[7], discussing each as follows:

• M2m: This method augments tail classes by transforming head-class samples into tail-class ones through perturbation-based optimization. However, the generated samples often closely resemble the originals (the head images), failing to truly enrich the diversity of tail classes.
• MiSLAS and Remix: These methods use Mixup for augmentation. However, MiSLAS experiments indicate that applying Mixup during classifier learning does not yield significant improvements and may even harm performance, requiring additional strategies to mitigate this issue.
• CMO and CSA: Both methods introduce rich contexts from majority samples into minority samples. CMO uses CutMix, while CSA employs a more precise segmentation approach. While effective in enriching context, they do not enhance the diversity of objects within tail classes.

Data richness, particularly in tail classes, is crucial for addressing the long-tail problem. While recent methods have acknowledged this and used data augmentation to tackle it, their effectiveness is limited to context enrichment. In contrast, diffusion models in our pipeline represent the most powerful data augmentation method. Unlike mixup-based approaches, diffusion models avoid domain shift by accurately estimating data distributions. Additionally, they enrich both context and objects, offering greater diversity since they combine the entire dataset rather than just two images like CMO or CSA. Therefore, with proper utilization, diffusion models hold the highest potential. Below is a performance comparison of various data augmentation methods, including ours, on CIFAR100-LT with an imbalance ratio of 100, using ResNet-32 as the backbone.

[1] RSG: A Simple but Effective Module for Learning Imbalanced Datasets. CVPR 2021.

[2] Feature Space Augmentation for Long-Tailed Data. ECCV 2020.

[3] M2m: Imbalanced Classification via Major-to-minor Translation. CVPR 2020.

[4] Improving Calibration for Long-Tailed Recognition. CVPR 2021.

[5] Remix: Rebalanced Mixup. ECCV workshop 2020.

[6] The Majority Can Help the Minority: Context-rich Minority Oversampling for Long-tailed Classification. CVPR 2022.

[7] How Re-sampling Helps for Long-Tail Learning? NeurIPS 2023.