Mining latent labels for imbalance classification: a regrouping perspective

We would like to heartfully thank the reviewer for their considerate comments and critiques of our work. In the following, we have addressed some of the questions raised in the review. Please feel free to follow up if any aspects are unclear, or if there are further questions.

• More datasets: We acknowledge the limitation that we only have one multi-class dataset, and have addressed this issue by extending our experiment to include a new multi-class task. Specifically, we take the financial phasebank dataset (Malo et al. (2014)), which consists of 4840 sentences of English-language financial news categorized by sentiment with a label frequency of 2,879 (neutral), 1,363 (positive), and 604 (negative). We have shown that in the new dataset, RG surpasses other baselines and remains dominant. For more information about the experiment setups and results, we would kindly refer to Appendix C for details.

• Choice of dataset for visualization experiments: We appreciate the reviewer’s comment on how binary HAM10000 can form a more realistic dataset for visualization experiments. The choice of the CIFAR-10 dataset for the visualization experiment is based on three arguments:

  • Popularity: CIFAR-10 is one of the most popular datasets in image classification, and the class labels are easily interpretable by the readers, and hence a natural choice for a motivating example
  • Relevance to the regrouping idea: The fact that the majority class is diverse makes it an ideal situation for regrouping. Hence, it helps in highlighting the failure of naively using clustering algorithms to separate samples of different classes, for example in raw image space and autoencoder latent representation space.
  • Difficulty: As seen from the results in Section 4, the classification of HAM10000 images is a much harder task in comparison to CIFAR10 image classification, and hence presents a more clear motivation for the readers.

  While a dataset like CATSvsDOGS could potentially be a more realistic setting for imbalance classification, the lack of fine-grained labels (for example: breeds of the animals) makes it difficult to visualize the effect of regrouping. Additionally, the discrepancy between how DNNs interpret label structures and human understanding further complicates the analysis.

• Presentation: We would like to thank the reviewer for the comments on the presentation, we have addressed them as follows:

  • Aggregation: We performed an ablation study to show the impact aggregation method in Appendix B, where max aggregation is found to be marginally better in comparison to sum aggregation, but since the difference is not too large it is recommended to try both methods. We have added some explanation for both aggregation methods in Sec 2.2 to avoid confusion.
  • Clustering Method: In Appendix B, we also explored the role of clustering algorithms in the performance of the proposed method, especially focussing on whether enforcing equal cluster sizes can be beneficial to the method. Two balanced clustering algorithms: constrained K-means (Bradley et al. (2000)) and Normalized Cut (Shi et al. (2000)) have been compared with the naive K-means method. We find that balanced clustering can lead to slightly better BA, but perform much worse in terms of AP metric, and hence the choice of a simple K-means algorithm is proposed. To avoid confusion, we have also added brief information about the clustering algorithm in Sec 2.2.
  • Image Caption: We have fixed the typo in the image caption, it should have been airplane instead of apple.

• Why WCE is required with RG?: WCE is the weighted version of cross entropy loss where each sample is weighted by the inverse of the corresponding class frequency, so that classes with fewer samples get higher weights, check Elkan (2001) for the general concept of reweighting. Since we are using K-means algorithm to perform clustering and assign pseudo labels, the resulting classes are not guaranteed to be balanced. Hence, along with CE, we have also reported results that use WCE loss function.

• AUPRC/AP metric clarification: AUPRC is a metric that calculates the area under precision-recall curve, where each point is based on a specific threshold (decision rule). Hence, AUPRC (similar to AUROC) is independent of the choice of decision rule. Average Precision (AP) is known to be an unbiased estimate of AUPRC (Boyd et al. (2013)). We have added the reference in the paper as well.

We would like to heartfully thank the reviewer for their considerate comments and critiques of our work. In the following, we have addressed some of the questions raised in the review. Please feel free to follow up if any aspects are unclear, or if there are further questions.

• Novelty: We agree with the reviewer that the proposed method shares a high-level overview with Classification using local clustering (COG), proposed by Wu, et. al. (2010) to tackle imbalanced classification. However, we would like to highlight some key arguments in the paper that differ from COG:

  • Motivation: COG was proposed in the context of linear SVM models in order to: a) decompose the majority class samples so that they become linearly separable from minority class samples, and b) produce sub-classes with relatively uniform sizes. Note that a) is irrelevant in the context of deep neural network (DNN) models, and b) is not supported (in Wu, et. al. (2010)) with any theoretical/empirical evidence. In contrast, our motivation for this method stems from the advantages demonstrated through the use of multiclass labels, for the one-vs-rest CIFAR-10 classification problem, over binary labels.
  • Insights: Wu, et. al. (2010) justify their method by showing how linear decision boundaries between pseudo-classes found by COG can separate linearly inseparable classes. However, for DNN classifiers it is known that even random labels can fit perfectly on a training set (Zhang et al. (2021)), and hence the insight from Wu et al. (2010) is not valid for DNNs. In comparison, in this paper, we show the ability of the proposed method to learn meaningful representations (Figure 3 bottom pane), synchronize training across different labels (Figure 5), and achieve better performance (Figure 6). These insights shed light on the training dynamics in binary vs multiclass setting for the first time to the best of our knowledge, which could also be of independent interest.
  • Implementation: In COG, pseudo labels are assigned by clustering the data in data space, we propose to cluster in the image encoding space. The choice of image encoder is central to the merits of the proposed method – it can be seen in Figure 3 (top pane) that naively clustering the data in raw image space, or even the representation space of autoencoder, will lead to very noisy pseudo labels and would result in poor final performance. We argue that this implementation detail of the proposed method is non-trivial for a practitioner implementing COG method for DNNs.

  To conclude, we argue that this work provides a fresh perspective for exploring and expanding on the method proposed by Wu et al. (2010) as a powerful baseline method to tackle imbalance.

• Clustering Method: We would like to thank the reviewer for pointing out the lack of clarity about the clustering algorithm in the method description, we have updated the paper to make it more clear. Essentially, the proposed method is agnostic to the choice of clustering algorithm, but the conventional K-means algorithm seems to perform well empirically. Note that the proposed method depends heavily on the choice of image encoder (which has been explored in detail in Section 2.1), but less so on the choice of clustering algorithm used in the image encoder space. A natural question could be if enforcing the clustering algorithm to generate relatively balanced clusters could be helpful, which is explored in the ablation study in Appendix B. Two balanced clustering algorithms: constrained K-means(Bradley et al. (2000)) and Normalized Cut (Shi et al. (2000)) have been compared with the naive K-means method. We find that balanced clustering can lead to slightly better BA, but perform much worse in terms of AP metric, and hence the choice of a simple K-means algorithm is proposed.

• Comparison for long-tailed learning dataset: Extension of the proposed method to long-tailed learning setting has been left open for future work. This requires careful handling of the tail classes, and addressing the explosion in numbers of pseudo-classes due to the extremely small sample size in tail classes. We have briefly discussed the merits of exploring classical imbalance classification methods in the last paragraph of Section 3: numerous practical applications still fall under classical imbalance setting, and classical methods often form a base for long-tailed learning methods. The competing methods have also been chosen accordingly. To show that our proposed method can work well in a classical imbalance classification setting, we added an extended experiment using the financial phasebank dataset (Malo et al. (2014)). The details of the experiment setup and results can be found in the revised appendix. For a quick summary, we observed consistent improvements of RG over other baselines, both in terms of BA and AP.

Thanks for the response. After reading the rebuttal, I appreciate the authors' effort and encourage the authors to further refine their works. The rating is remained given the current unsolved concerns.

Best,

The reviewer.

We would like to heartfully thank the reviewer for their considerate comments and critiques of our work. In the following, we have addressed some of the questions raised in the review. Please feel free to follow up if any aspects are unclear, or if there are further questions.

• Error Bars: We have rerun all the experiments with 3 random seeds and report the mean with standard deviation, please see the updated results in the revised manuscript.
• Gaussian Processes: In this paper, we mainly focus on classification based on deep learning models, especially in the image domain. We could find only two works that address Gaussian Process-based classification for image data (Blomqvist et al. (2020), Van der Wilk et al. (2017)), but their public code repositories have not been maintained and we are unable to implement these methods from scratch due to limited time of the discussion period. However, the method proposed by Wu et al. (2010) is similar to the proposed method for tabular data where results on linear SVM models are provided, which may be of interest to the reviewer.
• Meaningful subgroups: A potential way to check whether the data can be partitioned into sub-groups is to make use of a visualization tool such as t-SNE plot (For example: As used in Figure 3 top pane). When data cannot be put into meaningful subgroups, it might point to the choice of encoder model (i.e. CLIP model in our case), and other encoder models can be tried.
• Limitations: The key limitation of the proposed method is to handle long-tailed datasets, where the tail classes lie in the few-shot learning setting (i.e. tail classes have few training samples). This leads to an explosion in the number of pseudo-classes, and hence extension to this setting is non-trivial and left open for future work. Another limitation is that the number of pseudo-classes (K) in the current format is a tuning hyper-parameter, a future work could try to deduce this in a more methodological manner to save excess computation for tuning the hyper-parameter.

References:

• Blomqvist, K., Kaski, S., & Heinonen, M. (2020). Deep convolutional Gaussian processes. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16–20, 2019, Proceedings, Part II (pp. 582-597). Springer International Publishing.
• Van der Wilk, M., Rasmussen, C. E., & Hensman, J. (2017). Convolutional gaussian processes. Advances in Neural Information Processing Systems, 30.
• Wu, J., Xiong, H., & Chen, J. (2010). COG: local decomposition for rare class analysis. Data Mining and Knowledge Discovery, 20, 191-220.