Learning Label Shift Correction for Test-Agnostic Long-Tailed Recognition

We are grateful for the valuable reviews, and for the positive comments such as the method is simple yet effective. We will address the reviewer's concerns below.

Concern #1: How does the performance change as encountering more general label distribution shift?

Response: In this paper, we delve into the test-agnostic long-tailed recognition problem, aligning with previous research efforts such as SADE [NeurIPS 2021], LADE [CVPR 2021], and BalPoe [CVPR 2023]. To the best of our knowledge, no existing long-tail learning method can generalize to arbitrary test label distributions—a limitation inherent in the no-free-lunch theorem. To address this challenge, a pivotal issue is how to effectively simulate diverse label distributions, a facet that has not been extensively explored in the field. By developing the capability to simulate any label distribution, we believe our proposed method can encompass various simulations into the training data, thereby learning to estimate arbitrary label distributions.

Concern #2: The effectiveness of the training of neural estimator.

Response: We show that the training of our proposed neural estimator can be done efficiently. We first elaborate on the simulation of training data for the neural estimator. Given a class prior, we determine the corresponding number of samples for each class for simulation. Subsequently, we swiftly sample a subset of training samples for every class by pre-dividing the samples in the entire training set into their respective class pools. In general, the time complexity of the re-sampling process is approximately $O(QN)$, where $Q$ is the number of times of re-sampling and $N$ is the number of training samples. By default, we set $Q=2\times10^3$ in all experiments and the re-sampling process can be done in seconds for training data of size $N=10^6$ on modern computers. We also monitor the time consumption of the entire training process of the neural estimator, it takes 12.78 seconds and 355.15 seconds on CIFAR100-LT and ImageNet-LT, respectively. To demonstrate the effectiveness of the simulation of different label distributions, we attach the Python code for reference.

def multiple_subset_resampling(imbalance_ratio_list, img_idx_each_cls):
  """sample multiple subsets from a list of class priors"""
  subset_list = []
  for imbalance_ratio in imbalance_ratio_list:
    selected_idx_list = subset_resampling(imbalance_ratio, img_idx_each_cls)
    subset_list.append(selected_idx_list)
  return subset_list

def subset_resampling(imbalance_ratio, img_idx_each_cls):
  """sample a subset from a given class prior"""
  img_num_per_cls = produce_num_per_cls(imbalance_ratio)
  selected_idx_list = []
  for the_class, the_img_num in enumerate(img_num_per_cls):
    idx = img_idx_each_cls[the_class]
    selected_idx = torch.multinomial(torch.ones(idx.size(0)), the_img_num, replacement=True)
  	selected_idx_list.append(selected_idx)      
  return selected_idx_list

Concern #3: In Eq. (5), what does index $j$ mean?

Response: We are sorry for the confusion and correct Eq. (5) as follows:

$k = \arg \max _ {k \in \mathcal{K}} \mathbb{I}(\pi _ 0^{h} > \lambda \pi _ 0^{t}) \widehat{Z}^{h} + \mathbb{I}(\pi _ 0^{h} < \lambda \pi _ 0^{t}) \widehat{Z}^{t} \quad s.t. \quad \widehat{Z}=logitClip({Z}, k)$

Concern #4: Why the optimal $k$ maximizing Eq. (5) results in a higher value for the backward case and lower for the forward case?

Response: The rationale behind Eq. (5) is twofold.

Firstly, we observe that most long-tail learning methods generate logits that exhibit a bias towards tail classes. Specifically, the model may produce large (positive) logits for tail classes even when the ground truth corresponds to a head class. Conversely, the produced logits for head classes can be small (negative) values, excluding the ground-truth class. Consequently, without a higher value of $k$, it becomes easy to observe the backward label distribution, as tail classes dominate head classes in the logits matrix. However, we require a lower value of $k$ to clip negative logits of head classes, enabling the identification of a forward label distribution. This observation is visually depicted in Figure 2(b) of the paper.

Secondly, we utilize Eq. (5) to determine the value of $k$ because the predicted probabilities (i.e., $\pi _ 0$) can provide a rough sketch of the true label distribution. This observation is illustrated in Figure 2(a) of the paper. Combining these observations, Eq. (5) is capable of automatically searching for a suitable value of $k$.

Concern #5: Can the proposed method, which only experienced the imbalance ratio of range [1/100,100], correctly estimate the test label distribution?

Response: This is a very intriguing question. To answer this question, we test our method on the CIFAR100-LT dataset by varying the test label distribution from Forward100 to Forward200 and Backward100 to Backward200. We compare the performance of our method LSC with the base model. In general, the results show that our method consistently improves the base model across all settings. Notably, the performance of LSC remains stable even as the test label distribution becomes more imbalanced.

Concern #6: Does the number of sampling subsets of the training dataset affect the final performance?

Response: Thank you for this interesting question. We explore the impact of the number of sampled subsets on the ImageNet-LT dataset. We conduct simulations with subsets ranging from $\\{400, 1000, 2000, 4000\\}$ and analyze the resulting final performance. Generally, we observe an improvement in test accuracy with an increase in the number of simulated subsets. This is attributed to the aim of ensuring that the training set for $g _ \theta$ covers diverse label distributions. However, we only sample 2000 subsets for a balance between effectiveness and efficiency.

We appreciate the valuable reviews and the positive comments highlighting the importance of the question and the clarity of the motivation. Below, we will address the concerns raised by the reviewer.

Concern #1: The connection between LSC and GBBSE?

Response: Sorry for any confusion. This paper presents a general label-shift estimation framework called GBBSE. We then propose a practical implementation under the framework of GBBSE and the implemented method is called LSC. We will improve the writing in the next version soon.

Concern #2: Is $g _ \theta$ equal to NerualEstimator?

Response: Yes, we refer to $g _ \theta$ as the NerualEstimator in our paper.

Concern #3: Since the training dataset is imbalanced, how can you sample a subset of the training dataset when its smallest class should be the biggest class in the subset?

Response: Given a class prior, we utilize either undersampling or oversampling for simulation. In cases where additional training data is required for the smallest class, we implement resampling of the training data with replacement. To enhance diversity, we incorporate data augmentation strategies, such as RandAugment or AutoAugment, which generate distinct copies of the same image. To elucidate the sampling process, we provide the attached Python code for reference:

def multiple_subset_resampling(imbalance_ratio_list, img_idx_each_cls):
  """sample multiple subsets from a list of class priors"""
  subset_list = []
  for imbalance_ratio in imbalance_ratio_list:
    selected_idx_list = subset_resampling(imbalance_ratio, img_idx_each_cls)
    subset_list.append(selected_idx_list)
  return subset_list

def subset_resampling(imbalance_ratio, img_idx_each_cls):
  """sample a subset from a given class prior"""
  img_num_per_cls = produce_num_per_cls(imbalance_ratio)
  selected_idx_list = []
  for the_class, the_img_num in enumerate(img_num_per_cls):
    idx = img_idx_each_cls[the_class]
    selected_idx = torch.multinomial(torch.ones(idx.size(0)), the_img_num, replacement=True)
  	selected_idx_list.append(selected_idx)      
  return selected_idx_list

Concern #4: Can you explain which step achieves the alignment?

Response: As illustrated in the pseudo-code, the training of $g _ \theta$ is conducted on simulated subsets with various label distributions, rather than the global label distribution. The objective is to minimize the discrepancy between the output of $g _ \theta$ and the ground-truth label distribution of our simulated subsets. Ultimately, the well-trained $g _ \theta$ is able to generalize to the test data and estimate the test label distribution. We will modify the pseudo-code to make the notations more clear.

Concern #5: Can you theoretically analyze why this method can align the predicted and the true test label distribution?

Response: Theorem 3.2 in our paper ensures that we can acquire a sufficiently trained base model, i.e., $\epsilon _ {L}(g^{\star})$ is small. Assuming that the error gap $\epsilon _ {L}(g _ {\theta}) - \epsilon _ {L}(g^{\star})$ can be bounded through the training of $g _ {\theta}$ using generated multiple subsets (ensured by the Bayes-risk consistency of the training), we can approximate the test label distribution precisely to adjust the model, especially when the test sample size is sufficiently large. More precisely, assuming that the multiple subsets are obtained by i.i.d sampling, we can bound the estimation error through classical generalization theories.

However, in the context of long-tail learning, generating multiple subsets for training $g _ {\theta}$ involves oversampling for tail classes, disrupting the i.i.d properties of sampling. This departure from i.i.d conditions complicates the application of traditional machine learning generalization analysis. To the best of our knowledge, analyzing the generalization bound in this scenario without additional assumptions proves to be extremely challenging.

Concern #6: What is the effect of $\lambda$ on the final performance?

Response: We examine the impact of $\lambda$ on the final performance by selecting values from the set $\\{1, 1.5, 2, 3\\}$. The experiments are conducted on the ImageNet-LT dataset, and the results are presented in the table. Overall, we observe comparable performance across different values of $\lambda$ in the majority of cases.

Concern #2: the choice of BBSE seems unjustified.

Response: In this paper, we present GBBSE -- a more general test distribution estimation framework built upon BBSE. We improve the theoretical results of BBSE in the context of long-tail scenarios. Furthermore, in our implementation, we enhance BBSE by substituting the confusion matrix estimation with the training of a neural estimator. While acknowledging that BBSE may not be the optimal choice for label-shift estimation, we opt for it due to its simplicity and robust theoretical properties, such as an appealing convergence rate of the size of the validation set $O(\frac{\log N}{N})$. It is worth noting that alternative approaches like MLLS also necessitate an additional validation set, a resource typically unavailable in long-tail learning scenarios. We plan to discuss this with MLLS and BCTS in the next version.

Concern #3: GBBSE requires re-sampling multiple subsets, which might take non-trivial time to train.

Response: We demonstrate that the re-resampling module can be implemented efficiently through the attached Python code. Specifically, given each class prior, we first determine the number of samples for each class for simulation. Subsequently, we sample a subset of training samples for each class by pre-dividing the samples in the entire training set into their respective class pools. In general, the time complexity of the re-sampling process is approximately $O(QN)$, where $Q$ is the number of times of re-sampling and $N$ is the number of training samples. By default, we set $Q=2\times10^3$ in all experiments and the re-sampling process can be done in seconds for training data of size $N=10^6$ on modern computers. We also monitor the time consumption of the entire process of GBBSE, it takes 12.78 seconds and 355.15 seconds on CIFAR100-LT and ImageNet-LT, respectively.

def multiple_subset_resampling(imbalance_ratio_list, img_idx_each_cls):
  """sample multiple subsets from a list of class pirors"""
  subset_list = []
  for imbalance_ratio in imbalance_ratio_list:
    selected_idx_list = subset_resampling(imbalance_ratio, img_idx_each_cls)
    subset_list.append(selected_idx_list)
  return subset_list

def subset_resampling(imbalance_ratio, img_idx_each_cls):
  """sample a subset from a given class prior"""
  img_num_per_cls = produce_num_per_cls(imbalance_ratio)
  selected_idx_list = []
  for the_class, the_img_num in enumerate(img_num_per_cls):
    idx = img_idx_each_cls[the_class]
    selected_idx = torch.multinomial(torch.ones(idx.size(0)), the_img_num, replacement=True)
  	selected_idx_list.append(selected_idx)      
  return selected_idx_list

Concern #4: You employ logit clipping to address overconfident logits for tail classes. Is that related to the miscalibration of neural networks?

Response: To validate the reviewer's assumption, we report both the Expected Calibration Error (ECE) and Accuracy of LSC and its counterpart without using logit clipping. We implement LSC on two distinct base models, namely, NCL and RIDE. The experiments are conducted on the CIFAR100-LT dataset. Overall, the results do not indicate a strong correlation between miscalibration and logit clipping. Specifically, NCL exhibits good calibration with a low ECE. However, NCL+LSC without logit clipping shows poor performance on both "Forward" and "Uniform" test distributions. This is attributed to the fact that the logits generated by NCL for head classes are still dominated by tail classes. In contrast, NCL+LSC with logit clipping significantly improves NCL performance in both "Forward" and "Backward" cases. Unlike NCL, RIDE is a model with poor calibration, characterized by a high ECE. Remarkably, LSC with logit clipping is still effective in enhancing its performance across all cases. These results highlight the efficacy of our proposed method for various base models, whether they exhibit good or poor calibration.

We express gratitude to the reviewer for their insightful reviews and encouraging remarks, including strong empirical performance and ample ablation study. Below, we address the concerns raised by the reviewer.

Concern #1: suboptimal performance on "Backward" test distributions compared to "Forward".

Response: Indeed, nearly every long-tail learning method experiences a decline in performance when transitioning from a "Forward" to a "Backward" test distribution. In our setup, where the training data adheres to a "Forward" long-tail distribution, the model demonstrates robust generalization when the test distribution aligns with the "Forward" pattern. Conversely, the model's performance may falter when faced with a "Backward" test distribution, as it lacks exposure to ample samples from tail classes during training. While some methods can be employed to address the challenges of "Backward" performance, their effectiveness remains suboptimal. In response to this issue, we introduce LSC in this paper, specifically designed to address label distribution shifts, and it surpasses previous methods by a significant margin. Acknowledging the no-free-lunch theorem, we recognize the impossibility of achieving perfect compensation for data scarcity and generalizing seamlessly to "Backward" as well as "Forward".

From a theoretical standpoint, the implications of Theorem 3.1 in our paper align with this issue. For the sake of simplicity, we assume the true posterior distribution $P(Y \mid X)$ is one-hot, and we focus on the first term in Theorem 3.1 as follows:

Assume the label shift estimation is perfect, i.e., $\widehat{P} _ {T}(Y)=P _ {T}(Y)$, then we have:

During the training of the base model, we minimize the empirical cross-entropy loss on the training set to address the aforementioned term. However, there exists an inherent gap between empirical error and generalization error, which is linked to the number of samples. In prior studies, including [1, 2], this gap is typically bounded by $O(\frac{1}{\sqrt{N}})$, where we have:

Hence, $\mathbb{E} _ {x \sim \mathcal{D} _ {S}(\cdot \mid Y=k)} \left[\left(1 - \widehat{P}(Y=k\mid x)\right) \right]$ increases as the number of $k$-th class training samples decreases. Under such observation, we have:

Given that we have $N _ {1} \geq N _ {2}\geq \dots \geq N _ {K}$, it is evident that the error gradually increases as the test data distribution leans toward the tail class. This phenomenon is solely linked to the long-tailed distribution of the training set, assuming perfect test label distribution prediction has been achieved during the derivation process. For a more in-depth exploration of this phenomenon, refer to [1, 2]. Numerous long-tail learning methods, such as [3, 4, 5], have been introduced to address this issue. Notably, LSC can seamlessly complement existing long-tail learning methods.

[1]. A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning. https://arxiv.org/pdf/2310.04752.pdf

[2]. Sharp error bounds for imbalanced classification: how many examples in the minority class? https://arxiv.org/abs/2310.14826.pdf

[3]. Long-tailed recognition by routing diverse distribution-aware experts. https://arxiv.org/abs/2010.01809.pdf

[4]. Parametric contrastive learning. https://arxiv.org/abs/2107.12028.pdf

[5]. Nested collaborative learning for long-tailed visual recognition. https://arxiv.org/abs/2203.15359.pdf

Thanks for the clarification, analysis, and experiments. Most of my concerns have been addressed, so I have increased my rating to 6. I look forward to your future work.