Rethinking Classifier Re-Training in Long-Tailed Recognition: Label Over-Smooth Can Balance

Q1: It is recommended that the authors include comparisons with the latest SOTA methods and discuss related works.

R1: Thank you very much for recognizing the value of our work! We also appreciate the suggested references that help us further improve our paper. [2][3] represent classical works in this field and the newer work [1] certainly deserves inclusion. Let us first discuss our relationship with [2][3], and then incorporate the SOTA results from [1] into our performance comparisons. We will add these works to the paper in the revison.

Adaptive Logit Adjustment Loss (ALA Loss)[2] innovatively applies an adaptive adjusting term to the logit. The adaptive adjusting term comprises two complementary factors: 1) a quantity factor that emphasizes tail classes, and 2) a difficulty factor that adaptively focuses on hard instances during training. This approach integrates data priors and Hard Data Mining concepts. In contrast, our method maintains purity and simplicity. We focus more on optimizing the overall class distribution without relying on artificial priors, demonstrating superior versatility and effectiveness. Moreover, our approach is orthogonal to the Difficulty Factor design mentioned in [2], suggesting potential for integration.

[3] begins from the perspective of feature space, aiming to expand the coverage of minority classes within this space. [3] improves feature calibration through logits. Their optimization involves shifting the predicted logits values $z$ based on class nums. From a motivation standpoint, we propose a more effective metric directly derived from the logits, around which we develop our work by optimizing this specific metric. In terms of optimization techniques, unlike the shift of $z$ during training in [3], our method actually implements label over-smooth, which introduces new predictive targets to replace the traditional one-hot ground truth vectors.

Our method and analysis focus on the properties and disparities in the training process, particularly the potential convergence behaviors induced by head and tail classes. Indeed, a key advantage of our method is its independence from class-specific hyperparameters to mitigate long-tail issues rather than the parameters which is handcrafted in the final equations of [2][3].

This table demonstrates the comparison between our method and ProCo[1]. ProCo[1] employs contrastive learning and requires AutoAugment, utilizing substantially higher computational costs and data augmentation compared to our approach. We position it as an Upperbound Model, similar to those in Tab.3 of our main paper.

Consequently, ProCo[1] can learn better feature representations, giving it an advantage on challenging datasets like ImageNet-LT. However, its classifier performance limitations become evident in CIFAR100-LT. Without employing any additional tricks, our method significantly outperforms [1]. Furthermore, leveraging ProCo's excellent feature representations, we can integrate our method as a plugin to adjust its classifier, achieving notable performance improvements across different tasks. This demonstrates the complementary nature of our approach and its ability to enhance even sophisticated methods through better classifier optimization.

[1] Probabilistic Contrastive Learning for Long-Tailed Visual Recognition. TPAMI 2024.

[2] Adaptive Logit Adjustment Loss for Long-Tailed Visual Recognition. AAAI 2022.

[3] Long-Tailed Visual Recognition via Gaussian Clouded Logit Adjustment. CVPR 2022.

Q1: In deep learning networks, the objective is always a non-convex optimization problem, how the conclusion is still valid in this case in Proposition 1?

R1: In proposition 1, we assumed that $W$ is given and determined and $b$ is the only variable. With such assumption, the optimization problem is convex with respect to $b$ as we proved. This shows that we don't need to pay too much attention to $b$.

Q2: The Proposition 1 is used to exclude the effect of b and to introduce Proposition 2 which illustrates that the weight norm-based method may not be reliable in some cases, am I right? But how the equation (4) is obtained? from s' to s.

R2: Right, Proposition 2 illustrates that the weight norm-based method may not be reliable and compare the weight across classes is unreliable. That is the reason that we proposed LoMa. In Proposition 2, We just construct a new converage point $(W', b^*)$. In such situation, the calculated new $s'$ is not changed from original $s$. By substituting into equation (2), we can get equation (4).

Q3:From line 313-314, how the assumption is determined" Consider random perturbations ∆i that independently affect each class zi and are proportional to the standard deviation of their logits, i.e., z′"? It is said "These perturbations ∆ are commonly introduced during the training process, particularly due to factors such as overfitting in the initial stage of training and bias in the sampling of the training set.", can you clarify more on this point?

R3:

Consider the basic derivative

They both converage as $\mathbb{E}[s_i] = \mathbb{E}[y_i]$,

In each step,

$\mathbb{E}_w[s_i]$ is widely present during the gradient descent process, so its inconsistent magnitude will lead to differences in convergence speeds in different directions and an overall directional bias relative to balanced datasets.

Due to the randomness of the sampling itself, or the bias of $W$ after first stage of training, even the SGD optimizer will introduce such perturbations.

We will add more explanation in the revision to help understanding.

Q4: If the point lies in $\Delta$ and $\sigma$ in section 3.1, what is the point to introduce definition 2? The point in definition 2 shoule be $R_i$, right?

R4: With definition 2, we observe $R_i$ is almost invariant between these methods as $\sigma$ is not. Based on the invariance of $\sigma$ and the balance of LoMa $L$, we can analyze the trend of $\sigma$ in following.

Q5: Figure 1 is also not easy to understand for the first time until figure 2 is given.

R5: Thank you for pointing out this issue. We will add more explanation in figure 1 in the revision to help understanding.

Q6: Typos Line 107.

R6: Thank you for pointing out this issue. We mistakenly used \cite when it should have been \citep. We have made this correction in the revised PDF version.

Q1: For Q2, will it satisfy naturally? Any proof?

R1: For Proposition 1, the proof is given in the Appendix B. Equation 4 is part of the proof for Propostion 2, the new converage points $(W', b^*)$ are constructed to illustrate the existence. Such converage points naturally exist. You may notice the definetion of $z$ in Section 2.1, from line 146 to 153, Equation 4 is obtained from the given definition. I may not understand what "it" refers to, I hope this solves your doubts.

Q2: For Q3, what do you mean "$R_i$ is almost invariant between these methods as $\sigma$ is not"? $\sigma$ is not invariant?

R2: Since $R_i$ is defined as

$R_i=\frac{\sigma_i}{L_i},$

where $L_i$ is logits magnitude. $R_i$ is invariant and $L_i$ is variant across methods as shown in Figure 3. So $\sigma_i=L_iR_i$ is not invariant as well.

Q3: It is suggested to provide the line number about the changes in the paper. It would be easier to track what and where have been revised?

R3: In the uploaded revision PDF, all modifications have been highlighted in blue. Below are the specific line numbers and corresponding changes:

1. Line80-87 & Figure1: Replace "Gaussian distribution diagram drawn with mean and variance of samples" with a "bar chart showing specific logits value for each data samples". Update the caption to reflect these changes.

2. Line168: Generalize Vanilla Cosine to a more general form, add temperature parameter $\tau$, conducte new experiments and update best results for this method.

3. Line325 & Appendix C: Add proof for the approximation in Equation 9.

4. Line215 & Line307: Modify expressions for improved readability.

5. Line440-445 (Table3): Add new experimental results combining the latest methods with our approach.

6. Line457-Line459, Line766, Line773-776: Add analysis of latest related work.

7. Line540-Line715 (Reference): Standardize paper citation format, update journal/conference names and abbreviations and replace arXiv versions with published versions.

Note: Figure sizes have been adjusted to maintain the 10-page limit without affecting crucial content.

I think the authors have addressed my concerns. I can raise the score to 6.

One last thing, please pay attention to R1 "Equation 4 is part of proof in for Propostion 2", the typo.

Thank you very much for your time and constructive feedback, which helped us effectively improve this paper. We truly appreciate your recognition and positive evaluation of our work.

P.S. We have modified the typo as suggested - thank you for pointing that out.

Q1: The proposed LoMa metric is hard to understand due to ambiguous language and notation.

R1: Each Loma $L_i \in R$ is uniquely associated with a single class $c_i$. As shown in Figure 2, the Loma $L_i$ is defined as the difference between the mean logits of the samples belonging to the current class $c_i$ and and those of the samples not belonging to $c_i$.

Q2: The suggested label smoothing approach doesn't directly fit into the story of achieving uniform LoMa.

R2:

1. We conducted an analysis of previous methods and concluded the following: A balanced LoMa typically leads to better performance. The Regularized Standard Deviation (RSD) is invariant, meaning it does not significantly vary across different methods(Section 2.2). Moreover, it remains balanced on datasets that are themselves balanced. 2）Building on these observations, we analyzed the differences in the gradient descent process between balanced datasets and long-tailed datasets (Section 3.1).
2. As a non-convex optimization problem, gradient descent inevitably deviates from the true optimal point and direction. On balanced datasets, these deviations can be considered consistent across classes (Eq. 2). However, on long-tailed datasets, noticeable inter-class inconsistencies emerge.
3. To address this issue, we propose a method that not only maintains a well-balanced LoMa but also eliminates inter-class inconsistencies.

Q3: Proposition 2 is meaningless. Section 3.1 on the deep dive into LoMa seems like a rambling section without concrete takeaways.

R3: Proposition 2 demonstrates that the convergence point is not unique, and the norm $||w_i||$ can have arbitrary length at convergence. Many methods attempt to constrain the length of $||w||$. However, based on Proposition 2，directly comparing $||w_i||^2_2$ across classes is meaningless. To address this, we propose LoMa，which enables unified comparison across classes. It is worth noting that LoMa is also related to $w$. For instance, simply doubling $w_i$ would result in the corresponding LoMa $L_i$ being doubled.

Q4: In Eq. 9, the expectation can't be decomposed like that, it looks artificial.

R4: In Eq. 9, we assume a balanced dataset scenario, meaning that $\Delta_i$ and $\Delta_j$ are identically distributed.

Analyzing the Approximation:

For convenience, let $p_j = \Delta_i-\Delta_j$， $q_j=z_j-z_i$,

$

\mathbb{E}\left[\frac{1}{\sum_j e^{p_j} e^{q_j}}\right]&= \mathbb{E}\left[\frac{1}{\mathbb{E}[e^{p_j}]\sum_j e^{q_j} + \sum_j (e^{p_j} - \mathbb{E}[e^{p_j}])e^{q_j}}\right]

$

Assuming that the fluctuations $e^{p_j} - \mathbb{E}[e^{p_j}]$ are small compared to $\mathbb{E}[e^{p_j}]$, we can use the first-order approximation of the reciprocal:

$

\frac{1}{A+\delta} &= \frac{1}{A} - \frac{\delta}{A^2} + \frac{\delta^2}{A^3} + \dots \\ &\approx \frac{1}{A} - \frac{\delta}{A^2}

$

$

\mathbb{E}\left[\frac{1}{\sum_j e^{p_j} e^{q_j}}\right]&\approx \mathbb{E}\left[\frac{1}{\mathbb{E}[e^{p_j}]\sum_j e^{q_j}}\right] - \mathbb{E}\left[\frac{\sum_j (e^{p_j} - \mathbb{E}[e^{p_j}])e^{q_j}}{(\mathbb{E}[e^{p_j}]\sum_je^{q_j})^2}\right] \\

$

Since $\mathbb{E}[e^{p_j} - \mathbb{E}[e^{p_j}]]=0$,

$

\mathbb{E}\left[\frac{1}{\sum_j e^{p_j} e^{q_j}}\right]&\approx \mathbb{E}\left[\frac{1}{\mathbb{E}[e^{p_j}]\sum_j e^{q_j}}\right]

$

Q5: The proposed (over) label smoothing technique is just a minor variant of prior work.

R5: Although our proposed method is a variant of prior work, its usage is completely different. We not only proposed the concept of over-label smoothing but also theoretically analyzed its impact on the model learning process. Label smoothing has not demonstrated a dominant position in the long-tailed (LT) field, and no one has attempted over-smoothing in other fields. The difference between 0.99 and 0.2 has already far exceeded mere formal consistency. Our method is simple but effective. It can be extremely effective in the LT domain, which constitutes a novelty.

I hope this message finds you well. I am following up on my previous response to your review comments on our manuscript. We deeply value your feedback and aim to ensure we address all your concerns thoroughly.

If there are any further questions or clarifications you would like us to address, we are more than happy to provide additional details.

Q4: The references are outdated. No papers from 2024, only 3 papers from 2023, and 4 papers from 2022.

R4: The main reason is that our method focuses on fundamental long-tailed distribution optimization without incorporating more complex training paradigms or introducing complicated data augmentation. Therefore, the methods that can be fairly compared are relatively classic.

Recent works are mostly combinations of complex and effective paradigms, such as those based on mixture of experts [1], contrastive learning [2][6], data augmentation [1][3][6], and pre-trained model tuning [4][5]. While incorporating these methods can certainly improve model performance, we focus more on fundamental components rather than using an all-encompassing approach for combination. Additionally, Q3 demonstrates that our method can further improve the performance of these methods, thereby enhancing their biased classifiers.

Of course, in the revised version, we will actively include these methods and more in the Related Work analysis, and add the methods from Q3 to the experimental comparisons. We welcome your suggestions for references and relevant literature that should be included in our article revisions.

Q5: Pay attention to the format of the references.

R5: Thank you for your valuable feedback. During the initial submission, we relied solely on the BibTeX entries provided by Google Scholar without further refinement. Following your suggestions, we have now standardized all citation formats and updated the references from arXiv preprints to their corresponding published versions in journals and conference proceedings where applicable. The revised PDF with these modifications has been uploaded for your review.

[1] Nested Collaborative Learning for Long-Tailed Visual Recognition. CVPR22

[2] Long-Tailed Recognition by Mutual Information Maximization between Latent Features and Ground-Truth Labels. ICML23

[3] Kill Two Birds with One Stone: Rethinking Data Augmentation for Deep Long-tailed Learning. ICLR24

[4] Parameter-Efficient Complementary Expert Learning for Long-Tailed Visual Recognition. ACM MM24

[5] Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts. ICML24

[6] Probabilistic Contrastive Learning for Long-Tailed Visual Recognition. TPAMI24

Q1: In Fig1, what do the red and blue lines mean? Why do they obey Gaussian distributions?

R1: The red line shows the distribution of samples that truly belong to the current class, while the blue line shows the distribution of samples that don’t.

It cannot be proven that they follow a Gaussian distribution, but their actual distribution is very similar to a Gaussian distribution. Here, as a qualitative demonstration, we directly used the mean and variance to plot a Gaussian distribution.

Q2: Why not directly use the Cosine classifier? Should include the Cosine classifier as a baseline.

R2: Thank you for your sincere suggestion! In fact, we have already included the Cosine classifier as a baseline for comparison in Tab.1 of the main paper, and analyzed its characteristics in the subsequent Fig.3. Here we rewrite the experimental results.

As shown in the table, our method significantly outperforms the Cosine method. Indeed, as you mentioned, feature magnitude and logit magnitude are not the same. We focus more on the relationships between computed Logits, directly optimizing a more straightforward objective, rather than pre-normalizing Weights and Features.

Q3: Comparison to recent works RIDE, BCL, NCL, GML.

R3: In the main paper Tab.3, we compared with these Upperbound models, which typically require larger models (e.g. Mixture of Experts) and more computational resources (e.g. Contrastive Learning). This includes models like RIDE, BCL, PaCo, and SSD.

Following your suggestion, we have supplemented the complete results for NCL and GML, and presented the unified results below.

These methods often have better feature representations. Taking the newly added NCL as an example, it integrates multiple feature learning methods and requires RandAugment as training data augmentation, making its training complexity and unfairness far exceed our method. Our method requires neither complex data augmentation nor contrastive learning.

However, these methods do not perform well in classifier learning. On the smaller CIFAR100 dataset, which is easier to fit, our method significantly outperforms these approaches. When dealing with larger datasets, where feature space plays a more crucial role, we use our method as a Plugin to adjust the Classifier based on these method backbone, achieving performance improvements and thus proving our method's effectiveness. Additionally, as shown in Tab.4 of the main paper, our method achieved the best improvement even with poorer feature representations, demonstrating its universality.

First of all, thank you for your quick response and so many constructive suggestions. With your help, we have been continuously improving the paper. Here are the specific responses to these issues.

Q1: Directly plotting a Gaussian logit distribution is not rigorous and a bit misleading, because you have not provided solid evidence that it follows a Gaussian distribution.

R1: I apologize for the confusion. The simplification to a Gaussian distribution was for intuitive visualization, but we don't need this assumption for the subsequent proofs. Following your suggestion, we have used the actual distribution of the original data and updated it in the paper. Objectively speaking, the current figure is more comprehensive than before and can effectively convey our ideas. Thank you for your continuous suggestions that help us further optimize the paper's readability.

Q2: The formulation of the Cosine classifier in Table 1 seems incorrect. According to your formulation, the logit $g_i$ will fall into a short interval of $[-1, 1]$. In this case, the prediction will be significantly limited. In practice, the Cosine similarity should be divided by a temperature, i.e., $g_i = \frac{W_i^T}{\|W_i\|} \cdot \frac{f(x)}{\|f(x)\|} / \tau$ where $\tau$ is often manually set to such as $0.1$, $0.05$, $0.01$, etc.

R2: This is a great question. Setting the temperature parameter indeed makes this method more flexible and adaptable. Based on your suggestion, we conducted urgent experiments, and here are the results - we have already updated the optimal results in the paper.

Q3: The paper limit is 10 pages across all stages (initial submission, rebuttal revisions, camera ready). You'd better update your manuscript to meet the rules.

R3: Thank you for the reminder. We have formatted the article according to the specified requirements.

Thanks for your explanation. I have run your code and find that label over-smoothing can be more surprising than expected. I find that setting $\delta=1$ still yields satisfactory results (just slightly lower than $\delta=0.98$). I think it is very interesting because it indicates that even with a uniform label distribution, label smoothing in stage 2 can still re-balance the biased predictions. But I cannot realize the underlying reasons. I guess it is because the model from stage 1 already has an initial ability. Indeed, I think your analyses in Section 3.1 have not touched the fundamental reason. Nonetheless, I believe this paper provides an interesting insight that can contribute to the community and inspire more future work. I will raise my score to 6.

Moreover, previous work [1] has highlighted the benefits of label smoothing in the second stage, but it is ignored and not discussed in this paper. Besides, is there any other paper that applies label smoothing for long-tailed learning?

PS: I find a mistake in your code. In main_stage1.py, you save a deep copy of the best model (Lines 58, 74). However, when it comes to main_stage2.py, you load the model using its state_dict (Lines 89-90). There is also a similar mistake in evaluate.py. If directly run, it will cause an error. It seems that we should use the code model = torch.load(pth) annotated in line 92.

[1] Improving calibration for long-tailed recognition.

In Fig. 4, while the x-axis extends to $\delta=1$, the sample points in the figure do not reach this value, with the rightmost point corresponding to $\delta=0.99$. It is also explained in Lines 375-377 of the paper: "From this perspective, $\delta$ approaching 1 can yield the best results. However, our method essentially acts as an auxiliary regularization technique to mitigate biases among different class samples and assist the model in converging. The model needs to exhibit both convergence and discriminability, meaning that the ground truth labels should remain distinct from the other labels."

1. In Appendix E, we conducted parameter sensitivity experiments on CIFAR100, which demonstrated that our method is relatively stable across different parameter settings. This stability allows us to confidently apply the best-performing parameter combination from CIFAR100 to other experimental settings.

2. Indeed, your understanding is correct. We have also provided the implementation details in our GitHub.

3. Regarding this question about the reweighting/rebalancing effect, while we have provided extensive justification in the paper and previous rebuttal responses, I can offer additional intuitive perspectives:

• As demonstrated in Figure 1, our method does not significantly alter the intra-class discriminative ability but rather reduces the majority class bias. The ultimate goal in addressing long-tailed recognition is to achieve equal treatment across all classes, which doesn't necessarily need to be accomplished through sample count manipulation.
• In the probability prediction space, Label Over-Smooth effectively reduces the disparity in total prediction probabilities across different classes, thereby achieving the concept of equal treatment.

These are preliminary insights, and we believe this phenomenon warrants various interpretations and further investigation.

Dear Han,

Thanks for your reply.

(1) I noticed that the backbone you used for cifar100-LT is ResNet34, which provides a baseline of around 47% accuracy using your code. This is much higher than the accuracy when using ResNet32, around 38%. Most of the methods in Table 2 use ResNet32 for CIFAR100-LT, so did Alshammari et al., 2022. For a fair comparison with the existing methods, ResNet32 should be used unless you run all the methods using ResNet34.

(2) When I use ResNet32 and use the common setup following the literature (Cao et al., 2019), the result I obtained when using the label smoothing is not as good as the case of not using it. I also could not reproduce your results with imagenet-LT. What do you think is the possible reason? Could you share your code for imagenet-LT and iNaturalist2018?

(3) In the two-stage training from the literature, the representation part of the model is typically fixed when retraining the classifier. However, in your work, when using the label smoothing for retraining the model, you don't fix any part of the pretrained model. In the case of CIFAR100, the label smoothing 0.98 will change the labels from 1 to 0.0298 or 0 to 0.0098. Would this strong regularisation destroy the pretrained representation model? Have you tried retraining the model for, e.g., 200 epochs instead of 20?