MaxSup: Overcoming Representation Collapse in Label Smoothing

We are grateful for the reviewer’s thoughtful and constructive feedback. We have carefully considered and incorporated the suggestion into our revision, as detailed below.

W1: The authors do not adequately address/discuss concurrent work [1]. Although it is briefly mentioned in L150, the fact that it performs a similar training loss analysis for the (different) task of selective classification would best be discussed in Sec. 2.2 Studies on Label Smoothing.

We thank the reviewer for pointing this out and are happy to expand our discussion of [1*] in Section 2.2. While [1*] does conduct a related gradient analysis of the training loss, it focuses specifically on the setting of selective classification, and examines a posthoc logit normalization technique to mitigate confidence calibration issues. However, this approach addresses only the overconfidence problem of label smoothing (LS), without tackling representation collapse and improving classification accuracy.

In contrast, our work presents a logit-level reformulation of LS that provides a deeper theoretical understanding of why LS amplifies errors and leads to representation collapse. This reformulation motivates MaxSup, a principled and effective alternative that directly regularizes the classifier during training, improves classification accuracy, and resolves the representation collapse issue, which is critical for feature learning and downstream generalization.

We will integrate this expanded comparison into the related work section in the revised version.

Q1: The authors should also add the underbraces from Eq. 6 to Eq. 2.

We are glad to incorporate the reviewer's suggestion into our revised version.

We sincerely thank the reviewer for the valuable feedback and deeply appreciate the time and effort invested in reviewing our work. Below, we address the mentioned concerns and suggestions in detail.

W1: Clarification of Figure 1.

• We focus the severely reduced intra-class variation caused by label smoothing [13, 26, 37]. This collapse leads to tightly clustered features for each class, which may seem beneficial but is known to harm downstream tasks such as transfer learning. MaxSup explicitly preserves intra-class variation, as can be observed for classes 281, 282, and 292.

• While class 282 appears more scattered under MaxSup, the inter-class separation—particularly between classes 281 and 282—is visibly better (smaller overlapped area) under MaxSup than under ls or ce. This supports the improved classification performance we report and confirms that MaxSup achieves larger inter-class separation while preserving the intra-class variation.

W2: Experiments with high label noise, severe class imbalance, or adversarial examples—settings.

We emphasize that $L_{\text{maxsup}}$ is orthogonal to strategies for handling these challenges, and can be readily combined with them.

For imbalanced data, we have conducted comprehensive experiments on CIFAR10-LT with varying imbalance ratios (50, and 100) following settings in [1*], as shown in Table R1. Across all imbalance ratios and splits (val/test), MaxSup consistently outperforms both the baseline and LS.

For out-of-distribution (OOD) settings, we conducted experiments on CIFAR10-C using ResNet-50 shown in Table R2 following settings in [2*]. These results validate that MaxSup remains effective on OOD datasets, achieving performance comparable to LS.

Table R1: Comparison of overall accuracy (%) (jointly considering many-shot, median-shot, and low-shot top-1 accuracy).

Table R2: Comparison of MaxSup and LS on CIFAR10-C using Resnet50 as backbone. Lower value is better.

[1*] Tang, K., Huang, J., & Zhang, H. Long-tailed classification by keeping the good and removing the bad momentum causal effect. NeurIPS 2020.

[2*] Heinonen, M., Tran, B. H., Kampffmeyer, M., and Filippone, M. "Robust Classification by Coupling Data Mollification with Label Smoothing." AISTATS 2025.

W3: Comparison to related techniques such as self-knowledge distillation, label perturbation, or mixup-based regularizers.

We have trained ResNet-50 with two representative self-knowledge distillation techniques using ResNet-50 on ImageNet, and below are the results:

Notably, DDSGD slows down training by 2× since it always reuses half of the previous batch.

[3*] Yuan, Li, et al. "Revisiting knowledge distillation via label smoothing regularization." CVPR 2020.

[4*] Shen, Yiqing, et al. "Self-distillation from the last mini-batch for consistency regularization." CVPR 2022.

W4: Sensitivity analysis across a wide α range (e.g., α→0 or α→1 extremes).

We appreciate the reviewer’s interest in the sensitivity of α. We note that extreme values (e.g., α → 0 or α → 1) are less informative and rarely used in practice: As α → 0, the method reduces to standard cross-entropy, which we already include as a baseline; As α → 1, the supervision signal vanishes, leading to degraded performance.

For ResNet-50 trained with MaxSup on ImageNet, when α = 0.5, the top-1 accuracy is 76.11%, while at α = 0.9, it drops sharply to 68.38%. Our analysis thus focuses on the practical range of α where MaxSup meaningfully improves performance.

W5: In tasks with extremely large label sets, the computational and memory overhead of max operation.

We appreciate the reviewer's concern on the scalability of MaxSup. However, even in extreme scenarios, the computation required to find the maximum logit is negligible compared to the total cost of a forward pass in modern deep networks.

Let $K$ denote the number of classes. Identifying the maximum logit requires approximately $1 \times 10^6$ FLOPs for $K = 10^6$. This amounts to only: $0.001 \text{ GFLOPs}$. By comparison, a single forward pass of ResNet-50 on a 224×224 input image requires about 4 GFLOPs. Therefore, it adds virtually no measurable runtime overhead.

Q1: How does MaxSup scale when the number of classes K is extremely large?

This is not an issue for MaxSup, please see our response to W5.

Q2: If the training set contains mislabeled examples, does MaxSup inadvertently amplify noise?

Thank you for the thoughtful question. Since MaxSup penalizes overconfidence in the top-1 prediction regardless of the ground-truth label, it does not specifically amplify label noise. In fact, by discouraging overly confident predictions, MaxSup can potentially reduce the harmful impact of mislabeled examples.

Q3: MaxSup with teacher–student distillation frameworks.

Following [21], we evaluated the effectiveness of distillation from ResNet-50 to AlexNet on CIFAR-10. Both teacher and student models are trained for 100 epochs using the Adam optimizer with a fixed learning rate of 0.001. The distillation process is conducted with a fixed temperature of 1.0 and an interpolation factor α of 0.1 across all settings:

Although the student distilled from the teacher using MaxSup (MS) does not outperform the student distilled with cross-entropy (CE), it surpasses the performance of the student distilled with label smoothing (LS). This result aligns with Table 2, which shows that MaxSup effectively mitigates the variation collapse issue.

L1: Effectiveness in NLP, multimodal tasks, or reinforcement learning remains untested.

We acknowledge the reviewer’s point and agree that a full exploration across domains is valuable future work. While our focus is on computer vision due to the well-documented issues of label smoothing in this domain, we provide preliminary evidence that MaxSup also generalizes to natural language processing. Specifically, we train a 12-layer Transformer [1] on the IWSLT 2014 German to English dataset, following the training setup of fairseq repository. The detokenized SacreBLEU scores of 3 runs are compared below. While this enhancement may not appear substantial, it likely stems from the constraints of downstream tasks. Nevertheless, the improvement is statistically significant, as it exceeds the standard deviation.

Table: Comparison of Label Smoothing and MaxSup on IWSLT 2014 German to English Dataset

[1*] Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).

L2: No formal study of training dynamics or convergence rates under MaxSup.

We appreciate the insightful comment. As briefly mentioned in our Limitation section, Guo et al. [9] demonstrate that label smoothing (LS) accelerates convergence through a conditioning number analysis. A similar approach could be applied to analyze the training dynamics of MaxSup. However, this lies beyond the current scope and is left as a promising direction for future work.

L3: Robustness to extreme label noise, long-tail distributions, adversarial attacks, or online learning scenarios is not assessed.

MaxSup applies the intended regularization when the model makes incorrect predictions, i.e., it consistently suppresses the max logit position even when it is not the class label position. This is independent of the assumption of label noise, or long-tail distributions. For the results on imbalanced dataset and dataset with noisy labels, please refer to our response to w2.

Paper Formatting Concerns.

We appreciate the reviewer's suggestion and are glad to improve this aspect in our revised version.

Thank you for the detailed response. However, my primary concern (Question 1) has not been fully addressed. A well-structured feature space is expected to exhibit small intra-class variance and large inter-class distance, which is not clearly demonstrated in Figure 1. I suggest that the authors consider replacing the illustration with a more convincing one, or alternatively, provide additional visualizations in the appendix. I will not change my score.

We sincerely thank the reviewer for the detailed and constructive feedback. In response to the reviewer’s concerns, we address the mentioned concerns below:

W1 – Indeed, the novelty is quite limited as the issues with LS on misclassified samples have been considered extensively in the prior work [1]. The authors should at least empirically compare this work.

1. Conceptual distinctions

We already cite and briefly discuss the distinctions with [1*] ([35] in our paper, lines 149–151), and are happy to expand on this comparison in more detail below.

While [1*] performs a related gradient analysis, it is conducted in the context of selective classification and evaluates a post-hoc logit normalization technique (previously suggested in [2*]) to address overconfidence at inference time. This approach:

• targets confidence calibration, not representation collapse,
• does not improve classification accuracy, and
• relies on additional hyperparameter tuning.

In contrast, our logit-level reformulation of the training objective reveals the root cause of error amplification in LS. This decomposition leads directly to MaxSup, a principled and training-time regularization method that:

• mitigates overconfidence during training,
• improves classification accuracy,
• and resolves representation collapse, a critical issue for transferable feature learning.

2. Empirical comparison (more results)

To address the reviewer's concern, we additionally compared to [3*], which proposed to mitigate overconfidence via logit normalization during training. With the default hyperparameter, it merely achieves an accuracy of 74.322 (lag far behind the 76.91 accuracy of label smoothing). The performance aligns with that of Logit Penalty [4] in our Table 4, which similarly minimizes the global L2-norm of logits. We also note that norm-based methods [1*, 3*, 4] are highly sensitive to hyperparameter choices, which limits their practical applicability. We will integrate this expanded comparison in Appendix H in the revised version.

3. Highlighting our novelty

Our method is not merely incremental; it represents a substantial rethinking of Label Smoothing by:

1. Theoretically uncovering and explicitly characterizing the "error amplification" mechanism within the LS-induced loss.
2. Introducing MaxSup, a simple yet effective modification of penalizing the top-1 logit rather than the ground-truth logit, thus resolving critical limitations of LS during training.
3. Demonstrating consistent, robust improvements across various model architectures (CNNs, Vision Transformers), datasets, and tasks (classification, segmentation, fine-grained recognition), highlighting the generalizability and practical value of MaxSup.

References

[1*] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It. ICLR 2025

[2*] Luís Felipe Prates Cattelan and Danilo Silva. How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks. In UAI, 2024.

[3*] Wei, Hongxin, et al. "Mitigating neural network overconfidence with logit normalization." International conference on machine learning. PMLR, 2022.

W2: Regarding the theoretical implications of penalizing the top-1 logit beyond the impact on error amplification, such as the influence on the geometry of the learned feature space or some simple, controlled experiments on even synthetic data.

We appreciate the reviewer’s thoughtful comment. While we agree that a deeper theoretical analysis of the geometric effects of penalizing the top-1 logit would be valuable, we respectfully note that this direction goes beyond the scope of our current study. Our primary focus is to address the well-established representation collapse issue induced by label smoothing [13, 26, 37]. We demonstrate the effectiveness of MaxSup in mitigating this issue through widely adopted qualitative [21] and quantitative [13] evaluations, which we believe are sufficient for supporting our claims.

Additionally, we would like to emphasize that representation collapse is a phenomenon that mainly arises in complex, real-world data settings. In contrast, prior work such as Neural Collapse [42] has shown that in simplified or synthetic settings—where models achieve near-zero training error—representations often collapse naturally, regardless of the loss function used. Therefore, we believe that synthetic experiments may not meaningfully reflect the challenges our method is designed to address.

W3: The paper isolates MaxSup's contribution by disabling CutMix and Mixup. A discussion or additional experiments on how MaxSup interacts when combined with other common regularization techniques and data augmentations (which are often used in practice) could provide more practical insights.

We appreciate the reviewer’s valuable suggestion. In fact, we have provided an extended logit-level analysis that incorporates CutMix and Mixup, which is documented in the Appendix (lines 641–657). As shown in Equation (39), MaxSup can be naturally extended to these settings by replacing the weighted average of two ground-truth logits with the maximum logit value.

Additionally, we actually performed experiments with CutMix and Mixup that we decided to leave in order to not overload the presentation. Nonetheless, for completeness, we share below the results with CutMix and Mixup enabled on DeiT-Small:

These results demonstrate that MaxSup continues to provide consistent improvements even when used in conjunction with strong data augmentations like CutMix and Mixup.

Q1: 1) Optimal strategies for dynamically adjusting (similar to adaptive LS variants) specifically for MaxSup. 2) Robustness to $\alpha$-scheduler. 3) Comparison to [2] which couples label smoothing and data mollification through schedulers.

We appreciate the suggestion to explore optimal strategies for dynamically adjusting MaxSup parameters, such as via adaptive α-schedulers. As shown in Table 14 (Appendix), incorporating the α-scheduler from [18] yields moderate additional improvements. However, our work primarily aims to address the core limitations of label smoothing—namely, error amplification and representation collapse—rather than optimizing performance through scheduler tuning. We view the design of optimal α-scheduling strategies as orthogonal and consider it a valuable direction for future work.

Importantly, as demonstrated in Table 8, MaxSup consistently outperforms standard label smoothing with or without an α-scheduler. While both methods benefit from scheduling, MaxSup’s gains are robust and not reliant on this tuning.

Regarding [2*], we note that while the paper also modifies label smoothing, it tackles a different problem by coupling LS with data mollification. This coupling leads to a different formulation and focus compared to our method. While a detailed comparison falls outside the scope of our paper, we will cite [2*] in the revised version to acknowledge its relevance in the broader context.

We sincerely thank the reviewer for acknowledging the significance of our contribution and valuable suggestions. We address the mentioned questions in detail, as outlined below:

Q1: The proposed method operates effectively when the CE with Hard Labels decreases sufficiently to recover $z_{gt}$ from $z_{max}$ for incorrect samples. How about an imbalanced dataset or out-of-distribution? In those cases, $L_\text{maxsup}$ would not be effective to minor classes?

Thank you for the insightful question. In our study, we had not explicitly addressed class imbalance or out-of-distribution (OOD). However, we emphasize that $L_{\text{maxsup}}$ is orthogonal to existing strategies for handling these challenges, and can be readily combined with standard technique. For instance, reweighting the total loss by the inverse class frequency can mitigate bias toward majority classes, while entropy-based regularization may enhance robustness under OOD conditions where predictions are often overconfident.

To specifically address the concern on imbalanced data, we have conducted comprehensive experiments on CIFAR10-LT with varying imbalance ratios (50, and 100) following settings in [1*], as shown in Table R1. The evaluation compares three setups: Focal Loss, Focal Loss + Label Smoothing (LS), and Focal Loss + MaxSup. Across all imbalance ratios and splits (val/test), MaxSup consistently outperforms both the baseline and LS in overall accuracy, which jointly reflects the many-shot, medium-shot, and low-shot (minor class) performance. For example, at an imbalance ratio of 50 on the test split, MaxSup achieves 81.4% accuracy, outperforming Focal Loss (76.8%) by 4.6 percentage points, and LS (80.5%) by 0.9 percentage points. These results indicate that MaxSup is robust under class imbalance, especially in challenging settings with fewer samples in tail classes.

To address the concern regarding the effectiveness of MaxSup on out-of-distribution (OOD) settings, we also conducted experiments on CIFAR10-C benchmark shown in Table R2 following settings in [2*]. Table R2 reports the performance of MaxSup and Label Smoothing (LS) on this benchmark using ResNet-50 as the backbone. Specifically, LS yields a better NLL (1.5730 vs. 1.8431), implying more confident probabilistic predictions. However, MaxSup achieves a better ECE (0.1479 vs. 0.1741), indicating better calibration of the predicted confidence scores. These results validate that MaxSup remains effective on OOD datasets, achieving performance comparable to LS across all three metrics.

Table R1: Comparison of overall accuracy (%) (jointly considering many-shot, median-shot, and low-shot top-1 accuracy) for different loss strategies (Focal Loss vs Label Smoothing (LS) vs MaxSup) across datasets, imbalance levels, and backbones. Best performances are in bold.

Table R2: Comparison of MaxSup and Label Smoothing (LS) on CIFAR10-C using Resnet50 as backbone. For all metrics, lower value is better, and best performances are in bold.

[1*] Tang, K., Huang, J., & Zhang, H. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In NeurIPS 2020, pp. 1513–1524.

[2*] Heinonen, M., Tran, B. H., Kampffmeyer, M., and Filippone, M. "Robust Classification by Coupling Data Mollification with Label Smoothing." AISTATS 2025.

Q2: How about integrating the proposed method into other label smoothing methods? It would be synergized enough to increase the model performance

Thank you for the suggestion. As shown in Table 14 in Appendix L (Ablation on the Weight Schedule), we combined the adaptive scheduler from Adaptive Label Smoothing [18] with our MaxSup method, which indeed led to consistent performance improvements. This indicates that there is potential synergy between MaxSup and existing label smoothing variants.

We leave a more comprehensive exploration of such combinations—especially with more advanced or task-specific label smoothing techniques—as an interesting direction for future work.

"Things to improve the paper that did not impact the score"

We are glad to resolve these formatting issues in the revised version.