MaxSup: Fixing Label Smoothing for Improved Feature Representation

We thank the reviewer for the invaluable feedback. We address the mentioned points in the following:

W1: The motivation of MaxSup is unclear. In Section 3.1, Label Smoothing Loss is decomposed into two components: the regularization term and the error-enhancement term. Preliminary studies validate that using only the regularization term is effective. If so, why not directly minimize the regularization term as the learning objective?

Replacing $z_{max}$ with $z_{gt}$ is indeed the optimal choice, and directly dropping the error-enhancement term is suboptimal. We have revised Section 3.1 of our manuscript to provide an extended elaboration on this aspect to improve the clarity. We would like to emphasize that the goal of regularization is to penalize the network for being over-confident about its prediction, i.e., the top-1 position $z_{max}$. However, Label Smoothing not only introduces an undesired error-enhancement term, but also fails to penalize the network's incorrect prediction, i.e., $z_{max} \neq z_{gt}$ in this case. Therefore, simply dropping the error-enhancement term only avoids enhancing the over-confidence in the incorrect prediction, but fails to regularize it. In contrast, replacing $z_{gt}$ with $z_{max}$ could solve both issues perfectly.

This is also supported by the superior performance of MaxSup: Using the regularization term alone ($75.98\%$) only brings marginal improvement ($+0.07\%$) over Label Smoothing ($75.91\%$), from Table 1. In contrast, MaxSup ($76.12\%$) leads to larger improvement ($0.21\%$) over Label Smoothing ($75.91\%$), from Table 6 (Table 8 in the revised version). In order to improve the clarity, we have added the result of MaxSup in Table 1 in our revised manuscript.

W2: The incorrect prediction samples may bring noise to MaxSup during optimization.

As explained in our answer to W1, the intended regularization, i.e., penalizes the network for being over-confident about its prediction (the peak position, i.e., $z_{max}$, should be regarded as the prediction of a classifier), can be only achieved by MaxSup when the prediction is incorrect, whereas the regularization term of Label Smoothing fails to penalize the network for being over-confident about its incorrect prediction (Inconsistent Regularization). MaxSup is also demonstrated to improve the feature representation over Label Smoothing, please refer to our answer to Reviewer RCT5 W3.

W3: provide the result of using only the regularization term in Table 2 and Table 3.

The results of using only the regularization term in Table 2 and Table 3 are 75.84±0.18 and 79.96±0.13, which are inferior to the results of using MaxSup, i.e., 77.08±0.07 and 80.16±0.09.

W4: Figure 2 shows a few samples that are correctly classified by the proposed method but incorrectly by the baseline. Are there any samples that are correctly classified by the baseline but incorrectly by the proposed method?

We measure the average top-1 accuracy on different class in ImageNet-1k dataset.

Caption: Class-wise results.

We provide three samples in butternut_squash class in validation set in ImageNet-1K.

Caption: Sample-wise results.

W5: Some notations are confusing.

We have resolved these issues in our revised manuscript and address your questions in the following:

W5.1: What is $q$ in Lemma 3.2?

$\mathbf{q}$ denotes the predicted probability vector.

W5.2: Where is the $\lambda$ in Eq. (7)?

The $\lambda$ was due to sloppy writing, and has been removed.

W6: Some typos, e.g., Line 266, "Similarity:".

We have corrected in the revised version.

We thank the reviewer for the constructive feedback. We address the mentioned concerns in the following:

W1: The overall contribution seems to be more of an engineering refinement.

We believe there exists some misunderstanding regarding our core contribution. Our work theoretically reveals the reason behind the empirically observed issue that label smoothing results in more confident errors [7]. Based on our analysis, we derive MaxSup as the optimal solution to such an issue. We have revised Section 3.1 of our manuscript to provide an extended elaboration on the superiority of MaxSup over Label Smoothing to improve the clarity.

We would like to emphasize that the goal of regularization is to penalize the network for being over-confident about its prediction, i.e., the top-1 position $z_{max}$. However, Label Smoothing not only introduces an undesired error-enhancement term, but also fails to penalize the network's incorrect prediction, i.e., $z_{max} \neq z_{gt}$ in this case. And MaxSup could solve both issues perfectly by replacing $z_{gt}$ with $z_{max}$.

The theoretically grounded design of MaxSup leads to its superior performance over empirical soft label approaches: From the extended evaluation of our approach using various architectures across multiple datasets in our revised manuscript, MaxSup consistently achieves the highest accuracy among label smoothing alternatives, whereas OLS and Zipf-LS fail to deliver stable performance, indicating that the previous empirical justification of such methods is limited to certain training schemes.

Caption: Performance comparison of classic convolutional neural networks on ImageNet-1K. The training script used was consistent with TorchVision V1 Scripts. Note that a larger batch size was employed to accelerate the experimental process, and the learning rate was adjusted based on the linear scaling principle.

Caption: Performance comparison of classic convolutional neural networks on CIFAR100. The training script used was consistent with TorchVision V1 Scripts.

Caption: Comparison of DeiT-Small (without CutMix&Mixup) accuracy (%) with Other Label Smoothing Variants.

W5: Providing the quantitative results regarding inter-class separability and intra-class variation would be beneficial in better elucidating Figure 2.

We appreciate the constructive feedback and have quantitatively validated the improved feature representation with MaxSup over Label Smoothing using intra-class variation measure $d_\text{within}$ (the larger, the better transfer learning and richer similarity information—reflecting how individual examples relate to different classes) and inter-class separability measure $R^2=1- \frac{d_\text{within}}{d_\text{total}}$ (the larger, the better classification) metrics from [6] (suggested by Reviewer 2YeD), respectively:

Caption: Quantitative measures for inter-class separability and intra-class variation of feature representations, using ResNet-50 trained on ImageNet-1K. The results are calculated on the ImageNet training set. As analyzed in [6], all regularization losses reduce the relative intra-class variance in the penultimate layer representation space, but our MaxSup suffers from the least reduction of intra-class variance, significantly improving the intra-class variance as well as the inter-class separability over Label Smoothing.

W6: Line 87: calibrated classifiers M¨uller et al. (2019). -> calibrated classifiers (M¨uller et al. 2019).

We have resolved the mentioned issue in our revised manuscript as suggested.

W7: Which dataset was used in Figure 2?

It was ImageNet-1K.

reference:

• [4] Shen, Yiqing, et al. "Self-distillation from the last mini-batch for consistency regularization." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.
• [5] Xu, Ting-Bing, and Cheng-Lin Liu. "Data-distortion guided self-distillation for deep neural networks." Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.
• [6] Kornblith et al. Why Do Better Loss Functions Lead to Less Transferable Features? NeurIPS 2021
• [7] Zhu et al. Rethinking Confidence Calibration for Failure Prediction, ECCV 2022

Thanks for addressing my concerns in detail. I acknowledge that this is an interesting work advancing the Label Smoothing (LS) method, especially in the detailed analysis of the LS method. However, I believe the following issues still need to be addressed:

1. The authors claim that MaxSup is the optimal solution to address LS’s two issues: (i) Error-Enhancement; and (ii) Inconsistent Regularization. However, those issues can also be tackled by the other approaches, e.g., $L = \frac{\alpha}{K} \sum\_{z\_m < z\_{gt}} (z\_{gt} – z\_m) + \frac{\alpha}{K} (z\_{\max} – z\_{gt})$. Also, many alternative solutions could be explored using other types of loss functions, e.g., hinge loss. How do you theoretically prove MaxSup is the optimal solution compared to those alternatives?

2. Another crucial issue is that the performance improvement of MaxSup over LS seems to be limited, despite the authors' clarification. In particular, the improvement of MaxSup over LS, and even vanilla CE, is often less than 1%. If this is the first method proposed to enhance CE loss for improving generalization, such an improvement could be sufficient. However, LS is the first work that is both simple and efficient, as it does not require an additional sorting operation to select the top-1 logit.

In summary, given the marginal performance improvement and the limited technical contribution to the LS field, I am inclined to maintain my original score.

Thanks for your thoughtful discussion on the “limited improvement” aspect. I appreciate your insight that simple approaches are crucial to the advancement of the machine learning (ML) community. Indeed, many simple yet effective methods have been developed over the years, significantly contributing to the progress of ML.

Label Smoothing (LS) is a prime example of this, as it was the first method to address the overconfidence and hindered generalization issues caused by one-hot label encoding. While I agree that MaxSup is a simple approach, I believe that it can be seen more as a modest adjustment to LS, resulting in limited improvement, rather than a transformative solution. It addresses certain issues but does not revolutionize LS to provide a general solution to one-hot label encoding or transform the current supervised learning paradigm. Therefore, I believe that there is still much room for exploration in this field, and I hope that future work will present more groundbreaking methods to overcome the inherent limitations of LS, rather than merely repairing or patching it.

We are grateful to the reviewer for engaging in this follow-up discussion. And we address the mentioned aspects as follows:

A1.1: Although the suggested loss is equivalent to MaxSup and Label Smoothing when the network makes correct predictions, it is indeed problematic when the network makes incorrect predictions:

• For the suggested loss: the gradient w.r.t. $z_{gt}$ is $\frac{\partial L}{\partial z_{gt}}= \frac{M-1}{K}\alpha$, whereas the gradient w.r.t. $z_{max}$ is $\frac{\partial L}{\partial z_{max}}=\frac{\alpha}{K}$.

• For our MaxSup: the gradient w.r.t. $z_{gt}$ is $\frac{\partial L_\text{MaxSup}}{\partial z_{gt}}= -\frac{1}{K}\alpha$, whereas the gradient w.r.t. $z_{max}$ is $\frac{\partial L_\text{MaxSup}}{\partial z_{max}}=\frac{K-1}{K}\alpha$.

The positive gradient $\frac{M - 1}{K} \alpha$ of the suggested loss w.r.t. $z_{gt}$ implies that the model will decrease the already underestimated $z_{gt}$ during backpropagation. Moreover, the gradient $\frac{\alpha}{K}$ of the suggested loss w.r.t. $z_{max}$ has a significantly smaller magnitude, which might not sufficiently suppress the overly large $z_{max}$. In contrast, the negative gradient $-\frac{1}{K}\alpha$ of MaxSup w.r.t. $z_{gt}$ encourages to increase the underestimated $z_{gt}$, and the significantly larger gradient $\frac{K-1}{K}\alpha$ of MaxSup w.r.t. $z_{max}$ correctly focuses on suppressing the overly large $z_{max}$.

A 1.2: While exploring these alternatives might be interesting, it lies beyond the scope of this study: our work focuses on the issues of Cross-Entropy loss with Label Smoothing. Since Cross-Entropy is the most widely adopted loss functions in the current deep learning landscape, focusing on it does not diminish the significance of our work.

A2.1: Regarding the reviewer's concern about the improvement in classification, it is essential to consider both performance and cost for evaluating the significance of a method. The "less than $1\%$" improvements over the highly optimized baselines are indeed substantial, especially considering that almost no extra cost is required (please refer to our explanation in A2.2): MaxSup works as a drop-in replacement of LS, whereas OLS has to maintain the statistics of the model prediction for each category, and Zipf employs a sophisticated pipeline to process the extra pixel-wise predictions, introducing additional overhead to the training process. As mentioned in our answer to W2, our MaxSup consistently outperforms OLS and Zipf-LS, whereas OLS and Zipf-LS fail to deliver stable performance. This indicates that such more sophisticated methods may rely on additional hyperparameter tuning to perform well.

Moreover, it is noteworthy that our contributions are threefold, as stated clearly in our introduction section:

1. Theoretical contribution: We reveal the underlying mechanism of the previously observed defects of Label Smoothing and derive MaxSup as a remedy.
2. Empirical contribution: MaxSup improves classification over LS.
3. Empirical contribution: MaxSup improves feature representation and transfer learning over LS.

In comparison, empirical soft label approaches such as OLS and Zipf-LS only demonstrate improvements on classification over LS, but lack both theoretical contribution and demonstration of improved feature representation.

A2.2: We would like to kindly remind that finding the top-1 logit doesn't require sorting, and the complexity of max() operation is just $\mathcal{O}(n)$, even for large values of $n$. Furthermore, the number of classes is typically small—for instance, 1,000 in the case of ImageNet—making this computational cost negligible in the context of training neural networks.

My impression is that the authors' use of the word "optimal" was unintentionally imprecise. The paper itself doesn't make any such claims and it is difficult in and of itself to make any meaningful claims about optimality (in the sense of maximising some criterion) when we are measuring performance according to generalisation on unseen data.

Thank you for your detailed response. I would like to take this opportunity to clarify the following points:

First, I appreciate your acknowledgment that the proposed MaxSup may not be a theoretically optimal solution to the issues of LS. Accordingly, my main point remains that while the authors have provided a theoretical analysis highlighting the issues of LS, there is a lack of theoretical proof for why MaxSup is presented as the optimal solution. Given that MaxSup has not been theoretically proven to be optimal, I regard it primarily as an intuitive engineering design, making minor adjustments to LS, resulting in a modest performance improvement. As such, I believe that MaxSup represents more of an engineering approach than an innovative solution. It does not fundamentally revolutionize LS, but rather serves as a corrective measure. Consequently, the novelty and contributions of MaxSup are limited.

Second, regarding my previous example of an engineering technique, I intended to illustrate that your method should be seen as an engineering solution rather than a novel theoretical breakthrough. I kindly request that we refrain from focusing on the differences between these methods. I fully recognize that they are distinct approaches, but their commonality lies in the fact that both are engineering tricks.

Third, to avoid any further misunderstandings, I will refrain from providing any additional examples this time.

We appreciate the Reviewer for acknowledging our analysis of the issues of Label Smoothing, as well as the rationale behind the choice of MaxSup as a solution. And we further address the questions in the following:

A1: We thank the reviewer for suggesting alternative solutions for discussion during the rebuttal. We agree that the new example provided by the reviewer for discussion would alleviate the issues of Label Smoothing as well, since this example follows the core design rules of MaxSup. However, we show that MaxSup would still be the preferable option compared to such alternatives, due to the simplicity of MaxSup:

First, the provided example is equivalent to the following:

which is essentially MaxSup (Eq. 7 in our paper) plus an additional term.

Moreover, as long as the additional term does not contain $z_{\text{max}}$ and $z_{\text{gt}}$, a family of such losses with the same gradients as MaxSup w.r.t. $z_{\text{max}}$ and $z_{\text{gt}}$ can be constructed:

the constants such as $\alpha$ and $\frac{1}{K}$ could be further varied, but these are just trivial details and thus negligible. For such alternatives, a concern arises that the impact of the additional term is unknown: MaxSup only modifies the max and gt positions of Label Smoothing (this is more clear by comparing the soft labels in Eq. 1 and Eq. 8 in our paper), which are essential to solve the error-enhancement and inconsistent regularization issues of Label Smoothing. Additional changes are avoided (a small amount of probability mass is still uniformly distributed to the other positions as done in Label Smoothing), since they are irrelevant to the studied issues and their impact is unknown (exploration of alternatives to the uniform label component $\frac{\alpha}{K}$ in Eq. 8 is orthogonal to our study). In line with the widely accepted heuristic of Occam's Razor, i.e., Entities should not be multiplied unnecessarily, we assert that MaxSup is an elegant solution.

As for the term "optimal", we totally agree with the Reviewer that reasoning must be provided for such a strong wording. Note that we never claimed optimality in the paper, and we apologize for the imprecise use of the word "optimal" in our answer to Reviewers during the rebuttal. As pointed out by Reviewer 2YeD in the reply to Reviewer 59zm, it is difficult in and of itself to make any meaningful claims about optimality (in the sense of maximising some criterion) when we are measuring performance according to generalisation on unseen data. And our contributions are independent of the assumption of optimality. Therefore, "optimality" shall be less relevant to the evaluation of our paper.

A2: We thank the Reviewer for the explicit explanation of the remaining concern, and we would like to show that MaxSup and the mentioned engineering trick are substantially different from each other regarding both concepts and mechanisms. We believe the Reviewer's claim might simply stem from a misunderstanding of our method, and we are glad to provide a detailed explanation:

• The concepts are different: MaxSup is a regularization method for penalizing overconfident predictions of a classification network, whereas the mentioned engineering trick is essentially a normalization method for preventing numerical overflow of Softmax function.

• The mechanisms are different: The mentioned engineering trick is based on the translation invariance of Softmax function, i.e., $Softmax_i(\mathbf{z}+\mathbf{c}) = \frac{\text{exp}(z_i + c)}{\sum_{j=1}^C \text{exp}(z_j + c)}= \frac{\text{exp}(z_i)\text{exp}(c)}{\sum_{j=1}^C \text{exp}(z_j)\text{exp}(c)} =\frac{\text{exp}(z_i)}{\sum_{j=1}^C \text{exp}(z_j)}=Softmax_i(\mathbf{z}) ,$ where $\mathbf{c}=(c,...,c)$ could be an arbitrary constant vector, which will be eliminated. Setting $c=-z_{\text{max}}$ normalizes the maximum exponentiated value to $e^0=1$, but the output remains unchanged.

  In contrast, the training objective of a classifier with MaxSup is: $L = H(\mathbf{y}, \mathbf{q}) + \alpha(z_{*max*} - \frac{1}{K} \sum^{K}_{k = 1} z_k),$ where $\mathbf{q}$ denote the predicted probability vector, and $\mathbf{y}$ denote the one-hot ground-truth label. Note that our MaxSup is a stand-alone regularization term in addition to the vanilla cross-entropy loss, which is not added to the input of softmax and can not be eliminated.

To avoid such a misunderstanding for future readers, we carefully inspected our manuscript and suspect that it probably occurs because we only presented the stand-alone Max Suppression Loss in Eq. 7, and we plan to improve the clarity by replacing it with the full training objective as above in our camera ready version.

Also, we would be glad if the reviewer can provide a specific reference for the heuristic so that we can add a short discussion to the paper.

That approach to modifying softmax is simply an engineering solution, and I am not aware of a specific reference for it. Additionally, as mentioned in my previous comment, there is no need to include that reference. Thanks.

We apologize if we misread your previous comment. Pointing out the different effects of the regularization and normalization approaches was meant to provide clarification on the contribution of MaxSup. Again, if you would provide us with a reference, we gladly refer to the work in our related work section.

We sincerely appreciate the time and effort the Reviewer has dedicated to the review process. We fully understand that, given the workload and tight time constraints, misunderstandings may occasionally occur. However, we would like to respectfully address what we believe to be an unfounded or irresponsible response.

• The claim that "Given that MaxSup has not been theoretically proven to be optimal, I regard it primarily as an intuitive engineering design" is unfounded. Instead of proving MaxSup as an optimal solution, we have provided theoretical proof (please refer to the gradient analysis in Appendix C) that MaxSup is an effective and concise solution to the studied issues of Label Smoothing. Note that even Label Smoothing and its alternatives, including Zipf-LS and OLS, are not proven as an optimal solution, yet Label Smoothing is widely applied in training classifiers.

• The claims that "regarding my previous example of an engineering technique, I intended to illustrate that your method should be seen as an engineering solution rather than a novel theoretical breakthrough" and "their commonality lies in the fact that both are engineering tricks" are unfounded, as the reviewer admitted "I fully recognize that they are distinct approaches" and "I am not aware of a specific reference for it".

• The claim that 'there is no need to include that reference' falls outside the bounds of a rigorous scientific discussion and reflects a lack of responsibility.

We thank the reviewer for the knowledgeable review and invaluable suggestions. We address the mentioned points in the following:

W1: Positioning relative to existing/concurrent work.

We address the discussions in the following:

W1.1. Existing work [1] empirically identified the more confident errors caused by Label Smoothing.

We have revised our contribution as "uncovering the underlying mechanism of the recently observed defect of Label Smoothing" to acknowledge their contribution in our revision. Indeed, we regard the existence of the empirical finding as rather positive, because our theoretical analysis reveals the reason behind their empirical finding, which supports the validity of our theoretical analysis in return.

W1.2. Concurrent work [2] similarly shows how the logits are suppressed differently depending on whether a prediction is likely to be correct or not.

The observation in [2] is made by analyzing the gradients w.r.t. logits, while our analysis is based on logit-level reformulation of the training objective. The findings from both studies can be seen as mutually validating. However, our decomposition offers an additional advantage, as it allows us to derive MaxSup as a direct solution to the observed problem, whereas [2] merely recommend an existing post-hoc technique as remedy, i.e., logit normalization [11]. We have added this discussion in Section 3.1 in our revised manuscript.

W1.3 Related work [3, 4, 5, 6, 7, 8] are missing.

We have included [3-8] in the related work in our revised manuscript. Following the reviewer's suggestion, we have also provided quantitative evaluation of feature separability/variation and transfer learning performance, following [6].

W2: Lacking analysis.

We address the suggestions in the following:

W2.1: Lack of quantitative measures over the training and test datasets, as well as comparison to vanilla CE.

We appreciate the insightful feedback and have quantitatively evaluated the intra-class variance $d_\text{within}$ and inter-class separation $R^2$ from [6], respectively:

Caption: Quantitative measures for inter-class separability and intra-class variation of features representations, using ResNet-50 trained on ImageNet-1K. The results are calculated on the ImageNet training set.

As analyzed in [6], all regularization losses reduce the relative intra-class variance in the penultimate layer representation space, but our MaxSup suffers from the least reduction of intra-class variance, significantly improving the intra-class variance as well as the inter-class separability over Label Smoothing.

W2.2: The claims should be supported by comparing the test accuracy vs transfer linear probing (representation learning) performance, following [6]. It is unclear whether or not the downstream segmentation loss also includes MaxSup/LS and the experiment lacks a vanilla CE baseline (which the authors should include).

Caption: Validation performance of different methods based on multi-nominal Logistic Regression with $l_2$ regularization in CIFAR10 validation set. We searched the strength of the regularization from $1e-4$ to $1e2$, the search step size is increasing by an order of magnitude.

We evaluated the linear transfer performance of a ResNet50 model, pre-trained on ImageNet, on the CIFAR-10 dataset using different methods following [6]. As for the segmentation task, the vanilla Cross-Entropy loss is used for fine-tuning. We have also included the result of using vanilla CE baseline in the revised manuscript as follows:

Caption: Comparison of Label Smoothing and MaxSup on on ADE20K validation set, and the best result on ADE20K with only ImageNet-1K as training data in pretraining.

W2.3: The differences in observed separation/variation in section 3.3 should be linked to the theoretical analysis in section 3.1 and 3.2.

We would like to emphasize that the goal of regularization is to penalize the network for being over-confident about its prediction, i.e., the top-1 position $z_{max}$. However, Label Smoothing not only introduces an undesired error-enhancement term, but also fails to penalize the network's incorrect prediction, i.e., $z_{max} \neq z_{gt}$ in this case. Since MaxSup consistently regularizes both the correct and incorrect predictions (top-1 probability), thereby it leads to even larger inter-class separability. Moreover, MaxSup eliminates the error-enhancement defect of Label Smoothing, which may be the cause of the severely reduced intra-class variance.

W2.4: Additional explicit discussion on how increasing confidence on training errors would lead to worse generalization in terms of top 1 test accuracy. For example adding a sentence like: LS weakens the strength of supervision to the GT class on errors, when a model really should be learning to correct itself rather than being regularized to prevent over-fitting.

We have revised Section 3.1 of our manuscript to provide an extended elaboration on this aspect to improve the clarity. Please also refer to our answer to W2.3 for more details.

W3: Presentation, missing information and clarity.

We address the suggestions in the following:

W3.1: The paper would benefit either from an expanded (and separate) discussion and analysis on the alpha schedule, or just simply from its removal from the paper.

Based on the reviewer's suggestion, we have revised the manuscript by removing the details of the alpha scheduler from the main text and adding a separate section in the appendix to discuss it.

W3.2: Improve the clarity of Sec 3.1 with updated wording and maybe some additional underbraces to the equations (like in Eq. 6).

To avoid the confusion, we have now adopted "Cross-Entropy Loss with Hard Label" and "Cross-Entropy Loss with Soft Label" to denote the vanilla training objective and the training objective with Label Smoothing, respectively. "Label Smoothing Loss" is kept to denote the the loss component introduced by Label Smoothing. In addition, we have added underbraces to each of these terms in Eq. 2 for improved clarity.

W3.3: Omit the decision boundaries and numerical measures [6,7] may convey the point more clearly than the plots.

We have put Figure 2 in the appendix and will omit the decision boundaries as suggested. Instead, we provided numerical measures as mentioned in our answer to W2.

W3.4: For CAM figures, the authors should add a comparison to the baseline cross entropy model as well. The figures would also benefit from subcaptions describing what to look out for, e.g. "label smoothing fails to consider the tail of the monkey".

We have added one row for baseline model to the CAM figures, and also improved the caption as suggested.

W3.5: The authors should make clear the difference between overconfidence in the context of model calibration and overconfidence in the sense of error enhancement.

In our paper, the "overconfidence" consistently regards the network's confidence in its top-1 prediction, regardless of whether the prediction is correct or not. As explained in our revised analysis in our answer to W2.3, LS indeed only reduces the overconfidence in the correct prediction, but fails to reduce the overconfidence in the incorrect prediction (Inconsistent Regularization) and even further enhanced it (Error Enhancement).

W3.6: Generally speaking, there are a number of grammar and spelling errors. The authors should use an automated grammar checking tool to correct these.

We have addressed the grammar issues as per the reviewer's suggestions.

W4: Suggestion to use a clean baseline without additional label augmentations such as hard-label distillation, cutmix and mixup.

We would like to kindly remind that the widely adopted Deit baseline is trained on ImageNet1K from scratch, without using Knowledge Distillation. However, we agree that using a clean baseline without cutmix and mixup is a better choice. Therefore, we have added the comparison to other approaches using Deit baseline without cutmix and mixup.

Caption: Comparison of DeiT-Small (without CutMix&Mixup) accuracy (%) with Other Label Smoothing Variants.

Reference:

• [11] Luís Felipe Prates Cattelan and Danilo Silva. How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks. In The 40th Conference on Uncertainty in Artificial Intelligence, 2024

Dear Reviewer 2YeD,

Thank you once again for your insightful feedback throughout the review process. Your input has significantly contributed to improving the quality of our paper, and we are very pleased with the results.

Best regards,

Authors

We thank the reviewer for the constructive feedback and address the mentioned points in the following:

W1: Why replacing $z_\text{gt}$ with $z_\text{max}$ is the optimal choice?

This can be explained by our decomposition of the Label Smoothing Loss in equation 6. Thanks to the reviewer's feedback, we have revised Section 3.1 of our manuscript to provide an extended elaboration on this aspect to improve the clarity. We would like to emphasize that the goal of regularization is to penalize the network for being over-confident about its prediction, i.e., the top-1 position $z_{max}$. However, Label Smoothing not only introduces an undesired error-enhancement term, but also fails to penalize the network's incorrect prediction, i.e., $z_{max} \neq z_{gt}$ in this case. Therefore, simply dropping the error-enhancement term only avoids enhancing the over-confidence in the incorrect prediction, but fails to regularize it. In contrast, replacing $z_{gt}$ with $z_{max}$ could solve both issues perfectly.

This is also supported by the superior performance of MaxSup: Using the regularization term alone ($75.98\%$) only brings marginal improvement ($+0.07\%$) over Label Smoothing ($75.91\%$), from Table 1. In contrast, MaxSup ($76.12\%$) leads to larger improvement ($0.21\%$) over Label Smoothing ($75.91\%$), from Table 6 (Table 8 in the revised version). In order to improve the clarity, we have added the result of MaxSup in Table 1 in our revised manuscript.

W2: The targets created by the approach are not a probability, and hence, using it with cross-entropy loss may not be correct.

Although the targets created by our approach are not a probability, they are guaranteed to be compatible with cross-entropy based on the decomposed loss component in equation 7: MaxSup is essentially a stand-alone regularization loss in addition to the vanilla cross-entropy loss, and its equivalent label-level formulation in equation 8 is merely intended to show its connection to Label Smoothing. Similarly, Negative Label Smoothing [8] also adopts non-probability targets with cross-entropy as a penalization mechanism, which has been demonstrated to be effective.

W3: Figure 2 is very confusing and difficult to understand.

Due to the lack of clarity in the qualitative evaluation in Figure 2, we rely on quantitative evaluation to demonstrate the improved feature representation with MaxSup in our revised manuscript, using intra-class variation measure $d_\text{within}$ (the larger, the better transfer learning) and inter-class separability measure $R^2 = 1 - \frac{d_\text{within}}{d_\text{total}}$ (the larger, the better classification) metrics from [6] (suggested by Reviewer 2YeD), respectively:

Caption: Quantitative measures for inter-class separability and intra-class variation of feature representations, using ResNet-50 trained on ImageNet-1K. The results are calculated on the ImageNet training set. As analyzed in [6], all regularization losses reduce the relative intra-class variance in the penultimate layer representation space, but our MaxSup suffers from the least reduction of intra-class variance, significantly improving the intra-class variance as well as the inter-class separability over Label Smoothing.

W4: The paper compares the approach across different modalities but should also include results with multiple datasets from the same modality and various architectures (Like done in OLS [2] and Zipf [3]).

We have revised the manuscript with extended evaluation of our approach using various architectures across multiple datasets. MaxSup consistently achieves the highest accuracy among label smoothing alternatives, whereas OLS and Zipf-LS fail to deliver stable performance, indicating that the previous empirical justification of such methods is limited to certain training schemes.

Caption: Performance comparison of classic convolutional neural networks on ImageNet-1K. The training script used was consistent with TorchVision V1 Scripts. Note that a larger batch size was employed to accelerate the experimental process, and the learning rate was adjusted based on the linear scaling principle.

Caption: Performance comparison of classic convolutional neural networks on CIFAR100. The training script used was consistent with TorchVision V1 Scripts.

W5: Label Smoothing is known to affect semantic class relationships [1]. The baseline approaches (OLS [2], Zipf [3]) also formulate their loss function to mitigate this, but it is not handled in the proposed approach.

MaxSup indeed improves the feature representation w.r.t. class relationships in a more elegant way, which can be proved by the improved intra-class variance and inter-class separability with MaxSup (Please refer to our answer to W3 and Q3). In contrast, both OLS and Zipf merely demonstrate improved classification performance over label smoothing, but it is not justified whether they have mitigated the issue of Label Smoothing's semantic class relationships.

Moreover, our work provides theoretical justification and thoroughly derives a solution to the inherent issue of LS resulting in more confident errors, which is empirically identified very recently [7]. Our consistently superior performance compared to OLS and Zipf (please refer to our answer to W4) also suggests that the potential of such empirical approaches could be inherently limited.

W6: In Tables 4 and 5, comparison is only done with the vanilla label smoothing. The improvement is minimal. It would be interesting to see the results of baseline approaches on them.

The BLEU score of the baseline approach in Table 4 is 34.3 ± 0.09, so the improvement of Label Smoothing itself is minor, probably due to the limited dataset size. And the $200$% relative improvement ($+0.3\%$ with MaxSup vs. $+0.1\%$ with LS) of MaxSup over Label Smoothing should be satisfactory.

The mIoU of the baseline approach is 43.4 in Table 5, so the improvement using Label Smoothing and MaxSup is $0.3$ and $0.7$, respectively. The over $100$% relative improvement of MaxSup over Label Smoothing is significant.

To better highlight our improvement, we have added the baseline results in Table 4 and 5, and emphasized the improvement of our MaxSup over the baseline.

Q1: In the ablation (Table 1), The paper showed removing the Error-Enhancement is beneficial. Why is that not compared/proposed as inplace for replacement?

When simply removing the Error-Enhancement term, the $+0.07\%$ improvement (baseline + Label Smoothing $75.91\%$ vs. baseline + regularization $75.98\%$) is marginal. We provide a detailed explanation on why replacing $z_{gt}$ with $z_{max}$ is optimal in our answer to W1.

Q2: Why is the error-enhancement term having a huge negative impact? The logits of the correct class are generally higher while training.

The $0.58\%$ accuracy drop (baseline $74.21\%$ vs. baseline + error enhancement $73.63\%$) is reasonable, because the error-enhancement takes effect since the training started and the accuracy gradually increases from near 0 to the final accuracy of $73.63\%$. Even when the training converges, there are still around $25$% incorrect predictions.

Q3: How does it impact the inter-class relationship?

As analyzed in our answer to W1, Label Smoothing only performs the desired regularization on the correct predictions (top-1 probability), whereas MaxSup regularizes both the correct and incorrect predictions (top-1 probability), thereby leading to even larger inter-class separability. Please refer to our answer to W3 for more details. Moreover, MaxSup eliminates the error-enhancement defect of Label Smoothing, which may be the cause of the severely reduced intra-class variance.

Reference:

• [6] Kornblith et al. "Why Do Better Loss Functions Lead to Less Transferable Features?" *NeurIPS 2021.
• [7] Zhu et al. "Rethinking Confidence Calibration for Failure Prediction." ECCV 2022.
• [8] Wei, Jiaheng, et al. "To smooth or not? When label smoothing meets noisy labels." arXiv preprint arXiv:2106.04149 (2021).

Thank you for the rebuttal. I was on a family emergency and could not get back in time. I have a few open questions after going through the rebuttal:

W1: Why replacing $z_\text{gt}$ with $z_\text{max}$ is the optimal choice? Rv: I see the benefits of using $z_\text{max}$ on performance and understand the intuition behind why the method chose $z_\text{max}$ but having a theoretical reasoning would back it up better. I think a better replacement could be explored or reasoning must be provided as to why is the optimal choice.

W2: The targets created by the approach are not a probability, and hence, using it with cross-entropy loss may not be correct. Rv: "Guaranteed to be compatible" is not the right approach. It also needs to make sense mathematically, which states that targets must be probability vectors.

W3: Figure 2 is very confusing and difficult to understand. Rv: Thank you for the analysis. It was helpful.

W4: The paper compares the approach across different modalities but should also include results with multiple datasets from the same modality and various architectures (Like done in OLS [2] and Zipf [3]). Rv: These results help a lot.

W5: Label Smoothing is known to affect semantic class relationships [1]. The baseline approaches (OLS [2], Zipf [3]) also formulate their loss function to mitigate this, but it is not handled in the proposed approach. Rv: The explanation still does not talk about/clarify the impact on class relationships.

W6: In Tables 4 and 5, comparison is only done with the vanilla label smoothing. The improvement is minimal. It would be interesting to see the results of baseline approaches on them. Rv: I agree it is an improvement over vanilla baselines but that does not mean it will be better than other alternatives. The baselines must be compared as well.

Q1: In the ablation (Table 1), The paper showed removing the Error-Enhancement is beneficial. Why is that not compared/proposed as inplace for replacement? Rv: I understand the improvements. But we need to compare the results on a common setup. Either the results without Error-Enhancement should be included for the results datasets (preferred way) or performance in this setup with the proposed approach.

Q2: Why is the error-enhancement term having a huge negative impact? The logits of the correct class are generally higher while training. Rv: Thank you for the explanation.

Q3: How does it impact the inter-class relationship? Rv: The explanation still does not talk about/clarify the impact on class relationships.

We sincerely thank the reviewer for the follow-up suggestions and questions. We address the raised points in the following:

W1: Why replacing $z_{\text{gt}}$ with $z_{\text{max}}$ is the optimal choice? Rv: I see the benefits of using $z_{\text{max}}$ on performance and understand the intuition behind why the method chose but having a theoretical reasoning would back it up better. I think a better replacement could be explored or reasoning must be provided as to why is the optimal choice.

A1: We sincerely appreciate the Reviewer for acknowledging the benefits of MaxSup, as well as the rationale behind it.

As for the term "optimal", we totally agree with the Reviewer that reasoning must be provided for such a strong wording. Note that we never claimed optimality in the paper, and we apologize for any imprecise uses of the word "optimal" in our answer to Reviewers during the rebuttal. As pointed out by Reviewer 2YeD in the reply to Reviewer 59zm, it is difficult in and of itself to make any meaningful claims about optimality (in the sense of maximizing some criterion) when we are measuring performance according to generalisation on unseen data. And our contributions are independent of the assumption of optimality. Therefore, "optimality" shall be less relevant to the evaluation of our paper.

However, we are pleased to provide more detailed reasoning for proposing MaxSup. Note that MaxSup only modifies the max and gt positions of Label Smoothing (Please refer to Eq. 1 and Eq. 8 for comparison), which are essential to solve the error-enhancement and inconsistent regularization issues of Label Smoothing: when the network makes incorrect predictions, the gradient of MaxSup loss w.r.t. $z_{gt}$ is $\frac{\partial L_\text{MaxSup}}{\partial z_{gt}}= -\frac{1}{K}\alpha$, whereas the gradient w.r.t. $z_{max}$ is $\frac{\partial L_\text{MaxSup}}{\partial z_{max}}=\frac{K-1}{K}\alpha$(please refer to our Appendix C.2 for more details). The negative gradient $-\frac{1}{K}\alpha$ of MaxSup w.r.t. $z_{gt}$ encourages to increase the underestimated $z_{gt}$, and the significantly larger gradient $\frac{K-1}{K}\alpha$ of MaxSup w.r.t. $z_{max}$ correctly focuses on suppressing the overly large $z_{max}$.

Additional changes are avoided (a small amount of probability mass is still uniformly distributed to the other positions as done in Label Smoothing), since they are irrelevant to the studied issues and their impact is unknown (exploration of alternatives to the uniform label component $\frac{\alpha}{K}$ in Eq. 8 is orthogonal to our study). In line with the widely accepted heuristic of Occam's Razor, i.e., Entities should not be multiplied unnecessarily, we assert that MaxSup is an elegant solution.

Please also refer to our follow-up discussion with Reviewer 59zm for the comparison to other alternatives.

W2: The targets created by the approach are not a probability, and hence, using it with cross-entropy loss may not be correct. Rv: "Guaranteed to be compatible" is not the right approach. It also needs to make sense mathematically, which states that targets must be probability vectors.

A2: We apologize for the lack of clarity in our previous answer, and we are glad to provide a more detailed explanation why MaxSup is mathematically guaranteed to be compatible with Cross-Entorpy Loss:

Let $\mathbf{s'}$ denote the soft label created by MaxSup, and $\mathbf{q}$ denote the predicted probability vector, and $\mathbf{y}$ denote the one-hot ground-truth label. It can be easily proven (similar to our proof of Lemma 3.2) that the Cross-Entropy Loss between $\mathbf{s'}$ and $\mathbf{q}$ can be decomposed as follows:

where the Max Suppression Loss $L_{*MaxSup*}$ is given by:

(sorry that we had problem to show \mathbb correctly at openreview, thus we omit it in our equation)

It can be seen that training with the soft targets created by MaxSup is mathematically equivalent to training with a weighted combination of three individual objectives, i.e., minimizing the Cross-Entropy with one-hot Label (vanilla loss) $\mathbf{y}$, where $y_k = 1_{k=*gt*}$ ($y_k = 1$ if class $k$ is the ground-truth class; otherwise $y_k = 0$), minimizing the Cross-Entropy with uniform Label (the effective component of Label Smoothing loss is kept), and maximizing the Cross-Entropy with one-hot Label (the problematic component of Label Smoothing loss is replaced) $1_{k=*Argmax*(\mathbf{q})}$ ($y_k = 1$ if class $k$ is top-1 prediction; otherwise $y_k = 0$). It is noteworthy that the minus and plus signs only affect the direction of the optimization, i.e., minimization or maximization, thus the individual Cross-Entropies and their sum are always valid.

In addition, our paper shows that Max Suppression Loss $L_{*MaxSup*}$ is mathematically equivalent to the logit-level form in Eq. 7, and the full training objective can be written as:

it can be also observed from the above equation that the compatibility issue doesn't arise in our approach, since the validity of the vanilla Cross-Entropy is not affected.

W5: Label Smoothing is known to affect semantic class relationships [1]. The baseline approaches (OLS [2], Zipf [3]) also formulate their loss function to mitigate this, but it is not handled in the proposed approach. Rv: The explanation still does not talk about/clarify the impact on class relationships.

A5: According to [1] (Section 2) and [6] (Section 3), the semantic class relationships are exactly characterized by inter-class separation / intra-class variation of the learned representations in the penultimate layer. Although the referred literature [1] is not explicitly given by the reviewer, it should refer to the paper "When does label smoothing help?" by Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton, because it revealed that "label smoothing encourages the representations of training examples from the same class to group in tight clusters. This results in loss of information in the logits about resemblances between instances of different classes".

If the reviewer did refer to the above issue, then it is both qualitatively and quantitatively validated that MaxSup alleviates such an issue, while OLS and Zipf-LS have not been demonstrated to mitigate it. We would like to provide a more explicit explanation:

The issue that "the representations of training examples from the same class to group in tight clusters" essentially means that Label Smoothing leads to small intra-class variation, which is qualitatively observed via visualization of the penultimate layer representations in [1]. In our visualization (We have improved the clarity of Figure 2, which is put in the appendix as Figure 3 in our revised version), MaxSup is shown to result in much larger intra-class variation (more scattered clusters of each class) compared to Label Smoothing, while maintaining larger distances between different classes (large inter-class separation benefits classification). Moreover, we have also demonstrated the improvement over Label Smoothing through quantitative evaluation. As suggested by Reviewer 2YeD, such quantitative metrics are also used for evaluating the learned semantic class relationships, but are more convincing than the illustrative plots in [1].

W6: In Tables 4 and 5, comparison is only done with the vanilla label smoothing. The improvement is minimal. It would be interesting to see the results of baseline approaches on them. Rv: I agree it is an improvement over vanilla baselines but that does not mean it will be better than other alternatives. The baselines must be compared as well.

A6: We thank the reviewer for the constructive suggestion, and are glad to provide the results of other alternatives for the machine translation and semantic segmentation tasks as suggested. We promise to include the results in the camera-ready version.

Table 1: Comparison of Label Smoothing and MaxSup on IWSLT 2014 German to English Dataset.

Table 2: Comparison of Label Smoothing and MaxSup on ADE20K Validation Set.

For the machine translation task, the improvements of Label Smoothing and all the alternatives are relatively small. As we have analyzed in our paper, this probably stems from the constraints of this particular task, e.g., limited dataset size. Thus the comparison on this particular task might be less informative compared to other tasks in our paper.

For the semantic segmentation task, the improvements of OLS and Zipf-LS over the vanilla baseline are limited compared to the improvement of MaxSup, indicating the benefit of the improved feature representation with MaxSup for downstream tasks.

Q1: In the ablation (Table 1), The paper showed removing the Error-Enhancement is beneficial. Why is that not compared/proposed as inplace for replacement? Rv: I understand the improvements. But we need to compare the results on a common setup. Either the results without Error-Enhancement should be included for the results datasets (preferred way) or performance in this setup with the proposed approach.

A1: We apologize for the lack of clarity in our previous answer. The comparison between the results without Error-Enhancement ($75.98\%$) and our MaxSup ($76.12\%$) in Table 1 in our revised manuscript is under the same setup, i.e., Deit-Small model without CutMix and Mixup. And the results for training Deit-Small (with CutMix and Mixup) on ImageNet1k without Error-Enhancement and with MaxSup are 79.96±0.13 and 80.16±0.09, respectively.

Q3: How does it impact the inter-class relationship? Rv: The explanation still does not talk about/clarify the impact on class relationships.

A3: According to [1] (Section 2) and [6] (Section 3),, the intra-class variation and inter-class separability of the learned representations reflect the impact on inter-class relationships w.r.t. the relationship between individual instances of a class to other classes and the relationship between the averaged features of each class, respectively:

• The inter-class separation, i.e., the distances between averaged class features, is further increased compared to Label Smoothing. As analyzed in [1], this indicates improved classification performance.

• The intra-class variation, i.e., "the information about the resemblances between instances of different classes" ([1]), is increased compared to Label Smoothing. According to [1], the heavily reduced intra-class variation, or the "overly tight cluster", "results in loss of information in the logits about resemblances between instances of different classes". Therefore, the increased intra-class variation with MaxSup indicates that such information are better retained, which benefits knowledge distillation [1] and transfer learning [6].

By examining the intra-class variation and inter-class separability both qualitatively and quantitatively (please refer to our answer to W3), MaxSup is demonstrated to have a positive impact on the inter-class relationship, w.r.t. both the relationship between individual instances of a class to other classes and the relationship between the averaged features of each class.

W1: Why replacing $z_\text{gt}$ with $z_\text{max}$ is the optimal choice? Rv: I see the benefits of using $z_\text{max}$ on performance and understand the intuition behind why the method chose $z_\text{max}$ but having a theoretical reasoning would back it up better. I think a better replacement could be explored or reasoning must be provided as to why is the optimal choice.
rCT5: I agree with 59zm. Without any theoretical proof/good logic, the proposed approach feels like a hack.

W2: The targets created by the approach are not a probability, and hence, using it with cross-entropy loss may not be correct. Rv: "Guaranteed to be compatible" is not the right approach. It also needs to make sense mathematically, which states that targets must be probability vectors.
rCT5: Thank you for clarifying this.

W5: Label Smoothing is known to affect semantic class relationships [1]. The baseline approaches (OLS [2], Zipf [3]) also formulate their loss function to mitigate this, but it is not handled in the proposed approach. Rv: The explanation still does not talk about/clarify the impact on class relationships.
rCT5: Yes, in the paper "When does label smoothing help?", the authors showed how the label smoothing destroys inter-class relationships. OLS discusses this in their paper they use a non-uniform vector to mitigate the negative effects. No such discussion /impact/analysis has been done in the paper.

W6: In Tables 4 and 5, comparison is only done with the vanilla label smoothing. The improvement is minimal. It would be interesting to see the results of baseline approaches on them. Rv: I agree it is an improvement over vanilla baselines but that does not mean it will be better than other alternatives. The baselines must be compared as well.
rCT5: Thank you for the results.

Q1: In the ablation (Table 1), The paper showed removing the Error-Enhancement is beneficial. Why is that not compared/proposed as inplace for replacement? Rv: I understand the improvements. But we need to compare the results on a common setup. Either the results without Error-Enhancement should be included for the results datasets (preferred way) or performance in this setup with the proposed approach.
rCT5: Thank you for the results. These numbers (without Error-Enhancement and with MaxSup are 79.96±0.13 and 80.16±0.09) are too close which makes me wonder if removing Error-Enhancement is a simpler/better solution as it achieves similar results and has a better reasoning.