$\epsilon$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise

Thanks very much for your valuable comments. We would like to offer the following responses to your concerns.

1. Response to Weakness 1 and Question 3

Thanks for your kind comment.

Previous work indicates that, for a fixed vector $\mathbf{v}$ and $\forall L \in \mathcal{L}$, we have $\sum _{k=1}^K L(\mathbf{u}, k) = C, \forall \mathbf{u} \in \mathcal{P} _{\mathbf{v}},$

where $k \in [K]$ denotes the label corresponding to each class, $C$ is a constant. This lemma suggests that when the network output $\mathbf{u}$ is restricted to a permutation set $\mathcal{P}_{\mathbf{v}}$ of a fixed vector $\mathbf{v}$, any loss function will satisfy the symmetric condition.

In this paper, we consider the fixed vector as a one-hot vector $\mathbf{e} _1$. The key challenge is that directly mapping outputs to a permutation set $\mathcal{P} _{\mathbf{v}}$ is a non-differentiable operation. Based on these motivations, we propose a simple yet effective scheme, $\epsilon$-softmax, to approximate one-hot vectors $\mathcal{P} _{\mathbf{e}_1}$. Thus, any loss function using $\epsilon$-softmax can mitigate label noise with a theoretical guarantee. We will include this detailed explanation in the revised version.

2. Response to Weakness 2 and Question 5

Thanks for your feedback. We would like to emphasize that our experiments include dozens of baselines and follow the same setting with previous works. On difficult noise and real world noise, our method achieves remarkable results. These comprehensive evaluations aims to provide a robust assessment of our method.

For Table 2, as suggested by other reviewer, we search more carefully about hyperparameters and obtain better performance. More details are available in the global rebuttal (see response to Q2). The new results are provided as follows, where "S" is symmetric noise, "AS" is asymmetric noise, and * denotes using better learning rate and weight decay.

As can be seen, we significantly outperform the SOTA approaches.

For Table 4, we used the t-test to determine if our CE$\epsilon$+MAE (Semi) statistically significantly exceeds the SOTA approaches. The p-values are as follows:

As can be seen, in all cases, our approach statistically significantly exceeds the previous sota approaches (p-value $<$ 0.05). This additional analyses will be included in the revised manuscript.

2. Response to Question 1

A loss function $L$ satisfies the symmetric condition is defined as (Eq. 1.1 in our Introduction):

where $k \in [K]$ denotes the label corresponding to each class, $C$ is a constant, and $\mathcal{H}$ is the hypothesis class. We will include this explanation in the abstract of the revised version to clarify this detail.

3. Response to Question 2

Thanks for your comment. $\mathbf{p(\cdot|x)}$ denotes the prediction probability vector for the sample $\mathbf{x}$. $p_k$ denotes the prediction probability value for the $k$-th class, i.e., the $k$-th element in the vector $\mathbf{p(\cdot|x)}$. We have followed common definitions of mathematical notations in academic paper, i.e., vectors are typically represented in boldface while scalars are usually represented in italicized font.

4. Response to Question 4

Thanks for your suggestions.

--- Guidance on Choosing $m$

An effective guideline is to use a larger $m$ for symmetric noise and a smaller $m$ for asymmetric noise. A larger $m$ imposes tighter symmetric constraints, which enhances robustness. However, if $m$ is too large, optimization on asymmetric noise can become challenging. For instance, on CIFAR-10, the symmetric MAE and CE achieve 45.36% and 18.95% accuracy under 0.8 symmetric noise, respectively. Conversely, MAE becomes difficult to optimize under asymmetric noise, resulting in 55.88% and 75.28% accuracy under 0.4 asymmetric noise.

--- More experiments on different $m$

We conducted further experiments on CIFAR-10 with various values of $m$. The results are summarized below. "S" is symmetric noise, and "AS" is asymmetric noise. We use m=1e5 for symmetric noise and 1e3 for asymmetric noise in the paper.

As shown above, using different values of $m$ can sometimes yield better performance, while the differences are not substantial.

Thanks very much for your positive comments. We would like to offer the following responses to your concerns.

1. Response to Weakness 1

Thanks for you kind comment.

--- About ablation study on gradient clipping

We fully followed the experimental setup of previous work [1], and we used gradient clipping because it was used in [1]. This PyTorch component stabilizes training without impacting the method comparison. We report the results without using gradient clipping as follows, where "S" is symmetric noise and "AS" is asymmetric noise.

As can be seen, whether gradient clipping is used or not does not affect the comparison of methods, and our method still shows excellent performance.

--- About different backbone

We conduct more experiments using VGG-13. The training settings are listed as follows: batch size 256, learning rate 0.1 with cosine annealing, weight decay 5e-4, and 200 epochs. The results with 3 random trials are as follows:

As can be seen, our approach still has outstanding performance.

2. Response to Weakness 2

Thanks for your kind comment. Our method, like most robust loss functions, requires different parameters for different datasets. Many approachs have been proposed to find optimal parameters quickly and economically. One efficient approach is to random select a subset of the training set as a smaller training set, for instance, 1/5 of the original set. We can then train on this smaller set to search for parameters efficiently.

3. Response to Question 1

Thanks for your insightful question. We chose MAE because it is the most classic symmetric loss, and there is no other reason. We recommend using any symmetric loss in combination with CE$_\epsilon$ rather than non-symmetric loss, because combining with symmetric loss does not increase the excess risk bound, as demonstrated by Lemma 2.

4. Response to Question 2

Thanks for your insightful question. Logit adjustment [2] modifies logits to encourage a large margin between rare and dominant labels for long-tail learning. However, logit adjustment requires prior knowledge of label distributions, which is not available for noisy labels. Therefore, it is not suitable for learning with noisy labels.

$\epsilon$-Softmax mitigates overfitting to noisy labels by adjusting the probabilities. We find that $\epsilon$-softmax is also effective for long-tail learning, where we can use larger $m$ for dominant labels. Specifically, we set $m_k$ for the k-th class as $m \cdot p_k^{label}$, where $p_k^{label}$ is the frequency of the k-th class labels among all labels. We use ResNet-32 for CIFAR-10-Lt and CIFAR-100-LT, following the setting in [2] with long-tail imbalance ratio 100. The average class accuracies with 3 random trials are as follows:

Thanks again for your kind comment, which has inspired us to explore the application of $\epsilon$-softmax in other areas. We are pleased to see the flexibility of our plug-and-play method.

[1] Asymmetric loss functions for noise-tolerant learning: Theory and applications, TPAMI, 2023.

[2] Long-tail learning via logit adjustment, ICLR, 2021.

Thanks very much for your positive comments. We would like to offer the following responses to your concerns.

1. Response to Weakness 1

Thanks for your insightful comment. In the following, we give the theoretical discussion for temperature-dependent softmax.

For model with temperature-dependent softmax, i.e., $f(\mathbf{x})= \tau$-softmax$\circ h(\mathbf x)$, we have: $\min _{\mathbf u \in \mathcal{P} _{\mathbf e_1}} \\|f(\mathbf x)-\mathbf u \\|_2=\sqrt{1-2p _{t}+\sum _{k=1}^K p_k^2} = \sqrt{1-p _{t}-\sum _{k=1}^K p_k(p_t-p_k)} \le \sqrt{1-p_t}\le \sqrt{1-1/K},$ where $\mathbf p(\cdot|\mathbf x) = f(\mathbf{x})$ and $t=\arg\max _{k\in[K]}p _k$. The equality holds when each term of $h(\mathbf x)$ is equal, no matter what value the temperature $\tau$ takes, we have $p_k = 1/K, \forall k$.

For $\epsilon$-softmax, we have $\min_{\mathbf u\in\mathcal{P}_{\mathbf e_1}}\\|f(\mathbf x)-\mathbf u\\|_2\le \epsilon = \tfrac{\sqrt{1 - 1/K}}{m+1}$ (Lemma 1). Our $\epsilon$-softmax make every output approximating one-hot vectors with a smaller error $\epsilon$. Thus, a smaller excess risk bound can be derived.

2. Response to Weakness 2 and Question 2

Thanks for your kind suggestion. Sparse regularization (SR) is a regularization method for probabilities, independent of labels and distinct from a loss function. We tend to compare with robust loss functions in this paper. Following the suggestion, we compared our method with SR. The results for 0.8 symmetric and real-world noise are provided.

As can be seen, our approach significantly outperforms SR in difficult and real-world situations. In addition, SR requires dynamic adjustment of hyperparameters at each epoch, making it difficult to train.

3. Response to Weakness 3

Thanks for your kind suggestion.

--- About ablation study on $\alpha$ and $\beta$.

We offer the ablation experiments. For simplicity, we fix $\alpha$ and then adjust $\beta$, $\beta=5$ for CIFAR-10 and 1 for CIFAR-100 is used in the paper. "S" is symmetric noise, and "AS" is asymmetric noise.

--- About the better trade-off

The better trade-off means that we achieve better performance on both fitting ability and robustness compared to previous works. Previous works like GCE and SCE increase fitting ability but reduce robustness due to the CE term. In contrast, our combination retains robustness, as demonstrated by Lemma 2, and also improves fitting ability. To verify this, we present comparison results on CIFAR-10 symmetric noise:

As can be seen, CE$_\epsilon$+MAE achieves better results on both the clean and noisy cases.

4. Response to Weakness 4

Thanks for your comment. The results for instance-dependent noise are available in the global rebuttal (please see response to Q1). Furthermore, we provide the excess risk bound for instance-dependent noise: $\mathcal{R} _L(f _\eta^*)\le 2\delta + \frac{2c\delta}{a},$

where $c = \mathbb{E} _\mathcal D\left(1-\eta _{\mathbf{x}}\right)$, $a=\min _{\mathbf x,k}(1-\eta _y-\eta _{\mathbf x,k})$, $f^* _\eta$ and $f^*$ denote the global minimum of $\mathcal R_L^\eta(f)$ and $\mathcal R_L(f)$, respectively.

The proof is similar to that for asymmetric noise. We will add the proof in the revised version, as there is not enough space.

5. Response to Question 1

Thanks for your kind comment. The answer is yes. We rename the $\alpha, \beta$ for CE$_\epsilon$+MAE loss as $1-\lambda$ and $\lambda$ for a better presentation, where $0 < 1 - \lambda \le 1$.

For $L=$ CE$_ \epsilon$+MAE, label is $\mathbf q$, and $f(\mathbf x)=\epsilon$-softmax$\circ h(\mathbf x)$, we have $L = \sum _{k=1}^K q _k(-(1-\lambda))\log f(\mathbf x) _k + \lambda(1-f(\mathbf x) _k)).$ By Lagrangian multiplier, we have

$\min _{f(\mathbf x)} \max _{\alpha, \beta _k \mathcal \geq 0} \sum _{k=1}^K q _k \left( -(1-\lambda) \log f(\mathbf x) _k + \lambda (1-f(\mathbf x) _k) \right) + \alpha \left( \sum _{k=1}^K f(\mathbf x) _k - 1 \right) - \sum _{k=1}^K \beta_k f(\mathbf x)_k$

Consider the stationary condition for $f(\mathbf x)$, we have $\frac{1}{f(\mathbf x)_k^*} + \frac{\lambda}{1-\lambda} = \frac{\alpha - \beta_k}{q_k(1-\lambda)}, 0<f(\mathbf x)_k <1$. By complementary slackness, we can get $\beta_k^* =0$ and $\alpha >0$. Hence, we have $\left( \frac{1}{f(\mathbf x)_k^*} + \frac{\lambda}{1-\lambda} \right) \propto \frac{1}{q_k}$ and consequently $f(\mathbf x)^*$ is rank consistent with $\mathbf q$, i.e., $L$ is All-$k$ calibrated. Since $L$ is non-negative, so $L$ is All-$k$ consistency.

Thanks very much for your positive comments. We would like to offer the following responses to your concerns.

1. Response to Weakness

Thanks for your kind comment. For Figures 2 and 3, we extract the high-dimensional features of the test set at the second-to-last fully connected layer, then project them into 2D embeddings using t-SNE. The x-label and y-label represent the first and second dimensions of the 2D embeddings, respectively. We will add the x-label and y-label in Figures 2 and 3 in the revised version.

2. Response to Question 1

Thanks for your kind comment. For regression tasks, the goal is to predict a continuous numerical output rather than a categorical probability. Therefore, the softmax function and our $\epsilon$-softmax are not suitable for regression tasks.

3. Response to Question 2

Thanks for your insightful comment. We evaluate the training time of different loss functions on CIFAR100 using ResNet34, and record the average training time of an epoch on NVIDIA RTX 4090. The results are summarized in the table below.

As can be seen, the training time required for all methods is almost the same.

4. Response to Question 3

Thanks for your insightful comment. We use random masking, Gaussian blur and solarisation to synthesize symmetric feature noise. The experimental results with 3 random trials are as follows:

We believe that the feature noise primarily tests the fitting ability of the loss function, and we can see that our CE$_\epsilon$+MAE also performs very well for feature noise.

5. Response to Question 4

Thanks for your kind comment. In the area of learning with noisy labels, it is commonly assumed that the training labels are noisy and the test labels are clean, which aligns with real-world applications of machine learning models. Having clean training labels and noisy test labels doesn't make sense in practical scenarios, and we don't recommend considering this scenario.

Thanks very much for your positive comments. We would like to offer the following responses to your concerns.

1. Response to Weakness 1

Thanks for your insightful comments.

--- About the better trade-off

The better trade-off means that we achieve better performance on both fitting ability and robustness compared to previous works. Previous works like GCE and SCE [1] increase fitting ability but reduce robustness due to the CE term. In contrast, our combination retains robustness, as demonstrated by Lemma 2, and also improves fitting ability. To verify this, we present comparison results on CIFAR-10 symmetric noise:

As can be seen, CE$_\epsilon$+MAE achieves better results on both the clean and noisy cases.

--- About issue (1)

We offer the ablation experiments for $\alpha$ and $\beta$. For simplicity, we fix $\alpha$ and then adjust $\beta$. $\beta=5$ for CIFAR-10 and 1 for CIFAR-100 is used in the paper. "S" is symmetric noise, and "AS" is asymmetric noise.

--- About issue (2)

Our experimental setup precisely follows that of previous work [2], ensuring a completely fair comparison. The results we reproduced are consistent with [2]. We note that the key difference between our results and those in SCE paper [1] lies in the recording of epoch accuracy. [1] recorded the best epoch accuracy, while we recorded the last epoch accuracy. Consequently, [1] avoided overfitting noisy labels in the later stage of training and reported better results. For example, for CIFAR-10 with 0.8 symmetric noise, CE achieved about 39% accuracy at the best epoch and about 18% at the last epoch.

2. Response to Weakness 2

Thanks for your kind comment. Ours, and most robust loss functions, are indeed somewhat sensitive to hyperparameters. For instance, NCE+AGCE [2] involves four hyperparameters.

As suggested and in line with [3], we provide a more extensive hyperparameter search for learning rate and weight decay. The results are available in the global rebuttal (please see response to Q2). As can be seen, our method achieves better results in many cases.

3. Response to Question 1

Thanks for your insightful comment. Due to the excellent fitting ability of the neural network, even if $m$ is about $1e4$, the gradient will still play a guiding role in optimization. According to your suggestion, we conducted experiments on sufficiently large and infinity $m$. The results on CIFAR-100 symmetric noise are as follows:

As can be seen, when $m$ is sufficiently large, there is no obvious difference between the results.

4. Response to Question 2

Thanks for your kind comment. As suggested, the results of CE$_\epsilon$ are offered as follows, where "S" is symmetric noise and "AS" is asymmetric noise:

[1] Symmetric cross entropy for robust learning with noisy labels, CVPR, 2019.

[2] Asymmetric loss functions for noise-tolerant learning: Theory and applications, TPAMI, 2023.

[3] Generalized jensen-shannon divergence loss for learning with noisy labels, NeurIPS, 2021.