Variation-Bounded Losses for Learning with Noisy Labels

Thank you very much for your positive comments. We would like to offer the following responses to your concerns.

Response to Weakness 1

Thank you for your valuable comment. We provide additional ablation experiments about NCE+VCE on CIFAR-100. The results over the last sota method are bolded.

As can be seen, for fixed hyperparameters ($\alpha$=5, $\beta$=1, a=0.4), we can also significantly outperform the last sota method in all scenarios.

Response to Weakness 2

Thank you for your valuable comment.

--- About hyperparameters

Our VBL have a more concise form than the existing asymmetric loss functions. The existing asymmetric loss functions [1], AUL, AGCE, and AEL, all have two hyperparameters. Conversely, our VCE, VEL and VMSE have only one hyperparameter.

--- About underfitting issue

Our VBL have a more concise form, making them easier to adjust the hyperparameters and optimize. We provide more experimental demonstrations. "S" denotes symmetric noise, and "A" denotes asymmetric noise. The top-3 best results are bolded.

As can be seen, our VBL significantly exceed the existing asymmetric losses, including the clean case. Since asymmetric losses and our VBL are theoretically proven to be noise-tolerant, the performance difference depends mainly on the fitting ability. The experimental results clearly demonstrate that our methods have a better fitting ability.

We will clarify the advantages of our VBL over existing asymmetric loss functions more explicitly in the manuscript. Thank you again for your constructive feedback.

Response to Question 1

Thank you for your kind comment.

--- About hyperparameters for Table 5

Please see Appendix B.2, Lines 797 and 805.

--- About choosing hyperparameters

For VBL, we can easily calculate a rough range based on Theorem 3, an example can be seen in Lines 242-244, then we will search hyperparameters based on empirical testing. For NCE+VBL, in most cases, we fix $\alpha=5, \beta=1$, and then search for $a$. In rare cases where $\alpha=5$ does not work very well, we also search for $\alpha$. Since $\beta$ is always fixed at 1, we actually have only two parameters to adjust. We will add these instructions to the manuscript for clarity.

[1] Asymmetric Loss Functions for Learning with Noisy Labels, ICML 2021.

Dear Reviewer qKQr,

We sincerely appreciate your valuable time and insightful comments on our work. As the deadline for the Author-Reviewer discussion draws near, we kindly request your feedback on whether our responses have effectively addressed your concerns. Thank you once again for your contributions, and we hope you have a wonderful day!

Best regards,

The Authors

Thank you very much for your positive comments. We would like to offer the following responses to your concerns.

Response to Weakness

Thank you for your valuable comment. In the following, we provide a detailed explanation about the combination of VBL with NCE.

--- Why is the combination beneficial

The combination of two different robust loss functions, such as VBL and NCE, can mutually enhance the optimization processes of each other, thus improving the overall fitting ability of the model. Previous works [1, 2, 3] have experimentally demonstrated that combining NCE often results in improved performance.

--- Why use the combination for CIFAR-100

In fact, NCE+VBL can achieve good performance on any dataset, not just CIFAR-100. To verify this, we conduct more experiments about NCE+VCE on other datasets, where "S" denotes symmetric noise and "A" denotes asymmetric noise. The results over the last sota method are bolded.

As can be seen, NCE+VCE also significantly outperforms the last sota method on other datasets. Moreover, VCE alone often surpasses the sota method in most scenarios other than CIFAR-100. Therefore, for datasets other than CIFAR-100, we opt to use VBL alone to better highlight the superiority of our method, even though it puts us at a disadvantage compared to methods that combine NCE.

Since CIFAR-100 is a more challenging dataset, the improvement from incorporating NCE is more pronounced. If we do not combine with NCE, we cannot completely exceed the previous sota methods that also utilize NCE. Therefore, we opt to combine NCE with VBL on CIFAR-100 to surpass them.

We will incorporate these more detailed interpretations into the manuscript to enhance clarity and completeness. Thank you once again for your valuable suggestions and for helping us improve our work.

Response to Question

Thank you very much for your insightful comment. We provide experiments about more epochs on CIFAR-10 with 0.8 symmetric noise. The best results are bolded.

The results demonstrate two points. (1) As $f_\eta$ converges to the global minimum of $R^\eta_{L}(f)$, GCE basically loses robustness. Conversely, VCE still retains excellent noise-tolerance. This is consistent with our theory. (2) Although VCE (a=10) did not exceed the best result of VCE (a=4, 120 epochs), VCE (a=10) achieved a better performance than VCE (a=4) in the case of both training 400 epochs. This shows that a smaller variation ratio provides better robustness, which is consistent with our theory.

Thank you again for your insightful comment, which has enriched our work.

[1] Normalized Loss Functions for Deep Learning with Noisy Labels, ICML 2020.

[2] Asymmetric Loss Functions for Learning with Noisy Labels, ICML 2021.

[3] Active Negative Loss Functions for Learning with Noisy Labels, NeurIPS 2023.

Dear Reviewer DhVJ,

We sincerely appreciate your valuable time and insightful comments on our work. As the deadline for the Author-Reviewer discussion draws near, we kindly request your feedback on whether our responses have effectively addressed your concerns. Thank you once again for your contributions, and we hope you have a wonderful day!

Best regards,

The Authors

Thank you very much for your valuable comments. We would like to offer the following responses to your concerns.

Response to Weakness 1 and Weakness 2

--- Comparison to symmetric loss functions

In the following, we explain why VBL address the slow convergence of symmetric loss functions

1. From the perspective of symmetric conditions, our method is actually a relaxation of the symmetric condition, which has a looser constraint than the symmetric losses. Our VBL, with bounded variation ratio, ensure sufficient and controlled gradient variation compared to symmetric losses like MAE. This enables our method to maintain robustness while significantly improving convergence speed.

2. From the perspective of the asymmetric condition, we clarify in Lines 241-248.

3. From the experimental perspective, we show symmetric losses in Table 1, such as MAE and NCE. As can be seen, our methods are significantly superior to these.

--- Comparison to the hybrid between CE and MAE

These hybrid loss functions, often result in reduced robustness due to their reliance on the CE component, as their variation ratio is $\infty$. In contrast, our variation-bounded losses achieve complete noise tolerance while maintaining good fitting ability. We discuss these points in the paper, such as Lines 315–323.

In addition, further response to your concern about the hybrid loss GCE can be found in the "Response to Weakness 3."

--- Comparison to existing asymmetric loss functions:

1. Simplifying the complexity: Our VBL have a more concise form than the existing asymmetric loss functions. The existing asymmetric loss functions [1], AUL, AGCE, and AEL, all have two hyperparameters. Conversely, our VCE, VEL and VMSE have only one hyperparameter.

2. Better fitting ability: Our VBL have a more concise form, making them easier to adjust the hyperparameters and optimize. We provide more experimental demonstrations. "S" denotes symmetric noise, and "A" denotes asymmetric noise. The top-3 best results are bolded.

As can be seen, our VBL significantly exceed the existing asymmetric losses, including the clean case. Since asymmetric losses and our VBL are theoretically proven to be noise-tolerant, the performance difference depends mainly on the fitting ability. The experimental results clearly demonstrate that our methods have a better fitting ability.

We will clarify the advantages of our VBL over other robust loss functions more explicitly in the manuscript. Thank you again for your constructive feedback.

Response to Weakness 3

Thank you for your valuable comment. Our theory measures the global minimization of the expected risk $R^\eta_{L}(f)$, i.e., the final convergence state of the model. Neural networks tend to fit clean labels at the early phases of training [2], hence the performance at the early phases of training cannot prove the noise-tolerance for the convergent state. We provide experiments about more epochs on CIFAR-10 with 0.8 symmetric noise. The best results are bolded.

As shown, with the increase in epochs, as $f_\eta$ converges to the global minimum of $R^\eta_{L}(f)$, GCE basically loses robustness. Conversely, VCE still retains excellent noise-tolerance. This is consistent with our theory.

Dear Reviewer PV3A,

We sincerely appreciate your valuable time and insightful comments on our work. As the deadline for the Author-Reviewer discussion draws near, we kindly request your feedback on whether our responses have effectively addressed your concerns. Thank you once again for your contributions, and we hope you have a wonderful day!

Best regards,

The Authors

Thank you very much for your valuable comments. We would like to offer the following responses to your concerns.

Response to Weakness 1 and Question 1

Thank you very much for highlighting this ambiguity in our definition. In both Lemma 1 and Theorem 3, we have explicitly stated that we only consider cases where $\ell_{active}$ is monotonically decreasing. To clarify, we will also incorporate this condition into Definition 2. Thank you once again for your valuable input.

Response to Weakness 2 and Question 2

Thank you for your meaningful comment.

--- About fixing the hyperparameters across different noise scenarios

We appreciate your valuable suggestion. As suggested, we provide additional experiments about NCE+VCE with fixed same hyperparameters across different noise scenarios. The results over the last sota method are bolded.

As can be seen, for fixed hyperparameters ($\alpha=5$, $\beta=1$, $a=0.4$), we can also significantly outperform the last sota method in all scenarios.

--- About the range of hyperparameters selection

For NCE+VBL, in most cases, we fix $\alpha=5, \beta=1$, and then search for $a$. In rare cases where $\alpha=5$ does not work very well, we also search for $\alpha$. Since $\beta$ is always fixed at 1, we actually have only two parameters to adjust, consistent with previous works [1, 2]. We will add these instructions to the manuscript for clarity.

[1] Normalized Loss Functions for Deep Learning with Noisy Labels, ICML 2020.

[2] Active Negative Loss Functions for Learning with Noisy Labels, NeurIPS 2023.

Dear Reviewer kQBp,

We sincerely appreciate your valuable time and insightful comments on our work. As the deadline for the Author-Reviewer discussion draws near, we kindly request your feedback on whether our responses have effectively addressed your concerns. Thank you once again for your contributions, and we hope you have a wonderful day!

Best regards,

The Authors

Dear Reviewer kQBp,

Thank you for your feedback. We would like to offer the following responses to your concerns.

Response to Comment 1: Thank you for your kind comment. In fact, there is no ambiguity here because, in our definition, $L$ belongs to $\mathcal{L}$, which is defined in Lines 99-101. The definition of $\mathcal{L}$ shows that the loss function $L$ is formulated as $L(\mathrm {u},\mathrm{e_y})=\ell(u_y,1)+\sum_{k\neq y} \ell(u_k,0)$. Therefore, the terms $\ell_{active}(u_y)$ and $\ell_{passive}(u_k)$ share the same basis loss function, ensuring the decomposition is unique.

Response to Comment 2: Thank you for your thoughtful suggestion. For CIFAR-10, the parameter $\beta$ denotes a scalar parameter, and we adjust this scalar parameter and $a$. Therefore, for CIFAR-10, we also only need to tune two parameters.

For your solution (1), we will include experimental results with the fixed scalar parameter of 1 for VBL alone in the final version.

For your solution (2), we acknowledge that, regardless of the specific experimental setting, our NCE+VBL involve three hyperparameters. This is more concise than the previous work [1], but they introduce one additional hyperparameter compared to previous works [2, 3].

[1] Asymmetric Loss Functions for Learning with Noisy Labels, ICML 2021.

[2] Normalized Loss Functions for Deep Learning with Noisy Labels, ICML 2020

[3] Active Negative Loss Functions for Learning with Noisy Labels, NeurIPS 2023.

We hope this message finds you well. The discussion period is coming to an end and we look forward to your feedback.

Best regards,

The Authors

I thank you for your efforts in answering my questions. I still have the following concerns:

1.It seems that adding the assumption that $l_{active}$ is monotonically decreasing does not solve the problem in Definition 2. The problem comes from the fact that the active/passive decomposition is not unique. Consider the exponential loss $L=e^{-u_y}$. If we choose $l_{active}(u_y)=e^{-u_y}-u_y$ and $l_{passive}(u_k)=\frac{1}{K-1}-u_k$, we get $v(L)=\frac{max|-e^{-u_y}-1|}{min|-e^{-u_y}-1|}=\frac{max(e^{-u_y}+1)}{min(e^{-u_y}+1)}= \frac{2}{1+e^{-1}}$. However, this is different from $v(L)=e$ obtained from $l_{active}(u_y)=e^{-u_y}$ and $l_{passive}(u_k)=0$.

Considering only loss functions with $l_{passive}(u_k)$ set to $0$ would solve this problem (like in Lemma 1 and Theorem 3). However, the decomposition active/passive would then not be relevant to the method of the paper. If this solution is used, some modifications would be required in the text. For example, starting in line 296: ''To date, no studies have explored asymmetric loss functions that have both active and passive terms, because this is a more complex scenario. In this paper, we extend the asymmetric loss function to include both active and passive terms''.

2.Thank you for providing results showing that good performance can still be achieved on CIFAR100 without too much tuning. It is claimed in your response that ''Since $\beta$ is always fixed at 1, we actually have only two parameters to adjust''. This is true on the dataset CIFAR100. However, the parameter $\beta$ is different on CIFAR10 from what I can see in Table 6. This means that the parameter $\beta$ is actually being tuned. In my opinion, two reasonable solutions could be:

a) Fix $\beta$ to 1 for all experiments and then the claim ''we actually have only two parameters to adjust, consistent with previous works'' is correct.

b) Accepting that the method has more hyperparameters to tune and argue that the search of hyperparamters is still fair by, for example, comparing the size of the grid search among the different methods.

Thank you very much for your positive comments. We would like to offer the following responses to your concerns.

Response to Weakness 1 and Question 1

Thank you for your kind comment. We determine hyperparameters $\alpha$, $\beta$, $a$ for NCE+VBL based on empirical testing. Specifically, in most cases, we fix $\alpha=5, \beta=1$, then search for $a$. In rare cases where $\alpha=5$ does not work very well, we also search for $\alpha$. Since $\beta$ is always fixed at 1, there are actually only two parameters to adjust. We will add these instructions to the manuscript for clarity.

Response to Weakness 2 and Question 2

Thank you for your valuable comment.

--- Why is the combination beneficial

The combination of two different robust loss functions, such as VBL and NCE, can mutually enhance the optimization processes of each other, thus improving the overall fitting ability of the model. Previous works [1, 2, 3] have experimentally demonstrated that combining NCE often results in improved performance.

--- Why use the combination for CIFAR-100

In fact, NCE+VBL can achieve good performance on any dataset, not just CIFAR-100. To verify this, we conduct more experiments about NCE+VCE on other datasets, where "S" denotes symmetric noise and "A" denotes asymmetric noise. The results over the last sota method are bolded.

As can be seen, NCE+VCE also significantly outperforms the last sota method on other datasets. Moreover, VCE alone often surpasses the sota method in most scenarios other than CIFAR-100. Therefore, for datasets other than CIFAR-100, we opt to use VBL alone to better highlight the superiority of our method, even though it puts us at a disadvantage compared to methods that combine NCE.

Since CIFAR-100 is a more challenging dataset, the improvement from incorporating NCE is more pronounced. If we do not combine with NCE, we cannot completely exceed the previous sota methods that also utilize NCE. Therefore, we opt to combine NCE with VBL on CIFAR-100 to surpass them.

--- Theoretical guarantee for NCE+VBL

According to our definition, variation ratio is suitable for loss functions where $\ell_{active}$ only depends on $u_y$. Since the $\ell_{active}$ of NCE depends not only on $u_y$ but also on $u_{k\neq y}$, NCE is not a function considered by the variation ratio. However, we can still prove that NCE+VBL is noise-tolerant.

Previous work [2] proved that symmetric loss functions are completely asymmetric, and the combination of asymmetric loss functions remains asymmetric. Because NCE is symmetric (i.e., also asymmetric), and we have already proved that VBL is asymmetric. So NCE+VBL is still asymmetric and therefore noise-tolerant.

We will incorporate these more detailed interpretations into the manuscript to enhance clarity and completeness. Thank you once again for your valuable suggestions and for helping us improve our work.

[1] Normalized Loss Functions for Deep Learning with Noisy Labels, ICML 2020.

[2] Asymmetric Loss Functions for Learning with Noisy Labels, ICML 2021.

[3] Active Negative Loss Functions for Learning with Noisy Labels, NeurIPS 2023.