Early Stopping Against Label Noise Without Validation Data

We answer the questions one by one as follows. We are really open to further discussion.

Question.1 - Class imbalanced dataset We have tested the performance of our method on class imbalanced datasets, by setting the Imbalance Factor to 0.1 and the Noise Ratio (Sym.) to 0.4, with other settings consistent with those described in Section 4.1. Our experiments were conducted using our method in Cross-Entropy (CE) and class imbalanced method LDAM [1].

The experimental results affirm that our method can effectively identify early stopping points when learning from class imbalanced noisy datasets. While we emphasize the effectiveness of the Label Wave method in such scenarios, we acknowledge, as your intuition, that it may not perform equally well in more extreme cases. For example, when applied to the CIFAR-100 training set with an Imbalance Factor of 0.01 and a Sym. Noise Ratio of 0.4, the model's test accuracy exhibited significant fluctuations, leading to the ineffectiveness of the Label Wave method. We wish to clarify that our method is intended to identify appropriate early stopping points; thus, scenarios where early stopping itself is not applicable fall outside the scope of our research.

[1] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019.

Question.2 - Difference between the previous studies Our research, along with the findings of [1, 2], aims to deepen the understanding of how a model's performance is impacted by fitting incorrectly labeled examples. However, we believe that these studies contribute to this understanding in different ways.

Both [1] and [2] have introduced new methods to handle noisy labels based on their findings, and thus have made notable contributions to the field. Specifically, [1] discusses the effects of Label Smoothing (LS) and Negative Label Smoothing (NLS) in the context of label noise, mainly focusing on how to adjust label smoothing strategies. [2] tackles instance-dependent label noise, and concentrates on effectively sieving out wrongly labeled instances from the training data.

While [1] and [2] have significantly contributed to the field by exploring how noisy labels affect a model's generalization performance; our research distinctly points out how fitting noisy labels can impact the model's fitting performance on the training set. In Section 3.2, we delve deeper into the specific effects of fitting mislabeled examples on the overall fitting performance on the training set. Our study, by designing metrics called ''stability'' and ''variability'', discusses how the model's fitting performance on the training set is influenced by fitting incorrectly labeled examples as the training progresses. Moreover, we highlight a transitional phase, which we term ''learning confusion patterns''. We believe that this perspective has not been fully explored in previous studies.

[1] Wei, J., Liu, H., Liu, T., Niu, G., Sugiyama, M., & Liu, Y. (2022, June). To Smooth or Not? When Label Smoothing Meets Noisy Labels. In International Conference on Machine Learning.

[2] Cheng, H., Zhu, Z., Li, X., Gong, Y., Sun, X., & Liu, Y. (2020, October). Learning with Instance-Dependent Label Noise: A Sample Sieve Approach. In International Conference on Learning Representations.

Question.7 - Real noise We respectfully wish to highlight that our testing on the CIFAR-10N [1] dataset involved human-annotated real-world noise, showcasing the Label Wave method's effectiveness in real-world scenarios. Here, we present the outcomes of applying our label wave method on the Clothing1M dataset.

[1] Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., & Liu, Y. (2021, October). Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations.

Question.8 - Semantic segmentation Considering that our aim is to demonstrate that the concept of "learning confusion patterns" helps detect the transitional point in learning with noisy labels, we did not consider tasks like semantic segmentation and focused solely on classification tasks.

We extend our sincere thanks for your patience in reviewing our work and deeply appreciate your informative comments towards improving our manuscript.

Question 3 - PC Fluctuations & Sensitivity Analysis over k Here, we address both Weakness 3 and Question 3. Based on your constructive suggestions, we have conducted additional analysis.

1. Monotonicity of PC Values: You inquired about the fluctuations in PC values and suggested setting some thresholds. We fully endorse your suggestion that applying a specific threshold to label wave in practice would further enhance the robustness of our method. In our existing experiments, we have found that Moving Averages and "Patience" are sufficient to accurately help us identity the appropriate early stopping point through PC. Therefore, to ensure the simplicity and straightforward of our proposed method, we did not add an additional threshold in this paper.

2. Sensitivity Analysis of k Values: In response to your suggestion, we conducted a sensitivity analysis on the k value for Moving Averages. Specifically, we analyzed the correlation between the PC after applying different k values of moving averages and test accuracy:

(Experiment take the same settings as Sec. 4.1 and Appendix B)

We observed that as the k value changes, the Pearson correlation coefficient between the moving average of PC values and test accuracy exhibits some fluctuations but overall maintains a very strong negative correlation. This suggests that while considering the setting of k values is necessary, different k values do not significantly alter the relationship between the moving average of PC values and test accuracy. We will provide detailed sensitivity analysis charts in the further revision.

Question.4 - Additional regularizations Based on the reviewers' detailed comments, we tested the effectiveness of our Label Wave method with additional regularization methods. It is worth noting that our proposed Label Wave method itself includes BN and Data Augmentation, i.e., Label Wave + B, D as described below. Beyond the reviewer's comments, we have also tested the setting where these two regularization methods are not included. These regularization methods include:

A. Mixup B. BN C. Dropout D. Data Augmentation. (All experiments take the same settings as Sec. 4.1)

The experimental results show that adding or subtracting these regularization methods in the experimental setting does not affect the effectiveness of the label wave method for identifying early stopping points.

Question.5 - LNL algorithms Yes, Tables 1 and 2 evaluate the effectiveness of the label wave method when using Cross-Entropy Loss under various setting modifications. In Section 4.2, Tables 3 and 4 examine a range of algorithms for managing noisy labels. The differences between the maximum test accuracy and the label wave outcomes are shown below:

Question.6 - More noise condition for table 3 and 4 Due to computational resource constraints, we will provide results with more noise condition for Tables 3 and 4 in subsequent Comments and further revision.

Thank authors for their sincere efforts to relieve my worries. I think this research has some interesing ideas, but I still have concerns as follows:

• Since this strategy cannot be theoretically supported, many experimental results would be required. e.g. Which type of CIFAR10N the authors experimented? All 5 types? If so, results of all types should be reported. Although I agree the noisy label of CIFAR10N is realistic, since the image itself is 32$\times$32 CIFAR, it may be limited to represent more realistic situations.
• I also like the result reporting type as Table 3 and 4. yet, results with more diverse settings of noise condition will be more convincing.
• I think results with clean dataset will make this study more convincing.

Therefore, currently I will keep my score as before.

Thank the authors for their sincere efforts to address my concerns. With the inclusion of several additional experimental results, I am inclined to increase my score by +1 (as a result, 6). However, I would like to clarify that my acceptance of this paper is contingent upon the inclusion of the total experiments. As the current manuscript, without any modifications, falls below the acceptance threshold based on my previous scoring, I look forward to seeing the complete set of experiments in the revised version.

Response to Question 3 (Validation set) We really appreciate your wonderful suggestion about using a clean validation set to align with a clean test set distribution. However, in the context of our research on learning with noisy labels, we have intentionally chosen to maintain consistency between the distributions of the validation and training sets. This decision is grounded in the following considerations:

1. Universality of Label Wave: Our proposed method for identifying early stopping points using the Label Wave mthods aims to offer a universally applicable solution for learning with noisy labels. A prevalent assumption in this field, as reflected in numerous studies [7-10], is the presence of a training set with unreliable labels alongside a test set with reliable labels. In these cases, the training set is often divided into two parts: one for training and the other serving as a noisy validation set for early stopping. By adhering to this widely accepted setting, our method aligns with and can be directly compared with the method that utilizes a noisy validation set for identifying early stopping points.
2. Label Wave in Real-World Applications: In many real-world applications, it is common to encounter datasets where all available data is unreliable or noisy. This means that we cannot obtain an additional clean validation set for identifying early stopping points. Therefore, we only compare methods that utilize a noisy validation set for identifying early stopping points, as this reflects a more realistic and practical situation.

In conclusion, we use a noisy validation set to ensure a fair and relevant evaluation of our method within the specific context of learning with noisy labels.

[7] Early-learning regularization prevents memorization of noisy labels, NeurIPS 2020. [8] Robust Early-learning: Hindering the Memorization of Noisy Labels, ICLR 2021. [9] Mitigating Memorization of Noisy Labels by Clipping the Model Prediction, ICML 2023. [10] Late stopping: Avoiding confidently learning from mislabeled examples, ICCV2023.

Response to Question 5 (Training data size) Although our method targets domains with limited training data, its fundamental principle of using all available data effectively applies universally when the training and validation sets have similar distributions. In such cases, incorporating the validation data into the training set can always be positive for the model performance, regardless of training data size or validation data split ratio.

We really appreciate your feedback as it significantly contributes to refining our research. We are committed to investigating these aspects to extend the applicability of the Label Wave method.

We would like to thank the reviewer for your detailed and insightful review. We are glad that you find the paper is interesting and has insights. Below we address the questions you’ve raised about the Label Wave method, and we remain open to further discussion if there are any additional comments on these questions.

Response to Question 1 and Question 4 (Applicability of the Label Wave method)

We appreciate the insightful concerns you have expressed about the applicability of the Label Wave method, particularly in scenarios with different types and amounts of label noise. We will detailedly address these concerns below in Part A - Types of Label Noise, and Part B - Amounts of Label Noise.

Part A: Types of Label Noise

1. Broad Applicability Across Noise Types: Our method is effective not only in the presence of random label noise but also in more complex scenarios. We have conducted extensive experiments to prove our method's applicability to both symmetric noise and instance-dependent label noise [1], as well as in real-world environments with label noise [2, 3] (refer to Section 4.1, Tables 1 and 2). These experiments demonstrate that our method is capable of effectively identifying early stopping points across various types of label noise studied in the community of learning with noisy labels.
2. Limitations in Specific Noise Types: While we highlight the effectiveness of the Label Wave method, it is consistent with your intuition that our method may not work equally well across all types of label noise. Recent research [4] highlights a type of noise known as subclass-dominant label noise, where early stopping has proven to be ineffective. We wish to clarify that our method is designed to identify appropriate early stopping points; hence, scenarios where early stopping itself is inapplicable fall outside the scope of our research.

Part B: Amounts of Label Noise

1. Effectiveness in Low Noise Scenarios: We would like to emphasize that our method remains effective in training sets with noise levels below 20%. Our experiments show that the method works well even with a noise rate as low as 10%. The choice to experiment with noise rates fixed between 20%-80% was based on the common range of dataset noise discussed in the research of learning with noisy labels.
2. Limitations at Very Low Noise Levels: However, we recognize that the concerns raised about the method's effectiveness at extremely low noise levels are valid. Our primary goal in this paper is to demonstrate how the concept of "learning confusion patterns" aids in detecting the transitional point in learning with noisy labels. To maintain simplicity, intuitiveness, and computational efficiency, we used the variability metric explored in Section 3.1, termed as prediction changes or PC (Section 3.3). Designing metrics that track the fluctuations of the model’s fitting performance on a broader range can be an effective way to improve our work. One initial intuition is to discuss the degree of fluctuations in prediction changes, which could help identify the early stopping point even when there is no obvious first local minimum in prediction changes.
3. 0% Label Noise: We have noted the suggestion to discuss scenarios where label noise is zero. However, as discussed in our paper, our method is not applicable to typical clean training sets. In cases of completely clean datasets, modern deep neural networks often exhibit benign overfitting [5, 6], making early stopping unnecessary. Scenarios where early stopping itself is inapplicable are beyond the scope of our method's intended use.

[1] Part-dependent label noise: Towards instance-dependent label noise, NeurIPS 2020. [2] Learning with noisy labels revisited: A study using real-world human annotations, ICLR 2022. [3] Learning from Massive Noisy Labeled Data for Image Classification, CVPR 2015. [4] Subclass-Dominant Label Noise: A Counterexample for the Success of Early Stopping, NeurIPS 2023. [5] Benign overfitting in linear regression, PNAS 2020. [6] Benign overfitting in classification: Provably counter label noise with larger models, ICLR 2023.

Response to Question 2 and Question 6 (More datasets & Real-world noise) We respectfully wish to highlight that our testing on the CIFAR-10N dataset involved human-annotated real-world noise, showcasing the Label Wave method's effectiveness in real-world scenarios. Furthermore, our experiments conducted on the NEWS and Tiny-ImageNet datasets have demonstrated our method's broad applicability across various domains. Here, we present the outcomes of applying our Label Wave method on the Clothing1M dataset. We are committed to supplementing more results using real-world datasets in the further revision.

We extend our heartfelt gratitude to the reviewer for your detailed constructive feedback, addressing both the content of our paper and the applicability of our proposed method. Below we address the comments you’ve raised. We are very open to further discussion.

Weakness.1 - Related Work

In revised manuscript, we have added the references [1] and [2] in Related Work.

[1] and [2] conceptualize model learning with noisy labels as two stages: the first stage learns from clean examples, and the second stage overfits to mislabeled examples. By identifying the largest safe set in the first stage and focusing on learning from that set in the latter stage, these methods effectively avoid the model's overfitting to mislabeled examples. Notably, [2] provides a method to identify the point where MP(t) equals MR(t) without additional supervision, thereby determining an early stopping point. [1] and [2] indeed provide good insights into the study of learning with noisy labels from two perspectives: the analysis of learning stages and the provision of practical learning with noisy label methods.

Here, we will emphasize the advantages of our proposed method from two aspects: the learning stages analysis and the model selection method.

Learning Stages:

We introduce a novel stage between the stage dominated by learning simple patterns and overfitting mislabeled examples, named "Learning Confused Patterns". In this stage, the model begins to learn from mislabeled examples, leading to simultaneous declines in generalization and fitting performance.

Model Selection:

The Label Wave method not only requires no additional supervision but also has the advantages of being simple and efficient:

1. Simple Concept: It is based on the principle that the first local minimum in prediction changes marks the best early stopping point. Hence, the Label Wave method does not need to estimate the noise rate and is effective across a wide range of noise rates.
2. Direct Application: It tracks prediction changes during the training process, allowing the model to halt before learning confusing patterns. This approach avoids the complexity and computational costs of statistical models like the GMM and EM algorithms used in [2] and [3].

[1] How does early stopping help generalization against label noise, arXiv 2019 [2] Robust learning by self-transition for handling noisy labels, KDD 2021 [3] Selc: self-ensemble label correction improves learning with noisy labels, IJCAI 2022

Weakness.2 - Strong Regularization Techniques

Based on the reviewers' detailed comments, we tested the effectiveness of our Label Wave method with additional regularization methods. It is worth noting that our proposed Label Wave method itself includes BN and Data Augmentation, i.e., Label Wave + B, D as described below. Beyond the reviewer's comments, we have also tested the setting where these two regularization methods are not included. These regularization methods include:

A. Mixup B. BN C. Dropout D. Data Augmentation. (All experiments take the same settings as Sec. 4.1)

The experimental results show that adding or subtracting these regularization methods in the experimental setting does not affect the effectiveness of the label wave method for identifying early stopping points.

In the end, we thank the reviewer for the two constructive comments mentioned above, and addressing them really improves our paper to a great extent.

Thanks for the response from my questions. I carefully read the provided explanation and additional experiments. However, I still have some ambiguity.

Weakness.1 - Related Work

[1,2] will act as strong competitors to identify the early stopping point. I agree with that the proposed approach has distinct merits over them, but I am not sure if the proposed method outperforms the two existing works.

Weakness.2 - Strong Regularization Techniques

It was good to show effectiveness of the label wave even after applying data augmentation. But I wonder if the improvement still remains compared with Existing Label Noise Training without Label Wave. If strong augmentation are applied, the movement of test error will significantly change, e.g., the test error keeps going down over all training period. If this is the case, the early stopping is not required.

I am also open to further discussion.

Appreciate your hard working. The provided answer cleared my first concern, but still have concern for the second question. I saw many situations where validation and test errors are consistently going down even when there are noisy labels in training data (especially for real-world datasets, or when noise ratio is not significant).

Assuming the test error is like Figure 1 sounds a bit strict. These test error movement trends may vary depending on the training setup (e.g. augmentation, optimizer, learning rate, architecture, etc.). I think it would be nice if the proposed method had more insights to add (or expand on) its advantages in all scenarios rather than simply mentioning something that is out of scope.

I increased my score +1.

We thank the reviewer for your comments. We are glad you point out that the ‘learning confusion patterns’ presented in our paper is intriguing. Below we address the concerns you’ve raised, and we’d be very open to further discussion.

1. SOTA semi-supervised LNL methods on real-world datasets.

We would like to humbly point out that the CIFAR-10N dataset we have tested is a dataset labeled with human-annotated real-world noise. Besides, the experiments on NEWS and Tiny-ImageNet have demonstrated the applicability of our method on datasets from diverse domains. Meanwhile, our method is very simple and straightforward that can be used to strengthen existing methods, and thus cannot be directly compared with the state-of-the-art semi-supervised learning with noisy labels methods.

To address the common concerns of the reviewers, and in order to improve the credibility of our method, we tested the efficacy of our method applied to semi-supervised methods on real-world datasets.

Here, we present the outcomes of applying our Label Wave method in state-of-the-art semi-supervised methods, on the Clothing1M dataset. Notably, we have already evaluated ELR [2] in our paper (i.e., in Tables 3 and 4 on Page 8). Therefore, our current focus is to apply our Label Wave method to the methods including CE (baseline), DivideMix [1], and AugDesc [3]. We are committed to supplementing more results using real-world datasets and semi-supervised methods in the further revision.

2. Maximum test accuracy during the training phase.

We would like to humbly point out that in Tables 1 and 2 (in Section 4.1), we presented a direct comparison between the test accuracy of models chosen through Label Wave and the maximum test accuracy model during the training phase. This comparison validates the effectiveness of our Label Wave method.

For results in Tables 3 and 4 (Section 4.2), the difference between maximum test accuracy and label wave selected model are shown below:

In the end, we thank the reviewer for your detailed further revision comments that helped to improve the credibility of our method.