Rethinking pseudo-labeling: Data-centric insights improve semi-supervised learning

References

[R1] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.

[R2] Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 191–195, 2021.

[R3] Niladri S Chatterji and Philip M Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. J. Mach. Learn. Res., 22:129–1, 2021.

[R4] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822–2878, 2018.

[R5] Maini, Pratyush, Saurabh Garg, Zachary Lipton, and J. Zico Kolter. "Characterizing datapoints via second-split forgetting." Advances in Neural Information Processing Systems 35 (2022): 30044-30057.

[R6] Wang, Yidong, et al. "Usb: A unified semi-supervised learning benchmark for classification." Advances in Neural Information Processing Systems 35 (2022): 3938-3961.

Dear Reviewer K9EF

Thank you for your thoughtful comments and suggestions! We give answers to each of the following points in turn and highlight the updates to the revised manuscript. In addition, we have uploaded the revised manuscript. We hope this response alleviates your concerns, but please let us know if there are any remaining concerns.

• A) Additional experiments in computer vision
• B) Analysis of conditions of success

(A) Additional experiments in computer vision

We thank the reviewer for this point on applying DIPS to images datasets. To address this, we have conducted experiments on $4$ additional image datasets : satellite images from Eurosat [R1] and medical images from TissueMNIST which form part of the USB benchmark [R6]. Additionally, we include the related OrganAMNIST, and PathMNIST, which are part of the MedMNIST collection [R2]. Given that these datasets are well-curated, we consider a proportion of $0.2$ of symmetric label noise added to these datasets.

We report the results in the following figure available at: https://i.imgur.com/NYmYGHs.png

As is shown, DIPS consistently improves the FixMatch baseline, demonstrating its generalizability beyond tabular datasets.

Additionally, we wish to reiterate that beyond CIFAR-10N, we have also conducted additional experiments on CIFAR-100N in the Appendix.

UPDATE: we have included these additional results in Appendix C.4.4 in our revised manuscript.

(B) Analysis of conditions of success

We thank the reviewer for the question on conditions of success of DIPS. To answer it, we conduct a synthetic experiment following a similar setup as in Sec. 5.1 in our manuscript. Our setup considers the dependencies between the amount of label noise and the amount of labeled data. Note that the experiment is synthetic in order to be able to control the amount of label noise. We considered the same list of label noise proportions as in Sec. 5.1, ranging from 0. to 0.45. For each label noise proportion, we consider $n_{\mathrm{lab}} \in$ {50,100,1000}, and fix $n_{\mathrm{unlab}} = 1000$. For each configuration, we conduct the experiment $40$ times. We report the results in the following plots available at: https://i.imgur.com/rzFeFsl.png

As we can see on the plots, PL+DIPS consistently outperforms the supervised baselines and PL in almost all the configurations. The performance gap between DIPS and PL is remarkably noticeable for $n_{\mathrm{lab}}=1000$, that is, when the amount of labeled samples is high, which hints at the fact that curation is made easier with more samples, and hence leads to improved performance. When the amount of labeled data is very low ($n_{lab} = 50$) and the proportion of corrupted samples is high ($0.45$), we can see that the different baselines have closer accuracies. We note, though, that DIPS consistently improves the PL baseline for reasonable amounts of label noise which we could expect in real-world settings (e.g. 0.1).

UPDATE: We have included these results in Appendix C.5 of the revised manuscript.

The reviewer also asks about any "theoretical connection" involving "statistical features of historical predictions". Some previous works in the supervised learning setting have shown that learning dynamics emit a strong signal about the nature of samples ([R3],[R4]). For example, Theorem 3 in [R5] (Appendix A.4 in the corresponding paper) gives some intuition about mislabeled samples in the supervised setting, by stating that the probability that mislabeled examples are classified with their given (incorrect) labels tends to $0$ in a mixture setting, as the model is trained until convergence. As such, this phenomenon directly impacts the agreement between the predictions of the model and the given label of a mislabeled sample, the agreement being worse as the model converges. This agreement is directly the quantity that we capture with the learning dynamics (confidence and aleatoric uncertainty) and DIPS' selection mechanism $s$.

We hope this answered your points, please let us know if there are any remaining concerns.

Dear Reviewer K9EF

We are sincerely grateful for your time and energy in the review process.We hope that our responses and appendix/manuscript updates have been helpful. Please let us know of any leftover concerns and if there was anything else we could do to address any further questions or comments!

Thank you!
Paper 3725 Authors

Thank you for your response, we are glad our additional experiments and analysis addressed your concerns. We would like to thank you for the encouragement to add further image datasets, which has improved the paper and more thoroughly illustrates the impact of DIPS and data quality across diverse settings --- which is now more extensive than other papers in this area (see below Table).

Please let us know if there are any remaining points that we could address that would lead you to increase your score. We are eager to do our utmost to address them!

--

References

[R4] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, 2013.

[R5] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations, 2021.

[R6] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. FlexMatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34, 2021

[R7] Kai Sheng Tai, Peter D Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi-supervised classification via annealed self-training. In International Conference on Machine Learning, pp. 10065–10075. PMLR, 2021.

[R8] Vu-Linh Nguyen, Sachin Sudhakar Farfade, and Anton van den Hengel. Confident Sinkhorn allocation for pseudo-labeling. arXiv preprint arXiv:2206.05880, 2022a

[R9]Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.

[R10]Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. In The Eleventh International Conference on Learning Representations, 2023

Dear Reviewer hGhE

Thank you for your thoughtful comments and suggestions! We give answers to each of the following points in turn and highlight the updates to the revised manuscript. In addition, we have uploaded the revised manuscript. We hope this response alleviates your concerns, but please let us know if there are any remaining concerns.

• A) Additional experiments with the USB benchmark
• B) Placement of pseudo-code

(A) Additional experiments with the USB benchmark

We thank the reviewer for this good suggestion to include datasets from the suggested USB benchmark, which, we believe, will strengthen our submission. To address this point, we have conducted experiments on $4$ additional datasets: satellite images from Eurosat [R1] and medical images from TissueMNIST which form part of the USB benchmark [R3]. Additionally, we include the related OrganAMNIST, and PathMNIST, which are part of the MedMNIST collection [R2]. Given that these datasets are well-curated, we consider a proportion of $0.2$ of symmetric label noise added to these datasets.

We report the results in the figure available at https://i.imgur.com/NYmYGHs.png

As is shown, DIPS consistently improves the FixMatch baseline, demonstrating its generalizability beyond tabular datasets.

Additionally, we wish to reiterate that beyond CIFAR-10N, we have also conducted additional experiments on CIFAR-100N in Appendix C.4.1.

UPDATE: we have included these additional results in Appendix C.4.4 in our revised manuscript.

(B) Placement of pseudo-code

Thank you for your suggestion to improve the presentation of our work. We agree with you and have moved the pseudo-code presented in Algorithm 1 to Section 4.4 in the updated manuscript.

UPDATE: Algorithm 1 moved to Section 4.4 in the revised manuscript.

We hope this answers your points, please let us know if there are any remaining concerns.

References

[R1] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.

[R2] Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 191–195, 2021.

[R3] Wang, Yidong, et al. “Usb: A unified semi-supervised learning benchmark for classification.” Advances in Neural Information Processing Systems 35 (2022): 3938-3961.

Dear Reviewer hGhE

We are sincerely grateful for your time and energy in the review process.We hope that our responses and appendix/manuscript updates have been helpful. Please let us know of any leftover concerns and if there was anything else we could do to address any further questions or comments!

Thank you!
Paper 3725 Authors

Dear Reviewer hGhE,

Thank you once again for your insightful feedback and for acknowledging the new experiments we have added to our manuscript. We would like to take this opportunity to make the purpose of our paper clearer, which is as follows:

• We identify the issue of data quality and mislabeling in the original labeled dataset and highlight the importance of this for pseudo-labeling.
• We propose a general framework to incorporate this insight within the pseudo-labeling paradigm and introduce an approach, DIPS, that evaluates the quality of the labeling using model confidence and aleatoric uncertainty.
• We demonstrate the application and impact of this approach in both the tabular and image modalities.

Your review has been very helpful for us in disentangling these three points and we have clarified them as a result in the manuscript.

In short, we don’t use experiments to demonstrate the universal superiority of our proposed approach, DIPS (point 2). Rather we seek to show the usefulness and applicability of our key insight (point 1 above) by applying DIPS to real-world cases (point 3 above), and in particular those where manual data audit might be particularly hard (e.g. tabular). We believe others will find this approach useful too and they will be able to use it and develop new approaches in other modalities, such as those you suggest.

So, our paper identifies a fundamental issue that has been overlooked previously in semi-supervised learning, introduces a new method to tackle this limitation, and then shows its usefulness with some real-world examples.

We hope this clarifies the goal of our paper and we have updated the paper (updates in purple) to reflect the clarity that your review has helped us to reach.

Regarding our evaluation, we believe that the scope of our experiments is substantially broad to achieve our stated aims in the previous comment. We have demonstrated the impact and pervasiveness of the identified issue across multiple modalities and 19 datasets. This is substantially higher than the median of 4 datasets (mean: 4.1) employed to evaluate methods published at leading machine learning conferences (Liao et al. [R11]).

We wish to draw the Reviewer's attention to the validation of other approaches. We checked the modalities and number of datasets employed by the baseline approaches used in our manuscript, which we have included in the table below. We confirmed our experiments were more extensive than any other approach, both in terms of the number of datasets and modalities. CSA is the only other approach assessed on non-image data, and no approach conducted experiments on NLP or speech data.

We agree that NLP and speech are interesting domains for future research building on the formalism and problem introduced by DIPS, and we have updated Section 5 of our manuscript accordingly. However, as noted above, they are not the central focus of our work, which was to demonstrate that explicitly accounting for labeled data quality is an aspect that helps pseudo-labeling.

We trust that this, together with clarifying changes included in our revised manuscript, allays any concerns the Reviewer had regarding the thoroughness of our evaluation of DIPS.

References

[R4] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, 2013.

[R5] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations, 2021.

[R6] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. FlexMatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34, 2021

[R7] Kai Sheng Tai, Peter D Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi-supervised classification via annealed self-training. In International Conference on Machine Learning, pp. 10065–10075. PMLR, 2021.

[R8] Vu-Linh Nguyen, Sachin Sudhakar Farfade, and Anton van den Hengel. Confident Sinkhorn allocation for pseudo-labeling. arXiv preprint arXiv:2206.05880, 2022a

[R9]Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.

[R10]Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. In The Eleventh International Conference on Learning Representations, 2023

[R11] Thomas Liao, Rohan Taori, Deborah Raji, and Ludwig Schmidt. Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning. NeurIPS Track on Datasets and Benchmarks, 2021

Dear Reviewer hxp2

Thank you for your thoughtful comments and suggestions!

We give answers to each of the following points in turn and highlight the updates to the revised manuscript. In addition, we have uploaded the revised manuscript. We hope this response alleviates your concerns, but please let us know if there are any remaining concerns.

• A) Contrasting DIPS w/ Schmarje et al.,2022 [Part 2/3]
• B) Connection to active learning [Part 2/3]
• C) Threshold parameters & trade-offs [Part 2/3]
• D) Computational trade-offs [Part 3 /3]
• E) VIME baseline [Part 3/3]
• F) Incorporating DIPS w/ Fixmatch [Part 3/3]

(A) Contrasting DIPS w/ Schmarje et al.,2022 [DC3]

Thank you for highlighting the paper [L. Schmarje et al.,2022]. While both DIPS and DC3 handle the data-centric issue of issues in data and share similarities in their titles, they tackle different data-centric problems which might arise in semi-supervised learning. The main differences are along 4 different dimensions.

(i) Problem setup/Type of data-centric issue: DIPS tackles the problems of hard noisy labels where each sample has a single label assigned in the labeled set which could be incorrect. In contrast, DC3 deals with the problem of soft labeling where each sample might have multiple annotations from different annotators which may be variable.

(ii) Label noise modeling: DIPS aims to identify the noisy labels, whereas DC3 models the inter-annotator variability to estimate label ambiguity.

(iii) Integration into SSL: DIPS is a plug-in on top of any pseudo-labeling pipeline, selecting the labeled and pseudo-labeled data. DC3 on the other hand uses its ambiguity model (learned on the multiple annotations) to either keep the pseudo-label or use a cluster assignment.

(iv) Dataset applicability: DIPS has lower dataset requirements as it can be applied to any dataset with labeled and unlabeled samples, even if there is only a single label per sample. It does not require multiple annotations. DC3 has higher dataset requirements as it relies on having multiple annotations per sample to estimate inter-annotator variability and label ambiguity. Without multiple labels per sample, it cannot estimate ambiguity and perform joint classification and clustering. Consequently, DIPS is applicable to the standard semi-supervised learning setup of limited labeled data and abundant unlabeled data, whereas DC3 targets the specific problem of ambiguity across multiple annotators.

UPDATE: we have added this discussion in realtion to Schmarje et al.,2022 to Appendix A.4 in our revised manuscript.

(B) Connection to active learning

We wish to clarify the difference in setting of DIPS vs active learning. We agree that the concept of 'usefulness' in the selection of data samples is a significant aspect of both active learning and our work. However, it is crucial to highlight the distinct contexts in which this term is used in both settings.

Active learning primarily focuses on the iterative process of selecting data samples that, when labeled, are expected to most significantly improve the model's performance. This selection is typically based on criteria such as uncertainty sampling which focuses on epistemic uncertainty [R1-R4]. The primary objective is to minimize labeling effort while maximizing the model's learning efficiency.

In contrast, DIPS does both labeled and pseudo-labeled selection and employs the term 'useful' in a different sense. Here, 'usefulness' refers to the capacity of a data sample to contribute positively to the learning process based on its likelihood of being correctly labeled. Our approach, which leverages training dynamics based on aleatoric uncertainty and confidence, is designed to flag and exclude mislabeled data. This distinction is critical in our methodology as it directly addresses the challenge of data quality, particularly in scenarios where large volumes of unlabeled data are integrated into the training process.

In active learning, these metrics are used to identify data points that, if labeled, would yield the most significant insights for model training. In our approach, they serve to identify and exclude data points that could potentially deteriorate the model's performance due to incorrect labeling.

UPDATE: We have included this discussion in Appendix A.5 in our revised manuscript.

(C) Threshold parameters & trade-offs

We thank the reviewer for the question on the choice of threshold parameters. To address this point, we conducted an experiment in the synthetic setup where we varied the thresholds used for both the confidence and the aleatoric uncertainty. In addition to our choice used in our manuscript (confidence threshold = 0.8, and adaptive threshold on the aleatoric uncertainty), we consider two baselines:

1. confidence threshold = 0.9 and uncertainty threshold = 0.1 (aggressive filtering)
2. confidence threshold = 0.5 and uncertainty threshold = 0.2 (permissive filtering)

We show the test accuracy for these baselines in the following plot available at: https://i.imgur.com/QCYEgzm.png.

As we can see, our configuration outperforms both the aggressive filtering configuration (red line) and a permissive one (blue line), which is why we adopt it for the rest of the experiments. We empirically notice in Section 5.2 that it performs well on the 12 real-world datasets we used.

UPDATE: we have included these experimental results in Appendix C.8 in our revised manuscript.

(D) Computational trade-offs

Thank you for asking a question about the computational overheads in our method, particularly in relation to large-scale datasets. While evaluating labeled and pseudo-labeled examples to compute the learning dynamics might incur additional overhead, we want to highlight that our selection mechanism yields a direct improvement in the quality of data used for training. This effect leads to a quicker model convergence. We refer to Fig. 8) a) in our manuscript, which gives the time efficiency of DIPS for the dataset CIFAR-10N. It shows that DIPS accelerates convergence by a factor 1.5-4X, despite the computation of learning dynamics.

(E) VIME baseline

We would like to thank the reviewer for suggesting a comparison to VIME. We believe its incorporation has helped to strengthen the paper. We report the results of VIME tested on the same setup as DIPS in the next table, where we use the same 12 datasets as in Section 5.2. These results demonstrate that DIPS outperforms VIME across multiple real-world tabular datasets.

UPDATE: we have included these experimental results in Appendix C.9 in our revised manuscript.

(F) Incorporating DIPS w/ non-iteration-based SSL

We thank the reviewer for asking this question on the integration of DIPS in non-iteration-based SSL. The DIPS framework necessitates two ingredients: a list of model checkpoints, and a label for each sample for which we want to compute the learning dynamics. In the case of non-iteration-based SSL methods, DIPS updates the learning dynamics at every training step and performs its selection of labeled and unlabeled data every $k$ training steps (for example, a training step could denote an epoch). For the unlabeled data, DIPS uses the pseudo-labels computed by the model at the beginning of every learning dynamics cycle. Then, after $k$ training steps, DIPS curates the labeled and unlabeled data using its selector function and then updates the pseudo-labels used for the computation of learning dynamics on unlabeled data for the next $k$ training steps.

We hope this answers your points, please let us know if there are any remaining concerns.

References

[R1] Stephen Mussmann and Percy Liang. On the relationship between data efficiency and error for uncertainty sampling, 35th International Conference on Machine Learning, PMLR.

[R2]Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Mate Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745, 2011.

[R3] Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems, pp. 7024–7035, 2019.

[R4] Nguyen, Vu-Linh, Mohammad Hossein Shaker, and Eyke Hüllermeier. "How to measure uncertainty in uncertainty sampling for active learning." Machine Learning 111, no. 1 (2022): 89-122.

Dear Reviewer hxp2

We are sincerely grateful for your time and energy in the review process.We hope that our responses and appendix/manuscript updates have been helpful. Please let us know of any leftover concerns and if there was anything else we could do to address any further questions or comments!

Thank you!
Paper 3725 Authors

Thank you for your comprehensive comments, including detailed clarifications, discussions, and follow-up experiments. I have also carefully checked the discussions with other reviewers. I have a few additional comments as following.

(A) Comparison with DIPS and Schmarje et al., 2022 [DC3]

The conceptual comparison with existing work effectively highlights the uniqueness of the DIPS framework. I recommend extending this comparison to the other work [X. Wang et al., 2022], which also focuses on selective labeling. A broader comparison would further enhance the conceptual novelty of the proposed work.

(E) VIME Baseline

The experimental comparison with the Tabular-specific SSL method is appreciated. For a more comprehensive understanding, could you also include the numerical results of the supervised baseline and the vanilla PL method in the table? In addition, it would be beneficial to incorporate additional SSL methods specific to the tabular domain. While time constraints may limit this during the review period, please consider addressing this in your post-review revisions.

(F) Incorporation of DIPS with Non-Iteration-Based SSL

There seems to be a fundamental difference between DIPS’s approach and continuous training mechanisms like FixMatch, which generate instant pseudo-labels within the same training step. This distinction in the application and updating of pseudo-labels requires further clarification.It would be important to understand whether DIPS, when integrated with FixMatch, utilizes two distinct types of pseudo-labels: those derived from self-training and those from continuous training, similar to the way with [C. Wei et al., 2022]. A more explicit explanation of how DIPS can integrate with continuous training SSL methods, such as FixMatch, would be beneficial.

—

References

[X. Wang et al., 2022] Unsupervised Selective Labeling for More Effective Semi-Supervised Learning, in ECCV 2022.

[C. Wei et al., 2022] CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning, in CVPR 2022.

Thank you for your feedback and further clarification. I believe there are no major issues remaining for me, and the proposed concept is both versatile and distinct from previous related literature. Therefore, I intend to maintain my favorable rating.

(C) Importance of aleatoric uncertainty

We thank the reviewer for the suggestion. We conducted an ablation study where we removed the aleatoric uncertainty and only kept a confidence-based selection (with threshold = 0.8). We term this confidence ablation to highlight if there is indeed value to the aleatoric uncertainty component of DIPS.

We report results in the following table, for the $12$ tabular datasets used in Section 5.2, which shows the benefit of the two-dimensional selection criterion of DIPS. Of course, in some cases there might not be a large difference with respect to our confidence ablation --- however we see that DIPS provides a statistically significant improvement in most of the datasets. Hence, since the computation is negligible, it is reasonable to use the 2-D approach given the benefit obtained on the noisier datasets.

UPDATE: we have included these results in Appendix C.10 of our revised manuscript.

While DIPS builds on a notion of uncertainty, we highlight that this uncertainty differs in nature from the uncertainty metrics used in the pseudo-labeling literature. One key difference is that DIPS computes the aleatoric uncertainty based on some given labels (provided labels for $D_{lab}$ or pseudo-label for $D_{unlab}$), and then computes the agreement between the prediction of a model and these given labels as is captured by the quantity $[f_{e}(x)]_{y}$. This is very different in nature from the uncertainty metrics used in pseudo-labeling, which traditionally do not rely on a given label as they are applied to unlabeled data (as is done when using an ensemble of models and computing a standard deviation to define an uncertainty metric). This explains why these uncertainty based filtering methods only focus on unlabeled data, while ours can handle both labeled and pseudo-labeled data.

(D) Selection of unlabeled data

To answer the Reviewer's question regarding the selection of unlabeled data in DIPS, we refer to Figure 10 in our submitted manuscript (Appendix C.1). This figure shows results when our selection mechanism is only applied to the labeled data and we only use prior selection methods for the unlabeled data (this baseline is denoted as "A1"). The results show DIPS outperforms this baseline by a large margin. This justifies why we apply our selection method both to labeled and unlabeled data. We have updated our manuscript (Section 4.1) to more clearly highlight this result - thank you for the suggestion!

(A) Dependency between label noise level and amount of labeled data

We thank the reviewer for the suggestion to investigate the dependency between label noise level and the amount of labeled data. To answer this point, we conduct a synthetic experiment following a similar setup as in Section 5.1 in our manuscript. Note that the experiment is synthetic in order to be able to control the amount of label noise. We considered the same list of label noise proportions, ranging from 0. to 0.45. For each label noise proportion, we consider $n_{\mathrm{lab}} \in$ {50,100,1000}, and fix $n_{\mathrm{unlab}} = 1000$. We conduct the experiment $40$ times for each configuration.

We report the results in the plots available at: https://i.imgur.com/rzFeFsl.png.

As we can see on the plots, PL+DIPS consistently outperforms the supervised baselines in almost all the configurations. When the amount of labeled data is low ($n_{lab} = 50$) and the proportion of corrupted samples is high ($0.45$), PL is on par with the supervised baseline. This mirrors the reviewer's intuition that pseudo-labeling is more difficult with a very low amount of labeled samples (and a high level of noise). We note, though, that DIPS consistently improves the PL baseline for reasonable amounts of label noise which we could expect in real-world settings (e.g. $0.1$). The performance gap between DIPS and PL is remarkably noticeable for $n_{\mathrm{lab}}=1000$, i.e. when the amount of labeled samples is high.

UPDATE: We have included these results in Appendix C.5 of the revised manuscript.

(B) LNL + pseudo-labeling

In the following table, we report the results when applying the LNL baselines to the labeled data only, and then performing standard teacher-student training.

The results show that DIPS outperforms these baselines (Small-loss, Fluctuation, FINE) in almost all the datasets, and suggest that that DIPS' curation method based on learning dynamics should be preferred, both on the labeled set and on top of standard teacher-student training.

References

[R1] Vu-Linh Nguyen, Sachin Sudhakar Farfade, and Anton van den Hengel. Confident Sinkhorn allocation for pseudo-labeling. arXiv preprint arXiv:2206.05880, 2022.

• "why do we use checkpoints for the learning dynamic being epoch and not some parameter based on number of iterations? For very large data we will never do 1 epoch, or only 1 epoch, or only few and thus measure based on epochs will be weak."

We first note that DIPS only needs models which are trained in an iterative manner. For example, in our tabular experiments in Section 5.2, we use an XGBoost backbone, where the learning dynamics are computed over the different boosting iterations. Hence what we call a checkpoint in our manuscript (Section 4.2) needs not be an epoch. For neural networks, we do not necessarily need to define checkpoints as epochs, when the dataset is large; we can instead define checkpoints after every $k$ gradient steps.

• "did authors try to exclude some first epochs from confidence and uncertainty definition as they could be not very informative?"

We thank the reviewer for this suggestion. We have conducted an additional experiment to investigate the choice of the range of iterations used to compute the learning dynamics. We consider ignoring the first $25$%/$50$%/$75$% iterations and use the remaining iterations to compute the learning dynamics.

The plot available at https://i.imgur.com/QzNnHba.png

It shows the mean performance difference (on test accuracy) by using the truncated iteration windows versus using all the iterations, and averages the results over the 12 datasets used in Section 5.2. As we can see, it is better to use all the iterations window, as the initial iterations carry some informative signal about the hardness of samples. This motivates our choice of computing the learning dynamics over the whole optimization trajectory, a choice which we adopt for all of our experiments.

UPDATE: we have added these experimental results in Appendix C.7 of our revised manuscript.

• "Why does aleatoric uncertainty capture mislabeling? Why def. 4.2 is aleatoric as we consider variation across checkpoints (= over learning process)?"

Def. 4.2 of aleatoric uncertainty builds on the variability that originates from the inability to predict the correct label with high confidence over the different checkpoints. Intuitively, a sample with high aleatoric uncertainty would be a sample for which we are not confident in predicting the ground-truth label at every checkpoint (e.g. $[f_{e}(x)]_{y} = 1/2$ for all iteration $e$ in the case of binary classification). This would mean that the model is not capable of learning the given label, which happens for mislabeled points.

• "What is the upper bound (when all data are labeled and when all data are labeled and correct / w/o mislabeling) in Fig. 4?"

We thank the reviewer for the question and we report in the following plot, available at https://i.imgur.com/gsLrb7r.png, the test accuracy when all the data are labeled in Section 5.2. Note that we cannot control the amount of mislabeling in this setting, as we only have access to the training labels in the datasets.

Training a model on all the labeled data often yields the best results, which is intuitive, as we consider the pseudo-labeling setting which uses only $10\%$ of labeled data. We notice that in some datasets (e.g. cutract, compas, drug), DIPS improves the fully supervised baseline, which may hint at greater inherent noise in the datasets. We can't quantify exactly the level of noise as we don't have the ground-truth, but the performance gap obtained by using DIPS could be seen as a proxy for it.

• "How many iterations are done for teacher-student baselines in Fig. 4?" "I see that aleatoric uncertainty is adopted per dataset."

We used $T=5$ iterations for the experiment in Fig. 4. This follows the guideline of the sensitivity experiment conducted in [R1] (Appendix B.2 in the corresponding paper) and explains why we kept this parameter fixed for the datasets used in Section 5.2 in our manuscript. Regarding the aleatoric uncertainty threshold, it is chosen to be adaptive, and is not hand-crafted for each dataset. Furthermore, we stress that DIPS is applied on top of the baselines.

• "What about Fig 11 with vanilla PL but w/o any data selection?"

We give the radial diagram showing the target dataset and how the DIPS selector and vanilla PL selector differ at the following link: https://i.imgur.com/f4VV76N.png.

We see that DIPS is closer to the test data than vanilla PL selector.

UPDATE: we have added this plot in Appendix C.3 in our revised manuscript.

We hope this answers your points, please let us know if there are any remaining concerns.

(E) Terminology of pseudo-labeling

We thank the reviewer for the comment about the terminology of teacher-student pseudo-labeling and apologize if our description of this paradigm was not clear enough in our manuscript. We decided to describe the common teacher-student pseudo-labeling methodology adopted in the tabular setting. As a consequence, we used the implementation provided by [R1], which grows the training set with pseudo-labels generated at the current iteration, thus keeping old pseudo-labels in the training dataset in subsequent iterations.

In addition to adopting this practice, we investigated this choice experimentally, by comparing between two versions of confidence-based pseudo-labeling:

• Version 1): with a growing set of pseudo-labels (as followed by the implementation of [R1] and our paper)
• Version 2): without keeping old pseudo-labels, as suggested by the reviewer.

We evaluate these two methods in the synthetic setup described in Section 5.1 of our manuscript, and report the test accuracy in the plot available at: https://i.imgur.com/VSPg1mD.png.

The red line corresponds to Version 1) (the implementation we used in the manuscript), while the green line corresponds to Version 2) (suggested by the reviewer). As we can see, in this tabular setting, growing a training set by keeping the pseudo-labels generated at each iteration leads to the best results, motivating our adoption of this pseudo-labeling methodology used in the tabular setting.

UPDATE: we have included this experiment in Appendix C.6 of our revised manuscript.

We apologize if the terminology of greedy-PL was not made sufficiently clear in our original manuscript. We refer to confidence-based PL as "greedy-PL", following the same terminology as in [R1]. As such, all the baselines in Section 5.2 fall under the umbrella of " teacher-student PL methods", where the models (i.e. XGBoost in Section 5.2) are trained from scratch at each iteration in a teacher/student fashion.

Finally, to address the query about Fig 3 with zero corrupted samples, we add to the results in Fig.3 the zero corruption setting in the updated plot available at: https://i.imgur.com/EBUYDkY.png.

In the zero corruption setting, all methods almost achieve perfect test performance.

UPDATE: we have updated Fig. 3)c) in our revised manuscript.

We also highlight that pseudo-labeling does always increase performance over the purely supervised approach, which is natural since PL has access to unlabeled data. However, we are interested in a setting where the labeled data is subject to mislabeling. Hence, this explains why the margin between PL and Supervised in Fig. 3)c) is not bigger: noise propagates to the unlabeled data, which limits the effectiveness of PL and justifies the data-centric lens that DIPS adopts, hence giving a big performance gap with the baselines.

(F) Answers to the other questions

• "I do not agree entirely with statement "labeled data are noisy & if small amounts we can request the gold standard"

The issue of labeled data errors are prevalent in many industries from healthcare to finance. This is even the case in seemingly curated datasets as shown by Northcutt et al., where label error rates of widely-used benchmark datasets can reach up to 10%. As mentioned in Section 1, while it might appear possible to manually inspect the data to identify errors in the labeled set, we note that this requires domain expertise and is human-intensive, especially in modalities such as tabular data where inspecting rows in a spreadsheet can be much more challenging than reviewing an image. In other cases, updating labels is actually infeasible due to rerunning costly experiments in domains such as biology and physics, or indeed impossible due to lack of access to either the underlying sample or equipment.

• "application: these works use pseudo-labeling as a tool for supervised learning, whereas DIPS extends the machinery of pseudo-labeling itself." I don't see really huge difference here, as noisy labels means - we don't know the correct label, so mathematically it is very close tasks and solutions.

We agree that mathematically, the auditing task is similar. The main difference is how semi-supervised learning is used. In the first setting SSL is a solution to noisy labels, whereas we study and resolve the effect of noisy labels on SSL.

Dear Reviewer yF2p

Thank you for your thoughtful comments and suggestions! We give answers to each of the following points in turn and highlight the updates to the revised manuscript. In addition, we have uploaded the revised manuscript. We hope this response alleviates your concerns, but please let us know if there are any remaining concerns.

• A) Dependency between label noise level and amount of labeled data [Part 2/5]
• B) LNL + pseudo-labeling [Part 2/5]
• C) Importance of aleatoric uncertainty [Part 3/5]
• D) Selection of unlabeled data [Part 3/5]
• E) Terminology of pseudo-labeling [Part 4/5]
• F) Answers to the other questions [Parts 4,5/5]

Dear Reviewer yF2p

We are sincerely grateful for your time and energy in the review process.We hope that our responses and appendix/manuscript updates have been helpful. Please let us know of any leftover concerns and if there was anything else we could do to address any further questions or comments!

Thank you!
Paper 3725 Authors

Dear Reviewer yF2p,

Thank you for your time and questions! We answer your questions in what follows.

On VIME results: In order to get these results, we used the code repository released by the authors of VIME (https://github.com/jsyoon0823/VIME) to avoid any difference in implementation. We note that VIME necessitates a neural network as it builds on pretext tasks which require training an encoder. This contrasts the framework of pseudo-labeling, which is general-purpose, and enables us to use tree-based models such as XGBoost, which are traditionally the models giving the best performance for tabular data (Grinsztajn et al., 2022), and which might explain the performance gap with the supervised baseline.

Grinsztajn, Leo, Edouard Oyallon, and Gael Varoquaux. "Why do tree-based models still outperform deep learning on typical tabular data?." Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2022.

Filtering out noise from labeled data + training PL baseline: We apologize if this was not clear: as mentioned in our previous response, filtering out noise from labeled data first and training the PL baseline (which is confidence-based filtering, and does not use DIPS) corresponds to the A1 baseline in Appendix C.1. As we show in Figure 10, DIPS outperforms A1.

On LNL results: The Reviewer is right that some of the results in Appendix C.2 are worse than the supervised baseline. We attribute this to the unsuitability of these selectors for the setting DIPS tackles. While these methods are part of the LNL literature, they traditionally assume access to large labeled datasets, which contrast our pseudo-labeling setting, in which none of these baselines have been used before. Hence, there are no guarantees of performance translations during the iterative procedure.

On adding confidence filtering: We want to emphasize that the baselines we consider in Figure 4 already use confidence filtering in their selection of pseudo-labels. For example, greedy-PL solely uses a confidence threshold, while UPS adds to it an uncertainty threshold. We integrate DIPS on top of them in the experiment of Figure 4.

Uncertainty metrics designed for unlabeled data applied to labeled data: We wish to clarify that the uncertainty metrics focused on unlabeled data necessitate us to train a supervised model in the first place. Thereafter, we predict on the unlabeled set and apply the uncertainty metric to decide which pseudo-labels to keep. This is unsuitable for the labeled data as we would then have to use the same supervised model to predict on the very same labeled data --- yet having to "ignore" the label of the labeled sample. For example, in tabular data, the uncertainty proxy is often obtained with an ensemble. However, training this ensemble on labeled data and evaluating it on the same labeled data would reduce diversity in the predictions of the ensemble and lead to bad calibration of this uncertainty on labeled data (with the ensemble being very overconfident).

On ablation in Appendix C.6: We will conduct these experiments on the datasets of Section 5.2 for our post-review revisions considering the limited time before the end of the rebuttal window.

Baseline implementations: The implementation of the baselines in our tabular experiments (Section 5.2) trains the models from scratch, while FixMatch in our images experiment (Section 5.5) is continuous model training. Hence, the methods in Section 5.2 are not implemented as a mixture of these two types of methods, they are implemented as teacher-student methods. We used their implementation provided by the authors of [R1] at the following link: https://github.com/amzn/confident-sinkhorn-allocation. We will clarify this distinction between teacher-student training and continuous model training in our revised manuscript.

On iterations for learning dynamics: Let us clarify: in Section 4.2, we define the learning dynamics by studying how the model predicts samples over different checkpoints. For example, for a neural network, a checkpoint can be an epoch. For an XGBoost, this can be a boosting iteration.

These checkpoints are not to be confused with the teacher-student iterations of the baselines that we use in Figure 4 and Section 5.2.

On ablation in Appendix C.7: In the ablation of Appendix C.7, the "iterations" denote the boosting iterations of the XGBoost model. For example, 25% (on the x-axis of the plot) means that in order to compute the learning dynamics of the XGBoost model trained at every teacher-student iteration, we use the last 75% of the boosting iterations of the XGBoost.

On memorization in the model: Memorization may impact the computation of the learning dynamics. However, with learning dynamics we compute the metrics by looking at the behavior of the model over multiple checkpoints, and not only the last one. In the latter case, memorization would indeed be a problem. Furthermore, we want to point out to theoretical results that show (in some simple scenarios) that mislabeled points are unlikely to be classified with their (incorrect) label. For example, Theorem 3 in (Maini et al., 2022) (Appendix A.4 in the corresponding paper) gives some intuition about mislabeled samples in a mixture setting, by stating that the probability that mislabeled examples are classified with their given (incorrect) labels tends to $0$, as the model is trained until convergence.

FixMatch+DIPS parameters: For the image experiments, we define one checkpoint every $1024$ gradient steps, and compute the learning dynamics over $10$ checkpoints.

We thank you for your time and energy in the reviewing process, please let us know if you additional questions!

References:

Maini, Pratyush, Saurabh Garg, Zachary Lipton, and J. Zico Kolter. "Characterizing datapoints via second-split forgetting." Advances in Neural Information Processing Systems 35 (2022): 30044-30057.

Dear Reviewer yF2p,

Thank you for your engagement during this discussion period! Let us clarify in this response the baselines used in our experiments.

In Sections 5.1, 5.2, 5.3, 5.4 (Tabular experiments)

For all these baselines: we use the implementation provided by https://github.com/amzn/confident-sinkhorn-allocation. Also note that all these methods are implemented as teacher-student models in the tabular experiments of Sections 5.1, 5.2, 5.3, 5.4, and that we do not use FixMatch for these tabular experiments.

DIPS integration: DIPS is integrated as a plug-in on top of these baselines, for both labeled samples and pseudo-labeled samples selected by these baselines, before each teacher-student iteration. Since DIPS is a plug-in, it means that these baselines still select pseudo-labeled samples based on their respective unlabeled data selection mechanism (e.g. "Confidence thresholding"). DIPS is simply applied on top of them, for both labeled and pseudo-labeled data. That is why we refer to PL+DIPS for example in Figure 5 (i.e. we use the first baseline in the above table and plug-in DIPS on top). Similarly, in Figure 4, on the grouped bar plots, DIPS is plugged on top of each baseline (giving PL+DIPS, UPS+DIPS, etc.).

In Section 5.2 (Image experiment)

DIPS integration: DIPS updates the learning dynamics at every checkpoint. We define one checkpoint every $1024$ gradient steps. DIPS performs its selection of labeled and unlabeled data every $10$ checkpoints. For the unlabeled data, DIPS uses the pseudo-labels computed by the model at the beginning of every learning dynamics cycle. Then, after every $10$ checkpoints, DIPS curates the labeled and unlabeled data using its selector function and then regenerates the pseudo-labels used for the computation of learning dynamics on unlabeled data for the next learning dynamics cycle.

Note that the frozen pseudo-labels used for the learning dynamics are not to be confused with the online pseudo-labels which are used to train the WideResnet model: those are updated after every gradient step of the model.

We hope this clarifies your point, please let us know if you have any other questions!