Rethinking Confidence Scores and Thresholds in Pseudolabeling-based SSL

We appreciate the feedback and questions. Our response is as follows,

On baselines that do not rely on confidence scores and thresholds. While this would be interesting, our paper's focus is on pseudolabeling methods based on confidence scores and thresholds. For this reason, we chose baselines necessary to evaluate our claim — using our scores and thresholds in the common methods (FixMatch, FreeMatch) can improve the performance significantly, in contrast to using them with ad-hoc choices of scores and thresholding techniques. We also compared the variations of these with recent methods (BAM, MR) designed to encourage calibrated scores in SSL settings.

On validation data requirements. We clarify as follows.

• it is common practice in SSL methods to use validation data to find the best model checkpoint by evaluating validation accuracy. The baselines use all $N_{val}$ validation samples for this purpose.

• In our method, we split this $N_{val}$ data into three parts, $N_{val}'$, $N_{cal}$, $N_{th}$ and use $N_{val}'$ for the usual model checkpoint evaluation and use $N_{cal}, N_{th}$ for learning confidence function and estimating thresholds respectively. Note $N_{cal}$ and $N_{th}$ are much smaller than $N_{val}$.

• The idea of using the validation data for training is natural. Common benchmarks have been extensively studied in the literature for which there is a reasonable understanding of hyperparameters and expected performance. However, this overlooks the generality of SSL methods. In general, most of the SSL methods need validation data for model selection, tuning hyperparameters, etc. This cost is often overlooked in the prior works. Our work mentions it transparently.

Clarification on Table 6 results. These results are to study the effect of sizes of calibration $N_{cal}$ and threshold estimation sets $N_{th}$. When these are too small, it will result in high variance in learning scores and thresholds, resulting in unreliable pseudolabeling (high excess pseudolabeling error). Using uniform convergence results it can be shown that the excess pseudolabeling error will scale as $O(\frac{1}{\sqrt{N_{cal}}} +\frac{1}{\sqrt{N_{th}}})$. Thus we see as $N_{cal}, N_{th}$ increases the performance gets better.

Inverse U Shape curve for error tolerance. In general, we expect an inverse U shape for test accuracy vs epsilon curve. In the Fig. 3. setting, we did not see this. We have run this experiment in one more setting where we see the inverse U shape curve. We summarize our findings in two points,

• We note that zero error tolerance is not equivalent to classical supervised learning. We see sizeable pseudolabeling taking place even with 0 error tolerance in certain settings, e.g., as in Figure 3, CIFAR-10 with 250 labeled points for training. Please see the new results here . This means reasonable models are found early in the training that are 100% accurate on part of the space and our scores, and thresholds are able to find that space.

• Second, in settings with an even smaller amount of labeled data for training, we start to see the expected inverse U shape curve. We ran the procedure in the CIFAR-10 setting with 40 labeled points for training with various choices of $\epsilon \in$ { 0.1%, 1%, 5%, 20%, 40% }. See new results for the plot of test accuracy corresponding to each $\epsilon$. The test accuracies are 74.6%, 91.5%, and 83.8% corresponding to $\epsilon$ 0.1%, 1%, and 5% respectively.

We hope our response resolves your queries. We are happy to answer any further questions.

Thanks for the careful review and positive feedback. We appreciate the recognition of the strengths of our work — a flexible and principled approach for learning confidence scores and thresholds for pseudolabeling and its empirical effectiveness. Our response to the queries is as follows,

Differences in insights and techniques compared to [1]

The prior work [1] studies a procedure to create labeled datasets with accuracy guarantees. Thus, [1] is only concerned with labeling the data correctly and not with how good the end model is. In contrast, our work is on semi-supervised learning, where our goal is to learn a classifier with good generalization error. In this procedure, the pseudolabels are never "committed", i.e., in each iteration, the assigned pseudolabel to a point can change. In contrast, in [1], the label assigned to a point is never changed. Furthermore, our approach and insights are novel within the area of pseudolabeling-based semi-supervised learning (SSL) [4,5]. We provide two points on this:

First, our work settles the question of the right choices of confidence functions for SSL. Recent works [2,3] have highlighted the problem of miscalibrated confidence functions, leading to inefficiencies in pseudolabeling-based SSL. Although these works offer solutions to this problem, they still fall short of addressing the core issue: the confidence function is not specifically tailored to the needs of SSL. To provide a principled solution to this problem, we adapt the framework for learning confidence functions in the TBAL setting from [1] to the SSL setting.

Second, we show how this framework for learning confidence functions can work in concert with popular SSL methods such as Fixmatch [4], Freematch [5], etc., and conduct an extensive empirical evaluation demonstrating that using confidence function learned from our method can yield significant improvements in the test accuracy. The flexibility to be integrated into a variety of techniques is novel to this work; works like [1] did not need to offer it. As a result, our work provides a flexible solution to practitioners using SSL, freeing them from the trial and error of selecting various confidence functions or hand-crafting one.

[1] Vishwakarma, H., Lin, H., Sala, F., and Vinayak, R. K. Promises and pitfalls of threshold-based auto-labeling. NeurIPS, 2023

[2] Loh et al., Mitigating confirmation bias in semi-supervised learning via efficient bayesian model averaging. TMLR, 2023.

[3] Mishra et al., Do not trust what you trust: Miscalibration in semi-supervised learning, arXiv, 2024.

[4] Sohn et al., Fixmatch: Simplifying semi-supervised learning with consistency and confidence, NeurIPS, 2020.

[5] Wang et al., Freematch: Self-adaptive thresholding for semi-supervised learning, ICLR, 2023.

We hope our response resolves your queries. We are happy to answer any further questions.

We thank the reviewer for the constructive feedback and the noted strengths — a lightweight, intuitive, and theoretical framework to learn scores and thresholds that can be integrated with existing SSL approaches to improve their performance. Our response to the queries is as follows,

More labeled data. We have evaluated the methods on settings with smaller $N_l$. Specifically, using $N_l=$ 40 for Cifar-10 and SVHN and 400 for Cifar-100. The results are available here. The results are consistent with claims in the main paper and even more pronounced than those in Table 2.

On pseudolabel accumulation. For the accumulation procedure our hypothesis is that it may help the high-precision pseudolabeling methods but it may not be useful for methods with noisy pseudolabels. We do not expect it to help baselines and the results in Table 4 are for completion — to show the performance of baselines with accumulation as well. The accumulation method is generally helpful when used in concert with our confidence scores and thresholds that can ensure high-precision pseudolabels. More sophisticated accumulation strategies accounting for the staleness of pseudolabels might be interesting to explore for future works.

On hyperparameters. While several of the SSL baselines also require choosing hyperparameters, we provide guidance on selecting the hyperparameters $N_{cal}$, $N_{th}$, $\epsilon$. The results in Table 6, suggest setting $N_{cal}, N_{th}$ to as high value as possible will be favorable. For $\epsilon$ a small value ( $\le$ 1%) is favorable when $N_l$ is not too small (i.e. when the initial models are not expected to be too bad); when we expect the initial models to be bad (or have $N_l$ too small), then using slightly higher epsilon around 5% is favorable. We will include a detailed discussion on these recommendations in the paper.

Class-wise vs global thresholds. Class-wise threshold estimation is suitable when there are less number of classes and global threshold estimation is suitable when the number of classes is large. We use class-wise for CIFAR-10 and SVHN settings and global for CIFAR-100 cases. The rationale behind this is, that when the number of classes is large it is hard to have sufficient samples for each class to learn meaningful thresholds. Thus in such settings, it is more useful to learn a global threshold.

Moving related works section earlier. We will move this in the camera-ready version.

Thresholds over iterations. Since we learn new scores in each pseudolabeling iteration, the scale of scores is not consistent across the iterations and thus we do not see a pattern in thresholds over iterations. However, Figures 4 and 5, provide insights into how the quantity and quality of pseudolabels evolve over time.

We hope our response resolves the queries. We are happy to answer any further questions you may have.

We appreciate the feedback and the noted strengths of our paper. Our work is well-positioned in the literature on SSL and confidence calibration. Our principled methods to learn confidence scores and thresholds with error bounds replace the heuristic-based choices and enhance the prior SSL methods. Our response to the queries is as follows,

Generalization/transfer to unlabeled data. Yes, we assume that the validation data and the unlabeled data are independent and identically distributed (i.i.d). Under this assumption, it is easy to show using standard uniform convergence results that the optimal solution on the validation set will transfer (generalize) to the unlabeled set. More specifically, our method uses parts of the validation data: $N_{cal}$ samples for learning the confidence scores and $N_{th}$ samples for estimating thresholds. Using the uniform convergence results, we can show that the excess pseudolabeling error when transferring the solution to the unalabeled data will be $O(1/{\sqrt{N_{cal}}} + 1/{\sqrt{N_{th}}})$. Thus, as long as $N_{cal}$ and $N_{th}$ are sufficiently large and the i.i.d assumption is satisfied, the solution will generalize to the unlabeled data.

The setting where the distributions of unlabeled and validation data are not aligned is interesting, and adapting our methods to these settings would be a fruitful direction for future work.

On pseudolabel accumulation. For the accumulation procedure, our hypothesis is that it may help the high-precision pseudolabeling methods, but it may not be useful for methods with noisy pseudolabels. We do not expect it to help baselines, and the results in Table 4 support this. The accumulation method is generally helpful when used in concert with our confidence scores and thresholds that can ensure high-precision pseudolabels.

More baselines and datasets. Our focus is on principled choices for confidence scores and thresholds for pseudolabeling based SSL, and we proposed solutions to this end. The goal of our experiments is to show that modern and commonly used methods, such as Fixmatch, Freematch, can be adapted to use our scores and thresholds, and the performance of the resulting method is better in comparison to using their standard versions. Since our focus is on confidence functions, we included baselines Margin Regularization (MR) and Bayesian Model Averaging (BAM) that are aimed at improving the calibration of confidence scores for pseudolabeling.

We used common benchmark datasets in SSL, that are sufficient for our empirical analysis. Introducing stronger benchmarks for SSL would be interesting future work.

Fixmatch + SVHN Case. We point the reviewer to Figure 2 (top, right), where we plot the test accuracy over iterations for this case. This plot suggests that all the methods for the Fixmatch + SVHN case have similar performance.

Related works. We have updated the related works with a discussion on the shared references.

Discussion on runtime. It is correct that additional training for confidence functions and calculating thresholds increases runtime. For this reason, we adjust the training iterations of the baselines so that all the methods are run for the same amount of time. We have included the details on runtime in Appendix C. We are happy to provide any specific details the reviewer is interested in.

On running the CIFAR-100 experiment longer. For fair comparison, we fixed our experiment protocol, i.e., run the variations with our method for 25K iterations and the baselines for the equivalent number of iterations in the same amount of time. Thus, the evaluation is fair. Prior works run till 1 million iterations to reach near convergence. Note that this takes an enormous amount of time, and one of the advantages of using our method is that it can achieve high accuracy earlier and does not require running very long.

Observed pseudolabeling errors. We plot the pseudolabeling error and coverage for the Cifar-10 setting. Please see the plot on this link. We see that the observed pseudolabeling error is very close to the target $\epsilon$ when it starts to give non-trivial pseudolabeling coverage, and it approaches $\epsilon$ as the training progresses.

Impact of confidence function architecture and its alternatives. Our framework to learn confidence scores is flexible to work with any choice of $\mathcal{G}$. While the choice of the function class $\mathcal{G}$ is up to the user, in general, it makes sense to use a flexible non-linear function class. A multi-layer neural network with any activation function could be a good fit here. We chose a 2-layer NN since the classification model is doing the heavy lifting of learning the features, so for $g$ we do not need a highly complex network.

We hope our response resolves the queries. We are happy to answer any more questions you may have.