Evaluating multiple models using labeled and unlabeled data

Thank you for your comments and the positive review! We appreciate that you recognize the importance of semi-supervised evaluation and believe our method is theoretically sound.

We address your questions and respond to the highlighted weaknesses below.

Q1. How does the author’s contribution improve the state of the art? How do authors ensure the lack of data leaks in the datasets they used?

Our approach improves upon the state-of-the-art in semi-supervised performance estimation by framing the problem as semi-supervised mixture model estimation; we show empirically that our method outperforms eight baselines across a wide range of tasks and metrics. We ensure there is no dataset leakage by using distinct sets of data to train the classifiers (the training split), to estimate performance (the estimation split), and to measure ground truth performance (the evaluation split). We detail this procedure in Section B.1 and will better incorporate this information into the main text.

Q2. Adding a limitations section

Thank you for this suggestion. We will add a dedicated limitations section to clearly discuss the scope and potential constraints of our approach, including the difficulty of estimating KDEs in high dimensions and the assumption that the labeled and unlabeled data are drawn from the same distribution.

Q3. Paper organization: moving the related work to be right after the problem setting. Addressing missing bibliography for the appendix.

Thank you for the suggestion and for flagging the reference issue in the appendix. We will shift the related work to appear after the introduction. The appendix references map to the main text bibliography (we manually split the files to submit the supplementary material). We will ensure that the references properly accompany the appendix.

W1. Outperforming baselines

We’d like to clarify: SSME consistently outperforms the eight baselines we compare to across metrics. In particular, SSME outperforms the next-best baseline in every metric tested (2.1x gain in accuracy estimation, 1.5x gain in ECE estimation, 1.2x gain in AUC estimation, and 1.1x gain in AUPRC estimation) averaged across tasks. We will clarify our statement in the checklist, which was intended to transparently state that we discuss limitations of our work and settings where baselines could outperform SSME (e.g. at Line 310 we discuss settings that would favor Bayesian-Calibration). Overall, however, we decisively outperform all baselines we compare to.

W2. Clarifying relationship to work in semi-supervised learning and active learning

Thanks for raising this! Our work indeed builds off prior work in semi-supervised learning and, to a lesser extent, active learning. All the baselines we compare to use either semi-supervised (Bayesian-Calibration, AutoEval, SPE, Dawid-Skene) or active learning (Active-Testing) to derive their performance estimates. Unlike prior methods, we are able to simultaneously incorporate continuous scores, labeled and unlabeled data, and multiple classifiers, whereas previous works only use at most two of these three data sources. We will further clarify this point in the related work.

W3. Improve writing to make clear the sources of the tasks; include references to datasets.

Thanks for the suggestion; while we do include references for each dataset in Section 4.1, we will make the connections more explicit. Our code, once public, will also support downloading and processing each of the five public datasets used in our experiments.

W4. Inclusion of source code

Our apologies, we misunderstood the checklist to mean the code must be made public upon publication. We will make the code available upon publication, including all code required to reproduce experimental results on five public datasets. Unfortunately, NeurIPS reviewing guidelines do not allow anonymized code URLs during the rebuttal process, so we cannot provide the code right now. The codebase will also support practitioners who wish to apply SSME to new settings.

Please let us know if our response addresses your primary concerns; if so, we would kindly appreciate it if you could raise your score. If not, are there additional concerns we can address?

Thank you for your comments! We appreciate that you recognize that SSME is able to utilize multiple sources of information simultaneously, has wide applicability, and outperforms baselines in estimating performance.

We address your questions and respond to the highlighted weaknesses below.

Q1. It seems that currently, this method is mainly applied for Natural Language classification tasks, how will it performs on image classification tasks?

Thanks for this question! SSME outperforms baselines across three settings we test: natural language data, tabular data, and graph data. In response to your question, we have added an experiment where we analyze SSME’s performance on image data in the context of CIFAR-3 [1, 2, 3] (i.e., a dataset with three classes from CIFAR-10), where classifiers are models pre-trained on ImageNet and finetuned on CIFAR. We found SSME outperforms baselines that can be applied in this setting, achieving an mean absolute error in accuracy estimation of 4.50 (SD: 2.12), while the next best baseline (labeled data alone) achieves a mean absolute error of 5.33 (SD: 2.77). We find similar results for ECE estimation, where SSME achieves a mean absolute error in ECE estimation of 3.75 (SD: 1.21), while the next best baseline (pseudo-labeling) achieves a mean absolute error of 6.36 (SD: 4.78).

[1] https://ewsn2022.pro2future.at/paper/sessions/ewsn2022-final3.pdf

[2] https://kclpure.kcl.ac.uk/ws/portalfiles/portal/332763034/2501.06066v3.pdf

[3] https://proceedings.mlr.press/v189/pandey23a/pandey23a.pdf

Q2. How will the diversity and the coverage of the classifiers being tested impact the estimation accuracy? Intuitively, more classifiers you considered, the more unbias your joint distribution should be, I encourage authors to conduct a experiments to test this.

Thanks for this question. We ran an experiment to measure the benefit of each additional classifier, as well as how much performance varies across classifier sets, and include the results below. In particular, for each number of classifiers between 2 and 7 (inclusive), we use SSME to estimate performance for each unique set of classifiers (so for 7 classifiers, there is exactly one set; for 4 classifiers, there are 7 choose 4 = 35 sets). We performed this experiment on the CivilComments dataset.

Your intuition is correct: additional classifiers do produce better estimates of performance, with diminishing returns. As shown in the table below, we find that SSME’s performance improves as the number of classifiers increases, which accords with our theoretical results. As a caveat, our theory predicts that SSME is likely not to perform well if one adds an inaccurate (e.g., worse than random) classifier to the set.

We also find that SSME's performance does not vary greatly across classifier sets, suggesting that SSME’s performance is robust to using different classifier sets. In particular, the standard deviation in accuracy estimation error across random classifier sets is always under 0.011, and the standard deviation in ECE estimation error is always under 0.015, indicating consistent performance across classifier sets.

(Dataset: CivilComments) MAE for ECE and accuracy across classifier set choice, along with the standard deviation in MAE across classifier sets.

W1. Comparison to additional methods

Thanks for your comment! In the original manuscript, we compare to and outperform eight state-of-the-art baselines designed for semi-supervised evaluation. We agree that unsupervised accuracy estimation methods are related, and Appendix A describes connections to past work in this literature. However, these baselines cannot be directly applied to estimate the metrics we wish to estimate. For example: three of the baselines you mention ([1], [3], and [4]) discuss heuristics that correlate well with model accuracy, but it is not clear how to map these heuristics to absolute estimates of other metrics like ECE, AUC, and AUPRC, as SSME does. It is also unclear how to extend the ideas in the other baseline you mention [2], which fits a custom pseudo-labeling function to image data specifically, to our setting, where it is unclear what such a pseudo-labeling function should be for a generic dataset. We agree the connection is interesting and believe it merits future work, and will include the citations you provided in Appendix A; thank you for bringing them up!

[1] Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation, ICML 2023.

[2] Unsupervised Accuracy Estimation of Deep Visual Models using Domain-Adaptive Adversarial Perturbation without Source Samples, ICCV 2023.

[3] Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density, ICCV 2021.

[4] Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data, KDD 2023.

W2. Limitation to classification While our method primarily focuses on semi-supervised performance estimation in classification, we note that we can very naturally extend our approach to incorporate regression settings as well. The key idea behind our method is to estimate a joint distribution for P(s, y) with a semi-supervised model. In classification settings, it is natural to model this using a mixture model. In a regression case with a continuous y, however, there are other joint density estimators for P(s,y) that could be reasonable, including KDEs or autoencoders. Thanks for flagging this, which we agree is a natural extension of our method and an interesting direction for future work!

W3. Proposed method is consistently worse than Bayesian-Calibration Thank you for this comment. To clarify, our results show that SSME decisively outperforms Bayesian-Calibration across the diverse datasets that we study. In particular, our results show that SSME outperforms Bayesian-Calibration in every metric tested (2.1x gain in accuracy estimation, 1.5x gain in ECE estimation, 1.2x gain in AUC estimation, and 1.1x gain in AUPRC estimation), averaging across datasets and tasks. Our tables in the appendix provide further details substantiating this result. Bayesian-Calibration also does not generalize to settings with more than two classes, an important limitation SSME overcomes.

Please let us know if our response addresses your primary concerns; if so, we would kindly appreciate it if you could raise your score. If not, are there additional concerns we can address?

Thank you for your thoughtful engagement with our rebuttal and for the helpful references. We will incorporate these citations into the paper. We cannot compare to these methods on the tasks we consider because neither method directly estimates model performance (though they are applicable to the related and easier task of model selection). We will clarify the distinction between the two tasks in the paper and expand our justification for the baseline comparisons; thanks again for your reply! If you are convinced by our rebuttal, we would kindly appreciate if you could raise your score.

Thank you for your positive review and recognizing both our theoretical contributions and in-depth empirical analysis. We respond to your questions and suggestions below.

Q1. Do different initializations affect the estimation of parameters?

Good question. We find that SSME’s performance remains strong when using an alternative plausible initialization method: using a majority vote among k=5 nearest neighbors among the labeled examples to initialize cluster assignment for unlabeled examples. We compare this alternate initialization method to our original initialization method (which we refer to as "draw" in the table below) on the CivilComments dataset with three different amounts of labeled data (20, 50, and 100 points), just as in our original paper. We include our results on ECE and accuracy estimation error in the table below, where we find that the two initialization methods perform comparably on the CivilComments dataset (with KNN slightly beating the original initialization method in 2 of 3 settings, and original beating KNN in the other). Notably, we continue to outperform all the baselines even with the alternative KNN initialization. This is useful practical guidance for implementing SSME, which we will incorporate into the paper.

Q2 + W3. Why use a different algorithm (UL+) in the theoretical analysis?

Thank you for this question. We use UL+ for the theoretical analysis as it enables us to build on prior results in semi-supervised learning for mixture models, which use this as a standard assumption. To our knowledge, there are no results establishing analogous performance bounds (i.e., bounds relating the amount of data to parameter estimation error) for EM algorithms. The bound derived from analyzing UL+ is likely conservative, as UL+ first uses the unlabeled data to identify decision boundaries then uses the labeled data to assign regions to classes. In general we expect EM, which directly maximizes the log-likelihood of both the labeled and unlabeled data, to perform even better.

Q3. Error bounds tighten as the number of classifiers increases; but adding classifiers introduces expenses. Given this tradeoff, is there an optimal number of classifiers?

Thanks for raising this question! Empirically, we find that the computational expense of an additional classifier is very small. Concretely, fitting SSME with nine classifiers takes 17.4 seconds; in comparison, fitting SSME with one classifier takes 14.3 seconds. We provide theoretical conditions under which an additional classifier provides benefit in Section C.1: specifically if the separation between components exceeds $\sqrt{d+1}$, where $d$ represents the dimensionality, any additional classifier whose accuracy is better than random will tighten the bound on performance estimation error.

Q4. Robustness of SSME to bandwidth h?

Thanks for this question. We ran an experiment to test the robustness of our results to the choice of bandwidth. In the paper, we use the Sheather-Jones algorithm [1] to automatically identify a bandwidth. We compare this approach to (1) another automated bandwidth selection algorithm (the Silverman algorithm [2]) and (2) two fixed bandwidths, chosen to be larger but within an order of magnitude of the Sheather-Jones selected bandwidth. We conduct experiments using the CivilComments dataset and 3 labeled dataset sizes (20, 50, and 100 labeled datapoints). While results remain strong in all cases, we achieve the best performance when using the Sheather-Jones bandwidth selection procedure, as is done in the original manuscript. We will incorporate these results into the manuscript.

[1] https://www.stat.cmu.edu/~rnugent/PCMI2016/papers/SurveyBandwidthSelectionWand.pdf

[2] https://sites.stat.washington.edu/wxs/Stat593-s03/Literature/silverman-81a.pdf

W1: Assuming that unlabeled samples are drawn from the same distribution as the labeled samples may violate reality.

Thanks for this point, which we agree is important. We have added an experiment to explore how SSME performs under violations of this assumption. Specifically, we test SSME’s performance when the unlabeled data for CivilComments is drawn from the unlabeled split provided by the WILDS benchmark [1], which is specifically designed to reflect real-world distribution shifts. SSME achieves an average estimation error (in MAE, scaled to same 0-100 scale as the paper) of 2.12 (averaging across 20, 50, and 100 labeled datapoints), compared to 4.15 for labeled data alone and 4.44 and 4.56 for the two best-performing baselines (Bayesian-Calibration and Dawid-Skene). This demonstrates that SSME still provides an advantage over baselines even in the presence of real-world distribution shifts.

Furthermore, we believe the assumption that the unlabeled data is drawn from the same distribution as the labeled data is realistic in several important settings. First, the assumption is met if the annotator/practitioner chooses samples to label at random. For instance, if the labeling budget is small, a practitioner may select a random subset to label. There are also settings where this assumption approximately holds. For instance, in radiology, if one considers the “labeled” images to be all images that have been labeled in the past (e.g., for a particular hospital) and the “unlabeled” images to be the images a radiologist has yet to look at, there is no reason to expect a distribution shift.

[1] https://arxiv.org/abs/2112.05090

W2. Providing running time and computational complexity.

We will include a discussion of computational cost to accompany our empirical results; thank you for the suggestion! Specifically, we will include the following. Fitting SSME is computationally cheap and faster than the best baseline. For every dataset and experiment configuration reported in the paper, SSME can be fit in under 5 minutes, using only 1 CPU. SSME inherits the computational complexity of kernel density estimators (O(n^2)) and EM. In a matched comparison (using 20 labeled examples, 1000 unlabeled examples, and 7 classifiers) the next best-performing baseline (Bayesian-Calibration) takes on average 32.1 seconds to estimate performance, while SSME takes 21.5 seconds.

Please let us know if our response addresses your primary concerns; if so, we would kindly appreciate it if you could raise your score. If not, are there additional concerns we can address?

Thank you for your comments! We appreciate that you believe SSME is able to utilize multiple sources of information simultaneously, has wide applicability, and outperforms baselines in estimating performance.

We address your questions and respond to the highlighted weaknesses below.

Q1. Do the authors consider the selection of classifiers to be a practical issue that is adequately addressed in the paper, or do they believe it is straightforward enough not to pose a concern?

Thanks for this question. To address it, we ran an experiment where we measured SSME’s performance as the classifier set expands and the mix of classifiers within that set changes. In particular, for each number of classifiers between 2 and 7 (inclusive), we use SSME to estimate performance for each unique set of classifiers (so for 7 classifiers, there is exactly one set; for 4 classifiers, there are 7 choose 4 = 35 sets). We performed this experiment on the CivilComments dataset.

As shown in the table below, we find that SSME’s performance improves as the number of classifiers increases, which accords with our theoretical results. As a caveat, our theory predicts that SSME is likely not to perform well if one adds an inaccurate (e.g., worse than random) classifier to the set.

We also find that SSME's performance does not vary greatly across classifier sets, suggesting that SSME’s performance is robust to using different classifier sets. In particular, the standard deviation in accuracy estimation error across random classifier sets is always under 0.011, and the standard deviation in ECE estimation error is always under 0.015, indicating consistent performance across classifier sets.

First, we find that performance does not vary greatly across classifier sets, suggesting that SSME’s performance is robust to using different classifiers. In particular, the standard deviation in accuracy estimation error across random classifier sets is always under 0.011, and the standard deviation in ECE estimation error is always under 0.015.

Second, we find that SSME’s performance improves as the number of classifiers increases, which accords with our theoretical results. As a caveat, our theory predicts that SSME is likely not to perform well if one adds an inaccurate (e.g., worse than random) classifier to the set.

(Dataset: CivilComments) MAE for ECE and accuracy across classifier set choice, along with the standard deviation in MAE across classifier sets.

Q2 + W2 On the assumption that labeled and unlabeled data are IID: do the authors believe that such distributional mismatch would not pose a serious issue in real-world applications? Alternatively, do they see this as a limitation that could be addressed through future methodological improvements?

Thanks for this question! We performed an experiment to assess SSME’s performance under violations of this assumption and found that SSME continues to outperform baselines at small amounts of labeled data. The experiment draws the unlabeled data for CivilComments from the unlabeled data provided by WILDS, which is specifically designed to exhibit realistic distribution shifts. We report the average error in accuracy estimation for each method. SSME achieves an average estimation error (in MAE, scaled to same 0-100 scale as the paper) of 2.12 (averaging across 20, 50, and 100 labeled datapoints), compared to 4.15 for labeled data alone and 4.44 and 4.56 for the two best-performing baselines (Bayesian-Calibration and Dawid-Skene). This demonstrates that SSME still provides an advantage over baselines even in the presence of real-world distribution shifts.

Furthermore, we believe the assumption that the unlabeled data is drawn from the same distribution as the labeled data is realistic in several important settings. The assumption is met if the annotator/practitioner chooses samples to label at random. For instance, if the labeling budget is small, a practitioner may select a random subset to label. There are also settings where this assumption approximately holds. For instance, in radiology, if one considers the “labeled” images to be all images that have been labeled in the past (e.g., for a particular hospital) and the “unlabeled” images to be the images a radiologist has yet to look at, there is no reason to expect a distribution shift.

W1. Simplifying assumptions in theoretical analysis.

Thank you for this question. We use UL+ for the theoretical analysis as it enables us to build on prior results in semi-supervised learning for mixture models, which use this as a standard assumption. To our knowledge, there are no results establishing analogous performance bounds (i.e., bounds relating the amount of data to parameter estimation error) for EM algorithms. The bound derived from analyzing UL+ is likely conservative, as UL+ first uses the unlabeled data to identify decision boundaries then uses the labeled data to assign regions to classes. In general we expect EM, which directly maximizes the log-likelihood of both the labeled and unlabeled data, to perform even better.

W3. Performance variability in specific metrics and scenarios

The reviewer points out that, although SSME performs well overall, it reduces the error more for some metrics than for others. The reviewer is correct that SSME’s performance is strong, outperforming baselines across all metrics and tasks. In particular, SSME outperforms the next-best baseline, Bayesian-Calibration, in every metric tested (2.1x gain in accuracy estimation, 1.5x gain in ECE estimation, 1.2x gain in AUC estimation, and 1.1x gain in AUPRC estimation) averaged across tasks.

We agree that the performance gains are not uniform across metrics. This behavior is consistent with our theory, which shows that error bounds on AUC are looser than those on accuracy (because the error bound on estimated AUC scales differently with separation, see the $\sqrt{2}$ denominator instead of 2 in Equations 3 and 4).

Furthermore, the reviewer points out that under certain assumptions, Bayesian-Calibration might outperform SSME. We agree that Bayesian-Calibration may outperform all other methods if its strong assumptions on the classifier set (monotonic relationship between classifier scores and Pr(Y=1)) exactly hold. However, as our empirical results demonstrate, in real-world settings the assumptions Bayesian-Calibration requires do not hold well enough for it to outperform SSME, which achieves better performance for every metric. Additionally, Bayesian-Calibration is restricted to settings with binary classifiers, a significant limitation that SSME addresses.

Please let us know if our response addresses your primary concerns; if so, we would kindly appreciate it if you could raise your score. If not, are there additional concerns we can address?

Dear Reviewer,

The author have provided their rebuttal with additional results. Please kindly reply to the authors as soon as possible before the discussion period ends.

Thanks a lot.

Best regards,

AC