Pearls from Pebbles: Improved Confidence Functions for Auto-labeling

We appreciate your feedback and acknowledgment of the paper's strengths. Our response:

Clarification on thresholds

The vector $\mathbf{t}$ denotes the thresholds over $k$ classes. $T^{k}$ stands for the space of threshold vectors $\mathbf{t}$. Because we solve a relaxed version of the optimization, using $\hat{\mathbf{t}}'_i$ does not guarantee auto-labeling error below $\epsilon_a$. We estimate the thresholds again in line 16 to ensure the auto-labeling error constraint is strictly followed by the estimated threshold. We have updated the draft to further clarify this point.

Discussion on computation overhead

The wall clock time of our method is similar to other post-hoc methods. For instance, a single run on CIFAR-10 setting on NVIDIA RTX A6000 takes roughly 1.5 hours with post-hoc methods and roughly 1 hour without post-hoc methods. We have included a discussion on the computation overhead in the paper.

Answers

A1. Yes, our current implementation uses a grid in $[0,1]$ as the set $T$. We emphasize that this is not a necessity. We chose it to favor a simple implementation. It boils down to evaluating the number of thresholds equal to the number of validation points given in the round, so the effective size of $T$ is the number of validation points $N_v$. Moreover, a binary search based implementation can reduce the effective size to $\log(N_v)$ for each class.

A2. See the above discussion on thresholds.

A3. It is an interesting idea for future works to make thresholds instance-specific. We chose the current thresholding technique as it is backed by the theoretical guarantees on auto-labeling error and coverage [1].

Limitation. Please see our response A1 above.

[1] Vishwakarma et al., Promises and Pitfalls of Threshold-based Auto-labeling, NeurIPS 2023,

Thanks for the clarification on the thresholds and the additional experiments! My questions and concerns regarding the proposed method have been addressed and I decided to raise the presentation score.

Thank you for your insightful review and for recognizing the key strengths of our paper. Our responses:

Details and effects of hyperparameter selection

Due to space constraints, we deferred a detailed discussion on hyperparameter search to Appendices C.3 to C.7. We plan to incorporate more details in the main paper given the additional space in the camera-ready version. In the rebuttal pdf (Table 5) we provide results with various choices of the penalty term $\lambda$. We observe our method is robust to a wide range for $\lambda$, except around extreme values.

Effect of model architecture choice

In TBAL it is not a priori clear what model the practitioner should use. The overall system is flexible enough to work with any chosen model class. Our focus is on evaluating the effect of various training time and post-hoc methods designed to improve the confidence functions for any given model. To answer the query, we ran experiments with Resnet18 and ViT models in the CIFAR-10 setting (see Table 6 in the rebuttal pdf). As we expected there are variations in the results in the baselines due to model choices but our method maintains high performance irrespective of the classification model used. This is due to its ability to learn confidence scores tailored for TBAL.

Limitations and future work

We discussed these in the conclusions sections and briefly reiterate them here. Colander, like other post-hoc methods, relies on validation data to learn the confidence function. The use of validation data is in general a requirement in TBAL systems as noted in [1]. Reducing or eliminating this dependence is an important direction for future work. It is also related to the general problem of model evaluation in machine learning where solutions based on active testing have emerged [2]. We have updated the draft with additional discussions.

[1] Vishwakarma et al., Promises and Pitfalls of Threshold-based Auto-labeling, NeurIPS, 2023.

[2] Kossen et al., Active Testing: Sample–Efficient Model Evaluation, ICML, 2021.

Thank you for your detailed feedback and for noting the strengths of our paper. Our responses are the following:

On ST/AL works in the experiments

Our response involves the following 3 points:

1. The focus of our paper is to study confidence functions for existing TBAL techniques [1,2,3]. Our main contribution is a method to learn the optimal confidence function for TBAL. Given this focus, we have used various natural and widely used choices of confidence functions as baselines. In particular, we have considered both post-hoc methods and train-time methods used for calibrating confidence functions and thoroughly evaluated our proposed method for learning confidence function vs the combination of train-time and post-hoc methods. ST/AL is not a method for learning or calibrating confidence functions, and hence cannot be used as a baseline in our setting.

2. However, given the overlapping components, we addressed the fundamental differences between TBAL and ST/AL in our paper. Our experiments demonstrate that methods like ST/AL, which first aim to learn the best possible classifier from a given function class before performing auto-labeling, can be severely constrained by the choice of function class, especially when it does not contain a high accuracy classifier. In contrast, for TBAL it is not necessary to learn a classifier with high accuracy. Indeed, TBAL can iteratively auto-label and cover much of a dataset with classifiers that are moderately accurate.

3. Our aim is not to establish TBAL as the state-of-the-art method for data labeling or learning a good model with less data. For this reason, the comparison with ST/AL and other methods is not useful in our work: what we seek to know is whether TBAL itself can be improved with our new confidence funtion approach. An extensive evaluation of TBAL, ST/AL, and other similar techniques would be an interesting work. The differences in the goal -- labeling data with reliability guarantees versus learning the best model in the class with fewer labels are important distinctions and these are discussed in detail along with thorough empirical comparisons in [1].

On indirect comparison due to multiple rounds

We agree that multiple rounds of auto-labeling and data selection make the comparison indirect. However, it is evident that the gains are due to using Colander since we keep all the other components as it is and only change the confidence functions. To make this more clear, we have run experiments in the single round (passive learning) setting as well, and our results are consistent with the multi-round setting. Reinforcing the fact that the performance improvements are due to Colander, Table 4 in the pdf shows results in single round setting. The evolution of coverage and error over multiple rounds in the multi-round setting is shown in Figure 1.

[1] Promises and Pitfalls of Threshold-based Auto-labeling, Vishwakarma et al., NeurIPS, 2023,

[2] MCAL: Minimum Cost Human-Machine Active Labeling, Qiu et al. ICLR, 2023.

[3] AWS Sagemaker Ground Truth https://aws.amazon.com/sagemaker/data-labeling/

Thank you for your positive and encouraging review. We are delighted with your assessment and recognition of our work’s contributions. Our response:

Dependence on validation data. Yes, Colander, like other post-hoc methods, relies on validation data to learn the confidence function. In fact, this is in general a requirement in TBAL systems as noted in [4] and your reference [1]. Reducing or eliminating this dependence is a useful direction for future work. It is also related to the general problem of model evaluation in machine learning where solutions based on active testing have emerged [3].

Question on a related work. We appreciate the suggestion of the related paper [2]. This work uses softmax scores in a similar pipeline augmented with external knowledge sources. In our experiments, we have a comparison with the softmax scores. On the other hand, we believe using scores from Colander could be helpful in [2]. We have included a discussion on this idea in the draft.

[1] https://aclanthology.org/2023.acl-long.796.pdf

[2] https://aclanthology.org/2021.naacl-main.84.pdf

[3] http://proceedings.mlr.press/v139/kossen21a/kossen21a.pdf

[4] https://openreview.net/pdf?id=RUCFAKNDb2

We appreciate the detailed review. We have updated our work to account for points on notation and writing. Our response:

Contributions

We believe the reviewer has missed our paper's key contribution, specifically with respect to prior work on TBAL [1]. Our work does not reiterate [1]. Instead, the weaknesses of the TBAL technique in [1] inspired our work. That is, [1] uses a simple, fixed confidence function that leads to poor performance in terms of error rate and coverage. Our goal is to understand the impact of this choice and to produce innovations that resolve these weaknesses, as illustrated in our draft's Figure 2. This is important given wide industry adoption of TBAL [2]. We:

• demonstrate the importance of confidence functions in TBAL, evaluate several choices of functions, and find them to be severely limited---as they are not well-aligned with TBAL objectives.
• propose a framework to learn the optimal confidence function and provide a practical version of it.
• provide extensive empirical evaluations, showing that using confidence functions learned via Colander in TBAL improves auto-labeling coverage significantly.

Theoretical Analysis

Our main contribution lies in demonstrating the issues with common choices of confidence functions in TBAL, proposing a principled solution to learn the optimal confidence functions for TBAL, and showing its effectiveness empirically. A rigorous theoretical analysis of our method is left as future work.

Completeness

• We have fixed notation and typos.
• ActiveQuery is discussed in Appendix B.3 (lines 583-603). We have added a detailed algorithm.
• We use scoring and confidence functions synonymously.
• The terms inherent tension and trade-off indeed refer to similar behaviors.
• Train-time and post-hoc calibration methods are standard terms: Train-time methods modify the training procedure while post-hoc methods operate on trained classifiers.
• Fig. 1 shows a simplified workflow of the standard procedure highlighting the role of confidence functions. We have set out to make the figure readable without the notations as well; we kept the notations for consistency with later figures.
• It is the Appendix B.2 (line 571 onwards)
• $\hat{\sigma}$ is the standard deviation of the error estimate. We have included the formula.

Active Learning vs. TBAL

Our answer:

1. The details of the AL setup and experiment are in Appendix A.1. We added further detail.
2. The aim of the AL experiment is to highlight the fundamental difference between TBAL and procedures (such as AL) that seek to first learn the best possible classifier from the given function class and then do auto-labeling. These methods are severely limited by the choice of function class. On the other hand, TBAL succeeds since it iteratively auto-labels and eliminates the auto-labeled space. The comparison is fair as the methods use same amount of labeled data, same function classes, and training to learn the model.
3. We adhered to the workflow of TBAL from [1]. While other ways to involve the human labeler are possible, these would not permit a comparison to [1].

On Colander

1. Because we solve a relaxed version of the optimization, $\hat{\mathbf{t}}'_i$ is not guaranteed to ensure auto-labeling error below $\epsilon_a$. We estimate the thresholds again in line 16, to ensure the auto-labeling error constraint is strictly followed by the estimated threshold.
2. The optimization process does not require any specific performance assumptions on the classifier. It has to be better than a random classifier to get any meaningful output from the procedure; it is not a strong requirement.
3. Colander does not learn multiple functions to choose from. In each round, it takes the current classifier and learns a confidence function by solving P3.
4. Colander can use any function class for $g$. In experiments, we chose 2-layer nets and successfully used the same across all datasets, thus we may not need an exhaustive architecture search. Intuitively, we do not need a large network for $g$ since $h$ already performs the heavier representation learning work. As a result, simple models are preferable to avoid overfitting and to reduce training time (since post-hoc methods should be fast).
5. TBAL learns a new classifier in every round, thus the confidence function also needs to be relearned. We comment on the time below.

Baselines

We use prominent and recent post-hoc and train-time baselines. Thank you for the references. Of these [3] fits our setting. We provide results in the attached pdf (Table 1); these are consistent with the draft. The key insight is that baselines are not tailored to the TBAL objective. Colander is designed to maximize TBAL performance. Figure 2 and its discussion make this point.

Empirical results

1. Our primary focus is on confidence functions, and to avoid confounding, we have chosen not to compare various active querying strategies within this work. Including multiple variables would obscure the effects of our method.
2. Could you please clarify this question?
3. The wall clock time of our method is similar to other post-hoc methods: a single CIFAR-10 run on an NVIDIA RTX A6000 is roughly 1.5 hours (post-hoc) and 1 hour (no post-hoc).
4. Our framework is flexible to work with any choice of $\epsilon_a \in (0,1)$. In experiments, we used a fixed \epsilon_a=0.05\. We provide additional results in Table 2 in the pdf with varying $\epsilon_a$ to demonstrate its effect.
5. Results with $C_1 \in${$0.0, 0.25$} in 20-newsgroup setting with vanilla train-time and all post-hoc methods are in Table 3. $C_1=0.0$ leads to higher variance in auto-labeling error, consistent with previous works [1].

[1] Vishwakarma et al., Promises and Pitfalls of Threshold-based Auto-labeling, NeurIPS 2023.

[2] https://aws.amazon.com/sagemaker/data-labeling

[3] Joy et al., Sample-dependent Adaptive Temperature Scaling for Improved Calibration, AAAI 2023.