Weak Supervision Performance Evaluation via Partial Identification

Dear reviewer, thank you for your dedication to our paper. We addressed the issues you raised below. Please let us know if you have any other questions.

• Regarding weak label pruning: Thank you for raising this point. In the following, we explain our point of view, which we will make clearer in the main text. Pruning not-so-informative weak labels (even though they carry some signal) could be beneficial in practical situations. This is true for at least one reason (besides the one you mentioned): the number of parameters in the label model can increase exponentially fast with the number of weak labels, making estimation very hard (convergence rates are slow in parametric inference when the number of parameters is big, for example). As a consequence, having too many weak labels that are not very informative can lead to problems analogous to overfitting if the data is not big enough.

• Applications with end-to-end weak supervision methods: Our method works with all of these methods given that we can fit a label model (e.g., Snorkel) separately and apply it just to the evaluation step. Thank you for your comment. We will write this observation in our main text and cite the papers you suggested.

Dear reviewer, thank you for your work on our paper. We addressed the issues you raised below. Please let us know if you have any questions.

• Clarity and appendix material: Thanks for your suggestion. We will revise to reference the appendix where appropriate and give some ideas on the basic tools used to obtain the results.

• Applications or circumstances in which this method could be used for real-world problems: We will provide additional details on this point in the conclusion section. In summary, applications primarily focus on performance estimation and model selection, particularly in areas such as image or text classification, as demonstrated in the experiments section. Additionally, the label model does not need to be perfect to offer useful bounds, as evidenced in the experiments, where we compare a very simple label model (e.g., Snorkel) vs the oracle estimation of $P\_{Y|Z}$. Concerning the number of weak labels, Appendix A.2 indeed provides results that help determine when we have a sufficient quantity of weak labels and their level of informativeness.

Dear reviewer, thank you for your dedication to our paper. We realized that the main issues you raised are related to clarity and presentation. We revised the main parts you brought up and are willing to further improve other parts depending on your feedback. Please let us know if you have more issues/questions.

PWS over noisy labels approaches

Our method could be potentially applied to noisy label applications as well if $P\_{Y|Z}$ can be estimated without the conditional independence assumption that $Y$ is independent of $X$ given $Z$. However, we are not aware of any paper that proposes such a method in the noisy labels literature. If the reviewer has any suggestions, we would be happy to include a note in our paper about this.

Interpretation of Theorem 2.4

Thanks for the feedback. We have added the following sentences right after the theorem statement to make the interpretation clear:

Theorem 2.4 tells us that, if the label model is consistent (Assumption 2.3), under some mild regularity conditions (Assumption 2.2), our estimators $\hat{L}\_\varepsilon$ and $\hat{U}_\varepsilon$ will be asymptotically Gaussian with means $L\_\varepsilon$ and $U\_\varepsilon$ and variances $\sigma^2\_{l,\varepsilon}/n$ and $\sigma^2\_{u,\varepsilon}/n$. That is, the estimates will be close to $L\_\varepsilon$ and $U\_\varepsilon$ up to a Gaussian estimation error with a standard deviation of order $\mathcal{O}\_P(1/\sqrt{n})$.

Figure 1

Thanks for the feedback. We have added the following sentence as the caption of Figure 1 (and a slightly different version of it to the main text) to make its interpretation clear:

We apply our method to bound test metrics such as accuracy and F1 score (in green) when no true labels are used to estimate performance. In the first row ("Oracle"), we use true labels to estimate the conditional distribution $P_{Y\mid Z}$, thus approximating a scenario in which the label model is reasonably specified. On the second row ("Snorkel"), we use a label model to estimate $P_{Y\mid Z}$ without access to any true labels. Despite potential misspecification in Snorkel's label model, it performs comparably to using labels to estimate $P_{Y\mid Z}$, giving approximate but meaningful bounds.

Regarding quality, we will work on the quality of Figure 1 by making it larger and with better resolution (let us know if there is any other issue with that figure that we will be happy to solve).

Introduction

Thank you for your feedback. We have revised the introduction to make it clearer:

Programmatic weak supervision (PWS) is a modern learning paradigm that allows practitioners to train their supervised models without the immediate need for clean labels $Y$ [37, 35, 34, 36, 43, 47]. In PWS, practitioners first acquire cheap and abundant weak labels $Z$ through heuristics, crowdsourcing, external APIs, and pretrained models, which serve as proxies for $Y$. Then, the practitioner fits a label model, i.e., a graphical model for $P\_{Y,Z}$ [37, 36, 17, 12], which, under appropriate modeling assumptions, can be fitted without requiring $Y$'s. Finally, a predictor $h:\mathcal{X}\to\mathcal{Y}$ is trained using a noise-aware loss constructed using this fitted label model [37]. Using weak labels for prediction is usually not possible because they are not observed during test time; thus training a final classifier $h$ that depends only on features $X$ is needed.

One major unsolved issue with the weak supervision approach is that even if we knew $P\_{Y, Z}$, the risk $R(h)=\mathbb{E}[\ell(h(X), Y)]$ or any other metrics such as accuracy/recall/precision/$F\_1$ score are not identifiable (not uniquely specified) because the joint distribution $P\_{X,Y}$ is unknown while the marginals $P\_{X, Z}$ and $P\_{Y, Z}$ are known or can be approximated. As a consequence, any performance metric based on $h$ cannot be estimated without making extra strong assumptions, e.g., $X$ is independent of $Y$ given $Z$, or assuming the availability of some labeled samples. Unfortunately, these conditions are unlikely to arise in many situations. A recent work, Zhu et al [48] , investigated the role and importance of clean labels on model evaluation in the weak supervision literature. They determined that, under the current situation, the good performance and applicability of weakly supervised classifiers heavily rely on the presence of at least some high-quality labels, which undermines the purpose of using weak supervision since models can be directly fine-tuned on those labels and achieve similar performance. Therefore, in this work, we develop new evaluation methods that can be used without any clean labels and show that the performance of weakly supervised models can be accurately estimated in many cases, even permitting successful model selection. Our solution relies on estimating Fréchet bounds, explained below, for bounding performance metrics such as accuracy, precision, recall, and $F\_1$ score of classifiers trained with weak supervision.

Fréchet bounds Consider a random vector $(X,Y,Z)\in \mathcal{X}\times\mathcal{Y}\times\mathcal{Z}$ is drawn from an unknown distribution $P$. We will add more information about Fréchet bounds here (omitted due to lack of space).

Contributions In summary, our main contributions are:

1. Developing a practical algorithm for estimating the Fréchet bounds in Equation (1). Our algorithm can be summarized as solving convex programs and is scalable to high-dimensional distributions.
2. Quantifying the uncertainty in the computed bounds due to uncertainty in the prescribed marginals by deriving the asymptotic distribution for our estimators.
3. Applying our method to bounding the accuracy, precision, recall, and $F_1$ score of classifiers trained with weak supervision. This enables practitioners to evaluate classifiers in weak supervision settings without access to labels.

Dear reviewer, thank you for your work on our paper. We addressed the issues you raised below. Please let us know if you have any other questions.

• Why not estimate performance metrics directly? In fact, we work under the assumption that no true labels are observed, which is a realistic scenario in the programmatic weak supervision literature. Therefore, it is impossible to compute metrics such as accuracy, recall/precision, F1 score, etc. However, using the label models proposed in the weak supervision literature, it is still possible to estimate $P_{Y|Z}$ (even with no labels $Y$'s; see for example [1]). Therefore, obtaining lower and upper bounds for performance metrics (accuracy, recall/precision, F1 score) is the best that one can do; we focus on that problem. Moreover, even if some labeled samples are given, our method can enjoy superior performance in some tasks (see Section 4.2.1).

• Label space size and the method applicability: Whether the label space needs to be small or not, really depends on the sample size you have. The key point is that the label model needs to be well estimated. This will happen (no matter the size of the label space) if the sample size is big enough for a given label model complexity (e.g., label models with fewer graph edges need fewer samples to be well estimated).

• High probability results: It is possible to obtain a high-probability version of Thm 2.4. We opted for the asymptotic version because we rely on it to construct confidence intervals later. Although it is possible to form confidence intervals based on finite-sample high probability bounds, the resulting intervals are typically very conservative. This is why we opted for asymptotic results.

• Variance of the estimates: Yes, that is correct. At a high level, there are two main error terms in ${\\widehat{L}}\_{\\epsilon}$ and $\\widehat{U}\_\\epsilon$: an $O\_P(n^{-1/2})$-term and an $O\_P(m^{-\\lambda})$ term. In Thm 2.4, we assume $n = o(m^{2\\lambda})$, which implies the $O\_P(m^{-\lambda})$ error term is asymptotically negligible. This simplifies the theorem because it allows us to state its conclusions solely in terms of $n$. In your scenario, $\lambda = \frac12$ so the two error terms vanish at the same square-root rate.

References

[1] Alex Ratner, Braden Hancock, Jared Dunnmon, Roger Goldman, and Christopher Ré. Snorkel metal: Weak supervision for multi-task learning. In Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning, pages 1–4, 2018.