Learning with Selectively Labeled Data from Multiple Decision-makers

We appreciate your valuable feedback. We now address each of your comments below.

• Evidence of (Quasi-)randomly Assigned Heterogeneous Decision-Makers: The selective label literature has mainly considered two examples for randomly assigned heterogeneous decision-makers:

1. Judicial Decision-Making – In US, Judges are often randomly assigned to cases, yet they exhibit heterogeneous decision-making styles and biases (Kleinberg et al., 2018; Marcondes et al., 2019). This example is widely cited in the selective label literature.
2. Loan Approval – In fact, loan applications are also randomly assigned to different loan officers in some places. For example, there even exist platforms, such as OppFi (https://ortooapps.com/case-study/oppfi), that implements the random assignment to ensure fairness and equity. Even if the loan applications are not randomly assigned, they may be viewed as quasi-random as long as the loan officers roughly face the same distribution of applications given the observed features. Moreover, Bushman et al. (2021) demonstrate that individual loan officers significantly influence loan contract terms and performance, highlighting the heterogeneity across decision-makers.

• Real-world Data: Kleinberg et al. (2018) indeed uses real-world data on all arrests made in New York City between 2008 and 2013, but the data are not publicly available. Existing selective label literature also considered a few other datasets. But unfortunately these are also not public. This is why we use synthetic or semi-synthetic data in our paper. However, we hope to clarify that synthetic data actually have advantages over real-world selective label data. This is because in the real data, the outcomes of the unlabeled units are unobservable so evaluating the results from a given algorithm is not straightforward. By using synthetic or semi-synthetic data, we do observe the true outcomes for all units so we can more easily assess different algorithms.

• Decision-maker Assignment as IV: we understand your concern about the point identification assumption and your questioning the validity of view decision-maker assignment as IV. We hope to clarify that the key defining characteristics of IV are exogeneity and exclusion restriction. These two are satisfied by the decision-maker assignment under our assumptions, and many existing selective label works also leverage these two properties in their heuristic solutions. In contrast, the homogeneity assumption (NUCEM assumption) is not the defining characteristic of IV. There exist many works on IV that does not require this assumption, such as the local average treatment effect (LATE) analysis in Angrist et al. (1996) and the Balk-Pearl partial identification bounds. Moreover, one purpose of our point identification is exactly to reveal that it needs strong assumption, which motivates our IV partial identification. In experiments, we do observe that the partial identification approach tends to perform better.

• Comparison with Rambachan et al. (2023): The focus of Rambachan et al. (2023) differs fundamentally from ours. Their work primarily studies evaluating various error measures of a given binary classifier based on IV-based partial bounds, whereas we focus on learning a robust classification rule under both point and partial identification settings. We note that it is not clear how to optimize their error measure estimators to effectively train robust classifiers. In contrast, our work develops a unified cost-sensitive learning (UCL) algorithm for both point and partial identification settings. This requires in-depth analyses of the minimax formulation and cost-sensitive learning problems. So their approach is not directly comparable to ours. We will further clarify this in our revision.

• Specification of Range $a_k$ and $b_k$: In our experiments, we set $a_k(x) = 0$ and $b_k(x) = 1$ for all cases. The partial bounds $l_k(x)$ and $u_k(x)$ are then computed using the estimated nuisance functions along with the specified values of $a_k(x)$ and $b_k(x)$. These nuisance functions are learned from the observed data using the Gradient Boosting algorithm. We will clarify this point in future revisions.

We sincerely thank you for your insightful comments and positive feedback for our work.

• Computational Cost: The computational cost of our method is higher than vanilla method which directly learn a classifier from the observed (selectively labeled) data. That is because our method consists of two steps: The first step is estimation of nuisance functions through a cross-fitting technique, which are used to construct the classification weights; The second step is a weighted / cost-sensitive classification procedure, which can be efficiently solved. The nuisance estimation indeed entails some additional computational costs. But this typically only involves a series of standard regression/classification fitting. This is usually manageable and also widely adopted in the causal inference literature (Chernozhukov et al., 2018). Moreover, this can be accelerated by fitting the nuisances on different folds of data in parallel. We will clarify the computational aspect in our revision.

• Weight Estimation Accuracy: The accuracy of weight estimation can directly impact the performance of our method. In fact, our learning guarantee in Appendix E already captures this. Our excess risk bound in that part involves a term capture the weight estimation error. Notably, the weights to be estimated are different for the point and partial identification settings. Estimating the weight for point identification involves estimating two nuisances and their ratio. In contrast, the weight in the partial identification only involves the sum of a series of nuisance functions and does not involve any ratio. The latter is generally more insensitive to the nuisance estimation error. During the rebuttal period, we conducted some additional experiments for the simulation setting with confounding strength $\alpha_Y=0.5$ and $\alpha_D = 0.7$ (see setting details in Appendix F) to assess the impact of nuisance estimation error. We introduced Gaussian noise into the estimates of nuisance functions to inflate their errors, defined as $\tilde{\eta}(X_i) = \hat{\eta}(X_i) \cdot [1 + \sigma^2 \mathcal{N}(0,1)]$ with noise levels $\sigma = \{0.0, 1.0, 2.0, 3.0, 4.0 \}$. The experiment is repeated for 10 times. We then computed weights based on these noisy estimates and analyzed their effect on resulting classification accuracy. Our findings reveal that both partial and point learning remain stable under small perturbations.
  However, beyond a certain noise threshold, performance degrades significantly. Notably, partial learning exhibits higher tolerance ($\sigma=4.0$) compared to point learning ($\sigma=2.0$). This demonstrates that the partial learning approach is indeed more resilient to nuisance estimation errors. In our revision, we will clarify the impact of nuisance estimation errors and add the extra numerical results.

• First Work to Use IV for the Selective Labels Problem: Thank you for your suggestion on emphasizing the novelty of our work. Previous literature on the selective labels problem (SLP) also leverages the random assignments of heterogeneous decision-makers but their approaches are largely heuristic, as discussed in Appendix A. Our work uses the IV framework to provide principled point and partial identification analyses and derive rigorous learning algorithms. We remark that a closely related work by Rambachan et al. (2023) also considers IV-based partial identification in the context of selective label problem. However, our work is substantially different from theirs and makes many unique contributions. Please see our detailed responses to reviewer aAyP. While we already mentioned these in our paper, we will further highlight them in our revision.

We sincerely appreciate your valuable feedback and your references. Our work indeed builds on these previous literature and is therefore closely related to them. However, our work is not “a special case of these prior works”. Instead, our work significantly generalizes these existing literature and makes contributions beyond them. Below we provide detailed explanations, which we will also clarify in our revised version.

• The Exact Identification in Cui and Tchetgen (2021). They study policy learning under unmeasured confounding using IV, focusing on a binary IV and binary treatment. They aim to learn a confounding robust treatment rule that maps features to a binary treatment. This is more like a binary classification problem with causal structure. In contrast, our work considers a multi-valued IV and a multi-class label, aiming to learn a classification rule that maps features to a multi-class label. This setting strictly generalizes their framework. Notably, their identifying NUCEM assumption involves the difference of certain conditional expectations given the two different IV values. This heavily relies on the binary nature of IV. Instead, we consider a more general assumption that can tackle general IV $Z$. Moreover, their learning procedure only uses hinge loss but we explore more surrogate losses for both binary and multi-class classification.

• The Partial Identification in Pu and Zhang (2021). They explore policy learning under unmeasured confounding with a binary-valued IV, binary treatment, and binary outcome, again learning a robust treatment rule that maps features to a binary treatment. In contrast, we consider multi-valued IVs and multi-class labels. Pu and Zhang (2021) directly uses Balke and Pearl’s partial bounds for the binary IV and binary outcome setting. We extend these bound to accommodate multi-valued IVs and any bounded multi-class outcomes (see Assumption 4.1). Moreover, we rigorously prove the tightness of our IV-based partial identification bounds (Appendix C.1), which to our knowledge is new to the literature. Pu and Zhang (2021) also considers minimax learning. Their inner maximization problem can be easily solved in closed-form for the binary outcome and their final minimization is based on hinge surrogate loss. However, we consider multi-class classification where the inner maximization involves much more complex simplex constraints. We managed to give a closed-form solution by carefully analyzing the problem structure.

• Contextual bandits in Bietti et al. (2021). This paper conducts an empirical analysis of several algorithms for online contextual bandits, which studies a fundamentally different problem from ours. We acknowledge that cost-sensitive classification and surrogate losses are not new problem ideas. But we hope to clarify that our contribution lies in in-depth analyses that transform the problems in both point and partial identification settings into a unified cost-sensitive classification form. Under our general framework, we can explore a range of different surrogate losses while Cui and Tchetgen (2021) and Pu and Zhang (2021) focus on only the hinge loss.

We now further respond to your other comments.

• Efficiency bound under exact identification: We appreciate your suggestion but we think this is beyond the scope of our current paper. The main focus of this paper is to provide a unified learning framework for the selective label problem in both point and partial identification settings. As our work is dense already, we leave the efficiency bound for future study.

• Extension from Discrete to Continuous Outcomes: Thanks for the great question. Extension from discrete to continuous outcomes is indeed an important problem but we think this should be left for a separate future study. Notably, all existing selective label literature focuses on binary outcomes (see references in Appendix A). Our study of multi-class outcomes already constitutes an extension. Moreover, as we discuss above, our work also strictly generalizes Cui and Tchetgen (2021) and Pu and Zhang (2021), instead of being their special cases with restricted outcomes. In our extensions of these prior literature, our paper overcomes many new technical challenges, generalizing many assumptions, analyses and results in these literature. Finally, we hope to briefly touch on the challenges with continuous outcome. Our partial identification analysis involves specifying some bounds $a_k(X)$ and $b_k(X)$ in Assumption 4.1. For classification problems, these can be naturally set as 0 and 1. But for continuous outcome, we may need to specify the range for $E[Y^* | X, U]$, which may has no natural ranges. Moreover, in solving the inner maximization problem, we heavily rely on the simplex structure of the constraints. But such structure no longer applies to continuous outcomes.