When to Act and When to Ask: Policy Learning With Deferral Under Hidden Confounding

Related work: Thank you for pointing this paper out! As this very relevant paper has also been brought up by another reviewer, we choose to comment on it in detail in the “general” comment at the top. We should definitely have caught on to it, and we will refer to it, revise our claims, and discuss it in the revised version of our paper as detailed in the aforementioned comment. We will also make sure to refer and discuss the earlier work by Gao et al. (2021) which deals with the unconfounded case.

Binary action: Indeed our method can be readily extended to multiple actions. We found the presentation to be simpler (including visualizing the intervals as we do in the appendix) for the binary case.

Theoretical results - the bounds validity assumption: The Theorem’s claims hold for any $x$ for which the bounds include the true potential. Thus, if the bounds obey this for a fraction of $1-\delta$ of the $x$’s, the Theorem will hold for the same fraction of cases. Bounds that include quantile components (as the B-learner bounds do) could be robust to a fair degree of noise in the random variable. Oprescu et al. show (in their Corollary 2) that the B-learner bounds are valid on average. More generally, the goal of the theorem is to show that the costs we derived make sense and would lead a model to choose the best course of action.

Experiments - baselines and data: Thank you for the suggestions. We wish to point out that we did compare with an IPS based method: the method of Kallus & Zhou 2020, denoted CRLogit and CRLogit L1 in our experiments. We further attempted to compare with the method of Gao & Yin; however, as we detail in the general comment to all reviewers at the top, we unfortunately could not obtain an implementation of their code or experimental setup, and could not replicate it fully based on the details in the arxiv paper.

We wish to point out that experimenting with real human responses would require an “active” experiment: since this is a causal problem we cannot merely use historical human decisions, but rather we would need to recruit real humans that act as decision makers, in a setup where we know the causal ground truth. While we completely agree that this would be an ideal experiment, we believe that most causal inference papers in the community are not required to attain such a high standard, and we humbly ask the reviewer to take this into account. For example, in the Gau & Yin paper, while data of real human responses is used, they synthesize the risks and the hidden confounding aspects of the data. Similarly, we use the IHDP dataset which includes real human subjects, and induce hidden confounding in it.

Related work presentation - Expanding on Mozannar et al.'s work: Thank you for this important comment. We will make sure to improve our presentation of Mozannar and Sontag [2020] and ensure the paper is self-contained.

Cost function alternatives - conservative vs. optimistic costs: This is an excellent question. Under the validity assumption, the conservative approach is guaranteed to be correct whenever the expert is correct, while the optimistic costs do not have this guarantee. Thus, intuitively the conservative costs would be better in cases where the experts are generally correct. More generally, neither the conservative nor the optimistic case are always superior to each other, as we show now: Let $(x,a,y)$ be a sample where w.l.o.g. $Y(1)>Y(0)$. We have the following sufficient conditions:

1. When the expert is wrong, the conservative costs are strictly better than the optimistic costs when the following holds: $2\hat{Y}^{-}(x, 0) < \hat{Y}^{+}(x, 1)+Y(1) < \hat{Y}^{-}(x, 0)+\hat{Y}^{+}(x, 0)$.

2. When the expert is wrong, the optimistic costs are strictly better than the conservative costs, when the following holds: $2\hat{Y}^{-}(x, 1) < \hat{Y}^{+}(x, 0)+Y(0) < \hat{Y}^{-}(x, 1)+\hat{Y}^{+}(x, 1)$.

If none of these conditions hold we might have that both are wrong or both are correct.

Bounds validity assumption: The Theorem’s claims hold for any $x$ for which the bounds are valid. Thus, if the bounds are valid for a fraction of $1-\delta$ of the $x$’s, the Theorem will hold for the same fraction of cases. The goal of the theorem is to show that the costs we derived indeed make sense and would lead a model to choose the best course of action.

Related work: Thank you for pointing this paper out! As this very relevant paper has also been brought up by another reviewer, we choose to comment on it in detail in the “general” comment at the top. We should definitely have caught on to it, and we will refer to it, revise our claims, and discuss it in the revised version of our paper, as detailed in the aforementioned comment.

Expert’s action space: You are right, this is a typo. The action space is the same as the action space $\mathcal{A}$.

The source of observed data: In principal the only requirement we have is that the process generating the actions in the historical data is the same one generating them at test time; and our method is geared toward the case where the process generating these actions has access to extra information that is not reflected in $X$, but which is correlated with the outcome $Y$. Thus, while we are motivated by human experts, the method is not limited to that case.

CARED policy performance across $\Lambda$ Values: In Figure 2b we see that for large enough $\Lambda$ the CARED policy does indeed change and its value decreases; this can also be seen to a lesser degree in Figure 2a. We will add a line corresponding to the true $\Lambda$ value in Figures 1 and 2, following your suggestion. Notavelyt for the IHDP hidden confounding dataset this $\Lambda$ value will be estimated from the propensity scores, as this dataset was created by hiding a confounder from the original IHDP dataset.

[3] Jesson, Andrew, et al. "Quantifying ignorance in individual-level causal-effect estimates under hidden confounding." International Conference on Machine Learning. PMLR, 2021.

References on Learning to Abstain: Thank you for bringing this paper to our attention. While our paper has been inspired by the work of Mozannar & Sontag on deferral which adopts an end-to-end approach, this work presents an alternative, post-hoc approach to the problem of (non-causal) deferral. We leave it to future work to consider how the approach of Yin et al. above can be adapted to the causal case, where no ground-truth labels are available.

Alternatives of the cost function: This is a very important research direction which can be further extended to a general question - how to choose the best set of costs for a specific problem? Each choice (example average cost) would imply different trade-offs which would be interesting to explore.

Rademacher complexity of NN architectures: Indeed the specific complexity would depend on the architecture. Our point was merely to state that this condition is not vacuous and is fulfilled by a range of widely used approaches. We refer the reader to the following papers which discuss this subject with far more nuance. We will mention this point in the paper.

• Bartlett, Peter L., Dylan J. Foster, and Matus J. Telgarsky. "Spectrally-normalized margin bounds for neural networks." Advances in neural information processing systems 30 (2017).
• Neyshabur, Behnam, et al. "Towards understanding the role of over-parametrization in generalization of neural networks." arXiv preprint arXiv:1805.12076 (2018).
• Golowich, Noah, Alexander Rakhlin, and Ohad Shamir. "Size-independent sample complexity of neural networks." Conference On Learning Theory. PMLR, 2018

Impact of limited expert sample processing: This is an important point. We believe that in many cases the deferral rate would be set apriori according to such limits on the experts’ time and capabilities. This motivates the comparison in Figure 2b which asks “given a fixed deferral rate, which policy would perform the best”: for example, assuming we know that experts can only deal with 30% of the samples, we can fix the deferral rate accordingly.

Reliance on B-Learner for estimating bounds: We assume we are working with observational data, which is the case with real-world applications. In this case, for each sample, we have the observed treatment, and the observed corresponding outcome. That is, for each sample, we only observe one treatment, and will need to estimate the other potential outcomes in order to know what is the right treatment for each sample. Our theoretical guarantees hold for any model that provides bounds on the CAPO and for which Assumptions 2-5 hold. In practice, the algorithm would work with any model that gives bounds on the CAPO even if these assumptions do not hold, though for such a model our theoretical guarantees might not carry over. Thus, the B-Learner is just one possible model that our method can use. However, our choice of the B-Learner is due to the strong guarantees it provides such as validity, sharpness, robustness, and the ability to perform well on moderate amounts of data.

Implicit assumption on the confounded behavior policy: We would be grateful for a further clarification of this comment, as it seems to have been cut short.

Relationship between implicit assumptions on regimes where this approach "does well": This is a fascinating connection, as both our work and the work cited consider explicitly the case of human experts with access to information which the model does not. The work by Stensrud et al. considers a different scenario, where using an instrumental variable and some clever identification results one can “override” the human expert. The main difference is the reliance on instrumental variables for identification, which is a serious drawback as such variables do not always exist. However, in future work we are considering merging their approach with MSM-type bounds. We include a mention of their work and this connection in the revised version.

Exploring policy improvement conditions and expert-ness assumptions impact on loss performance:

1. In the proof of Theorem 1 we outline conditions for when the policy improvement is strict, which relies on whether inequality (12) holds or does not hold. Unfortunately this condition, which can easily occur in practice (as we see in the experiments), does not have a straightforward interpretation in terms of CATE bounds. Our best attempt of explaining it is as follows: Assuming w.l.o.g.that Y(1)>Y(0), then this event occurs when the negative of a “narrow” CATE bound (narrow in the sense of taking the lower bound for the higher potential outcome minus the upper bound for the lower potential outcome) is larger than the lower bound minus the actual potential outcome. We will refer to this condition in the main paper.
2. In general we cannot know for sure whether the experts are mostly correct or not, due to the fundamental problem of causal inference. However, if we assume they are generally correct, then one might wish to use conservative approach and defer more often to the experts.

Comparison to random deferral baseline: Thank you for this excellent suggestion! We attach the new Figure 2b in the general comment at the top. Following your suggestion, we have implemented a random deferral strategy, where the action that is taken in non-deferred cases is based on a T-learner, using the same outcome base-learners as those that were used in the B-learner. The comparison with the random baseline shows that generally speaking, both the B-learner policy and CARED defer “correctly”, i.e. defer cases where they might make incorrect actions, with CARED significantly outperforming the B-learner on a wide range of deferral rates.

Figure 2a results: You are correct to point out that for large log(Λ) B-learner slightly outperforms our approach, and we will be more careful in our presentation of this result. However, we wish to point out that as seen in Figure 2b, what is actually happening is that for similar log(Λ) values the B-learner defers much less than our approach, which explains the difference in performance. When comparing per deferral rate, our approach outperforms B-learner. In addition, it seems that in order to get the B-learner to defer at high-rates one would need to use extremely high values of log(Λ). We will add this more nuanced presentation of the results.

Figure 2b results: Estimating the policy value of a policy which includes human experts and hidden confounding can only be done using a real-world experiment. In principle, when such an experiment is conducted different deferral rates can be compared and the optimal one chosen. Alternatively, one can make various assumptions about the human expert and try to estimate policy values with hidden confounding for such hybrid policies. This is an interesting question which we leave for future research.
In practice, in many cases the deferral rate could be set or constrained a-priori to some narrow interval based on economic and administrative constraints and preferences, for example the capacity of human experts. We will mention this point in the discussion.

Limitation: Static Expert Behavior: A good deal of observational data includes expert actions; for example many electronic health care datasets would include the treatments and medications prescribed by expert clinicians. These same clinician populations might be assisted by an algorithmic model based on our approach.

However, you raise an excellent point regarding the way humans might react to the presence of a model-based action recommendation system. We believe this is a fascinating avenue for future work in collaboration with experts on human decision making: do human experts change their actions when they know their decisions are those that were deferred to them by a machine?

Baselines Presenting: Thank you for this useful suggestion, we will make sure to better explain the baselines and justify the merits of our approach See also our reply to the major question.

Clarification of baseline and differentiation from proposed method: Thank you for this important comment - we indeed should have explained this point better. We will revise and explain in more detail the baseline policy we denote by $\pi_{bounds}^Q$, which defers if the estimated CATE interval crosses 0. This approach is indeed natural and has been used for deferral in many previous papers (including papers which only deal with statistical uncertainty). It was also used in the B-Learner paper.

We will also give a more detailed discussion of the ramifications of Theorem 1: in particular, the Theorem shows that whenever the bounds include the true potential outcome, the action that minimizes our proposed loss function is at least as good as the action implied by the baseline “bounds” policy, as well as the human expert policy. We further show that under very reasonable conditions there will be a non-trivial proportion of cases where the action minimizing our loss function will be strictly better than the one chosen by the bounds policy and expert policy. Finally, our experiments show empirically the added value of our approach compared to these baselines. The reasoning for why our approach might perform better is that it directly optimizes for the deferral classification, as opposed to classifying a sample as “deferred” solely based on some uncertainty measure. This last point has already been made by Mozannar & Sontag 2022 for the non-causal case.

We will make sure to clarify these important points in the revised text.

Human expert access to hidden confounders clarification: When learning how to act based on historical data where the actions in the data were made by human experts, the confounders are by definition factors that affected both the action choice and the outcome. Thus, if at test time we have a human decision maker drawn from the same distribution as those in the historical data, they will by necessity have access to all the confounders. Notably, this is true even if those confounders only affect the decision maker unconsciously: the point is that we have access to actions taken by the “true” policy which generated the historical data. Consider for example the case where a human clinician’s decisions are affected by the demeanor of the patient: this would typically be considered a hidden confounder, as it is not recorded in health records. However, at test time the clinicians can still see the patients’ demeanor and be affected by it - in that sense they have access to a confounder which is hidden from the eyes of any system which is merely based on recorded data.

Deferral costs notation: Indeed, while our approach can be extended to multiple actions using the max operator as in L181, in our paper we focus on the binary case. We will correct this and use the notation for binary actions as in L190.

Algorithm 1: Thank you for your note, we will consider whether we can do without the algorithm, saving the corresponding space.

Bounds validity: This sentence refers to Corollary 2 of Oprescu et al. 2023 [2], which gives a precise statement. Informally it means that with high probability, the average of the estimated upper bounds over the dataset is larger than the average of the true upper bounds given by the MSM. A symmetric result is true for the lower bounds.

[2] Oprescu, Miruna, et al. "B-learner: Quasi-oracle bounds on heterogeneous causal effects under hidden confounding." International Conference on Machine Learning. PMLR, 2023.

CATE bounds' deferral policy explanation: Thank you for the comment. You are correct in pointing this out and we will make sure this point is explained more clearly.

Theorem 2 - Optimization objective: L is a non-convex loss function that is hard to optimize, so we propose optimizing L_CE which is convex, and easier for optimization. We show that L_CE converges to the optimal solution of L. Then, since L_CE is easier to work with, and converges to the desired solution, we show the theoretical guarantees on L_CE

Theorem 2 - Generalization meaning Theorem 2 implies that our optimization objective is reasonable in the sense that attaining good performance on the train set by minimizing it can indeed lead to good performance over unseen samples. In principle one could have had an objective which cannot be learned, or where the errors cannot be controlled even for arbitrarily large samples.

The reason this result is not completely standard is the fact that we optimize a cost sensitive objective where the costs themselves are estimated from data. This is similar in spirit (though not at all in the specifics) to cases in causal inference where objectives weighted by the inverse propensity score are considered, and one has to account for the error in estimating the propensity scores themselves. Our approach using estimated costs turns out to be tractable, but still requires some care in its derivation and proof. As is usually the case in generalization bounds, the result does not translate immediately into a practical guide, but it does imply that the method is expected to respond “well” (i.e., similar to standard supervised learning) to changes in sample size and the complexity of the function class.

Thank you authors for providing responses to reviews. I have updated my rating from 5 to 6.