Defining Expertise: Applications to Treatment Effect Estimation

Dear Reviewer uYNJ, thank you for your thoughtful comments and suggestions! Below, we aim to address all the individual points raised in your review and hope that this alleviates any remaining concerns — please also see the revised paper for changes (highlighted in blue).

(W1) Specific focus on balancing representations

We chose to focus on this specific strand of literature because it has emerged as the predominant way of tackling confounding biases in the heterogeneous treatment effect estimation problem in machine learning over the last years — balancing representations have appeared in a majority of the publications on heterogeneous treatment effect estimation in machine learning conferences (e.g. Johansson et al., ICML 2016; Shalit et al., ICML 2017; Yao et al., NeurIPS 2018; Hassanpour & Greiner, ICLR 2020; Du et al., 2021; Assaad et al., AISTATS 2021; Huan et al., 20222), and also been extended to other settings, such as survival data (e.g. Chapfuwa et al., CHIL 2021; Curth et al., NeurIPS 2021) and time-series data (e.g. Bica et al., ICLR 2020; Melnychuk et al., ICML 2022; Seedat et al., ICML 2022). Given the widespread use of this technique in the machine learning literature, we believe a focus on balancing representations to be warranted for the audience targeted by this venue.

(W2) Relationship to doubly robust estimators

Our work is largely orthogonal to the work on doubly robust estimators from the (bio)statistics and econometrics literature. In particular, the majority of approaches considered in this literature, and in particular also those cited in the review, target the average treatment effect, while we study conditional average treatment effects (CATE, i.e. personalized effects, instead of population-average effects). We focus on estimators based on neural networks because these have been the focus of study in the machine learning literature on CATE estimation since the seminal papers by Shalit, Johansson, and Sontag on the topic in 2016/2017.

Nonetheless, some doubly robust approaches have recently been adapted to the CATE estimation setting, for instance giving rise to DR-learner in Kennedy EH, “Towards optimal doubly robust estimation of heterogeneous causal effects,” arXiv preprint arXiv:2004.14497, 2020. The approaches considered in our paper could be used therein as plug-ins for the outcomes, where improved outcome estimation by making use of expertise would likely improve estimation of CATEs by the DR-learner downstream — for instance, Curth & van der Schaar (2021a,b) showed experimentally that using neural networks with better architectures to estimate the outcome portion of the DR-learner improves the performance of the DR-learner as a whole.

(Q1) Consistency issues with balanced representations for domain adaptation

Thank you for highlighting this interesting connection. The issues arising with balancing representations for treatment effect estimation in the presence of predictive expertise are indeed due to information loss in the representation as analyzed theoretically in Johansson et al. (2019).

Changes: We have now added a footnote in Section 4.1 to highlight that their results could be adapted from their domain adaptation context to our treatment effect estimation context to provide generalization bounds for this setting.

(Q2) Relationship between expertise and mutual information

Our definitions of expertise are exactly equal to the mutual information between actions and outcomes. For predictive expertise, $E^{\pi}\_{*pred*}=I(A^{\pi};Y\_1-Y\_0)/\mathbb{H}[A^{\pi}]$, and for prognostic expertise, $E^{\pi}\_{*prog*}=I(A^{\pi};Y\_0,Y\_1)/\mathbb{H}[A^{\pi}]$. We chose to express expertise in terms of conditional entropies rather than in terms of mutual information because we believe that the former type of expression makes it easier to understand why these definitions are intuitive and why they capture mathematically what we might describe as “expertise” in language.

Changes: We now highlight that our definitions can equivalently be expressed in terms of mutual information in a brand-new appendix (Appendix A).

(Q3) Why does predictive expertise imply prognostic expertise?

Predictive expertise implies prognostic expertise because $Y_1-Y_0$ is a deterministic function of $(Y_0,Y_1)$. If a policy depends on the treatment effect $Y_1-Y_0$ and hence has predictive expertise, we can also say that the same policy depends on the pair $(Y_0,Y_1)$ and hence has prognostic expertise. We can not say the converse — for instance, a policy that only depends on $Y_0$ would have high prognostic expertise, but since it does not depend on $Y_1-Y_0$ directly, it would generally have low predictive expertise.

Formally, the fact that $Y_1-Y_0$ is a deterministic function of $(Y_0,Y_1)$ means that it is generally less informative than $(Y_0,Y_1)$ and at most as informative as $(Y_0,Y_1)$ — knowing the two potential outcomes $Y_0,Y_1$, we can easily compute the treatment effect $Y_1-Y_0$, but not the other way around. We have $\mathbb{H}[Y_1-Y_0|Y_0,Y_1]=0$ therefore $H[A^{\pi}|Y_1-Y_0]\geq H[A^{\pi}|Y_0,Y_1]$ and therefore $E^{\pi}\_{*pred*}\leq E^{\pi}\_{*prog*}$. Since predictive expertise is a lower bound for prognostic expertise, having predictive expertise would also imply having prognostic expertise.

The example of a constant shift in the potential outcomes is not particularly helpful in understanding this relationship. In the case of a policy that depends only on the treatment effect, the treatment assignments would not be affected by such a constant shift. As long as the treatment assignments remain the same, neither predictive expertise nor prognostic expertise would be sensitive to the constant shift — that is the policy would still have the same predictive expertise and prognostic expertise under any constant shift. This is a desirable property of expertise: Expertise should capture how differences in outcomes from one subject to another impact treatment assignments for those subjects independent of the unit/scaling and the zero-point/mean of outcomes.

Changes: We have now stated the fact that predictive expertise is a lower bound for prognostic expertise (hence having predictive expertise implies having prognostic expertise) as Proposition 3 with an accompanying proof (see new Appendix B).

(Q6) Could you please provide a more detailed description of your pipeline process, in particular whether the same samples are used for estimating expertise and treatment effect? Do you anticipate pre-testing problems in this case?

(W4) It is unclear in the proposed pipeline whether the expertise is estimated on same samples used for training or on some hold-out set and what the consequences of this choice have on the outcomes.

When practically implementing Expertise-informed, given a training dataset, we always train two models: Action-predictive and Balancing using the same training dataset. We then use the trained Action-predictive model to estimate predictive and prognostic expertise, and use those estimates to select which model to use during test time on a held-out testing dataset. This means that Expertise-informed never sees the testing dataset until training, selecting, and committing to a particular model for testing. Here, the fact that the same training dataset is used both for model training and model selection should have minimal impact on results because model selection relies on estimating expertise for the entire training dataset while model performance is measured with respect to the accuracy of estimating treatment effects for individual data points. It would have been a different story if model selection also relied on performance in estimating treatment effects (the same task we want our models to achieve).

(Q7) How sensitive are the outcomes of the pipeline to the threshold 1\2?

Varying the threshold used in Expertise-informed would shift its performance between that of Balancing and Action-predictive. As the threshold approaches 0, we expect the performance of Expertise-informed to converge to that of Balancing (as all datasets would be classified as prognostic), and as the threshold approaches 1, we expect its performance to converge to that of Action-predictive (as all datasets would be classified as predictive). An important point to consider is that the performance of Expertise-informed would never drop below the minimum performance between Balancing and Action-predictive as it always selects one of these methods or the other.

Dear Reviewer obyv, thank you for your thoughtful comments and suggestions! Below, we aim to address all the individual points raised in your review and hope that this alleviates any remaining concerns — please also see the revised paper for changes (highlighted in blue).

(Q1) The entropy is defined with log to base 2. Do you decide on this because the actions are binary, i.e. $A^{\pi}\in\{0,1\}$?

Yes, indeed the entropy is defined with $\log_2$ because the actions are binary. This is a fairly arbitrary decision. When $\log_2$ is used, the variability of a uniformly random policy happens to be one.

(Q2) Could you formally prove that $C^{\pi}=0$ implies the violation of the overlap assumption?

(W1) The relationship between in-context action variability and the overlap assumption is just intuitively explained. A formal proof for this relationship is missing.

Yes, it is possible to formally prove this relationship. Thank you for the suggestion.

Changes: We have now stated the fact that having zero in-context action variability implies that the overlap assumption is violated as Proposition 4 with an accompanying proof (see new Appendix B).

(Q3) The comment (i) after Proposition 2 is unclear to me. Could you provide an example for that?

An example can be found in the paragraph titled “How does expertise differ from optimality?” in Section 3.1. There, we considered $\mathbb{E}[Y^{\pi}]$ to be our success measure. While $\pi_{*risk*}$ was suboptimal with respect to this success measure, it happened to be a perfect predictive expert. This is because $\pi_{*risk*}$ was aiming to achieve a different objective: making risk-averse decisions.

(W2) In Figure 1 the axis tick values are missing. (The ticks are also confusing as for $C^{\pi}=0$, the expertise is somewhere between ticks.)

Changes: We have now added tick labels to Figure 1. However, the point of Figure 1 is more conceptual than quantitative, the important information is the relative positions of the three regions (smaller/larger expertise, smaller/larger in-context action variability).

(Q4) Is it possible to give more formal arguments how the discussed CATE estimators (TARNet, IPW, CFRNet, DragonNet) are related to the entropy definition of expertise?

(W3) The authors explain in great detail why some CATE estimator perform intuitively better under some type of expertise than others. A formal proof for this intuition is however missing.

We believe that a theoretical analysis of existing CATE estimators is out of the scope of this initial paper on defining expertise, which not only provides and motivates a formal definition of expertise for the first time, but also theoretically analyzes its relationship to the CATE estimation problem (Propositions 1, 2, and now 3, 4), empirically explores its relationship to the existing CATE estimators, and even highlights model selection as one of its practical use cases.

(Q5) Are the results sensitive to the choice of network based CATE estimators in the analysis?

We have used the same network architecture for all CATE estimators. This means that any similarities and differences that are observed between the estimators should be the result of how the estimators are trained rather than the capacity/complexity of the network architecture they use.

Dear Reviewer ScD8, thank you for your thoughtful comments and suggestions! Below, we aim to address all the individual points raised in your review and hope that this alleviates any remaining concerns — please also see the revised paper for changes (highlighted in blue).

(W1) Prognostic-predictive distinction

We would like to emphasize that the distinction between prognostic and predictive variables is made within the medical and biostatistics literature (e.g. Ballman, 2015; Sechidis et al., 2018) outside of our work. We simply extend the same distinction to policies — that is to say, we differentiate between policies that depend on prognostic variables and policies that depend on predictive variables).

Policies that have predominantly predictive expertise — meaning policies that largely depend on the difference between the two potential outcomes (but not on the potential outcomes individually) — can arise naturally when the goal of a decision-maker is to maximize outcomes. For instance, consider the policy that always selects the action that would lead to the best estimated outcome: $\pi(x)=1$ if $\mathbb{E}[Y_1|X=x]>\mathbb{E}[Y_0|X=x]$ and $\pi(x)=0$ otherwise — it has maximal predictive expertise. Since achieving better outcomes for patients is often the primary goal of clinicians when treatment decisions (e.g. Graham et al., 2007; Caye et al., 2019), such policies can be more prevalent (compared with policies with predominantly prognostic expertise but no predictive expertise) in the healthcare domain.

(W2) Relationship between features, actions, and outcomes

In our framework, features cause actions, and actions (together with features) cause outcomes. Features $X$ have a distribution $\alpha$ independent of any policy, actions $A^{\pi}$ are determined on the basis on features only such that $A^{\pi}\sim\pi(X)$, and outcomes are a joint consequence of both: $Y=\rho_{A^{\pi}}(X)$.

When we say “to what extent the actions of a decision-maker are informed by what a subject’s potential outcomes could be,” our intention is not to imply that a decision-maker’s actions are caused/influenced by potential outcomes, which would be unknown at the time of decision-making. Of course, we agree that in any practical setting, a decision-maker’s actions can only depend on the information that is available to them — in our formulation, this would be features $X$. Some of these features would cause/influence/determine the potential outcomes $Y_0,Y_1$ (let those features be $X_{*critical*}$) and some would not. According to our definition, an expert would mainly consider features $X_{*critical*}$ over $X\setminus X_{*critical*}$, estimate what $Y_0,Y_1$ could be based on these features, and make a decision accordingly. With expertise, we aim to capture to what extent a decision-maker follows this process — meaning to what extent their actions are informed by estimated outcomes $\hat{Y}\_0,\hat{Y}\_1$ based on $X\_{*critical*}$.

Changes: Thank you for bringing the possibly confusing phrasing of our intuitive definition of expertise to our attention. We have now rephrased the sentence to make it clear that actions never directly depend on potential outcomes.

(Q1) Why not use the graphical framework of Pearl?

We refrained from using the graphical framework of Pearl because it is specifically not well-equipped to distinguish between prognostic and predictive variables. This is because DAGs are notoriously bad at representing effect modification (i.e. predictive variables) natively. There have been proposals to extend the graphical framework to better depict effect modifiers (Weinberg CR, “Can DAGs clarify effect modification?” Epidemiology 18, 2007), but to the best of our knowledge, no solution has been well established in the literature to this data.

Changes: Thank you for bringing Pearl’s framework to our attention as an alternative formulation. We have now included this discussion in a brand-new appendix (Appendix A).

Dear Reviewer 7Wu5, thank you for your thoughtful comments and suggestions! Below, we aim to address all the individual points raised in your review and hope that this alleviates any remaining concerns — please also see the revised paper for changes (highlighted in blue).

(Q1) Why entropy over statistical measures?

Although variance, similar to entropy, could also be indicative of how “random” a random variable is, the two quantities measure fundamentally different things: Variance quantifies the amount of spread around a mean while entropy quantifies uncertainty. When defining expertise, our aim is to capture the uncertainty of outcomes when the actions of a decision-maker are known vs. not known to an outside observer (the bigger the difference between the two cases, we say the larger the expertise is). Hence, a more direct measure of uncertainty is naturally more suitable to our aim and use case.

On a more technical level, attempting to define expertise through variance has two immediate shortcomings:

1. First, variance is not well defined for categorical variables, such as binary actions in our work, unless we assign numerical values to each category. Even if we were to assign such numeric values (for instance, $A^{\pi}=0$ for the negative treatment and $A^{\pi}=1$ for the positive treatment), the resulting variance would be sensitive to our assignments (for instance, the variance of $A^{\pi}$ would have increased if we were to represent the negative treatment as $A^{\pi}=-1$ instead of $A^{\pi}=0$, which should not be a meaningful change as far as expertise is concerned).

2. Variance depends on the scale of variables whereas entropy does not. For instance, the variance of $2Y$ would be double the variance of $Y$, while their entropies would be the same: $\mathbb{H}[Y]=\mathbb{H}[2Y]$. Such sensitivity to scale is undesirable when defining expertise — the fact that outcomes are recorded twice as large in a dataset (maybe due to a change of units) should have no effect on expertise.

Changes: Thank you for bringing statistical measures to our attention as an alternative to entropy. We have now included this discussion in a brand-new appendix (Appendix A).

(Q2) Data-driven approaches to discerning domain expertise

Our experiments reveal that different methods are more or less suitable to settings with different types of domain expertise. We strongly agree that this makes it extremely important to accurately discern the type of expertise present in a domain or dataset when selecting between different methods. While machine learning traditionally relies on domain knowledge in making such decisions, our framework enables a more data-driven and quantitative approach: We propose the pipeline “expertise-informed” in Section 4.2 precisely to demonstrate one such approach.

Notice that Expertise-informed first identifies the prominent type of expertise present in a dataset by estimating the measures of prognostic and predictive expertise (and considering their ratio). Then, it takes advantage of this identification by selecting the most appropriate algorithm for estimating treatment effects: Action-predictive for datasets with predominantly predictive expertise vs. Balancing for datasets with predominantly prognostic expertise.

(Q3) Randomized controlled trials

You are correct that, in a randomized controlled trial, when the propensity scores are uniform across treatments, there would be no expertise — and appropriately, both the predictive and the prognostic expertise would be equal to zero for datasets collected through a randomized controlled trial (this can be inferred from Proposition 1, as we mention in the paragraph above Proposition 1, a uniform policy would have $C=1$ hence $E_{*pred*}=0$ and $E_{*prog*}=0$). Consequently, expertise as an inductive bias would of course be less helpful in estimating treatment effects using such datasets, however in that case, the datasets would already be ideal for treatment effect estimation with no confounding bias (the best case scenario in Figure 1). This is why we mainly focused on the high expertise, low overlap setting (the amber region in Figure 1).

While taking advantage of expertise would not be possible for datasets collected through a randomized controlled trial, estimating expertise (as in Section 4.2) can still act as a data-driven way to determine whether the data we have is effectively randomized or not: The closer both $E_{*pred*}$ and $E_{*prog*}$ are to $0$, the closer the data would be to trial data, in which case we might prefer conventional supervised learning methods over algorithms specialized for treatment effect estimation.

Changes: We have now included this discussion in a brand-new appendix (Appendix A).

Many thanks to the AC for the message. The reviewer’s comments have helped us a great deal in improving the paper, and we hope the reviewers will benefit from the discussion as much as we did. We look forward to hearing the reviewers’ replies.