Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets

[W1] In our paper, we follow the same notation used in Elements of Causal Inference by Peters, Janzing and Schölkopf, a standard book in causality research with over 2,000 citations in Google Scholar, also used in [18] (ICML 24). In particular, under the notation used by Peters et al. to denote interventions in a counterfactual distribution in Chapter 3.3, we think our current definition is correct. In Definition 1, the distribution $P^{\mathcal{M} ; do(\Lambda = \lambda) | \hat{Y} = \hat{y}, X = x, Y=y}(\hat{Y})$ denotes the counterfactual distribution entailed by the structural causal model $\mathcal{M}$ under the intervention $do(\Lambda = \lambda)$ given the observation $\hat{Y} = \hat{y}, X = x, Y=y$. We would prefer to keep the notation as is because it explicitly notes the (counterfactual) distribution used in the expectation. However, to avoid misunderstandings, we will describe our notation in the revised paper.

[W2] In the notation we use, following Peters et al. as noted above, the definition by Richens et al. is as follows: $h_{\lambda}(x, y, \hat{y}) = E\_{\hat{Y} \sim P^{\mathcal{M} | \hat{Y} = \hat{y}, X = x, Y=y}(\hat{Y})}[ \max\\{0, \Bbb{1}\\{\hat{Y} = y\\} - \Bbb{1}\\{\hat{y} = y\\}\\} ],$ where $x, y, \hat{y} \sim P^{\mathcal{M} ; do(\Lambda=\lambda)}$ and we consider $\lambda=1$ to be the default action (in Richen's language). In contrast, our definition is as follows: $h_{\lambda}(x, y, \hat{y}) = E_{\hat{Y} \sim P^{\mathcal{M} ; do(\Lambda = \lambda) | \hat{Y} = \hat{y}, X = x, Y=y}(\hat{Y})}[ \max\\{0, \Bbb{1}\\{\hat{y} = y\\} - \Bbb{1}\\{\hat{Y} = y\\} \\} ],$ where $x, y, \hat{y} \sim P^{\mathcal{M}}$. Under Richen's definition, counterfactual harm depends on an observation $x, y, \hat{y} \sim P^{\mathcal{M} ; do(\Lambda=\lambda)}$, which comprises a setting in which the human expert makes a prediction from the prediction set $\mathcal{C}\_{\lambda}(x)$ (i.e, the decision support system $\mathcal{C}_{\lambda}$ is deployed). In contrast, under our definition, counterfactual harm depends on an observation $x, y, \hat{y} \sim P^{\mathcal{M}}$, which comprises a setting in which the human expert makes a prediction on their own.

In the notation suggested by the reviewer, the definition by Richens et al. is as follows: $h_{\lambda}(x, y, \hat{y}) = \mathbb{E}[\max\\{0, \Bbb{1}\\{\hat{Y}\_{1} = y \\} - \Bbb{1}\\{\hat{y} = y \\} \\} | X=x, Y=y, \hat{Y}\_{\lambda}=\hat{y}],$ where $x, y, \hat{y} \sim P(X, Y, \hat{Y}\_{\lambda})$ and the expectation is over the prediction $\hat{Y}\_{1}$ made by the expert on their own and we consider $\lambda=1$ to be the default action. In contrast, our definition is as follows: $h_{\lambda}(x, y, \hat{y}) = \mathbb{E}[\max\\{0, \Bbb{1}\\{\hat{y} = y \\} - \Bbb{1}\\{\hat{Y}\_{\lambda} = y\\} \\} | X=x, Y=y, \hat{Y}\_{1}=\hat{y}],$ where $x, y, \hat{y} \sim P(X, Y, \hat{Y}\_{1})$ and the expectation is over the prediction $\hat{Y}\_{\lambda}$ made by the expert using the system $\mathcal{C}\_{\lambda}$.

Following the reviewer's advice, we will clarify this in a separate Appendix.

[W3] As noted above, we follow standard notation in causality research, which we would respectfully prefer to keep, though, we will explicitly clarify that the mechanisms are not equations that can be manipulated in the revised paper.

[Q1] To avoid misunderstandings, we will rephrase lines 147-148 to clarify that we cannot expect to find $\Lambda (\alpha)$ from observational data as counterfactual harm is not identifiable from observational data without any further assumptions. However, we would like to clarify that, since $\hat{Y}$ and $\hat{Y}_{\lambda}$ are the same variable under no intervention $(\Lambda=1)$ and under the intervention $do(\Lambda=\lambda)$, it is not possible to simultaneously measure them, therefore it is not sensible to consider their joint distribution as argued in Pearl, J. (2010). An introduction to causal inference. The international journal of biostatistics, 6(2) (page 25).

[Q2] We would like to clarify that our goal is not to minimize counterfactual harm but rather to find the largest harm-controlling set $\Lambda(\alpha)$ for a user-specified harm bound $\alpha$, as described in lines 144-150. Among the $\lambda$ values in $\Lambda(\alpha)$, one could later on find the $\lambda$ value that maximizes some utility, e.g., accuracy by using the methods in [17,18], or the counterfactual benefit, though this remains outside the scope of our work.

Removing the max from our definition of counterfactual harm would result in measuring the accuracy of the human expert under no interventions minus the accuracy of their counterfactual predictions had they used the system. That said, removing the max would not solve the identifiability issue, as this is due to the counterfactual predictions in the definition.

To the best of our knowledge, we have found no prior work on conformal risk control using $\lambda = \lambda(x)$ to construct the prediction sets, however, a recent work by Gibbs et al. (Conformal Prediction With Conditional Guarantees, Arxiv 2023) uses threshold values that depend on x in the context of conformal prediction. We will add this in our discussion in Section 7.

[Q3] We agree with the reviewer that different incorrect classifications may have different consequences—a cold is much more similar to a flu rather than lung cancer in terms of symptoms and treatment. In that context, we acknowledge that, for simplicity, our definition of counterfactual harm treats all incorrect classifications as equally harmful. Expanding our definition of harm to weigh different incorrect classifications according to their consequences would be a very interesting avenue for future work, however, it may be challenging to quantify/measure the consequences. We will discuss this in Section 7 under Assumptions.

[Q4] We will expand the discussion in lines 127-130 to include the suggested sentence.

Dear authors,

Thanks for your detailed response. You have addressed some of my concerns, and I will just share some follow-ups:

(W1) Although I am familiar with the textbook you mention, I was not aware of the do() notation being used for counterfactuals. It does seem that your definition is then correct, I am a happy for you to keep the notation you prefer (I thought there was a genuine mistake). However, two things to keep in mind: (1) I am not sure if there is any place where you mention where your notation is coming from, which would be a very good thing to add. (2) My comment about the notation is still pertinent, in the sense that readers familiar with the framework of the Causality book (Pearl, 2000) will be confused by the notation (this book has 27,000 citations on Google Scholar, if you consider that a relevant metric). For this reason, it may be helpful to have a sentence in the preliminaries that mentions the difference in the notation.

(Q2)

We would like to clarify that our goal is not to minimize counterfactual harm but rather to find the largest harm-controlling set ...

I am aware of the scope of the paper. However, given that you are spending significant energy on studying this topic, and are cleary doing some good work, I would expect a better justification for how this effort fits into the bigger picture. In other words, I would expect some simple example/illustration to demonstrate how harm-controlling sets you discuss help solve a problem that is not handled by other existing methods. I hope you can share some thoughts, since I would be quite interested to learn about this.

[Eq. 4] In Eq. 4, the true label $Y$ is not a function of the predicted label $\hat{Y}$. The expectation is over the predicted label $\hat{Y}$ and, following standard notation used elsewhere, the distribution $P^{\mathcal{M} ; do(\Lambda = \lambda) | \hat{Y} = \hat{y}, X = x, Y=y}(\hat{Y})$ denotes the counterfactual distribution entailed by the structural causal model $\mathcal{M}$ under the intervention $do(\Lambda = \lambda)$ given the observation $\hat{Y} = \hat{y}, X = x, Y=y$.

[Line 164] The distribution of the noise $v$ is conditioned (only) on $X=x$ because, in Eq. 6, the variable $X=x$ is fixed but the variables $Y$ and $\hat{Y}$ are not fixed.

[Lines 127-130] We state the former.

[Proof of Proposition 1] We used the CF monotonicity assumption in the 5th line of the proof where we used that $\Bbb{1}\\{\hat{Y} = y\\} = 1$ if $y \in \mathcal{C}\_{\lambda}(x)$. We will clarify this step in the revised version of the paper.

[On CF monotonicity] Under the CF monotonicity, any conformal predictor that guarantees $1-\alpha$ coverage is guaranteed to cause average counterfactual harm smaller or equal than $\alpha$. However, there may be conformal predictors that guarantee $1-\alpha$ coverage and cause average counterfactual harm smaller than a strictly smaller bound $\alpha' < \alpha$. This is because the average counterfactual harm does not only depend on the coverage probability but also the probability that a human predicts the ground-truth label successfully on their own.

[Examples for CF monotonicity violations] In general, since counterfactuals lie within level three in the “ladder of causation” [19], it is not possible to identify CF monotonicity violations directly from observational data. A violation of counterfactual monotonicity may happen in the following situation. Let $\mathcal{S} \subseteq \mathcal{T}$ be two prediction sets and assume both sets contain the true label $y$. If there is a label value $y' \in \mathcal{T} \backslash \mathcal{S}$ that help the human expert realize that $y$ is the true label, the expert may succeed at predicting $y$ in $\mathcal{T}$ but may fail at prediction $y$ in $\mathcal{S}$.

[Tradeoff between CF harm and conformal accuracy] We would like to clarify that, if one sets the value of the threshold $\lambda$ to be roughly the $1-\alpha$ quantile of the empirical distribution of the scores $1-m_{y}(x)$ in the calibration set, then, the set valued predictor defined by Eq. 1 is equivalent to a vanilla conformal predictor with nonconformity scores $1-m_{y}(x)$ and coverage $1-\alpha$. This type of conformal predictors are the state of the art for decision support systems based on prediction sets [17, 18]. Under this view, it becomes apparent that, given the same amount of coverage guarantee, there is not a trade-off between prediction set size and average counterfactual harm and, by searching for $\lambda$ values in $[0,1]$ that are harm controlling, we are essentially searching for vanilla conformal predictors that are harm controlling.

[Conformal baselines] As discussed in our previous answer, the class of set valued predictors defined by Eq. 1 are equivalent to the class of vanilla conformal predictors with nonconformity scores $1-m_{y}(x)$. Moreover, since our focus is on (counterfactual harm of) decision support, we have used the average accuracy of the predictions made by humans using prediction sets constructed by the above set-valued predictors as evaluation metric, similarly as the state of the art [17,18]. However, following the reviewer's suggestion, we have additionally evaluated all the set-valued predictors considered in our experiments in terms of empirical coverage (conformal accuracy) and prediction set size against average counterfactual harm. In the pdf attached with the rebuttal, we summarize the results for the set-valued predictors used in Figure 2.

[Conformalizing $\hat{Y}$] Given a set of set-valued predictors and a user-specified bound $\alpha$, the goal of our work is to identify the subset of them that cause average counterfactual harm smaller than $\alpha$. To this end, we have considered the set of set-valued predictors defined by Eq. 1, which as discussed above, are equivalent to the set of vanilla conformal predictors with nonconformity scores $1-m_{y}(x)$ and are the state of the art for decision support systems based on prediction sets [17, 18].

Consequently, given the set of set-valued predictors suggested by the reviewer, or other conformal predictors with different nonconformity scores, one could think of extending our framework to identify the subset of these predictors that cause average counterfactual harm smaller than $\alpha$, however, we leave this as future work. We will clarify this in Section 7 under Methodology.

We would like to thank the reviewer for their follow-up message. Below, we provide a point-by-point reply. Please, do not hesitate to follow up further if something is unclear.

1. In Eq. 4, the true label is not a function of the predicted label. Here, note that $(\hat{Y})$ is not part of the superscript. The expression refers to the probability of the random variable $\hat{Y}$ under the distribution $P^{\mathcal{M}; do(\Lambda=\lambda) | \hat{Y} = \hat{y}, X = x, Y=y}$.

2. In the definition of CF monotonicity we follow from Straitouri et al. [18], for a fixed $X=x$, the inequality in Eq. 6 needs to hold for any $u$ and $v$ such that $X=x$. However, for different $v$’s such that $X=x$, one may have different $Y$ values since, in the SCM in Eq. 3, we do not make any assumption on $f_X$ and $f_Y$.

3. and 5. Given a conformal predictor with coverage $1-\alpha$, our framework can be used to predict whether the conformal predictor causes average counterfactual harm smaller than a user-specified bound $\alpha' \in [0, 1]$, with $\alpha' \leq \alpha$. In our experiments, we applied our framework to the SOTA conformal predictors by Straitouri et al. [17,18]. Therefore, in Figure 2 in the one page pdf attached with the rebuttal, we show the (a) average set size and (b) coverage vs the average counterfactual harm achieved by the SOTA conformal predictors by Straitouri et al. [17, 18]. Given this clarification, we are not sure what it would mean to compare these SOTA conformal predictors with our method in terms of prediction set size.

4. In the benchmark datasets we used in our experiments, we cannot identify counterfactual monotonicity violations using the mixture of MNLs because the mixture of MNL is not a causal model—it is just a prediction model used to estimate the accuracy of the predictions made by humans using prediction sets. However, we would further like to clarify that counterfactual harm may occur even if there are no violations of counterfactual monotonicity. If there are no violations of counterfactual monotonicity, the average counterfactual harm is given by Eq. 8. Therein, it is apparent that, as long as there are samples for which a human expert successfully predicts the true label on their own but the corresponding prediction sets do not include the true label, counterfactual harm will occur. Further, if there are violations of counterfactual monotonicity, the right-hand side of Eq. 8 is a lower bound on the counterfactual harm. Then, based on the results shown in Figures 2 and 3, we can conclude that there exists counterfactual harm irrespective of whether counterfactual monotonicity holds.

[y-hat & Y-hat] The random variable $\hat{Y}$ always denotes the expert prediction from a prediction set $\mathcal{C}\_{\Lambda}(X)$ following the SCM specified in Eq.3. Under no intervention, $\Lambda=1$ and thus $\mathcal{C}_{\Lambda}(X) = \mathcal{Y}$, as noted in Line 131-132, i.e., the human expert predicts on their own. The variable $\hat{y}$ is always a realization of the random variable $\hat{Y}$, as clarified in Page 3, Footnote 2. Consequently, in Line 142, the variable $\hat{y}$ is the realization of the random variable $\hat{Y}$ under no intervention, i.e., $\Lambda = 1$.

[Estimated accuracy] To compare the average accuracy estimated using the mixture of MNLs to the average accuracy estimated using predictions made by human participants, we resort to the dataset made publicly available by Straitouri et al. [18]. Unfortunately, this dataset only comprises human predictions for $\omega = 110$ and a single classifier. However, since the reported cost for collecting such data was 7,150 British pounds [18, Section 6], we could not afford gathering additional data to assess the quality of the average accuracy estimated using the mixture of MNLs for other classifiers and $\omega$ values.

While it is clear that the average accuracy estimated using the mixture of MNLs is higher than the average accuracy estimated using predictions made by human participants, as acknowledged in Line 303-306, the estimated accuracy follows the same trend and supports the same qualitative conclusions.

[‘Good’ values in $\Lambda(\alpha)$] We would like to highlight that, in Figures 2 and 3, the average counterfactual harm $H(\lambda)$ is estimated using data from $50$ random realizations of a test set. Here, for each realization of the test set, we apply Theorem 1 and Corollary 1, and include a $\lambda$ value in the set $\Lambda(\alpha)$ if the empirical estimate $\hat{H}_n(\lambda)$ of the average counterfactual harm using data from a calibration set, disjoint from the test set, is smaller than the user-specified bound $\alpha$.

Consequently, the closer $H(\lambda)$ is to $\alpha$ from below, the higher the probability that the empirical estimate $\hat{H}_n(\lambda) > \alpha$ due to estimation errors and thus $\lambda \notin \Lambda(\alpha)$. In Figures 2(a) and 3(a), for the particular choice of $\alpha = 0.01$, it turns out the harm-controlling $\lambda$ values with the highest accuracy $A(\lambda)$ (the 'good' values) are those for which $H(\lambda)$ is closer to $\alpha$ and thus are less often included in $\Lambda(\alpha)$. However, this is an inherent limitation of using conformal risk control and, more broadly, distribution-free uncertainty quantification, not a specific drawback of our framework.

That said, during the rebuttal period, we have conducted additional experiments with larger calibration sets. The results are summarized in Figure 1 in the one page pdf attached with the author rebuttal and show that, for larger calibration sets, the ‘good’ values of $\lambda$ are more often included in the set $\Lambda(\alpha)$ since the estimation error is lower.

[Line 105] There is a typo in line 105; $\mathcal{C}_{\lambda}(x)$ should have been $\mathcal{C}(x)$. We will fix the typo in the revised version of the paper.

[Theorem 1 and 2] We will present an intuitive explanation of Theorem 1 and 2 in the revised version of our paper.

[Figure 1 and 2] We agree with the reviewer and will first present the results currently shown in Figure 2 and then the results in Figure 1 in the revised version of the paper.

[Classifiers worse than human experts] We would like to point out that, in contrast with decision support systems that provide single label predictions, decision support systems that provide prediction sets such as ours can help humans improve their predictions even if the classifier used to construct the prediction sets has lower average accuracy than the humans, as (empirically) shown by Straitouri et al., ICML 2023 [17; Tables 1 and 2, Figure 3]. In fact, note that, for the three classifiers with worse average performance than the human experts, there always exist values of $\lambda$ for which the average accuracy of the predictions made by humans using our system, as estimated using the mixture of MNLs, is greater than the average accuracy of the predictions made by humans on their own, as estimated using real human data.

[Practical examples] We would like to first clarify that, in the medical domain, prediction sets are often referred to as differential diagnoses. A prominent example of a decision support system that uses patient history and skin condition images to provide differential diagnoses (prediction sets) for skin diseases has been recently developed by Google (see references [1, 2]). This system has been CE-marked as a Class I medical device in the European Union and a study by Jain et al. [3] has concluded that the assistance of this tool was associated with improved diagnoses by physicians and nurses for 1 in every 8 to 10 cases, indicating potential for improving the quality of dermatologic care. We will include this example in the introduction of the revised version of our paper.

[1] Google Health blogpost. Improving access to skin disease information.

[2] Liu, Y., Jain, A., Eng, C. et al. A deep learning system for differential diagnosis of skin diseases. Nature Medicine 26, 900–908 (2020).

[3] Jain, A., et al. Development and assessment of an artificial intelligence--based tool for skin condition diagnosis by primary care physicians and nurse practitioners in teledermatology practices. JAMA network open, 4(4), e217249--e217249 (2021).

[Complexity and variability of real-world decision-making scenarios] Decision systems based on prediction sets such as ours and the example highlighted in the previous reply aim to help decision makers make more accurate predictions about outcomes of interest in real-world decision making scenarios. However, more research is needed to evaluate the impact that these systems and their predictions may have on patient care pathways or policy decisions.

[Limitations of predictive accuracy] Since Wang et al. (2024) and Liu et al. (2024) were published shortly before the NeurIPS deadline, we were unintentionally unaware of their critique. However, during the rebuttal period, we have carefully read both papers and we think our work actually aligns with the critiques against prediction optimization raised by Wang et al. (2024) and Liu et al. (2024). More specifically, in lines 115-117 of our paper, we argue that improving accuracy may come at the cost of causing harm. Motivated by this observation, the goal of our work is to identify systems that are guaranteed to cause less harm, on average, than a user-specified bound. In Section 7, we discuss other critiques raised by Wang et al. (2024), such as addressing fairness considerations and the challenge of distribution shift. In the revised version of the paper, we will cite and discuss Wang et al. (2024) and Liu et al. (2024) in the introduction and related work.