A hierarchical decomposition for explaining ML performance discrepancies

We thank the reviewer for a careful reading of the work. We are encouraged to hear that the reviewer found the manuscript well-organized and the method widely applicable.

1. Binary classification: While the manuscript focuses on binary classification, the framework can be readily extended to multi-class classification problems as well as to regression problems. The only section of this work that is specific to binary classification is importance estimation of $s$-partial outcome shifts with respect to the binned probabilities $\Pr(Y=1|Z,W) = E[Y|Z,W]$. However, one can just as easily define $s$-partial outcome shifts with respect to the binned expectations, which would make the framework applicable to regression and multi-class settings. We will add a section to the appendix to describe how such an extension can be achieved. Thank you for this suggestion!

2. Detailed decomposition of $W$: As the reviewer noted, the current framework does not provide a detailed decomposition of $W$ variables. We had previously considered simply rotating the roles of $W$ and $Z$, which would give us detailed decompositions for all variables. However, it was unclear how to interpret the resulting values. As such, we have left this open question for future work.

3. Fast convergence rate assumption: Thank you for the question on the fast convergence rate assumption! While we agree that fast convergence rates are not achievable in all settings, it does hold, for instance, when the set of $x=(w,z)$ with probabilities near the bin edges $b$ is measure zero. That is, super fast (exponential) convergence rates hold if the set \Xi\_{\epsilon} = \left\\{x: \left|Q(x) - b\right| \le \epsilon \right\\} is zero for some $\epsilon > 0$, which is a relaxation of Condition 3.1 in the manuscript. We do not need $q_{\text{bin}}$ to satisfy the Hölder condition; instead, we just need $q$, the function prior to binning, to satisfy the Hölder condition. In fact, our convergence rates follow from reference [2] quite directly, as they similarly threshold the estimated probability function and demonstrate fast convergence of the binned function.

We believe that this margin condition is quite reasonable in practice, as there may very well be some small $\epsilon$ for which the margin condition holds. More importantly, we have found that the method performs quite well, even under violations of this assumption. As shown in the empirical results, our method is able to achieve the nominal Type I error rates, even though the generated data does not satisfy the margin condition. We will revise the manuscript with this discussion of the fast convergence rate assumption.

References:

2. Jean-Yves Audibert and Alexandre B Tsybakov. Fast learning rates for plug-in classifiers. Ann. Stat., 35(2):608–633, April 2007.

We thank the reviewer for their time and positive feedback. We appreciate the constructive criticism to improve the clarity of the paper, and have included a high-level description of the procedure in the global response. We will also revise the manuscript to focus more on the high-level explanation.

• Definition of VI: Yes, the Shapley value formula in equation (2) defines the variable importance.

• Notation E000: $\mathbb{E}\_{D\_W, D\_Y, D\_Y}$ denotes an expectation with respect to the joint distribution $p\_{D\_W}(W) p\_{D\_Z}(Z|W) p\_{D\_Y}(Y|W,Z)$. Thus $\mathbb{E}\_{000}$ and $\mathbb{E}\_{111}$ denote the expectations with respect to the source and target environments, respectively.

• Which version of method to use: We generally recommend using the HDPD debiased method because it provides valid uncertainty quantification. In particular, its confidence intervals have valid coverage, asymptotically.

We thank the reviewer for a thorough reading of the work and the constructive feedback. We are excited to hear that you liked the work and hope that our response will convince you to raise the score.

• Selecting $W$ and $Z$: Thank you for asking this question on how to partition variables into $W$ and $Z$! Your question has prompted us to think more deeply about the selection procedure, and we sincerely thank the reviewer for the opportunity to clarify our thinking in this area. We have responded in the global response with both causal and non-causal approaches for choosing $W$ versus $Z$. We believe these options provide users many practical approaches to applying our method.

• Choice of $W$ and $Z$ in experiments: Our reasoning was as follows:

  • Hospital readmission: The primary reason for setting demographic variables to $W$ and diagnoses to $Z$ is that demographics are causally upstream of diagnoses, and diagnoses are causally upstream of readmission status $Y$. As the reviewer rightly noted, other variables can also cause diagnosis codes. However, this is not an issue as long as we correctly interpret the results. In particular, Figure 3(a) reports how well shifts in disease diagnoses explain the variation in the performance gaps (between the two environments) across demographic groups $W$. Omitting variables from $W$ simply changes the performance gaps we are trying to explain (see Subbaswamy et al. 2021 for a formal discussion of why there is a causal interpretation of distribution shifts with respect to $W$, even if it does not contain all possible confounders). At the same time, because one is unlikely to include all possible variables that cause $Y=$readmission in $Z$, the estimated variable importances should be viewed as relative to variables included in $Z$ and not as absolute importances with respect to all possible variables. We will discuss this nuance in more detail in the manuscript.
  • ACS insurance: Most of the variables available in the ACS dataset could be considered as demographics. Nevertheless, the selected $W$ (sex, age, race) are still causally upstream of the remaining variables we assigned to $Z$ (health condition, marital status, education, etc). One can then interpret the explanations similar to the aforementioned example. Alternatively, one can think of this from an ML fairness perspective, where our goal is to explain the performance gap overall as well as performance gaps within groups defined by the protected attributes of sex, age, and race. Again, as the causal graph may be missing variables, the results should be interpreted per the discussion above.
  • Simulation setup described on line 243: This simulation tests our explanation method for conditional outcome shifts. Because the outcome distribution $Y|W, Z$ is (variationally) independent of the conditional distribution of $Z|W$, both the estimand and the estimation procedure are not affected by the dependence structure between $Z$ and $W$. As such, we considered a simple simulation where $Z$ and $W$ are completely independent. Nevertheless, we see how this can be confusing to the reader and will revise the manuscript so that $Z$ depends on $W$.

• Bias-variance tradeoff between $W$ and $Z$: This is an interesting question! When more variables are assigned to $W$, the performance gap with respect to $W$ (i.e. $\Delta_{\cdot 10}(W)$) is a more complex function. Thus we may have more uncertainty in our estimate of $\Delta_{\cdot 10}(W)$, which may lead to wider confidence intervals for the variable importance values. On the other hand, more variables in $W$ lead to higher variance of $\Delta_{\cdot 10}(W)$, so it allows one to better distinguish the relative importance of variables in $Z$ for explaining its variability. We will include this nuance in the manuscript as an additional consideration when choosing $W$ and $Z$.

• Partial retraining experiments: We have included the two suggested baselines from the reviewer in the table below. Partial retraining based on the proposed framework is better than the two baselines: it results in models with a 2% improvement in AUC. This is because targeted retraining is more statistically efficient (i.e. requires less training data), so it can quickly adapt to the new environment.

• We thank the reviewer for the suggested terminology changes such as risk to conditional outcome, domain to environment, and target to shifted environment. We will incorporate these into our revision.

Table: AUC and accuracy of retrained models are reported. Target-only model is retrained with all features ($W,Z$) but only on the target data. Weighted source-target model is retrained on pooled source and target data with all features, where source data are assigned weight $r$ to optimize AUC in the target environment. Retrained models based on the proposed method achieve the best performance.

References:

Subbaswamy, Adarsh, Roy Adams, and Suchi Saria. “Evaluating Model Robustness and Stability to Dataset Shift.” International Conference on Artificial Intelligence and Statistics 2021 https://proceedings.mlr.press/v130/subbaswamy21a.html.

We thank the reviewer for helpful comments and are glad that they appreciated the novel aspects of the work. We summarize the framework in the global response for more clarity and respond to the reviewer's questions below.

• Aggregate decomposition: Each term in the aggregate decomposition $\Delta = \Lambda_W + \Lambda_Z + \Lambda_Y$ quantifies the performance change when we vary only one factor of the joint distribution of $(W,Z,Y)$. They are defined as: $\Lambda_W = \mathbb{E}\_{100}[\ell] - \mathbb{E}\_{000}[\ell] = \int \ell(f(w,z), y) p_0(y|w,z) p_0(z|w) \left(p_1(w) - p_0(w)\right) dy dz dw$ $\Lambda_Z = \mathbb{E}\_{110}[\ell] - \mathbb{E}\_{100}[\ell] = \int \ell(f(w,z), y) p_0(y|w,z) \left(p_1(z|w) - p_0(z|w)\right) p_1(w) dy dz dw$ $\Lambda_Y = \mathbb{E}\_{111}[\ell] - \mathbb{E}\_{110}[\ell] = \int \ell(f(w,z), y) \left(p_1(y|w,z) - p_0(y|w,z)\right) p_1(z|w) p_1(w) dy dz dw$ where $\Lambda_W$ corresponds to the effect of varying $p(W)$, $\Lambda_Z$ corresponds to the additional effect of varying $p(Z|W)$, and $\Lambda_Y$ corresponds to the additional effect of varying $p(Y|Z,W)$. From the equations, one can see that the terms sum up to $\Delta$ by definition; no assumptions are needed.

• Multiple variables: As the reviewer rightly noted, multiple variables can shift to induce a performance gap. This, in fact, is one of the major contributions of this work, as it provides a framework that (i) formally defines an $s$-partial shift for variable subsets $s$ and a value function $v(s)$ that quantifies how well a partial shift with respect to subset $s$ explains the performance gap and (ii) a procedure for estimation and statistical inference of the value function. Shapley values should be viewed as an abstraction that lies on top. Its primary use is to provide a simple digestible summary at the level of individual variables. Shapley values also account for interactions between variables, as it defines the importance of variable $j$ as the average increase in the value function when a variable $j$ is added to a subset $s$. Our framework allows one to directly analyze the importance of variable subsets or Shapley values of individual variables.

• Scability: The method is highly scalable. Scalability is achieved in two ways. First, we use a subset sampling procedure to estimate Shapley values that maintains statistical validity. This allows us to compute hundreds of Shapley values. Second, one can group together variables and use the framework to quantify the importance of variable groups (rather than individual variables). For instance, variables may be grouped based on prior knowledge (e.g. genes in the same pathway) and a clustering procedure (e.g. grouping together variables that are highly correlated).

• Discrete: The method handles discrete variables, as illustrated in the real-world experiments section. In the submission, we obtain Shapley values for binary variables as well as categorical variables. For categorical variables, we can either obtain importance for each category (e.g. report importance of different marital statuses) or group together all the categories to report a single importance score (e.g. report importance of marital status as a whole). We opted for the former but our procedure can also do the latter.

• Fig 3b question: For binary variables, the framework reports a single importance; there is no consistency problem. For categorical variables, the importance of different values may not necessarily be similar, which should be expected. A shift with respect to a specific category is not directly connected to a shift with respect to a different category. This is why the different categories of marital status have different values; marital status has five categories in the ACS dataset.

• We appreciate the suggestion to surface the causal interpretation in the main paper rather than the appendix. Indeed, our explanation framework can be interpreted from the view of stochastic interventions, assuming causal sufficiency, positivity, and SUTVA hold.