Explanation Shift: How Did the Distribution Shift Impact the Model?

Dear reviewer, many thank for the reviewer and its valuable and insightful comments. Please consider the main points of our rebuttal above and our specific points that follow now:

Results are sensitive to choice of prediction model and detector model

Yes, different explanation shift $g$ will perform differently on different model $f$. The pipeline is susceptible to both. In the Appendix Table 7, we have compared several estimators $f$ and detectors $g$

Question 1. Is there some way to know which predictor and detector model should be used based on prior assumptions about the data itself or the expected shift, if any, that you are trying to detect? Do you have any insights into when different models agree/disagree, for example?

Not to the best of our knowledge. Note that this question is similar to how we could know what was the best ML estimator prior to fitting.

Question 2. Top of page 8 referencing Figure 4 says the PR18 is the most disparate. Should this be KS18 as it results in the largest AUC for the shift detector? Or what is meant by “disparate” here?

Yes, this is a typo, which we will fix—many thanks for noting.

Question 3. Can you comment on the difference between the results shown in Figure 4 and those from Figures 6-8 in Appendix E? It seems employment, travel time, and mobility follow similar patterns, but differ from Figure 4.

Our main analysis in this experimental design is that, given similar data, the choice of what to predict is relevant. Depending on what the model has learned, there are different results. As the reviewer has correctly noticed, the most disparate state is not always PR. This shows an example of when the explanations learned representation varies but changes on the input data are mostly constant. We will further refine the paragraph at the end of section 5.2.2 to make this clearer.

Dear reviewer, many thanks for the comments, please consider th points of our main rebuttal above and our specific points that follow now:

Item I.1 I am not convinced that the results demonstrate the empirical superiority of the proposed method relative to the baselines. The authors only compare the baselines in Figure 2. Here, there are several other methods that are competitive with explanation shift. In addition, the authors do not show confidence intervals in this figure. I also contest that "good indicators should follow a progressive steady positive slope", as if the goal is distribution shift detection, the only thing that should matter is the outcome of the hypothesis test.

Answer A.1: The reviewer may note that we also compare baselines B4, B5, and B6 in Figure 3. Wrt our evaluation, we fully agree with the reviewer that the approach's effectiveness is key. This raises the question of how best to measure effectiveness. We have decided to construct experiments in Figures 2 and 3 in a way such that we can control the level of distribution shift by continuously increasing $\rho$, the co-variance (Fig 2), and the level of OOD data (Fig 3). A good metric for distribution shift should be able to mirror this control parameter. By the design of our experiment, step changes or fluctuations point towards less reliable metrics for recognising distribution shifts. We will make this experimental design clearer in the text. Also, see the extra evaluation above (2. quantifiable metrics).

I.2 All of the datasets considered are tabular datasets. How would the authors adapt their method to images (where X could be pixels) and text (where X is a sequence of tokens and f is an LLM)? It seems like Shapley values would be less meaningful and harder to compute in these settings.

Our method is limited to tabular data. While this constitutes a limitation to our approach, we posit that understanding the distribution shift of tabular data is of problem hugely relevant for both research and practice.

I.3 In the real-world datasets, it is unclear what the ground truth should be, and so it is hard to say whether the proposed method is behaving as intended. For example, how do we know in Section 5.2.2 that the distribution in CA18 is actually different than CA14, in a way that affects model performance?

A.3 As described in section 2.2 (Impossibility of model monitoring), (Garg 2021)’s theorem shows that it is not possible to relate distribution shift to a change in model performance, particularly for tabular data where concept shift cannot be detected by properties of the visual domain. Therefore, our approach focuses on the change in how a given model works on data.

I.4 One important aspect of distribution shift detection is to isolate the shift to particular distributions (i.e. particular shifts in Definitions 2.1-2.5). This is underexplored in most of paper. The authors do explore this by examining the feature importances in Section 5.2.2, but this seems quite ad-hoc and should be characterized further. For example, how would this behave theoretically in the covariate shift case in Example 4.1?

A1.4 We fully agree with the reviewer, in the last paragraph of section 2.1 we have stated: "In practice, multiple types of shifts co-occur together and their disentangling may constitute a significant challenge that we do not address here."". We have only analyzed particular shifts on synthetic data where we can fully characterize the type of shift.

I.5 The authors should consider examining the scenario where the number of samples on the target domain is limited, both empirically and theoretically. How does the power of the two-sample test scale with the number of target domain samples?

A1.5: It is correct that our recognition procedure is a binary classifier and the reliability of the classifier depends on the number of samples on which it is trained. Previous work has studied the limitations of Classifier Two Sample Tests in the particle physics domain, in which they suggest using bootstrap permutation tests to get a better handle on the test's accuracy.

We've taken this methodology. In our own experiments (check out Section 5.2.2), we have included bootstrap permutation testing between the in-distribution (CA14) bootstraps and other states bootstraps. Furthermore, in our experiment in section 5.2.1 where an increasing fraction of data is from OOD, we that from a small fraction onwards our metrics picks up the changes in the model due to distribution shift.

[3] Model-independent detection of new physics signals using interpretable SemiSupervised classifier tests, Chakravarti, Purvasha and Kuusela, Mikael and Lei, Jing and Wasserman, Larry, The Annals of Applied Statistics, 2023

Dear reviewer, many thanks for the comments, please consider the main points of our rebuttal above and our specific points that follow now:

I.1 The authors provide some interesting case studies and examples of the utility of Shapley-based explanations in identifying distribution shifts. One aspect of this discussion that could be done better is trying to emphasize what additional assumptions could be required so that explanations such as Shapley can indeed be used for detecting, say, concept shift. This is not an impossible task for other methods, see for example: Liu et al [1]. See also experiments in this paper.

A.1: Paper [1] can deal with concept shift because the authors assume that labels $D^{new}_Y$ are given. In our problem scenario, we make the realistic assumption that $D^{new}_Y$ are unknown. Thus, we address a different problem. Also cf. our Main Point 1 at the beginning of our rebuttal.

We will contrast our work against [1] in our discussion. This will further clarify and strengthen our work.

I.2 The overall empirical evaluation, in my view is weak. May be at least discuss how general the approach will be (can it be used on other data-modalities, beyond tabular data?)

A.2: ​​We have systematically varied models, model parametrizations, and input data distributions. As indicated in Main Point 2 (at the beginning of the rebuttal), we have designed carefully controlled experiments (which are novel by themselves). Additionally, the appendix contains supplementary experimental results, providing details on synthetic data experiments, extending to further natural datasets, showcasing a broader range of modelling choices, and different explanation mechanisms. Thus, we believe that one of the paper's strengths is an extensive empirical evaluation.

As the reviewer correctly notes, the Classifier Two Sample Test on the explanations distributions is limited to tabular data and is not able to detect concept shift (we have stated this in the discussion). For non-tabular data, it needs further modifications that are out-of-scope for the paper. Characterizing distribution shift on tabular data constitutes an economically hugely important problem that still lacks appropriate treatment.

I.3 Question 1 What is $\alpha_i$ in Example 4.3? I am not sure this example is valid based on what I think will be in this example. Can authors clarify?

A.3: The Shapley values are a product of a weighting (here: alpha_i for feature i) multiplied by the actual value of the feature (here: x_i). Thus,Shapley values explain a model f by a linear model with coefficients $\alpha_i$. We will further clarify that the Shapley values can be directly calculated for a linear model and IID data by this formula (Aas, 2021)

The specific example shows that there is a feature that is not used by the model and is shifted. In this case, we have a distribution shift, that does not affect the model f nor the explanation shift, and any “distribution shift” methods based on input data will flag it.

I.4 Question 2 Why aren't methods that don't actually need a causal graph such as [2] compared to in the paper? [2] Namkoong, Hongseok, and Steve Yadlowsky. "Diagnosing Model Performance Under Distribution Shift." arXiv preprint arXiv:2303.02011 (2023).

A.4: We do not compare to this paper for two reasons: This paper works in a different problem setup. It requires labels $D^new_Y$, whereas we assume that these are not available (cf. answer A.1; Main Point 1) We will be happy to acknowledge the recent work that is reported in the two arxiv papers. However, given that they have not been published at a conference yet, we think that their absence in our discussion should not be held against our submission.

I.5: Question 3: The overall empirical evaluation, in my view is weak. May be at least discuss how general the approach will be (can it be used on other data-modalities, beyond tabular data?)

A.5 Classifier Two Sample Test, as we have done for tabular data, is not directly applicable to text or images. We have focused on tabular data since it constitutes an economically hugely important domain that brings up unique challenges. We have stated this in the discussions, but we will further clarify.

I.6 Question 4 I see a discussion on using LIME as a possible explanation for addressing the same, however, I am not sure whether crucial properties of Shapley are necessary for this method to succeed. I think additional discussion on which explanations have the desirable properties for use in detecting explanation shifts will make the paper stronger.

A.6: The theoretical properties that we require from a feature attribution method are efficiency and uninformativeness (cf section 2.3). We use and compare Shapley values and Lime because these approaches (and their implementations) fulfil or at least approximately fulfil these properties.

Dear reviewer, many thanks for the comments, please consider the main rebutall and the specific rebutal:

I.1 Motivation for considering shifts in explanation is not convincing. This, I believe, is because the goal is not stated concretely to then motivate the solution. The stated goal of investigating interaction between distribution shifts and learned model needs to be further characterized in quantifiable forms. I agree that not every type of distribution shifts are important to detect. The ones that change model behaviour in some meaningful way are important. However, it is unclear to me why explanations, that too Shapley values, is a meaningful or useful summary of model behavior to consider.

A.1: Main Points 1 and 2 at the beginning of our rebuttal address these concerns. To summarize: AUC or likewise metrics cannot be applied in our problem setting, which is frequent in the real world, where new data $D_x^{new}$ comes without labels $D_y^{new}$.

As (Garg 2021) have shown, correctly estimating the performance $\text{perf}(D^{new})=\sum_{(x,y)\in D} \ell_\text{eval}(f_\theta(x),y)$ is impossible in the presence of distribution shift. Their results imply that AUC and likewise methods are inapplicable for discovering distribution shift when $D_y^{new}$ is not given.

Given the model predictions $f_\theta(x)$, we use the Shapley values, as they attribute the contribution of input features to the prediction (more information in section 2.3).

We will use your question and our response to make the introduction clearer. Thank you.

I.2 Evaluation setup can be improved in terms of quantifying evaluation metrics like sensitivity. The magnitude of AUC does not matter that much I believe. The authors are right in asking for sensitivity in the method but it is not directly evaluated. For instance in Figure 2, it is unclear between (ours) and (B7) methods in the left figure and between (ours) and (B2) which one is better. All seem sensitive in the sense that different correlation coefficients result in different AUCs.

A.2 In order to clarify and provide a quantifiable metric, we have extended the experimental evaluation in the main rebuttal. We say baselines are underperformant because they achieve a lower correlation with $\rho$ or are independent of the model used.

I.3 Presentation of experiment can be improved. The case study on ACS data in Sec 5.2.2 is a good place to showcase how the question in the paper title (how the shift impacted the model) is addressed by the method. The findings are stated in a matter-of-fact way. These could be contextualized in the data making the significance of the method and its results more apparent. The questions asked through the experiments could be described at the start of Sec 5 and what we learned from the results could be described more directly at the end of the section.

Many thanks for the suggestion; we fully agree with the reviewer. These minor edits will improve the paper's clarity. We will take them into account.

I.4 Q1 Please clarify the problem statement more formally. What aspects of the model behavior under distribution shifts (interaction between distribution shift and learned model) is of interest?

A.4 Section 4 covers all the different types of shifts: Uninformative shifts vs Informative Shifts. Among the latter, problems in practice have to deal with Covariate shifts, Prediction shifts, and Concept shifts. Discovering concept shift (if not domain assumptions can be made, nor labeled data can be obtain), which is why we are interested in covariate and prediction shifts. These are all defined in section 4 and related to explanation shifts.

I.5 Q2 Please motivate the approach (focus on explanations) in terms of the problem statement. How do Shapley values capture the desired aspects of the model behavior?

The Shapley values, weigh the influence of feature values towards a prediction, $\sum_{j \in N} S_j(f, x^{\star}) = f(x^{\star}) - E[f(X)])$(Efficiency Property - cf. Section 2).

The strength lies in the decomposition of predictions, offering insight into how each feature contributes to the overall outcome. This attribute is valuable when dealing with high-dimensional distributions, providing a more informative distribution than prediction distributions.

By employing feature attribution explanation techniques, specifically Shapley and LIME, we exploit this efficiency property. These techniques facilitate a granular understanding of how shifts in specific feature distributions impact the model. he decomposition of the prediction across the contribution of each features allows for a higher dimensionality distribution than the predictions distributions. Therefore the explanation distribution compasses always more information.