Selective Explanations

Thank you for your review! It was really insightful and will definitely improve the final version of our paper. We will implement your suggested changes, and hope you engage with us during the discussion period. We address your comments below.

Q1) “The terminology is misleading: the paper calls expensive-to-obtain explanations high-quality explanations.”

This is a great point. We do agree with you that expensive-to-obtain explanations is a more precise term and will change “high-quality” to “expensive-to-obtain” everywhere in the revised version of our paper. Throughout the paper we assume that “high-quality” explanations are the ones computed until convergence – for example, SHAP with exponentially many inferences. These computationally expensive SHAP explanations have been extensively tested both quantitatively and via user studies [1, 2] – although SHAP does have limitations as well [3]. We will add the following paragraph on the quality of expensive-to-obtain explanations to the limitations section:

“Selective explanations aim to approximate expensive-to-obtain explanation methods such as SHAP with an exponential number of inferences. Therefore, its efficacy is limited to the performance of the explanation method being approximated. The general methodology behind Selective Explanations is flexible, and can be applied to approximate other expensive-to-obtain feature attribution methods. We encourage the reader to use Selective Explanations to approximate the computationally expensive explanation method that best fits their application, noting that, ultimately, the quality of explanation will be limited by the chosen approximand.”

Q2) “The methodology is based on never-discussed assumptions, i.e. a more uncertain/less stable explanation is a low-quality explanation.”

Thank you for raising this important point. We address it in the global response to all reviewers due to its importance, and we kindly ask you to refer to the text above (see answer to Q2 above). TL;DR: Figure 3 illustrates the relationship between uncertainty and explanation quality, showing that explanations with the lowest uncertainty also have the lowest MSE (and vice-versa), where MSE is with respect to "higher-quality explanations" (SHAP computed until convergence). We agree that this correlation would be better supported by more direct evidence. To address this, we will add a new Table 1 (see attached rebuttal PDF) to the main paper, which presents Pearson and Spearman correlations between our uncertainty measures and MSE. This table provides evidence of strong positive correlations and monotonic relationships between MSE and uncertainty estimates, particularly for language datasets. The monotonic relationship underlies our approach's effectiveness in identifying lower-quality explanations by using uncertainty estimates. For more details, please refer to the global answer (Q2).

Q3) “The experimental evaluation is limited to quantitative measures.”

The main reason for the lack of user evaluation is that our main goal is to approximate widely-studied yet expensive-to-obtain explanation methods such as LIME (with a large number of model inferences) and SHAP (computed until convergence) [1, 2]. Given their established results, we follow the methodology in recent papers focused on accelerating attribution methods [5 – 8] and report both MSE and Spearman’s Correlation to these expensive-to-obtain explanations. When MSE is low and Spearman Correlation is high with converged SHAP explanations (which our Selective Explanation method achieves), the performance of our method is essentially the same as SHAP.

Q4) “The paper does not take into account relevant work on uncertainty quantification of explanation methods. For instance, see [Zhao et al, 2021] and [Slack et al., 2021].”

Thanks for suggesting these papers. We'll discuss them in our related work section. Our approach differs in two key ways from the mentioned papers:

1. These papers study uncertainty for SHAP and LIME variants. We calculate uncertainty for the amortized explainer - a model that takes data points as input and directly outputs explanations with only one inference.

2. Main contribution: Our focus isn't on the uncertainty measures themselves; they only serve the intermediate purpose of finding amortized explanations that severely differ from expensive-to-obtain explanations.

Q5) “As far as I know, an unstable explanation might be due to the underlying uncertainty around a prediction, but also due to the approximations required to compute the explanation. Is there a way to disentangle these two kinds of uncertainty?”

We also believe that prediction uncertainty and approximations are sources of unstable explanations. However, we are computing the uncertainty for the amortized explainer, not the black-box model being explained ($h$ in our notation). Our method prioritizes practical improvement of the approximation of expensive-to-obtain explanations over the disentanglement of uncertainty sources – which we believe is an important direction of future work.

[1] Lundberg et al. A Unified Approach to Interpreting Model Predictions. NIPS 2017.

[2] Antwarg et al. Explaining Anomalies Detected by Autoencoders Using SHAP. Expert Syst. Appl. Vol 186.

[3] Slack et al. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. AIES 2020.

[4] Ribeiro et al. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. KDD 2016.

[5] Jethani et al. FastSHAP: Real-Time Shapley Value Estimation. ICLR 2022.

[6] Schwab and Karlen. CXPlain: Causal Explanations for Model Interpretation under Uncertainty. NeurIPS 2019.

[7] Yang et al. Efficient Shapley Values Estimation by Amortization for Text Classification. ACL 2023.

[8] Covert et al. Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution. arXiv:2401.15866.

Thank you for your review. We appreciate that you found our work novel. We also appreciate that you found our evaluation comprehensive and our method flexible. We answer your questions and comments next.

Q1) “How does the performance of selective explanations scale with extremely large models (e.g., models with billions of parameters)?”

Our results indicate that Selective explanations scale well to very large models. We observed a better performance improvement when using Selective Explanation for larger models - the language models for text classification (Figures 3 and Figure 4 (c) and (d)). This is because we use the higher quality embeddings of these larger models to train the amortized explainer and the uncertainty metrics.

Q2) “Have you explored using more advanced uncertainty estimation techniques, such as those based on Bayesian neural networks?”

This is an excellent suggestion. We agree with you that other techniques for uncertainty quantification such as those based on Bayesian neural networks could potentially improve uncertainty estimation by providing more calibrated uncertainty estimates. In fact, our Deep uncertainty metric (Eq. 4) is directly inspired by this approach. However, Bayesian methods induce a higher cost that may be prohibitive in some applications. For example using ensembles of models as in deep ensembles [1] requires retraining the same model pipeline multiple times – a limitation we also observe in our proposed Deep uncertainty metric (Eq. 4). Moreover, our ultimate goal is to select the examples that receive higher (lower) quality explanations relative to a computationally-expensive-to-obtain method such as SHAP. In Learned uncertainty we “learn” from data which points will receive higher (lower) quality explanations instead of using a Bayesian approach. This leads to a method that is optimized to achieve our goal because it is trained to predict the examples with higher MSE relative to the high-quality explanations and decrease the computational overhead by only training a model once.

[1] Lakshminarayanan et al. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NIPS 2017.

Q3) “How sensitive is the method to the choice of amortized explainer? How might it perform with different types of amortized explainers?”

The main objective of selective explanations is to improve the quality of the explanations provided by both amortized and Monte Carlo explainers by combining both methods when the amortized explainer fails. Therefore, in the case that the amortized explainer has poor performance, selective explanations would require more usage of the Monte Carlo explainer to achieve favorable performance (i.e., small MSE from high-quality explanations), thus increasing the computational cost per explanation. Hence, the choice of amortized explainer heavily impacts the quality vs. computational cost of selective explanations.

Q4) “Could the selective explanation approach be extended to other types of explanation methods beyond feature attribution (e.g., example-based explanations)?”

The core idea of selective explanations is identifying low-quality predicted explanations and using a more computationally expensive method to improve their quality. Therefore, selective explanations could be extended to other explanation types. For example, with example-based explanations, an uncertainty metric could be developed to identify cases where selected examples are likely to be unrepresentative, and thus a different explanation method should be used.

Q5) “How might the method be adapted to handle streaming data or online learning scenarios where the underlying model is continuously updating?”

To handle streaming data or online learning, the components of the selective explanation method would need to be updated incrementally. Specifically, the amortized explainer, the combination function (Eq. 12), and the uncertainty measures (Eq. 4 and 5) would need to be continuously updated to reflect the changes in the model being explained.

Q6) “Computational overhead: While more efficient than full Monte Carlo methods, the selective approach still requires additional computation compared to pure amortized methods.”

The selective explanation method combines amortized and Monte Carlo explanations, introducing an additional computational cost. However, we believe this is a feature instead of a flaw, since it allows flexibility in choosing the fraction of samples that receives extra computations. Figure 5 shows that selective explanations significantly improve the performance of the amortized explainer, especially for the samples with worst-performing explanations. For example, Figure 5 (a) demonstrates that providing explanations with an initial guess to 50% of the data improves the Spearman’s correlation of the worst 10% amortized explanations from 0 (no correlation) to almost 0.6 (strong correlation).

Thank you very much for your thoughtful review. Your feedback will positively impact the final version of our paper! We address your questions below.

Q1) ”Although the experimental results show that the proposed method can improve the accuracy of explanations, those on computational efficiency are not investigated.”

Thank you very much for this comment! Please kindly refer to the global answer to all reviewers above. TL;DR: the trade-off between computational complexity (given by number of model inferences) and quality of explanations (MSE from higher-quality explanations) is captured in Fig. 4 and 10 of the paper. Particularly, Fig. 10 (reproduced in the attached PDF) shows that our method performs very close to an oracle that "knows" which explainer to use for a given MSE and constraint on number of inferences. We will make this clearer by bringing Fig. 10 and the associated discussion to the main body of the paper instead of the appendix. Please, check the global answer (Q1) for more details.

Q2) “The generated explanations are quantitatively evaluated. However, due to the lack of qualitative evaluation, it is unclear how good the explanations generated by the proposed method are actually from a user perspective.”

Thank you for this great point. The main reason for the lack of user evaluation is that we are approximating expensive-to-obtain yet established explanation methods that were extensively tested and evaluated by humans such as LIME [1] and SHAP [2]. Given these established results, we follow the methodology in most papers that aim to approximate feature attribution methods [3, 4, 5, 6] and report both MSE and Spearman’s Correlation to SHAP explanations computed until convergence. For low MSE and high Spearman Correlation with SHAP (which our Selective Explanation method achieves), the performance of our method is essentially the same as SHAP. In other words, our goal is to approximate expensive-to-obtain SHAP explanations as closely as possible, while lowering the required number of inference and, thus, computational cost.

Nevertheless, we acknowledge this point, and recognize that – since we aim to approximate methods such as SHAP – our method is ``high quality"' insofar that SHAP is high-quality, and will note this caveat in the paper.

[1] Ribeiro et al. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. KDD 2016.

[2] Lundberg et al. A Unified Approach to Interpreting Model Predictions. NIPS 2017.

[3] Jethani et al. FastSHAP: Real-Time Shapley Value Estimation. ICLR 2022.

[4] Schwab and Karlen. CXPlain: Causal Explanations for Model Interpretation under Uncertainty. NeurIPS 2019.

[5] Yang et al. Efficient Shapley Values Estimation by Amortization for Text Classification. ACL 2023.

[6] Covert et al. Stochastic Amortization: A Unified Approach to Accelerate Feature and Data. Attribution. arXiv:2401.15866.

Q3) “If (5) is MSE, should use ||⋅|| instead of |⋅|.”

Thank you for this careful point. Notice that $\ell(\text{AMOR}(x;y), \text{MC}(x; y))$ is a scalar. Hence, $||.||$ and $|.|$ are equivalent.

Q4) “Learned uncertainty: The loss in (5) is the same as the objective of the amortized explainer in (3). Therefore, if the amortized explainer is fitted enough to the training data, the loss is consistently near zero, resulting in the learned uncertainty functions being consistently near zero, too. This learned uncertainty function does not seem to work well at an inference phase. What is the justification for (5)?”

Thank you for this great point. In the case there is overfitting, the selection function might not be sufficiently accurate (neither the amortized explainer), as you mentioned. In such cases, we advise the users to reserve a validation dataset to fit the uncertainty function. We highlight that we did not observe overfitting in any of our experiments where we train the uncertainty measure using the training dataset, and train the amortized explainers until convergence on the same training dataset, and evaluate their performance on a test set. However, this does not preclude the risk of overfitting in other settings. We will add your comment to our limitations section and advise on the use of a validation set instead of the same training dataset.

Q5) "What does 'Random' in Figure 3 mean?"

"Random" means that we select explanations uniformly at random instead of using the uncertainty metrics. This leads to an average MSE that is near the MSE of the amortized explainer and is independent of the coverage. We will add this description in line 255 of our paper.

Thank you for your thoughtful review. Your feedback will positively impact the updated version of our paper and we will include all of your comments. We also appreciate that you found our manuscript well-written, our method intuitive and easy to understand, and that our idea of Selective Explanations is compelling and timely. We address your comments below. We hope you can engage with us during the discussion period.

Q1) “The experimental design does not strongly support the major claims on the reduction of computational cost. Evaluations on the tradeoff between explanation quality and computational cost would be helpful.”

We appreciate this comment and we kindly ask that you refer to Q1 in the global answer to all reviewers above. The TL;DR is: the trade-off between computational complexity (given by number of model inferences) and quality of explanations (MSE from high-quality explanations) is captured in Fig. 4 and 10 of the paper. In particular, Fig. 10 shows that our method significantly reduces the number of inferences required to achieve a given explanation quality in comparison with Monte Carlo explanations – specifically SVS. Remarkably, Fig. 10 shows that Selective Explanations performs very close to an oracle that "knows" which explainer to use for a given MSE and constraint on number of inferences. We will make this clearer by bringing Fig. 10 and the associated discussion to the main body of the paper instead of the appendix. Please, check the global answer (Q1) for more details.

Q2) “The relationship between uncertainty and explanation quality is unclear, it would be better to have empirical or theoretical proof on their correlations.”

Again, thank you for this thoughtful comment. Here, again, we kindly ask for you to refer to the global answer for all reviewers above and the attached pdf, which empirically shows a high correlation between uncertainty and explanation quality. TL;DR: Figure 3 shows the relationship between uncertainty and explanation quality by demonstrating that the amortized explanations with smallest uncertainty also have the smallest MSE from high-quality explanations and vice-and-versa. However, we agree with you that more direct evidence is needed. To address this, we computed this correlation, in a new Table 1 (attached in the PDF) and will add it to the main paper. This table presents positive Pearson and Spearman’s correlations between our uncertainty measures and MSE and a strong monotonic relationship between both quantities, especially for the language datasets. This evidence reinforces the effectiveness of our approach in detecting lower-quality explanations in the main tasks of interest. Again, please, check the global answer (Q2) for more details.

Q3) “In addition, the proposed uncertainty measurement also introduces computational overhead when “run the training pipeline for the amortized explainer described in (3) k times” (line 139).”

We agree with you, the Deep Uncertainty measure introduces a computational overhead. Please note that this computational overhead is exactly the reason we proposed a second method, referred to as "Learned Uncertainty," which only needs to be trained once. Our experiments suggest that the Learned Uncertainty provides a better performance for selective explanations of language models – our main use case of interest. We will make this distinction clearer in the paper by adding the following paragraph after line 148:

“To address the computational overhead of Deep Uncertainty and directly target low-quality explanations, we introduce Learned Uncertainty. This alternative method requires only one training run, drastically reducing computational costs. Learned Uncertainty is optimized to predict discrepancies between amortized and high-quality explanations.”

We proposed Deep Uncertainty because this method is directly inspired by Deep Ensembles [1] used for distribution-free uncertainty quantification. It achieves favorable performance (see Figs. 3, 4, 5, and 6), but it indeed comes with a computational overhead. We introduce it because of its connection with Deep Ensembles traditionally used for uncertainty quantification.

[1] Lakshminarayanan et al. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NIPS 2017.