Prediction via Shapley Value Regression

We appreciate the reviewer's time and feedback. Please find our responses below.

A) It's unclear to me what it means to run KernelSHAP until convergence... B) KernelSHAP produces estimates which are slightly biased...

We agree with the reviewer that KernelSHAP can be biased, therefore, we employed the unbiased KernelSHAP [1]. For the tabular datasets, we allowed unbiased KernelSHAP to continue sampling and updating the learned values iteratively until convergence, meaning that the sampling is not restricted to a fixed number of coalitions but continues until the estimates are stable.

The bias of KernelSHAP shrinks to zero as the sample size grows [1,2]. Furthermore, it has been shown that the unbiased KernelSHAP converges to the true Shapley values given a sufficiently large number of samples [1,3], which we provided for the tabular data. We employ the same evaluation setup that has been employed in [3].

I'm concerned about how they compute ground truth...Use a small value of $2^n$...

We included datasets with a small number of features $n$, such as Phonemes (5 $n$), Pollen (5 $n$), Mozilla 4 (5 $n$), Abalone (8 $n$), Electricity (8 $n$), and MagicTelescope (10 $n$). For these datasets, the total number of possible coalitions is small enough that the unbiased KernelSHAP solution provides the exact Shapley values, as all possible coalitions are generated.

... Use a linear model.... Use a decision tree

We appreciate the reviewer's perspective. However, training a linear model or TreeSHAP to obtain the exact Shapley values may not be applicable to evaluating the explainability of ViaSHAP, as ViaSHAP explains its own predictions rather than the predictions of a separate model, e.g., a decision tree where exact Shapley values can be obtained. Also, ViaSHAP, in the current form, cannot be implemented using decision trees.

They state three lemmas and a theorem, but I would describe these as "decorative theory"...

We respectfully disagree that the lemmas and the theorem are of marginal value for the following reasons:

1-The proofs of Lemmas 3.2 and 3.3 provide upper bounds on the error associated with satisfying the missingness and consistency properties under realistic optimization assumptions.

2-Without the theoretical results, there would be no formal guarantee that ViaSHAP computes Shapley values, given it differs from post-hoc explainers and computes the Shapley values before making predictions.

simple baseline is missing: Train a decision tree model...

We want to clarify again that ViaSHAP explains itself rather than acting as a post-hoc explainer. Since ViaSHAP, as currently proposed, can only be implemented with algorithms that can be optimized using backpropagation (i.e., cannot be implemented using decision trees), we cannot see how training a decision tree-based model and explaining it could help evaluate ViaSHAP's explainability.

"There are $2^n - 1$ possible coalitions", I guess this is semantic but doesn't the empty set trivially count as a coalition?

We mean that there are $2^n - 1$ possible coalitions that can be used to compute the exact Shapley values, since we cannot add a new player $i$ to the grand coalition $S$ of all players as follows: $\frac{|S|! (n - |S| - 1)!}{n!} (v(S \cup \\{i\\}) - v(S))$. We will update the sentence to be clear.

Notation of $p(S)$ and $p(x)$ in 2.4 is confusing..

We agree with the reviewer and will update the notation to eliminate confusion.

Figure 3 is very strange...

Figure 3 is a standard visualization for summarizing the results of Friedman-Nemenyi statistical significance tests. Similar plots have been used in [4,5,6]. For a detailed breakdown of the results, please refer to Table 1.

are you computing the Shapley values for each pixel...

We computed Shapley values using super-pixels of size 2×2. However, this is a hyperparameter, and users can choose between pixel-wise explanation or larger super-pixel sizes.

title "Prediction via Shapley Value Regression" is very broad...

Methods like KernelSHAP indeed involve Shapley value regression. However, a key difference is that they do not formulate predictions using the regressed Shapley values, which is central to our approach. We appreciate the reviewer’s suggestion for an alternative title and are open to modifying the title to be more specific.

[1]-Covert, I., et al. Improving kernelshap: Practical shapley value estimation using linear regression. AISTATS 2021.

[2]-Covert, I., et al. Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution. NeurIPS 2024.

[3]-Jethani, N., et al. FastSHAP: Real-time shapley value estimation. ICLR, 2022

[4]-Dedja, K., et al. BELLATREX: Building Explanations through a LocaLly AccuraTe Rule EXtractor.

[5]-Werner, H., et al. Evaluating Different Approaches to Calibrating Conformal Predictive Systems.

[6]-Pugnana, A., et al. Deep Neural Network Benchmarks for Selective Classification

We appreciate the reviewer's time and feedback on our paper. Below, we provide our responses and clarifications.

I'm a bit concerned with how the comparisons between ViaSHAP and the ground truth are made...

We appreciate the reviewer’s concern and would like to clarify our experimental setup. In the explainability evaluation, we treated ViaSHAP models as standard black boxes and computed the ground truth Shapley values using the unbiased KernelSHAP [1], which involves solving an optimization problem for each prediction. We then compared the explanations generated by ViaSHAP to the ground truth obtained from the unbiased KernelSHAP.

As established in previous work, the bias of KernelSHAP shrinks to zero as the sample size grows [1, 2], and with a sufficiently large number of samples, it converges to the true Shapley values [1, 3]. For the tabular datasets, we allowed the unbiased KernelSHAP to continue sampling and updating the learned values until convergence. We followed the same evaluation setup using the unbiased KernelSHAP that has been employed in [3].

The paper focuses on the classification case with regression being briefly discussed in the appendices...

We appreciate the reviewer’s feedback and acknowledge that incorporating additional regression datasets could further strengthen the experiments. However, due to the constraints of the rebuttal phase, we are unable to add new results at this stage.

We would also like to clarify that the term regression in the paper title, Prediction via Shapley Value Regression, specifically refers to the fact that we solve the regression task of Shapley values and use Shapley values in the prediction task itself. In other words, the title is intended to convey "(Classification and Regression) via Shapley Value Regression."

incorporating more discussion about when this method would be best used...

We sincerely appreciate the reviewer’s suggestion and will update the manuscript accordingly. ViaSHAP is particularly advantageous in settings where computational resources or time are limited, as it removes the need to train and run separate models for prediction and explanation. Moreover, it is well-suited for scenarios where high-fidelity explanations are crucial, as the model’s predictions are inherently tied to the explanations.

Post-hoc methods might be more suitable in cases where a pre-trained black-box model is already employed or when users cannot access or modify the black-box model.

Why is it the case that the results are particularly weak over certain datasets (such as pollen)?

Similar to other machine learning models, ViaSHAP's performance can vary depending on the characteristics of the dataset. Certain datasets pose inherent challenges, such as a limited number of training examples, a high-dimensional feature space, complex data distributions, or noisy data, all of which can impact model performance. While ViaSHAP's performance on the Pollen dataset is relatively weaker compared to other datasets, it is noteworthy that it also outperforms XGBoost, Random Forests, and TabNet on this dataset, which suggests that the Pollen dataset is inherently challenging, rather than an issue specific to ViaSHAP.

...why is it the case that the AUC of XGBoost does not have a +- value...

All compared models are trained and evaluated using the same data splits to ensure a fair comparison. The $\pm$ values indicate variations in performance due to different random initializations, as stated in the experimental setup (Lines [312, 315]): "If the model’s performance varies with different random seeds, it will be trained using five different seeds, and the average result will be reported alongside the standard deviation." Since XGBoost with the default settings is deterministic and its performance does not vary across different random seeds, its AUC is reported without a $\pm$ value.

What exactly does Figure 5 show? I'm not sure how much this illustration contributes to the main paper.

Figure 5 illustrates that the explanations generated by ViaSHAP are more precise compared to those from FastSHAP when applied to the same image. Specifically, FastSHAP tends to highlight broader regions, whereas ViaSHAP primarily focuses on the shape of the classified instance within the image, providing a more precise and sparse explanation.

[1] Covert, I. and Lee, S.-I. Improving kernelshap: Practical shapley value estimation using linear regression. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics.

[2] Covert, I., Kim, C., Lee, S., Zou, J., Hashimoto, T. Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution. In The Thirty-eighth Annual Conference on Neural Information Processing Systems 2024.

[3] Jethani, N., Sudarshan, M., Covert, I. C., Lee, S.-I., and Ranganath, R. FastSHAP: Real-time shapley value estimation. In the International Conference on Learning Representations, 2022

We appreciate the reviewer's feedback. We provide answers in the following part.

My major concern is its positioning...

The proposed method does not fall strictly into the categories of model-specific or model-agnostic approaches. Instead, we introduce a framework for training by-design explainable models, where predictions and their Shapley value explanations are learned simultaneously. This distinguishes our approach from existing methods that compute Shapley values post-hoc or in a separate forward pass.

... authors should sufficiently compare it with prior works...

We appreciate the reviewer’s suggestion. Our primary objective, however, is to demonstrate that the proposed method is inherently explainable and that the generated explanations are accurate. To this end, we compared our approach to the unbiased KernelSHAP, as well as FastSHAP, which directly inspired our method and represents the most closely related work. Unfortunately, due to the constraints of the rebuttal phase, we are unable to add new results at this stage. However, we acknowledge the value of broader comparisons.

...on complex datasets such as CIFAR-10...

We have indeed reported the model’s predictive performance on complex datasets such as CIFAR-10, along with the approximation error of the explanation, and compared the results to FastSHAP. We kindly refer the reviewer to Appendix F, which provides detailed results for CIFAR-10. The evaluation of Shapley value estimates on images using the root mean square error requires access to ground truth Shapley values, which is computationally infeasible for image datasets. Instead, we assess explanation quality using inclusion and exclusion curves, which measure an explanation’s ability to identify informative image regions. This evaluation strategy is commonly used for image datasets, as seen in [1,2,3].

Is the second categorical loss in Eq. (7) incorrectly written?...

We thank the reviewer for pointing out that the notations in Eq.7 are confusing. Indeed, j in Eq.7 represents the true probability to be estimated for each category. We will correct and clarify this in the revised paper.

...explicitly explain the physical meaning of Eq. (6)...

We appreciate the reviewer’s suggestion. We explained the intuition behind Eq.6 in lines [173,186] and illustrated the main idea in Figure 2 and Algorithm 1. However, as recommended by the reviewer, we will further refine our explanation to enhance clarity.

...using KernelSHAP as the ground-truth...

We agree with the reviewer that KernelSHAP is unreliable, particularly for high-dimensional data. Therefore, we employed an unbiased version of KernelSHAP [4], which aligns with the reviewer’s suggested approach [5], as mentioned in lines 358 and 359 " the ground truth Shapley values ($\phi$), computed by the unbiased KernelSHAP". The same evaluation setup, using unbiased KernelSHAP, has been employed in [3]. For the tabular datasets, we allowed the unbiased KernelSHAP to keep sampling until convergence. The bias of KernelSHAP shrinks to zero as the sample size grows [4,6]. It has been shown that the unbiased KernelSHAP converges to the true values if provided with a large enough number of samples [3,4], which we did for the tabular data in the experiments.

Using cosine similarity as the metric...

We completely agree with the reviewer that cosine similarity alone is insufficient for comparing explanations. Therefore, we employed three metrics: Spearman rank correlation, $R^2$, and cosine similarity. The results are reported using the three metrics in Tables 7, 9, 12, 14, and 16.

... include a broader set of model-agnostic approaches...

We acknowledge that incorporating additional experiments can enhance the quality of the paper. We want to clarify that the method in reference [5] provided by the reviewer (Covert et al. Improving KernelSHAP...) is the unbiased KernelSHAP method, which we have already compared our proposed method with in our evaluation.

lacks a discussion of related works...

We appreciate the reviewer’s suggestions for additional related work to discuss. We kindly clarify that the papers (Mitchell et al. Sampling Permutations for Shapley Value Estimation) and (Covert et al. Improving KernelSHAP) are already cited multiple times in our manuscript.

[1] Hooker, S., et al. A benchmark for interpretability methods in deep neural networks.

[2] Jethani, N., et al. Have we learned to explain?: How interpretability methods can learn to encode predictions in their interpretations. AISTATS 2021

[3] Jethani, N., et al. FastSHAP: Real-time shapley value estimation. ICLR, 2022

[4] Covert, I. et al. Improving kernelshap: Practical shapley value estimation using linear regression. AISTATS 2021

[5] Castro, J., et al. Polynomial calculation of the shapley value based on sampling. Computers & Operations Research

[6] Covert, I., et al. Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution.

We thank the reviewer for their time and valuable feedback on our manuscript. We provide answers and clarifications below.

The authors argue that one major drawback of existing methods is that Shapley values are computed post-hoc....

We do not intend to suggest that computing Shapley values in a post-hoc setup is inherently a drawback. Rather, our work builds on existing post-hoc methods and aims to further reduce computational costs by eliminating the need for training a separate explainer model and running two models at inference time. In particular, it is true that in cases where a pre-trained model already exists or when users cannot access or modify the black-box model, post-hoc methods might be more suitable. Nonetheless, this is not always the case. ViaSHAP offers an alternative for users who prioritize developing an inherently explainable model, as well as for scenarios where computational efficiency and resources are important considerations.

There are already several works on computing Shapley's explanation faster. Again, even though this work is new from the angel learning to explain rather than postdoc explanation, still the methologies is not very distinct from FastSHAP.

Our proposed method is indeed inspired by FastSHAP. However, a key difference is that we do not start with a pre-trained black-box model that requires post-hoc explanations. This difference is fundamental in the settings, i.e., building a by-design explainable model vs. explaining a pre-trained model. In the adapted setting, ViaSHAP thus reduces the computational cost needed from training two models to a single one. The use of a by-design explainable model also ensures that the model explains itself, thus providing explanations that align with the prediction. On the other hand, explaining a black-box model through post-hoc methods leaves the explanations open to the inherent randomness issues of statistical learning [1].

It would be good to discuss the recent papers on amortized SHAP...

We thank the reviewer for pointing out the missing recent paper. We will indeed add it to the discussion.

In what problem settings should one decide to use ViaSHAP instead of training a regular model and then apply FastSHAP?

ViaSHAP is particularly beneficial in environments with limited time or computational resources, as it eliminates the need to train and run separate models for prediction and explanation. Additionally, it is well-suited for scenarios where high-fidelity explanations are a central concern, as the model’s predictions are inherently tied to its explanations.

On the other hand, post-hoc explanation methods remain necessary when users cannot access or modify the black-box model or when the model does not support optimization via gradients and backpropagation. We do not propose to replace or discard post-hoc methods, but rather to offer an alternative approach that promotes explainability while addressing computational efficiency.

how much computation time will be saved using ViaSHAP...

When comparing ViaSHAP to KernelSHAP, ViaSHAP significantly reduces computational cost at inference time by avoiding the need to solve a separate optimization problem for each prediction. Therefore, computational cost is reduced by an order of magnitude, as demonstrated in Appendix L and Table 15.

When compared to real-time explanation methods such as FastSHAP, ViaSHAP further reduces computation at training time (by avoiding the training of a separate explainer model) and at inference time (by running inference on a single model instead of two).

More importantly, the computational cost of ViaSHAP at inference time remains the same as a similar model that does not compute Shapley values. Detailed computational cost analyses for ViaSHAP can be found in Appendix N.

Can ViaSHAP be used while training LLMs? What are the tradeoffs and fundamental computational limits?

In principle, ViaSHAP could be adapted to train LLMs. However, there are significant challenges. For instance, LLM training is exceptionally computationally intensive, and incorporating Shapley-based explanations during training would add more computational burden at training time, as ViaSHAP requires sampling from the input and propagating gradients. From the previously evoked results (Appendix N), the inference time should not be harmed by this procedure.

Another possible solution is to use ViaSHAP on pre-trained LLMs, where all the learned parameters of the model are fixed, and one or two output layers are modified and optimized to approximate the explanation.

That being said, we cannot provide an accurate answer without a proper study similar to what we did with tabular data and images.

[1] Rudin et al., Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, 2019