Prediction Via Shapley Value Regression

Response to weakness 6 (A comparison on non-tabular data, such as images, is missing. This would be particularly interested, since FastSHAP unfolds its benefits on such data. A comparison on the benchmarks from the FastSHAP paper would be beneficial.):

We again thank the reviewer for highlighting a limitation that can be addressed and improving the quality of the paper.

In order to address this limitation in the paper, we conducted an experiment where we implemented ViaSHAP using 3 image recognition architectures (ResNet50 [5], ResNet18, and U-Net[6]). We evaluated the predictive performance of the 3 models using 1-Top Accuracy (The results are displayed in Table 2). All the models were trained on the CIFAR10 dataset without pre-trained weights (trained from scratch) using 2 masks (samples) per instance and with early stopping after 10 epochs without improvement on the validation split (10% of the training data). The results show that ViaSHAP can have high predictive performance on classical image classification tasks.

Table 2: Comparison of the predictive performance of the implementations of ViaSHAP using U-Net, ResNet18, and ResNet50.

Afterward, we evaluated the accuracy of the Shapley values by following the same approach employed in [1], i.e., by selecting the top 50% important features (according to the explainer) and evaluating the predictive performance of the explained model using only the selected top features (Inclusion accuracy) and without the top features (Exclusion accuracy). We compared the accuracy of approximating the Shapley values of the 3 models as well as FastSHAP as an explainer to the 3 ViaSHAP implementations. The results show that ViaSHAP implementations are more accurate in approximating the Shapley values compared to the explanations obtained through FastSHAP.

Table 3: The table compares the accuracy of the Shapley values using the top 50% of the most important features (according to their Shapley values). It reports the Top 1 accuracy of the explained model. The Inclusion AUC (where higher values are better) and the Exclusion AUC (where lower values are better).

[1] FastSHAP: Real-Time Shapley Value Estimation, Jethani et al., ICLR2022.

[5] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770– 778.

[6] Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. arXiv. https://arxiv.org/abs/1505.04597

Questions:

1- What are the consequences of the novel training scheme. How does the performance compare to the same architecture but with usual training scheme?

We thank the reviewer for highlighting the missing comparison. We conducted an experiment to evaluate the effect of Shapley loss on the predictive performance. We compare $*KAN*^{\mathcal{V}ia}$ to a model with the same architecture but does not compute the Shapley values. The results provided in Table 1 (below) show that the performance of the ViaSHAP is generally better than a KAN model with the same architecture, which suggests that the Shapley component of the loss function may have a regularization effect on the training of the model.

Table 1: A comparison between ViaSHAP ($*KAN*^{\mathcal{V}ia}$) and a KAN model with the same architecture (but not constrained to compute the Shapley values) with respect to the predictive performance (as measured in AUC).

2- Moreover, how does the training time of the novel training scheme (with maskings) compare with the usual training scheme to achieve similar performances for the same architecture?

We reported the training time of a model with the same architecture but does not employ sampling or compute Shapley values, which can be found in Table 4 under columns (No Sampling) in Appendix G.

3- Is there any guarantee to obtain exact SHAP explanations with ViaSHAP (fully agreeing with KernelSHAP). How does the training scheme needs to be modified to guarantee this? Is there a trade-off between accuracy and computational feasability?

The accuracy of the approximated Shapley values depends on several factors, including the number of features, i.e., the number of possible coalitions, the amount of training data available, the representational capacity of the chosen architecture, as well as the nature and distribution of the data. Given all the mentioned variables, we cannot guarantee the exact computation of the Shapley values.

4- In the proof of Lemma 2, why should the global loss be minimized at zero and not some other non-negative value?

The assumption of Lemma 2 is that the global loss is minimized at zero. Obviously, being a sum of square, 0 is the lower bound of this term's values overall. The reviewer asks how we can guarantee that this lower bound is indeed reached. Our argument in this regard is the same as that used by FastSHAP [1]. That is, since ViaSHAP relies on an MLP to learn its values, the universal approximation theorem (Cybenko [2]; Hornik [3]) states that we can learn the Shapley values up to an arbitrary accuracy.

Nonetheless, to push this answer further, we also added a proof for a relaxed version of Lemma $2$. We prove that, as the loss converges to $0$, so do the attributed importances of non-influential criteria. In particular, if the loss has value $\epsilon^2\geq0$, the importance attributed to a non-influential criterion is at most $2\epsilon$. The proof was added in the paper in the appendix, after the proof of Lemma 2.

In practice, it is unlikely for a loss to exactly reach its global optimum. Instead, it approximates it. We assume here that the loss has reached a value $\epsilon^2 \geq 0$. We propose an upper bound on $\phi_i^{\mathcal{V}ia}(x; \theta)$ conditioned on $\epsilon$.

Since the loss is composed only of non-negative terms, this means that:

Thus, in particular, we have the two following cases:

\left| \mathcal{V}ia^{*SHAP*}(x^{S}) - \mathcal{V}ia^{*SHAP*}(0) - 1^{\top}\_{S}\phi^{\mathcal{V}ia}(x; \theta) \right| \leq \epsilon

\Rightarrow \left| \mathcal{V}ia^{SHAP}(x^{S \cup {i}}) - \mathcal{V}ia^{SHAP}(0) - 1^{\top}_{S \cup i}\phi^{\mathcal{V}ia}(x; \theta) - \mathcal{V}ia^{SHAP}(x^{S}) + \mathcal{V}ia^{SHAP}(0) + 1_S^{\top}\phi^{\mathcal{V}ia}(x; \theta) \right| \leq 2\epsilon

\Rightarrow \left| \mathcal{V}ia^{SHAP}(x^{S}) - S\cup I^{\top}\phi^{\mathcal{V}ia}(x; \theta) - \mathcal{V}ia^{SHAP}(x^{S}) + 1^{\top}_S\phi^{\mathcal{V}ia}(x; \theta) \right| \leq 2\epsilon \quad \text{by equation 8}

\Rightarrow \left| \sum_{j \in S \cup {i}} \phi_j^{\mathcal{V}ia}(x; \theta) - \sum_{j \in S} \phi_j^{\mathcal{V}ia}(x; \theta) \right| \leq 2\epsilon

\Rightarrow \left| \phi_i^{\mathcal{V}ia}(x; \theta) \right| \leq 2\epsilon

\Rightarrow \left| \phi_i^{\mathcal{V}ia}(x; \theta) \right| \leq 2\sqrt{\mathcal{L}_{\phi}(\theta)}

5- In the proof of Lemma 3, and in general, I do not understand why the prediction on the masked output $\mathcal{V}(x^S)$ should be equal to the sum $1_S^{\top}\phi^{\mathcal{V}ia}(x; \theta^*)$ - this should only be the case, if the loss is minimized at zero, which is highely unlikely. Does there exist any empirical evidence for this, or a formal assumption?

In the same way as for the previous Lemma (2), this assumes perfect minimization of the loss. Thus, we propose a relaxed variant, where the loss term $\mathcal{L}_{\phi}(\theta)$ was minimized down to $\epsilon^2$ with $\epsilon\geq 0$. Thus, following similar reasoning as in the proof of Lemma $2$, we have that $\forall S$:

$\Big|\mathcal{V}ia^{*SHAP*}(x^S) - \mathcal{V}ia^{*SHAP*}(0) - 1_S^{\top}\phi^{\mathcal{V}ia}(x; \theta)\Big| \leq \epsilon$

We also have:

$\Big|\mathcal{V}ia^{*SHAP*}(x^S) - 1_S^{\top}\phi^{\mathcal{V}ia}(x; \theta)\Big| = \Big|\mathcal{V}ia^{*SHAP*}(x^S) - 1_S^{\top}\phi^{\mathcal{V}ia}(x; \theta) - \mathcal{V}ia^{*SHAP*}(0) + \mathcal{V}ia^{*SHAP*}(0)\Big|$

By the triangle inequality on the right-hand side:

$\Big|\mathcal{V}ia^{*SHAP*}(x^S) - 1_S^{\top}\phi^{\mathcal{V}ia}(x; \theta)\Big| \leq \Big|\mathcal{V}ia^{*SHAP*}(x^S) - 1_S^{\top}\phi^{\mathcal{V}ia}(x; \theta) - \mathcal{V}ia^{*SHAP*}(0)\Big| + \Big|\mathcal{V}ia^{*SHAP*}(0)\Big|$

But observe that all features in 0 are non-contributive since, $\forall S\subseteq N$, $0^S=0$ by definition of the masking operation. Thus, by the bound found in Lemma $2$: $\forall i\in N, \Big|\phi_i(0,\theta)\Big|\leq 2\epsilon$. Thus $\Big|\mathcal{V}ia^{*SHAP*}(0)\Big| \leq 2n\epsilon$.

Thus:

$\Big|\mathcal{V}ia^{*SHAP*}(x^S) - 1_S^{\top}\phi^{\mathcal{V}ia}(x; \theta) - \mathcal{V}ia^{*SHAP*}(0)\Big| + \Big|\mathcal{V}ia^{*SHAP*}(0)\Big| \leq \epsilon + 2n\epsilon$

and we thus derive the following upper bound on the $\phi_i$-wise error as:

$\Big|\mathcal{V}ia^{*SHAP*}(x^S) - 1_S^{\top}\phi^{\mathcal{V}ia}(x; \theta)\Big| \leq \epsilon(2n+1) = (2n+1)\sqrt{\mathcal{L}_{\phi}(\theta)}$.

Response to weakness 5 (ViaSHAP only considers baseline imputations as the masked inputs in the learning paradigm. A discussion for marginal and conditional imputations (as given in FastSHAP) is missing. What are the consequences of this decision? How does ViaSHAP behave under other value functions?):

Thanks for pointing out this weakness. We indeed have to motivate our decision to use the interventional approach to approximate the Shapley values. Our decision is based on the work of Chen [4], which suggests that the interventional approach to computing the Shapley values results in explanations that tend to be more "true" to the data and can be more computationally tractable as well.

[4] Hugh Chen, Joseph D Janizek, Scott Lundberg, and Su-In Lee. True to the model or true to the data? arXiv preprint arXiv:2006.16234, 2020.

We appreciate the reviewer's thorough feedback. We are grateful for the time and effort dedicated to reviewing our paper. We provide some comments and answers to the questions in the following part.

Regarding weakness 1 (One weakness is the novelty of the contribution. Training a black-box classifier, which incorporates the SHAP explanations, has already been proposed with FastSHAP. The main novelty in this work, to fit FastSHAP on the (FastSHAP-)predictions of masked training data, instead of the predictions of the black-box model is not a major contribution.):

We respectfully disagree with the reviewer.

To the best of our knowledge, Shapley values have always been computed post-hoc, requiring the prediction and the black-box model to be available first, which applies to all the known methods, such as, FastSHAP [1], KernelSHAP[2], or the unbiased KernelSHAP[3]. In contrast, a key contribution of our work is that ViaSHAP computes Shapley values before the game itself, i.e., before the prediction takes place.

FastSHAP and our approach, ViaSHAP, are designed for different scenarios. FastSHAP is a post-hoc explanation technique, relying on a pre-trained black-box model to generate predictions, which FastSHAP then explains. In contrast, ViaSHAP is a standalone, explainable-by-design model. It inherently provides Shapley values as explanations for its predictions, without depending on or explaining the behavior of a separate pre-trained model.

[1] FastSHAP: Real-Time Shapley Value Estimation, Jethani et al., ICLR2022.

[2] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 4768–4777, 2017.

[3] Ian Covert and Su-In Lee. Improving kernelshap: Practical shapley value estimation using linear regression. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pp. 3457–3465, April 2021.

We thank the reviewer for the helpful questions that clarify our contribution.

as I understood, you have not used the interventional approach (marginal expectations), but instead used baseline imputations (masking all features not available with the baseline 0), which corresponds to Baseline SHAP [1]? Could you clarify this? Is there any reason not to use the same (conditional) variant as FastSHAP? What are possible implications on the interpretation of these scores?

Indeed, we used the same approach as FastSHAP. We use standard normalization so that the average value over each feature is $0$. Therefore, applying a mask over feature $i$ and setting its normalized value to $0$ is equivalent to setting its value to $E(x_i)$. Consequently, what we explain is $f(x) - f(E(x))$. This will indeed be clarified in the experimental setup.

I think the provided experiments regarding performance should be included in the paper for all architectures, as these seem crucial and intriguing. With the novel training scheme, it is important to understand how it affects the model's performance, which might also differ across architectures. Moreover, I think it would be valuable to compare these performances given similar computational resources.

We totally agree with the reviewer. The results from the additional experiments highlighted by the reviewers have significantly enhanced the quality of the paper. We are in the process of updating the manuscript to incorporate the new experiments, as discussed during the discussion. The updated version will be uploaded before the revision deadline.

I understand that such guarantees are difficult to provide. However, it seems plausible that increasing the number of sampled masks would yield values closer to the actual Shapley values, possibly at the cost of performance. Do you have any insights on such a trade-off? I think together with the performance of your approach, a crucial point of your contribution should address the interpretability aspect your method, i.e. how much your attributions coincide with the SHAP scores, and how to achieve higher interpretability, possibly at the cost of performance. Experiment 4.3 is good first step towards evaluation but does not answer this question.

Our empirical observations support the reviewer's argument, as the similarity of the approximated Shapley values to the ground truth witness more degradation with a higher number of features, i.e., a higher number of required coalitions to compute the exact Shapley values, given a constant number of samples per data instance. An example is the Indian Pines dataset (220 features). We are working on addressing such a trade-off.

We thank the reviewer very much for the engaging discussion and the thoughtful comments. From the sources you proposed [1,2], and another relevant one [3], we found the three following ways of describing interventional Shapley values:

[1]: "One type of what-if analysis (e.g. BShap) performs interventions on the feature, while another (e.g. CES) marginalizes the feature over the training data. The former may construct out-of-distribution inputs, but regularization can ensure reasonable model behavior on these inputs"

[2]: "Interventional: we “intervene” on the features by breaking the dependence between features in S and the remaining features. We refer to Shapley values obtained with either approach as either observational or interventional Shapley values."

[3]: "Notably observational SHAP takes the underlying data distribution with its dependencies into account by using (observational) conditional expectations. Whereas interventional SHAP breaks up feature dependencies via interventions and therefore puts more emphasis on the model."

You are right that we do not use the conditional approach used in [2]. If our terminology is correct, our approach is then interventional, in that that it replaces feature values without taking into account the inter-features dependencies, as opposed to the observational one. This means that we do construct out-of-distribution inputs. In fact, we used the expected values of the features as a baseline for feature removal since we used standard normalization to normalize all feature values and center them around zero.

If you see any confusion wrt our terminology, we apologize, and we will be happy to correct it in the paper in order to avoid any ambiguity wrt that specific point.

On the other hand, in FastSHAP, the authors evaluated 3 approaches in section 5.2 (Surrogate/In distribution, Marginal/Out of distribution, Baseline removal), and we employ an approach similar to number 3 (baseline removal). In particular, the code provided by the FastSHAP authors [4] uses baseline removal as a default. We agree that it would be relevant to evaluate the effect of several masking strategies on the performance of our approach.

In our experiments, we applied the same baseline removal to all compared models (ViaSHAP, FastSHAP, and KernelSHAP) in order to ensure fairness in the evaluation.

[1] Sundararajan, M. & Najmi, A.. (2020). The Many Shapley Values for Model Explanation

[2] Hugh Chen, Joseph D Janizek, Scott Lundberg, and Su-In Lee. True to the model or true to the data? arXiv preprint arXiv:2006.16234, 2020.

[3] Artjom Zern, Klaus Broelemann, Gjergji Kasneci, Interventional SHAP Values and Interaction Values for Piecewise Linear Regression Trees, AAAI-23

[4] https://github.com/iancovert/fastshap, line 186 in fastshap/fastshap.py

We thank the reviewer for the positive feedback and helpful comments. We provide answers to their question below.

1- Does the “# samples” in figure 5 refer to the number of samples used to estimate the loss?

Yes, the figure shows the effect of the number of samples used to optimize the loss functions on the training time. We also show the time required to train a model with the same architecture but does not compute the Shapley values, i.e., does not apply sampling and Shapley loss, which is shown in the figure at # samples = 0.

2- What was your intuition for using KANs? Did you try different models and found that KANs worked well, or do you have an idea as to why they may be particularly suited to this problem?

The reason for using KANs is twofold. At some level, you are right that, by trying them, we noticed their superior performance. On the other hand, a slightly longer-term motivation is that KAN was introduced with interpretability in mind [1, 2]. In particular, their behaviour as a combination of only univariate functions allows for more transparency when adequately leveraged, for instance, through some symbolic regression. While we do not exploit this property here, we believe that showing that KANs perform well in this situation opens interesting future perspectives. For instance, since our Shapley values here are functions of the input, being able to interpret that input would allow us to determine why a criterion is particularly important for a given instance.

3- Could you elaborate on your view of the main contributions of your method compared to FastShap?

We thank the reviewer for this question. Our approach (ViaSHAP) and FastSHAP are designed for different scenarios. FastSHAP is a post-hoc explanation technique, relying on a pre-trained black-box model to generate predictions, which FastSHAP then explains. In contrast, ViaSHAP is a standalone, explainable-by-design model. It inherently provides Shapley values as explanations for its predictions, without depending on or explaining the behavior of a separate pre-trained model.

To the best of our knowledge, the Shapley values were always computed post-hoc, i.e., the prediction and the black-box must be provided first. In contrast, ViaSHAP computes the Shapley value before the game, which is one of the main contributions of the paper.

We hope this clarification adequately addresses the reviewer’s concerns and highlights the unique contributions of ViaSHAP.

4- Nitpicks & Suggestions:

We thank the reviewer very much for the helpful remarks. We will make the necessary updates to the paper to address your points.

[1]- Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks, 2024. URL https://arxiv.org/abs/2404.19756.

[2]- Liu, Z., Ma, P., Wang, Y., Matusik, W., & Tegmark, M. (2024). KAN 2.0: Kolmogorov-Arnold networks meet science. arXiv. https://arxiv.org/abs/2408.10205

In response to the Weakness point number 3 (Unclear Practical Scope and Complexity):

In Appendix K, we presented the computational cost in terms of training time, inference time, and the impact of varying the number of samples on training time. However, we need to clarify that users are free to employ any deep learning architecture, and the complexity of the model will depend on their choice. Moreover, any extra computational cost arising from sampling feature coalitions is incurred solely during the training phase. During inference, the computational cost remains identical to that of a similar architecture that does not compute Shapley values.

We thank the reviewer for their time and the helpful review. In the following, we provide answers to the raised questions.

Question 1:

Indeed, and as described in section 2.2, the Shapley values do not explain $f(x)$ on its own, but the difference in output between $f(x)$ and a baseline output (which we called $f(**0**)$). Usually, the baseline is chosen, as you suggest, as the average output of the model over the training set. In our case, we use linear normalization so that the average value over each feature is $0$. Thus, applying a mask over a feature, and setting its normalized value to $0$, is equivalent to setting this feature's value to the average value over the dataset. Thus, what we explain is actually $f(x) - f(E(x))$. In the context of image classification, one could see this as classifying a fully gray image, which is not expected to belong to any class. Thus it is true that our model assumes that the average input $E(x)$ belongs in no class (akin to a 'grey picture'), since the prediction for this average example is $f(E(x))-f(E(x)) = 0$.

Nonetheless, it is true that, in some cases, particularly with heavily imbalanced datasets, the "average value" $E(x)$ might still be strongly representing one class over others. We agree that we need to address this limitation more clearly in the paper, and thank the reviewer for pointing it out. We see that, from our results, the model still performs remarkably well with this assumption.

Question 2:

Since we allow for the application of a link function to accommodate a valid range of outcomes ($\big(y = \sigma(1^{\top}\phi^{\mathcal{V}ia}(x; \theta)\big)$) as mentioned in section 3.1, the Shapley values computed by ViaSHAP are not on the same scale of KernelSHAP values, i.e., the values of the two models can have different magnitudes but maintain relative importance. Therefore, $R^2$ is not a suitable metric to measure the similarities between the different approximations. We also based our selection of the similarity metrics on the following work [1], which argued that Spearman’s Rank correlation is a suitable measure for comparing explanations in general.

However, to address the valid concerns of the reviewer, we conducted an ablation study to address the effect of applying a link function on the predictive performance of ViaSHAP as well as the accuracy of approximating the Shapley values. The results (Table 1) show that a link function does not affect the predictive performance significantly, while the removal of the link function significantly improves the accuracy of the Shapley values as shown in Table 2 below. This experiment allowed for the use of $R^2$ as a similarity metric, which turned out to be consistent with the results reported using Spearman’s Rank and the cosine similarity.

Table 1: A comparison of the predictive performance (measured in AUC) between $*KAN*^{\mathcal{V}ia}$ with and without a link function applied to the output.

[1]- Rahnama, A. (2024). The blame problem in evaluating local explanations and how to tackle it. In Artificial Intelligence. ECAI 2023 International Workshops (pp. 66–86). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-50396-2_4

Table 2: The Accuracy of approximating the Shapley values of $*KAN*^{\mathcal{V}ia}$ with and without a link function applied to the output.

Question 3:

We thank the reviewer for pointing out this limitation in the paper. In response, we evaluated the explanations of ViaSHAP in comparison with FastSHAP, where ViaSHAP is provided as a black box in FastSHAP. The evaluation metrics, including $R^2$, cosine similarity, and Spearman's rank correlation, demonstrate that ViaSHAP significantly outperforms FastSHAP in terms of the accuracy of the computed Shapley values. The results are shown in Table 3.

Table 3: A comparison between the accuracy of ViaSHAP and FastSHAP in approximating the Shapley values

Thank you once again for your thoughtful review of our work. Based on your review, we have conducted additional experiments and gathered new evidence to address the concerns you raised. We believe the results strengthen the manuscript. We would greatly appreciate your perspective on the latest findings.

Q3- Thanks as well, for using your method as a black-box in FastSHAP. This should illustrate how relatively good your method is. It is very impressive that ViaSHAP is generating better SHAP values than FastSHAP. There are also datasets like MC1 and Ada Prior that FastSHAP’s $R^2$ compared to ground truth is really off while ViaSHAP is doing great.

1. What do you think explains this amount of difference? FastSHAP’s $R^2 \approx 0$ in some of the datasets. It means it does not explain any of the variability in the ground truth SHAP values. Or put simply, it is the same as a simple baseline model that always predicts the mean vector for each data point. It was unexpected to see such a performance.

2. You mentioned you used the default setting for all algorithms. If I’m not wrong, the default setting for FastSHAP is their surrogate model that computes the conditional SHAP, as opposed to KernelSHAP that computes the interventional SHAP, used for generating ground truth values.

3. Would be great if you could also include the setting for the FastSHAP you used, including the network details and the hyperparameters. Please excuse me if it is already in the Appendices but I overlooked!

W3 / Unclear Practical Scope and Complexity - My point is more on the complexity of controlling the training dynamics of the predictor. That is true that the model is not bound to use a specific model, but the predictions are always eventually going to be the sum of SHAP values. So comparing it to the alternative of having the full control on directly predicting the output from the inputs, it is naturally a more restrictive class of predictors with less degree of freedom. If the authors could highlight more on the practicality of such a method, it can strengthen their arguments.

Thanks a lot for going through my review and providing the extra information I asked for. I will go through each part individually.

Q1- Thanks for your answer and referring me to the corresponding section. There are still points that I would like to discuss with the authors and hopefully clarify.

Indeed, and as described in section 2.2, the Shapley values do not explain $f(x)$ on its own, but the difference in output between $f(x)$ and a baseline output (which we called $f(**0**)$).

Reading line 122 I understand you are using the vector 0 as your baseline, so you are assuming your baseline value to be $f(0)$. The baseline value is dependent on the version of SHAP values you are computing (this is discussed in many papers including [1], [2]). As mentioned in line 327 and given that in your experiments you are comparing your values with KernelSHAP’s values as ground truth, I can infer that interventional SHAP is what you are computing with your method (by the way, Chen et al. (2020) suggested that interventional SHAP is more true to the model than data, contrasting with what is mentioned in line 328). In the interventional setting, the baseline value is $E_{X\sim D}[f(x_S, X_{-S})]$, where $D$ is a background dataset. This means that you are assuming this baseline value to be zero. Which I believe is not obvious at all and requires extensive justifications. Assuming $E_{X\sim D}[f(x_S, X_{-S})]$ to be a proxy for $E[f(x)]$, this is as if you assume the expected value of the predictor to always be zero. Imagine you expect this from a regressor that is supposed to predict the height of people based on some features.

 In our case, we use linear normalization so that the average value over each feature is 0. Thus, applying a mask over a feature, and setting its normalized value to 0, is equivalent to setting this feature's value to the average value over the dataset.

I could not find the details on this “linear normalization” in the paper. If you are referring to the batch normalisations done in your MLPs, this was not part of your assumptions in your Section 2.2. In line 111, it is said that $f$ is “a trained model” with no extra constraints. Even for KANs that you used this is not an obvious assumption. It would be great if you could detail more on what do you mean by “we use linear normalization” and “setting its normalized value to 0”.

applying a mask over a feature, and setting its normalized value to 0, is equivalent to setting this feature's value to the average value over the dataset.

Again, here it is not obvious why setting the value of the masked values to zero is as if they are set to the average value over the dataset. This assumption that all the features in the dataset is centered at zero is something that neither I see in the paper nor is realistic.

 Thus, what we explain is actually $f(x)-f(E(x))$.

Let’s imagine we accept $E[x]=0$. If you are assuming $1^T \phi_x = f(x)$ as ViaSHAP is supposed to be a predictor that generates the final predictions, then you are also assuming $f(0)=0$. I can accept that with the constraints you considered in your trainings, ViaSHAP might become an estimator with this attribute. But this does not necessarily sounds a realistic assumption for an arbitrary predictor and needs to be both clearly stated and justified in the paper.

Nonetheless, it is true that, in some cases, particularly with heavily imbalanced datasets, the "average value" E(x) might still be strongly representing one class over others. We agree that we need to address this limitation more clearly in the paper, and thank the reviewer for pointing it out. We see that, from our results, the model still performs remarkably well with this assumption.

I strongly advise the authors to work out the theoretical aspects of their assumptions before looking into the experimental results. High accuracies in empirical results do not necessarily guarantee the soundness of the method. I believe the theoretical aspects needs to be clarified first, and then be backed up with strong empirical results. I also wish I could see the limitations being mentioned and addressed in the updates as promised by the authors, but I believe they are still missing.

Q2- Thanks a lot for providing $R^2$ scores between ViaSHAP’s outputs and ground truth. It is impressive how $R^2$ scores are also high in most of the cases. I still have few questions.

1. I did not find the details on what has been used as the background dataset?

2. Why do you think on Helena the $R^2$ achieved is that low?

[1] Sundararajan, M. & Najmi, A.. (2020). The Many Shapley Values for Model Explanation [2] Hugh Chen, Joseph D Janizek, Scott Lundberg, and Su-In Lee. True to the model or true to the data? arXiv preprint arXiv:2006.16234, 2020.

We greatly appreciate the reviewer's questions and feedback, as they contribute to refining and clarifying the paper.

considering the sum of SHAP values to be $f(x)−f(0)$, how are you ensuring $f(0)=0$?

Since $f$, in our case, is a ViaSHAP model, $f$ has been optimized to minimize the following loss function and predict $f(\mathbf{0}) = 0$.

$\sum_{x \in X} \underset{p(S)}{\mathbb{E}} \Big[\Big(\mathcal{V}ia^{*SHAP*}(x^S) - \mathcal{V}ia^{*SHAP*}(\mathbf{0}) - S^{\top}\phi^{\mathcal{V}ia}(x; \theta)\Big)^2\Big].$

Let us assume a perfect minimization of the loss:

$\Big(\mathcal{V}ia^{*SHAP*}(x^S) - \mathcal{V}ia^{*SHAP*}(\mathbf{0}) - S^{\top}\phi^{\mathcal{V}ia}(x; \theta)\Big)^2 = 0$

Therefore, if $x$ is a vector of zeros (or mean values): $\Big(\mathcal{V}ia^{*SHAP*}(\mathbf{0}) - \mathcal{V}ia^{*SHAP*}(\mathbf{0}) - S^{\top}\phi^{\mathcal{V}ia}(x; \theta)\Big)^2 = 0$

$\Rightarrow \mathcal{V}ia^{*SHAP*}(\mathbf{0}) = 0$

In practice, it is unlikely for the loss to reach its global optimum. Consequently:

$\Big(\mathcal{V}ia^{*SHAP*}(\mathbf{0}) - \mathcal{V}ia^{*SHAP*}(\mathbf{0}) - S^{\top}\phi^{\mathcal{V}ia}(x; \theta)\Big)^2 \leq \epsilon$

However, as the loss function converges to 0, so does $\mathcal{V}ia^{*SHAP*}(\mathbf{0})$

As we mentioned before, in some cases, specifically with heavily imbalanced data, the "average value" $\mathbb{E}(x)$ may still predominantly represent one class over others. This limitation also applies to FastSHAP when using a baseline removal value function. However, unlike ViaSHAP, which explicitly optimizes $f$ to ensure that $f(\mathbf{0}) = 0$, FastSHAP does not guarantee this property as FastSHAP has no control over the black-box model. Regardless of the values of $f(\mathbf{0})$, the explanations of both ViaSHAP and FastSHAP are $f(x)−f(\mathbb{E}(x))$. 

Nevertheless, our results demonstrate that ViaSHAP performs remarkably well under the previous assumption. In future work, we could add a learned or fixed bias to account for the shift of value $f(\mathbf{0})$ without losing any model properties.

That is true if we only optimize the Shapley component of the loss function. In fact, what we optimize is the following:

$\sum_{x \in X} \sum_{j\in M}\Big(\beta\cdot\Big( \underset{p(S)}{\mathbb{E}} \Big[\Big(\mathcal{V}ia^{*SHAP*}_j(x^S) - \mathcal{V}ia^{*SHAP*}_j(0) - S^{\top}\phi^{\mathcal{V}ia}_j(x; \theta)\Big)^2\Big]\Big) - \Big( j\text{log}(\hat{j})\Big) \Big),$

where $\hat{j}$ is the predicted probability of class $j \in M$. Therefore, if forcing $f(0)=0$ results in a poor predictor, the loss value will rise, and the optimizer will shift $f(0)$ to a more suitable value for the predictor.

Again, regardless of the values of $f(0)$, the explanations of ViaSHAP are $f(x)−f(E(x))$.

It is true that the regression tasks are also in ViaSHAP's scope of application. For instance, the cross-entropy component in the loss function can be replaced with mean squared error to train a ViaSHAP model for regression.

In an ideal scenario, such as a balanced classification dataset, perfect minimization of the Shapley loss, and convergence of the prediction loss to a minimal value, the model ensures that $f(0)$ equals 0.

As mentioned earlier, such ideal conditions are rare in practice. However, $f(0)$ will converge closer to 0 as long as doing so does not lead to an increase in the combined loss values (Shapley loss + prediction loss).

Only iterating what is suggested by you; I see that the Shap component of the loss results in a predictor $f$ with $f(0) \approx E[f(x)]$ of zero at the global optimum. On the other hand, your prediction loss results in a predictor $f$ with $E[f(x)]$ somewhere around the expected value of the distribution of data, in its global optimum. These are two optimums with fundamental differences. I am not convinced minimising their sum will guarantee stable training and fundamentally sound optimisation. Especially when the expected value of the label distribution is different than $0$. I believe ensuring correct SHAP values and predictions at the same time is not as easy as adding the two losses and optimising the sum.

I appreciate the attempt of the authors for trying KANs on computing SHAP values. However, the core idea behind the method, which seems to only be the addition of the prediction loss to FastSHAP while heavily constraining the original method by limiting it to only zero baselines as the value function, does not sound very novel to me. For such a limited novelty, at least a thorough analytical analysis on methodological soundness and optimisation dynamics is needed.

I am very happy that the authors could still present good empirical results on the classification datasets tested. However, I think the work is prone to fundamental issues while not also being novel enough to be accepted in this conference. I will not reduce my score in favour of the good empirical results provided and experimenting KANs for computing SHAP values.

The previous set of experiments will be added to the paper using the complete set of 25 datasets and all corresponding statistical significance tests.

However, the core idea behind the method, which seems to only be the addition of the prediction loss to FastSHAP...

It is possible that we oversimplified the description of the proposed approach, that it sounds as if we do nothing beyond adding the prediction loss to FastSHAP. While we acknowledge that our work is based on FastSHAP, there are key differences that distinguish our approach:

1- There is no black-box model provided to explain, which is a fundamental assumption of any Shapley-based explanation method

2- Shapley values are provided prior to the predictions

3- The model learns to explain itself

Thanks for your answer and providing new empirical results on the classification tasks tested.

Unfortunately, the analytical discussion is still missing and I have nothing new to add to my last comment. I still find the novelty of the method limited, and see the restriction on using zero baseline for the value function a weakness that I do not see a fundamental reason for. This is something that was also mentioned by reviewer aGxt.

While I appreciate the effort the authors put on providing more empirical results, my opinion stays unchanged and I think the work in its current form does not pass the acceptance threshold.

We thank the reviewer for their positive feedback and helpful comments. We will answer the questions in the following part.

1- When should I use Shapley value regression instead of the traditional approach?

We thank the reviewer for pointing out the need for clarification regarding the novelty and relevance of our approach. Regarding KernelSHAP, the obvious advantage of our approach (and of FashSHAP) is to avoid running an optimization problem for each example one wishes to explain. Now, to explain the difference with FastSHAP: FastSHAP and our approach, ViaSHAP, are applied in two different settings. FastSHAP is a post-hoc explanation method; it requires a black-box, pretrained model to provide the predictions, over which FastSHAP is trained to explain its predictions. On the other hand, ViaSHAP is its own standalone model, explainable by-design through providing its own Shapley value. It does not explain the inner workings of a separate, pretrained model.

To the best of our knowledge, Shapley values have traditionally been computed in a post-hoc manner, meaning that the prediction and the black-box model must first be available. In contrast, the key contribution of our work is that ViaSHAP computes the Shapley values before the game itself.

2- From what I can see the main reason for using Shapley value regression is to avoid additional computational overhead at inference time. However, to avoid that, we need to constrain our choice of architecture, adapt the training process, etc.

The choice of architecture is constrained in exactly the same way as any black-box used with FastSHAP, in the sense that both models are performing the same classification task, and can thus have similar architectures up to the last layer. The difference is that FastSHAP then needs to run its explainer on top of the black-box to draw the explanation. On the other hand, for ViaSHAP, the training of the predictor is conditioned on providing the explanation, thus making the entire process single-step. This means that ViaSHAP avoids running two models sequentially, but only runs one. Therefore, the user has the freedom to select any suitable deep learning architecture, e.g., MLP, transformers, computer vision, and image processing architectures.

3- I would like to see experiments that compare the training times and inference times of the two approaches, Shapley value regression, and the traditional approach. How much more time do I need for training when using Shapley value regression? How much more time do I need at inference time if I want to calculate Shapley values in the traditional way?

Regarding the training, FastSHAP also requires training the black-box first, then training the explainer by predicting the outputs of several maskings for all data points over several epochs. On the other hand, ViaSHAP does all of this in a single training (since there is a single model). Since, once again, ViaSHAP can have the same architecture as the black-box used in FastSHAP, training ViaSHAP is, at worst, as slow as training the black-box of FastSHAP. Furthermore, any additional computational cost associated with sampling feature coalitions is incurred exclusively during the training phase. At inference time, the computational cost of ViaSHAP is identical to the same architecture that does not compute Shapley values. In Table 4 in the paper, for each dataset, we report the time required to train a model using the same architecture as ViaSHAP, but the model does not compute the Shapley values, i.e., sampling and Shaply loss were not involved in the training, which can be found under columns (No Sampling).

In the following table, we report the time required to explain 1000 instances using KernelSHAP and ViaSHAP ($*KAN*^{\mathcal{V}ia}$) on 6 datasets using the same hardware setup as described in section 4.5 in the paper.

Table 1: The time required to explain 1000 predictions using KernelSHAP and ViaSHAP.

4- Clarifications:

Lines 430-432: Can you elaborate on what you mean by the sentence "KernelSHAP requires more than 2000 samples and model evaluations per data example to achieve the same accuracy level of ViaSHAP on the Adult, Elevators, and House16H datasets."? What is the "accuracy level"? Earlier it was mentioned that the ground truth is computed using KernelSHAP, so I am not sure against what the accuracy is measured.

Thank you for pointing out this caption that indeed requires more clarification. We computed the ground truth Shapley values using the unbiased KernelSHAP [1], which continously samples coalitions until convergence and updates the learned Shapley values. After convergence, we consider the values as the ground truth. Before convergence, the values are approximations that are being improved after each iteration of coalitions sampling and evaluation of the black-box model on the sampled coalitions. The figures demonstrate that KernelSHAP initially provides approximations that differ significantly from those computed by ViaSHAP. However, as KernelSHAP refines its approximations, the similarity to ViaSHAP's values increases. The figures further illustrate that KernelSHAP requires over 2000 samples and corresponding model evaluations to achieve a high level of similarity to ViaSHAP. In contrast, ViaSHAP computes these values within milliseconds, as shown in Table 1 above.

Lines 457-459: Why is that observation "remarkable"? It seems intuitive that the similarity will improve as increases.

What is remarkable is that the predictive performance on the classification task (not the accuracy of the Shapley values) remains mostly unaffected by the exponential increase in the scaling hyperparameter, $\beta$, which implies that users can substantially increase $\beta$ to learn more accurate Shapley values without compromising the model's high classification performance.

5- Minor: Figure 2. I think there is something wrong with the loss function. It is defined as a pair of two numbers.

In Figure 2 we show how the components of equation 6 are computed, i.e., ($\mathcal{V}ia^{*SHAP*}_j(x^S) - \mathcal{V}ia^{*SHAP*}_j(0)$) and ($S^T \phi^{\mathcal{V}ia}_j(x; \theta)$). We will update the Figure to eliminate the confusion.

[1]- Ian Covert and Su-In Lee. Improving kernelshap: Practical shapley value estimation using linear regression. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pp. 3457–3465, April 2021.

With the aim of engaging in the discussion of the paper, although it is not proposed by the authors, I think it is possible to extend the method straightforwardly to work with other non-neural based methods and thus it should not be considered a weakness of the framework proposed.

For example, one could imagine having one gradient boosting model per dimension of $\phi(x)$ and train it using the standard GBT training procedure for all trees. In particular, if $\phi(x)_i = GBT_i(x)$ then one can use the objective proposed by the authors ( equation (6) in the paper + model fit term) take the derivative with respect to each tree $d L / d \phi_i(x)$, and optimize. This strategy is often used by other methods and falls right in line with tree based models such as [1]. See for example [2] where this strategy is used.

With a bit more detail just to be super clear on what I mean, we model each dimension of the feature mapping $\phi(x) \in \mathbb{R}^d$ using a separate gradient boosting tree model. Specifically, we define:

where $\text{GBT}_j(x)$ denotes the gradient boosting tree model for the $j$-th dimension.

Boosting Process Handling Multidimensionality of $\phi(x)$

At each iteration $t$ of the boosting process, we have the current estimate of $\phi(x)$ denoted by:

Our goal is to minimize an objective function $L(y_i, \hat{\phi}(x_i))$ (for example eq (6) + model fit term, or eq (7) in the paper) over the dataset $\{(x_i, y_i)\}_{i=1}^N$. The boosting process involves the following steps:

1.Compute Negative Gradients:

For each sample $i$ and each dimension $j$, compute the negative gradient of the loss function with respect to the current estimate:

This results in a gradient vector $g_i = [g_{i,1}, g_{i,2}, \dots, g_{i,d}]^\top \in \mathbb{R}^d$ for each sample. (which should be straightforwardly computable with the loss proposed by the authors.)

2.Fit Base Learners for Each Dimension:

For each dimension $j$, fit a regression tree $h_j^{(t)}(x)$ to the negative gradients by solving:

where $\mathcal{H}$ is the space of regression trees.

3. Update Estimates of $\phi(x)$

Update the estimates for each dimension using the fitted base learners:

where $\eta$ is the learning rate.

The resulting method (or similar versions thereof) could be used in exactly the same way the authors propose it in the paper. Therefore, unless I am missing something (which is not impossible) I think the method can be easily extended to other paradigms without the need of backpropagation.

[1] Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5), 1189–1232.

[2] Duan, T., Avati, A., Ding, D. Y., Basu, S., Ng, A. Y., & Schuler, A. (2019). NGBoost: Natural gradient boosting for probabilistic prediction. CoRR, abs/1910.03225. Retrieved from http://arxiv.org/abs/1910.03225