Towards Faithful Neural Network Intrinsic Interpretation with Shapley Additive Self-Attribution

Dear Reviewer Ff8r,

Thank you for your time reviewing our paper and your valuable comments! Your concerns are addressed as follows.

Partial marginal contribution scores might not be meaningful

The purpose of our distillation loss is not to make $\triangle$ match the Shapley value, but rather to make the learned attribution value converge to the average of our intermediate value $\triangle$ (Proposition 3.3). This will eventually make the attribution value the Shapley value of the final output (Theorem 3.4). All our losses are direct training losses instead of regularizers.

The statement in condition 3, "partial sums of values should also equal the label $y$," is misunderstanding. In terms of the physical meaning of $\triangle$ and $f_c$, the objective of Equation 5 is to make the sum converges to the expectation of $y$ given these observable features (i.e., the conditional probability $p(y|x_S)$ in binary classification), which is not the label of one sample but the average in the entire dataset. This is proven in Appendix Proposition D.1. In the scenario you mentioned, "feature $x_1$ does not matter" statistically means its observation does not influence the posterior distribution of $y$. Consequently, the conditional probability $p(y|x_1)=p(y|\emptyset)$, and the contribution of $x_1$ in this case is exactly 0 because $p(y|\{x_1\})=\triangle(x_1, \emptyset) + \phi_0$ and $p(y|\emptyset)=\phi_0$.

In Appendix Figure 4, we also provided a plot of the model prediction using partial observation, which demonstrates that the model can make meaningful predictions with partial observations. Our paper rigorously proves that the Shapley module converges to the desired Shapley value, which demonstrates the effectiveness of these modules.

SASANet addresses traditional Shapley challenges in a potentially inefficient way

Modeling predictions under partially observed results is indeed a more challenging task. Our permutation-based training approach requires 2-3 times more training epochs to converge compared to directly fitting the final results. However, we believe that the benefits of obtaining accurate Shapley values, being able to model non-linear feature-label correlation, and having high robustness in predicting with missing features outweigh the additional training cost. We will try to find a space to clarify this point in our paper.

In this work, SASANet is only being applied to a few small tabular datasets. Especially given the potential brittleness of the marginal contributions and the potential inefficiencies (as described above), it is possible that this method does not translate well to larger tabular datasets (e.g. gene expressions) or non-tabular datasets (e.g. images, text).

In SASANet, our main objective is to model complex non-linear correlations between semantically meaningful features and labels, which is crucial for scientific analysis and discovery. The Shapley value provides a theoretically sound approach for evaluating this correlation. However, its intricate definition makes computation highly complex and involves permutations, making it challenging to handle large-scale features effectively. In the case of high-dimensional inputs like images, text, or gene expressions, where semantics depend on spatial interactions of multiple input points, the individual points (e.g., a single pixel) may lack specific conceptual meaning to explicitly contribute to the ouput. In this case, it is more reasonable to extract concepts from raw features using backbones and attribute concept contributions to the output instead of assigning an accurate Shapley value to each input point. To achieve explainable concept-based inference tasks for CV/NLP, we simply add an attribution layer on top of the concept extraction layer. Notably, depite broad application scenarios, our paper focuses on experimenting with tabular data. Tabular data naturally have explicit concepts with clear physical meanings, thus make the pure attribution ability more explicit to observe.

Minor typos

Thank you for pointing them out! We will correct it.

In the analysis of the accuracy of Shapley values, how were ground truth values obtained for RMSE?

We draw different orders for each sample and directly calculate the Shapley value of the output based on its definition, i.e, sample orders and get the marginal contribution of each feature and get the average. In our experiments, we have added some engineering tricks to reduce the time complexity. For example, we pre-calculate and store the attention matrix and value among features. Then, we inrementally update the representations for each prefix subset in an order. However, it still took us days to get the ground truth results for these 1,000 samples.

Dear Reviewer 2AVx,

Thank you for your time reviewing our paper and your valuable comments! Your concerns are addressed as follows.

Not sufficiently interpretable compared to SHAPNet

SHAPNet calculates the Shapley value for each layer between hidden units, but it cannot accurately calculate the inputs' Shapley value to the final output. Our work's significance lies in being the first to realize global Shapley self-attribution, which is significant considering Shapley has been widely reconized as effective attribution metric. Indeed, attribution aims to accurately quantify the contribution of each input feature to the output, and the Shapley value is explicitly defined as the average change in output when adding a feature. It has a clear physical meaning in showing the effect of features without the need for a transparent intermediate procedure. Extensive post-hoc interpretations studies have been using the Shapley value itself as a model interpretation instead of discussing the mechanism of generating it.

The presentation of the paper needs improvement.

Thank you for your suggestion and we will polish our paper accordingly!

Scalability of the proposed method to complex tasks

In SASANet, our main objective is to model complex non-linear correlations between semantically meaningful features and labels, which is crucial for scientific analysis and discovery. The Shapley value provides a theoretically sound approach for evaluating this correlation. However, its intricate definition makes computation highly complex and involves permutations, making it challenging to handle large-scale features effectively. In the case of high-dimensional inputs like images, text, or gene expressions, where semantics depend on spatial interactions of multiple input points, the individual points (e.g., a single pixel) may lack specific conceptual meaning to explicitly contribute to the ouput. In this case, it is more reasonable to extract concepts from raw features using backbones and attribute concept contributions to the output instead of assigning an accurate Shapley value to each input point. To achieve explainable concept-based inference tasks for CV/NLP, we simply add an attribution layer on top of the concept extraction layer. Notably, depite broad application scenarios, our paper focuses on experimenting with tabular data. Tabular data naturally have explicit concepts with clear physical meanings, thus make the pure attribution ability more explicit to observe.

Computational complexity in the inference stage

Only the Shapley module is used for inference. The purpose of learning marginal contribution is to converge the attribution value to the Shapley value of the final output, as shown in Theorem 3.4. Without learning marginal contribution and simply training a single Shapley module with a regularizer, we cannot obtain the Shapley value. This is because the optimization directions of the regularizer and prediction loss create a trade-off that makes it difficult to accurately converge to the constraint that the attribution value is the Shapley value of the final output.

Why learning $\triangle$ instead of $f_c(x_S)$

Both approaches are theoretically feasible. Modeling $f_c(x_S)$ directly and then taking the difference to obtain $\triangle$ is theoretically equivalent to our current method and does not affect convergence of $\phi$ to the Shapley value of $f$. However, in practical implementation, modeling $\triangle$ allows the model to focus more on the unique contribution of each added input, which aligns with Shapley's definition. On the other hand, modeling $f_c(x_S)$ requires combining the information of the entire feature set and potentially make the effect of each feature vague. Therefore, we chose to directly model $\triangle$ to simplify the operation when we design our model.

Typos about subsripts and loss equation (5):

Thank you for pointing this out and we will double check the notations.

Dear Reviewer JuUA,

Thank you for your time reviewing our paper and your valuable comments! Your concerns are addressed as follows.

About interpretable approaches

Table 1 aims to highlight the advantages of SASANet over other self-attributing models. The two black-box models and classic interpretable models widely used in tabular data are reported to position the expressiveness of these self-attributing models. We are not trying to prove SASANet is superior to self-interpreting models under any other interpretability paradigms. Since different paradigms have different goals and formulations, we believe they cannot be compared.

Under attribution paradigm, our focus is on accurately quantifying the contribution of features to the output, which has its unique significance for feature selection, debias, and discovering feature-outcome correlations. We believe that our current paper effectively demonstrates how SASANet excels other self-attribution models with its well-grounded theoretical basis and high accuracy.

About black-box models

We use these classic black-boxes commonly used for tabular data as references to demonstrate our superiority over other self-attribution methods. We have never claimed SASANet to achieve state-of-the-art black-box models' performance. It is clear that other self-attribution methods have struggled to match MLP and LightGBM, while SASANet does, showing it significantly enhances usability with guaranteed interpretability but acceptable sacrifice on expressiveness, at least nearly reaching MLP. We did not compare transformers because they are not common for tabular data. In terms of MLP size, as stated in the paper, we fine-tuned them to their best performance, since simply training an MLP has weaker supervision than SASANet's loss and more easily overfit. For instance, we find the best performance is obtained by $4 \times 64$ for CENSINCOME and $4 \times 512$ for HIGGS, which can be found in our code and we will summarize them in appendix later.

About Table 2:

Feature masking is common for evaluating the accuracy of an attribution method [1]. It assesses how well the method can distinguish important features that are necessary for accurate predictions. The rationale behind is that removing an important feature for an instance make the model lacks sufficient evidence to support the right prediction. Comparing the relative performance change equals comparing the absolute performance. Since the base performance are the same for all the interperters, which is the model's original performance without removing features.

[1] Alvarez Melis, et. al. "Towards robust interpretability with self-explaining neural networks." Advances in neural information processing systems 31 (2018).

About Fidelity experiments:

In the feature-masking experiment, the correlation between a feature and the outcome is determined by the model itself, not the specific interpreter. Each interpreter's attribution is compared in terms of how good they distinguish more important features, whose removal results in larger performance reduction, remain consistent regardless of the interpreter used. Therefore, removing the same number of features and compare which interpreter makes a better selection is a fair and intuitive comparison. It is statistically unlikely that removing important features on which a trained model relies for accurate predictions make all samples deviate towards a prediction that maintains the original performance. In reality, once important features are removed, the model lacks sufficient evidence, and the prediction becomes more akin to guessing. A guess can be difficult to perform as good as a reasonable inference based on strong evidences.

In addition to feature-masking experiment, Table 3 shows our fidelity in a different perspective, which directly indicates that we learned more accurate Shapley values than post-hoc methods.

Typos

Thank you for pointing out the typos! We will correct them in our paper.

Training time of SASANet

Post-hoc methods usually require no training, while incur high-cost computation for every sample. FastSHAP has a slow training process, preventing us from obtaining results on the HIGGS dataset in Table 2.

Compared to regular training procedure that directly fits the final results, SASANet requires 2-3 times more epochs to converge since it learns additional information, such as Shapley values and predictions under partially observable features. However, we believe this trade-off is reasonable in practice, as it allows us to gain more information for both higher interpretability and insights into non-linear feature-label correlation, as well as an ability to robustly handle any severe missing feature problem.

Page limitation

The Ethical Statements and Reproducible Statements are added according to the author guidelines, which says they are not counted towards the page limit.