Sum-of-Parts Models: Faithful Attributions for Groups of Features

We thank the reviewer for the constructive review.

W1.

We have added a table to show the similarity and differences between SOP and other methods in Appendix A Table 2.

W2.

Thank you for the suggestion of writing. We include more details for the method in Appendix C in the updated manuscript.

W3.

It is true that we can combine the a priori knowledge with the outcomes of any explainable model. However, the interpretation of different explanations are different. Our model gives faithful groups and their contributions to the final prediction, and the case study shows that the faithful groups correspond to the a priori knowledge, meaning that the semantically meaningful groups are contributing to the prediction.

Q1.

Although RISE is also combining the masks, their prediction is not based on the outputs from the masks. So the post-hoc masks are not directly contributing to the prediction, where SOP's masks are directly used in making the prediction.

Q2.

Section 2:

1. Yes. We apologize for the confusion. It should be $\sum{S \in P}$.
2. $\Delta f$ is the difference between predicting with the whole input x and x without subset $S$. \sum a is the sum of attribution scores for features in S. We would hope that attribution scores for each feature faithfully represent how much that feature contributes to the prediction of $f(x)$. Thus $f(x) - f(x_{\lnot S})$ should be similar to \sum a.
3. In the general case, they could have a real value contribution to the prediction.
4. The total deletion error is $\sum_{S\in\mathcal{P}} \mathrm{DelErr}(\alpha,S)$ where $\mathcal P$ is the powerset of $\{1,\dots, d\}$, as in Definition 1. Thus y-axis is the sum over the whole powerset.
5. Yes.
6. By definition here, only a linear model can achieve 0 error. SOP have 0 error. For SOP, we have access to the internal process of the new model, which is the linear combination of different passes of the old model. As a simple example, suppose $p(x) = x1 * x2 + x2 * x3$ where $x1, x2, x3 \in \{0, 1\}$, and SOP selects group $S1 = {x1, x2}$ and $S2 = {x2, x3}$, and assigns score $\alpha_1 = 1$, $\alpha_2 = 1$. Then, when $x1, x2, x3=1$, $InsErr = p(x_S) - p(0_d) - \sum_i \alpha_i = 2 - 0 - (1 + 1) = 0$. And if we only select $S1$, then $x1 = 1, x2 = 1, x3 = 0$, $InsErr = 1 - 0 - 1 = 0$. This is true for all cases. For images, this is also the same.
7. SOP first generates the groups, and aggregates the separately obtained results from the groups. In this way, the prediction from each group is inherently based only on the group and different from post-hoc methods such as LIME, RISE, and GradCAM. Also the groups in SOP overcome the inherent limitation of individual feature attributions.

Section 3:

1. The extra attention layers in GroupGen and GroupSelect are learned end-to-end and the model is trained on cross-entropy for classification.
2. We have not experimented with other Group Generators. It is true that our group generator is not the best one yet. We leave exploring different group generators for future work.
3. The W’s are learned end-to-end as part of the attention mechanism. The C is taken from the trained model’s classification layer, as it is the representation of the classes. The intuition behind this is to compute which group is most similar to each class and should be used for predicting that class. Sparsemax is used to completely zero-out some segments and some groups for a more human-interpretable set of groups.
4. $C z^T$ means doing matrix multiplication of the class weight matrix and the pooler output of each group, which is equivalent to $y_1, … y_G$. The comma means that there are two things outputted from GroupSelect. The first is the weights for each group for each class $c_1, …, c_G$. The second is the prediction from each group $y_1, …, y_G$.

Section 4:

1. We aggregate the grouped attributions with their group weights. Although this is not how our groups are intended to be used, we need to aggregate to be able to compare with other standard feature attribution methods.
2. For group insertion and deletion metrics, we use the same groups generated from SOP for evaluating other baseline, as they do not inherently come with groups.
3. We have added insertion and deletion examples for grouped and standard cases in Appendix D.3.

Case study

1. While voids and clusters are known, it has been unclear how much they contribute to omega and sigma for prediction. The research question here is how much they affect omega and sigma, and the weights from Group Selector offer some insights for that. Just using the backbone model’s output z is not faithful explanation of how this group contributes to the prediction.
2. One way to use SOP would be to look at top k groups affecting the prediction. We can also plot the histogram of different clusters of groups of groups that correspond to different a priori knowledge concept (such as voids and clusters) to obtain global explanations.

We thank the reviewer for the constructive review.

W1. We have added more images for the groups generated and their comparisons with other explanations in Appendix D.3.

W2. Thank you for the suggestions. The theoretical justification is the motivation for creating grouped attributions. It is true that the example we provide is a toy example. We only show the case for when x=1. In real cases of images, we will have much more complicated interactions between features and thus we do not only have one value of x that is affected by not accounting for interactions between features.

W3. We focus on images but our method is generic and should be able to extend to other modalities.

Yes, it is simple to implement other image baselines. We focus on the faithful-by-construction framework, while can easily modify to achieve other image baselines.

W4. Thank you for the suggestions and we will add formal definition of feature attribution and fix the typos and writings.

For the embedding layer used in GroupGen in ImageNet and VOC, for patches, we are training a new simple convolutional layer that is the same stride size as its patch size, so the layer does not depend on the model. For Cosmogrid, because we are using segments instead of patches, we use the hidden state from the last layer of the model. So it contains more information than just a first embedding layer. If the original model already has reasonable performance on the end task, it is likely that the embeddings from the last layer are good. But it is true that there is some dependence on the model's internal hidden states for the embedding.

Q1. For the monomial example, $\alpha\*$ is 1 for one group of $S_1 = \{1, …, d\}$ of all the features. For the binomial example, $\alpha\*_1 = 1$ for one group $S_1 \cup S_2$, and $\alpha\*_2 = 1$ for another group $S_2 \cup S_3$.

Q2. Shapley values are computed by the number of times including a feature while perturbing all other features that still lead to the same output. They have the property that Shapley values for groups can be computed by summing the Shapley values for individual features. However, it is impractical to compute real Shapley values for images as the space of all features in a 224x224 image with pixel values in 3 channels from 0 to 255 is too large. SHAP computes an approximate by doing Monte Carlo sampling. However, SHAP score is not the same as Shapley values. Also, although summing all the shapley values will lead to the same prediction as the original prediction, this is not the same decision process as the original model and thus not faithful.

Q3.Thank you for pointing out another related work. [Novello, et. al., 2022] assign scores to patches based on the kernel embedding of the distribution of each patch and the output y. As you also pointed out, they focus on individual feature importance, whereas we focus on importance of grouped features.

Q4.​​ Currently the candidate number of groups G is number of features x number of heads (chosen). The sparse multihead attention then choose a subset of it. It is true that some feature groups are redundant. If the two feature groups are exactly the same, then the predictions and scores are also the same. When we interpret the final results, we combine the groups that are equivalent.

It is also possible at test time to add an additional process to remove the redundant groups before going through the backbone model.

GroupSelect assigns a score to each group for each class. The feature group that contributes the most to the prediction of a class is the one with the highest score of that class.

Q5. Each mask/subset captures a different interaction within the group. Since the prediction does not depend on other features outside the group, it can still be interpreted as how this group of features contributes (without need for other features). If all the features are selected within a group, then it becomes uninterpretable. The groups generated by our group generator have around 60% non-zero features selected, so it should be interpretable.

We thank the reviewer for the constructive review.

W1. Thank you for pointing out the missing methods to compare. We havewill added experiments with Archipelago (Tsang et al 2020) for ImageNet and put in the revised Table 1. Due to time constraint, we have not compared with IDG and PLS as they only have code on text and IDG relies on a parser which is not available for images.

W2. Our goal is to have guaranteed faithfulness which none of the other approaches have. The purpose of the metrics is to be consistent with the literature, but none of these metrics can perfectly capture faithfulness. The fact that our approach can guarantee faithfulness while matching SHAP performance is a good thing.

W3. We acknowledge a mistake on our part and thank the reviewer for their careful reading. Indeed we have confirmed with the cosmologists and this was a mistake in the text—the comparison between Omega and Sigma for voids was intended to show that there was no difference (54% to 55%), in contrast to the Omega and Sigma for clusters where there is a significant difference (14.8% to 8.8%). We have updated this in the revised manuscript.

The previous gradient-based method does not give guarantee that the weights reflect how much the features contribute to the final prediction. Since gradient saliency methods are known to be unfaithful [Gilmer et al. 2020], this cast doubt into the cosmology community as for whether these results were true.

In fact, as we state in the third finding in Section 5, our findings are consistent with previous work's finding using gradient-based saliency maps. This can show that our findings are consistent with the findings of existing feature attributions in weak lensing maps, while we give more guarantees in how the weights faithfully reflect how much the groups are used by the model and validate the previous results.

It is important that we can validate their results (so we know that their results are correct), so we know for sure that voids can be more important than clusters for predicting these cosmological parameters. This is not to say that our method has to find the consistent results with fundamentally limited feature attribution methods. It would have been possible that we find opposite results.

Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2020). Sanity Checks for Saliency Maps. arXiv. https://arxiv.org/abs/1810.03292

W4. While our theorems and that of Bilodeau et al 2022 both present impossibility results for feature attributions, we kindly point out that posed characterization of our results is incorrect, both on the assumptions and the resulting theorem.

Bilodeau et al. 2022 put forth a result that says (put simply) that linear models cannot accurately capture complex models, where complexity is measured by having a large number of piece-wise linear components. Indeed, we agree that if we had shown that a linear model is not a good approximation of a highly non-linear model, then this would not be a novel contribution. This is also an unsurprising result (it is not surprising that a linear model cannot approximate a highly non-linear model).

However, our result paints a significantly bleaker picture: we show that a linear feature attribution is unable to model the extremely simple functions in our theorems. Our examples distill the problem to the fundamental issue in its purest form: correlated features. Specifically, we show feature attribution is impossible with only one group of correlated features. This is the polar opposite assumption than that of Bilodeau et al. 2022: our theorem shows that standard feature attributions cannot hope to explain even simple functions if they contain just a single group of correlated features, which is more surprising than being unable to explain complex neural networks functions (and subsumes that case).

Second, we provide not only a negative impossibility result for standard feature attributions, but also a positive result for grouped feature attributions that provides a path forward and motivates the approach in our submission. This is in contrast to Bilodeau et al. 2022, which only presents negative impossibility results standard feature attributions without clear suggestions on where to go.

In summary, our theoretical results differ in Assumption (we assume simple functions with a single correlation whereas Bilodeau et al. 2022 assume complex functions with many piece-wise linearities) Theoretical results (we show both positive and negative results, whereas Bilodeau et al. only show negative results) With this extended discussion, we hope the reviewer can better understand the theoretical results from both our paper and from Bilodeau et al. 2022. We have also included this discussion in the revision in Appendix B.1.

We thank the reviewer for the constructive review.

W1. Thank you for the advice on writing! We had a similar sentence in the abstract, but have used your suggestion to make this more clear in the updated revision.

Specifically, we modify black-box models to have faithful interpretations, by aggregating predictions over varying subsets of features.

W3. You bring up a great point: it is indeed possible that there is another model with the same accuracy that weights voids and clusters differently. However, in cosmology, we understand so little about the dynamics of dark matter and density fluctuations that understanding how any model (that achieves good performance) is weighting voids and clusters is an interesting problem. If there are multiple models that can use different patterns to achieve the same accuracy, then both of these are useful findings for cosmology. The baseline-study from [Matilla et al. 2020] is what our case study builds off from, which uses gradient saliency maps. However, the critiques of gradient saliency maps and their unfaithfulness [Gilmer et al. 2020, Sundararajan et al. 2017] led some of the cosmology community to have doubts on these initial results. Therefore the goal of our case study is twofold: (a) to confirm whether these saliency-based results are consistent with faithful explanations (finding #3) (b) to help cosmologists learn more about the cosmos (findings #1 + #2).

Matilla, José & Sharma, Manasi & Hsu, Daniel & Haiman, Zoltán. (2020). Interpreting deep learning models for weak lensing. Physical Review D. 102. 10.1103/PhysRevD.102.123506.

Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2020). Sanity Checks for Saliency Maps. arXiv. https://arxiv.org/abs/1810.03292

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017

Q1. The aggregation parameter C is learnt. We initialize it from an existing model but it is still being updated. The backbone model is an existing trained model that is completely frozen and not updated—this is typically a traditional deep learning model that has been trained on the same task. This backbone is used within GroupGen and GroupSelect to create the SOP model. The SOP model has additional trainable parameters (the attention weights in GroupGen and GroupSelect) which are trained end-to-end with the backbone model frozen.

Q2. For ImageNet, we take a trained Vision Transformer image classification model from HuggingFace as the backbone model. You can imagine this backbone is then sandwiched between the GroupGen and GroupSelect modules to create an SOP model. Specifically: The GroupGen module uses an attention layer to select the groups from the 16x16 patches in an 224x224 image. Then, each group is passed separately into the backbone ViT model as the original image with patches not in the group masked out. The GroupSelect module then uses an attention layer to select a subset of groups to use to make a prediction. This attention layer uses the last hidden states from these group's ViT model outputs as keys and values, while the query is a learnable parameter initialized with the class weights. These attention weights are then used to aggregate the predicted logits from each group for each class.

Q3. To evaluate how a structure is weighted in making the prediction, we would need to first have the groups of features and then check if they correspond to voids and clusters. For standard feature attribution methods like LIME, they attribute to individual features, and thus do not have a natural group created along the process. We could potentially do it with thresholded LIME. There is still the issue that since the weights from LIME are not faithfully representing how much the model is using these groups of features, even if we make a figure like Figure 5 using LIME, we do not know how to interpret the results.

In fact, as we state in the third finding in Section 5, our findings are consistent with previous work's finding using gradient-based saliency maps. This can show that our findings are consistent with the findings of existing feature attributions in weak lensing maps, while we give more guarantees in how the weights faithfully reflect how much the groups are used by the model and validate the previous results.

Q4. Counterfactual explanations do not inherently give a score for different groups of features. The paper by [Mothilal et al. 2021] you pointed out does give a way to obtain feature attributions from counterfactual explanations. The feature attribution score for one feature is averaged over multiple counterfactual explanations. Thus, it is still giving standard individual feature attribution which has limitations over grouped attributions.

We thank the reviewer for the constructive review. We want to apologize for the confusion on Table one. There was indeed a mistake with the bolding: the deletion score for integrated gradients should be bolded instead of italicized. This is corrected in the revised version. We originally intended the italicized numbers to indicate post-hoc methods, but acknowledge that in hindsight this was more confusing. We have cleaned this up in the revised version.

Q1: In the revision, we’ve updated the appendix (Appendix C) to contain more details on the group generated. We summarize this information for your convenience.

For Group Generator, we first use either a patch segmenter (for ImageNet and VOC) or a contour segmenter (for Cosmogrid) to divide the image into segments. For patches, we separate them into 16x16 patches, and use a simple convolutional layer with patch size 16 and stride size 16 to convert each patch into an embedding of the patch. For contours, we use watershed segmenter in Scikit-learn, and use the pooler output (last hidden state of the [CLS] token) of each segment (masking out everywhere that is not the segment) as the embedding of the segment.

Then, we use the multiheaded self-attention described in the paper to assign scores to each patch. For all the segments, we have attention to all other segments. For example, for an 224x224 image in ImageNet, we have 196 patches, so when we use 2 heads in the multiheaded attention, we have potentially 196x2 group candidates. In order to reduce computation, we randomly dropout others and select only 20 groups to use during training, and also cap it at 50 for testing.

We use sparsemax which projects softmax scores to a simplex so that some scores are zeroed out completely. We use this so that each group does not contain all the segments in the image.