Metric-Driven Attributions for Vision Transformers

Reviewer DmfJ, thank you for identifying clarity issues with our definitions, and raising questions about paramater selection. Here is our response.

Q1: In line 123 the notations for $MR^{ins}, MR^{del}$ were used before definition. I would recommend defining them before using them.

We have not been able to identify this issue in the text. Can you please provide more details so we may find this issue and correct it

Q2: In Eq (3) you defined "$|.|$ is the sum of all selected values" (line 129) - where it appears twice in the equation and in general overlaps with the norm symbol. I would recommend not shortening it and writing it explicitly: $\sum_{i=1}^M|A_i|$ where $M$ is total number of attributions.

Thank you for pointing out the confusing notation of Eq (3). We have updated the equation to provide clarity and remove the ambiguity of its original definition.

Q3. Can an ablation study on $\kappa$ be shown?

We have updated the supplementary material to include a $\kappa$ ablation evaluation as requested. Please see Appendix A.6 for the reasoning behind our selection of the $\kappa$ parameter. Through qualitative and quantitative analysis in this section, we reinforce that $\kappa = 0.5\%$ is the best selection for generating $A^{dense}$.

Q4. As I understand it, you are selecting an equal number of elements at each time-step ($D^2/N$). Could you provide an intuition why?

If this is in reference to the definition of the insertion and deletion tests on line 93, we are following the original authors in their definition of the tests [1]. They opt to perform perturbations of groups of pixels instead of single pixels, as selecting a group of pixels is more likely to select a feature than a single pixel would. If this is not what you meant, can you please provide clarity?

Q5. How would you suggest tuning the $\gamma$ parameter for an unknown image?

The selection of $\gamma$ for unknown or unseen images will be dependent on the desired application of the attribution. For downstream tasks such as confidence or out-of-distribution measures, we would recommend $\gamma = 0$, as we show a lower gamma will provide only the most important features, and thus the most important model information. For human-facing model evaluation, we would recommend $\gamma = 1$ as it may be preferable for human understanding. However, MDA with $\gamma = 0$, $\gamma = 1$, or any value in between can be calculated at the same time for nearly zero cost. Due to the design of the algorithm, the $\mathcal{O}(M^2)$ ordering process is performed only once, and the patch values are assigned for $\gamma = 0$ and $\gamma = 1$ without model calls. MDA $\gamma = (0,1)$ is then an interpolation between these sparse and dense attributions. Therefore, MDA $\gamma = 0$ and $\gamma = 1$ can generated for any unseen image without meaningful computation increases and thus an evaluator can have both sparse and dense interpretations of the model decision if necessary.

Q6. Explainability could be beneficial in debugging misclassified images. It would be nice if you provide examples of misclassified cases and demonstrate how MDA addresses them.

Thank you for this suggestion. There is a great amount of existing literature which explores the use of attributions for confidence measures [2], out-of-distribution detection [3], and model debugging [4]. In this paper, we focus solely on the computation of attributions as is common in the literature, and thus we decided to focus our revisions on the key points which question our method's quality. However, we think this is an interesting future MDA application that should be explored.

S1. Overall, I find similarities between your approach and general spectral analysis. I believe this is good direction for further research that has potential.

Thank you for this suggestion. We will take into consideration the relationship between MDA and spectral analysis when researching applications of MDA in the future.

[1] RISE: Randomized Input Sampling for Explanation of Black-box Models, BMVC 2018
[2] Attribution-Based Confidence Metric For Deep Neural Networks, NeurIPS 2019
[3] GAIA: Delving into Gradient-based Attribution Abnormality for Out-of-distribution Detection, NeurIPS 2023
[4] Explanation-based human debugging of nlp models: A survey, MIT Press 2021

Reviewer 7fM8, thank you for identifying areas in which our definitions could be clarified and for proposing interesting questions about our method. Here is our response.

Q1: Equation 2 does not correspond to the area under the MR curve, directly summing all $MR_k$ would not give the AUC.

Thank you for identifying the issues with Eq (2). We have corrected this such that Eq (2) properly calculates the AUC of the model response curve.

Q2: Line 97 says the "MR curve is formed on the range $[0,1]$". What variable is this range referencing?

By stating the "MR curve is formed on the range $[0,1]$", we are discussing the $MR_k$ values which form the $MR$ curve. In this line specifically, we are referencing the $MR^{ins}$ curve. We are explaining that this curve begins at $0$ and grows to $1$ as each $MR_k$ is determined throughout the perturbation test process. In addition, for $MR^{del}$ we use the $[1, 0]$ notation to describe that this curve begins at $1$ and decreases to $0$ as the perturbation test progresses.

Q3: Can this approach be applied to other models?

This approach can, in theory, be applied to other models as it only requires the softmax scores of the predicted model class. For a CNN model, the input would be separated into patches just like the ViT models, but the ideal patch size for each CNN model variation would have to be considered. This is in contrast to ViT models which have an obvious choice of patch size due to their architecture which is not present for CNN models. More realistically, this approach would be valuable to apply to other token-based models such as BERT where each token in the input is treated as a patch to order. We think this question is valuable to consider for future work with and exploration of the MDA method.

Q4: Why is the patch order found using two separate processes? I doubt the overall ordering found by the two processes are different between them.

We use two processes to find the ordering of the patches because the orders which optimize the insertion and deletion scores ($\mathcal{I}^{ins}$ and $\mathcal{I}^{del}$) are not guaranteed to be the same, and are often not. We provide proof of this by observing the results presented in Figure 7 and 8 as well as Tables 7, 8, and 9 in Appendix A.3. If insertion and deletion always yielded the same order, we would see no difference in the scores with respect to the choice of $\tau$ which changes the balance of insertion-only or deletion-only optimization. In addition, please see the newly added Figure 8 and its associated text in A.3 which provides visual proof and discussion of how the order varies between the two optimizations.

Q5: From what I understood, the only differences between the two are: i) the starting image is first a blurred and then a black image, and ii) the ordering is first found from most to least important, and then from least to most important. Is finding the "deletion-based patch ordering" therefore the same process as insertion?

We understand and recognize that the insertion and deletion processes both resemble an insertion process, but this for good reason, as we have shown a solo insertion optimization is not ideal. Beginning with insertion optimization, we follow the oringinal design of this test to find an order which maximizes the model response at each step. Because insertion measures the minimum features needed for classification, this is easy to solve. On the other hand, deletion, by definition, will ideally remove patches (replace them with black) from the original image which influence the largest change in model response first. For the reasons stated in Section 3.1, `"Deletion-Based Patch Ordering", it is challenging to find this order, so we perform an inversion of the deletion test. By inserting patches to a black image and selecting the patches which influence the lowest change in model response, we are still computing deletion, but in a reverse order. Once we invert this resulting ordering, we have done the equivalent of a deletion test.

Q6: Why start from a blurred image for insertion and from a black image for deletion?

For the implementation of these processes, we chose the blurred image and black image baselines for insertion and deletion as this are how the tests are defined by their original authors [1] as explained in Section 2.1. Since we aim to optimize for these tests, using the same baselines allows us to have proper evaluation of the model response.

[1] RISE: Randomized Input Sampling for Explanation of Black-box Models, BMVC 2018

I see that the other reviewers also share my concern regarding the same metric being used for optimization and evaluation. This is a major limitation. Besides this, the computational complexity is also a significant limitation.

Reviewer Q9o9, thank you for identifying areas for clarity improvement. Here is our response.

Q1: During insertion (page 4), due you use the strongest response with respect to the ground truth? Maybe give a precise definition of model response to avoid confusion.

In our method, there is no ground truth. The patch order is found strictly from the measured model response. Our patch ordering processes are performing the insertion or deletion tests as defined in the related work (Section 2.1), and thus at each step we calculate the model response via Eq (1) to indicate the value of the potential patch choices. Let us clarify this process for insertion. At each step $k$, we are starting from the image $P_k$ with $k$ patches inserted. From this image, we create $N - k$ unique images where each has $1$ of the currently unused $N - k$ patches inserted. We measure the $MR$ of each of these images via Eq (1), and the image with the highest $MR$ indicates that it had the best patch inserted. There is no ground truth available or used for this process.

Reviewer zur2, we thank you for your valuable questions and commentary. Here is our response.

Q1: Why did you choose to use perturbation-based metrics to optimize? Why are metrics such as localization ability, faithfulness, visual quality, sanity check, etc. ignored?

Thank you for this excellent question. There are indeed many different explainability metrics that could have been selected for optimization. The MDA framework could easily be adapted to optimize many of these alternative metrics. However, we selected to optimize the insertion and deletion metrics [1] and their MAS extensions because they are the most popular metrics in the literature. Moreover, unlike many faithfulness metrics [2, 3, 4, 5] the score of the MAS metric is not only dependent on the order of the attributions, but also the magnitude, which disambiguates many different attribution maps that would otherwise score similarly but visually look very different. Furthermore, we have revised the paper to show that optimizing for the MAS metrics gives very good results for additional explainablity metrics such as the ImageNet Segmentation pointing game [6] (Table 2), negative and positive perturbation [4, 6] (Table 3), and monotonicity [3] (Table 6), where it outperforms the SOTA methods on all tests. The inclusion of these tests illustrates MDA's ability to succeed in localization ability, faithfulness, and visual quality.

Some of the other metrics are not very practical for assessing the quality of an attribution map. For example, the Sanity Checks [7] or ROAR [8] methods require model editing or retraining and are therefore only used in a limited number of studies (while they might be cited substantially). In addition, we aim to provide single-image explanations, which would be not be as feasible with these methods since they often operate over an entire dataset. Due to the slow nature of performing these sanity checks, we could not include evaluation results in this response time frame.

[1] RISE: Randomized Input Sampling for Explanation of Black-box Models, BMVC 2018
[2] Is Attention Interpretable?, ACL 2019
[3] One explanation does not fit all: A toolkit and taxonomy of AI explainability techniques, ACL 2020
[4] ERASER: A Benchmark to Evaluate Rationalized NLP Models, ACL 2019
[5] Rethinking attention-model explainability through faithfulness violation test, ICML 2022
[6] Transformer interpretability beyond attention visualization, CVPR 2021
[7] Sanity checks for saliency maps, NeurIPS 2018
[8] A benchmark for interpretability methods in deep neural networks, NeurIPS 2019

Q2: The idea is similar to Shapley Value. Could the author discuss the advantages and unique contributions of MDA in terms of effectiveness and complexity?

We appreciate you raising the discussion of the similarities between our MDA method and the popular perturbation-based SHAP method [9]. We have included a direct comparison of these two methods in the new Appendix A.9. From those quantitative and qualitative results, we see that MDA is not only clearly outperforming SHAP in attribution quality, but it also has a significantly faster runtime.

[9] A unified approach to interpreting model predictions, NeurIPS 2017

Q3: As shown in Tables 1 and 2, the proposed MDA method exhibits performance degradation on Deletion. Is there any analysis of this phenomenon? Is this related to how the patch order is determined in step 1? Why does MDA perform worse in deletion?

The MDA framework does exhibit weaker performance in the deletion test as shown in the tables. However, this behavior is not surprising and it is indeed due to the patch order determined by step 1. We originally discussed this behavior with our $\tau$ ablation study. We will explain this behavior with reference to Figure 7 and Tables 7 - 9 in Appendix A.3 Since MDA optimizes for both insertion and deletion, the insertion - deletion score is maximized, but the two individual scores may be slightly worse as a result. This is illustrated in these figures and tables where we see that optimizing only for deletion will maximize the deletion score. Figure 7 shows that we can always select a value of $\tau$ (which balances the insertion and deletion joint optimization) that beats all tests in deletion or insertion if that is a desirable behavior. However, it is best to have an attribution which both tests prefer equally, rather than one which only performs well in one test. This leads to the choice $\tau = 0.9$ to balance insertion and deletion as seen in Figure 7.

First of all, thanks for the impressive additional comprehensive experiments, I think it is absolutely shed more light on the performance of your proposed approach.

1. Additional tests with different metrics - I think that the superior results based on common metrics are giving more credibility to the transferability of your approach. I tried to compare results with other papers (i.e. [1]) and found it difficult since the experiments are based on different architectures (you are reporting on ViT-B\32 whereas Chefer et al. report based on ViT-B\16). In general I think that compare against more architectures would be better, however, since also [1] did the same (report only on ViT-B\16) I think it is fair enough.
2. Timing evaluation - as you mentioned in the comment, I agree that better explanation can come in the expense of slower runtime. Nevertheless, following your additional timing experiment, MDA is order of magnitude slower than all other approaches, and I believe that this would exacerbate for larger architectures. If we take CLIP [5] as an example for a prominent architecture that ViT embedded in, then it trained on ViT-B\32, ViT-B\16 and ViT-L\14. Namely, that small and tiny are less in use in practice I believe.

Minor: Regarding the segmentation and perturbation tests - I see that older papers are reporting the results ranging from [0,100] ([1],[2]) and newer papers are already report ranging [0,1] ([3],[4]) - I can guess that it is due to the code shared by Chefer et al. (not sure what is better, but in general common conventions are always preferable).

[1] Hila Chefer et al. Transformer interpretability beyond attention visualization. CVPR 21 [2] Tingyi Yuan et al. Explaining information flow inside vision transformers using markov chain. eXplainable 21

[3] Alexandre Englebert et ak. Explaining through transformer input sampling. ICCV 23 [4] Weiyan Xie et al. Vit-cx: Causal explanation of vision transformers. IJCAI 23

[5] Radford et al. Learning transferable visual models from natural language supervision. PMLR 21.‏

Thank you for providing a detailed response along with extensive additional results and uploading your revised paper with the additional results. These additional results are indeed impressive and address my major concern related to optimizing and evaluating the same metric.

Regarding the run time, I agree that there are some applications that do not require realtime performance and can benefit from slower and more reliable attribution methods e.g. medical applications and model diagnosis.

With these changes, I think this would be a valuable paper for the research community.

Minor: I noticed some typos here and there is the added text. Please proof read and correct.

Thank you for taking the time to do more evaluation, and the running time comparisons.

• These additional evaluations are very valuable and better support the method.

• Although I agree with the authors and reviewer Q9o9, that some applications do not require real-time processing, I also agree with reviewer DmfJ that we are talking here about order of magnitude slower. This is a major limitations that can impact the possible applications of the method. For instance, this means that such method cannot be used in conjunction of any kind of method using attribution maps at training time.

That being said, I also think that the paper is overall valuable for the community.