From Attention to Prediction Maps: Per-Class Gradient-Free Transformer Explanations

``Please clarify if your class-wise prediction map is indeed equivalent to the logit lens. If they are equivalent, I still like this paper — it introduces a novel combination of existing interpretability techniques. If they are not equivalent, please clarify.’’

Thank you for bringing these works to our attention; we weren’t aware of them. We agree with you that while the prediction map is indeed addressed by the logit lens, our approach of combining it with attention maps remains novel. This combination, as we demonstrate, enhances explainability measures. The Logit Lens blog [1] focuses exclusively on the NLP domain, whereas our method adapts it for vision transformers. The Tuned Lens paper [2] introduces an optimization step that significantly increases computational overhead compared to prediction maps. Vials et al. [3] extend the Logit Lens to ViTs but rely on a gradient-based explanation map. Fuller et al. [4] primarily aim to improve positional encoding methods for handling high-resolution images during test time, rather than contributing to explanation techniques. The ViT-Prisma repository [5] aligns closely with our prediction maps but visualizes a segmentation map where each patch is marked with an emoji representing the most probable class. In contrast, our visualization operates on a per-class basis. Lastly, as you point out, none of these works explore the combination of prediction maps (or their counterparts, such as the logit lens) with attention maps. We agree that this combination provides an insightful perspective on the prediction process, yielding explanations that would not be possible otherwise.

``For the perturbation tests, patches are masked based on their order of importance. Is this importance estimated using each interpretability method — allowing you to compare how well the interpretability methods can predict the classification importance of patches?’’

Precisely. We’ll try to better clarify this.

``2: The paper should better acknowledge and differentiate from such prior work, especially since the technical approach shares similarities.’’

We thank the reviewer for referring us to relevant prior work that we had overlooked. While the prediction map is indeed addressed by the logit lens, our approach of combining it with attention maps remains novel. This combination, as we demonstrate, enhances explainability measures. Moreover, while the Logit Lens blog is confined to the NLP domain, our method extends its applicability to vision transformers, broadening its scope. Finally, the Tuned Lens paper incorporates an optimization step that introduces considerable computational overhead, whereas prediction maps offer a more computationally efficient alternative. Moreover, none of the aforementioned works explore the combination of attention and prediction maps, nor do they provide deep insights into the evolution of these mechanisms throughout the different layers and heads of the network. We will add a discussion on these works in the final version, thanks.

``While Fig. 3 shows some qualitative examples, the paper lacks systematic analysis of how patch token predictions evolve across layers. A quantitative study tracking prediction accuracy, confidence, and class distribution for patch tokens across layers would provide crucial insights into how the network builds its representations.’’

We thank the reviewer for their suggestion. Kindly note that in Figure 6 (in the main paper) and Figure 7 (in the supplementary) we show evaluation per layer. Additionally, we have updated the PDF and added Appendix A.6 in the supplementary material, where we provide results from the suggested quantitative study. The preliminary results indicate that around the 10th layer, confidence begins to increase while entropy decreases.

In addition, we analyze the layer containing the attention head most similar to the prediction map in Fig. 22. This analysis uncovers an unexpected pattern: instead of a monotonically increasing trend, the fifth layer emerges as the second most similar layer, exceeded only by the final layer. A deeper understanding of these intriguing findings requires further study, which we plan to explore in future work.

Notably, these additional insights are enabled by the novel combination of attention and prediction maps introduced in our work.

``5: Design choices (e.g., dot product similarity in Eq. 4) lack thorough justification (e.g., cosine similarity, which normalizes for magnitude, or KL divergence since we're comparing probability-like distributions).’’

Please note that in Table 5 (in the supplementary) we experiment with another similarity measure (normalized dot-product, a.k.a cosine-similarity), which performs slightly worse than the method we show in the main paper.

``6: Looking at Tables 1-2, the performance gap between the proposed method and TransAttr appears to shrink with smaller models (ViT-B vs ViT-L). The authors should evaluate their method on even smaller architectures (e.g., ViT-S, ViT-Ti) to verify if the advantages persist. This would help understand if the benefits are truly architectural or simply emerge from scale.’’

Thank you for the insightful observation. Please refer to the newly added Table 6 in Appendix A.7 of the updated PDF, where we present the evaluation of our method on ViT-S/16. Even with this smaller architecture, our approach consistently demonstrates some improvement over TransAttr across most metrics.

``7: The BERT and CLIP extensions feel preliminary and lack quantitative evaluation (e.g .object localization tasks where ground truth bounding boxes are available, activation & pruning tasks [3]).’’

The illustrations on BERT and CLIP are not meant to be exhaustive. We included these results only to illustrate that our method can potentially be extended to other modalities as well. This is certainly an interesting avenue for future work.

``8: [3] and [4] are highly related related-works, which are not mentioned and compared against.’’

We’ll mention them in the final version, thanks.

``Questions: ...’’

Thanks for all the corrections and suggested exposition improvements. We incorporated them into the updated version.

Thank you for your responses. While I appreciate the authors' efforts to address some of my previous comments, I am still largely unsatisfied:

• Revision Clarity: I recommend using color-coding (e.g., blue/red text) to highlight the changes made in the revision. This would make it easier to track and evaluate the modifications.
• BERT and CLIP: The authors state that their method is applicable to BERT and CLIP, but this claim lacks empirical validation. For a paper making such claims, I believe it is essential to: (1) Provide quantitative results for these models (2) Include comparative benchmarks.
• Related Work: The authors have not adequately addressed the comparison with [3] and [4], which represent the current state-of-the-art in this field. A thorough comparison with these works is crucial.
• Performance Improvements: The reported improvement for ViT-Small appears to be marginal. Given that this is one of the main experimental results, the limited performance gain raises questions about the method's practical significance and real-world applicability.

Given the inadequate response to critical concerns and the lack of substantive improvements in the revision, I will keep my score at 3.

Thanks for the response. I have to admit I'm still not convinced, but perhaps the authors can clarify some additional points:

• "Our approach (as well as the vast number of other papers on this topic) aims to provides an answer to the two first types of explainability" If there is a vast number of papers on the "what" side, could you list 3-5 of them? And would it be possible to compare to those directly e.g. in Tab 1 and 2? If not, why not? Just to clarify, I'm looking for evidence that patch based classification (or something similar) is an accepted method in the explainability community. In this case (and if you can provide a better method) I would consider resting my case. But if the main motivation for this paper is to actually establish patch-based classification as a new form of explainability, then I would need a bit more theoretical underpinning beyond "it performs better on x".

• "our approach can provide a form of explanation for why the model thinks the class of a given image is what it says it is" I'm inclined to accept the "what", probably less the "why". In terms of why, I would rather see an advantage on the gradient based methods. So, probably, what is the theoretical reasoning behind the assumption that the patchwise classification of e.g. layer 6 for a certain ImageNet class explains "why" the network came up with this class? While I accept that this can happen through skip connections, although we are already some layer and transformation away from the final classification layer, it does not give too much information about the why.

• Random remark: It might make sense to look into training-free methods for VL transformers (CLIPSurgery, MASKClip) etc.

``But overall, I don't understand how such classification gives us any information on how the system came up with this class.’’

Thank you for this important discussion. The term “explainability” can indeed mean different things. There is the explainability that interests end-users, which is to explain what in the input image triggered the model to output the prediction it did. This type of explainability can assure the user that the model did not base the prediction on e.g. irrelevant parts of the image. There is the explainability that interests engineers, which is to explain why the model made certain mistakes (e.g. whether it is because the model attended to a wrong part of the image or because it attended the right part but just didn’t recognize the object correctly there). This type of explainability can help engineers improve models by understanding whether the training set should be enriched with certain types of data, or certain types of augmentations are required. Finally, there is the explainability that interests researchers, which is to explain how the model reached the conclusion that it did. This type of explainability can improve our understanding of the inner workings of deep models and can e.g. allow us to devise better architectures.

Our approach (as well as the vast number of other papers on this topic) aims to provides an answer to the two first types of explainability, but it also takes us a step forward in the third type. Specifically, our approach can provide a form of explanation for why the model thinks the class of a given image is what it says it is. In particular, since we examine both prediction maps and attention maps, as illustrated in Fig. 2, our approach helps explain what triggered certain misclassifications - was it because of not attending to the object or because of not recognizing the object? Kindly note that the perturbation and segmentation tests are widely used and considered established measures for these types of explainability. On these measures, our approach outperforms previous methods.

Regarding the third type of explainability, note that our method allows us to probe every token in every layer of the network, visualizing what classes are encoded in that token. This, together with the visualization of the attention maps, helps understand how hypotheses are formed during the forward pass in the network.

``Layer-wise patch classification … This might be worth a deeper analysis, but the ablation is a bit shallow in this respect.’’

In the main text, we provide only one example in Fig. 3, however in Fig. 8 in the supplementary, we provide several more examples. Qualitatively speaking, we can observe that the evolution is somewhat consistent and reaches optimality for layer 11 across a wide range of classes. Quantitatively speaking, we follow the suggestion of Reviewer hdPy to analyze the confidence and distribution of the predictions throughout the different layers. As a confidence measure, we utilize a common approach which computes the difference between the logits of the most confident and the second most confident classes. As a distribution measure we report the mean entropy of the prediction. The preliminary results indicate that around the 10th layer, confidence begins to increase while entropy decreases. For further details, please refer to the newly added Appendix A.6 in the updated PDF.

Additionally, we analyze which layer contains the attention head most similar to the prediction map, as shown in the histogram in Fig. 22. This analysis uncovers an unexpected pattern: instead of a monotonically increasing trend, the fifth layer emerges as the second most similar layer, exceeded only by the final layer and surpassing all other layers by a large margin.

``Does TransAttr refer to (Chefer et al., 2021b)? Please indicate somewhere which publication this refers to.’’

Yes. We clarified this in the updated PDF (line 369), thanks.

``A notable limitation of this method is that the classification head must accept tokens of a specific dimension, which won't hold for certain architectures like DINO or Swin.’’

We agree, the limitations of our approach are acknowledged in the Conclusion and Limitations section. Nonetheless, our method remains relevant due to the widespread use of ViTs for classification tasks, with CLIP as a main example. The DINO architecture, for example, involves concatenation of the CLS tokens of several layers. Thus, a straightforward adaptation of our method for DINO could involve concatenating patch tokens in the same way, and applying the classification head on the resulting concatenated vectors. We leave it to future research.