INViTE: INterpret and Control Vision-Language Models with Text Explanations

Quantitative experiment: Comparison with other interpretation methods

Fixing Typographical Attacks

We consider the following methods of producing saliency maps.

• Raw attention values from penultimate layer to final layer CLS token
• GradCAM [1] with the same implementation for ViT in [3]: raw attention values from penultimate layer to final layer CLS tokens * their gradient respect to similarity with prompt “this can be described as a text”
• Rollout attention from CLS token to input layer [2]
• Rollout attention from CLS token to input layer where each attention matrix is multiplied by its gradient respect to similarity with prompt “this can be described as a text” [3]

With all these saliency maps, we mask the parts of the image with map > threshold with 0. We test different thresholds and report best performance with each map.

We perform the editing on the ImageNet typographical attack experiment (Table 2). Here’s the result:

Our method outperforms the best saliency map based method (raw attention) by 7.8% on attack image accuracy.

Reducing spurious correlations

We conducted the removing spurious correlation experiment (Table 4) too with saliency maps. For raw attention and rollout attention, we mask out parts of the image with map > threshold with 0. For gradient based map, we take gradient respect to similarity to “this can be described as hair” and mask out parts of the image with map < threshold with 0. We report performance under the best threshold for each saliency map.

Our method outperforms best saliency map based method (raw attention) by 24.56% on worst class accuracy.

[1] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2016). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. arXiv preprint arXiv:1610.02391.

[2] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190–4197, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.385. URL https://aclanthology.org/2020.acl-main.385.

[3] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 782–791, June 2021.

Analysis of distribution shift

Thank you for asking for clarification on the distribution shift. We provide numerical and visualization analysis of distribution shift in appendix B. We applied random smoothing to address the distribution shift (Levine et al., 2019; Salman et al., 2019; Cohen et al., 2019) and found random smoothing improves experimental results. The experiment performance with and without random smoothing therefore provides a proxy evaluation of distribution shift’s effects on interpretation quality. Most of our experiments (causal intervention in Fig. 4a; fixing typographical attack in table 1,2) show that distribution shift’s effects on interpretation are small. In a few other cases (entity editing in table 3; debias CelebA in table 4), addressing distribution shift with random smoothing improved the result more significantly.

Comparison to other interpretation methods

Thank you for asking for more comparison with other interpretation methods. To our knowledge, our method is among the first to provide token level interpretations of concepts. Previously methods mostly either interpret model weights or attribute prediction to image areas. Token level interpretation provides unique advantages of understanding better model reasoning processes and more flexible model editing. Therefore, it is difficult to directly compare our method with previous interpretation methods.

In our paper, we use rollout attention saliency map to find the image region that is most influential on the token being interpreted. This provides a qualitative visual verification that our interpretation is correct, and allows us to quantitatively verify correctness through saliency overlap experiment.

However, saliency map only provides correspondence between image area and token of interest. It cannot provide a conceptual explanation of a token. Although one could mask out images to keep only saliency map regions and find a text description of that region, this method would provide a same explanation for tokens that have similar saliency maps. However, our method is able to interpret tokens corresponding to the same image area differently. For example, in the fifth row of Figure 3, L12T50, L12T42, L13T1 all correspond to similar image areas, but interpretation is different. This allows for interpretations of different levels of concepts (e.g. “L12T35: green and blue coloured spots” contains low level features; “L13T1: a person walking down the street” contains high level concepts).

We also conducted additional experiments that show our interpretation enabling better model editing in addressing typographical attacks and spurious correlations than other interpretation methods (see next response).

Effects of vocabulary

One advantage of our framework is that it could produce interpretation with any vocabulary of any type and size. In general, a larger and diverse vocabulary set provides more granular and comprehensive interpretations. We conducted an ablation study on how vocabulary size influences model editing results based on interpretations by measuring performance on reducing CelebA spurious correlation (Appendix F, Fig. 12). In general, a larger vocabulary (we tested up to 5000) performs better, but a balance between vocabulary and object word list size is important.

Choosing specialized vocabulary might help improve interpretations on domain-specific images. For example, we used 2088 unique nouns and adjectives extracted from remote sensing image captions of RSICD dataset (Lu et al.) to interpret satellite images.

Process of vocabulary creation

To interpret VAW, CelebA, ImageNet images, we used vocabulary extracted from MILAN annotations (Hernandez et al., 2022) and GPT description of ImageNet (Mao et al., 2022b) which contain descriptions of most daily life object attributes. To interpret satellite images, we used 2088 unique nouns and adjectives extracted from remote sensing image captions of RSICD dataset (Lu et al.).

Applications beyond demonstrated tasks

Beyond the demonstrated tasks, our method could be generalized to pinpoint the internal mechanism for any kind of model errors. One can then build models that emphasize self-consistency of internal concepts and avoid overemphasis on certain biased features.

We also conducted additional demonstrations to show that our interpretation allows for understanding how adversarial attacks impact the model in Appendix A. We show that adversarial attack impacts model the most starting from layer 10 and provide some example latent token interpretations before and after attack.

Openness of vocabulary

Our approach works with vocabulary of any type and size, which makes our approach work on open vocabulary. In contrast, Koh et al (2020)’s method interprets a small set of predefined concepts, where the concept set is built into model architecture that is task-specific.

For our method, we showed that large vocabulary (5000 descriptions) is beneficial for producing better descriptions in our framework in section 4.3. Our method also works on any type of vocabulary, which makes it easily adapt to different tasks. To enable better interpretations in domain specific tasks, one can simply choose a vocabulary set that contains domain-specific languages (for example, we used RSICD captions to interpret satellite images in UC Merced Land Use Dataset).

Improving model training with our interpretations

Thank you for your suggestion on extending our method to improve model training. Our approach can indeed be generalized to enhance model training. A few possible ways include: 1) regularize the model to emphasize self-consistency of internal concept representations for more concrete visual reasoning and enhanced robustness against adversarial attacks (we demonstrate how our method help understand adversarial attacks in Appendix A); 2) increase model robustness by steering the model to avoid overemphasis on certain classes of features.

Quantitative experiment: Comparison with previous interpretation method

Fixing Typographical Attacks

We consider the following methods of producing saliency maps.

• Raw attention values from penultimate layer to final layer CLS token
• GradCAM [1] with the same implementation for ViT in [3]: raw attention values from penultimate layer to final layer CLS tokens * their gradient respect to similarity with prompt “this can be described as a text”
• Rollout attention from CLS token to input layer [2]
• Rollout attention from CLS token to input layer where each attention matrix is multiplied by its gradient respect to similarity with prompt “this can be described as a text” [3]

With all these saliency maps, we mask the parts of the image with map > threshold with 0. We test different thresholds and report best performance with each map.

We perform the editing on the ImageNet typographical attack experiment (Table 2). Here’s the result:

Our method outperforms the best saliency map based method (raw attention) by 7.8% on attack image accuracy.

Reducing spurious correlations

We conducted the removing spurious correlation experiment (Table 4) too with saliency maps. For raw attention and rollout attention, we mask out parts of the image with map > threshold with 0. For gradient based map, we take gradient respect to similarity to “this can be described as hair” and mask out parts of the image with map < threshold with 0. We report performance under the best threshold for each saliency map.

Our method outperforms best saliency map based method (raw attention) by 24.56% on worst class accuracy.

[1] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2016). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. arXiv preprint arXiv:1610.02391.

[2] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190–4197, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.385. URL https://aclanthology.org/2020.acl-main.385.

[3] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 782–791, June 2021.

We thank the reviewer for their valuable comments. We are glad that the reviewer found our work to be exciting. We addressed the questions below:

Interpreting Non-Text Trained Models

Thank you for pointing this out. Our framework is general to all transformer architecture. For non-text trained models like DINOv2 and MAE, one can train a linear mapping from final CLS embedding (if available) or average pooling of all final layer embeddings to CLIP image-text embedding space. Then, one can interpret latent tokens in these models with our method.

Comparison to other interpretation methods

Thank you for pointing us to STAIR, which is an interesting extension of CLIP that provides better concept representation through sparse token space. We have revised our paper and included it in related work.

Thank you for asking for more baselines. To our knowledge, our method is among the first to provide token level interpretations of concepts. Previously methods mostly either interpret model weights or attribute prediction to image areas. Token level interpretation provides unique advantages of understanding better model reasoning processes and more flexible model editing. Therefore, it is difficult to directly compare our method with previous interpretation methods.

However, we conducted additional experiments to compare the effectiveness of addressing typographical attacks and spurious correlation between our interpretation and saliency mask based interpretation methods (see next response).

Motivation for ''Intervening in Reasoning Procedure'' Experiment

While the saliency map overlap and causal intervention experiments in section 4.1 demonstrates that our interpretation is faithful, we hope to show that our interpretation also explains the causal influence between a token and final image representation. By replacing car tokens with airplane tokens and successfully changing final image representation from encoding highway to airport, we show that our interpretation not only explains what each token represents but also the causal relationship between a token’s and final image’s representation in ViT’s reasoning process.

Quantitative experiment: Comparison with gradient-based method

Thank you for asking for more baseline comparison. We conducted the following additional experiments to compare our methods against saliency based methods.

Fixing Typographical Attacks

We consider the following methods of producing saliency maps.

• Raw attention values from penultimate layer to final layer CLS token
• GradCAM [1] with the same implementation for ViT in [3]: raw attention values from penultimate layer to final layer CLS tokens * their gradient respect to similarity with prompt “this can be described as a text”
• Rollout attention from CLS token to input layer [2]
• Rollout attention from CLS token to input layer where each attention matrix is multiplied by its gradient respect to similarity with prompt “this can be described as a text” [3]

With all these saliency maps, we mask the parts of the image with map > threshold with 0. We test different thresholds and report best performance with each map.

We perform the editing on the ImageNet typographical attack experiment (Table 2). Here’s the result:

Our method outperforms the best saliency map based method (raw attention) by 7.8% on attack image accuracy.

Reducing spurious correlations

We conducted the removing spurious correlation experiment (Table 4) too with saliency maps. For raw attention and rollout attention, we mask out parts of the image with map > threshold with 0. For gradient based map, we take gradient respect to similarity to “this can be described as hair” and mask out parts of the image with map < threshold with 0. We report performance under the best threshold for each saliency map.

Our method outperforms best saliency map based method (raw attention) by 24.56% on worst class accuracy.

[1] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2016). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. arXiv preprint arXiv:1610.02391.

[2] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190–4197, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.385. URL https://aclanthology.org/2020.acl-main.385.

[3] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 782–791, June 2021.

Advantage over interpreting token via gradient

Thank you for suggesting a comparison between our method and gradient-based method. If we understand the method you proposed correctly, given a token and a final layer CLS embedding, your proposed method calculates the gradient between the token and the cosine similarity between final layer CLS embedding and each text description. Then the text description for the token would be the description with the largest gradient. Please correct us if our understanding is inaccurate.

Intuitively, your proposed method assigns X as token description if with a small change in token, the final layer CLS embedding ’s similarity to X will change a lot. One advantage of our method over this method is that our method finds a direct description for the latent token. In contrast, your proposed method might fail to find a direct description of the token. For example, a token that encodes “tire” might influences CLS’s similarity to “car” a lot, but your proposed method can only find “car” but not a direct description “tire” for the token.

We are glad that the reviewer found our method to be novel. We ran the suggested experiments and addressed the concerns below.

Advantage of text descriptions of latent tokens over explaining with salient regions

When interpreting latent tokens with salient regions, if different latent tokens correspond to the same salient region, the resulting explanation would be the same. However, our method is able to interpret tokens corresponding to the same image area differently. For example, in the fifth row of Figure 3, L12T50, L12T42, L13T1 all correspond to similar image areas, but interpretation is different. This allows for interpretations of different levels of concepts (e.g. “L12T35: green and blue coloured spots” contains low level features; “L13T1: a person walking down the street” contains high level concepts).

We also conducted additional demonstrations to show that our interpretation allows for understanding how adversarial attacks impact the model in Appendix A. We show that adversarial attack impacts model the most starting from layer 10 and provide some example latent token interpretations before and after attack. Salient regions are unable to provide such understanding because there's no visible difference of image before and after adding adversarial attack.

We also conducted additional quantitative experiments to show that text descriptions of latent tokens with our method produce better model editing than saliency map based editing (see next response).

Benefits over saliency map

In our paper, we use rollout attention saliency map to find the image region that is most influential on the token being interpreted. This provides a qualitative visual verification that our interpretation is correct, and allows us to quantitatively verify correctness through saliency overlap experiment.

However, saliency map only provides correspondence between image area and token of interest. It cannot provide a conceptual explanation of a token. Although one could mask out images to keep only saliency map regions and find a text description of that region, this method would provide a same explanation for tokens that have similar saliency maps. However, our method is able to interpret tokens corresponding to the same image area differently. For example, in the fifth row of Figure 3, L12T50, L12T42, L13T1 all correspond to similar image areas, but interpretation is different. This allows for interpretations of different levels of concepts (e.g. “L12T35: green and blue coloured spots” contains low level features; “L13T1: a person walking down the street” contains high level concepts).

We also conducted additional demonstrations to show that our interpretation allows for understanding how adversarial attacks impact the model in Appendix A. We show that adversarial attack impacts model the most starting from layer 10 and provide some example latent token interpretations before and after attack. Salient regions are unable to provide such understanding because there's no visible difference of image before and after adding adversarial attack.

Thank you for your thoughtful review. We are glad that the reviewer found our experiments well-designed and convincing. We address the reviewer’s concerns below:

Motivation behind our methodology

Thank you for your suggestion on theoretical motivation.

Let $x_{cls}$ be the CLS embedding after final projection layer. Let $x_i$ be the $i$-th non-CLS embedding after the final projection layer. Let $v_j$ be the embedding of $j$-th text description.

We know that we can retrieve $\hat{j}$-th text as a meaningful text description for $x_{cls}$ by taking $\hat{j}= \arg\max_{j} x_{cls}^T v_j$.

Now consider the average of all non-CLS embedding: $x_{agg} = \frac{1}{N}\sum_{i=1}^N x_i$. This average aggregates information from all image patches. We will empirically show that we could retrieve accurate text descriptions from averaged non-CLS tokens by showing $\arg\max_{j} x_{cls}^T v_j = \arg\max_{j} x_{agg}^T v_j$.

Since the averaged token is a linear combination of each non-CLS token, $x_{agg}^T v_j = \frac{1}{N}\sum_{i=1}^N x_i^T v_j$, being able to retrieve correct text descriptions for $x_{agg}$ implies that we could retrieve correct text descriptions for each individual non-CLS token $x_i$.

We conducted a two-class zero-shot classification using CLS token and average of non-CLS tokens. CLS token embedding attains 99.5% accuracy; averaging non-CLS token embeddings on the final layer attains 97.5% accuracy. This experiment shows that indeed $\arg\max_{j} x_{cls}^T v_j = \arg\max_{j} x_{agg}^T v_j$, and non-CLS tokens also encode correct information for text retrieval. (The experiment is conducted on 100 images from each class from UC Merced Land Use Dataset with beach class and forest class).

Benefits of natural language descriptions

The text explanations produced from our method reveal information encoded in each latent token. This could help users easily pinpoint where do models make errors or produce bias, and users could address the issues with model editing guided by the interpretations (as we showed in fixing typographical attack and reducing spurious correlation experiments.)

Using our interpretations to understand visual reasoning

Thank you for mentioning our usage of text descriptions to understand visual reasoning. Using text descriptions of latent tokens to understand the model reasoning process in for example “motor vehicle + valley = overpass” is not a discretionary interpretation but grounded in both text descriptions and model mechanics. In figure 3, referenced by section 4.3, we show interpretation of tokens on layer K+1 and tokens on layer K that the K+1 layer tokens pays most attention to (based on attention weights). For example, “overpass” pays most attention to “motor vehicle” and “valley,” so we reason that this combination of local features into global features allows the model to understand the entire image as “overpass.”

We agree that attention value alone doesn’t fully demonstrate causal relationship in reasoning. Therefore, we also conduct the Object Entity Intervention experiment in section 4.2. By showing that replacing car tokens with airplane tokens changes a model's understanding of the image from highway to airport, we demonstrate that our interpretation of local concepts are indeed causally related to how model reasons about global concepts.

Quantitative experiments comparing to saliency map based editing

We also conducted additional experiments that show our interpretation enabling better model editing in addressing typographical attacks and spurious correlations than saliency-map based methods:

Fixing Typographical Attacks

We consider the following methods of producing saliency maps.

• Raw attention values from penultimate layer to final layer CLS token
• GradCAM [1] with the same implementation for ViT in [3]: raw attention values from penultimate layer to final layer CLS tokens * their gradient respect to similarity with prompt “this can be described as a text”
• Rollout attention from CLS token to input layer [2]
• Rollout attention from CLS token to input layer where each attention matrix is multiplied by its gradient respect to similarity with prompt “this can be described as a text” [3]

With all these saliency maps, we mask the parts of the image with map > threshold with 0. We test different thresholds and report best performance with each map.

We perform the editing on the ImageNet typographical attack experiment (Table 2). Here’s the result:

Our method outperforms the best saliency map based method (raw attention) by 7.8% on attack image accuracy.

Reducing spurious correlations

We conducted the removing spurious correlation experiment (Table 4) too with saliency maps. For raw attention and rollout attention, we mask out parts of the image with map > threshold with 0. For gradient based map, we take gradient respect to similarity to “this can be described as hair” and mask out parts of the image with map < threshold with 0. We report performance under the best threshold for each saliency map.

Our method outperforms best saliency map based method (raw attention) by 24.56% on worst class accuracy.

[1] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2016). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. arXiv preprint arXiv:1610.02391.

[2] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190–4197, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.385. URL https://aclanthology.org/2020.acl-main.385.

[3] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 782–791, June 2021.