Towards Interpreting Visual Information Processing in Vision-Language Models

We thank the reviewer for their comments on the paper, as well as their suggestions on how to improve the paper.

The contributions on the interpretability of visual information processing in VLMs are well recognized, while the contributions on how we can adopt such information processing pattern on better prompting or training for VLMs are limited.

We thank the reviewer for recognising the contributions we make towards the interpretability of VLMs. Regarding how these findings could be used for better prompting or training of VLMs, we did suggest that this could be used to improve the hallucination rate for VLMs by modifying the Paying More Attention to Image methodology by Liu et al. (2024) [1] to pay more attention towards object tokens, now that we know that the object information is quite local to the visual tokens that originally corresponded to the object.

In general, our focus was more towards rigorously showing that the object information is local and is refined towards interpretable tokens. Wendler et al (2023) [2] suggests that LMs think predominantly in their main language regardless of the input language; we think our results suggest a more general phenomenon where LMs think predominantly in their main language regardless of the input modality. We hence think that our findings have larger implications overall on understanding multimodality, and focused on understanding the model’s processing.

[1] https://arxiv.org/abs/2407.21771 [2] https://arxiv.org/abs/2402.10588

The investigations focus on object only, which are only important to the general VQA, captioning, and some specific object detection tasks.

The key investigation of our study is on the visual token set — on how they are structured, how they are processed and how their information is extracted. Hence, we focus exclusively on object identification because it gives us the clearest setting in which the required information to complete the task is almost exclusively in the visual tokens.

We note that Basu et al. (2024) [3] study some setting closer to the textVQA setup, which makes it a much more complicated process because it requires extracting some intrinsic knowledge about the LM. (e.g. they show the model a picture of Christopher Nolan, and expect the model to answer “what movies has this director made?”.) This confounds the information flow because now the information comes from both the processing of the visual information (Who is this person?) and intrinsic knowledge from inside the LM (what movies has Christopher Nolan made?). Hence we think studying the object identification setting is the clearest for understanding visual token processing, and we focus on this.

The definition of last row of visual tokens in line 435 is very confused to me... It is recommend to provide a lot of explanations on that and why such experiments on last row visual tokens are required?

Yes, if CLIP ViT-L/14 has 24*24 tokens, the last row visual tokens refer to the last 24 tokens. This is because Basu et al. (2024) [3] suggest that visual information could be summarised in the last row of visual tokens, because they found that only the last row token is involved in transferring information to the last token. However, they do so merely by looking at the attention pattern in the early layers (Section 4.2 of their paper). By doing attention blocking from the entire image to the last visual token row across all layers, we demonstrate that even in the most extreme case there is no performance degradation, suggesting that looking at the attention patterns may not be enough, or that their findings may not generalise past the factual VQA task they study.

We noted that this was not stated outside of the findings section, and should have been mentioned in the Method section to clearly give a motivation on why we do so. We thank the reviewer for pointing this out. We will add the following paragraph to the methods section in the Camera Ready version of the paper at line 435:

“...(iii) From all visual tokens except the last row to the last row of visual tokens.

We do this as recent work by Basu et al. (2024) suggests that the model may summarise visual information in the last row of visual tokens, though they study this in a different VQA task. If the model does use the last row of visual tokens as a ‘working summary’, then blocking the attention from all the visual tokens to the last row should prevent it from answering the question.”

[3] https://arxiv.org/abs/2406.04236

Once again, thank you for your feedback! Since the discussion period closes soon (in 1-2 days), please let us know if you have any questions about our response! We feel that we have addressed your concerns in our response, and would like to know if you have any further concerns. If you have no other questions and felt we have addressed your concerns, we would kindly ask you to consider increasing your score!

We thank the reviewer for their review and appreciate their thorough feedback on the paper.

The attention flow experiments appear somewhat inconclusive... I would have therefore expected an additional condition in Table 2 showing the performance of knocking out some variant of text token attention.

We note that the competing explanation was for another experimental setting, and we were checking if that explanation would apply to our case as well.

In the factual association framework, it covers cases where the model is asked a question like: “Beats Music is owned by ___” Where implicit information needs to be extracted from the transformer. In this case, it is the information that Apple is the owner of Beats Music. In this case, the model refines the information at the last subject token position “sic”.

In our case, the tokens that need to be refined and information extracted from are from the visual token set. This is even more so as we explicitly try to avoid objects that are directly related to the question, e.g. If we ask the model “What is on the bed?” We only ask that question if the object is not directly related to the bed (e.g. book, as opposed to pillow). Hence, our hypothesis is more of a “does the refine-then-extract mechanism in the factual association framework also apply for the object identification setting?“ for the attention blocking experiment we do not consider the text.

We note that the setup more similar to Geva’s setup we were referring to was by Basu et al. (2024) [1], where they gave the model a photo of Christopher Nolan and asked the model “What movie by this director won a Golden Globe?” In this case, they were able to find that the model used the MLP at the position of “this director”’.

In contrast, we maintain that our study focuses on the visual token set, and how the model processes them. This is why the attention blocking experiment is in a setup with less confounding by the text, so that it makes sense to only block attention from the object tokens.

Nevertheless, since the attention blocking worked less well than expected, it could be that the model does do some sort of aggregation at the token. But since our setup has questions of different formats (e.g. what is on the bed? What is to the left of the bus? What is the child holding?), this would necessitate a different experimental setup where the text prompts are more fixed. Hence, we leave further exploration of text token blocking to future work.

[1] https://arxiv.org/abs/2406.04236

Questions

L257 - 259: ... I'm curious if the authors tried ablating tokens by zeroing out embeddings rather than using the global average token.

After doing some exploratory analysis, we found that the image tokens form soft prompts that are of high magnitude (~10-80, as compared to ~0-1 for text tokens), which is generally the case for soft prompts. For this reason we felt that to do zero-embedding would push the input further out of distribution, and it would be best to do mean-ablation. We agree that mentioning this will add clarity to the design choice, and will include this in the camera-ready version of the paper.

L262-L266: How is it ascertained whether the target object o is mentioned in the generated description? Is this done manually or automatically? ... L242-L246: How is correct identification determined? Is this automated or manual?

In both cases, o is detected automatically based on whether class is in description, through a simple class_name in description detection. We found that this worked well enough, especially because COCO class names are sufficiently general.

L329-:330: I'm wondering what the precise definition of "comparable number of tokens" is here?

Yes, the average number of object tokens is used. To be specific, we were comparing the average number of object tokens ablated for +1 buffer (which is 33.4), and comparing them to the integrated gradients and random baselines where 40 tokens were ablated.

L350-L352: How are correspondences identified, is this done by hand?... I'm trying to get a sense of how reliably this behavior is observed in the model.

This was done by hand in the main paper, but we have done additional experiments to make sure that this behavior can indeed be reliably detected. Please refer to the general response for more details.

L369: "such numbers seemingly counting applies in background tokens." -- the wording is a little awkward, consider revising.

Thank you for the suggestion, we will revise this to ‘such as number tokens that could be referring to the number of apples in the images’.

Once again, thank you for your feedback! Since the discussion period closes soon (in 1-2 days), please let us know if you have any questions about our response!

We thank the reviewer for their feedback on the paper. We are pleased that the reviewer finds that the paper is well written and clearly addresses an important problem in the field of multimodal interpretability.

Evaluated insufficient VLMs: As the authors acknowledged, the analysis is performed on only two VLMs from the same (LLaVA) family. However, the methodologies should be easily extendable to other transformer-based VLMs.

We thank the reviewer for suggesting that we should try our analysis on models of other families. We have tried performing an adapted version of the logit lens on Qwen2VL, and have found that the visual tokens become refined towards interpretable tokens in the residual stream towards the late layer as well. Please refer to the overall response for more details.

Results of "logit lens" analysis are hard to interpret...

We thank the reviewer for pointing out that the logit lens findings could be an artifact of confirmation bias, and for suggesting concrete examples of further analysis.

We have done additional experiments to make our logit lens analysis more concrete. Please refer to the overall response for more details.

We also encourage the reviewer to look at the interactive demo provided in the paper (and here for easy access).

Question: In Figure 3, what layer do these unembeddings correspond to?..."

These unembeddings are from different layers that are in the middle-to-late layer range. We agree that this may have been ambiguous and will update the caption to be more specific, to (added sentence in bold): “ Examples of tokens and their positions that the logit lens yields in the late layers. The labelled tokens come from a range of layers around the middle-to-late layers. In our labelling for an image, tokens that are part of a word are completed with an educated guess e.g. “diam”(ond). “

Once again, thank you for your feedback! Since the discussion period closes soon (in 1-2 days), please let us know if you have any questions about our response!

We thank the reviewer for their feedback on the paper. We are pleased that the reviewer finds that our paper is well written with comprehensive experiments.

The experiments only focus on the LLaVA model. The authors can consider other prevailing VLMs.

We thank the reviewer for suggesting that we should try our analysis on other models.

We have tried performing an adapted version of the logit lens on InstructBLIP and Qwen2VL, and have found that the visual tokens become refined towards interpretable tokens in the residual stream as well. Please refer to the general response for more details.

The authors can provide more direct evidence (experiments) to support the potential applications of the proposed methods... Is it easy to apply the methods and findings in this paper to downstream tasks?

The methods and findings from this paper have limited direct applicability to downstream tasks in their current form. We do suggest one potential application - using our insights about object token localization to modify Liu et al.'s (2024) 'Paying More Attention to Image' [1] methodology to potentially reduce VLM hallucinations.

Our research was primarily focused on understanding the fundamental mechanisms of how VLMs process information, specifically showing that object information is localized to specific visual tokens and gets refined into interpretable representations. We think that our findings concretely contribute to the broader theoretical understanding of multimodal processing, and we leave further research into practical applications to future work.

[1] https://arxiv.org/abs/2407.21771

Once again, thank you for your feedback! Since the discussion period closes soon (in 1-2 days), please let us know if you have any questions about our response!