Visual Representations inside the Language Model

We thank Reviewer 4 for your time and detailed comments. We address them below.

Thank you for your feedback about the presentation of our paper. We will improve the writing and clarify details about the tables and figures, including the metrics defined in Table 3, the feature maps, and Figure 3.

Regarding our contribution, we have added further results in General Response to Reviewers, expanding significantly on our experiments in the paper on intervention to prevent model hallucination. Our contributions thus tackle a wide range of tasks related to perception capability of MLMs: hallucination detection, few-shot foreground segmentation, co-segmentation, semantic segmentation, referring expression segmentation, semantic correspondence and temporal correspondence. We would be happy to provide further results during the discussion period, per suggestions you may have.

Thank you again, and please let us know if you have any questions, or suggestions for further experiments. We will be happy to answer and provide the same.

Dear Reviewer 6is2,

The authors have responded to your review. Does their answer address your questions and reasons to reject?

We once again thank all the reviewers for their time. We believe we have addressed all questions and concerns in our response, and that our paper will certainly be stronger upon incorporating these new results.

We wanted to follow up in advance of the upcoming deadline — we hope our response may lead you to consider updating your rating of our work.

We have summarized our response and our contributions in "General Response #2" above.

Please let us know if you have any remaining questions or concerns that we could clarify!

We thank Reviewer 2 for your time and detailed comments. We address them below.

• Reasons to Reject #1: Connection between feature variance and hallucination + Hallucination intervention only done on one model and benchmark.

  • Connection between feature variance and hallucination: We draw inspiration from Darcet et al “Vision Transformers Need Registers” (CVPR 2024), which showed that middle layers of the transformer corresponding to background patches encode input-agnostic artifacts. Combined with our insight that several of the image key tokens were input-agnostic, we wanted to determine whether these, too, encoded input-agnostic artifacts. We chose a task that requires careful attention paid to the input: hallucination detection.

    We found that input-independent keys in the initial layers, while input-agnostic, encode important information (as removing them significantly reduced model performance), while input-independent keys in later layers encode artifacts (as blocking them mitigated hallucination). We will expand on this discussion in the paper.

  • Hallucination intervention only done on one model and one benchmark: Thank you for your suggestion. We have conducted further experiments with hallucination intervention, which are presented in the General Response to Reviewers. We will provide further results during the discussion period as well.

• Reasons to Reject #2: Table 1 should be compared with existing segmentation models to show how close the models can achieve.

  • Thank you for your suggestion. Segmentation models achieve much higher results on segmentation, having been designed and trained specifically for the task. Our goal here is to evaluate MLM’s intermediate representations on perceptual tasks such as segmentation (to study further MLMs’ poor performance on perception). While not close to state-of-the-art (e.g., state-of-the-art on ImageNet-S is 63.2 MIOU, whereas LLaVA-OneVision 7B intermediate representations achieve 54.6 MIOU, per Table 4), the results are strong enough to show that MLMs should be doing better on perception tasks, given their intermediate representations’ performance (as discussed in Section 5.5). We will add these numbers and a discussion to the paper.

• Reasons to Reject #3: For the prefixing setting, it is evident that writing a context-related prompt may improve MLM performance, which is the reason why MLMs were developed. The new insight is limited.

  • In this experiment, we are not evaluating the MLM overall, but its intermediate representations: specifically, the image value tokens from the MLM layers’ key-value cache.
  • It is also worth noting that almost all existing evaluations of MLMs (e.g., LLaVA) place the image first, followed by the text. We here show that perception capabilities (in the context of domain shift) of the intermediate representations can actually improve if this order is reversed.

• Reasons to Reject #4: In section 5.3, the work tried to compare MLM with CLIP and DINO, which is not a fair comparison considering different training data and model architectures.

  • In this experiment as well, we are not evaluating the MLM overall, but its intermediate representations (image value tokens from the MLM layers’ key-value cache). It is not a direct comparison to CLIP and DINO, but rather a point of reference to highlight the capabilities of the intermediate representations. We will clarify this in the writing, thank you for highlighting this point.

• Questions to Authors #1: In section 5.4, I don't get what the point the author is trying to make. Could you elaborate more on this?
  • Figure 3 shows the correlation between performance of the MLM’s intermediate representations on perception tasks (X-axis) and the performance of the MLM itself on perception tasks (Y-axis). We see a strong correlation between the two, highlighting the importance of studying intermediate representations of MLMs.
  • Note: To isolate the number of parameters, we also evaluate two models with the same number of parameters (Qwen2-7B and Qwen2.5-7B).

Thank you again, and please let us know if you have any questions, or suggestions for further experiments. We will be happy to answer and provide the same.

Dear Reviewer SMbp,

The authors have responded to your questions. Does their answer address your questions and reasons to reject?

We once again thank all the reviewers for their time. We believe we have addressed all questions and concerns in our response, and that our paper will certainly be stronger upon incorporating these new results.

We wanted to follow up in advance of the upcoming deadline — we hope our response may lead you to consider updating your rating of our work.

We have summarized our response and our contributions in "General Response #2" above.

Please let us know if you have any remaining questions or concerns that we could clarify!

We thank Reviewer #3 for your time and detailed comments. We address them below.

Thank you for your suggestion regarding further experiments in our hallucination study. We have conducted further experiments with hallucination intervention, which are presented in the General Response to Reviewers. We will provide further results during the discussion period as well, such as your suggestion to evaluate on other benchmarks beyond POPE.

Thank you again, and please let us know if you have any questions, or suggestions for further experiments beyond these. We will be happy to answer and provide the same.

Dear Reviewer GABW,

The authors have responded to your questions. Does their answer address your questions and reasons to reject?

We once again thank all the reviewers for their time. We believe we have addressed all questions and concerns in our response, and that our paper will certainly be stronger upon incorporating these new results.

We wanted to follow up in advance of the upcoming deadline — we hope our response may lead you to consider updating your rating of our work.

We have summarized our response and our contributions in "General Response #2" above.

Please let us know if you have any remaining questions or concerns that we could clarify!

The author's responses indeed addressed some of my concerns, it would be beneficial to see more evaluations on general-purpose benchmarks to better assess the practical applicability of the proposed approach; however, this was not provided. Having considered the other reviewers' comments and the author's response, from my point of view, this paper indeed provides a good insight for visual representations for VLM research. Therefore, my initial assessment remains unchanged.

We thank Reviewer #1 for your time and detailed comments. We address them below.

• Reasons to Reject #1: MLMs are trained using CLIP and DINO as their visual encoders (via a connector) and that MLMMs have way more parameters, so you’d expect them to be even better at vision tasks, if not just as good. In some sense, the results aren’t that surprising as a result.
  • We agree that MLMs would be expected to build upon the information provided by their component visual encoder, and should thus perform better than their visual encoder, and MLM intermediate representations (image value tokens). However, we actually find that the opposite is true: the visual encoder (SigLIP) outperforms the intermediate values of the MLM on three of four segmentation tasks (details under response to Reasons to Reject #3). Further, the intermediate values actually outperform the overall MLM on the semantic correspondence task (discussed in Section 5.5), showing that the language model does not retain the perceptual information from the visual encoding throughout its layers. We expect that this occurs because the MLM is primarily trained on vision-language tasks such as captioning and VQA, rather than perception-heavy tasks such as those requiring segmentation. These results underscore the need to study visual encodings at different parts of the model, to redirect research towards improving them where needed.
• Reasons to Reject #2: I also didn’t find the PCA outputs (e.g., Figure 8) very readable.
  • We are happy to modify our figures to be more readable, especially Figure 8, which has double the number of heads compared to the other models (and was hence placed in the Appendix). The main takeaway is that some keys are input-independent, whereas some keys are input-dependent, which is leveraged in our subsequent intervention experiment to mitigate model hallucination.
• Reasons to Reject #3: I was particularly surprised that the authors didn't use Siglip as well, as it's what was used in LLaVA Onevision, which they compared against in the paper.
  • Thank you for your suggestion. We have obtained results with SigLIP.
  • On segmentation, we find that SigLIP actually beats intermediate representations of LLaVA-OV 7B on three of four segmentation tasks, as shown below: | | Ref. Exp. Seg. | Semantic Seg. | Co-Seg. | Foreground Seg. | |---|:---:|:---:|:---:|:---:| | SigLIP | 61.7 | 59.1 | 62.0 | 27.1 | | LLaVA-OV Intermediate Values | 64.8 | 54.6 | 61.6 | 26.3 |
  • As discussed under our response to point #1, we believe that this occurs due to the vision-language training of the MLM causing forgetting of perception capabilities through the layers. The only segmentation task in which LLaVA intermediate representations beat SigLIP is Referring Expression Segmentation, which requires more vision-language capabilities.
  • On semantic correspondence (SPair71K), we find that SigLIP performs worse than intermediate representations of LLaVA-OV 7B, as shown below: | | Semantic Correspondence | |---|:---:| | SigLIP | 41.6 | | LLaVA-OV Intermediate Values | 46.1 |
  • This could in part be due to the more semantic nature of the task (similar to referring expression segmentation), and in part due to the fact that LLaVA was trained on video data in addition to images, and is more capable of comparing multiple images.
  • We will certainly add these numbers to the paper, alongside a discussion on how further research is required to ensure the language model enhances vision encodings rather than degrades them, across various tasks.

We now address the Questions to Authors:

• Questions to Authors #1: Can you please clarify the way that you think about the interaction between visual encoders (SigLip, CLIP, DINO) and MLMMs? And how that factors into the results? I would expect at least a discussion in the paper on this. MLMMs should be strictly better.

  • Our work shows that although visual encoders capture perception-related information, that information is not necessarily retained by the language model layers, resulting in the MLM outputting incorrect answers on perception-related tasks.
  • In Section 5.5, we show that in 7.5% of semantic correspondence inputs, the MLM output was incorrect despite the intermediate MLM value representation being correct. We ran further experiments and find that in 6% of semantic correspondence inputs, the MLM output was incorrect despite the SigLIP representation being correct.
  • This makes a strong point calling for further research to ensure the layers of the language model retain perception-related information from the visual encoder. We will add these numbers to the paper, along with a detailed discussion. Thank you for this suggestion.

• Questions to Authors #2: What does the table look like if you add SigLip to Table 3?

  • Thank you for your suggestion. SigLIP obtains 41.6 PCK on SPair71K, compared to the intermediate representations of LLaVA-OV 7B, which obtain 46.1 PCK (reported and discussed under Reasons to Reject #1).

• Questions to Authors #3: Input-dependence and input-independence in this setting wasn't really defined until much later in the paper than needed (Section 4)

  • Thank you, we will update the writing to clarify.

• Questions to Authors #4: I don't understand the caption (or value) of Table 1. What is the average segmentation performance of a DINO or CLIP when trained for that task? What is state of the art?

  • Table 1 shows the performance of the image value tokens (i.e., intermediate representations within the language model from the key-value cache) averaged across 4 segmentation tasks. Their performance on each task is reported in Appendix Table 4. They may not reach state-of-the-art, but these results suggest that MLMs should have good perception capabilities — however, we show in Section 5.5 that MLMs do not retain the perception capabilities of their intermediate representations. Our SigLIP results, as discussed above, further show that the language model layers in fact degrade perceptual capabilities of the input visual encoding. We will add this discussion to the paper. Thank you for your suggestion.

• Questions to Authors #5: Some examples of the before and after in Table 2 would be nice. What is state of the art for Table 2?

  • Thank you, we will add these examples and results to the paper, and respond further during the discussion period.

• Questions to Authors #6: In Section 5.5 it's not clear to me whether it's the value tokens or the initial encoder that matters here. What is the performance of the visual encoder alone?

  • Thank you for your suggestion. We evaluate the visual encoder SigLIP on the experiment in Section 5.5 and find: | | Semantic Correspondence| |---|:---:| |SigLIP| 37.5% | | Intermediate Values | 41% | | Overall MLM | 36% |

  • In this task, it is clear that language model layers can initially build upon the perceptual capabilities of the input visual encoding, but then lose it by the time of the MLM output. (in contrast to segmentation, where the language model just degrades the perception). This finding shows the need to focus on researching retention of perception by language model layers. We will add these results to our paper.

• Questions to Authors #7: Isn't Figure 3 just saying that if you have more parameters, you do better at vision? What am I missing here?

  • Figure 3 shows the correlation between performance of the MLM’s intermediate representations on perception tasks (X-axis) and the performance of the MLM itself on perception tasks (Y-axis). We see a strong correlation between the two, highlighting the importance of studying intermediate representations of MLMs.
  • To isolate the number of parameters, we also evaluate two models with the same number of parameters (Qwen2-7B and Qwen2.5-7B), one of which performs better than the other.

• Questions to Authors #8: Why was the "intervention" done on the particular layers as described? It seems like there are many ways this could have been done.

  • Thank you for your suggestion. We have conducted further experiments with hallucination intervention, which are presented and discussed in detail in the General Response to Reviewers.

Thank you again, and please let us know if you have any questions, or suggestions for further experiments. We will be happy to answer and provide the same.

Dear Reviewer RSnv,

The authors have responded to your questions. Does their answer address your questions and reasons to reject?

We once again thank all the reviewers for their time. We believe we have addressed all questions and concerns in our response, and that our paper will certainly be stronger upon incorporating these new results.

We wanted to follow up in advance of the upcoming deadline — we hope our response may lead you to consider updating your rating of our work.

We have summarized our response and our contributions in "General Response #2" above.

Please let us know if you have any remaining questions or concerns that we could clarify!

After reading the authors response, I believe that there is still substantial work to be done to disentangle the performance of the visual encoder from the larger MLMM. As a result, I hold my rating and think this paper could benefit from another round of review.