Hidden in plain sight: VLMs overlook their visual representations

We thank the reviewer for their thoughtful response to our work. Below, we reference and respond to each concern:

W1: The reviewer raises a valid point that this task is the only one whose visual evaluation involves training on data that the VLM evaluation does not use. We maintain that the Depth Estimation task contributes meaningfully to the thesis of the paper. The DPT head is commonly used to read out spatial structure already encoded in a vision model, not as an external source of “extra knowledge.” To the best of our knowledge, learning a lightweight depth decoder is the standard method to evaluate the representations from a generic vision model incapable of producing depth maps on its own, and we make sure to train the probe on a different dataset (NYUv2) from the model evaluation (Omni3D).

This observation also gets at another important point: language is a flexible way to allow users to specify tasks without bespoke procedures such as training a depth head or computing texture features. Finding solutions that close the visual-VLM gap within this closed set of tasks can lead to improved performance on tasks that are difficult to specify zero-shot without language.

W2: We agree with the reviewer’s point that, if the visual encoder does not contain sufficient information to perform a task, the rest of the VLM should not be expected to “rescue” performance. We point out a concrete example of this when evaluating InternVL (L161-164). On the other hand, it’s not completely impossible for the VLM to outperform our visual evaluations, which are designed to be as simple as possible. For example, the LLM backbone could learn a more complex computation along the vision representations to outperform a simple pointwise cosine similarity function for correspondence tasks. In theory, the VLM could also identify paintings by name or artist, and reason through the Art Style task in a more robust way than texture comparison.

W3: Our data for Figure 5 was computed over the whole benchmark, so there was one trial. We will add shaded intervals for standard deviation in the final revision of the paper by subsampling 80% of the benchmark data for 10 trials. While we cannot provide an updated figure in OpenReview comments, we provide a table with some stats when computing standard deviation in this bootstrapped way. We include the min, max, and mean STD values computed across layers, as well as the last-layer STD to contextualize its variability against the variability in other layers. Hopefully these stats provide some more background for the results in Figure 5.

Semantic Correspondence

Low-level Matching

Object Affordance

Art Style

Dear Reviewer Nkzv,

The authors have responded to your questions. Does their answer address your reasons to reject?

We thank the reviewer for the observation that identifying and diagnosing a problem is alone an important step in the process of improving VLMs. We will also revise the citation format to make references more readable.

Although the paper provides evidence for the LLM's underutilization of visual information, is it possible that there might be other contributing factors that were not fully explored?

Great question! Our investigations fully cover the main architectural components of open-source VLMs to draw a general conclusion applicable across models, but there can be more nuanced reasons for how and why a particular model or training scheme leads to the failure of LLMs to use their input vision representations. For example, contributing factors may include issues in the alignment or instruction-tuning data that do not train the LLM to look carefully at its vision inputs. We will clarify in the discussion that our analysis does not preclude additional contributing factors, but highlights the LLM’s underutilization of visual inputs as the most consistent and actionable bottleneck.

We thank the reviewer for their thoughtful response to our work. Below, we reference and respond to each concern:

the paper claims that fine-tuning the LLM is very important but then they train the VLM, they freeze the LLM and only train the VLA. I think it would be helpful to see experiments where this is not frozen.

In Section 4.3, we report results on VLMs after fine-tuning on 1) vision encoder only, 2) projector only, and 3) language model only. These three settings cover the VLM components at the level of abstraction that we study them. We are happy to discuss this point further however if this section of the paper did not address the reviewer's concerns.

I have two somewhat conclicting thoughts here about the premise/conclusion of the paper. [...]

Both of these are very valuable discussion points. We agree that the main observation - that VLMs underperform vision encoders on vision-centric tasks - is unsurprising. The results supporting that point, in fact, exist when multiple separate papers are combined (e.g., [1] + [2] + [3] + [4]) though to our knowledge, we are the first to directly point out this persistent phenomenon.

What is not apparent, however, is to what part of the VLM these performance drops should be attributed. By understanding that the bottleneck for visual understanding within VLMs lies in the LLM’s usage of vision representations, the community can concentrate efforts to better exploit the visual abilities of stronger vision models rather than focus on overoptimizing for a particular vision model (e.g., CLIP-based ones).

We appreciate the reflection on Sutton’s bitter lesson, and agree that the storyline of progress often favors end-to-end systems trained at scale. We observe the reality to be that many open-source VLMs are still constructed from existing single-modal components, so understanding how these components interact - and where integration breaks down - remains critical for diagnosing limitations in today’s systems. Even if future VLM developments phase out late-fusion architectures, we hope that our experimental procedures and findings can serve as a framework for auditing multimodal models, evaluating representation usage, and guiding training data/task choices. Without such mindfulness toward diagnosing where VLMs fail, future models risk scaling blindly over persistent inefficiencies.

I'm a bit unclear on the choice of task. [...]

This is a great point - most VLM evaluations depend on proper understanding of the visual input. Among the large pool of possible tasks, we have a few core desiderata that narrows us down to the measures in the paper:

• We don't evaluate for complex reasoning or alignment to language description, since such tasks require more ability than is encoded in the vision model.
• To ensure fair comparison, we focus on tasks with wide-accepted evaluation methods for vision models (e.g., correspondence and depth) which eliminates some benchmarks such as object counting, the jigsaw task in [2], and more complicated spatial relationship tasks than the Depth Estimation task.
• We aim to cover a variety of visual understanding capabilities in both single-image and multi-image settings with little redundancy: these six tasks together cover object identity/pose (3D Object Awareness), scene understanding (Depth Estimation), correspondence within the same scene (Low-Level Matching), and correspondence between different scenes (Semantic Correspondence). Additionally, since these tasks include a localization component, we include Art Style that emphasizes texture and whose visual evaluation explicitly removing spatial information.

We will make our definition for “vision-centric tasks” more precise and differentiate them from other benchmarks used by the VLM community in our paper revision.

References

[1] Tong, Peter, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." Advances in Neural Information Processing Systems 37 (2024): 87310-87356.

[2] Fu, Xingyu, et al. "Blink: Multimodal large language models can see but not perceive." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[3] El Banani, Mohamed, et al. "Probing the 3d awareness of visual foundation models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Fu, Stephanie, et al. "Dreamsim: Learning new dimensions of human visual similarity using synthetic data." arXiv preprint arXiv:2306.09344 (2023).

Dear Reviewer 34qn,

The authors have responded to your review. Does their answer address your concerns?