Why are Visually-Grounded Language Models Bad at Image Classification?

We thank Reviewer CuMn for providing thoughtful feedback on our work. We address Reviewer CuMn’s questions below.

---

Limited novelty

The novelty is limited. This article assesses how well current VLMs perform on classification datasets and fine-tunes them using the same datasets. These contributions are insufficient to warrant publication of this paper.

We want to clarify that our paper is not a method paper proposing new techniques to achieve better performance on standard benchmark leaderboards. Instead, it is an analysis paper.

Object recognition is fundamental to the general reasoning capability of VLMs, yet current VLMs perform poorly in this area. We are the first to thoroughly investigate this critical issue and propose a potential solution to improve it.

Our primary contribution lies in identifying the problem (i.e., VLMs are inadequate image classifiers, a significant weakness that has been overlooked) and conducting an in-depth investigation (i.e., understanding why VLMs are poor image classifiers and how to address this issue). This contribution goes far beyond merely assessing current VLMs on classification datasets and fine-tuning them using the same datasets.

In summary, our contributions are threefold:

1. Thorough Evaluation to Identify the Problem: We thoroughly evaluated current public and proprietary VLMs on four common classification benchmarks and discovered that their performance significantly lags behind CLIP. This finding is counterintuitive because VLMs often use CLIP as their vision encoders. This finding reveals a weakness in VLMs that previous works have not noted. Understanding the limitations of VLMs is crucial given their increasing deployment in various scenarios.
2. Hypothesis Testing to Understand the Problem: Given the poor performance of VLMs in classification, we investigated the underlying reasons. We considered multiple plausible hypotheses related to VLM inference, training, and data perspectives. For example, essential information for classification could be lost during the vision encoder’s propagation through multiple LLM layers; VLMs might be inherently poor at classification due to their text generation training objective compared to the standard cross-entropy objective. Our thorough investigation ruled out these reasons but revealed the major issue: the lack of alignment data. This finding is counterintuitive because other alternative hypotheses also seemed plausible.
3. Improving VLMs based on Our Understanding: Based on our findings, we explored ways to enhance VLM performance. We believe that classification is foundational to more advanced capabilities. For example, if a VLM cannot accurately classify mushroom species, it will also struggle with follow-up questions, such as whether a particular mushroom is poisonous. Indeed, we found that simply adding classification data not only improves VLMs’ classification performance but also their general capabilities, demonstrating that accurately identifying objects is a prerequisite for answering complex questions about these objects.

Impact: recent work from Google DeepMind, PaliGemma [1], supports our main conclusion that data is the critical factor in improving VLM performance. They also found that most VLM tasks benefit significantly from longer pre-training with more data (Figure 4, Appendix K). This demonstrates that our analysis can inspire researchers to build better VLMs in the future.

[1] PaliGemma: A versatile 3B VLM for transfer.

---

Fine-tuned VLM performance on other benchmarks

I would like to know how well the fine-tuned VLMs perform on the original benchmarks such as MME, TextVQA, and others. Can we achieve similar results if a classification dataset is included in the second stage of training with LLaVA?

Thank you for your question. Following your suggestion, we evaluated both the original VLM and our fine-tuned VLM on three additional benchmarks and found that the VLM’s performance remain the same.

The table below shows the performance of the original LLaVA1.5-7B and the fine-tuned LLaVA1.5-7B on TextVQA, POPE, and MMVet. After fine-tuning, the LLaVA1.5-7B achieves the same accuracy as without fine-tuning. This result is intuitive because we included all the original LLaVA training data during our instruction tuning.

We did not include MME because the dataset is not publicly available, and our access request has not yet been approved.

We will release all the codes and model checkpoints to reproduce these results and include them in the revised paper.

---

Thank you again for your feedback. Please let us know if you have further questions or concerns!

Dear Reviewer,

We kindly request your feedback on our rebuttal. We believe we have thoroughly addressed your concerns regarding other VLM benchmarks and the novelty of our work.

If our rebuttal has resolved your concerns, we would greatly appreciate it if you could reconsider your scores for our paper. Should you have any further questions or concerns, we are eager to discuss them.

Thank you for your time and consideration!

We thank Reviewer 8fB5 for providing detailed and thoughtful feedback on our work. We address Reviewer 8fB5’s questions below.

---

Multiple textual labels

There may be multiple textual labels for many ImageNet classes. For example, n01496331 can be electric ray, crampfish...

Thank you for your suggestion. We have now considered ImageNet label synonyms [1] in our evaluation process. When evaluating with synonyms, the accuracy only improves by 1%-3%, which still results in a significant performance gap compared to CLIP.

We will include these results in the revised paper.

[1] https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a

---

Instruction-following capacity

It may harm the VLM's instruction-following capacity to incorporate the ImageNet 1.28M classification data into the original LLaVA instruction-tuning data.

Great question! We have now evaluated the original VLM and our fine-tuned VLM on three additional instruction-following benchmarks: TextVQA, POPE, and MMVet. We found that fine-tuned VLM achieves the same accuracy as without fine-tuning.

We will release all the codes and models to reproduce these results and include them in the revised paper.

---

Expected conclusion

It is expected that the information necessary for classification is largely preserved in the visual encoders in the VLMs.

We agree that it is apparent that the information necessary for classification is preserved in visual encoders. However, it is not unclear whether the information still remains after propagating through all the LLM layers (e.g., 32 layers for Vicuna-7B). Our results highlight that information is preserved after the LLM propagation rather than preserved in visual encoders.

If the instruction-tuning data for VLMs do not contain class-specific information, it is unlikely that the VLMs can automatically align the class information in the visual encoder's output to the text generation

This claim is actually unknown and controversial. The key question is: if both vision encoders and language models have seen a specific class during their uni-modal training, do VLMs need to see the exact class in a multi-modal format to align them during multi-modal training, and how much data is required? Many previous works have shown that this alignment stage is very data-efficient and even unnecessary [1, 2].

Our paper demonstrated that 1) multi-modal data is necessary for alignment, and 2) increasing the data amount leads to linearly improved performance. This provides a critical data-centric view for VLM training. Recent work from DeepMind echoes our findings. They found that most VLM tasks benefit significantly from longer pre-training with more data, as shown in Figure 4 and Appendix K [3].

We will add these clarifications in the revised paper.

[1] Visual Instruction Tuning

[2] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models

[3] PaliGemma: A versatile 3B VLM for transfer

---

ImageWikiQA clarification

It mentions that the authors randomly sampled at most 3 images for each question, but then it mentions each with an image. How many images are there per question?

There is only one image per question (see Appendix Table 11 for examples).

The confusion arises from our two-stage creation pipeline:

• 1st stage: generate questions at the class level. For any given class out of the 1000 ImageNet classes, we create questions such as “What is the native region of <class name>?”.
• 2nd stage: generate questions at the image level. Each ImageNet class has 50 images, from which we randomly sample 3 images and create 3 questions like “What is the native region of <image i>?”

We will revise the text for clarity.

---

Data split clarification

Are the results in Table 3 and line 279 are the accuracy on the validation/test sets? The accuracy on ImageNet looks very high to me.

Both Table 3 and Line 279 are evaluated on the validation set. The data splits use the official data split (e.g., ImageNet contains 1.28M training images and 50K validation images). Detailed data splits are provided in Appendix Table 6.

The very high accuracy on ImageNet is one of the significant contributions of our paper. We found that, after fine-tuning the VLM on ImageNet, there is no more gap between VLM and CLIP, with VLM now being the state-of-the-art classifier.

---

Low accuracy of LLaVA1.5-7B with GT class

In Table 4, LLaVA1.5-7B with GT Class still has a relative low accuracy. Why?

Table 4 reports the accuracy on ImageWikiQA. These questions come from Wikipedia and require extensive knowledge, which is very challenging even for humans (e.g., what is the native region of the guinea pig?)

The low accuracy (55.9%) is because LLaVA1.5-7B lacks sufficient world knowledge for these ImageNet classes. It is well known that smaller LMs like Vicuna-7B lack world knowledge compared to larger LMs like GPT4.

---

ImageWikiQA test only

No training data in ImageWikiQA?

Yes, ImageWikiQA only contains a test set. This dataset serves as an important resource to evaluate VLM’s fine-grained classification capabilities as well as its knowledge and reasoning abilities.

---

Thank you again for your feedback. Please let us know if you have further questions!

Dear Reviewer,

We kindly ask for your feedback on our rebuttal. We have conducted all the requested experiments (ImageNet multiple textual labels and instruction following capacity) and clarified concerns that due to misunderstanding.

If our rebuttal has addressed some of your concerns, we would appreciate it if you could reconsider your scores for our paper. Should you have any further questions, please do not hesitate to reach out to us.

Thank you for your time and consideration!

We thank Reviewer fPsb for their positive comments and thoughtful feedback. We address Reviewer fPsb’s questions below.

---

Analysis of proprietary VLMs

Can the authors address the hypotheses related to inference, training objectives, and data for proprietary VLMs?

Thank you for your question. Proprietary VLMs haven’t released the training details and data usage, so it is impossible to directly address training-related or data-related hypotheses. For inference-related hypotheses, we conduct additional experiments with GPT4 (see PDF in general response), and the conclusion is the same as the public VLMs.

Recent work from DeepMind [1] shows that most VLM tasks benefit significantly from longer pre-training with more data (in Figure 4 and Appendix K of [1]). This aligns with our main conclusion that data is the critical factor in improving VLM performance.

[1] PaliGemma: A versatile 3B VLM for transfer.

---

Open-world vs closed-world setting

It is not clear “open-world” vs “closed-world” settings are being used for sections 3 and 4.

Great question! Sec. 3 and 4 all use “open-world” settings (not providing the label set in the prompt), except for prompt variation analysis and label set size analysis in Sec. 3.1, using “closed-world” settings (providing the label set in the prompt).

Naturally, the “closed-world” setting narrows the model’s generation space and increases accuracy. When the model can easily predict the class in the label set by modifying the inference space (probabilistic inference in Sec. 3.1 and probing VLM in Sec. 3.2) or training VLMs with more classification data (Sec. 4), the advantage of the “closed-world” setting doesn’t exist. Thus, for these experiments, we use the “open-world” setting.

We will clarify this in the revised paper.

---

Clarification for Table 2

What is the difference between the “Closed-World Setting” performance of Table 1 and the “Base Prompt w/ Label (Fixed Order)” in Table 2? Why are they different?

Table 1’s “Closed-World Setting” is equivalent to Table 2’s “Base Prompt w/ Label (Random Order)”. In this setting, we concatenate label candidates in a random order for each question: “What’s in the image? Choose one from random_shuffle([cat, dog, pig])”. The accuracy for these two settings is identical between Table 1 and Table 2.

Table 2’s “Base Prompt w/ Label (Fixed Order)” concatenates label candidates in a fixed order for each question: “What’s in the image? Choose one from [cat, dog, pig]”. This setting is used to rule out the possibility that the order of labels might affect model performance.

---

CoT with larger VLMs

The LLaVA-7B and BLIP2-7B models are not large enough to demonstrate intrinsic capabilities such as chain-of-thought reasoning [1].

Thank you for your suggestion! We have now evaluated GPT-4’s performance on ImageNet using chain-of-thought (CoT) reasoning.

In Table 2, GPT-4 achieves 60.6% accuracy on ImageNet without CoT. With CoT, GPT-4 achieves 62.2% accuracy, demonstrating that CoT has a limited impact on image classification, even for large VLMs.

We will add these results and citations in the revised paper.

---

Cannot decode information

If the VLMs have the essential information, then why are they not able to decode it?

Great question! One possible reason is that the VLM’s decoding space is too large and not aligned with the visual features. Specifically, VLM decoding is performed through next-word prediction, which usually involves a vocabulary of over 10,000 words. The output text embedding is not aligned with the visual features due to insufficient data to align these spaces.

In general, having information in a model does not necessarily mean the model can express that information. This phenomenon is also observed in other research areas. For example, after training ResNets or ViTs with self-supervised learning methods like SimCLR, the model acquires discriminative information for different classes. However, this information can only be decoded by adding and training a linear layer on a new dataset.

Similarly, we demonstrate that VLMs possess classification information, but it is not readily expressible. Adding classification-related data helps bridge the gap between “information possession” and “information expression” (refer to Table 3 in our paper).

We will include this discussion in the revised paper.

---

Fine-tuning results

In Table 4, why is the “Finetuned on ImageNet” model performance (30.6) worse than the non-finetuned model performance (37.8 [should be 38.0])? Does finetuning always enhance the general capabilities of the VLMs?

Fine-tuning often improves in-distribution performance but does not necessarily enhance out-of-distribution or general capabilities.

Table 4 presents results of models fine-tuned on ImageNet and evaluated on ImageWikiQA, which are vastly different datasets. ImageNet questions only require classification, such as “Q:  <image> What is in the image? A: dog,” while ImageWikiQA questions demand both classification and knowledge/reasoning, such as “Q:  <image> What is the native region of this object? A: South America.”

Fine-tuning solely on ImageNet trains the model to classify, but it can lead to a loss of general capabilities like reasoning and knowledge, resulting in lower performance on ImageWikiQA (30.6 fine-tuned vs. 38.0 pre-trained). This phenomenon is known as “catastrophic forgetting” [1].

However, when we fine-tune on both ImageNet and instruction tuning data, the model learns classification while retaining its original capabilities, leading to a significant performance improvement on ImageWikiQA (49.8 fine-tuned vs. 38.0 pre-trained).

We discussed this in Lines 281-285 and will add further clarification.

[1] Overcoming catastrophic forgetting in neural networks.

---

Thank you again for your feedback. Please let us know if you have further questions!

Dear reviewer,

We would kindly ask for a response to our rebuttal. We believe that some of your concerns are due to a misunderstanding and are not weaknesses, and we would appreciate it if you would revisit your evaluation of our work. Thank you for your time and consideration!

Dear Reviewer fPsb,

With just 1 day left in the response period, we would greatly appreciate it if you could kindly review our responses soon. We are still looking to discuss any remaining concerns.

Thank you for your time!

We thank Reviewer SHnR for their positive comments and thoughtful feedback. We address Reviewer SHnR’s questions below.

---

Main reason and solution

I'm not completely convinced with the paper's final conclusion of data being the reason why CLIP models are superior to VLMs… The data argument for explaining the gap might not be the only, or even the strongest reason.

We clarify that data is not the primary reason. The primary reason is that the information required for classification is encoded in the VLM’s latent space but cannot be effectively decoded. Data serves as an effective solution to decode this information, as our experiments show that with sufficient data, VLMs can match the accuracy of state-of-the-art classification models.

This implies that either what's missing is a task-aligned fine-tuning or perhaps even a modification of how these are inferred on for image classification which can lead to a more a task-aligned inference.

Thank you for the thought-provoking question. We agree that task-aligned fine-tuning or inference, such as adding a linear probing loss on the VLM output space for training and using KNN in the VLM output space for evaluation, can improve VLM classification performance.

However, our focus is on enhancing VLM’s general capabilities in solving a variety of tasks rather than the specific classification task. The only natural inference interface for different tasks is text generation. Task-aligned inference, like KNN, is only applicable to classification and not other tasks.

For task-aligned fine-tuning, such as adding a new token and a linear probing loss, we observed no improvement when evaluating this fine-tuned VLM on ImageWikiQA. This shows that task-aligned fine-tuning does not improve VLM’s overall capabilities.

In summary, without changing VLM training or inference to maintain its general capabilities for different tasks, adding data is the most promising and effective approach. We will add these clarifications in the revised paper.

---

Solution to other problems

If this VLM limitation stems for other reasons such as multimodal confusion and text feature interference or hallucination problem specific to the generated text space, adding more data for different tasks as a solution might not generalize to similar VML issues.

Thank you for your question. We agree that for issues like multimodal confusion or hallucination, adding data might not resolve the problem, but addressing these is beyond the scope of our paper.

The main contribution of our paper is the formulation of the problem that VLMs are poor image classifiers and a thorough investigation into why this is the case and how to solve it. Our paper demonstrates that the poor classification performance of VLMs is due to the inability to decode the information encoded in VLMs. Given this reason, data serves as an effective solution to decode the information.

---

Label set size analysis

Do you think the label set size experiment shows the evaluation difference between CLIPs and VLMs can explain some of the gap?

Thank you for your question. The label set size experiment can partially explain the gap but not entirely.

By reducing the number of labels for classification, we can narrow the gap between VLM and CLIP, but a gap remains across all label set sizes, even with just two labels (two-way classification).

Moreover, we find that while the absolute gap between VLMs and CLIPs narrows with reduced label size, the relative gap increases (Appendix Figure 4). For example, in two-way classification on ImageNet, VLMs have a 5.7% error rate, while CLIP has a 0.2% error rate, resulting in a 28.5x gap; for 20 classes, VLMs have an 18.0% error rate, while CLIP has a 2.5% error rate, resulting in a 7.2x gap.

These results indicate that the VLM-CLIP gap cannot be fully explained by label set size.

Why do you think the Caltech and ImageNet datasets behave a little differently than Cars and Flowers in Figure 2?

The absolute gap on Flowers and Cars is larger than ImageNet and Caltech when reducing the label set size. This may be because Flowers and Cars are more fine-grained classification datasets, while ImageNet and Caltech are more coarse-grained. VLMs are weaker in fine-grained classification compared to CLIP, resulting in a larger gap.

We will add these clarifications in the revised paper.

---

Class prevalence bias

Why do you think CLIP does not suffer from the class prevalence bias but LLava does in Figure 3?

Great question! In Figure 3, the class frequency on the x-axis is computed based on the LLaVA pre-training and instruction-tuning dataset, not the CLIP training dataset. If we change the x-axis to the CLIP pre-training data frequency, CLIP should also show prevalence bias. Please refer to Figure 2 in [1].

[1] No “Zero-Shot” Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance.

What happens if we plot this for linear probed LLava models?

We plotted the line for the linear probed LLaVA model, which aligns with the fine-tuned LLaVA (see PDF in general response). The zero correlation is expected because fine-tuning or probing on the ImageNet dataset equalizes class frequencies, as ImageNet classes are evenly distributed. We will include this line in the updated paper.

What happens if we plot this for linear probed Llava models which are fine-tuned on similarly biased data having similarly skewed frequencies?

We have not yet trained a new model on skewed data, but we found that training on the ImageNet classification dataset does not improve performance on the Flowers dataset, which has drastically different class frequencies. This further verifies that data, rather than fine-tuning, is the key factor determining performance. We will add this in the revised paper.

---

Thank you again for your feedback. Please let us know if you have further questions!

Dear Reviewer,

We kindly request your consideration of our rebuttal. We believe we have thoroughly explained why the data presents a promising solution rather than the root of the problem, and why altering the training or inference objective would not be appropriate. Additionally, we have addressed the other questions raised.

Thank you for your time and consideration!

Dear Reviewer SHnR,

With just 1 day left in the response period, we would greatly appreciate it if you could kindly review our responses soon. We are still looking to discuss any remaining concerns.

Thank you for your time!