On Erroneous Agreements of CLIP Image Embeddings

Thank you for your detailed comments on our paper! We hope that our response below will help you better understand our paper.

1. Your concerns about not showing that a two-layer MLP is important in distinguishing two erroneously agreeing images.

   We did not suggest that the two-layer MLP is the sole key factor in the performance gap. The argument in our paper is that the visual information extraction strategies of VLMs, by which we mean techniques applied on top of the image encoder, are important. In LLaVA-1.5, this refers to both the two-layer MLP connector and the language model used for answer generation.

   Although separating the functions of the connector and the language model in MLLMs is an interesting question for future exploration, this is not the focus of our paper. To avoid future confusion, we clarified the meaning of visual information extraction in the introduction of our revised submission (L74-L76).

2. Your concerns regarding our use of Spearman's rank correlation and why ρ = −1 show that the embeddings are "fully opposed."

   In our paper, Spearman's rank correlation coefficient serves as a toy example to show that cosine similarity does not depict all aspects of vector pairs, and we could still get different information from vectors with high cosine similarity.

   For the vector pairs in our example ($[10,11,12]^\top$ and $[12,11,10]^\top$), their Spearman's rank correlation coefficient is -1, indicating that they have a perfectly inverse order. Hence, their order information is fully opposed in this sense.

   For your example of interpreting these two vectors as the embeddings of two dog images, we did not intend to show that the dogs "look opposite in the general sense." Instead, we show that in the specific sense (order information), the two embeddings are opposite, and we can extract this opposite information through Spearman's rank correlation coefficient. If they are indices of the dog's features you mentioned, the two embeddings could yield opposite answers to the question, "Are the indices of the dog's features in an ascending order?"

   We also rephrased the relevant description (L243-L244) for the toy example in our submission to better express our intention.

Feel free to ask further questions, and we are willing to address them as well!

Thank you so much for your detailed feedback! Below, we offer responses to assure your concerns:

1. Your feedback that the general storyline of our paper is confusing.

   Thank you for your valuable suggestion in our writing!

   Our storyline is as follows: Previous research attributes LLaVA-1.5's low performance on the MMVP benchmark to deficiencies in the CLIP vision encoder, specifically erroneous agreements. We first show that LLaVA-1.5 is able to perform well on image pairs with erroneous agreements as shown on the What'sUp benchmark, whereas CLIP cannot. Then we conduct an ablation study to identify the key difference between CLIP and LLaVA-1.5 that causes their performance gap. Finally, we revisit the MMVP benchmark and provide insights into the true cause for LLaVA-1.5's shortcomings.

   In our paper, What'sUp is considered to be easier for LLaVA-1.5 than MMVP, and the latter remains unsolved. Since the What'sUp and MMVP benchmarks are less known than the benchmarks you mentioned, it is useful to introduce and explain more on model performance and what people expected on them previously. In our submission, we introduced this difference in the introduction (L86) and in task setup (L223). Now we added more context to Figure 1 (both the image and the caption), highlighted this difference more in the introduction (L52-L54, L67), and revised the relevant description (originally in L186, now in L193-L195) to improve reader experience.

2. Your concerns that our scope, task, and dataset choices are not enough to derive general conclusion on the visual reasoning abilities of LLaVA and MMLMs.

   Our goal is not to draw conclusions about the general visual reasoning abilities of LLaVA or other MMLMs or to prove their supremacy over CLIP generally. We focus on the specific type of task that requires the VLMs to distinguish image pairs with highly similar CLIP embeddings (erroneous agreements), which were claimed to cause VLMs' shortcomings in the paper "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs." This motivates our use of the What'sUp and MMVP benchmarks for comparisons and ablation studies, as they consist of such image pairs. On these benchmarks, we show contrary evidence that LLaVA-1.5 can extract distinct information even from these similar embeddings.

   The other benchmarks you mentioned, including VQA, GQA, OK-VQA, VCR, and those used by LLaVA (VisWiz, SQA, TextVQA, POPE, MME, MMB, Chinese MMB, SEED, LLaVA-Bench, MM-Vet), are valuable for evaluating general visual reasoning capabilities. However, since they do not focus on distinguishing similar paired images, they fall outside the scope of our study.

   Although our scope is limited to this specific task type, our findings highlight important insights: CLIP and LLaVA-1.5 paradigms employ inherently different mechanisms for extracting visual information, and there is still room to enhance VLMs with a fixed, pretrained image encoder. These findings contribute to the broader goal of designing more capable VLMs for general visual reasoning tasks.

   We further clarified our scope in the related work section (L104-L107) and add more captions to Figure 2 to avoid confusion.

If you have further concerns or questions, please let us know and we are willing to reply as well!

Thank you for your reply! Below, we offer further explanation and evidence to assure your concerns:

1. "The benchmarks the authors used are not enough to support this claim. Specifically, I asked for experiments on other datasets, including the ones used in the original paper [R1] that the authors counteract, but the authors did not address this during the discussion period."

   The benchmarks used in the original paper [R1] include MMVP/MMVP-VLM, LLaVA-Bench, POPE, LLaVA-In-the-Wild, MMBench, TextVQA, VQA-v2, and MM-Vet. Apart from MMVP/MMVP-VLM we used, the rest do not focus on distinguishing two images with erroneous agreements, and thus do not have paired similar images with opposing answers to questions. Hence, results on these benchmark do not contribute to illustrate that LLaVA-1.5 shows a better visual information extraction ability than CLIP on images with erroneous agreements.

   To address the need for more generalizable insights, we tested CLIP and LLaVA-1.5 on another benchmark, NaturalBench [R2], to show the generalizability of our argument. It is a vision-centric benchmark that challenges vision-language models with pairs of simple questions about natural imagery. This dataset contains 1,900 test cases, each consisting of two images and two questions with opposing answers. Since NaturalBench follows MLLM's format but not CLIP's, we converted the questions and options into captions using GPT-4o-mini for CLIP evaluation. We report the individual accuracy and pairs accuracy as we did in Table 3 of our submission. Below are the results:

   Table 1: Results of CLIP-ViT-L/14-336px and LLaVA-1.5-7B on NaturalBench. The results for LLaVA-1.5-7B come from the Acc. and Q-Acc. in Table 1 of the NaturalBench paper.

   	Indiv. Acc.	Pairs Acc
   CLIP-ViT-L/14-336px	56.5	20.5
   LLaVA-1.5-7B	67.3	37.7
   Random Chance	50.0	25.0

   Given that the images were not collected by high CLIP cosine similarity, we further consider their performance on 320 testcases in NaturalBench with the highest CLIP image cosine similarity (>0.85):

   Table 2: Results of CLIP-ViT-L/14-336px and LLaVA-1.5-7B on 320 testcases from NaturalBench.

   	Indiv. Acc.	Pairs Acc
   CLIP-ViT-L/14-336px	53.7	13.9
   LLaVA-1.5-7B	63.0	28.8
   Random Chance	50.0	25.0

   From the results, we can see that the performance gap between LLaVA-1.5 and CLIP on similar image pairs generalizes to this benchmark, which has larger size than MMVP and has more diverse visual content than What'sUp, showing the generalizability of our finding.

2. "Moreover, only LLava is considered, and we cannot get generalizable insights from one model. The results could be of the way Llava is trained or because of its architecture."

   First, we showed by additional results of LLaMA-3-V-8B and Phi-3-V-3.8B that our findings generalize to other LLaVA-like models of varying scales and language backbones.

   Second, we acknowledge the possibility that the way LLaVA-like models are trained or their architecture might be important to realize strong extraction ability. We did not intend to prove the advantage of all MLLMs over their image encoders, since we note that MLLMs adopt various model architectures and training pipelines, and some of them might not be effective in visual information extraction. Nevertheless, our findings on LLaVA-like models show the feasibility of extracting distinct information from highly similar image embeddings through MLLMs, together with the effectiveness of LLaVA's methodology compared with CLIP on such image pairs.

We hope these additional analyses and clarifications address your concerns. Thank you for your valuable feedback!

[R2] NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples.

Thank you for your feedback on our paper and your insightful questions! We really appreciate your suggestions, which helps improve our work. Here are our responses to your questions:

1. How do these findings generalize to other MLLMs beyond LLaVA-1.5?

   We first would like to clarify that we did not intend to show that all MLLMs are better than CLIP. Instead, our main focus is to explore whether erroneous agreements are bottlenecks for VLMs (whether we can extract distinct information from highly similar CLIP embeddings or not), and we found that LLaVA-1.5 can capture such nuance, while CLIP cannot.

   Regarding your question, we agree that generalizing our finding to other MLLMs is an interesting direction and good supplementary to our work. For serving our purpose above, we focus on MLLMs that have CLIP as the image encoder and freeze it during training (otherwise, this variable is not controlled). Hence, we evaluate two other MLLMs, LLaMA-3-V-8B and Phi-3-V-3.8B, which use frozen CLIP-ViT-L/14-336px as the vision encoder. We use the model weights provided in https://github.com/mbzuai-oryx/LLaVA-pp.

   Table 1: Results of LLaMA-3-V-8B and Phi-3-V-3.8B on What'sUp Subset A. For comparison, we also include the results for CLIP-ViT-L/14-336px and LLaVA-1.5-7B from the Table 1 in our submission.

   	Left/Right, Indiv.	Left/Right, Pairs	On/Under, Indiv.	On/Under, Pairs
   CLIP-ViT-L/14-336px	49.0	1.9	61.7	23.3
   LLaVA-1.5-7B	99.0	98.1	80.1	60.2
   LLaMA-3-V-8B	90.3	80.6	57.8	20.4
   Phi-3-V-3.8B	100	100	85.4	70.9
   Random Chance	50.0	25.0	50.0	25.0

   Table 2: Results of LLaMA-3-V-8B and Phi-3-V-3.8B on What'sUp Subset B. For comparison, we also include the results for CLIP-ViT-L/14-336px and LLaVA-1.5-7B from the Table 1 in our submission.

   	Left/Right, Indiv.	Left/Right, Pairs	Front/Behind, Indiv.	Front/Behind, Pairs
   CLIP-ViT-L/14-336px	54.9	10.8	51.5	7.8
   LLaVA-1.5-7B	100	100	98.5	97.1
   LLaMA-3-V-8B	71.1	46.1	69.1	41.2
   Phi-3-V-3.8B	100	100	56.9	13.7
   Random Chance	50.0	25.0	50.0	25.0

   Table 3: Results of LLaMA-3-V-8B and Phi-3-V-3.8B on What'sUp (four-way classification) benchmark. For comparison, we also include the results for CLIP-ViT-L/14-336px and LLaVA-1.5-7B from the Table 2 in our submission.

   	Subset A, Indiv.	Subset A, Pairs	Subset A, Set of 4	Subset B, Indiv.	Subset B, Pairs	Subset B, Set of 4
   CLIP-ViT-L/14-336px	28.9	1.0	0.0	27.2	1.0	0.0
   LLaVA-1.5-7B	62.1	41.3	14.6	74.0	61.8	23.5
   LLaMA-3-V-8B	60.0	36.4	17.5	70.1	43.6	20.6
   Phi-3-V-3.8B	58.0	36.4	15.5	71.8	55.4	12.8
   Random Chance	25.0	6.3	0.4	25.0	6.3	0.4

Thank you for your comments and suggestions.

First of all, we want to point out a misunderstanding in your review. In the "Strengths" section of your review, you concluded that,

• "Therefore, it suggested a complementary metric, Spearman's rank correlation coefficient."

However, Spearman's rank correlation coefficient is only used as a toy example in Section 3.2 for explaining why we hypothesize that distinct information might still be preserved in embeddings with high cosine similarity and can be extracted. This coefficient does not appear in other contexts in our paper, and is not part of our core contribution.

After clarifying this, we address your concerns as follows:

1. Your concerns on what our main contributions are and how our contributions provide novel insights beyond the CVPR 2024 paper.

   Our contributions are summarized as follows:

   (1) First, we show evidence on the What'sUp benchmark that LLaVA-1.5 can distinguish images with CLIP embeddings of high cosine similarity, indicating that erroneous agreements are not the bottleneck of their visual reasoning performance on image pairs.

   (2) Second, since we observe that CLIP has poor performance when image embeddings have high cosine similarity, we conduct ablation studies to see what difference between CLIP and LLaVA-1.5 causes their performance gap. We find that evaluation methods, language encoder, and training data are not the key factors that cause the gap, and we suggest that their difference in model paradigm (training and inference methods) plays an important role here.

   (3) Third, we explore the true bottleneck of LLaVA-1.5 through two attempts in the discussion section. We find that LLaVA-1.5 does not attend enough to the visual input, and more visual information is preserved in CLIP image embeddings and aligned correctly than the pair accuracy suggested on the MMVP benchmark.

   Our contribution (1) counters the claim in the CVPR2024 paper "Eyes Wide Shut" that MLLMs fail in simple questions on image pairs because their pre-trained CLIP vision encoders overlook crucial visual details in images as they encode them into highly similar embeddings. While this previous paper states that "vision models might become a bottleneck in multimodal systems," we suggest the potential of improving current MLLMs with fixed vision models through better utilizing visual information.

2. Your suggestion on including Spearman's rank correlation in Table 1 and providing Spearman's rank correlation coefficient and cosine similarity for the questions with the highest- and lowest-accurate answers.

   As we explained at the begining, Spearman's rank correlation coefficient is not a metric we proposed to use on these benchmarks. It only serves as a toy example to show that cosine similarity does not depict all aspects of vector pairs, and we could still get different information from vectors with high cosine similarity. This motivates us to explore whether, in image embeddings with high cosine similarity (erroneous agreements), distinct information is actually preserved and can be extracted by MLLMs.

We hope that our response could help you better understand the content and the contributions of our paper. If you have further questions or concerns, we are willing to address them as well!

Thank you for the detailed response. After thoroughly reviewing your explanation and the feedback from other reviewers, I have identified additional concerns that warrant clarification.

The performance comparison between LLaVA-1.5 and its vision encoder, OpenAI-CLIP-L/336, on the MMVP-VLM benchmark is interesting. However, the analysis appears limited, as it includes only a single comparison. It would be better if the authors provided a more diverse and consistent comparison, not limited to OpenAI-CLIP-L-336 vs. LLaVA-1.5, but also EVA-01-CLIP-g vs. InstructBLIP (Vicuna7b, Vicuna13b, Flan T5xl). In this way, the authors can analyze the influence of the different Visual Encoders and the LLM architectures on the performance boost.

Furthermore, while it is interesting to observe improvements in the instructional model, the gains may result from LLaVA training. It adapts the language model (LLM) to leverage the frozen vision encoder better. This could enhance the LLM’s ability to extract visual information from sequence embeddings compared to CLIP models relying on the CLS token to align with smaller and limited language models. However, despite these improvements, the overall performance remains suboptimal. Consequently, I do not find the claim that “LLaVA-1.5 can distinguish images with CLIP embeddings of high cosine similarity, indicating that erroneous agreements are not the bottleneck of their visual reasoning performance on image pairs” to be fully substantiated.

Given these considerations, I maintain my initial score.

In response to Reviewer fpLb's suggestion on providing a more diverse and consistent comparison (not limited to OpenAI-CLIP-L-336 vs. LLaVA-1.5), we further updated the Appendix B.5 with evaluation for InstructBLIP-Vicuna-7B and Otter-Image-MPT7B on What'sUp, COCO-spatial, and GQA-spatial benchmark.