Visual Data-Type Understanding does not emerge from scaling Vision-Language Models

Thank you for your insightful and constructive feedback. We greatly appreciate the effort you have invested in reviewing our paper, and we find your comments very helpful for improving our paper’s quality. Before we address all your concerns and questions, could you please further clarify your question on the language-image mismatch? Specifically, this question:

3. The reasoning for weakness: Given how simple supervision helps a lot, the weaknesses could be attributed to a language-image mismatch in these models. Particularly, if the train set rarely contains these language terms. Can any simple experiments directly negate / verify this hypothesis?

We are happy to provide additional experiments if possible, but are unsure what you meant.

The reasoning for weakness

For visual-data type identification, the paper shows weak performance of VLMs (e.g. Figure 4), especially LMMs. Section 5 provides 2 reasons (CLIP image encoder and training datasets). However, fine-tuning experiments with CLIP image encoder frozen (e.g. Table 1) provides strong performance for data-type identification. This indicates some feature separation by data-type (contrary to Sec 5 and Figure 5). Does this mean CLIP image encoder already has good task awareness (features are separated by data-type), but there is simply a mismatch between language features causing the observed weaknesses (for zero-shot probing)?

My doubt here is that, maybe better language-based probing could solve the identified weaknesses. For example, if instead of simply using the manual prompt description (Appendix B.2) for evaluation, if we use more descriptive prompts for data types (it is shown how CLIP performance improves a lot with this [1]), this may overcome the VLM weaknesses identified. I would like this concern to be addressed.

Please let me know if further clarification is needed.

[1] Visual Classification via Description from Large Language Models (ICLR '23), https://cv.cs.columbia.edu/sachit/classviadescr

• Regarding LMM evaluation details contd. This log-likelihood evaluation strategy is common for evaluating LLMs on multiple choice questions (Brown et al., Sanh et al., https://blog.eleuther.ai/multiple-choice-normalization/) and LMMs on classification tasks (InstructBLIP, OpenFlamingo). Further, as suggested by (Brown et al., https://blog.eleuther.ai/multiple-choice-normalization/) , we explored various length normalisation strategies to compute the log-likelihood scores and observed no significant changes to the results. Hence, we used the standard log-likelihood scoring procedure outlined above for all our results. We now have added new Appendix Section B.4 clearly describing this evaluation strategy and for concreteness showcase the exact code snippets for computing the log-probabilities of three of the models tested (See Appendix B.4 pages 25, 26 and 27). We think adding this information has significantly improved the clarity of our paper, thank you very much for this question.

• Regarding KNN and Linear Probing of CLIP visual features depicted in t-SNE Fig. 5. We now have run KNN on the CLIP-RN50 image embeddings as features and animal-types / data-types as labels (shown in Fig. 5). While the KNN classifier performed well on identifying the animal class (0.96 mean informedness for k=1), performance significantly dropped in predicting the data-type (0.29 mean informedness for k=1; see new Appendix Table 5 for more values of k). Similarly, a linear probe (multinomial logistic regression with LBFGS optimizer akin to what was used for CLIP) showed high linear separability of animal classes (mean informedness=0.99), whereas data-types were less linearly separable (mean informedness=0.84). These values were obtained on the train set only, as for the question you raised, we want to study linear separability of the data points and are not interested in test set generalisation. The number of 1,350 data-points in our dataset is similar to the dimensionality of the CLIP-RN50’s feature space (1,024 dimensions), which might in part account for the higher performance of the logistic regression classifier. While we feel this is somewhat a limiting caveat of the linear probe analysis, we believe this analysis adds important information. Overall, the worse performance of these classifiers when applied on data-types compared to animal-types suggest CLIP is somewhat invariant to data-types. We now have added these results to Appendix F and new Table 5.

• Regarding your suggestion to use semantic similarity metrics to analyse the VLM pre-training datasets. We agree that a purely text-based search might not fully capture a comprehensive data-type vocabulary from the pre-training dataset. Hence, we selected the sentence-transformers/all-MiniLM-L6-v2 text embedding model for performing semantic-similarity based search. This model is light-weight and optimised for effective sentence similarity matching in large corpora (Reimers and Gurevych, EMNLP 2019). With this model, we first encode the data-type text descriptions to get embeddings. We next encode the text captions in the LAION-2B-en pre-training dataset (due to compute limitations, we run this analysis only on 36 randomly selected parquet files out of 127, i.e. ~30% of the data). Leveraging these embeddings, we computed the cosine similarity between encoded data-type prompts and LAION text captions, keeping those samples that had a cosine similarity larger than 0.85. We chose 0.85 as our matching threshold by manually inspecting similarity scores on a curated set of sentences containing data-type descriptions---below 0.85 we noted that the concepts within two sentences no longer matched. The resulting rank correlation between abundancy scores and averaged model informedness evaluated on SyntheticTypeIdent were much higher compared to the simple text-search: r=0.798 for all models and r=0.776 for CLIP-based models only. These results support our previous results suggesting that lack of data-type rich samples in the pre-training data might lead to poor VLM performance for data-type identification. We have added these results into Appendix Section G.1.

Thank you once again for all of your suggestions, we truly believe that they have helped significantly strengthen the results of our paper. We hope our added experimental results and analyses have addressed your questions and comments. We are happy to add further clarifications if needed.

Thank you for thoroughly reviewing our paper, and providing your helpful comments and insightful questions. We highly appreciate your fast response and want to apologise for not replying earlier to you---we misunderstood the review process in that regard. We address all your comments and questions below.

• Regarding your first comment on prompting the LMMs and your third question on the reasoning for weakness. We agree that reasonably prompting the LMMs is crucial for prompt-based model evaluation. Following your suggestions, we now have run additional experiments, verifying that the results reported in our paper did not emerge from a peculiarity in the prompts we used.

1. We created two additional sets of alternate prompts for the 27 data-types capturing language diversity. Inspired by VisDesc (Menon and Vondrick ICLR, 2023) and CuPL (Pratt et al ICCV, 2023), we prompted GPT-4 to suggest alternative prompts for each of the original data-type prompts and manually verified that they actually capture the correct data-type concept. For instance, for the PATCH_AND_RESHUFFLE data-type, in our initial experiments we probed VLMs with the prompt “This is a patched and reshuffled image of an animal”. For the two alternative prompt-sets we probe them with “This is a scrambled image of an animal.” and “This is a jumbled image of an animal.” (the full set of alternate prompts for all data-types is listed in Appendix B.3). Evaluating a representative subset of models on SyntheticTypeIdent revealed highest model performance for the prompts we originally chose, while the alternative prompts performed slightly worse (see new Appendix C and Figure 7 for results). Further, we were curious how the newer LLaVA 1.5 model you explored would perform on our task and included it into our systematic evaluation (both the 7B and 13B variants). In line with what you found in your experiments with the web-API, we found an improvement over LLaVA 1.0 (see Figure 7). At the same time, the absolute performance is comparable to the other LMMs tested.
2. To further extend this analysis, we followed your suggestion and used the GPT-3 prompting method introduced by the paper you cited (VisDesc (Menon and Vondrick ICLR, 2023)), to generate a large set of data-type descriptors (5-7 descriptors per data-type). To probe C-VLMs with these descriptors, we followed the evaluation scheme by VisDesc: we averaged the corresponding text embeddings for each data-type and predicted the data-type whose embedding had the highest similarity with the test image. LMMs cannot be probed by comparing to averaged embeddings. Hence, we averaged the log-likelihood across descriptors for each data-type, akin to an evaluation strategy used for OpenFlamingo (see Github). This much more sophisticated prompting method led to similar results as in our main analysis and the alternative prompt experiments (see new Appendix section C and Figure 8).

In conclusion, while the exact prompting style showed some effect on model performance, overall prompt-based results on our data-type identification tasks (C-VLM outperforming LMMs, LMMs performing generally poor, only marginal scaling behaviour) robustly stayed the same across prompting strategies.

• Regarding LMM evaluation details. We thank you for pointing out that this information was not clearly included in our paper. To compute the zero-shot performance of C-VLMs, we computed the matching score of each image $I$ with each of the 27 data-type text prompts $\{D_1, D_2, …, D_{27}\}$ and predicted the data type with the highest score, i.e. $\text{predicted data-type} = \arg \max_{i\in{1, …, 27}} I_{\text{enc}}(I) \cdot T_{\text{enc}}(D_i)$. Since LMMs are auto-regressively trained, evaluating them in the same way is impossible. For a fair comparison of LMMs to C-VLMs, we instead computed the LMMs log-likelihood of a prompt joining the image and the data-type text prompts. By default, we prompted LMMs to compute the log-likelihood for the query prompt $P_i =$ “<image> Q: Describe the image. A: $D_i$”, where $D_i$ is replaced by the specific data-type text prompt. Specifically, we tokenized the prompt $P_i$ and retrieved the model log-probabilities for each input token. Then, we summed the log-probabilities across tokens yielding the final log-probability for the particular data-type, $L_i(P_i(I, D_i))$. We predicted the data-type of an image $I$ by the highest log-probability across data-types, $\text{predicted data-type} = \arg \max_{i\in{1, …, 27}} L_i(P_i(I, D_i))$. Some LMMs were trained with a particular prompt pattern for instruction tuning, and for those, we used their specific prompt template $P$ (see Appendix B.5 for full list).

Thank you for your positive assessment of our paper. As you inferred, we formulate the hypothesis that weak alignment and the discriminative-generative gap could explain why LMMs underperform C-VLMs. Although beyond the scope of our study, related work provides some initial support for this hypothesis that could form the basis for future work (Vapnik IEEE 1999, Ng and Jordan, NeurIPS 2001, Saunders et al. arXiv 2022, Bavishi et al. 2023). We now mention this more explicitly in the Results (section 4.2) and the Conclusion (section 7) of our paper.

We thank the reviewer for the helpful comments and questions. We answer both below.

• Regarding the comment about prompt sensitivity. Thank you for this great question! In our initial zero-shot results, we had used a default, fixed set of prompts for all our experiments. However, we now have run additional experiments to check for prompt sensitivity, verifying that the results reported in our paper did not emerge from a peculiarity in the prompts we used.

1. We created two additional sets of alternate prompts for the 27 data-types capturing language diversity. Inspired by VisDesc (Menon and Vondrick, ICLR, 2023) and CuPL (Pratt et al, ICCV, 2023), we prompted GPT-4 to suggest alternative prompts for each of the original data-type prompts and manually verified that they actually capture the correct data-type concept. For instance, for the PATCH_AND_RESHUFFLE data-type, in our initial experiments we probed VLMs with the prompt “This is a patched and reshuffled image of an animal”. For the two alternative prompt-sets we probe them with “This is a scrambled image of an animal.” and “This is a jumbled image of an animal.” (the full set of alternate prompts for all data-types is listed in Appendix B.3). Evaluating a representative subset of models on SyntheticTypeIdent revealed highest model performance for the prompts we originally chose, while the alternative prompts performed slightly worse (see new Appendix C and Figure 7 for results).
2. To further extend this analysis, we followed the GPT-3 prompting method introduced by VisDesc (Menon and Vondrick, ICLR, 2023), to generate a large set of data-type descriptors (5-7 descriptors per data-type). To probe C-VLMs with these descriptors, we followed the evaluation scheme by VisDesc: we averaged the corresponding text embeddings for each data-type and predicted the data-type whose embedding had the highest similarity with the test image. LMMs cannot be probed by comparing to averaged embeddings. Hence, we averaged the log-likelihood across descriptors for each data-type, akin to an evaluation strategy used for OpenFlamingo (see Github). This much more sophisticated prompting method led to similar results as in our main analysis and the alternative prompt experiments (see new Appendix section C and Figure 8).

In conclusion, while the exact prompting style showed some effect on model performance, overall prompt-based results on our data-type identification tasks (C-VLM outperforming LMMs, LMMs performing generally poor, only marginal scaling behaviour) robustly stayed the same across prompting strategies.

• Regarding in-context learning with larger models. For Otter, which we previously investigated (Figure 6b), unfortunately no in-context learning model versions larger than the investigated 7B model exist. Thus, we repeated the same experiments as for Otter with the LLaVA models, where both 7B and 13B variants are available. Interestingly, both LLaVA versions underperformed Otter-LLaMA7B and Otter-MPT7B, and surprisingly, LLaVA-13B even underperformed its smaller LLaVA-7B version (new Appendix Section H.3 and Figure 11). This experiment further corroborates the claim of our paper that in-context learning seemingly does not provide benefits for improving the performance of LMMs like Otter and LLaVA on data-type identification. Thank you again for this great suggestion.