Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity

Dear Reviewer jHnf,

We appreciate your feedback and are thankful for your recognition of the value and novelty of our work. We would like to address your concerns and questions below.

• The differences between our observations and the CHiLS [1] paper as follows: CHiLS demonstrates that mapping predictions from leaf classes across granularity levels 1-6 can enhance results for ImageNet classification (Table 1 in [1]), aligning with our findings in a multi-label classification context (Figure 3 in our paper). However, our analysis extends to all levels of the ImageNet hierarchy, up to 16. We observed distinct behaviors when the leaf classes are extremely fine-grained. In such cases, propagating their predictions to parent classes may not be beneficial. For instance, certain classes at level 16 exhibited negative changes when predictions were propagated from children at level 17, as illustrated in our Figure 3-Right.

• Regarding the potential improvement in Specificity through modified training, our Appendix (Section C) explores this. We finetuned CLIP-B on COCO with diverse text samples generated by our benchmarking strategy. While fine-tuning with challenging text samples shows promise, it falls short of a comprehensive solution. Notably, the models struggle with cases outside the augmentation strategy's scope, underscoring the limitations of relying solely on fine-tuning with hard samples. Moreover, achieving certain augmentation goals, like enhancing alt-text to increase correct information, presents significant challenges.

• To clarify Section 4.2 (Impact of Pre-training Data), we compared the performance of UniCL models trained on distinct datasets. These datasets include ImageNet21K, featuring classification prompts (e.g., “a photo of {}”) and class names, and image alt-text data (YFCC-14M and GCC-15M).

  • UniCL trained solely on image alt-text data (UniCL_YFCC) outperforms UniCLIN21K in FG leaf label classification (35.75 vs. 26.28 mAP).
  • UniCL_IN21K, trained on ImageNet21K, excels in CG ancestor raw performance due to the inclusion of all CG classes in IN21K, surpassing CLIP and OpenCLIP trained on larger-scale alt-text data.
  • Training UniCL on a combination of IN21K with Alt-text Data (YFCC14M and GCC15M) improves FG classification but slightly worsens CG classification, leading to a larger CG-FG discrepancy.
  • Scaling up alt-text training data: Larger models, such as OpenCLIP-H-2B trained on LAION2B, show inferior CG ancestor raw performance and a more pronounced CG-FG discrepancy than UniCL_IN21K, indicating that merely scaling up the training data or model sizes is insufficient to resolve this discrepancy effectively.

  We will modify our draft to make this section clearer.

Dear Reviewer stdj,

Thank you for your time for reviewing our submission.

We acknowledge your concern regarding the straightforward nature of our technical evaluation. However, we wish to emphasize that the straightforwardness of our approach was crucial for ensuring the clarity and replicability of our results. Our methodology, while seemingly intuitive, was designed to rigorously test VLMs in scenarios that have not been explicitly showcased or examined in existing literature.

Regarding the novelty of our work, we believe our contribution is significant in several aspects: Empirical Evidence and Novel Benchmarks: We have developed new benchmarks and conducted comprehensive analyses, providing empirical evidence of the VLMs' performance across different granularity and specificity levels. Development of a New Evaluation Protocol: A major novelty of our research lies in the creation of a new evaluation protocol and the modification of existing datasets to fit this protocol. This effort was non-trivial and has not been previously undertaken to our knowledge. We would greatly appreciate it if the reviewer could provide references to previous works that are dedicated in the same topic of our work. We also provide deep analysis on how the training data impact the observed granularity issue and if finetuning can solve the specificity issue.

We believe that our research fills a crucial gap in the current understanding of VLMs, and we hope that our responses clarify the novelty and depth of our contributions to this important area of study.

Dear Reviewer pPRh,

Thank you for your valuable feedback on our submission. We've carefully considered the concerns and questions you've raised and would like to offer the following responses.

• Lack of Novel Insights:

  • Although some issues relevant to detailed language understanding may have been studied in the NLP community, our research brings a novel perspective by investigating these issues in the context of VLMs. Unlike LMs that process only textual content, VLMs require an intricate understanding of how visual elements correspond with textual descriptions with unique challenges. For instance, understanding the granularity in the context of VLMs involves not only textual recognition of 'leopard' vs 'animal' but also the correct visual identification of these categories in diverse images.
  • While we acknowledge the relevance you've drawn with studies [1] and [2], we would like to underscore the distinct contributions of our research in the realm of Vision-Language Models (VLMs). [1] primarily examines the memorization and retrieval capabilities of LMs in QA tasks in relation to the frequency of information in training datasets. In contrast, our work evaluates VLMs' ability to interpret and relate visual concepts with varying levels of granularity and text of various specificity, a challenge that inherently involves interpreting visual stimuli alongside textual data. Reference [2] ("Can Large Language Models Truly Understand Prompts?") focuses on the comprehension of negated prompts in LMs. Our research on specificity, however, is concerned with how the VL matching score can be distorted by the amount of information in the text, a distinct and unexplored area in the field of VLMs.
  • We acknowledge the intuitive aspect of our finding that "VLMs are better at matching images with commonly occurring captions." However, our research adds significant value by grounding this intuition within the newly formulated benchmarks for granularity and specificity evaluation. This framing provides a structured and empirical approach to understanding how VLMs handle varying levels of granularity and specificity in the complex interplay of visual and textual data, offering insights that extend beyond the general understanding of VLMs' capabilities based on their training data distribution.

• Textual Bias in Specificity Benchmark:

  • We would like to clarify a critical aspect of our study that seems to have been misunderstood. Our language-only study was not aimed at demonstrating the competitiveness of language-only methods on the VLM benchmark. Instead, its purpose was to investigate whether the granularity issue observed in VLMs also manifests in language modeling alone.
  • We specifically designed a text-classification task based on two-level granularity concepts, derived from our VLM benchmark, to assess if language models exhibit similar granularity issues. Our findings revealed that while text encoders in VLMs and embedding-type LMs struggle with this task, larger generative Language Models (LLMs), such as GPT-3.5, show better performance. However, it remains unclear whether this improved performance is attributable to the generative modeling approach or simply the larger scale of these models.

• On the Importance of Granularity from Vision WordNet for its structured and widely-accepted semantic relationships, which provide a clear framework for evaluating VLMs' understanding of granularity. We believe overall it defines the most widely accepted conceptual relations. To clarify, the example “road-cone” and “barrier” we used to explain the granularity issue is not part of WordNet. We believe WordNet’s semantic relationships provide a solid foundation for evaluating VLMs' understanding of granularity while it may not capture every nuance of real-world visual categories. We will include the discussion in the limitation section.

• Summary Metric: our study does not employ a single summary metric as it evaluates VLMs across two separate benchmarks: granularity and specificity, each testing different aspects of VLM capabilities. Consequently, each benchmark has its specific set of metrics that appropriately capture the distinct properties we aim to assess. This approach ensures a more nuanced and accurate evaluation of VLM performance in diverse scenarios.

• How Should Training be Amended? We acknowledge that altering the training regime to fit a specific benchmark can introduce biases. Using LLM to augment more diverse text examples may improve it. We also recognize that this is not a universal solution to the inherent challenges in embedding matching based models. Our language-only study indicates that generative models, such as GPT-3.5, show promising results. Therefore, exploring the use of generative VLMs might be a fruitful direction for future research, despite the trade-off in computational cost.

We will carefully review our draft to correct any spelling or grammatical errors.

Dear Reviewer PhzX,

We appreciate your thorough review of our paper and insightful comments. Below, we address each of your concerns:

• The influence of prompt design:

  We appreciate your comments on the influence of prompt design in classification, particularly regarding granularity. Our study utilized popular prompt templates from the CLIP paper, considering its broad application in vision-language (VL) model research. To further explore this aspect, we conducted additional analyses using a simplified prompt format: “a picture of {}.” Our updated results, which are in the table below, affirm that our conclusions on granularity issues remain consistent under this varied prompt design approach.

  Model	Leaves	Ancestor$_{raw}$	Ancestor$_{child}$	Ancestor$_{leaf}$
  CLIP ViT-B/32	48.17	22.96	43.01(+20.05)	55.93(+32.97)
  CLIP ViT-L/14	60.77	29.73	52.98(+23.25)	67.13(+37.40)
  OpenCLIP B-400m	50.93	22.07	43.99(+21.92)	58.66(+36.59)
  OpenCLIP B-2B	57.04	25.8	49.20(+23.40)	64.27(+38.47)
  OpenCLIP ViT-L-14 laion2b	67.42	31.98	57.76(+25.79)	74.16(+42.19)
  OpenCLIP ViT-H-14 laion2b	68.87	32.19	58.44(+26.25)	74.64(+42.45)
  UniCL_YFCC	30.73	16.26	30.99(+14.73)	42.06(+25.80)
  UniCL_IN	22.84	34.93	35.56(+0.63)	35.53(+0.60)
  UniCL_YFCC+IN	40.92	32.09	45.34(+13.25)	54.27(+22.18)
  UniCL_all	44.43	28.78	44.13(+15.35)	55.33(+26.55)
  KLITE	45.82	26.93	45.26(+18.33)	58.03(+31.10)
  BLIP	44.87	21.14	41.21(+20.07)	54.45(+33.31)
  BLIP_ft	45.88	23.62	43.99(+20.37)	57.12(+33.51)
  FLAVA	36.05	17.57	33.99(+16.42)	45.68(+28.10)

• Going Beyond Common Knowledge: We understand your concern that our findings about performance improvement under training-like scenarios may appear intuitive. Nevertheless, our contribution lies in the empirical evidence and systematic exploration of this phenomenon, which has not been explicitly explored in previous research. Our benchmarks provide a foundation for future in-depth studies in this area. Furthermore, we explored the efficacy of simple fine-tuning with hard examples to address the specificity issue. Detailed in Appendix (Section C), we fine-tuned the CLIP-B model on COCO using various text samples generated through our benchmark strategy. While fine-tuning shows some promise, it falls short as a comprehensive solution, particularly in instances outside the augmented sample scope. This finding underscores the limitations inherent in relying solely on fine-tuning with hard samples.

• Connection to Specific Vision Tasks:

  We are grateful for your suggestion to link our benchmarks to specific vision-language tasks. Downstream VL tasks like open-vocabulary detection/tracking and text-to-image generation often leverage pre-trained VL models assessed in our benchmarks. They often leverage the pretrained VL models tested in our benchmarking by knowledge distillation or apply similar image-text embedding alignment strategies on task-specific models with comparable image-text data. Consequently, they are subject to the same granularity and specificity issues we identified. Our improvements in these benchmarks should, therefore, translate to advancements in these tasks, yielding more robust pre-trained VL models and improved training methodologies. We will incorporate this discussion into our manuscript to underscore our benchmarks' practical implications in enhancing VLM performance across various applications.

• Regarding your question on Evaluation Using COCO: We believe the original captions are more effective due to their similarity in information content with the alt-text in VL training data, which is typically shorter than Localized Narratives captions. While there is overlap between COCO images and web-crawled datasets like LAION and YFCC, COCO's distinct annotation process separates its captions from these datasets. Recognizing your concern about potential model exposure to COCO images and captions during training, we note that such exposure is a common challenge in VL model evaluations. However, given the vast scale of pretraining data (billions of image-text pairs) compared to the size of the COCO dataset (hundreds of thousands), the models' behavior is predominantly shaped by the broader training data spectrum, where COCO's portion is minimal.

Dear Reviewer 1Zb8,

Thank you for your detailed and insightful review of our submission. We address your questions as follows:

• Explore Simple Ideas to Mitigate Granularity and Specificity Issues by Lightweight Fine-tuning:

  We appreciate your suggestion to explore lightweight fine-tuning methods. Our appendix (Section C) investigates this for the specificity issue, where we finetuned CLIP-B on COCO using various text samples generated by our benchmarking strategy. While fine-tuning with hard text samples shows some promise, it’s not a comprehensive solution. In particular, we found that the models tend to fail on cases not covered by the augmentation strategy. This highlights the inherent limitations of relying solely on fine-tuning with hard samples. It’s also important to note that some augmentation goals, such as "augmenting the alt-text with LLM to increase the amount of information", can be challenging to achieve. There is no guarantee that the LLM-generated information will be well aligned with the image, posing a significant challenge for ensuring model accuracy and relevance. We concur that the ideas you have suggested are intriguing and will discuss them as potential future directions, acknowledging the complexities involved.

• Translating Findings to Downstream Applications:

  Our study provides essential insights into the limitations of current VLMs in terms of granularity and specificity, forming a basis for enhancing these models for practical applications such as open-vocabulary object detection. We acknowledge the potential of applying our findings to practical applications, as you kindly suggested, and will include a discussion in our revised draft on this promising yet extensive topic, which requires further investigation beyond our current scope.

• Regarding the "Not Surprising" Nature of Findings:

  We believe that the seemingly intuitive nature of findings does not diminish their importance. Our work, like recent studies on VLMs' compositional understanding limitations, provides vital empirical evidence, despite some findings being in line with expectations. These insights are crucial for advancing the field. We have discussed the limitations in Appendix Section E and will expand this discussion to further articulate the significance of our findings.

• Additional Experiment - As the reviewer suggested, we further studied the impact of prompt design on classification for granularity study. We conducted supplementary experiments using a simplified prompt “a picture of {}.”. The results reconfirm our initial findings regarding granularity issues in VLMs and further solidify our conclusions.

  Model	Leaves	Ancestor$_{raw}$	Ancestor$_{child}$	Ancestor$_{leaf}$
  CLIP ViT-B/32	48.17	22.96	43.01(+20.05)	55.93(+32.97)
  CLIP ViT-L/14	60.77	29.73	52.98(+23.25)	67.13(+37.40)
  OpenCLIP B-400m	50.93	22.07	43.99(+21.92)	58.66(+36.59)
  OpenCLIP B-2B	57.04	25.8	49.20(+23.40)	64.27(+38.47)
  OpenCLIP ViT-L-14 laion2b	67.42	31.98	57.76(+25.79)	74.16(+42.19)
  OpenCLIP ViT-H-14 laion2b	68.87	32.19	58.44(+26.25)	74.64(+42.45)
  UniCL_YFCC	30.73	16.26	30.99(+14.73)	42.06(+25.80)
  UniCL_IN21K	22.84	34.93	35.56(+0.63)	35.53(+0.60)
  UniCL_YFCC+IN21K	40.92	32.09	45.34(+13.25)	54.27(+22.18)
  UniCL_all	44.43	28.78	44.13(+15.35)	55.33(+26.55)
  KLITE	45.82	26.93	45.26(+18.33)	58.03(+31.10)
  BLIP	44.87	21.14	41.21(+20.07)	54.45(+33.31)
  BLIP_ft	45.88	23.62	43.99(+20.37)	57.12(+33.51)
  FLAVA	36.05	17.57	33.99(+16.42)	45.68(+28.10)