Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification

Thank you for your thoughtful comments on our manuscript. Below we address your comments and questions about our work.

The experiments only include one type of VLM, i.e., CLIP. The reviewer would like to know whether FuDD is compatible with more advanced VLMs.

Thank you for your suggestion. To further evaluate FuDD, we run additional experiments with OpenCLIP, which is much stronger than CLIP, and BLIP2, which uses an entirely different training strategy. We observe similar performance improvement in this new set of experiments, which further supports the argument that FuDD benefits a wide range of VLMs.
Please see our general comment to all reviewers for the results and further discussion.

There is no analysis on the trade-off between performance gain by FuDD and additional (computational or financial) cost. Since generating pair-wise class description is more expensive than generating independent description for each class, such analysis is important.

The number of LLM queries directly depends on the number of ambiguous classes, which is a hyperparameter set by the user. In Figure 3, we report the performance gains with different numbers of ambiguous classes (i.e., different values for k). As discussed in Section 4.1, the five most ambiguous classes account for most of FuDD’s performance gains. As a result, to further decrease the costs, one can use smaller values for k, while preserving most of FuDD’s benefits.
Please also see our general comment to all reviewers for exact API costs and further discussion.

Thank you for your thoughtful comments on our manuscript. Below we address your comments and questions about our work.

The method is incremental, as it is a small trick to boost zero-shot image classification results.

We believe that our work contributes significantly to the field, especially as the need for efficient approaches to adapting VLMs to downstream tasks increases. We propose technical contributions that outperform previous methods, provide analysis of characteristics of effective natural language descriptions, study the utility of open-source LLMs and their fine-tuning, and contribute to answering open questions in the field on the potential of natural language class descriptions for VLM adaptation compared to other sources of information (e.g., synthetic images) or alternative approaches like soft prompt tuning.

Please see our general comment to all reviewers for an extended discussion on our contributions.

The baseline is not strong enough. It is therefore not clear if the method can still improve state-of-the-art models on these tasks.

To address the reviewer’s concern, we run additional experiments on two OpenCLIP model variants, which are stronger than their CLIP counterpart, as well as the BLIP2 model, which uses a different style of pre-training than CLIP. In both cases, we observe similar improvements, where the additional information provided by FuDD improves the performance over the naive LLM-generated descriptions.

Please see our general comment to all reviewers for additional experiments and further discussion.

More diverse and challenging tasks are missing from the experiment, e.g. image-text retrieval. It will be nice to show wider applicability of the method.

There are many interesting vision and language tasks that could benefit from additional semantic information. However, in keeping with most of the closely related work, here, we also focus on image classification, which remains a challenging and important problem (please refer to the manuscript for an extended discussion of previous work). Focusing on image classification allows more diverse comparisons and insights in this line of work, without confounding factors like task variation. For example, the comparison of image classification results in Section 6 uncovers the potential of natural language descriptions for VLM adaptation compared to other modalities or few-shot approaches proposed by previous work.

Although outside of the focus of our work, we would like to discuss FuDD in the context of information retrieval (IR). We draw an analogy between FuDD and reranking methods for IR, where for each query, a low-cost approach selects a subset of potentially related documents (similar to Section 3.2 in FuDD), and a more complex method reranks this smaller subset of documents (similar to Section 3.3 and 3.4 in FuDD). In the context of IR, because of the large number of documents, it is financially and computationally impossible to generate alternative descriptions for all documents regardless of choosing naive or differential descriptions. In such a setting, focusing on ambiguous documents is the only feasible option (we have extended discussion and experiments on ambiguous classes in Sections 3.2 and 4.1). Now the question is, "How effective are naive descriptions for reranking compared to FuDD?." Constrained by the rebuttal time window, we design new experiments to simulate text retrieval with our current descriptions and datasets. In these new experiments, we use test images as equivalent for queries and classes as the documents to be retrieved in a text-retrieval setup. We first retrieve the ambiguous classes as described in Section 3.2 and then use differential descriptions and naive descriptions to rerank these ambiguous classes and compare the results. We report the accuracy and nDCG@10, which represents the improvement in the ranking of the documents (i.e., class names). Compared to classification, it shows how much closer the true class label is to the top of the list, even if it is not the top prediction.

As expected, we observe that better representations for each class coming from differential descriptions actually improve the performance in this case, where both naive and differential descriptions are used for reranking.

Thank you for your thoughtful comments on our manuscript. Below we address your comments and questions about our work.

If this is the first paper presented in this field, I would rate a strong acceptance.

Our work is the first to propose a zero-shot, training-free approach for generating class descriptions that better differentiate between the target classes, as well as providing analysis of what the characteristics of such descriptions are. Previous zero-shot approaches dismiss the commonalities between classes, and the closest work that aims for differentiating descriptions is LaBo[1] which uses training data and further optimization to select a subset of naive class descriptions that are more differentiating.

The section 5: I wonder why would the Llama2 can generate informative sentences for satellite images? Especially given the model only been finetuned on the ImageNet dataset sentences.

We believe that during fine-tuning, the LLM learns what types of features are useful for differentiating the target classes, while the world knowledge comes from the pre-training stage. As shown in Figure 4 and Figure 5, after fine-tuning, the LLM is describing more low-level visual features instead of higher-level semantic concepts. As another piece of evidence in support of this argument, in Table 3, we observe that the correctness of the descriptions increases moderately (i.e., fine-tuning has limited impact on LLM’s world knowledge), while their helpfulness (i.e., being differentiating) increases significantly, which we attribute the increased performance to.

What if the caption is so long and so complex that the image-text model could not understand? I wonder if the author encountered this scenario, if so, how to address it?

That was exactly the case in our early experiments, where we got long, complex descriptions, sometimes expanding an entire paragraph. To alleviate this problem, we take advantage of the in-context learning capabilities of LLMs, and use two examples as part of our LLM prompt to guide the model to focus on concise visual features and follow the description style that is known to work well for VLMs, i.e., short photo captions that start with prefixes like "a photo of a."

Inspecting the generated descriptions across all datasets, we find that, on average, each description contains 14 words and 99 percent of the descriptions start with simple prefixes like:
"a photo of"
"a photograph of"
"a video of"
"a close-up of"
"a screenshot of"
"a close-up photo of"
"a microscopic image of"

Excluding the prefix and the class name, there are approximately 10 remaining words in each description, which does not allow for much complexity in descriptions.

Moreover, to keep the descriptions simple but at the same time benefit from a diverse set of described attributes, we guide the model in both the task instruction and the in-context examples to generate multiple descriptions, and then we average over the generated descriptions to compute the class representations. On average, over all datasets, we use approximately 4 descriptions per class.

[1] Yang, Y., Panagopoulou, A., Zhou, S., Jin, D., Callison-Burch, C., & Yatskar, M. (2023). Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR 2023.

Thank you for your thoughtful comments on our manuscript. Below we address your comments and questions about our work.

No significant method improvement

We respectfully disagree with the reviewer as our work makes technical contributions, achieves superior results, provides analysis on characteristics of effective natural language descriptions, studies the use of off-the-shelf and fine-tuned LLMs in the process, and positions natural language descriptions as a promising approach for adapting VLMs. This is especially important for future work as it advances the debate over promising approaches for VLM adaptation in favor of natural language descriptions compared to other sources of information like synthetic images or alternative approaches like soft prompt tuning.
Please find our extended response in the general comment to all reviewers for further discussion.

The paper focuses only on getting differential descriptions; however, it is not guaranteed that CLIP can accurately interpret them. The effectiveness of such models hinges critically on and is bounded by the vision-language models.

Here, we focus on generating class descriptions that provide the most useful information for the target set of classes and how we can use LLMs to do so effectively and efficiently. Of course, different VLMs use this information differently based on their capabilities and pre-training datasets. However, as shown in our experiments, VLMs are capable of taking advantage of this information to improve performance. Please see our general comment to all reviewers for experiments with additional VLMs, where we observe similar improvements. Clearly, VLMs are able to take advantage of this information. However, we believe an additional attempt to improve VLMs’ understanding of natural language is a different question and is out of the scope of this work.

[...] I suspect that this could be why the performance improvement is not very significant.

Respectfully, considering the challenging nature of the task, we believe our method makes significant improvements in most cases (e.g., 13.95 and 5.58 percentage points for EuroSAT and Oxford Pets). Especially, considering that without using any labeled data, FuDD’s performance as a zero-shot method is comparable to other few-shot learning methods that use labeled images for adapting VLMs.

[...] how many of the mistakes are due to insufficient / erroneous differential descriptions and how many of the mistakes are due to CLIP’s insensitivity to descriptions?

As an attempt to understand how the correctness and helpfulness (i.e., being differentiating) of descriptions impact the performance, we perform a manual evaluation as reported in Table 3, and observe that improving the helpfulness has a significant impact on performance even if the level of correctness of descriptions stays roughly the same. Another related experiment is described in Section 4.2, where we show that differentiating attributes are superior to generic attributes and that VLMs are sensitive to the described differences and can use them to their advantage.

As explained in Section 3.3, we use an ensemble of descriptions for each class, and often the ensemble is a mixture of helpful and correct and unhelpful or incorrect descriptions. As a result, it is challenging, if at all possible, to decide if the wrong prediction is caused by incorrect or unhelpful information or by the model’s inability to understand the correct and helpful descriptions. Therefore, this hinders such a precise analysis of the cause of the error for each image as suggested by the reviewer.

API cost for a single inference on an image

Please, find our response in the general comment to all reviewers.

For the equation in 3.2, what does the summation mean? [...]

As mentioned by the reviewer, this equation selects the k-most similar classes to each image. In other words, the argmax is over sets of classes, and to score a set, we have to sum the similarities of all classes in each set.

Would it be possible to give examples of differential descriptions for each of the evaluation datasets?

Thank you for your suggestion. We include examples of differential descriptions for all datasets in the updated manuscript. It is interesting to see that differential descriptions are more effective in cases that describe low-level visual characteristics than high-level semantic features, which is consistent with our previous observations described in Section 5.

Would it be possible to prompt the LLM to come up with differential attributes given >2 classes in a single prompt?

We think it is really interesting to investigate these different approaches, and we do not see an immediate reason that it does not work. Since this is the first work on generating differentiating, dataset-specific class descriptions, we focused on two classes per query. But, this is an interesting direction for future work.

Other VLMs

Multiple reviewers raised concerns about our choice of CLIP as our VLM. We emphasize that our main focus is on generating informative natural language descriptions that improve image classification.

To address reviewers’ concerns, we show that the generated descriptions provide useful information that is compatible with a wide range of VLMs. Here, we provide additional results on two OpenClip models[2], which are significantly stronger than their CLIP counterparts (e.g., 4 percentage points improvement on ImageNet with a single template compared to CLIP for ViT-L/14), as well as BLIP2[1], which uses an entirely different approach for training VLMs.

As shown below, the generated descriptions are useful for these models as well and further improve the performance of naive descriptions. Please see the updated manuscript for experiments on other datasets.

DComp is the checkpoint trained on the datacomp_xl_s13b_b90k dataset, and Laion is the checkpoint trained on laion2b_e16 [3].

[1] Li, J., Li, D., Savarese, S., & Hoi, S. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML.
[2] Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., … Jitsev, J. (2023). Reproducible scaling laws for contrastive language-image learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2818–2829.
[3] https://github.com/mlfoundations/open_clip

API Costs

Although FuDD queries the LLM more than naive approaches, the API calls are pretty affordable and do not hinder the wider adoption of FuDD. On average, one input prompt and model response are 380 and 199 tokens, respectively. With OpenAI pricing at the time of writing (0.001 and 0.002 dollars per 1k tokens for input and response), the cost is $0.78 per 1000 queries, leading to affordable prices as mentioned below (prices are in dollars).

FuDD also accommodates datasets with a large number of classes like ImageNet by recognizing the more significant role of ambiguous classes, reducing the costs for the ImageNet dataset from 388 to 60 dollars (See appendix B for details).

In addition, as studied extensively in our work, we can use off-the-shelf or fine-tuned LLMs like Llama 2 to generate differential descriptions using in-house hardware to avoid API costs or accommodate other issues like working with private and sensitive data.