ICE: Image-Caption Encoding for Improved Out-Of-Distribution Generalization In Vision-Language Models

Comment: There is no comparison to other ensembling techniques that could provide diversity. While the motivation behind using captions to improve the OOD is interesting, the improvements from the model are small which raises two questions - a) are the captions generated the main problem, or b) are the way they are used to correct the prediction? A more descriptive SOTA captioning model would help in answering the first question and hence lead the way to design better ensembling techniques.

Response: Our method is complementary to other ensembling methods, such as ensembling image augmentations or ensembling multiple pretrained models, because we leverage a different source of diversity. From our experiments, we believe that we have performed a comprehensive analysis of our method stacked on top of prior SoTA few-shot OOD methods. As an aside, we claim ICE to be different from other ensembling techniques because we do not use other independently trained models as part of our predictions; we specifically take advantage of captioning properties inherent in the CoCa architecture to better guide downstream classification.

To address whether the captions being generated are the main problem or the way they are being used: the BLIP experiment above confirms that merely more "descriptive'' captions do not increase the zero-shot accuracy. However, this was likely because BLIP-2 captions do not contain more information relevant to the classification problem. We believe that captions which contain more specific information about objects in the image could increase the performance of zero-shot ICE. Hence, automated prompting of large VL models like BLIP-2 to elicit more specific information would be an interesting direction for future work.

Comment: As discussed in the limitations, the method relies on captions providing useful supplementary information. Generating captions from a stronger vision-language model (such as BLIP) and then combining those captions with image predictions could help to make caption selection more robust.

Response: Thank you for your suggestion, this is an interesting topic of investigation. Here are some supplementary results on ImageNet and 10 other datasets using BLIP-2 captions instead of CoCa captions:

We use the largest available BLIP-2 model, which contains a ViT-g image encoder with a FlanT5-XXL LLM. We observe a similar accuracy gain when applying ICE with these BLIP-2 captions. BLIP-2 captions achieve slightly lower numerical results than CoCa captions on average, despite using a much larger model. We think this is because BLIP-2 sacrifices fine-grained information in favor of a more descriptive caption. For example, for one image in the Pets dataset, the CoCa caption is: "a photo of a abyssinian cat'', while the BLIP-2 caption reads: "a photo of a cat standing on a table''. The CoCa caption gives us the correct classification for the pet in the image, while the BLIP-2 caption describes the scene in general terms without providing the information necessary for pets classification. These results illustrate that simply using a better captioner does not necessarily lead to better ZS accuracy under our ICE framework. However, we believe that using LLM prompting techniques to elicit more fine-grained information from large captioning models and incorporating that information into ICE is an interesting direction for future work. We will include our analysis with BLIP in our revision as part of the supplementary material.

[blip2] Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." arXiv preprint arXiv:2301.12597 (2023).

Comment: Determining the optimal weight between image and caption probabilities seems challenging. Would a learning-based approach that adaptively learns weights for each branch work better? Exploring other ways to set this weight adaptively could strengthen the approach and give more insights into how failure can be handled.

Response: Thank you for your suggestion, this is an excellent point. We emphasize that ICE is a zero-shot method, i.e. we assume no additional training or access to test-time information. However, if we relax the constraints to allow for some learning, there can definitely be room for improvement.

One method we attempted was adaptively learning the value of $\xi$ as a function of image and caption embeddings. $\xi$ determines the weight between the image and caption predictions according to Equation (3). We use a simple Multi-Layer Perceptron with ReLU activations and sigmoid output to predict $\xi$ and train the network on few-shot ImageNet data. We use the same $\xi$ prediction network for all datasets at test time. The results are provided in the tables below:

We observe that the adaptive $\xi$ improves accuracy on all the ImageNet datasets, including both the ImageNet validation set and the domain-shifted versions. This improvement is relatively small due to the limited capacity of the $\xi$ network, but it is consistent. This is a promising result that demonstrates that the $\xi$ can be learned. The adaptive $\xi$ degrades the accuracy on some datasets due to the large label-space shift relative to ImageNet classes (which the $\xi$ network was trained on). However, the promising results on the domain generalization targets could warrant further work. We welcome further discussion on alternative methods to find better weights between the image and caption embeddings.

Thanks to the authors for the detailed response and new experimental results. My main concern is regarding the new BLIP-2 experiments as it does not lead to improvements in performance. As also pointed out by reviewer oC1d, this could be due to the caption itself or LLM prompting. As for fine-grained classes it is hard to get captions that are useful for improving classification accuracy, this method will fall short. Maybe combining this with other sources of knowledge and such would lead to better gains

Given the response from other reviewers and the limitations of the method, I would like to keep the borderline score.

Comment: The paper should also discuss in more detail the applicability of the proposed approach to video data.

Response: Thank you for your suggestion, video data is indeed an important component of the original CoCa paper (Yu et. al., 2022). One of the limitations of our method is that it requires additional forward passes on the base model to generate captions, creating a time bottleneck. Thus, it would be difficult to scale ICE to video data where we must perform online predictions on a stream of data. However, we note that our current focus is to provide gains for zero-shot image classification, and we believe that our method provides a novel perspective on how to correctly pivot toward the correct class. Future work can focus on reducing the computational load of our method. For example, one potential direction is tuning the captioner to give more descriptive captions (e.g. color of the image, textures, etc.) rather than directly state the objects in the scene. If we can generate an informative caption in the first pass, we can reduce the number of captions needed for ICE. Another potential direction is detecting change in video data, and only regenerating captions when necessary.

We were not able to perform video classification experiments in the short span of the rebuttal period, but we propose the following more scalable way of extending ICE to video recognition. Captioning each frame of a video is impractical, since tokens must be generated one after another. However, we can consider captioning multiple frames at a time. The CoCa decoder takes as input a spatial token map from the image encoder. We could pool the spatial tokens from multiple video frames before sending them to the text decoder. The authors of CoCa already proposed an attention pooling mechanism to ensemble the spatial tokens from different frames; we could leverage this to generate a video caption. We can then use ICE to aggregate the predictions based on the pooled video image embedding and the generated caption. We welcome additional discussion for how our approach can be extended toward video data.

Comment: As the approach looks only at the Top-K classes, its performance is inherently limited. Is the proposed approach helpful in case the correct class is not within the initial Top-K predictions?

Response: Correct, the proposed method only allows for pivoting within the Top-$K$ predictions, which means that we are unable to correctly reclassify to far-away classes. However, we believe that this is a reasonable assumption to make, since we are motivated by the observation in Figure 2a that in most cases, the correct class can already be found in the Top-$K$ predictions. Thus, in those cases, we wish to provide additional information to the model to pivot toward the correct class.

Comment: A detailed analysis of inference speeds with and without the use of the proposed ICE method would be helpful.

Response: Thank you, we provide a supplemental analysis on inference speed in this response. This information is provided in the following table. We use a V100 GPU with batch size 32, no gradient tracking, image size 224x224, and the CoCa ViT L/14 model.

ICE evaluation is indeed slower than zero-shot inference with only image embeddings, due to the additional time for generating captions. However, we emphasize that our method does not focus on inference speed, but rather demonstrating a simple method to improve zero-shot accuracy by incorporating captions into the prediction. It is important to weigh the gains in zero-shot accuracy against the additional inference time. Furthermore, there is extensive work in the area of fast LLM inference we can leverage to speed up caption evaluation (such as quantization methods like [smoothquant]). We can also use simple tricks such as decreasing the number of tokens in the generated caption. This is outside the scope of the current work.

[smoothquant] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." International Conference on Machine Learning. PMLR, 2023.

Comment: The paper should discuss in more detail in which scenarios ICE provides a performance boost, as in the case of many datasets, e.g., Caltech, Food, SUN, and UCF, the performance improvement is less than 0.1%.

Response: We acknowledge that ICE provides smaller gains on several datasets for few-shot classification in Table 2, especially for domain generalization. However, we first note that ICE is a zero-shot method, and thus our gains of 0.5% on average in Table 1 better reflect the capability of our approach. We include supplementary few-shot training results in Table 2 to show that even when deploying a zero-shot method to a few-shot context, we are able to enjoy gains of 0.5% on average on cross-dataset generalization. Furthermore, on several of the datasets where we fail to see gains, we note that there was little room for improvement to begin with. This can be observed in the table we display here. One hypothesis that can be made is if the classification accuracy is already very high, ICE may underperform because it is more likely to incorrectly reclassify a previously correct sample than vice versa.

We would like to highlight that ICE achieves a 2% gain over few-shot baselines, and a 3% gain over zero-shot baselines on the aircraft dataset. This is likely because aircraft classification requires attending to details in the image (such as wingtip styles, numbers on the fuselage etc.). Traditional zero-shot CLIP uses the global image feature for classification, which may ignore these details. On the other hand, the CoCa decoder considers all spatial image tokens. Thus the caption is better at capturing local details. We can clearly see this when asking CoCa to describe an image with writing in it, since the resulting caption usually faithfully transcribes the writing. Therefore, it is intuitive that using ICE to take into account caption information would be beneficial to classification problems where discriminative visual attributes are small. We will include this additional analysis in our revision as part of the supplementary material.

Comment: In-domain performance: the proposed method is evaluated primarily on cross-dataset and domain generalization settings. However, the in-domain performance, e.g., on ImageNet, is not evaluated (cf Figure 4 in CoCa).

Response: Thank you for pointing this out. We have conducted an in-domain evaluation of ImageNet (labeled as INet) in Table 1 and Table 2. We will reorganize our results before publishing to clarify any confusion.

Comment: Can authors explain why CoCa is used, instead of an arbitrary caption model? If we replace CoCa with SoTA caption model, will the performance be significantly improved?

Response: Good question. We would first like to reiterate our rationale for choosing CoCa as our base model; then we will present supplementary results with the SoTA BLIP-2 Vision-language (VL) model.

In the context of ICE, CoCa has the following nice properties as compared to other VL models:

• CoCa is optimized to perform zero-shot classification (same as image-to-text retrieval), while many SoTA VL models such as BLIP-2 and Flamingo place more emphasis on generating human-like responses to queries. Consequently, SoTA VL models do not output a caption that is fine-grained enough for certain classification tasks, at least not without additional prompting.

• CoCa contains a text encoder, not just a text decoder. This allows us to efficiently perform zero-shot classification with just one model.

We present results on ImageNet and 10 other datasets using BLIP-2 captions instead of CoCa captions in the following table.

We use the largest available BLIP-2 model, which contains a ViT-g image encoder with a FlanT5-XXL LLM. We observe a similar accuracy gain when applying ICE with these BLIP-2 captions. BLIP-2 captions achieve slightly lower numerical results than CoCa captions on average, despite using a much larger model. We think this is because BLIP-2 sacrifices fine-grained information in favor of a more descriptive caption. For example, for one image in the Pets dataset, the CoCa caption is: "a photo of a abyssinian cat'', while the BLIP-2 caption reads: "a photo of a cat standing on a table''. The CoCa caption provides the correct classification for the pet in the image, while the BLIP-2 caption describes the scene in general terms without providing the information necessary for pets classification. These results illustrate that simply using a better captioner does not necessarily lead to better ZS accuracy under our ICE framework. However, we believe that using LLM prompting techniques to elicit more fine-grained information from large captioning models and incorporating that information into ICE is an interesting direction for future work. It would be very helpful for us to hear your thoughts on our analysis here.

[blip2] Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." arXiv preprint arXiv:2301.12597 (2023).

Comment: The performance gain in the few-shot setting seems marginal (Table 2). Does this mean that in the few-shot adaptation setting, we may just need a few images, instead of using an additional decoder?

Response: Thank you for pointing this out. From Table 2, we observe that the Domain Generalization (DG) targets have the lowest average gain; the Cross-Dataset (CD) Evaluation targets still enjoy 0.5% performance boosts on average. However, we still see consistent improvements on hard DG datasets such as ImageNet Adversarial (INet-A) and ImageNet Rendition (INet-R). We note that similar trends can be found in the zero-shot data in Table 1, and thus believe that it is more important to evaluate the importance of using an additional decoder based on the properties of the target dataset (as elaborated in Section 3.4 Caption Properties). In addition, we note that ICE is designed to be a zero-shot method, making Table 1 results more relevant to our target domain, but we think it is promising that ICE can provide benefits even in a few-shot context.

Thank you for providing detailed responses and the additional experiments with BLIP-2.

It is interesting to see that a better captioning model such as BLIP-2 does not necessarily lead to superior zero-shot classification performance. It remains a bit unclear to me if is it because the fine-grained information from large captioning models is not accurate or the language encoder cannot properly handle such information. This brings some further questions such as what properties are desirable from image captions for zero-shot visual classification and how to better exploit a general caption model beyond utilizing a specific pre-trained model such as CoCa? For example, does BLIP-2 really do not understand what an abyssinian cat is? I appreciate the authors' efforts in showing this addition results and it is a good starting point for answering one of the core scientific questions of this work.

I understand that the motivation for using CoCa is that it allows authors to "efficiently perform zero-shot classification with just one model". However, this framework can be interpreted as both an advantage (simplicity) and disadvantage (e.g., post-hoc adjustment, strong reliance on CoCA and therefore would be difficult to generalize to different domains or general visual reasoning tasks).

As for efficiency, the proposed method uses three components: image encoder -> text decoder -> text encoder (as shown in Figure 3). Alternatively, if we use simple vision-language models such as LLaVA [1], we still have the same number of components: image encoder (in LLAVA) -> text decoder (in LLAVA) -> text encoder (in CLIP). Therefore, it might be hard to argue which one is more efficient.

Given the above considerations, other reviewers' comments, and the empirical performance, my judgement for the work remains borderline (either marginally below or above the acceptance threshold).

[1] Liu et al., Visual Instruction Tuning, NeurIPS 2023

Comment: Image captioning poses a greater challenge ...

Response: Thank you for providing valuable insight into this problem. We agree that image captioning poses a greater challenge than image classification, and captions can describe several objects in the scene that may conflict with the single-label classification objective. We first note that if multiple objects exist in the image which correspond to different labels in the dataset, then it is likely a poor problem formulation for single-label image classification to begin with. Thus, dealing with these conflicting scenarios would be out of the scope of our method. Second, our method works only to pivot toward the correct class in the top-$K$ predicted classes. This means that even if the caption provides descriptions about unrelated objects, we can only select a class that was already predicted with high probability by the original image encoder. The combination of our image and caption embeddings ensures that we will not stray too far from the initial prediction for image classification which provides a useful constraint. Finally, Tables 1 and 2 show that our method can give good zero-shot improvements to existing state-of-the-art models.

Comment: In Section 4.3, "Understanding Why ICE Provides Improvements," the explanation is merely qualitative, based on just four examples. It's difficult to gauge the representativeness of these cases. A deeper quantitative analysis—for example, determining how common these cases are in standard image datasets—would be necessary to fully understand why and how ICE is effective.

Response: Thank you for suggesting a quantitative analysis of the four situations presented in Section 4.3 and Fig. 5; we agree that this enhances our discussion. Accordingly, we provide the quantity of images in each category as a percentage of total test images in the tables below, for each dataset. In all the datasets, most images are either correctly predicted by both methods or incorrectly predicted by both methods. This is expected, since every dataset contains a large amount of easy images and a large amount of impossible images given a constant model capacity. However, we do observe that in all datasets except Caltech and Pets, the percentage of images where ICE successfully reclassifies an initially incorrect prediction exceeds the percentage of images where ICE incorrectly reclassifies an initially correct prediction. Furthermore, the percentage of failed re-classifications is small when compared to the percentage of successful re-classifications on datasets such as DTD (0.5% compared to 3.1%) and EuroSAT (0.8% compared to 3.4%). Finally, we note that images incorrectly classified by ICE may be images that are hard to classify without additional context or ambiguous images. Future work can help identify which samples to apply ICE to decrease the number of correct-to-incorrect ICE re-classifications, and we welcome further discussion about alternative directions. We will include this additional analysis in our revision as part of the supplementary material.