Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP

Thanks for your helpful suggestions to improve our work. Below is our point-by-point response.

Whether the prompt aggregator can suppress noisy text prompts.

The attached Rebuttal.pdf (Fig. 1) visualizes the attention score for some prompt samples. We do observe low scores for those noisy or redundant prompts (see the example of car image), which will be suppressed during aggregation. We plan to add a larger-scale visualization and a correlation analysis between the attention score and image-prompt similarity.

Inference efficiency, FLOPs, evaluation time on ImageNet1k.

Please refer to our Response to Common Concern for comparisons and discussions of inference cost in terms of # params, GFLOP and FPS. AAPE evaluation on ImageNet1k takes a reasonable amount of time (12.6 min).

Application to more vision-language models, such as SigLIP.

We have shown the benefits of the language priors learned in our AAPE for two categories of vision-language models: 1) contrastive CLIP and 2) generative LiMBeR that connects the vision encoder with an LLM. For future work, we do plan to study more multimodal models to test the generality of AAPE, as mentioned in L318-320. We can go the contrastive route and apply AAPE to e.g. SigLIP as suggested. What's more interesting is the application to larger-scale generative models like BLIP-2 and LLaVA (work in progress) for general purpose visual and language understanding.

Thanks for your recognition of our work and the detailed feedback! We respond to specific comments below.

Does AAPE $h(x)$ mitigate the image-text modality gap?

Great question! Short answer is yes. Essentially, AAPE belongs to those text prompt tuning methods that only operate at the text branch, and one tuning signal is from some aggregation of external text knowledge. This sounds like we may risk breaking the image-text feature alignment after finetuning. However, AAPE is directly predicted from the image feature $x$ via our prompt generator $h(\cdot)$ (or an image-to-text mapping function) — AAPE does not involve separate token vectors as learned in CoOp and CoCoOp. Then $h(\cdot)$ can be viewed as a bridge between the image and text modalities, encouraging their alignment via an explicit feature mapping function. This way, our finetuning process in Fig. 2(b) can be re-interpreted from the perspective of modality gap mitigation: first we compute two text embeddings that are already image-aligned — $w_{i}$ (from frozen CLIP text encoder) and $h(x)$; they are then combined using a projection $g$ to compute an image-text alignment loss to further reduce modality gap.

Note our image-text mapping function $h(\cdot)$ is similar to that in MaPLe, only that MaPLe learns both image and text prompts (vs. our unimodal prompts). The key difference between the two methods is that our $h(\cdot)$ maps to some natural language equivalent while MaPLe does not introduce any language priors. This makes the text distribution for our finetuning not deviate too much from the free-form texts used for CLIP pre-training. As a result, our learned image-to-text mapping is expected to preserve CLIP's feature alignment level after finetuing. For empirical evidence, we compare the average cosine similarity score for the ground-truth image-prompt pairs on ImageNet. The higher the similarity score, the better the image-text alignment. We show AAPE scores 0.91, slightly higher than CLIP (0.89) which confirms our hypothesis. AAPE is also comparable to multimodal prompting method MaPLe (score 0.92), yet with language priors, AAPE achieves better classification accuracy and generalizes to more vision-language tasks.

Natural language may not be able to handle particular distribution shifts e.g. on Terra Incognita and WILDS datasets

Thanks for this insightful comment. We do rely on the language priors learned in AAPE to specify descriptive class details that are often invariant to image-based distribution shifts. Table 4 confirms our robustness across ImageNet variants. Intuitively, the description of "boxy shape and sloping roofline" for a "Jeep car" is useful to distinguish it from other similar classes, regardless of the style of input image, e.g., sketch images in ImageNet-Sketch or paintings in ImageNet-R. That said, we acknowledge that our model robustness (to domain shifts) depends on the descriptiveness of training image prompts and their coverage of discriminating image information in the considered domain. The suggested Terra Incognita and WILDS datasets are both good examples where one might need customized image prompts (preferably generated by LLM at scale) with domain knowledge, in order to generalize across e.g. different structural scaffolds of molecules in WILDS dataset. We leave such investigations of LLM prompting to future work, and will add the above discussions in main paper.

Ablation on distillation loss coefficient $\lambda$

The attached Rebuttal.pdf includes the ablation results in Table 2. It is shown that our AAPE performs robustly (with overlapping confidence intervals) when $\lambda \in [3,9]$, and still outperforms the strong baseline OGEN (H: 80.34) in this wide range of $\lambda$. By default, $\lambda=5$ is used for classification task.

L176-180: is it true that learning h(x) is more parameter-efficient than learning a sequence of token embeddings?

Sorry about the confusion. We meant for the purpose of augmenting $w_i$, how to efficiently learn an image-conditional embedding that is equivalent to a full prompt sentence. So the comparison is only under the context of learning sentence embedding conditioned on image $x$. We could simply generate a single embedding for the sentence of length $L$ based on $x$ using $h(x)$. Or we could generate $L$ embeddings for individual word tokens based on $x$. Obviously the token-wise prediction is much less efficient, since we need to train either $L$ conditional generator networks {$h_l$}$_{l=1}^L$ or a large-scale autoregressive network $h$. Will clarify in the main paper.

Thanks for the positive feedback and constructive suggestions. For the efficiency concern, please refer to our Response to Common Concern. Answers to your other questions below.

More details on the CLIP reward

As mentioned in L160, we use the same CLIP reward as in [17], formulated as $`CLIP-S`(x,p^a) = w \cdot \max(\cos(x,p^a),0)$ - will add in paper. This reward works well in our experiments. Studies of more advanced loss/reward functions are left for future work.

To show the benefits AAPE, use $h(x)$ only but no $w_i$ and $g$ for classification

In response, we have successfully applied AAPE $h(x)$ alone to the tasks of image-to-text retrieval, image captioning and VQA, which proves the efficacy of AAPE thanks to the distilled language priors in it. But for classification, it is not possible to only use $h(x)$ to fulfill the task. We argue that it's necessary to include $w_i$ and $g$ to build a well-functioning classifier.

Specifically, the image-conditional $h(x)$ can be viewed as an ``image captioning latent vector''. It does not necessarily encode the explicit class name, which prevents it from acting as a good text classifier for each class. Also, it's easy to imagine that directly matching the image features $x$ to $x$-conditional $h(x)$ would always lead to a high similarity score, which is not discriminative. Therefore, we start with a basic template ($w_i$) that allows us to manually encode the $i$-th class name. Then we combine $w_i$ with $h(x)$ so that $h(x)$ can provide extra class descriptions that are adapted to input image. For combination strategy, we found it not working when linearly combining $w_i$ and $h(x)$ based on element-wise addition or using a linear projection $g$, since that tends to ignore $h(x)$ if we match the linear combination to $x$ for classification (see L187-188). Hence we choose to use a nonlinear $g$ on top of the concatenation of $w_i$ and $h(x)$ for their non-trivial fusion. Because of such design choices, to isolate the impact of natural language priors under the classification setting, we always use $w_i$ and $g$ and compare $h(x)$ learned with and without the language prior distillation loss. Figs. 3-5 all confirm the positive contributions of language priors. Will add the above clarifications in paper.

Thank you for your response. Most of my concerns have been addressed. I have reviewed all the reviews and rebuttals.

However, I still have a few questions:

• What is the computational cost during the training phase?
• I'm still curious about the performance of the proposed method with no $w_i$ and $g$ on the classification task. I think that the proposed method can perform well on at least base (seen) classes without using $w_i$ and $g$.

Thanks.

Compare with ArGue (CVPR 2024)

ArGue and our AAPE both learn text prompts that distill language priors from LLM. However, the prompt learning mechanisms are different: ArGue learns individual prompt token vectors. They are combined with the embeddings of the class name and class-wise visual attributes generated by LLM. On the other hand, we learn to directly generate the embedding of a full prompt sentence which we call AAPE, supervised by both LLM-generated natural language prompts and task loss. Note our AAPE prediction is conditioned on the image feature $x$ via a prompt generator $h(x)$.

We share the following benefits:

• 1. Our input-conditioning mechanism focuses on extracting prompt features from the input image, while ArGue learns unconditional text prompts for each class. Without considering input information, ArGue has a higher risk of overfitting to training class distributions. 2. Our image-conditional prompt generator $h(x)$ can be viewed as an image-to-text mapping function, which helps to bridge the modality gap between image and text feature spaces. This further gives rise to improved generalization during prompt tuning.
• Here we provide empirical support for AAPE's good generalization on the few-shot classification task. In the base-to-new class generalization setting, we have comparable Base/New/H class accuracy averaged on 11 datasets: ArGue-N (83.77/78.74/81.18) vs. AAPE (84.72/77.54/80.97). While in the domain generalization setting, we have better accuracy for ImageNet/-V2/-Sketch/-A/-R: ArGue-N (71.84/65.02/49.25/51.47/76.96) vs. AAPE (73.56/65.97/50.12/51.62/77.52).
• 3. More importantly, our AAPE is designed to be a universal embedding directly applicable to various vision-language tasks. For example, in our text retrieval, image captioning and VQA experiments, AAPE $h(x)$ is used as a standalone ``image captioning latent'' that achieves SOTA performance (Table 2 in main paper). This is not possible with ArGue.

Compare with latest methods like BLIP-2 for the image-to-text retrieval task

As mentioned in Section 5.1 and Appendix B, we treat the image-to-text retrieval task as a small-scale proof of concept of our prompt learning method. Specifically, we learn AAPE on COCO dataset only, and test its zero-shot/finetuning performance on Flickr30k. The goal is not to push for SOTA performance, but to verify if the learned AAPE can successfully distill text knowledge from COCO captions which can be useful for downstream text retrieval.

Table 3 provides supportive results as AAPE obtains strong performance with both zero-shot and finetuned models on Flickr30k. What's most interesting is that our zero-shot Flickr30k performance is nearly on par with that of SOTA zero-shot models, including CLIP/SigLIP and the latest work Llip. This proves good generalization of AAPE. We will add results of more recent/competitive methods like BLIP-2 as suggested. But note this won't be a fair comparison since we only train on COCO while other methods like BLIP-2 are trained on billions of image-text pairs.

In the future, we plan to scale up training data to learn universal AAPE. This not only enables fair comparisons with SOTA multimodal models for image-to-text retrieval, but also on more complex vision-language tasks e.g. using AAPE+LiMBeR which proves promising already.

AAPE overfits base classes, with lower performance on novel classes; poor generalization on unseen data

We humbly argue that overall, AAPE improves generalization over unseen classes and data distributions. We acknowledge there exists a gap between the base and new class accuracies (7.18 points averaged across 11 datasets) for AAPE's few-shot classification experiments. One reason is that our prompt generator $h(\cdot)$ won't be trained sufficiently well on some datasets with limited class data, while we rely on a good $h(\cdot)$ to provide useful language priors for new class generalization. For example, DTD and EuroSAT datasets have only 47 and 10 training classes respectively (16 shots per class), and their base-new accuracy difference is high (20.33 and 19.10 points). While the SUN397 dataset has 397 classes (hence better learned $h(\cdot)$), and the base-new accuracy gap is reduced to 3.06 points. This motivates us to scale up training data in real-world tasks to learn universal prompt generator that generalizes over unseen categories.

Nevertheless, under the few-shot classification setting, our AAPE learning does not significantly widen the average base-new accuracy gap (7.18) in comparison to other methods, e.g. the competitive OGEN (gap 7.31) that doesn't leverage language priors. With the language priors, AAPE can actually improve both the base and new class accuracies for each of the 11 datasets (see Fig. 3), sometimes by a large margin. When well trained on ImageNet, AAPE also achieves SOTA results for domain generalization (Table 4). Furthermore, we have successfully trained a competent $h(\cdot)$ on COCO to describe complex scenes. It shows impressive zero-shot generalization to multiple tasks with varying data distributions, including image-to-text retrieval (on Flickr30k), image captioning (on NoCaps) and VQA (on VQA2).

Fig 1: what is the input to GPT-3? Why does the last sentence of the description contain "the image is..."?

As mentioned in L120-127, we follow the CuPL method to query GPT-3 with more than one LLM-prompt templates (i.e., GPT-3 inputs). Examples are "How can you identify a(n) {}?" and "Describe an image from the internet of a(n) {}". The latter example leads to the GPT-generated image prompts that start with "this image is...". Will clarify in the paper.

Where are human-generated image captions obtained from?

As mentioned in L128-136, we use human-annotated captions (5 per image) from COCO dataset to represent complex scenes. We train on COCO captions for 3 vision-language tasks, i.e., image-to-text retrieval, image captioning and VQA.

Thank you for the constructive feedback on our work. Regarding the efficiency concern, please refer to our Response to Common Concern. For other comments, our point-by-point response is as follows.

Compare and discuss how this work differs from ProText (arXiv:2401.02418)

Thanks for bringing this paper to our attention! ProText and our AAPE both learn text prompts that distill natural language priors from LLM, with the same goal of improving generalization to novel classes or data distributions. However, the prompt learning mechanisms are different: ProText learns individual prompt token vectors. They are then combined with a text template to generate the embedding of a full prompt sentence, which stays close to those LLM-generated sentences. On the other hand, we learn to directly generate the prompt sentence embedding which we call AAPE, supervised by both LLM-generated prompts and task loss. Note our AAPE prediction is conditioned on the image feature $x$ via a prompt generator $h(x)$.

We share the following benefits:

• 1. Our input-conditioning mechanism focuses on extracting prompt features from the input image, while ProText learns unconditional text prompts for each class. Without considering input information, ProText has a higher risk of overfitting to training classes (thus worse generalization to new classes). 2. Our image-conditional prompt generator $h(x)$ can be viewed as an image-to-text mapping function, which helps to bridge the modality gap between image and text feature spaces. This further gives rise to improved generalization during prompt tuning.
• Here we provide empirical support for AAPE's better generalization on the few-shot classification task. In the base-to-new class generalization setting, we have the average Base/New/H class accuracy on 11 datasets: ProText (72.95/76.98/74.91) vs. AAPE (84.72/77.54/80.97). While in the domain generalization setting, we have the accuracy for ImageNet/-V2/-Sketch/-A/-R: ProText (70.22/63.54/49.45/51.47/77.35) vs. AAPE (73.56/65.97/50.12/51.62/77.52).
• 3. More importantly, our AAPE is designed to be a universal embedding directly applicable to various vision-language tasks. For example, in our text retrieval, image captioning and VQA experiments, AAPE $h(x)$ is used as a standalone ``image captioning latent'' that achieves SOTA performance (Table 2 in main paper). This is not possible with ProText.

Will add the above discussions and comparisons in revised paper.

Is the projection $g$ necessary for classification? Is $g$ used for VQA/captioning tasks?

In response, we apply AAPE $h(x)$ alone to the tasks of image-to-text retrieval, image captioning and VQA, which proves the efficacy of AAPE thanks to the distilled language priors in it. But for classification, it is not possible to only use $h(x)$ to fulfill the task. We argue that it's necessary to include both $w_i$ and $g$ to build a well-functioning classifier.

Specifically, the image-conditional $h(x)$ can be viewed as an ``image captioning latent vector''. It does not necessarily encode the explicit class name, which prevents it from acting as a good text classifier for each class. Therefore, we start with a basic template ($w_i$) that allows us to manually encode the $i$-th class name. Then we combine $w_i$ with $h(x)$ so that $h(x)$ can provide extra class descriptions that are adapted to input image. For combination strategy, we found it not working when linearly combining $w_i$ and $h(x)$ based on element-wise addition or using a linear projection $g$, since that tends to ignore $h(x)$ if we match the linear combination to $x$ for classification (see L187-188). Hence we choose to use a nonlinear $g$ on top of the concatenation of $w_i$ and $h(x)$ for their non-trivial fusion.

We acknowledge that $g$ introduces more parameters for prompt learning. One might wonder whether the increased model size contributes significantly to our classification accuracy gains. To this end, we compare with a variant of the CoCoOp method that similarly conditions prompt generation on input image. Our implemented CoCoOp variant has a $g$ attached after the text encoder, such that its total parameter count is similar to ours. We show such CoCoOp variant achieves an average H of 75.32 on 11 datasets, even worse than that of projection-free CoCoOp (75.83) mainly because of the worse generalization on new classes. This indicates that adding projection $g$ with increased capacity does not necessarily mean higher performance for few-shot classification. On the other hand, our similar-sized AAPE with $g$ achieves H 80.97, which benefits a lot from the learned language priors. Figs. 3-5 all confirm the positive contributions of language priors in AAPE. Will add the above clarifications in paper.