AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models

We thank the reviewer for the constructive review and helpful feedback. We appreciate that the reviewer finds that "The idea and motivation of AutoCLIP make sense", "it is easy to apply AutoCLIP to existing zero-shot classifiers for improving accuracy", "the experiments are thoroughly conducted covering many classification datasets and ranging from CLIP to CoCa.", and "the paper presentation is clear for audience.". We provide responses to the reviewer's questions and critical remarks below:

The is one limitation: Current experiments suggest AutoCLIP can only be used for zero-shot classification, which is only one useful aspect of large vision and language model. Large vision and language model like CLIP is not about just doing zero-shot classification. For real applications, the impact of these models also lies in downstream tasks. For example, finetuning from CLIP pretrained parameters or using CLIP to directly assist the downstream tasks. It's better to explain some zero-shot classification applications. In view of the above limitations, another concern comes out about the result. If a model has 1% improvement on ImageNet, it may bring more improvements when the pre-trained model is applied in downstream task. In case AutoCLIP is not intended for downstream task, the 1% improvement may look not significant. What this 1% improvement can bring?

In this work, we do focus on zero-shot classification. We are aware that this is only one potential usage of VLMs. However, it is an important enough use-case that several prior works have focused on it, such as for instance DCLIP (Menon & Vondrick, 2022) and WaffleCLIP (Roth et al., 2023). We also think that an improvement by 1 percent point accuracy is significant, particularly given that it comes without any retraining or custom prompt engineering by a pure change in the way the zero-shot classifier is constructed. However, we will further motivate zero-shot classification applications in the introduction as proposed by the reviewer.

The experiment in fig.3 is not clear. What does this experiment verify for? Any explain why ViT-L is worse? Does it mean AutoCLIP cannot well be scaled to larger models?

Figure 2 provides ample evidence that AutoCLIP also brings benefits for larger models (ViT-L-14s) on multiple datasets. Figure 3 shows that in the specific combination of large models and image corruptions, AutoCLIP does not bring benefits compared to baseline zero-shot classifiers. But still, for 3 out of the 5 VLMs in Figure 3 (and averaged across models), AutoCLIP is better than the baseline zero-shot classifier.

It is highly suggested to show evidence for "This is inspired by the observation that class descriptors having higher similarity in the embedding space describe the image better". This claim support the rationale of the proposed AutoCLIP. To my understanding, if this claim does not hold, then AutoCLIP does not hold.

We provide evidence for this in the paper in Figure 6 and in Figure 7 (in the appendix): Figure 6 shows that on images from the Food dataset, prompts starting with “A photo of...” get higher weights by AutoCLIP than prompts starting with "Art of ...", while on ImageNetR in Figure 7, it is the other way around. This corresponds to the Food dataset containing actual photos while ImageNetR containing more artistic renditions etc. Thus higher weights correspond to better class descriptor-image fit.

Is there any speed comparison? How much more inference time does the AutoCLIP need?

We provide some numbers for the case of a ViT-L-14 on the Oxford Pets dataset. Here, encoding an image takes 12.64ms on a V100 (minimum over 100 images). The baseline "averaging" zero-shot classifiers takes additional 0.08ms (average over 640 samples) on top to classify a sample. AutoCLIP takes additional 1.54ms (average over 640 samples) for classification when running bisection for autotuning the step size. For a fixed step size, the overhead of AutoCLIP is only 0.45ms. Thus, AutoCLIP raises inference from 12.64ms to a bit more than 14ms. In contrast, TPT (Shu et al., 2022) and RLCF (Zhao et al., 2023), which did not report compute or memory requirements, require encoding multiple image augmentations. TPT states "We augment a single test image 63 times using random resized crops and construct a batch of 64 images, including the original one.", which means that the image encoding time (for instance the 12.64ms from above) is increased by a factor of 64x, plus additional overhead for backpropagating through the text encoder, which likely brings the inference time per sample close to 1s. RLCF also requires multiple image augmentation but does not state the number of augmentations used. In contrast, AutoCLIP does not require any image augmentations or backpropagating through the text encoder and increase inference time from 12.64ms to ~14.0ms instead of ~1 second.

We thank the reviewer for the constructive review and helpful feedback. We appreciate that the reviewer considers the proposed AutoCLIP a "simple and effective method for increasing the model performance." and highlights that "Extensive experiments on multiple datasets [...] highlight the performance of the proposed method.". We provide responses to the reviewer's questions and critical remarks below:

The main concern of this work is its novelty. The idea of weighing the text features from different prompts has been proposed [a]. It is unclear why is the proposed method different than [a].

We thank the reviewer for bringing the work [a] to our attention; we will include a discussion of it in the related works in the next revision of our paper. We note that ZPE (the method from [a]) has more restrictive prerequisites than our AutoCLIP: The "max logit scoring" of ZPE determines prompt weights based on logit statistics in the target domain, which must be computed from multiple images from the target domain. This limits ZPE to operate only with a batch of images instead of a single image. Additionally, the "logit normalization" of ZPE requires availability of image features which represent the feature distribution in the pre-training domain, that is: ZPE with logit normalization is not "source-free". In contrast, our AutoCLIP requires no additional statistics so that we can do the inference with AutoCLIP in a source-free setting from a single input image alone. We note that ZPE and AutoCLIP are orthogonal and could be combined: first normalize logits with ZPE and then construct a auto-tuned zero-shot classifier with AutoCLIP on top.

While adopting the weighting strategy in [a] during test time is a solution for boosting the classification performance, one could simply average the topK text prompts that have the topK cosine similarity with the image feature. Is the proposed method better?

Thanks for proposing TopR as a baseline (we denote the number of selected queries by R to avoid confusion with the number of total prompt templates that we denote by K in the paper). It is indeed a stronger baseline than max or mean aggregation for suitably chosen R (we note that choosing R is non-trivial in a zero-shot setting). We have run an additional study comparing AutoCLIP against this TopR-CLIP for $100$ DCLIP prompt template, across the same VLMs and datasets as in Figure 2. We provide results for different choices of R, both in terms of the percentage of settings in which AutoCLIP is better as well as the Δ Accuracy compared to TopR-CLIP below.

We note that even for the optimal choice of R (R=20), which would require tuning R on labeled data, AutoCLIP is better than TopR-CLIP in 86% of the cases. We will include a more detailed visualization of this study in the appendix of the revised version of the paper.

The author is suggested to put the argument explanation of scipy.optimize.bisect to the appendix

Thanks for the suggestion, we will do so.

The x-axis of Figure 2 is not clear. Why are there so many lines on the x-axis? Are they meaningful?

These are just the default ticks of matplotlib for a logarithmic x-axis. We did evaluate for $K \in$ {2, 4, 10, 20, 50, 100, 200, 400, 1000} prompt templates in this figure.

Unclear: “from ∆ = 0.06 for K = 4 over ∆ = 0.33 for K = 10 and ∆ = 0.49 for K = 50 to ∆ = 0.57 for K = 200”

Here K denotes the number of prompt templates sampled from CLIP/DCLIP/WaffleCLIP and ∆ the average increase in accuracy when using AutoCLIP compared to the baseline "averaging" zero-shot classifier. Did this explanation clarify the statement?

Thanks to the authors for providing the rebuttal. I carefully checked all the reviews and rebuttals. The authors clarified the novelty of the proposed method compared to [a] and provided additional experiments on the suggested TopR baseline. As shown in Figure 9, AutoCLIP generally performs better than this baseline. This addresses my concerns, and I am prone to increase my rating after the rebuttal.

We thank the reviewer for the constructive review and helpful feedback. We appreciate that the reviewer finds our work has "good writing and [is] easy to understand", our method having "intuition [that] is also clear and provides significant progress on the zero-shot image classification". We provide responses to the reviewer's questions and critical remarks below:

Even though the proposed method is simple and effective, the method looks like a naive modification method to replace the uniform-weighted average descriptor encodings with weighted average encodings based on the existing algorithms which may limit the paper's novelty.

We agree that AutoCLIP is simple and effective; however, we do not think AutoCLIP is naive since it is a principled solution (gradient-based maximization of an objective) based on a clear intuition: prompt templates resulting in class descriptors more similar to the image on average should get higher weights. Being a simple method is in our opinion a feature not a bug of our contribution.

I'd like to know if the order of the queries is different, will the performance remain the same?

AutoCLIP is by design permutation-invariant, that is: it treats queries as a set and order of queries does not matter. We also run an empirical sanity check and randomly permuting queries did not change performance.

If each query has different classes in the inference time, will the weighted average encodings need to be changed/calculated all the time?

We interpret the reviewer's question that the reviewer likes to know whether the weights can be reused. In fact, the weights in AutoCLIP are recalculated for every sample during inference. If queries or classes change at inference time, the main overhead would be that the class descriptors need to be re-encoded by the VLM's text encoder. This, however, is the case for standard CLIP zero-shot classifiers as well and thus no overhead specific to AutoCLIP.

Similarly, will the proposed method still have the same speed in the inference process or it will be dramatically different?

AutoCLIP zero-shot classifiers have overhead at inference time compared to standard zero-shot classifiers that simply average encoded class descriptors. However, this overhead is still very small compared to the encoding of the image with the VLM's image encoder in most cases. For instance, in the case of a ViT-L-14 on the Oxford Pets dataset, encoding an image takes 12.64ms on a V100 (minimum over 100 images). The baseline "averaging" zero-shot classifiers takes additional 0.08ms (average over 640 samples) on top to classify a sample. AutoCLIP takes additional 1.54ms (average over 640 samples) for classification when running bisection for autotuning the step size. For a fixed step size, the overhead of AutoCLIP is only 0.45ms. We note that AutoCLIP's overhead is very small compared to other VLM test-time adaption approaches (see below).

Since the authors claim that the proposed method has the benefit of significantly lowers the test-time computation and memory overhead, it'll be more convincing if adding the comparison with tables, and figures, etc.

Unfortunately, TPT (Shu et al., 2022) and RLCF (Zhao et al., 2023) did not report compute or memory requirements. However, TPT states "We augment a single test image 63 times using random resized crops and construct a batch of 64 images, including the original one.", which means that the image encoding time (for instance the 12.64ms from above) is increased by a factor of 64x, plus additional overhead for backpropagating through the text encoder, which likely brings the inference time per sample close to 1s. RLCF also requires multiple image augmentation but does not state the number of augmentations used. In contrast, AutoCLIP does not require any image augmentations or backpropagating through the text encoder and increase inference time from 12.64ms to roughly 14ms instead of the other method's ~1 second (for the setting discussed above)

Is there more discussions about why 'mean' in Figure 5 on Oxford dataset has the worst result?

In general, Oxford Pets is the dataset with the strongest differences between methods (see also y-scale in Figure 2). Why "mean" performs particularly poorly in this setting is unclear.

Minor: with the logsumexp providing the best result compared to entropy, mean, and max, is the idea similar to focal loss with multi-class?

Relating AutoCLIP's logsumexp objective function to other losses is interesting. We do not see an immediate connection to focal loss (maybe the reviewer can provide further details on the possible relationship?). However, as stated in the paper we note that −logsumexp has been interpreted as the energy function of a data point (for appropriately trained classifiers) (Grathwohl et al., 2020); in this view, AutoCLIP can be interpreted as minimizing the energy and maximizing the probability density p(x) of x under the zero-shot classifier.