Frustratingly Easy Test-Time Adaptation of Vision-Language Models

W1-Q1: Novelty and relationship to over-under-confidence. To clarify our novelties: (1) we provide theoretical tools to understand the pitfalls of Marginal Entropy Minimization; (2) the proposed baseline only requires manual tweaking of a single parameter, a single forward pass of the vision encoder, and no optimization.

We politely disagree with the statement “the motivation is not novel, as the phenomenon of over-under-confidence has been already revealed”. Here are the reasons:

1. We know that over-under-confidence are widely known phenomena in the fields of uncertainty estimation and model calibration. Hence, we do not claim their discovery in any passage of the manuscript.
2. Our core motivation is not over-under-confidence, but: (1) demonstrating that the $\arg\max$ of the marginal probability distribution is largely invariant to MEM (Sec 2.2), and (2) establishing that the marginal probability distribution can be regarded as a lower bound for the error of the standard inference protocol (Sec 2.3). These findings are not readily available in the existing literature; they are key contributions and novelties of our work. As also acknowledged by Wz3n, we believe these are “great motivations for bringing insights into Test-Time Augmentations”.

Q2: Setting the temperature to 0 is "straightforward and naive". We agree that Zero is indeed simple, but we see this as a strength rather than a weakness, as pointed out by Wz3n, 7eTE, and a5fT. We firmly believe that simplicity should be rewarded, since it allows more practitioners to use our work.

Q6: Why is the temperature set to 0? Fig. 1(b) suggests that incorrectly classified views tend to suffer from overconfidence. When marginalizing, a highly confident but wrong prediction may greatly influence the resulting marginal distribution, leading to an overall incorrect prediction. Adapting the temperature addresses this by removing the dependency on confidence. Please, see the response to a5fT (Q3) for further clarifications.

cont. - learnable temperature. We are aware of works treating the temperature as a learnable parameter. Other than the provided reference, this is also done in [7], a pioneer of modern research on calibration, as well as in CLIP. Our goal, however, largely differs from learning parameters; in contrast, we aim to show that a surprisingly strong TTA baseline emerges with no optimization at all.

W2: Method lacks flexibility and guarantees. On flexibility: our method is fairly simple and can be easily integrated into a variety of VLMs (as also Wz3n points out). Its only prerequisite is a softmax-based classification pipeline. On guarantees: as also acknowledged by 7eTE, our work features a theoretical and empirical analysis (Sec. 2.2 to 3.1) that tries to justify its design. This is missing even in some influential work in this field, e.g., [52, 37, 32]. We are happy to expand our answer in case the reviewer points to specific flexibility/guarantees issues we did not cover.

Q3: Unusual pseudocode. We provided the pseudocode in a code-like fashion following some influential examples from the literature, including CLIP (see [a-h]). While traditional pseudocode may hide how functions are implemented, code-like snippets provide an intuitive way to grasp how simple and easy to adopt the proposed methodology is.

Q4: Too few comparisons, many TTA methods exist [3]. Do they have the same problem? We strove to compare with the state-of-the-art among MEM-based methods (TPT, PromptAlign) as well as the latest state-of-the-art on TTA for VLMs “Reinforcement Learning from CLIP Feedback” [53], published at ICLR 2024. This year ICLR ended 4 days before the abstract submission deadline for NeurIPS. We believe this is a good effort to keep up with the pace of AI research. Our work also features an analysis of Marginal Entropy Minimization, hence, to answer the question, our theoretical insights extend to strategies relying on this objective, including those listed in the provided survey paper. Other than TPT and PromptAlign, some examples are [18,26,41,52].

We are happy to enrich our manuscript in case the reviewer points to specific comparisons that may be relevant to this work.

Q5: Missing results. Our focus is on VLMs, hence we followed the established experimental protocol of the field of TTA with VLMs by evaluating different CLIP variants (ViT-B-16 and MaPLe), following [37,32]. Additionally, we experimented with a newer combination of CLIP-ViT-B-16 and CLIP-ViT-L-14, for which [37,32] did not present results. We did not omit any results and believe that our evaluation is fair.

Q7: Label smoothing. In our context, applying label smoothing as in [i] would not bring benefits since it would not affect the $\arg \max$ of the marginal probability distribution. Thus, it would incur the same issues highlighted in the example of the response to a5fT (Q3).

$\bar{p}^{LS} = 1/N \sum p_i^{LS} = 1/N \sum \left[(1-\alpha) * p_i + \alpha/C\right] = (1-\alpha) * 1/N \sum p_i + \alpha/C$
$\bar{p}^{LS} = (1-\alpha) * \bar{p} + \alpha/C \implies \arg \max \bar{p}^{LS} = \arg \max \bar{p}$

cont. SAM. A core contribution of our work is to show that Marginal Entropy, a very popular objective for TTA, has some pitfalls that can be circumvented with a simple and optimization-free approach. While we agree that sharpness-aware strategies can synergize with TTA, proposing an optimization-based alternative is out of the scope of our work.

Q8: Unusual terms like "prevalent class". We thank the reviewer for pinpointing this potential misunderstanding! To clarify, we will replace “prevalent class” with “most probable class”. This refers to the class with the highest probability within a distribution. We notice the reviewer mentioned "many uncommon words." Could the reviewer provide examples? We are willing to clarify and update the manuscript accordingly.

W1 - Q1: Invariance to Entropy Minimization. Invariance to MEM is strongly related to the uncertainty of the marginal probability distribution pre-TTA. The lower the initial entropy, the lower the impact of MEM on the $\arg \max$.

Theory: this is related to the proof of Proposition 2.1. To guarantee invariance, we express the post-MEM embeddings as a function of the pre-MEM embeddings through a Taylor expansion, which implies that the variation is small. If the initial entropy is high, the gradients from MEM (and, thus, the variation between pre- and post-MEM embeddings) can be larger than what a Taylor expansion can accurately approximate, so Prop. 2.1 cannot be guaranteed.

Empirical evidence: To visualize the aforementioned relationship, we compute pre- and post-MEM marginal probability distributions. We sort the pre-MEM distributions in order of descending entropy and quantize them into 10 bins. Bins shall be interpreted as follows:

1. the leftmost bin contains the top 10% of samples with the highest entropy;
2. the second bin contains samples outside the top-10% percentile but within the top-20%, and so on;
3. the rightmost contains the bottom 10% of samples with the lowest entropy.

For each bin we compute the invariance ratio, measuring how often the $\arg \max$ of the pre-MEM $\overline{p}$ does not change after MEM (the higher the better). Finally, we display a histogram with this data in Figure 2 of the PDF attachment.

A trend appears: as the entropy decreases (left to right), invariance holds more often. Hence, to answer: the most likely cases where invariance to MEM does not hold are those of high uncertainty in the marginal probability distribution. However, this may still be rare: even within the top 10% of most uncertain samples, invariance holds more than 82% of the time (leftmost bin).

The reported experiment was conducted on the validation set of ImageNet-1k with OpenAI’s CLIP-ViT-B-16, but identical trends were observed for all Natural Distribution Shifts datasets. We plan to report these results in the revised appendix, together with this discussion and a few examples. We are thankful to the reviewer for this comment.

W2 - Q2: Overconfidence and "always". We agree. Following the suggestion, we report the same experiment with LAION-pretrained CLIP models, using the code of [j]. In the PDF attachment, the reviewer can find results for LAION-2B and LAION-400M.

Notably, these CLIP variants comply with the already observed patterns, i.e.: the ECE increases with augmented views, with overconfidence being the leading cause. We propose to include these results in a dedicated Appendix.

We further propose to tone down ll. 192-197 (shared with 7eTE):

1. line 192, paragraph header: “Poor calibration is frequently linked to overconfidence.”
2. lines 195-196: “Notably, in the scope of our experiments, overconfidence is the primal factor leading to an increase of the ECE.”
3. line 197: “In Appendix B, we also experiment across all datasets for Natural Distribution Shifts and different CLIP models pretrained on LAION. Importantly, this phenomenon further persists within this extended experimental suite.”

We thank the reviewer for this comment.

W3: Cross-datasets generalization. These results are already in the manuscript, in the 2nd comparison group of Tab. 2 (MaPLe). We pinpoint that there are slightly different meanings to the term “cross-datasets generalization”, according to the recent literature on TTA, and that confusion may arise from these. We recap them here:

1. This experiment was first designed in the TPT paper to compare supervised prompt learning (e.g. CoOp [55]) vs instance-specific prompt learning. In our case there is no learning, so this would not apply.
2. The PromptAlign paper extends this setting to see how prompt learning methods, trained on a source dataset like ImageNet, can benefit from instance-specific TTA when evaluating “cross-datasets”.

This second experiment is already in Table 2: MaPLe prompts are learned on ImageNet and the model is adapted by Zero “cross-datasets”. We referred to these as the "Fine-grained classification" experiments.

However, we notice that we did not explicitly mention that our MaPLe initialization comes from ImageNet. We sincerely apologize. As a remedy, we propose the following:

1. Before l. 252: “MaPLe prompts are learned on ImageNet, following [32]”;
2. In l. 267, we will insert: “When adapting MaPLe, we stick to the ImageNet-learned prompts and evaluate it cross-datasets as in [32].”

Q3: zero temperature. Consider this example in a 3-way problem. Let $x_1$, $x_2$, and $x_3$ be three views and $p_1$, $p_2$, and $p_3$ be their probabilities with the default temperature. Let $y=2$ be the correct label.
$p_1 = [0.9, 0.1, 0.0]$ - incorrect
$p_2 = [0.3, 0.6, 0.1]$ - correct
$p_3 = [0.25, 0.6, 0.15]$ - correct

This simple example is designed to meet the observations of the manuscript:

1. the error rate of the model is low, but
2. the model may suffer from overconfidence ($p_1$).

The resulting marginal probability distribution $\bar{p} = [0.483, 0.433, 0.083]$ would be wrong because of the influence of a wrong overconfident example.

Setting the temperature to zero before marginalizing, it would be (approximating):
$p_1 = [1, 0, 0]$
$p_2 = [0, 1, 0]$
$p_3 = [0, 1, 0]$

with a resulting $\bar{p} = [0.33, 0.67, 0]$, which would be correct and avoid the influence of overconfidence.

The takeaway is that due to the effect of data augmentations, confidence information can be misleading, but we can still rely on the fact that the predictions of the model, on average, are accurate. Setting the temperature to zero discards confidence information, but retains the $\arg\max$ (i.e., the prediction).

Q4: single-test point. To clarify, we will include the following before l. 252: “Similarly to [37, 32, 53], we always work with a single test point at a time.”

Dear Authors,

Thank you for your response, which has addressed the initial concerns. I tend to maintain my score of “accept.” Please ensure the revision incorporates all the responses.

Best,

Reviewer a5fT

Dear Authors,

Please clarify two major points to help me understand the effectiveness of ZERO.

• Cross-dataset setting. Thanks for the response. Why not include Caltech in Tab 2 as TPT [37] and PromptAlign [32] do? Also, Ref Table 1 of RLCF [53] reports the comparison with TPT on natural distribution shifts, using CLIP-ViT-B-16. Why not include it in Table 1? MaPLe (ref Table 5 in [14]) reports the results on ImageNet, why not report in Table 1 ?

• Zero temperature. Follow the suggested case, let $p_1 = [0.1, 0.9, 0.0]$- correct, $p_2 = [0.6, 0.3, 0.1]$- incorrect, and $p_3 = [0.6, 0.25, 0.15]$- incorrect, then $\bar{p}=[0.43, 0.48, 0.08]$. After using zero temperature, $\bar{p}_{zero}=[0.67, 0.33, 0.00]$. This gives the wrong class prediction.

I understand there are two points here: 1) 'confidence information can be misleading', and 2) 'the fact that the predictions of the model, on average, are accurate'. How to make sure the second point persists? Moreover, using zero temp, makes the prediction scores aggregation (a kind of) the majority vote of the predicted classes.

Kind regards,

Reviewer a5fT

W1 - Q1: Independence among views. We agree and we thank the reviewer for this remark, which allows us to enrich our manuscript further.

The theoretical framework of Section 2.3 models an ideal scenario, where independence holds among different inputs. To clarify, this means that the model's error on view $\mathbf{x}_i$ should not be correlated with the error on another view $\mathbf{x}_j$, which allows writing the compound error with a binomial distribution.

In practice, achieving perfect independence is challenging, if not impossible. Hence, a suitable approximation strategy to mitigate this issue is to promote diversity. In classical ensembling theory, a well-established approach is to train different models on different subsets of the available data. Similarly, our augmentation scheme of random cropping aligns with this approach by presenting the model with different portions of the image each time.

Moreover, ideally, the augmentation pipeline should not change the underlying label of the original input and guarantee that the model’s error rate on augmented views remains comparable to the error rate on the original inputs belonging to the same category ($\epsilon(y)$ in the caption of Figure 1(a) of the manuscript). In practice, this entails that augmentations should not disrupt the visual appearance of the image, and, consequently, some views may result in a slight or moderate correlation, because some “parts” of the source image will overlap among them. An analogy with classical literature can be drawn also in this case. Specifically, when not enough data are available, overlaps among the training sets of different models are required to ensure convergence. As a consequence, models producing slightly or moderately correlated predictions are more likely to emerge.

While we have tried to highlight this potential limitation in the dedicated section, we believe a tailored discussion may be helpful for the readers. Hence, we propose to replace lines 139-140 with a pointer to an in-depth appendix on this subject, where we will include this discussion.

W2: Overconfidence and "always". We agree with this remark. As we acknowledge in the Limitations section, this is an empirical observation stemming from the combinations of models and datasets that we tested, and could not extend to the space of all existing VLMs and datasets (or those that will arise in the future).

As suggested by Reviewer a5fT, Figures 1(a) and 1(b) of the PDF attachment show additional experiments with LAION-pretrained CLIP models, which confirm our initial observations. We hope that these will strengthen the analytical section, and plan to include these results in the Appendix.

Finally, we propose detailed changes to the manuscript to tone down some passages from lines 192 to 197 (shared with reviewer a5fT):

1. line 192, paragraph header: “Poor calibration is frequently linked to overconfidence.”
2. lines 195-196: “Notably, in the scope of our experiments, overconfidence is the primal factor leading to an increase of the ECE.”
3. line 197: “In Appendix B, we also experiment across all datasets for Natural Distribution Shifts and different CLIP models pretrained on LAION. Importantly, this phenomenon further persists within this extended experimental suite.”

Thank you for this suggestion.

About Limitations. We are glad that our efforts in highlighting the limitations of our work were appreciated! We agree that, ideally, a Limitations section shall be part of the main body of the paper. This year NeurIPS is granting an extra page for the final revision of the manuscript upon acceptance. We will use it to follow this suggestion if this is our case. Thank you.

W1 - Performance on Fine-grained datasets vs Natural Distribution Shifts.
We thank the reviewer for raising this interesting point, allowing us to further investigate our method.

A possible explanation may be linked to Sections 2.3 and 3.1 of the manuscript. Specifically, Zero improves over the zero-shot baseline if the error rate does not largely increase with augmented views. As Fig.1(b) of the manuscript displays, this is the case for all Natural Distribution Shifts datasets. For Fine-grained classification, we discuss here three distinct datasets for which Zero exhibits different behaviors:

1. $\text{Flowers102}$. Zero does not improve over the baseline here.
2. $\text{Caltech101}$. Zero marginally improves here.
3. $\text{SUN397}$. Zero largely improves here.

To understand these different behaviors, we repeat the same experiment of Section 3.1 for the entire Fine-grained suite and report here the results for the aforementioned datasets, formatted as follows: $[ \text{zero-shot accuracy}, \text{augmented version accuracy}, \text{error gap}]$.

By $\text{error gap}$, we refer to: $\text{error gap} = \text{zero-shot accuracy} - \text{augmented version accuracy}$.

We additionally include the accuracy of Zero.

Results [%]

1. $\text{Flowers102} = [67.44, 66.19, 1.25]$ | $\text{Zero} = 67.07$ ($-0.37$ w.r.t. zero-shot baseline)
2. $\text{Caltech101} = [93.35, 92.62, 0.73]$ | $\text{Zero} = 93.51$ ($+0.16$ w.r.t. zero-shot baseline)
3. $\text{SUN397} = [62.59, 62.97, -0.38]$ | $\text{Zero} = 64.49$ ($+1.90$ w.r.t. zero-shot baseline)

For completeness, all results are reported in Table 1 of the PDF attachment of the general response.

Overall, there is a strong correlation between the error gap and the improvement provided by Zero, with Spearman’s coefficient being $-0.95$ across all datasets. This result shows that the correlation is negative, i.e., the lower the error gap, the larger the improvement. This pattern is also consistent with the experiments on EuroSAT reported in the Appendix of the manuscript (App.E, Tab.5).

Hence, the reviewer’s question is directly linked to this follow-up question: Why do augmentations increase or decrease the error gap with different datasets? While this may be a case-by-case matter, we pinpoint two possible reasons:

1. The semantic space of the ImageNet variants of the Natural Distribution Shifts benchmark comprises many common categories, which may have appeared frequently during CLIP’s pretraining. Hence, it seems reasonable that CLIP is robust w.r.t. augmented views of images belonging to these categories. In the Fine-grained classification suite, datasets such as SUN397 and Caltech101 also contain common object categories, which is consistent with the results shown above. Other datasets, such as Flowers102 and Oxford-Pets, span much less frequent concepts, and CLIP is less robust w.r.t. their augmented views.

2. Other than the semantic classification space, also the visual appearance of images plays an important role. For example, datasets such as FGVC-Aircraft and Stanford Cars still contain rare concepts, but Zero largely improves over the baseline nonetheless (main paper, Tab.2, first comparison group). Our augmentation setup is simple, and only contains random resized crops and random horizontal flips, which can constitute a “zoom-in” to a random portion of the image. For some benchmarks, this is useful as it may trigger CLIP’s capabilities to recognize small details, such as logos, or even reading text, such as the car brand or the airline name. In contrast, for Flower102, these may lead to some loss of precious visual features, such as the stem.

To conclude: In our work we did not search for the best data augmentations but rather stuck to an established setting, using the same augmentations setup for all datasets. Nevertheless, the performance of Zero is linked to the impact that data augmentations have on how the model perceives images, and we believe this is an interesting research direction to pursue.

We also think that this may be a useful discussion, and plan to include it in the Appendix of the revised manuscript.