BAT-CLIP: Bimodal Test-Time Adaptation for CLIP

• Ablation: Please report Zero shot CLIP results followed by the loss components. It appears that $L_{tent}$ sometimes worse than Zero shot CLIP for some cases. What happens if you only use $L_{tent}$, $L_{pm}$ and $L_{sp}$ individually. A better ablation study is where you study all the loss combinations, which I encourage the authors to present.
• Bimodal Adapatation: As the primary motivation and difference from prior works is the need to do bimodal adaptation, experiments demonstrating the effectiveness of bimodal adaptation is missing. Some experiments like: What if you use $L_{pm}$ without updating the text encoder? How do you show bimodal adaptation is better than unimodal update of LN parameters of Vision encoder?, could be done to highlight the role of bimodal adaptation.
• Lacking strong experimental protocol: All the experiments are done on only corruption benchmarks. How would BAT-CLIP perform in other domain shifts in datasets like DomainNet, VisDA comprising of Cartoon, Sketch kind of domains and ImageNet-variants like IN-R/Sketch/V2/A? While not a necessity, a stronger set of baseline methods would be to compare with more recent works like[4,5,6].
• Projection matching loss:: In eqn 2: What happens in low batchsize setting or when some classes are absent in the batch? How would the prototypes be computed for this loss? The prototypes are computed for every batch? Also, performing experiments on varying batchsizes would further demonstrate you method's effectiveness. Why is this a projection based loss where unnormalized protypes are used and not cosine similarity based, as in $L_{sp}$? Is there any specific reason for this choice?

1. Limited benchmarking against prior methods: The current paper primarily compares its approach to TTA methods from 2020 to 2022. It would provide a clearer understanding of BAT-CLIP's performance within the broader TTA landscape if the authors were to include comparisons with a wider range of state-of-the-art TTA methods in 2023 and 2024.
2. Limited Exploration of Text Encoder's Role: While the text encoder is adapted alongside the vision encoder, the paper doesn't deeply explore cases where the text encoder might contribute disproportionately to misalignment. For instance, how does BAT-CLIP handle noisy or ambiguous text class descriptions during adaptation? This could be important in real-world applications where text inputs may not always be clean or well-formed.
3. Lack of hyperparameter sensitivity analysis: The method introduces projection matching loss and separation loss, but no details are given on how sensitive the method is to hyperparameters such as the weighting between these two losses, or the cosine distance threshold for class prototypes. This sensitivity needs to be explored experimentally to ensure the method is robust.
4. Limited justification for LayerNorm adaptation: The paper chooses to adapt the LayerNorm parameters of CLIP’s encoders, as inspired by the other method. However, there is no strong theoretical or empirical justification for why adapting LayerNorm parameters is the optimal choice for both the vision and text encoders. It’s not clear if other layers might benefit from adaptation as well.

• The claim that the proposed method is the first to perform a bimodal test-time adaptation of CLIP for classification tasks is imprecise, see, e.g., Section 4 of [1]. Moreover, since maximizing the similarity between visual and text features (e.g., CLIP original training objective) and increasing inter-class separability are common practices, I do not consider the approach as particularly novel.
• Experimental evaluation should be improved. (1) Some highly related SOTA methods [2, 3, 4, 5] lack detailed discussion and comparison. (2) Additional datasets are needed to further validate the effectiveness of the approach, such as ImageNet-V2, ImageNet-A, ImageNet-R, and ImageNet-Sketch, which are commonly used in the TTA of CLIP [3].
• Contributions are not fully supported by experimental evidence and should be clarified. (1) In the ablation study (Table 4), it is unclear whether the significant improvement with the two additional loss terms can also be achievable without updating the text encoder. (2) To support the claim of efficient adaptation, comparisons with previous baselines, especially [3], in terms of FLOPs or at least forward/backward computation times are needed. It is important given that updating the text encoder seems computationally intensive, as all text prompts seem to be processed again through the text encoder at each update step.

[1] Döbler, Mario, et al. "A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models." CVPR Workshop.

[2] Sreenivas, Manogna, and Soma Biswas. "Effectiveness of Vision Language Models for Open-World Single Image Test Time Adaptation." arXiv preprint arXiv:2406.00481 (2024).

[3] Niu, Shuaicheng, et al. "Test-Time Model Adaptation with Only Forward Passes." ICML, 2024.

[4] Ma, Xiaosong, et al. "SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models." NeurIPS, 2023.

[5] Zhang, Jingyi, et al. "Historical Test-Time Prompt Tuning for Vision Foundation Models." NeurIPS, 2024.

The method design and experiment have fatal problems.

1. The test-time adaptation task assumes that there is no any class label for test images, because we need to use the model to predict the label for test images. However, this work directly leverage the groundtruth class label of test images to train the model, e.g., ``we compute the mean feature of all the support visual features constituting a class c. . yˆ refers to the predicted labels computed via Eq. 1 and v¯c is the class prototype of class c.'' in L333-334. It is obvious that the method is trained using the class label of test images.
2. The test-time adaptation task typically assume that the model can process limited samples during test-time adaptation, e.g., one test image, or up to 16 images. However, this work takes 200 images as a mini-batch for TTA, as shown in L419, ``The batch sizes are set to 200, 200, and 64 for the datasets''. In contrast, other methods, e.g., TPT (Shu et al., 2022) and VTE (Dobler et al., 2024) process a single test image at a time.
3. The novelty and contribution is limited. The proposed losses are the popular entropy minimization, projection loss and contrastive loss. The model adaptation method is the classical normalization layer adaptation.
4. The authors claim that ''image class prototype are computed using pseudo-labels'' in the abstract. However, I cannot find any ``pseudo-labels'' in the method part. It is obvious that the authors did not produce and use the pseudo labels. Instead, the authors directly use the groundtruth class labels of test images to train the model.