BaFTA: Backprop-Free Test-Time Adaptation for Zero-shot Vision Language Models

We appreciate the valuable feedback! Here are our responses to your concerns:

1. ReCLIP focuses on source-free adaptation, employing a label propagation algorithm for pseudo label generation. However, this method relies on a substantial number of examples to yield satisfactory results, a condition that may not be met in many practical scenarios. In our exploration of alternatives to the label propagation approach, we found that online clustering performs satisfactorily. This approach alleviates the need for an extensive pool of available test data while maintaining performance standards to a reasonable degree.
2. In response to your suggestion, we have incorporated additional methods (CoCoOp, CoCoOp+TPT, ProGrad, SuS-X) into the supplementary materials. However, we opted not to include CoCoOp+BaFTA for two reasons: 1) The authors of CoOp/CoCoOp have not released the weights for CoCoOp, preventing us from running BaFTA+CoCoOp; 2) CoOp+TPT outperforms CoCoOp+TPT on both RN50 and ViT-B/16, and we believe that comparing with the stronger CoOp+TPT is more informative. For a comprehensive examination of results and analysis, kindly refer to Section D and Table 4. We are planning to reorganize and integrate some of these methods into the main paper in future revisions.
3. To offer more insights into the sensitivity of the Renyi Entropy parameter, we have performed extensive experiments with detailed results in the supplementary materials. The analysis suggests that BaFTA's performance is not highly sensitive to the choice of $\alpha$. Please refer to Section B, Figure 1 and Table 2 for a more detailed analysis.
4. Yes, BaFTA can be easily saved at any moment. The only evolving parameters are the moving averaged centroids and the cluster member counts, making them space-efficient for storage.
5. The projection can be conceptually described in two steps: 1) SVD extracts the orthonormal basis U of the text embedding subspace. The projection matrix $P=UU^T$ projects everything onto the text embedding subspace, removing dimensions not participating in the classification calculation, since they are orthogonal to the text embeddings. This step removes the category agnostic information from visual embeddings and makes them easier to cluster; 2) $P^*=U’U’^T$ discards the first axis from U where all text embeddings overlap the most. This step removes shared information between text embeddings from different classes, emphasizing their differences and separating clusters from different classes.

We value your constructive feedback and would like to address your comments with the following responses:

1. In light of the concern regarding ablation experiments on various components of BaFTA, we've introduced additional ablation results in the supplementary materials. Specifically, we've included BaFTA-single, utilizing a single-template CLIP as the base model, and BaFTA-Avg, which employs a simple average function to merge predictions. Notably, the results from BaFTA-Avg exhibit a 1.68% decrease compared to the regular BaFTA, underscoring the effectiveness of Renyi Entropy Aggregation. For a comprehensive exploration of these results, please refer to Section C.1, Section C.2, and Table 3 in the supplementary materials.
2. The consistent decent zero-shot classification accuracy achieved by CLIP across ImageNet and ImageNet-A/V2/W/R/Sketch using the same text embeddings generated from the CLIP text encoder suggests that the distributions of these datasets are not distinctly different from each other. Notably, ImageNet-R itself comprises a dataset with mixed distributions, including art, sketch, cartoon, plastic toys, etc. BaFTA demonstrates a significant improvement even in this mixed-distribution scenario. This indicates that BaFTA can effectively handle a certain level of distribution mixture, given that CLIP embeddings being sufficiently accurate, thereby ensuring that the distributions are not too dissimilar. It is possible to improve BaFTA with dynamic clustering mechanisms to handle an indefinite number of clusters or mixtures of severely different distributions. While this modification is beyond the scope of the current paper, we appreciate the suggestion and will consider it in future work.
3. In response to the request for an inference time analysis, we have included a comparison of the inference time for BaFTA and TPT in our supplementary materials. The results indicate BaFTA is roughly 5 times faster than TPT. For detailed results and analysis, please refer to Section A and Table 1.

Hi! Thanks for the reply. Since it is approaching the end of the rebuttal period, we will focus on sharing our insights in this response. Hopefully these answers will help you better understand BaFTA!

Augmentation Views: Although TPT only uses a single view for prediction, it requires the computation of augmented views for its test-time training before inference for every example. BaFTA does not introduce extra computation or memory cost on augmentations in comparison with TPT. Both TPT and BaFTA require augmentations as essential components to enhance performance, while TPT utilizes them during test-time training, BaFTA directly utilizes them during inference.

Effectiveness of Online Clustering: The effectiveness of online clustering can be evidenced by results from Table 5 from the main paper. BaFTA-RA uses $p_i$ for prediction (Renyi Entropy aggregated CLIP predictions on augmented views), and BaFTA uses $p_i + \hat{p_i}$ for prediction (Renyi Entropy aggregated CLIP predictions and Online Clustering predictions, on augmented views). BaFTA exhibits a 1.38% improvement over BaFTA-RA, underscoring the effectiveness of online clustering predictions ($\hat{p_i}$).

Why Renyi Entropy: Entropy functions are good estimators of prediction confidence. For example, an uncertain uniform distribution $p_u = [1/k, 1/k,..., 1/k]$ will have Renyi Entropy $Re(p_u)=-1$ and a confident one-host distribution $p_c = [1, 0, 0, 0, 0]$ will have Renyi Entropy $Re(p_c)=0$. As a general form of entropy functions, we can control the sharpness of Renyi Entropy curves for it to mimic the behaviour of other entropy functions by tuning its entropy order $\alpha$. Small $\alpha$ corresponds to more sharpened entropy curves where only extremely confident predictions get high outputs. According to our ablation results in Section B of the supplementary materials, we find $\alpha=0.5$ reaches a good balance for predictions generated by CLIP embeddings, and therefore we use Renyi Entropy with $\alpha=0.5$ for prediction aggregation.

Thank you for your valuable comments! Here are our responses to your questions:

1. In response to the request for an efficiency analysis, we have included a detailed comparison of the inference time for BaFTA and TPT in our supplementary materials. Notably, BaFTA demonstrates an approximately 5 times speed improvement over TPT. For detailed results and analysis, please refer to Section A and Table 1.
2. Regarding the assumption about CLIP visual embeddings, results in Table 2 of the main paper indicate that CLIP's linear evaluation results on all datasets are significantly higher than the corresponding zero-shot results. Linear evaluation is a protocol that assesses the quality of representations. By freezing the CLIP visual embeddings and training a fully-supervised linear layer classifier, linear evaluation obtains the upper bound accuracy achievable by the frozen visual embeddings. Table 2 results suggest that the CLIP performance is likely constrained by the quality of text-generated classification weights. While CLIP visual embeddings are not perfect, we argue that, in the unsupervised test-time adaptation setting where only minor modifications can be made, focusing on text-side updates is more efficient due to the substantial improvement space available on the text side.
3. To offer more insights into the sensitivity of the Renyi Entropy parameter, we have added updated plots and results in the supplementary materials. The analysis suggests that BaFTA's performance is not highly sensitive to the choice of $\alpha$. Please refer to Section B, Figure 1, and Table 2 for a more detailed analysis. In terms of the embedding projection, it does not involve additional hyper-parameters except for the number of basis used to construct the projection matrix. For the majority of datasets, we set this number equal to the number of classes, considering it as the rank and the maximum number of basis for the text embedding subspace. However, for datasets with more than 150 categories, such as SUN397 and ImageNet, we observed that utilizing the top 150 basis is sufficient to capture most important dimensions from the embeddings. Therefore, we use the top 150 basis for SUN397 and ImageNet, as mentioned in the implementation detail section.

Thank you for addressing my main concerns. I will vote for weak accept.

Thank you for your valuable comments! Here are our responses to your questions:

1. Regarding the implementation details, we carefully followed the official implementation from the GitHub Repository provided in the TPT paper. In the actual implementation of TPT, both RandomResizedCrop and RandomFlip augmentations were used to produce the reported scores. Therefore, the comparison of BaFTA and TPT is fair.
2. In response to the concern about ablation experiments on prompts, we evaluated BaFTA with single-template CLIP as the base model. The BaFTA-single configuration resulted in a noteworthy 6.22% improvement in averaged accuracy across 15 datasets. You can find detailed results and analysis in Section C.1 and Table 3 of the supplementary materials.
3. To provide a more comprehensive understanding of the effectiveness of the projected embedding space, we have included t-SNE plots of CLIP visual embeddings in the supplementary materials. The t-SNE plot visually demonstrates that the projection space can effectively transform the sparse clusters into denser formation. Please refer to Section E and Figure 2 for detailed results and analysis.

Fixed some more typos in the main paper.