BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping

Q1, Q9, Q12. Technical insights and difference with existing baselines.

Please refer to Q1 in Global Response.

Q2, Q3, Q4. Figure 2 (b), Figure 3 and typo.

Thanks for your advice. We will revise the figures, rewrite the corresponding part, and fix the typo in the revision.

Q5. Ablation study on threshold $\tau$.

We follow the setting in TPT [1] and empirically set the threshold $\tau=0.1$. To have a better understanding of the threshold, we provide the ablation results on the Aircraft dataset in Figure 1 of the rebuttal PDF.

The results are consistent with the conclusion from Figure 4(b) in the original TPT paper, showing that a threshold near $0.1 \sim 0.2$ contributes to the best results, while higher thresholds lead to noisy samples that may result in misleading predictions.

With the default setting that utilizes 64 augmented views, we can find that $6$ high-quality boosting samples are sufficient to bring significant improvements to the cache models.

Q6. More details about entropy minimization.

When implementing entropy minimization of the training-required methods, we follow the pipeline of TPT [1] and update the learnable prompts in the input while freezing other weights of the model. Taking entropy as a self-supervised objective, we perform gradient descent over both historical and boosting samples. Prompt tuning is more lightweight than full model tuning but still requires a large computational cost during model optimization.

It is observed from Figure 4 (b) of the manuscript that either training-required or training-free methods benefit from both historical and boosting samples, in line with the theoretical analysis provided in Propositions 2 and 3. This further highlights our contribution to bridging the gap between training-required and training-free methods.

Thanks for your advice and we will modify these parts to provide a clearer description.

Q7. Hand-crafted Prompt.

We follow the pipeline of TDA and adopt hand-crafted prompts in the training-free adapters. Take the Action Recognition dataset UCF101 for instance, TDA utilizes the prompt "a photo of a person doing {}." to better incorporate the prior knowledge of the dataset into the model.

To study the influence of these hand-crafted prompts, we further equip CLIP, TDA, and BoostAdapter with different prompts (the standard prompt "a photo of {}." and the hand-crafted prompt "a photo of a person doing {}.") and compare their performance on the UCF101 dataset.

As can be seen from the results in Figure 3 of the rebuttal PDF, BoostAdapter surpasses TDA and CLIP with both the standard prompt and the hand-crafted prompt. Additionally, the hand-crafted prompt brings improvements in performance to all the methods to some extent.

Q8, Q10. Comparison with zero-shot clip when the shot capacity is less than 3.

Our methods could be categorized into TTA since it adaptively makes predictions for streaming data during test time based on feature retrieval of historical and boosting samples.

Besides, the reviewer points out that BoostAdapter cannot perform as well as CLIP when the shot capacity is less than 3. However, this is not always the case. As shown in Figure 2 of the rebuttal PDF, BoostAdapter outperforms CLIP across all 4 tasks on the OOD benchmarks. On the Aircraft dataset, the low-entropy samples stored in the boosting cache with a small shot capacity may be biased and not diverse enough, thereby leading to the performance drop.

Q11. Independent cache v.s. Joint cache.

The independent cache will retain all the knowledge from both historical and boosting samples, whereas in the joint cache, we update the cache of historical samples with boosting samples. Due to the limited cache size, the historical samples in the joint cache will be replaced by lower-entropy boosting samples when necessary.

In most cases, BoostAdapter performs better with the joint cache compared to the independent cache. However, in some cases, when the test sample benefits from a sufficient amount of diverse cross-sample interactions, the replacement of boosting samples in the joint cache may lead to a slight performance drop.

Generally, it is preferred to utilize the joint cache rather than the independent cache due to lower storage cost and better performance on average.

Reference

[1] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models." Advances in Neural Information Processing Systems 35 (2022): 14274-14289.

Q1. Technical insights and differences with existing baselines.

Please refer to Q1 in Global Response.

Q2, Q5. Computation overhead and efficiency.

Please refer to Q2 in Global Response.

Q3, Q11. Number of augmented views.

The analysis of the augmented views can be found in Table 12 of the Technical Appendix. It can be observed that BoostAdapter shows superior performance on average compared to TDA with 32 views on the OOD benchmark and only 16 views on the Cross-Domain Benchmark.

We would like to point out that TDA also utilizes augmentation to obtain high-quality embeddings of test samples to store in the cache, as depicted in Table 1 of the rebuttal PDF. Furthermore, most of the existing TTA methods use 64 views of augmentation by default. Therefore, using 64 views is not considered particularly large and is acceptable from a computational cost perspective.

Q4. Cross-Domain Benchmarks results with RN-50 backbone.

The marginal improvement is mainly due to the limited performance of the RN-50 backbone. To verify the robustness of BoostAdapter, we further provide results with more checkpoints in Tables 2 and 3 of the rebuttal PDF. BoostAdapter shows promising improvements and consistently outperforms TDA in 7, 8, and 7 out of 10 tasks on the Cross-Domain Benchmarks with RN101, ViT-B/32, and ViT-L/14 backbones, respectively.

Q6. Shot capacity.

We provide ablation results of the shot capacity over all four datasets on the OOD benchmarks in Figure 4 of the rebuttal PDF. The results are consistent with the conclusion in the manuscript that the boosting cache will achieve a balance of diversity and relevance as the shot capacity increases.

Q7. Applicability on Vision-Only Backbones.

Yes. We believe it is possible to apply BoostAdapter to vision-only TTA, but transferring might be non-trivial since vision-language TTA and vision-only TTA are generally two distinct research sub-fields with different settings. Our paper focuses on the adaptation of the vision-language model, while addressing vision-only models could be considered for future work.

We appreciate your suggestion on refining the title of this paper and will incorporate it in the revision.

Q8. Fine-grained knowledge.

We provide qualitative results in Figure 5 of the appendix and Figure 4 of the rebuttal PDF to investigate how BoostAdapter leverages boosting samples to extract fine-grained knowledge. As depicted in the figures, boosting samples with low entropy incorporate the prior of label systems to filter out the noisy parts of the test images and guide the model on where to focus. Most importantly, we only need to perform random cropping and random horizontal flipping to achieve this, making BoostAdapter more applicable in real-world scenarios.

Q9. Storage in Boosting Cache.

In practice, we store the features of both the historical and boosting samples to construct the key-value cache, which is privacy-preserving and time-efficient. Thanks for your advice, and we will modify the figure for better clarity in the revision.

Q10. ImageNet-C.

Please refer to Q3 in Global Response.

Dear reviewer,

We would like to thank you for your insightful feedback. We hope that your concerns are addressed with our rebuttal. As we are getting really close to the deadline of the discussion phase, please let us know if there are any further questions that need clarification.

Many thanks,

Authors

Thanks for the authors’ response. Regarding the computational efficiency, could the authors provide more details of the experimental setup, including but not limited to the GPU, and batch size?

For FPS, I am confused about why BoostAdapter (64 forward passes of image encoder) achieves 11.23 fps and CLIP only achieves 82.3 fps which only needs 1 forward propagation. Meanwhile, the memory consumption of BoostAdapter and CLIP is the same. Do you test Inference Speed (fps) using batch size 64 (64 views also equal to batch size 64), and test Memory using a different batch size?

Moreover, the performance could also be directly reported in Table 1 of the PDF.

Q1. Computation overhead and efficiency.

Please refer to Q2 in Global Response.

Q2. Specific Datasets.

As shown in Figure 5 in the appendix, BoostAdapter benefits from boosting samples to capture fine-grained knowledge of the test samples. We further provide more qualitative results on the ImageNet-A, Aircraft, and EuroSAT datasets in Figure 4 of the rebuttal PDF to investigate how BoostAdapter performs on these datasets.

ImageNet-A consists of real-world examples that are misclassified by ResNet models, while Aircraft is a benchmark dataset for the fine-grained visual categorization of aircraft, and EuroSAT consists of satellite images for land use and land cover classification.

These datasets require fine-grained information for classification, and the boosting samples filter out noisy parts of the test samples while retaining useful information, contributing to significant performance enhancements.

Q3. More backbones.

In order to further validate the robustness of our method, we compare BoostAdapter with baseline models across various backbones including both RN and ViT checkpoints.

The results in Table 2 and Table 3 of the rebuttal PDF indicate that BoostAdapter shows strong compatibility and consistently outperforms TDA in most of the cases over different backbones. For instance, BoostAdapter shows superior performance to TDA over 7, 8, and 8 out of 10 tasks with RN-101, ViT-B/32, and ViT-L/14 checkpoints on the Cross-Domain Benchmark, respectively.

Q4. ImageNet-C.

Please refer to Q3 in Global Response.

Dear reviewer,

We would like to thank you for your insightful feedback. We hope that your concerns are addressed with our rebuttal. As we are getting really close to the deadline of the discussion phase, please let us know if there are any further questions that need clarification.

Many thanks,

Authors

Dear authors,

I thank you for the responses, and additional experiments provided in the rebuttal. I have read fellow reviewers comments and authors responses. I find that the method is slightly better than TDA consistently across all benchmarks, particularly helpful for ImageNet-A. However, my major concern on limited novelty still exists and hence I tend to keep a borderline rating. I will discuss with my fellow reviewers for a final decision.