Historical Test-time Prompt Tuning for Vision Foundation Models

[Response 1] Further explanation of how and why the proposed knowledge memory banks effectively address the knowledge forgetting issue:

Thank you for pointing out this issue.

As discussed in Lines 37-45, HisTPT introduces three types of knowledge banks to memorize the previously learnt knowledge which helps mitigate the 'forgetting' issue. Specifically, the forgetting is largely caused by the accumulation of prediction errors over unlabelled test samples along the tuning process. HisTPT exploits the three types of complementary knowledge that collaborate to denoise the prediction of test samples along the tuning process, alleviating error accumulation and ultimately mitigating the forgetting issue.

We analyze how the three memory banks help mitigate the forgetting by visualizing the change of their stored features along the test-time adaptation process. As shown in Figure 2 of the attached PDF, the three types of knowledge banks store complementary historical features:

1. The global prototypes exhibit slow and gradual shift from the initial feature prototypes, preserving the knowledge of pre-trained vision foundation models and facilitating stable test-time adaptation.
2. The features in the local knowledge bank change rapidly, validating their effectiveness in capturing fresh and up-to-date distribution changes along the test-time adaptation process.
3. Most features in the hard-sample knowledge bank lies around inter-category boundary, indicating their effectiveness in capturing difficult and rare corner cases along the tuning process.

In this way, HisTPT can leverage the comprehensive and complementary knowledge stored in the three memory banks to denoise predictions of test samples along the tuning process, reducing error accumulation and effectively mitigating the issue of forgetting.

In addition, we follow prior test-time adaptation study [a] and conduct experiments to analyze the forgetting mitigation ability of HisTPT. Specifically, we randomly select one of the five datasets in Table 6 as the reference domain and perform continual adaptation toward the other four datasets. During the continuous adaptation process, we evaluate HisTPT's ability of preserving the knowledge of vision foundation models by measuring its performance on the reference domain. As shown in Figure 1 of the attached PDF, HisTPT shows clearly less performance degradation on the reference domain consistently, demonstrating its effectiveness in mitigating forgetting during the adaptation process.

We will include above analysis in our revised manuscript.

[a] Efficient test-time model adaptation without forgetting, ICML 2022.

[Response 2] The texts and figures need to be improved:

Thank you for your comments! We will check through the manuscript carefully to remove repetitive text as suggested. In addition, we will improve Figure 2 of the main manuscript to illustrate the framework of the proposed method more clearly.

Dear Reviewer CdiG,

Thank you for your insightful feedback. We have carefully considered your questions and suggestions and have addressed them accordingly. We sincerely appreciate your constructive comments, which have helped strengthen our paper. As the discussion phase is nearing its conclusion, we would appreciate it if you could let us know if there are any additional questions or suggestions.

Best regards,

Authors

[Response 1] Further analysis of the three knowledge banks:

Thank you for your suggestion! We analyze the three knowledge banks by visualizing their stored features along the test-time adaptation process. Three points can be drawn as illustrated in Figure 2 in the attached PDF:

1. The global prototypes exhibit slow and gradual shift from the initial feature prototypes, preserving the knowledge of pre-trained vision foundation models and facilitating stable test-time adaptation.
2. The features in the local knowledge bank change rapidly, validating their effectiveness in capturing fresh and up-to-date distribution changes along the test-time adaptation process.
3. Most features in the hard-sample knowledge bank lies around inter-category boundary, indicating their effectiveness in capturing difficult and corner cases along the tuning process.

With the three types of complementary knowledge, HisTPT enables adaptive regularization for the prediction of current test samples. We will include the above analysis and the attached visualization in the updated manuscript.

[Response 2] Comparisons with more related works:

Thank you for sharing the two prior studies! We will discuss and compare with them in the updated manuscript. Specifically, [1] leverages self-supervised contrastive learning to facilitate the test-time prompt adaptation, while [2] introduces a training-free dynamic adapter that caches category-specific feature for efficient test-time adaptation. Differently, HisTPT focuses on mitigating the knowledge 'forgetting' problem in test-time prompt tuning, and it achieves it by constructing comprehensive memorization that captures useful historical knowledge. As shown in Table below, HisTPT performs clearly better than [1,2], demonstrating its effectiveness in tackling test-time prompt tuning challenge. The experiments are conducted on Cityscapes semantic segmentation task with SEEM-Tiny.

[Response 3] Running time of HisTPT:

Thank you for your suggestion! The table below shows the run time of the proposed HisTPT with CLIP ResNet-50 on a single GPU over dataset Flowers-102. We can see that HisTPT is clearly more efficient than TPT. The better efficiency is largely attributed to two factors: 1) TPT involves heavy augmentations of test images (e.g., 64 augmentations for each test image), while HisTPT does not; 2) HisTPT introduces memory to store historical information efficiently as described in Lines 155-157, 169-172 and 541-550.

Dear Reviewer ejR5,

Thank you for your insightful feedback. We have carefully considered your questions and suggestions and have addressed them accordingly. We sincerely appreciate your constructive comments, which have helped strengthen our paper. As the discussion phase is nearing its conclusion, we would appreciate it if you could let us know if there are any additional questions or suggestions.

Best regards,

Authors

[Response 1] Evaluation of HisTPT over the continuous test-time adaptation task:

We would clarify that we evaluated HisTPT over continuously changing test domains in Table 6 of the main manuscript. As discussed in Lines 280-293, HisTPT can tackle challenging scenarios when the domain of test samples changes continuously.

[Response 2] Comparisons with other memory-based TTA methods:

Thank you for sharing the prior studies [2-5] and we will review and compare with them in the revised paper. Our HisTPT differs in two major aspects:

1. Memory Types - HisTPT designs three types of knowledge banks for capturing and storing both fresh and representative features;
2. Memory Retrieval - HisTPT designs an Adaptive Knowledge Retrieval Mechanism for retrieving the memorized information adaptively for each test image.

As a comparison, the shared studies generally employ a single type of vanilla memory that stores either samples or class centroids. Due to the very different designs, HisTPT outperforms the shared work clearly as shown in the table below (on Cityscapes semantic segmentation task with SEEM-Tiny).

[Response 3] Quantification of the forgetting mitigation ability of HisTPT:

Thank you for your suggestion! Since the source domain of vision foundation models (pre-training) is generally huge with data from multiple sources, it is very challenging to measure the forgetting by directly testing the performance over such source domain. Instead, we randomly select one of the five datasets in Table 6 as the reference domain and perform continual adaptation toward the other four datasets. During the continuous adaptation process, we evaluate HisTPT's ability of preserving the knowledge of vision foundation models by measuring its performance on the reference domain. As shown in the Figure 1 in the attached PDF, HisTPT shows less performance degradation on the reference domain consistently, demonstrating its effectiveness in preserving the knowledge of vision foundation models and mitigating forgetting during the adaptation process.

[Response 1] Robustness of HisTPT to different confidence metrics:

Thank you for your suggestion! We conduct the suggested studies by adopting different confidence metrics, i.e., Softmax probability [a], MC dropout [b], and Mahalanobis distance [c], over Cityscapes semantic segmentation task with SEEM-Tiny. As shown in the table below, HisTPT can work with different confidence metrics. We chose entropy as it is simple and widely adopted.

[a] A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks, ICLR 2017.

[b] Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, ICML 2016.

[c] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, NeurIPS 2018.

[Response 2] Clarification of the ablation study design:

Thank you for pointing out this issue. For the ablation experiments without Adaptive Knowledge Retrieval, we compute the self-supervised loss $L_{self}$ with the regularized prediction $\hat p_n$ which is obtained without the adaptive weighting as in Eq.7.

Specifically, for experiments using a single knowledge bank in Table 4, we compute $L_{self}$ by directly adopting the prediction of respective knowledge bank by Eq.6 as the final regularized prediction. For the rest experiments using more than a single knowledge bank, we compute $L_{self}$ by averaging the predictions of respective knowledge banks as the final regularized prediction.

We will clarify this point in Section 4.4 in the updated manuscript.

[Response 3] How does HisTPT regularize boxes for object detection task:

We would clarify that, as discussed in Section Method in the main manuscript and Section Limitations in the appendix, HisTPT achieves effective text prompt tuning by regularizing the category prediction only, which is general and applicable to various vision foundation models on image classification, semantic segmentation and object detection.

On the other hand, the bounding box prediction is the task-specific design for object detection task. We did not consider it during method design as we aim to build a general test-time prompt tuning framework that can work for various vision tasks. Nevertheless, we believe that introducing additional task-specific designs like regularizing bounding boxes for object detection tasks may improve the performance. We will investigate how to incorporate task-specific designs (e.g., regularizing bounding boxes) in our future work.

[Response 4] Computational and storage overhead compared to TPT:

Thank you for your suggestion! We conduct the suggested efficiency benchmarking with TPT in terms of run time and GPU memory usage, under the same setting with CLIP ResNet-50 on dataset Flowers-102 with a single GPU. As the table below shows, TPT has much longer inference time and higher GPU usage because 1) TPT involves heavy augmentations of test images (e.g., 64 augmentations for each test image) while HisTPT does not and 2) HisTPT efficiently memorizes the historical information by storing compacted data within fixed-size memory banks as described in Lines 155-157, 169-172 and 541-550.

Dear Reviewer 43p7,

Thank you for your insightful feedback. We have carefully considered your questions and suggestions and have addressed them accordingly. We sincerely appreciate your constructive comments, which have helped strengthen our paper. As the discussion phase is nearing its conclusion, we would appreciate it if you could let us know if there are any additional questions or suggestions.

Best regards,

Authors