Efficient Test-Time Prompt Tuning for Vision-Language Models

We sincerely appreciate your constructive feedback. Please allow us to address your concerns below.

W1 (part1): The proposed method faces a fundamental technical flaw when adapting models to new target domains that share the same label space as the source domain.

Thank you for your comment. We acknowledge that Self-TPT is not specifically designed for domain adaptation. However, we respectfully disagree that this aspect constitutes a fundamental technical flaw in our paper, for the following reasons:

• Purpose of our study: This paper primarily investigates test-time new class understanding (as opposed to domain adaptation). We demonstrate that our proposed method achieves sota performance with high efficiency in this area by benchmarking it against two popular open-vocabulary settings.
• Orthogonality to domain adaptation methods: It is crucial to highlight that Self-TPT can be seamlessly integrated with TPT-like methods, which performs single-sample prompts adaptation. It offers the flexibility for users to choose between prioritizing computational efficiency (using Self-TPT alone) or enhanced domain adaptation performance (combining Self-TPT with TPT).
• Decent domain adaptation performance: Although Self-TPT is not tailored for domain adaptation, it performs well in such benchmarks (shown in Table 4). This performance makes the term "a fundamental technical flaw" an unsuitable descriptor for the Self-TPT's limitations in domain adaptation contexts.

W1 (part2): This raises doubts about the results reported in Table 4. I conjecture the improvements may instead stem from prompt ensembling.

Thank you for your inquiry. To address your concern, we have compared our method with the two prompt ensembling techniques you referenced. The table below shows that although all methods incorporate prompt ensembling, our approach consistently outperforms others. This suggests a significant performance gap between Self-TPT and prompt ensembling techniques.

W2: The ablation studies do not include the performance of comparable baselines that use similar ensembling techniques.

Thank you for your comment. We have indeed included the experiment you mentioned in our original submission. Please refer to Line 2 of Table 5(a) for the results.

As stated in Lines 419-425 of our paper, Table 5(a) provides the following breakdown of our method: Line 1 shows results for our baseline CoOp, Line 2 describes CoOp enhanced with CPT which incorporates four prompt perspectives and employs contrastive learning. Line 3 further employs CPT at test time for new class adaptation, while Line 4 incorporates the gradient matching loss we proposed. Notably, Self-TPT surpasses the results of Line 2 by margins of 0.9%, 1.3%, and 4.4% on the "Generic," "Fine-Grained," and "Specialized" datasets, respectively.

We sincerely appreciate your insightful feedback. Please allow us to address your concerns below.

W1: Compare with more recent prompt learning approaches (Any-shift prompting, PromptKD etc).

Thank you for your suggestion. We will incorporate Any-shift prompting into Tables 2-4. Additionally, we will discuss PromptKD in our related work section, as a direct comparison with our method would not be appropriate due to its use of a larger pre-trained model for knowledge distillation. We will also include more recent prompt learning studies in accordance with your recommendations.

W2: Can CPT also be combined with TPT?

Yes, your observation is correct. Our method is orthogonal to existing test-time prompt tuning methods, such as TPT. Below are the results demonstrating their integration:

Thank you for your response and experiments. I think CPT does compliment existing techniques. But I'm not sure whether it is best presented as an efficient test-time prompt tuning technique. I maintain the original score.

We appreciate the time and effort you have dedicated to reviewing our paper. Please allow us to address your concerns below.

W1: The novelty is not significant, as leveraging multiple prompts is proved to be useful in Kgcoop and Promptalign paper. The Gradient matching loss provide very little improvement and may loss performance in some settings, as shown in table 5c.

Thank you for your comment.

First, we would like to clarify that our primary novelty is not in using multiple prompts, but in developing the first text-only, self-supervised test-time adaptation framework that effectively balances performance with efficiency. Within this framework, we implemented CPT as a specific self-supervised task. Additionally, we studied the gradient correlation phenomenon to improve the understanding of our framework's effectiveness.

Second, our study fundamentally differs from prior works. KgCoOp utilizes hand-crafted prompts for distillation and PromptAlign employs multi-layer and multi-modal prompts, while our innovation lies in adjusting the class token positions for contrastive learning, which has not been proposed before. Moreover, our method surpasses KgCoOp and PromptAlign in the base-to-new setting by 3.86% and 1.59% respectively, signifying the substantial progress we have made.

Lastly, Table 5(c) details our trial-and-error experiments, thus, some configurations did lead to decreased performance. [1, 2] have shown that manipulating gradients to meet specific expectations is challenging. Our GM loss both improves the task gradient similarity and generally boosts performance. While the improvements in some scenarios are modest, we believe our results could motivate future exploration of test-time prompt tuning.

W2: Do you need more prompts than others, since you may need three more prompts for CPT.

Thank you for your inquiry. Our approach does not require additional prompts; we utilize the same number of prompts as our baseline, CoOp. Specifically, we employ four $1\times512$ vectors as prompts. Importantly, the class token is inserted at varying positions within the prompt sequences to create distinct views, while the prompts themselves are shared across these views.

Q1: Can it be applied with existing prompt learning methods?

Yes, Self-TPT can be integrated with existing prompt learning methods. Applying Self-TPT involves two steps: 1) conducting CPT during the training phase on source data and optionally adding $\mathcal{L}_{GM}$ to enhance performance; 2) applying CPT to new classes during test-time for adaptation prior to making predictions.

Q2: In Table 5c, what is the experiment setting for the first line?

You are correct, and I apologize for any confusion. Line 1 of Table 5c and Line 3 of Table 5a have identical settings. However, these two experiments were conducted on different machines, which led to slight differences in the results. These variations are within the expected range, as detailed in our error bar analysis in Table 9.

[1] Gradient Projection Memory for Continual Learning. ICLR 2021.

[2] Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization. ICML 2022.

Thank you for your detailed review and insightful comments. Please allow us to address your concerns below.

W1 (part1): The authors vary the insertion points of the class token, and fail to give a convincing argument as to why that should lead to a better performance.

Thank you for your comment. Our CPT brings performance gain in both the source training and test-time adaptation stages, which we detail as follows:

• Source training: We alter the insertion points to generate diverse views of a class. This is analogous to image data augmentations such as random rotation. Such augmentation forms the basis for contrastive learning. This not only aids in learning more robust prompts with enhanced generalization ability (as you mentioned in W2) but also ensures inter-class distinguishability (as demonstrated in Figure 6).
• Test-time adaptation: [1] theoretically substantiates that positive gradient correlation between loss functions contributes to improved TTA performance. We discuss this in Section 3.4 and empirically demonstrate in Figure 3(b) that our proposed CPT consistently exhibits a positive gradient correlation with the classification task in 10 datasets.

W1 (part2): How did the authors decide to position the class token at the front, middle, and end, rather than choosing the positions randomly?

Thank you for your question. We aimed to ensure the simplicity of our method by making it straightforward to demonstrate and easy to replicate, so we chose to insert the class token at fixed positions. The random insertion is certainly feasible for Self-TPT, and we present the results below.

W2: Contrastive loss is a more robust one possibly with better generalization ability, while the CE loss has the absolutely right discriminative information. I cannot see obviously strong relation. Exp in Table 5c, cannot convince me.

Thank you for your comment. Self-supervised loss, to which contrastive loss belongs, can indeed display a strong relation with classification loss, as evidenced by studies [1, 2, 3]. We demonstrate the relation between CPT and CE loss by analyzing their backpropagated gradients across 11 datasets (Section 3.4 and Figure 3(b)). Additionally, Table 5(c) is not intended to illustrate the relationship between contrastive loss and CE loss; rather, it serves as a trial-and-error experiment to explore the effective design for gradient matching loss.

W3: How are the weights among losses distributed? The authors need to provide more experiments. I would also suggest authors to compare their method with more recent studies.

Thank you for pointing this out. Currently, the weights for three losses are each set to 1. In response to your suggestion, we have conducted an ablation study to explore different weight distributions:

The results suggest that increasing the weight for CE loss exacerbates overfitting on base classes. Meanwhile, increasing weights for CPT or GM loss slightly hampers the learning of classification knowledge. Our default setting provides the best balance across the configurations tested.

Additionally, we plan to compare our method with more contemporary studies as you suggested.

W4: There may be a writing error in Table 5.

Thank you for pointing this out. We apologize for this oversight and will ensure it is corrected in our final version.

[1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. ICML 2020.

[2] TTT++: When Does Self-Supervised Test-Time Training Fail or Thrive? NeurIPS 2021.

[3] Test-Time Training with Masked Autoencoders. NeurIPS 2022.