DynaPrompt: Dynamic Test-Time Prompt Tuning

Your suggestions help improve our manuscript a lot. Thanks for your updates and prompt encouragement.

We thank Reviewer kMEy for the constructive feedback and insightful comments. We hope to address the concerns of the reviewer with the responses below.

Questions

Experiments on ImageNet

We provide ImageNet results of our method, based on the ViT-B/16 backbone, in the following table, Our method achieves better performance than CLIP and TPT through dynamic prompt tuning. Combined with the prompt learning methods CoOp and MaPLe, the performance of our method is further improved. We added the ImageNet results in Table 1.

Experiments with different backbones

To evaluate the proposed method on different backbones, we conduct experiments for ImageNet-based datasets with the ResNet-50 and ViT-B/32 backbones. The experiments are provided in the following table. No matter the backbone, our method outperforms CLIP and TPT. We added these results to Appendix C.

Results for CoOp + DynaPrompt in the cross-dataset setting

We provide the requested results on the cross-dataset setting. Our method outperforms TPT based on the CoOp pretrained prompt for 9 out of 10 datasets. We included these results in Table 2.

Online prompts tuning with identical initialization

The reviewer is correct that our buffer is initialized with only one prompt, and we append a new prompt initialized with “a photo of a” when no previous prompt is selected during online tuning. To demonstrate the effect of identically initializing a set of prompts, we conduct experiments on ImageNet-A with 10 initialized prompts in the prompt buffer. The prompts are initialized by the embedding of “a photo of a” with random noise. We use our selection strategy to select and optimize the prompts online. When no prompt is selected, we use the initial prompt “a photo of a” for prediction.

As shown in the following table, this variant outperforms online TPT and CLIP, which demonstrates that it reduces collapse since it provides diverse prompt options for different test samples through dynamic selection. However, it underperforms TPT and our method. The reason can be that the negative influence of previous test samples during online updating is still not entirely solved by the limited number of predefined online prompts, which leads to error accumulation and suboptimal predictions. We added this experiment and discussion to Appendix C.

We thank Reviewer qkjX for the constructive feedback and insightful comments. We hope to address the concerns of the reviewer with the responses below.

Weaknesses

Time costs of dynamic selection and appending

We acknowledge our improved prediction performance comes with an increased computational cost. We note that most of the additional time cost stems from optimizing multiple prompts rather than the selection and appending strategy. For instance, on ImageNet-A, with a prompt buffer size of 10, the total processing time per test sample is approximately 0.39 seconds, of which the selection and appending steps account for only 0.004 seconds. We will clarify this in Section 6.

Ablations on prompt length

We thank the reviewer for sharing the insight. To investigate the prompt length effect, we experiment with longer prompts for both TPT and Online TPT. We set the prompt length to 40, which is 10 times longer than the 4-item original “a photo of a”. We consider two types of long prompts: (A) 10 times copy of “a photo of a”, (B) A prompt generated by GPT-4o: “Let us solve an image classification task: a photo of a distinct object, animal, plant, or scene, captured in diverse environments and representing meaningful categories. Carefully analyze its features; the exact category of the photo is a”.

As shown in the following table, longer trainable prompts do not solve the problem of prompt collapse for online learning, even worsening the problem. Since the online testing fails due to error accumulation and prompt collapse, simply improving the length of the prompts does not help. Specifically designed long prompts (B) perform better on the optimization-free CLIP model. However, it may lead to more difficult optimization for test-time tuning, resulting in worse TPT performance. By contrast, based on the prompt selection and appending strategy, our method achieves better performance while reducing error accumulation and prompt collapse. We added the experiments in Appendix C.

Questions

Prompt deleting

In our method, deleting means the entire prompt is removed from the buffer, we clarified the text accordingly.

Dear Reviewer qkjX,

We sincerely thank you for the insightful review. We appreciate the time and effort you put into reviewing our work. We have carefully considered your comments and made improvements based on your suggestions. As the discussion period will end in the next two days, please feel free to let us know if you have any further comments. We are willing to engage in further discussion.

Best regards,

Authors

Dear reviewer qkjX,

I hope this message finds you well. Thank you for your time and efforts in reviewing our submission. Your insights and expertise are greatly appreciated.

We submitted our rebuttal on November 20 and value your evaluation and feedback. As the discussion period is nearing its conclusion in two days, we kindly follow up for your review of our response.

Please feel free to let us know if you have any additional questions to discuss. We are more than willing to provide further clarification or engage in discussion to address any concerns.

Best regards,

Authors

We thank Reviewer tKkt for the constructive feedback and insightful comments. We hope to address the concerns of the reviewer with the responses below.

Weaknesses

Effect of initial prompts

Thank you for sharing the insight. We conducted experiments on ImageNet-A using various initial text prompts. As shown in the following table, the initial prompts affect the performance of CLIP (Radford et al. 2021), TPT (Shu et al. 2022), as well as our method. The reason can be related to the initial predictions of the original CLIP model. Nonetheless, our method consistently outperforms TPT, showing robustness despite variations in initialization. We added this experiment and discussions in Appendix C.

Multimodal test-time prompt tuning

We clarify that our approach can be evolved into a multimodal setting based on MaPLe. When integrated with MaPLe, our dynamic test-time prompt tuning is already applied on both the textual and visual branches. We clarified the corresponding implementation details in Section 6.

Questions

How to set sample orders during dynamic prompt tuning

We shuffle the sample order in the dataloader with different random seeds at test time, which leads to different sample orders. We added this description in Section 6 and will release the corresponding PyTorch code.

Thanks for your responses. Most of my concerns have been addressed and I would like to maintain my original rating. This is a meaningful work, which tends to be accepted.

Thanks for your updates and encouragement. Your suggestions have helped us improve the manuscript.

We thank Reviewer FeBK for the constructive feedback and insightful comments. We hope to address the concerns of the reviewer with the responses below.

Weaknesses

Comparisons with AdaPrompt

Thank you for pointing us to this related work on AdaPrompt by Zhang et al. We provide methodological and performance comparisons:

1. Methodological Comparisons: AdaPrompt performs test-time prompt tuning on a batch of 64 test samples per step, leveraging a buffer to store confident, class-balanced samples for improved tuning and prediction. In contrast, our method dynamically selects and tunes prompts for each single test sample using augmentations.

2. Performance Comparisons: As the benchmarks in AdaPrompt are not exactly the same as in our paper, we reproduced the method on the missing datasets using their released code. The comparisons are shown in the following tables, with reproduced results indicated in italics. Our method performs competitively in the cross-dataset setting and outperforms AdaPrompt in the domain generalization setting. This performance gap in the domain generalization setting may arise from the large-scale label space (1000 for ImageNet-V2/S and 200 for ImageNet-A/R), which prevents AdaPrompt's sample buffer from storing class-balanced test samples, leading to degradation. By contrast, our method dynamically tunes the prompt for each sample with its augmentations, therefore achieving consistently good performance.

We highlighted AdaPrompt in Related Work (Section 5) and added the comparisons in Section 6.

Dynamic prompt selection metrics and strategy

Indeed, the proposed two metrics are designed on prediction confidence, but they measure different properties of the model predictions.

1. Prediction entropy measures the relevance of the prompt and the sample: A lower prediction entropy means the prompt is more confident in its prediction (Niu et al. 2022; Zhang et al. 2024), indicating the prompt has more prior information about the sample. In this case, the prompt and sample are more relevant to each other.

2. Probability difference measures the prompt’s sensitivity to the sample augmentations: A larger probability difference means the prompt is more sensitive to the sample augmentations, which implies the prompt predictions are more diverse and effectively avoids over-confident prompts.

We clarify that since we use the measurements of the initial prompt $v_0$ as thresholds to select subsets for both metrics, the selections are independent. By taking the intersection of the two subsets, the selected prompts have both lower entropy and larger probability differences. Therefore, no matter our selection with the two measures, be it separately or sequentially, the results are the same, simultaneously satisfying both conditions. We included these discussions in Section 4.

Details of data augmentation

We follow the typical data augmentation strategy in TPT (Shu et al. 2022). Specifically, we use AugMix (Hendrycks et al. 2020) to augment the original test image into 63 different augmentation samples, leading to 64 samples in total for each test image. Each test image is first augmented by “resize” and random “crop”, then fed into the AugMix strategy with several augmentation methods including auto contrast, equalization, posterization, rotation, solarization, shearing, and translating. We added the data augmentation clarification in Appendix B.

Hendrycks D, Mu N, Cubuk E D, et al. AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. ICLR 2020.

Dear Reviewer FeBK,

We sincerely thank you for the insightful review. We appreciate the time and effort you put into reviewing our work. We have carefully considered your comments and made improvements based on your suggestions. As the discussion period will end in the next two days, please feel free to let us know if you have any further comments. We are willing to engage in further discussion.

Best regards,

Authors

Dear reviewer FeBK,

I hope this message finds you well. Thank you for your time and efforts in reviewing our submission. Your insights and expertise are greatly appreciated.

We submitted our rebuttal on November 20 and value your evaluation and feedback. As the discussion period is nearing its conclusion in two days, we kindly follow up for your review of our response.

Please feel free to let us know if you have any additional questions to discuss. We are more than willing to provide further clarification or engage in discussion to address any concerns.

Best regards,

Authors