Echoes of the Visual Past: Test-Time Prompt Tuning with Multi-Scale Visual Memory

We thank the reviewer for the constructive comments and suggestions. We appreciate the reviewer’s recognition of the great performance improvements achieved by our work. Below, we respond to all weaknesses (W) and questions (Q) point by point.

Q1: How is multi-scale visual memory initialized?

The multi-scale visual memory is empty at the beginning of the test phase. It accumulates visual memory progressively as test data streams in. Once the memory reaches its capacity, the patch with the lowest confidence is removed to maintain the fixed memory size.

Q2 & W1: There is a lack of explanation for the methods between lines 145 and 151. The authors should explain how these methods from [45, 46, 49] contribute to their work.

For [45], we adopt its proposed exponential scaling function to convert the computed similarity in Eq. (3) into non-negative values and to modulate the sharpness of the similarity scores. For [46] and [49], we follow their approach to take into account the relationship between the memory and the test image when using memory for prediction. We will further clarify these connections in the revised manuscript.

Q3 & W1: The author should strengthen the checking of symbolic definitions. For example, the size of the matrix P_{ada} is not given; L_{pt} in Formula 5 is not defined.

p_{ada} denotes the optimized textual prompt. Its size depends on the embedding size of the CLIP backbone used, and remains constant throughout the process. While not explicitly specified, its size does not affect other mathematical formulations in the framework. As for L_{pt}, it refers to the prompt optimization loss, which is defined on the right-hand side of Eq. (5).

Q4 & W1: The author needs to confirm the correctness of the formula. In Eq.8, p(y|v,p_{ada}) represents the probability of belonging to y, which should be a value rather than a distribution.

Thank you for pointing out the typo. The expression in Eq. (8) should be $P$($\mathbf{y}$|$\mathbf{v}, \mathbf{p}_{ada}$) ($P$ instead of $p$). It denotes the predicted distribution over classes with the adapted prompt.

Q5 & W2: The author believes that the bottleneck of existing methods is that they only use visual information from the current image, and introduces memory to solve this problem. To fully illustrate this problem, the author's baseline should add a multi-scale patch from the current image to illustrate the gain brought by memory through historical information.

We have added such a baseline, named Multi-scale Patch, in the table below. This baseline uses multi-scale patches from the current test image for prompt tuning. The comparison demonstrates that our framework achieves notable performance improvements, confirming the effectiveness of the historical visual memory.

Dear Reviewer e7b1,

We sincerely appreciate your valuable feedback and the time you have taken to review our work. As the discussion period will end within 48 hours, we just wanted to kindly check if you might have any further questions or points you would like to discuss.

We would be very happy to provide additional clarification or details should you need them.

Best regards,
The Authors

Dear Reviewer e7b1,

Thank you again for your valuable review of our work. We understand the concerns raised in your initial review, primarily including the need for an ablation study, further explanation of some related works, and a formula typo. We have addressed each of these points in our rebuttal by adding the requested ablation study and providing detailed clarifications.

We just wanted to follow up to see if you have any further questions or comments. We would be happy to discuss them here before the end of the discussion phase.

Best regards,
The Authors

Dear Reviewer e7b1,

This is the last call for author-reviewer discussion, which will ends on Aug 8. Could you please read authors' rebuttal and confirm whether your concerns have been addressed asap? Thank you.

Best,

AC

We thank the reviewer for the constructive comments and suggestions. We appreciate the reviewer’s recognition of our work for its novelty, thorough ablation studies, and clear writing. Below, we respond to all weaknesses (W) and questions (Q) point by point.

W1: Could the final results be sensitive to the order of test-time inputs? If so, this would compromise the robustness and reproducibility of the method, as test-time performance might fluctuate depending on input ordering.

We have included an analysis of different input test data orders in Appendix B, and we present the results in the table below. Specifically, we run three experiments with different random seeds that change the input test data order. The results show acceptable variance. We will also add a note in the main body of the paper. That said, we acknowledge a relevant limitation, discussed in Appendix D: memory-based methods initialize the memory with early input data, so a high volume of low-quality or difficult early samples may lead to suboptimal memory updates and introduce significant noise into the memory.

W2: If mispredictions occur early in the test phase, they may lead to cascading memory contamination, where future retrieval is misled by accumulated errors, reinforcing incorrect representations. How does the method mitigate this risk?

Memory-based methods indeed face the risk of memory contamination due to mispredictions during test-time updates. Unlike previous methods that update the memory using fixed CLIP representations, our method performs memory updates after adapting CLIP via prompt tuning to the current test data. This reduces the risk of early-stage mispredictions overwhelming the memory and leads to more robust memory updates. Experimentally, our proposed method is superior compared to DMN[46] and TDA [18] where fixed CLIP representations are utilized to construct the memory.

W3: The framework appears to lack a long-term memory management strategy, specifically a limit on memory size and a mechanism for discarding outdated samples. Without such mechanisms, there is a risk that the memory will become bloated or conflicted.

We would like to clarify that our method does include a memory management mechanism with a fixed memory size of 50 entries per class, as described in lines 168–170 of the Method section and in the Implementation Details. During the memory update step, if a class’s memory has not reached its capacity, the current sample is stored directly. If the memory is full, we remove the sample with the lowest prediction confidence (highest entropy) among the current and existing entries.

Q1 & Q2: How do additional memory overhead and runtime introduced by the memory part (including memory retrieval) increase with class number increasing? Would a large number of classes lead to retrieval accuracy degradation?

We analyze the additional memory overhead and runtime introduced by the memory component (including retrieval) on both low- and high-class-count datasets, shown in the table below. As expected, the overhead increases with the number of classes. However, it remains a small fraction of the overall system cost compared to the prompt tuning part, which adds ~15.8 GB when scaling from 47 to 1000 classes.

Memory retrieval accuracy influences the contribution of the memory part to the overall system. To assess whether memory retrieval accuracy degrades as the number of classes increases, we report the performance gains from the memory part across datasets with varying class counts. The table below shows consistent improvements, suggesting that memory retrieval remains effective and scalable even with 1000 classes.

Q3: The method seems to assume a closed-set scenario. This limitation should be discussed, especially in the context of real-world open-set scenarios.

We have noted this limitation in the Limitations section: this work addresses the closed-set setting, in line with most previous test-time prompt tuning methods. The current framework does not consider open-set test data streams. As future work, one promising direction is to incorporate recent out-of-distribution (OOD) detection techniques, which can identify and filter out samples not belonging to the predefined classes [1].

[1] Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey. In Transactions on Machine Learning Research. 2025.

Dear Reviewer rbTk,

Thank you for your response and for your thoughtful feedback throughout the review process. We're glad to hear that our rebuttal addressed your concerns and deeply appreciate that you remain positive on our submission. If you have any further questions or thoughts, we’d be happy to discuss them before the end of the discussion phase.

Best regards,
The Authors

We thank the reviewer for the constructive comments and suggestions. We appreciate that the reviewer finds our work to have good performance and fair comparisons. Below, we respond to all weaknesses (W) and questions (Q) point by point.

W1: I found this paper quite hard to digest; it introduces a convoluted method with many subcomponents, which add a non-negligible complexity. I would suggest moving the pseudocode of the appendix to the main body of the paper;

We will follow this suggestion and move the pseudocode to the Method section to improve readability. Moreover, we have revised the overview figure to enhance the correspondence between the figure and the descriptive paragraphs, making the method easier to follow. Overall, our proposed method follows the procedure of visual memory retrieval (patch or holistic) -> prompt tuning with suppression -> memory updates. We hope it makes the overall pipeline more clear.

W2_1 & Q1: Report computational burden on other datasets, like ImageNet-1k. W2_2 & Q1: Report computational burden of backpropagation-free TTA methods apart from test-time prompt tuning methods.

W2_1 & Q1. We report the computational burden on ImageNet-1K in the table below. Compared to Table 3, all prompt-tuning methods consume more resources as the number of classes increases due to the increased cost of backpropagation through the text encoder. However, the results still support our claim that M²TPT significantly improves performance with only moderate additional cost—e.g., only 9% additional memory compared to TPT [38].

W2_2 & Q1. We also report the computational burden of recent backpropagation-free test-time adaptation (TTA) methods. We acknowledge that these methods are computationally cheaper than prompt-tuning-based methods. However, we pursue the prompt-tuning paradigm in this work for two main reasons:

• Automation and practicality beyond manual prompt engineering. As shown in Fig. 1(a), prior backpropagation-free methods often outperform TPT on CLIP, largely because they use hand-crafted or LLM-generated prompts. However, such prompts lack automation and scalability, requiring manual engineering for each test scenario and prior knowledge of the test data [38][48]. In contrast, TPT methods enable on-the-fly adaptation to unseen test data. In addition, please note that the effort and time consumed for constructing or searching proper and more optimal prompts is always discarded and not included in the table.
• Limitations of previous prompt-tuning methods. Prior TPT methods rely on limited visual information from the current test image, which restricts the adaptability of the learned prompt. Our method addresses this by enhancing prompt learning with past visual memory, fully exploiting the potential of prompt tuning at test time.

[38] Test-time prompt tuning for zero-shot generalization in vision-language models. NeurIPS2022.
[48] Learning to prompt for vision-language models. IJCV2022.

W3: I think this work is missing a discussion about some important related works on efficient TTA [a, b, c] and, potentially, respective comparisons. I believe this is especially critical since in the past year, the community has observed that simple, optimization-free baselines can work very well for TTA.

To enable a comprehensive comparison, we present results for these methods [a, b, c] on all datasets—with and without prompt engineering (PE) (i.e., hand-crafted and LLM-generated prompts). The table shows that our method consistently outperforms others on average, regardless of whether PE is used.

Notably, M$^2$TPT without PE outperforms ZERO + PE, TPS + PE, and MTA + PE, showing that test-time learned prompts can surpass human-crafted ones. While prompt tuning involves additional computation, it is more practical in real-world scenarios where manually engineering prompts for every test domain is infeasible [38][48]. As noted in the Introduction, prior prompt-tuning methods fall short compared to prompt-engineering-based methods. Our work advances test-time prompt tuning and demonstrates its ability to outperform strong manually crafted prompt baselines.

[38] Test-time prompt tuning for zero-shot generalization in vision-language models. NeurIPS2022.
[48] Learning to prompt for vision-language models. IJCV2022.

W4 & Q2: The paper contains no explanation about how the hyperparameters of M2TPT were selected

Following TDA [18], we select hyperparameters using the ImageNet validation set and fix them across all datasets. For $\beta$, we choose a relatively small value to maintain a low weight for the irrelevant suppression loss when the number of classes is small. We also include a sensitivity analysis on 10 fine-grained datasets, shown below. The performance varies only slightly, indicating that our method is not highly sensitive to $\beta$.

W5: I would suggest considering different initializations (i.e., sticking to the ViT-B/16 vision encoder, but using OpenCLIP models [d]), and/or scaling to ViT-L/14 as well.

We follow your suggestion and scale our evaluation from ViT-B/16 to ViT-L/14 using the plain prompt and unchanged hyperparameters. The results below show that M$^2$TPT achieves a 2.77% average improvement over other methods across 10 datasets, indicating robust performance gains across model scales. We will include it in the revision.

Dear Authors,

Thank you for the detailed rebuttal! Let me reply below:

Computational complexity of M2TPT. Thank you for reporting these numbers! Unfortunately, as anticipated, what puzzles me is the increased computational cost of M2TPT compared to recent efforts in TTA. According to the numbers above, compared to recent efforts such as DMN and MTA, the proposed solution increases both space and time requirements by significant margins ($4.3\times$ and $7.5\times$ slower than MTA and DMN, respectively, while consuming $5.9\times$ and $9\times$ more memory). I understand the merits of increased accuracy, but the computational requirements look a bit too excessive to me. I am aware that "the effort and time consumed for constructing or searching proper and more optimal prompts is always discarded and not included in the table" when it comes to related works, but I believe this is for a valid reason, i.e., such a cost is amortized after deployment (if you think of it, for research benchmarks this was done once in the CLIP paper and never again in a multitude of follow-up works). This is not the case if inference on a single test-point is significantly more demanding, as the gap in compute spent will constantly increase over time.

M2TPT vs efficient TTA baselines. Thank you for reporting these numbers! For the "Fine-Grained (Avg of 10)" column, it looks like these results were directly reported from the provided references, both with and without PE. I guess there's a big caveat here, which is that both [a] and [b] (MTA and ZERO) do not use dataset-specific templates, but rather just ensemble a set of generic prompts provided by the CLIP authors for ImageNet alone. In contrast, this paper claims to use the same templates of DMN, which are dataset-specific. This is, unfortunately, not a fair comparison and gives an advantage to the method proposed in this paper. I think it would be beneficial to examine such a comparison.

About all other points, I have no further comment, and I appreciate your responses. The two aforementioned points are, unfortunately, what make me refrain from switching from a reject to an accept score at the current stage.

All the best,
Reviewer D3JX

We sincerely thank the reviewer for the continued engagement and thoughtful feedback. We greatly value the reviewer’s perspective on the importance of presenting a comprehensive view of the field, including alternatives with their respective strengths and limitations. Although this paper focuses on the prompt-tuning line of work, we have already included substantial discussion of the training-free TTA line in the Introduction, Related Work, and Experiments sections. In addition to the existing discussion, we will also provide a detailed comparison and discussion of both efficiency and accuracy between the two lines of work in the main paper.

We would also like to clarify that the performance improvement achieved by our work is not minimal but significant. Specifically, compared to MTA and ZERO, M$^2$TPT shows a 3.18% average accuracy improvement across 15 datasets without PE. Compared to the most recent SoTA test-time prompt tuning method (DynaPrompt), M$^2$TPT without PE achieves a 2.43% average improvement across the same 15 datasets, while also requiring much less computational overhead.

We agree that both accuracy and efficiency are important across all lines of work, and it is expected that the community will continue to advance on both fronts. Ultimately, it is the end user who decides which method to use—depending on the application, even a 1% accuracy improvement might justify additional computational cost. Our method primarily focuses on improving accuracy, with less emphasis on efficiency. As future work, we aim to better integrate training-free and optimization-based approaches, fostering their joint development to achieve a more balanced and practical trade-off between accuracy and efficiency.

We thank the reviewer for the constructive comments and suggestions. We appreciate that the reviewer finds our method novel and mathematically well-grounded, with rigorous logic. Below, we respond to all weaknesses (W) and questions (Q) point by point.

W1: The overall pipeline figure is overly complex and lacks clarity, making it hard to follow the method at a glance.

We have improved the overview figure by making the three components—memory retrieval, prompt tuning, and memory update—more modular and visually distinct. Additionally, we enhanced the correspondence between the illustration and method descriptions by adding equation indices to the figure, thereby improving readability.

W2: The descriptions of the two main improvements (holistic memory and irrelevance suppression) are too brief, making their motivations and contributions difficult to fully grasp.

We first introduce the motivations for these two components in detail in lines 57–67 of the Introduction. These motivations are then recalled and reinforced in the Method section—specifically in lines 182–186 and lines 198–201. We would like to add detailed descriptions as:

The holistic memory complements the multi-scale memory, particularly for holistic visual recognition tasks such as scene classification. These tasks require global visual descriptions, which may be overlooked by using only multi-scale memory. To address this, we design the holistic memory to provide comprehensive image-level context for each class during memory retrieval.

The irrelevance suppression component aims to mitigate the adverse effects of high-confidence mispredictions during test time. Since the memory retrieval and update processes rely on test-time predictions, the visual memory is inevitably noisy due to prediction errors. To reduce the impact of this noise, we first filter out low-relevance memory entries during retrieval. Furthermore, we construct a class-irrelevant memory to store the most “conspicuous” noise—i.e., high-confidence but incorrect patches—and suppress their influence during prompt tuning.

Moreover, we will add a more detailed mathematical formulation of the holistic memory in the Appendix. Additionally, we include a pseudocode of the full pipeline in Appendix A, which we will move to the main body to enhance clarity and accessibility.

Q1: Why does the method ultimately choose to use only one memory from multi-scale and holistic memory rather than integrating both?

The use of holistic memory is motivated by tasks requiring global visual understanding. In such tasks, relying on local patches (as in multi-scale memory) may lead to low-confidence or distracting cues. For instance, in scene classification which relies on global visual description, an isolated object patch may introduce confusion due to its limited local context. Therefore, in these cases, our method selects the high-confidence holistic memory and disregards the multi-scale one—and vice versa—depending on the entropy-based confidence measure.

Q2: In the comparison with prompt-tuning-based methods, the proposed method still underperforms on ImageNet-V2 and ImageNet-R. Could the authors provide insights into the potential causes of this gap, and whether there are ways to further improve performance on these challenging datasets?

We hypothesize that the performance gap arises from the high intra-class variation in datasets like ImageNet-R. Such variation can adversely affect the memory retrieval process, which relies on visual similarity. To mitigate this issue, future work could explore strategies to explicitly disentangle intra-class variation from inter-class differences, enhancing the robustness of memory retrieval.

Dear Reviewer CxW2,

We sincerely appreciate your valuable feedback and the time you have taken to review our work. As the discussion period will end within 48 hours, we just wanted to kindly check if you might have any further questions or points you would like to discuss.

We would be very happy to provide additional clarification or details should you need them.

Best regards,
The Authors

Dear Reviewer CxW2,

Thank you for your response and for your thoughtful feedback throughout the review process. We're glad to hear that our rebuttal addressed your concerns and deeply appreciate that you remain positive on our submission. If you have any further questions or thoughts, we’d be happy to discuss them before the end of the discussion phase.

Best regards,

The Authors