DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation

We appreciate the reviewer's feedback and address the specific concerns raised:

Q1. Hyperparameter Analysis

We determine hyperparameters using four disjoint validation corruptions from ImageNet-C and CIFAR10-C. The same hyperparameters (detailed in Sec 4.1, Appendix C.3) are used across all experiments. Additional results on ImageNet-C:

1. Impact of Temperature τ

The temperature τ in Eq.6 controls weight assignment softness. We evaluated τ in [0.1, 10.0]:

DPCore is stable for τ between 1.0 and 5.0. We used τ=3.0 for all experiments.

2. Impact of Update Weight α

We use exponential moving average in updating mechanism (Eq.7). We didn't tune α but set α=0.999 reported in [1] (comparable to ViDA: 0.999, CoTTA: 0.99). We evaluated α in [0.7, 0.999] and found stable performance when α≥0.9:

3. Impact of Coreset Size K

The parameter K is not fixed but grows dynamically as new domains are encountered, better aligning with real-world scenarios where the number of unseen domains is usually unknown. It is not a hyperparameter that needs to be specified but rather an outcome of the algorithm. More results and discussion can be found in Fig.3b and our response to Reviewer te3N Q4.

Q2. Contribution of Visual Prompt Adaptation (VPA)

We agree that VPA itself is not novel, as noted in our paper with citations. Our key contribution is not the VPA component in isolation but how it is integrated within the dynamic coreset framework:

1. VPA as a Practical Component: While not new, VPA offers a lightweight, efficient adaptation mechanism well-suited for our dynamic coreset approach. As shown in Table 3 (Exp-1), using VPA alone achieves only +5.0% improvement, but when combined with our dynamic coreset, it achieves +15.9%.
2. VPA is Replaceable: Table 3 (Exp-3) shows our dynamic coreset approach isn't dependent on VPA. Replacing VPA with NormLayer parameters still achieves strong performance (+10.7%), highlighting that our main contribution is the dynamic coreset approach rather than VPA.
3. Efficient Domain Alignment: Our VPA implementation offers computational advantages in changing environments. It requires minimal parameters (0.1% of model parameters) and few source examples (300 unlabeled), making it practical for real-world deployment.

In summary, while VPA itself isn't our core contribution, its integration in our dynamic coreset framework is key to DPCore's effectiveness. Our novelty lies in how we manage domain knowledge through the dynamic coreset, not in the specific adaptation method (e.g., VPA).

Q3. Updating Mechanism Concerns

Our updating mechanism is novel and effective for several reasons:

1. It works in tandem with our objective function (Eq.5), which creates a meaningful mapping between feature statistics and prompts, effectively reflecting the domain gaps. Empirical results support this: Table 5 shows that prompts learned for similar domains (with similar distances) transfer effectively between them; Fig.3a shows our prompts consistently reduce domain gaps; Fig.3c confirms stability across diverse domain orders.
2. The mechanism only activates when the current batch belongs to a seen domain or is similar to seen domains. For completely new domains (with large domain gaps), it is not used. Instead, existing core elements stay unchanged while a new prompt is learned from scratch and added to the coreset to prevent negative transfer.
3. Evaluating each prompt separately and selecting the closest one poses challenges (Sec 3.3): 1) It processes same test batch multiple times (linear to the coreset size), increasing computation. Linear combination reduces computation to constant time (only once); 2) Nearest neighbor selection might diminish coreset power, particularly for unseen domains which could be viewed as decompositions of known domains; 3) Increasing temperature τ to prioritize the nearest neighbor does yield heavier weighting, but performance drops (see Q1.1 τ table).
4. The linear combination used in Eq.7, while simplifying domain relationships, is highly effective. Table 1 and Fig.4b show DPCore consistently outperforms more complex methods across both CSC and CDC settings. We chose this approach for its efficiency in handling not only visual prompts but also other parameters (e.g., NormLayer parameters from Q2.2), balancing effectiveness and simplicity, and achieving SOTA performance with computational efficiency.

We appreciate the reviewer's feedback on our work. We believe DPCore makes a significant contribution to continual test-time adaptation, particularly for the challenging CDC setting that better reflects real-world scenarios.

[1] "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results" NeurIPS2017

We sincerely thank the reviewer for their positive feedback and insightful questions. We appreciate the thorough review and the recommendation to accept our paper. Below, we address the specific questions and concerns raised:

Q1. Memory Efficiency Analysis

In Lines 216–219, we compare with [1] where test samples are directly stored in a buffer. Our approach stores only test batch statistics, requiring significantly less space (e.g., from 64×3×224×224 per batch to just 2×768, about 0.02% of the original size). Additionally, our memory analysis on ImageNet-C using a single RTX 6000 Ada GPU:

DPCore achieves minimal memory footprint by 1) storing only learned prompts (0.34 MB total) and compact statistics, 2) requiring no test sample storage, and 3) sharing core elements across similar domains.

Q2. Sensitivity Analysis of ρ Across Datasets

We've analyzed ρ sensitivity across datasets, with results in this figure. DPCore maintains stable performance on ImageNet-C, CIFAR10-C, and CIFAR100-C for ρ values between 0.7 and 0.9. We fix ρ=0.8 for all main experiments.

Q3. Potential Failure Cases

This is an excellent question that helps us better understand the limitations of our approach. However, "Overlapping Distributions", "Noisy or Small Domain Shifts" and "Inconsistent Weighting Effects" are not necessarily failure cases for our method. The goal of DPCore is not to learn an identical number of prompts as the number of domains or to identify all different domains. Instead, DPCore aims to update the same prompt for similar domain groups. Specifically:

1. This design is motivated by our findings in Sec 3.3 and Appendix D.1, where we observed that a prompt learned on one domain (e.g., Gaussian Noise) could work effectively on a similar domain (e.g., Shot Noise), and performance could be further improved by slightly updating the prompt on the second domain.
2. Since distance-induced weights are used to generate the weighted prompt, a new domain can be effectively represented as a decomposition of existing domains. The weighted prompt might perform well on a new domain even when derived from different domains, which is acceptable in our framework since domain similarity is evaluated by distance and prompts are learned through distance minimization.

One potential failure case for our method would be when each test batch contains data from multiple domains. In this scenario, the statistics of each batch would not be stable, and our method might treat each batch as a different domain. This would reduce efficiency since the algorithm would need to learn prompts for each test batch from scratch, similar to what we show in Table 3 Exp-2 (learning prompts from scratch for each batch but the entire batch comes from the same domain). This scenario would pose challenges to most CTTA methods, as most assume each test batch contains data from a single domain. We plan to explore this direction in future work.

Q4. Memory Optimization for Prompt Coreset

Regarding memory overhead concerns, maintaining prompts in memory is quite negligible in practice. For example, the 14 prompts learned for ImageNet-C domains only require 0.08M parameters (0.34MB). If we were to allocate the same number of parameters as ViDA (7.13M), we could store approximately 1,247 prompts, making DPCore highly memory-efficient in practice.

While our current implementation has modest memory requirements, exploring potential optimizations aligns well with practical deployment considerations. We have explored two approaches to maintain a fixed-size coreset on ImageNet-C with K=15 (matching the number of domains). Once the number of prompts exceeds K, we either 1) discard the oldest prompt or 2) merge the most similar prompts (computing the average of the two prompts whose statistics distance is smallest). We evaluated these strategies in the 10-different-order setting (Fig.3c) and report the average error rates:

Both strategies still significantly improve upon the source model, though merging existing prompts performs slightly better than simply discarding one of them.

In real-world scenarios, the number of domains is typically unknown, and fixing K at a small value could lead to suboptimal performance. Since the coreset grows only when encountering unseen domains and the memory overhead is negligible, our default approach is to allow K to evolve naturally as adaptation progresses.

We appreciate the reviewer's thoughtful questions, which have helped us articulate our approach's strengths and limitations. We believe addressing these points will further strengthen the paper.

[1] "Learning to Prompt for Continual Learning" CVPR 2022

We sincerely thank the reviewer for their careful reading of our paper and constructive feedback. Below, we address the specific concerns and questions raised:

Q1. Source Data Requirement

We appreciate this concern and would like to clarify:

1. DPCore requires source data only before adaptation starts. In practice, feature statistics could be provided by the model publisher alongside the pre-trained model. Moreover, several SOTA methods (e.g., ViDA, EcoTTA, VDP, DePT, BeCoTTA) also require source data for preparation. The key difference is that these methods need to warm-up their parameters on the entire source dataset for several epochs, which demands significantly more data and computation than our approach. They require both forward and backward propagation on the entire labeled dataset, whereas our method needs only a single round of forward passes on 300 unlabeled examples to extract features.
2. When no statistics are provided during preparation, they could be computed from the test data stream. In real-world scenarios, ID data is usually mixed with OOD data in the test batch stream. Filtering out 300 ID samples is straightforward using simple metric thresholds such as entropy.
3. As demonstrated in Fig.5c, our method's performance remains stable even with as few as 50 unlabeled source examples.
4. Most importantly, we show in Fig.5c and Appendix F.2 that DPCore can function effectively without ANY source data by using public datasets (e.g., STL10) as proxy reference data. Even in this extreme scenario, DPCore still outperforms the source model by +10.2% on ImageNet-C.

Therefore, while we technically use source examples during preparation, our approach is source-data free during test time and can even operate without source data entirely, making it practical for real-world scenarios with limited or no access to source data.

Q2. Performance on CNNs

While our paper primarily focused on ViTs due to their strong representation capabilities, we have conducted additional experiments on ResNet for the datasets mentioned. The choice of models for each dataset follows EcoTTA. For CNNs, instead of learning visual prompts, the coreset stores NormLayer parameters (same approach as Table 3 Exp-3):

These results demonstrate that DPCore consistently outperforms existing methods across all CNN architectures and datasets, confirming our approach's effectiveness beyond ViT architectures.

Q3. Novelty of Dynamic Prompt Coreset

The dynamic prompt coreset approach in our paper is indeed novel for test-time adaptation. While coresets have been explored in various machine learning tasks (e.g., continual learning), their application to managing prompts in TTA scenarios has not been previously investigated. The key novelty lies in:

1. The design of a coreset specifically for storing and managing visual prompts in dynamic CTTA. Prior methods continuously update the same parameters across different domains, which leads to convergence issues with brief domain exposures, risks forgetting previously learned knowledge, or misapplies it to irrelevant domains. DPCore manages domain knowledge through a dynamic prompt coreset to mitigate these issues.
2. The dynamic coreset approach is not restricted to prompts but can also be used for other types of parameters such as NormLayer parameters. It still improves the performance of the source model by +10.7% when applied to NormLayer parameters (Table 3 Exp-3). These results show that our dynamic coreset can be applied to other model architectures beyond ViT and integrated with other TTA methods.
3. Our novel decision mechanism determines whether to update existing prompts or create new ones based on domain similarity, mitigating negative transfer between dissimilar domains while allowing coreset prompts to be updated on similar ones. This approach leverages domain similarities through weighted prompt generation, computing efficient combinations that ensure constant evaluation time regardless of coreset size.

Related work such as [1] explores prompt management for supervised continual learning, but requires labeled data and addresses a fundamentally different problem (discussed in lines 216–219). Similarly, online K-means clustering has been used in various applications, but our adaptation of these ideas to prompt-based CTTA with a dynamic update mechanism is novel.

Others

We will correct the Table 3 error (gain for Exp-3 should be +10.7, not +8.7) in the camera-ready version. The code will be published upon acceptance.

We appreciate the thorough review and constructive feedback and hope these clarifications address all concerns.

[1] "Learning to Prompt for Continual Learning" CVPR2022

We thank the reviewer and address each point:

Q1. Value of Proposed CDC Setting

Our CDC setting models real-world scenarios with irregular distribution shifts where domains recur unpredictably (Fig.1), unlike existing CTTA approaches that assume uniform changes. For example, an autonomous vehicle driving through a mountainous area would encounter brief tunnels, extended sunny stretches, and intermittent fog or rain with unpredictable patterns and durations. SOTA methods degrade significantly in CDC (ViDA's error rate increases from 43.4% to 52.1%), highlighting that CDC presents new challenges not adequately addressed by existing settings. CDC amplifies three critical limitations in existing settings: convergence issues with brief domains, catastrophic forgetting with irregular patterns, and negative transfer between dissimilar domains.

These findings suggest CDC better reflects real-world challenges and provides a rigorous testbed for evaluating CTTA methods. Multiple reviewers recognized our approach: te3N noted CDC is "practical and aligns well with real-world dynamic changes" and "the proposed new setting is valuable" while jKst called it "novel and interesting." These independent assessments validate our motivation. Differences between CDC and existing settings (e.g., SAR) are discussed in Appendix E.2.

Q2. Visual Prompt (VP) Implementation Details

We follow standard shallow VP implementation from [1], prepending prompt tokens to image tokens after the CLS token (Fig.2). The CLS token is invariant to the location of prompts since they are inserted after positional encoding ([1]). We explored using entropy minimization to learn VP at test time before settling on distribution alignment, but it led to degenerate solutions (increasing error rate by ~10% over the source model), which aligns with findings in recent work ([2]).

It's important to note that our main contribution is not how to learn VP, but rather DPCore's ability to manage domain knowledge through a dynamic coreset for CTTA. Our method remains effective when VP is replaced with NormLayer parameters (Table 3 Exp-3). For additional details, please see our response to Reviewer jKst's Q2.

Q3. Relationship between Feature Statistics and Prompts

While feature statistics and prompts exist in different spaces, our objective function (Eq.5) creates a meaningful mapping between them through optimization. When we learn a prompt to minimize the distance between source and target statistics, we're effectively creating a prompt space that aligns with the statistical relationships in feature space. Our empirical evidence supports this approach: Table 5 shows prompts learned for similar domains transfer effectively between them; Fig.3a demonstrates our prompts consistently reduce domain gaps across all corruption types; and Fig.3c confirms stability across diverse domain orders. Essentially, the optimization creates a reliable bridge between statistical relationships and prompt functionality.

Q4. Comparison with SAR

There's a misunderstanding in the comparison. SAR addresses single domain (evaluating each domain independently) while we focus on continually changing domains (15 domains as one sequential task). SAR isn't designed for CTTA, so its performance drops. We use consistent hyperparameters (e.g., batch size=64) across all methods for fair comparison (detailed in Appendix B).

The three challenging settings in SAR are originally designed for single domain, not changing CTTA scenarios. But we further address these for CTTA in Appendix E.3 and F.1:

1. Batch size=1: Fig.5d shows all CTTA methods struggle with small batches in changing domains. Our DPCore-B variant (Appendix F.1) addresses this by using a negligible buffer to accumulate sample features, achieving 41.2% error rate with single-sample batches (vs. 39.9% with batch size=64).
2. Imbalanced label shifts: Though our method wasn't specifically designed for this challenge, it achieves better performance in this setting (improving the error rate to 43.9% as shown in Table 10).
3. Mixed domain: In the CTTA setting, batches containing data from different domains usually occur near domain boundaries. We verified our method in this case with DPCore-B, which can still achieve an improvement of +14.6% over the source model (Table 11).

Q5. Computational Efficiency Concerns

We have carefully analyzed our method's computational efficiency (Table 4 and Appendix F.3). While DPCore does require additional computation compared to some simpler methods (e.g. Tent), it remains significantly more efficient than many SOTA approaches (e.g., CoTTA, ViDA). Our method requires a similar number of backpropagation operations (~1) but significantly fewer forward propagations (3.1) than CoTTA (11.7) and ViDA (11.0), which require forwarding extra augmented test data.

[1] "Visual Prompt Tuning" ECCV2022

[2] "Test-Time Model Adaptation with Only Forward Passes" ICML2024