ReservoirTTA: Prolonged Test-time Adaptation for Evolving and Recurring Domains

We appreciate the reviewer’s constructive feedback and address the main concerns below:

1. “This paper introduces a VGG network and a hyper-parameter, , to identify changes in the domain. It makes stronger assumptions than conventional TTA methods: 1) A pre-trained VGG network that can discriminate between different test domains (which are unknown in advance). 2) a dataset-specific hyperparameter, , which requires source data to be obtained.

We acknowledge this concern and would like to clarify:

Role of the VGG‑19 style extractor. We use a frozen, pruned VGG‑19—pre‑trained on ImageNet1K solely for extracting low‑level “style vectors” (log‑variance of early activations)—to detect domain shifts. This network is not fine‑tuned on any task or dataset, making it broadly applicable. In Appendices D/E, we show our method is robust to which VGG layers are used and that it outperforms ViT‑based identifiers (e.g., CoLA, DPCore) in clustering quality and stability (Figure 16). VGG’s early features capture texture statistics—a property long exploited in style transfer—so we leverage it without tying our framework to any particular task, as our framework is architecture-agnostic. Importantly, all task models (CNNs, ViTs, SegFormers) continue to use standard TTA objectives (TENT, EATA), with VGG only supplying style quantification.

Hyperparameter τ is data‑driven and lightweight. We compute τ automatically as the q‑quantile of pairwise distances among a small subset of source styles (Eq. 7), without labels or full source access. Appendix E (Figure  7) demonstrates τ’s stability across datasets and corruption types. This mirrors prior TTA practices—e.g., TTN tunes BN‑blend weights post‑training (warm-up stage), and ETA/EATA introduce Fisher‑information regularization weights derived from source data (Niu et al., ICML ’22)—so our requirement is no stronger than established methods.

Why not use the source model for style features? Following the reviewer’s comment, we replaced VGG with the source model for style calculation. On CIFAR100‑C with ResNeXt‑29 (CSC scenario), replacing VGG with early layers of the source model raises average error by +2.12 % (see table below). Source backbones lack the broad texture priors of ImageNet‑trained VGG and would tie our method to each source architecture. By contrast, VGG delivers consistent gains across ResNeXt‑29, ViT‑B‑16, and ResNet‑50 (GN) (see Tables 15–18, 20).

In sum, the frozen VGG‑19 style extractor and automatically derived τ introduce minimal, dataset‑agnostic overhead, matching common TTA assumptions, while delivering substantially better domain detection and adaptation performance.

2. “The paper's focus is on mitigating the risk of catastrophic forgetting in long-term TTA. The authors argue that a model pool would enable the use of different model parameters in different domains. However, earlier TTA work has shown that adapting a model for a long time within a single domain can still cause catastrophic forgetting problems. To address this, they introduced tools such as Fisher information and weight ensembling, which constrain the updated parameters to be as close as possible to the source model. Does this conflict with your starting point, where you want to update to the domain optimum within each domain? They constrain the parameters of each domain to be as close as possible to the source parameters.”

Excellent question. In Section 3.2, we analytically show that Fisher regularization or weight ensembling controls parameter variance but cannot resolve domain interference when |θ_Task,i – θ_Task,i+1| > \beta (stability radius), and regularization alone fails to protect previously learned domains.

In contrast, ReservoirTTA partitions adaptation across domains, allowing each model to converge toward its own domain-optimal parameters. Our theoretical results (Theorem 1, Lemma 1) explain how variance grows with adaptation time, and why variance regularization is insufficient alone. Empirically, Table 1 and Fig. 1 demonstrate that even Fisher‑regularized methods like EATA suffer significant performance drops when domains reoccur.

Importantly, we do not abandon regularization—in fact, each reservoir specialist continues to use EATA‑style Fisher penalties (or analogous constraints) during its own adaptation. The key novelty of ReservoirTTA is to complement these measures by maintaining a pool of specialists: regularization ensures stability within each domain, while the reservoir ensures retention across domains (focusing on the forgetting issue).

3. “If you detect domain changes using VGG, what is the performance of simply resetting to the source model?”

We thank the reviewer for this insightful question. In fact, we did evaluate reset‑based baselines to isolate the benefit of reusing adapted specialists versus simply resetting to source parameters:

EATA with blind resets (RDumb). Periodically resets every 1 K iterations, yielding an average error of 32.17 % on CIFAR100‑C (CSC). Domain‑aware resets (RDumb w/ VGG). Triggers the same reset only when our VGG‑based detector signals a domain change, improving to 31.17 % (–1.0 %).

While using VGG to drive resets is more effective than fixed‑interval resets, both discard all previously learned adaptation. As illustrated in Figure 1 (right), EATA’s error actually increases when revisiting “snow” after 15 intervening corruptions, because it has forgotten its prior specialization.

ReservoirTTA (EATA + ReservoirTTA). Instead of resetting, we reactivate the exact specialist previously trained on that domain, avoiding costly re‑adaptation and forgetting. This yields an average error of 28.57 %, substantially outperforming both reset strategies.

This shows that reusing a previously adapted specialist via ReservoirTTA is far more effective than resetting to the source model, even when resets are driven by a perfect domain‑change detector.

Dear Reviewer nnEp,

Thank you once again for taking the time to review our submission and share your insights. As the discussion window approaches its close, we wanted to check in to see if our responses have adequately addressed the points you raised. If there’s anything that remains unclear or could benefit from additional clarification, we’d be glad to elaborate.

We’re grateful for your thoughtful engagement.

Warm regards, The Authors

We thank the reviewer for their constructive feedback. Below, we address some of the concerns:

1. The additional VGG-19 and online clustering increase computation by ~30%... may raise concerns in resource-constrained environments.

We fully acknowledge this concern. While ReservoirTTA does introduce some computational overhead—primarily from the auxiliary VGG‑19 style extractor and the online clustering module—we have designed each component to be as efficient as possible:

Reduced Overhead. As noted in our response to the reviewer SQXd, ReservoirTTA increases the runtime by 30% for extremely lightweight methods like EATA. This overhead is lower than many other TTA schemes—for instance, ROID incurs >1.5× latency versus Tent, while delivering substantially better long‑term stability and accuracy.

Lightweight style extraction. We freeze and prune VGG‑19 down to its first few convolutional blocks, extracting only the log‑variance of feature maps “style vectors” rather than running full forward passes. This is highly parallelizable and light compared to full-model forward passes, and can be extended to faster style‑extraction backbones as an alternative to VGG.

Low‑cost clustering. In practice and real-time systems, the online clustering computations can be offloaded or scheduled asynchronously, as it is not a critical path of forward inference.

Asynchronous execution. In real‑time pipelines (e.g., autonomous driving), style extraction and clustering can run in parallel with sensor data preprocessing or post‑processing tasks, smoothing out latency spikes. When implemented on embedded GPUs or dedicated accelerators, the additional data movement (style vectors ↔ clustering buffer) is minimal—on the order of kilobytes per batch—so DRAM transfers and power draw remain low.

Efficient Triggering. While we currently update the parameters of each specialized domain at every batch, it is reasonable to expect that some domain-specific models may reach a point of convergence. In such cases, continued adaptation offers diminishing returns. To avoid unnecessary computation, future work could explore mechanisms to detect convergence and suspend adaptation when appropriate. This would enable a smarter, more resource-efficient triggering mechanism that reduces redundant updates and helps mitigate the ~30% computational overhead.

Nonetheless, we will include additional latency and energy measurements in the revised draft and discuss practical trade-offs for mobile deployment.

2. "The evaluation focuses on scene-level corruptions... would be effective for object-level style changes (e.g., VisDA, PACS DomainNet)..."

We appreciate your suggestion. Our current study primarily focuses on scene-level corruptions as they closely simulate the recurring and evolving conditions found in many real-world deployments (e.g., weather changes in autonomous driving). We agree that object-level domain shifts present an important and orthogonal challenge.

Notably, our framework is agnostic to the nature of the shift: it leverages style-based cues that are effective whether the distributional change stems from scene texture, object appearance, or drawing style. To this end, and following the reviewer’s excellent suggestion, we conducted additional experiments on both DomainNet (126 classes) and PACS (7 classes), which feature strong object-level domain gaps. On DomainNet, we perform CSC evaluation by training a ResNet‑50 on each of the four source domains—real, painting, clipart, and sketch—and then test its adaptation over a recurring sequence of the remaining three target domains in a cycle repeated 20 times. We use an identical protocol on PACS with source domains photo, art painting, cartoon, and sketch. The tables below report the error rates after the 20th recurrence.

Method Comparison on DomainNet (Error Rates %)

Method Comparison on PACS (Error Rates %)

As shown in the above Table, integrating ReservoirTTA with EATA reduces error rates by 0.79 % on DomainNet and 0.70 % on PACS, with the largest gains of 1.60 % when adapting from Real on DomainNet and 1.93 % when adapting from Cartoon on PACS. VisDA‑C, however, lacks a multi‑domain recurrence protocol, so it falls outside the scope of our ReservoirTTA evaluation. These new results further support the versatility of our method and its robustness across both scene- and object-level distribution shifts. Full per-round plots and analysis, and new Tables will be included in the revised manuscript’s Section 5 and Appendix L.

3. "Experiments are conducted at a fixed severity level of 5. Can the method adapt to varying levels of severity?"

Thank you for raising this point. Indeed, while our primary evaluations use severity level 5 to ensure comparability with prior work (e.g., CoTTA, EATA), our style-based domain detection is capable of capturing severity variations, as shown in the gradual domain evolution scenario (CCC setting). Specifically, our online clustering mechanism dynamically forms new clusters when severity transitions cause sufficient shifts in the style representation (as controlled by the quantile-based threshold \tau from Eq. 7).

To demonstrate this, we constructed a “gradual domain shift” stream on CIFAR100‑C (CSC) with a ResNeXt‑29 backbone, where each corruption type cycles severity 1 → 2 → 3 → 4 → 5 → 4 → 3 → 2 → 1 over 20 shifts. Even under these subtle transitions, ReservoirTTA consistently improves over EATA:

Compared to EATA’s 25.41 % average error, ReservoirTTA (Kₘₐₓ=16) reduces it to 25.06 % (–0.35%), and increasing Kₘₐₓ to 64 yields a further drop to 24.95 % (–0.10 %). This confirms that our method can detect and adapt to varying corruption severity—not just major shifts—and we will include these results in the revised manuscript.

4. Minor comment: a duplicate citation".

Thank you for catching this. We’ll remove the duplicate [31] citation in line 65.

We thank the reviewer for their constructive feedback and for recognizing the strengths of our work, particularly the utility of maintaining a specialist model reservoir and the robust online clustering approach for domain discovery. Below, we address some of the concerns:

1. New models are cloned from the specialist, maximizing mutual information on the current batch. It might be beneficial to compare this to initialization from the source model or an ensemble of top-k specialists, to verify optimality.

We agree that the strategy for initializing new models in the reservoir plays a critical role in facilitating efficient and stable adaptation to emerging domains. In our current implementation (Section 4.2), we instantiate a new specialist by cloning the model in the reservoir that yields the highest mutual information (MI) on the current test batch. This design biases initialization towards models already exhibiting confident and diverse predictions on similar domains, thereby reducing adaptation iterations and initial errors made by the model.

To verify the efficiency of this MI-based initialization, we have already conducted a comparative study (Appendix F, Figure 12) across different initialization schemes, including: 1. source-based initialization, i.e., cloning the original source model for all new domains; and 2. MI‑based cloning (our default).

We evaluated these on CIFAR100‑C under recurring CSC and CDC shifts (20 recurrences) with a ResNeXt‑29 backbone, and on the large‑scale CCC benchmark and ImageNet‑C under recurring CSC and CDC shifts (20 recurrences) with a ViT‑B‑16 backbone. We conclude that the optimal strategy of model initialization depends on the base TTA method and the backbone. For convnets (ResNeXt‑29), TENT and ETA benefit from source initialization—likely because, without strong regularization, drifting too far from the source harms stability—while EATA remains largely unaffected by initialization choice. Conversely, for ViT backbone initialization (ViT‑B‑16), MI‑based cloning substantially reduces error rates after 20 recurrences for TENT, ETA, and EATA, indicating that transformers are less prone to parameter drift and can better leverage the specialist most similar to the new domain, whereas source initialization may overwrite valuable acquired knowledge.

Finally, in response to the reviewer’s suggestion, we include results using the proposed top‑k (k = 3) ensemble strategy for ReservoirTTA+EATA. On CIFAR100-C under the CSC setting with a CNN backbone, EATA remains unaffected by the choice of initialization. This is consistent with our earlier observation in Appendix F, Figure 12.

On ImageNet-C (CSC setting) with a ViT backbone, initializing with top‑k (k = 3) improves performance by 3.37% compared to source initialization, but degrades performance over MI initialization by 0.27%.

2. Sect. 4.2: Soft assignment qt uses Euclidean distances in d-dim space—have you tried cosine similarity?

This is a valuable point. While we currently use Euclidean distances for centroid assignment due to their compatibility with our quantile-based new-domain detection threshold \tau (Eq. 7)- cosine similarity could indeed offer more robust comparisons in high‑dimensional spaces where vector norms vary. Importantly, the magnitude of our style vector (the log‑variance) also carries discriminative power between domains, which Euclidean distances capture but cosine would discard. To test this, we replaced Eq. 7’s Euclidean soft assignments with a cosine‐based variant (analogous to Eq. 12). Below is a comparison with cosine similarity-based softmax assignments (Eq. 7 and Eq. 12) for ReservoirTTA + ETA on Cifar100-C (CSC setting).

Based on these results, we retained Euclidean distances. In the final version, we will include this ablation in the revised Appendix for full transparency.

3. Eq. (7): Can you clarify choice of quantile for τ and discuss trade-off between sensitivity and specificity? We set \tau using the q-th quantile of pairwise distances among source-domain style vectors to ensure that only substantially shifted distributions spawn new clusters. This balances sensitivity (detecting genuine new domains) and specificity (avoiding over-fragmentation).

As shown in Appendix E (Figure 8), performance is stable across a wide range of q values (e.g. 0.9 to 0.999). Lower q increases sensitivity but risks false positives (unnecessary model creation), while higher q is conservative and may delay a new domain detection. In the following table, we show the number of domains discovered across q-threshold values. Lower value of q-threshold overestimates the number of domains; its higher value (1.0) detects fewer domains (merging similar domains like [zoom, motion, and glass blur], [gaussian, shot, impulse noise], etc.). The experiment is run for 1 sweep of 15 domains on CIFAR-100C.

Dear Reviewer kojJ,

We are happy that our additional experiments and clarifications were helpful. We appreciate the reviewer’s follow-up and the suggestion to explore richer distance metrics. To clarify, in our current formulation (Section 4.1) the style representation encodes only the log-variance of early-layer feature maps, without mean information. Nonetheless, in our formulation, the style vector can indeed include different statistics depending on the variant (see Appendix Table 6), e.g., s = [x_mean, x_var] when both mean and variance are used.

To compute the Mahalanobis distance between two style vectors s1,s2,

d_maha(s1, s2) = (s1 - s2)^T Σ_s^{-1} (s1 - s2),

one must estimate the covariance matrix Σ_s over style features. In our online, non-stationary setting, Σ_s cannot be precomputed at source training time (since test-time styles are unseen), and estimating it reliably on the fly is challenging due to small, evolving batches and shifting distributions. This makes stable and unbiased inversion Σ_s^{-1} non-trivial, particularly for high-dimensional style vectors.

However, when the style vector contains both mean and variance statistics, we can instead treat each as the parameters of a Gaussian distribution and compute the closed-form KL divergence:

D_KL(p || q) = 0.5 * Σ_i [ log(σ²_q,i / σ²_p,i) - 1 + (σ²_p,i / σ²_q,i) + ( (μ_p,i - μ_q,i)² / σ²_q,i ) ]

This does not require global covariance estimation and remains well-defined in our online scenario.

Given that our domain detector has proven robust to reasonable changes in the metric (Appendix E), we do not expect the KL divergence to significantly alter results. This experiment is currently running on CIFAR-100-C (expected to take several hours), and we will try to post the results before the discussion phase deadline.

Nevertheless, in the final version, we will include this variant in Table 6 to empirically confirm its consistency with our current Euclidean-based approach.

As we promised, below is a comparison of different distance metric–based softmax assignments for ReservoirTTA + ETA on CIFAR-100-C (CSC setting) using a ResNeXt-29 backbone for 20 reoccurance:

The results show that KL divergence yields a lower error rate than cosine distance, validating its effectiveness in our online setting, although Euclidean distance on log-variance remains the strongest among the tested metrics.

We hope this additional empirical evidence addresses your suggestion and provides a clear comparison. We would greatly appreciate it if, in light of these results and our clarifications, you might consider raising your score.

We thank the reviewer for their feedback and recognition of our contributions in ReservoirTTA. Here, we address the concerns raised in the weaknesses section.

1. In this paper, both the multi-model strategy and the online clustering technique introduce some additional computational overhead, which may limit the scalability of the proposed method to very large datasets or real-time applications.

We acknowledge that the multi-model design and online clustering in ReservoirTTA introduce additional overhead. However, as detailed in Table 4 (and detailed further in Appendix H), we minimize this cost by storing only trainable parameters- not full model copies—and by enforcing a hard cap on the number of reservoir models (via K_max). Our experiments demonstrate that this design results in only a 30% runtime increase over extremely lightweight methods like EATA. This overhead is actually lower than many other test‑time adaptation (TTA) schemes—for instance, ROID incurs >1.5× latency versus Tent—while delivering substantially better long‑term stability and accuracy. In Table 2, we further demonstrate that VPT + ReservoirTTA runs faster than DPcore, underscoring that, compared to previous multi‑model strategies, we have improved computation time even as we boost accuracy. Furthermore, the modularity of our framework supports plug-in usage with lightweight parameterizations (e.g., LoRA modules), which can be extended using faster‑style architectures as an alternative to VGG, which would make ReservoirTTA practical for many real-world applications and scalability.

Finally, on the large‑scale CCC benchmark (5.12 M test images, ~80 K adaptation steps), ReservoirTTA remains practical for real‑world deployment. We will include these additional discussions in the revised manuscript.

2. The experimental results are not convincing. The authors should compare the proposed algorithm with more recent and relevant methods for prolonged and recurring test-time adaptation.

In our opinion, labeling our experimental results as “not convincing” is too stringent. We appreciate the suggestion and, to the best of our knowledge, we have compared extensively with all of the recent, state‑of‑the‑art prolonged and recurring TTA methods available at the time of our NeurIPS ’25 submission—including CoTTA (CVPR ’22), RoTTA (CVPR ’23), EATA (ICML ’22), PeTTA (NeurIPS ’24), RDumb (NeurIPS ’23), ROID (WACV ’24), SAR (ICLR ’23), CoLA (NeurIPS ’24), DPCore (ICML ’25), and BECoTTA (ICML ’24). We furthermore plug ReservoirTTA into multiple TTA frameworks (TENT, ETA, EATA, ROID, SAR), demonstrating consistent gains across single‑target, continual, persistent, and prompt‑based adaptation, with results detailed in Tables 1–3 and Appendix K. Our ablations further validate performance gains across both CNN and ViT network architectures, classification and segmentation tasks, and diverse domain shift types (CSC, CDC, CCC).

Could the reviewer please clarify which “more recent and relevant” methods they feel are missing from our evaluation? We’d be happy to incorporate any additional baselines they have in mind.

3. A more thorough discussion of the potential weaknesses of the proposed method would strengthen this paper.

We appreciate the reviewer’s call for a more thorough discussion of potential weaknesses, and we agree that a candid limitations section is important. As a general remark, this request is quite broad and does not identify specific concerns, but we have explicitly discussed (in Section 6, “Limitations”) that maintaining multiple reservoir models and running online clustering does incur extra cost ( ~30 % overhead) but this overhead remains modest compared to other TTA approaches — for example, ROID runs at over 1.5× the latency of Tent, yet delivers significantly better long‑term stability and accuracy (kindly see our detailed response to Reviewer dxze). We also note that updating each domain model every batch can lead to diminishing returns once a model converges. Future work could incorporate a convergence detector to suspend updates for stabilized domains—enabling efficient triggering and cutting the ~30 % overhead. Aside from this, we are not aware of other significant limitations. We will make sure the revised manuscript covers this aspect.