Class-aware Domain Knowledge Fusion and Fission for Continual Test-Time Adaptation

Thank you for your detailed review and recognition of our work’s strengths. Your valuable feedback on technical positioning relative to DPCore, comprehensive efficiency analysis, and clarifications on data stream construction (including model performance under different modes) is deeply appreciated. We’ve made additional experiments and analyses to address your concerns as well as strenghten the paper, detailed below:

1. Elaborate on the conceptual and technical contributions of the proposed method in comparison to DPCore.

We respectfully clarify that our contributions extend beyond incremental modifications to DPCore, with key distinctions addressing critical limitations in prior prompt-based CTTA methods:

First, a correction: DPCore does not employ Online K-Means clustering for prompts, and its prompt pool grows unbounded without grouping mechanisms, leading to redundancy and noise accumulation. In contrast, our MST-based clustering is not a "replacement" but a novel design to actively merge similar prompts, which directly mitigates the parameter explosion and noise issues in DPCore.

Second, the introduction of a class prompt pool is not merely an "additional" component. It explicitly models class-level shared knowledge across domains, a dimension entirely absent in DPCore’s domain-only prompts. This enables our method to leverage cross-domain class commonalities, a capability validated by our ablation study in Table 3, showing significant performance gains (4.7%) from this design.

Together with our knowledge fusion mechanism and class prompt introduction, these innovations form a cohesive framework that outperforms DPCore by 5.1%, which is a substantial improvement. These designs address challenges in prompt-based CTTA (scalability and knowledge utilization), establishing clear technical contributions.

2. A more comprehensive efficiency analysis beyond the limited parameter count comparison.

We fully agree that a comprehensive computational analysis is critical to clarify the performance-efficiency trade-off, especially for prompt-tuning-based methods. To address this, we have supplemented detailed comparisons across multiple metrics, including learnable parameters, memory usage, runtime, and FLOPs, evaluated on a single NVIDIA 4090 GPU . Results are presented in two tables: one for ImageNet-to-ImageNet-C (first-round adaptation) and another for the 10th round under the repeating domains setting.

Our findings demonstrate that despite introducing additional operations during online adaptation, our method achieves a favorable balance between performance and efficiency:

• Compared to baselines like CoTTA and ViDA, our method leads in both efficiency and performance.
• Against DPCore, we achieve a 5.1% performance gain with comparable efficiency metrics.
• Compared to Tent, which only modifies batch normalization layers, has superior raw efficiency, our method achieves a 15.2% higher performance gain with little additional cost, a worthwhile trade-off.

A key efficiency advantage emerges in long-term adaptation: In later rounds (e.g., R10), our KFI module stops generating new prompts when no new domains appear, reducing computational load over time. In contrast, DPCore—lacking fusion and prompt selection—suffers from escalating parameters, FLOPs, and runtime due to unconstrained prompt accumulation, alongside rising error rates. This dynamic efficiency gain further validates our design’s practicality for continuous adaptation scenarios.

These results confirm that our method’s additional operations are justified by its performance gains and long-term efficiency stability, making it suitable for real-world deployment.

Table A: Computational analysis on ImageNet-to-ImageNet-C.

Table B: Computational analysis on ImageNet-to-ImageNet-C with repeating domains, round 10.

3. Clarifications on data stream construction and evaluations considered temporally correlated data streams.

Following common practices in CTTA, our data stream is constructed via independent and identically distributed (i.i.d.) sampling, consistent with regular training protocols. This aligns with most baselines to ensure fair comparisons.

Additionally, we further supplemented experiments on temporally correlated data streams, following the "practical TTA" setting in RoTTA[47]. Results in Table C show that our method achieves a further 4.7% improvement in this scenario.

This performance gain stems from the ability of our knowledge fission and fusion mechanisms to model temporal dependencies: fission captures time-varying knowledge, while fusion preserves stable temporal patterns across sequential data. These results validate the generalization of our method to real-world temporally correlated streams.

Table C: Average error rate(%) over practical TTA settings on ImageNet-to-ImageNet-C.

Dear Reviewer 8gyU,

Thank you for your time and effort in reviewing our work. As the rebuttal period concludes within three days, we wish to respectfully confirm whether our responses have adequatelly addressed your initial concerns.

Any feedback would be greatly appreciated. We truly value your expertise and are happy to continue the discussion if you have further questions.

Sincerely,

The Authors

Dear Reviewer 8gyU,

Thank you sincerely for your thoughtful reconsideration and the increased score. We deeply appreciate your candid feedback on the conceptual positioning of our prompt pool design, which provides valuable perspective for our future research. We will continue to explore more conceptually impactful innovations in this direction, building on the foundations laid here.

Thank you again for your time and insights.

Sincerely,

The Authors

Dear Authors,

Thank you for your clarification and the additional numbers provided! I acknowledge that this work designs a more comprehensive prompt pool method and achieves better performance than the previous prompt pool-based method (DPCore [52]), albeit with increased computational burden. Given the comprehensive evaluation and the accuracy improvements demonstrated, I have decided to increase my score by 1.

The reason I do not lean toward a higher score is that the “prompt pool” has been a classic technique in continual learning for several years, and this paper is not the first to apply it to continual test-time adaptation. The technical improvement over DPCore, while effective, does not significantly impress me at the conceptual level.

Best regards,

Reviewer 8gyU

Thank you sincerely for your thoughtful assessment of our KFF method and your recognition of its strengths. Your constructive feedback, particularly regarding efficiency metrics beyond parameters and the analysis of repeated domain performance, is incredibly helpful. Addressing these, we’ve made additional experiments and clarifications to strengthen the paper, as detailed below:

1. More comparision on efficiency.

We have supplemented comparisons of learnable parameter count, GPU memory usage, and computational efficiency with baselines, evaluated on a single NVIDIA 4090 GPU. Results are provided in two tables: table A for the first round and table B for the 10th round under the repeating domains' setting.

1. Comparison with Tent. Tent, which only modifies batch normalization layers, exhibits superior raw efficiency (faster computation, lower memory). However, our method delivers a 15.2% performance improvement with little additional cost, making this trade-off worthwhile.

2. Comparison with other baseline models. Against CoTTA and ViDA, our method outperforms them in both efficiency and performance. Against DPCore, our method achieves a 5.1% performance gain while maintaining comparable efficiency: computation cost, memory usage, and parameter count remain on par in early rounds.

3. Advantage of long-term adaptation. A critical efficiency edge emerges in later rounds (R10, shown in Table B): our KFI module stops generating new prompts when no new domains appear, reducing computational load over time. In contrast, DPCore, lacking fusion and prompt selection, sees escalating parameters, slower inference, and higher memory usage due to unconstrained prompt accumulation (accompanied by rising error rates). This validates our design’s efficiency and stability in continual test-time adaptation.

Overall, our method balances performance gains with efficiency, outperforming both prompt-based and non-prompt-based baselines in performance while maintaining comparable efficiency, and offering far greater performance improvements than ultra-lightweight methods like Tent with minimal additional cost.

Table A: Computational analysis on ImageNet-to-ImageNet-C.

Table B: Computational analysis on ImageNet-to-ImageNet-C with repeating domains, round 10.

2. A deeper analysis on why the repeated performance levels appear to be merely preserved without noticeable improvement.

The stable performance across repeated rounds does not stem from the strong reliance on source statistics. In our setup, R1 samples already provide sufficient information for the model to capture robust patterns, establishing a strong baseline, and our methods could maintain that performance. This is a deliberate strength, as many baselines struggle to even maintain initial performance due to the inevitable mixing of conflict domain knowledge (e.g., Tent drops by 48.9% from R1 to R10, while DPCore drops by 6.9% in Table 2 in our paper).

Our knowledge fission/fusion mechanisms focus on retaining and refining knowledge rather than forcing unnecessary "improvement" when domains stabilize. When we reduced per-domain samples to 1/10, as shown in Table C, clear improvements emerged across rounds, showcasing KFF’s ability to actively acquire new knowledge when needed. This validates its balance of stability (with sufficient data) and adaptability (with limited data).

Table C: Classification error rate(%) for ImageNet-to-ImageNet-C online CTTA task in 10 repeated rounds with limited samples(R1-R10).

3. Explain more about Fig1 (c).

Fig. 1(c) illustrates the performance of existing methods during a domain change: fog → bright → fog. Specifically, when the model encounters the fog domain again (after adapting to the bright domain), its classification accuracy drops sharply by 14.6% compared to the first fog domain, even when tested on identical fog samples.

This decline indicates that knowledge learned from the bright domain conflicts with prior knowledge from the fog domain, impairing the model’s ability to reuse previously acquired fog-specific patterns. In contrast, our method mitigates such conflicts through knowledge fission strategy, preserving accuracy across repeated domains, which further validates its effectiveness.

Dear Reviewer TUaD,

Thanks again for your expertise and invaluable feedback on improving the quality of our paper!

Best Regards,

The Authors

Thank you for your detailed evaluation of our KFF framework and recognition of its strengths. The constructive points you raised are highly valuable. To address these, we conducted supplementary experiments and analyses detailed below:

1. More comparative experiments.

We further supplemented experiments comparing our method with EATA[34] and BECoTTA[24] on the ImageNet-to-ImageNet-C benchmark, with results presented in Table A.

It can be seen that our method achieves an average error rate reduction of more than 10% compared to both EATA and BECoTTA. This performance gain can be attributed to the limitations of the competing methods: EATA and BECoTTA do not account for the dynamic evolution of domain knowledge, which reduces the efficiency of knowledge utilization. In contrast, our knowledge fission and fusion mechanism not only significantly controls model size but also promotes the integration of similar knowledge, thereby enhancing the generality of learned representations. Additionally, our class-specific prompts and selective fusion strategy better leverage shared features of the same/similar classes across domains, further boosting classification accuracy. These results provide additional evidence for the effectiveness of our approach in CTTA scenarios.

Table A: Average error rate(%) for ImageNet-to-ImageNet-C CTTA task across different methods.

2. Comparison with knowledge fusion frameworks similar to KFF.

We have conducted two supplementary experiments by integrating the knowledge fusion module proposed in CoLA[a]: (1) integrating CoLA’s knowledge fusion module into our baseline DPCore; (2) replacing our KFU module with CoLA’s fusion mechanism. The results were presented in Table B, which show that incorporating CoLA into the baseline yields only a 0.8% improvement, while replacing our KFU with CoLA leads to a 3.0% gain—both significantly lower than our method’s 5.1% enhancement. Compared to these two setups, our approach achieves additional gains of 4.3% and 2.1%, respectively.

The gap arises because CoLA aggregates weights unoptimizedly based solely on domain similarity, ignoring class-level relationships. In contrast, our KFF uses global similarity to merge both domain- and class-specific knowledge, better leveraging cross-domain shared features of same/similar classes—validating its superiority.

Table B: Average error rate(%) for ImageNet-to-ImageNet-C CTTA task across different fusion strategies.

[a] Cross-Device Collaborative Test-Time Adaptation. NeurIPS 2024.

3. Deeper analysis on knowledge fusion.

We have extended analyses on how N_d and N_c balance pool expansion control and retention of subtle, future-valuable knowledge. Experiments, spanning a broader parameter range under the "CTTA with Repeating Domains" setting (R1 and R10 for the first and 10th round), are in Tables C and D.

Results show that excessively small N restrict prompt diversity, losing subtle knowledge critical for future adaptation, while overly large N cause uncontrolled pool expansion, leading to redundant prompts poorly optimized by limited samples.

Our method selects optimal N_d/N_c, balancing these trade-offs to avoid both indefinite expansion and valuable knowledge loss, performing best in both R1 and R10.

Notably, in R1, large N_d/N_c yield stable performance because the Knowledge Fission Module (KFI) generates too few prompts to trigger fusion, letting the framework dynamically avoid unnecessary early fusion and preserve initial subtle knowledge.

In summary, our extended analyses confirm that properly tuned N_d and N_c effectively prevent both unchecked pool expansion and loss of valuable subtle knowledge, validating the robustness of our knowledge fusion design.

Table C: Effect of N_c. Average error rate(%) for ImageNet-to-ImageNet-C CTTA task.

Table D: Effect of N_d. Average error rate(%) for ImageNet-to-ImageNet-C CTTA task.

4. Ablation experiments for γ_h.

We further conducted an ablation study on the entropy threshold (γ_h), with results in Table E. The analysis reveals that γ_h impacts performance through two key mechanisms: While too low γ_h reduces valid samples for updates, hindering adaptation, too high γ_h includes high-entropy (low-confidence) samples as noise, hurting performance. This finding is consistent with EATA [34], which also demonstrated that adaptation on test samples with very high entropy may hurt performance.

Our method selects an optimal γ_h that balances these trade-offs, ensuring sufficient valid samples for updates while avoiding noise from high-entropy instances, thereby achieving robust performance. This ablation study confirms the effectiveness of our threshold selection strategy.

Table E: Effect of γ_h, we choose 2 in our paper. Average error rate(%) for ImageNet-to-ImageNet-C CTTA task.

5. Explain the differences between Domain Knowledge Fission and DPCore.

In terms of Domain Knowledge Fission, the core distinction from DPCore lies in our Domain Prompt Selection Strategy:

DPCore aggregates all existing domain prompts via similarity-weighted fusion during adaptation, which inevitably introduces noise from irrelevant domain prompts. In contrast, our Domain Knowledge Fission selectively picks only the most relevant domain prompts to guide the generation of new domain prompts, significantly reducing noise interference. A controlled experiment (Table F) shows that adopting DPCore’s full domain prompt weighting strategy ("Ours*") leads to a 1.8% performance drop, validating the effectiveness of our selective domain prompt selection in fission.

Beyond fission, our method differs from DPCore in two other key aspects: (1) We introduce class prompts (absent in DPCore) to explicitly capture cross-domain class-level commonalities, boosting performance by 4.7% (ablation in Table 3). (2) Our prompt fusion merges similar prompts to avoid redundancy, whereas DPCore’s prompt pool grows unbounded, alleviating noise accumulation and parameter explosion.

Table F: Performance Impact of different Domain Prompt Selection Strategies: Ours vs. DPCore-style (denoted as Ours*).

6. Analyze KFF’s performance under overlapping scenarios.

We supplemented experiments on overlapping domain distributions, following CCC[38] to construct scenarios with controlled feature overlap between domains.

Results (Table G) show our method outperforms previous SOTA DPCore by 3.7% on average under such conditions. This shows our knowledge fission/fusion mechanisms can still accurately distinguish and integrate knowledge types despite overlaps, maintaining strong generalization, validating KFF’s practicality for real-world scenarios.

Table G: Average error rate(%) on ImageNet-C, CCC-Medium and CCC-Hard.

7. Experiments on real-world complex distribution shifts.

We validated the effectiveness of our method on datasets with natural domain shifts (Cityscapes-to-ACDC). The experimental results are shown in Table H, which demonstrate that our method outperforms the existing SOTA method (DPCore) by 1.1%. This is because our knowledge fusion and fission can adaptively adjust according to the shift degree of different domains and select the most appropriate prompt for optimization, which indicates that our knowledge fission and knowledge fusion still maintain high generalization on datasets with real-world complex distribution shifts.

Table H: Average mIoU score(%) for Cityscapes-to-ACDC, with SegFormer-B5 model pre-trained on the Cityscapes dataset as backbone.

Q1. Would the performance decrease if the prompt pool were based on pseudo-labels?

Yes, saving all class prompts via pseudo-labels reduces performance: Appendix Table 4 shows a 1% accuracy drop, as pseudo-labels fail to capture shared traits among similar classes, resulting in suboptimal knowledge representation. By contrast, our design leverages inter-class commonalities, achieving the highest accuracy among compared approaches.

Q2. Can the KFF method be transferred to other models such as ResNet or MobileNet?

Our KFF method, like many other prompt-based approaches (e.g. DePT[12], DPCore[52]), is initially designed around Vision Transformers (ViTs)[10, b] due to their token-based structure, which naturally supports prompt integration. Conventional CNNs (e.g., ResNet, MobileNet) lack such token mechanisms, making direct adaptation of existing prompt frameworks challenging.

To align with common practices in CTTA and ensure fair comparisons (most SOTAs[25, 26, 52] use ViTs), we adopt ViTs as the backbone. However, we plan to explore CNN-compatible prompt designs in future work to verify KFF’s transferability across architectures.

[b] SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. NeurIPS 2021.

Thank you sincerely for your insightful feedback and constructive suggestions, which have been invaluable in refining our work. We greatly appreciate your recognition of the method’s strengths in CTTA performance, parameter efficiency, and the clarity of our presentation. Addressing your key concerns, we have supplemented critical experiments and analyses to strengthen the paper, as detailed below:

1. Evaluation on Larger, Real-World CTTA Benchmarks.

Following the common practice in the CTTA community, we have conducted fair comparisons with existing methods on ImageNet-to-ImageNet-C, CIFAR10, and CIFAR100. In addition, we further validated the effectiveness of our method on datasets with natural domain shifts (Cityscapes-to-ACDC). The experimental results are shown in Table A. The results demonstrate that our method outperforms the existing SOTA method (DPCore) by 1.1%. This is because our knowledge fusion and fission can adaptively adjust according to the shift degree of different domains and select the most appropriate prompt for optimization. The result on Cityscapes-to-ACDC dataset indicates that our knowledge fission and knowledge fusion still maintain high generalization on datasets with natural domain shifts.

Table A: Average mIoU score(%) for Cityscapes-to-ACDC, with SegFormer-B5 model pre-trained on the Cityscapes dataset as backbone.

2. Analyze Sensitivity and Robustness of Hyperparameters (γ and α).

We conducted further analyses on the influences of key hyperparameters (γ and α) on knowledge fission/fusion, with results summarized in Table B-E. Numbers with * represent the original default parameter tuned on the disjoint validation set from ImageNet-C.

For γ: As shown in cross-dataset experiments (Table B,C), while the optimal γ value varies slightly between ImageNet-C, CIFAR10, and CIFAR100, the performance remains stable within a reasonable range (±0.5%) across all datasets. Excessively small or large γ degrades performance consistently: a small γ introduces irrelevant knowledge, while a large γ limits knowledge reuse. This consistent cross-dataset behavior highlights γ’s robustness: though not universally optimal, its core tuning logic, balancing knowledge relevance and reuse, generalizes well, simplifying deployment via systematic adjustment rather than arbitrary search.

For α: Across all tested datasets, α shows minimal impact on performance. Our experiments (Table D,E) demonstrate that varying α within a typical range (e.g., 0–0.5) leads to performance fluctuations of less than 0.2% on ImageNet-C, CIFAR10, and CIFAR100. This insensitivity stems from the fact that our knowledge fission mechanism selects similar prompts for updates, promoting the reuse and evolution of similar knowledge while balancing the consistency of knowledge before and after fission. Thus, hyperparameter α has minimal impact on consistent knowledge, further verifying the robustness and generalization of our method.

Regarding the performance on CIFAR10 and CIFAR100: We have organized additional experiments to clarify this, as shown in Table F (and Appendix Tables 5/6). Without further parameter tuning, our method already achieves improvements of 2.6% and 3% over DPCore on CIFAR10 and CIFAR100, respectively. These are not marginal gains.

Furthermore, with simple parameter tuning (specifically adjusting γ_c to a larger value and N_c to a lower value) on CIFAR10 and CIFAR100, the average performance on the two datasets is further improved, resulting in an average gain of 3.3% over DPCore (denoted as "Ours+"). This adjustment is reasonable because CIFAR10/100 have smaller data scales compared to ImageNet-C; a stricter γ_c, through more rigorous knowledge matching, better models the discriminative knowledge between samples of different classes. This further demonstrates the generalization of our method across datasets of varying sizes.

Table B: Effect of γ_d across different datasets. Average error rate(%) for CTTA tasks.

Table C: Effect of γ_c across different datasets. Average error rate(%) for CTTA tasks.

Table D: Effect of α_d across different datasets. Average error rate(%) for CTTA tasks.

Table E: Effect of α_c across different datasets. Average error rate(%) for CTTA tasks.

Table F: Average error rate(%) for CTTA tasks across different datasets.

3. A more thorough comparison or differentiation from DPCore.

Our method differs from DPCore in three key aspects, with experimental evidence supporting the effectiveness of each innovation:

1. Knowledge Fusion Mechanism: Unlike DPCore, which allows unrestricted prompt growth (leading to escalating model size and potential redundancy), we introduce a knowledge fusion mechanism that merges similar prompts during adaptation. This not only controls parameter explosion but also promotes the integration of analogous knowledge, enhancing the generality of learned representations. As shown in our ablation study (Table 3 in our paper), this mechanism alone yields a performance improvement of 2.1%, validating its role in balancing efficiency and knowledge utility.

2. Class-Specific Prompts: DPCore relies on a single type of prompts. In contrast, we introduce class prompts to explicitly capture discriminative features shared by the same/similar classes across domains. This design enables the model to leverage cross-domain commonalities of classes for better classification. Our ablation results (Table 3) demonstrate that adding class prompts contributes an additional 4.7% improvement, highlighting its effectiveness in exploiting inter-domain class-level consistency.

3. Dynamic Prompt Selection Strategy: DPCore aggregates all existing prompts via similarity-weighted fusion for each test batch, which inevitably introduces noise from irrelevant prompts. Instead, our method selects only a small subset of the most relevant prompts for fusion, significantly reducing noise interference. To verify this, we conducted a controlled experiment where we modified our method to adopt DPCore’s full prompt weighting strategy (denoted as "Ours*"), shown in Table G. The observed performance dropped 1.8% in "Ours*" compared to our original design, which confirms that our selective fusion strategy is critical for avoiding noise and maintaining robustness.

These innovations collectively distinguish our approach from DPCore: by controlling parameter growth, enhancing class-level knowledge utilization, and reducing noise when using prompts, our method achieves superior performance as validated by both ablation studies and comparative experiments.

Table G: Performance Impact of different Domain Prompt Selection Strategies: Ours vs. DPCore-style (denoted as Ours*)

Dear Reviewer er6C,

Thank you for your time and effort in reviewing our work. As the rebuttal period concludes within three days, we wish to respectfully confirm whether our responses have adequatelly addressed your initial concerns.

Any feedback would be greatly appreciated. We truly value your expertise and are happy to continue the discussion if you have further questions.

Sincerely,

The Authors

Dear Reviewer er6C,

Thank you again for your invaluable feedback on improving the quality of our paper. We hope our responses have addressed your initial concerns thoroughly. Could you please kindly share your feedback on the supplementary experiments and clarifications provided? Thank you for your time and consideration.

Sincerely,

The Authors