Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation

Thank you for taking the time to review our paper and providing valuable feedback. We would like to answer your questions below.

Q1: Experimenting on continual TTA settings performed in recent TTA studies (e.g., EATA) would strengthen the efficacy of ReCAP in wild TTA settings.

A1: Thank you for your constructive suggestion to evaluate CTTA settings. We agree that such experiments would further strengthen the efficacy of our ReCAP. We conduct extensive experiments on CTTA for both classification and segmentation tasks.

For classification, CTTA setup (Tab. 3 in response to Reviewer 6Q1i) and additional PTTA (CTTA + label shift) setup (Tab. 3 in response to Reviewer eZDc) demonstrate that ReCAP consistently outperforms prior methods. For segmentation, results in Tab. 1 (this response) further confirm that ReCAP maintains robust adaptation performance in continual scenarios. These results underscore the broad applicability of ReCAP across continual and wild TTA settings.

Table 1: Semantic segmentation results (mIoU) on the Cityscapes-to-ACDC CTTA setup based on the Segformer-B5 architecture.

Q2: Sampling Number $N$ Sensitivity.

A2: We appreciate your question and apologize for any confusion. Our method does not need any sampling due to the finite-to-infinite approximation in Propositions 4.3 and 4.4. Therefore, our method is entirely unaffected by the value of $N$. We will provide additional clarifications in the revised version to enhance clarity.

Q3: Comparing ReCAP with a simple Monte Carlo approximation of the original region confidence in Eq. (2) would make the proposed method more convincing.

A3: We sincerely appreciate your valuable suggestion. We conduct a comprehensive comparison with the Monte Carlo (MC) approximation using different sampling numbers. As shown in Tab. 2, while MC provides a direct estimate of region confidence, its accuracy is highly sensitive to the number of samples, leading to increased variance and a computational cost that scales linearly with the sample size.

In contrast, ReCAP achieves significantly higher accuracy with lower variance during adaptation, demonstrating its superior stability and efficiency. These results further reinforce the motivation behind our finite-to-infinite approximation. We will incorporate this comparison into the revised version.

Table 2: Comparison between the MC approximation and our finite-to-infinite approximation under 3 independent runs.

Q4: Ablation on the sample weighting and selection in Eq. (9) would be helpful.

A4: We appreciate the reviewer's suggestion and have conducted an additional ablation study to analyze the impact of different sample selection and weighting strategies. As shown in Tab. 3, the results lead to the following key observations:

1. Our region-based confidence optimization consistently enhances performance, surpassing the previous SOTA even when combined with the simplest entropy-based selection and weighting strategies.
2. The combination of our proposed selection and weighting achieves the best overall accuracy, further validating the effectiveness of our design.

Table 3: Ablation study on selection and weighting strategies.

We deeply appreciate your positive comments and constructive suggestions on improving our paper. We will address your questions below.

Q1: I checked the ablation experiments and did not find the one ablating the impact of $\mathcal{L}_0$.

A1: Due to space constraints, we provide the ablation study on $\mathcal{L}_0$ in Appendix C.1. As shown in Appendix Fig. 7, ReCAP consistently enhances performance across a broad range of $\mathcal{L}_0$ values, demonstrating its robustness to different selection boundaries. This result confirms that ReCAP does not rely on precise tuning of $\mathcal{L}_0$ and remains effective across varying settings. To improve accessibility, we will incorporate this ablation study into the main paper.

Q2: When ReCAP is combined with SAR, is the data selection mechanism of SAR employed or the proposed one?

A2: When integrating ReCAP with SAR, we replace the original entropy selection with our proposed strategy, allowing for a direct evaluation of ReCAP's effectiveness. Likewise, when combining ReCAP with DeYO, we follow the same replacement strategy. We will clarify this in the revised version to eliminate any ambiguity.

Q3: Efficiency comparison with EATA and ETA.

A3: Thank you for raising this point. While EATA performs fewer backward passes on test samples, it requires additional computation for Fisher regularization on extra source samples, resulting in a higher runtime compared to ReCAP.

For comparison with ETA, we provide additional evaluations in Tab. 1. Although ETA offers a slight runtime improvement, it struggles to adapt to dynamic shifts in wild TTA scenarios. In contrast, ReCAP effectively balances efficiency and performance, achieving superior accuracy with marginal additional computation cost.

Table 1: Running time for 50,000 images and accuracy on ImageNet-C under label shifts using ResNet.

Q4: It is important, given the efficiency of ReCAP, to show the performance gain under computational budgeted evaluation.

A4: Thank you for your valuable suggestion. We agree that this evaluation is essential and realistic for assessing TTA methods. As shown in Tab. 2, ReCAP benefits from the computational efficiency of the upper-bound proxy, resulting in minimal performance degradation while achieving more significant gains under strict time constraints. This demonstrates ReCAP's ability to provide efficient adaptation under time limitations, making it well-suited for real-world deployments with computational budgets.

Table 2: Error rate on ImageNet-C under computational time constraints.

Q5: One extra [Optional] experiment is to extend the evaluation to the Practical TTA setting which is closely related to the wild TTA setting.

A5: Thank you for this insightful suggestion. We agree that the PTTA setting (Continual + Label Shifts) is closely related to the wild TTA setting. As shown in Tab. 3, despite ReCAP not incorporating any additional design specifically for continual adaptation, it still outperforms entropy-based methods and specific-design RoTTA in PTTA setup. This result further validates the effectiveness and robustness of ReCAP across diverse test-time conditions.

Table 3: Accuracy on ImageNet-C under the PTTA setup, evaluated on ResNet50.

We appreciate your detailed review and positive feedback on our contributions, including meaningful findings, novel region-based confidence optimization, and comprehensive evaluation. Building on your comments, we provide additional explanations and experiments to further demonstrate ReCAP's effectiveness and efficiency.

Q1: Generalization ability in more complex tasks.

A1: Thank you for raising this important point. While our current experiments focus on classification, the core idea of region-based confidence optimization is inherently versatile. Additional evaluations on segmentation (Tab. 1 in response to Reviewer ynrW) and object detection (Tab. 1 in this response) show consistent improvements of ReCAP over entropy-based methods, indicating that ReCAP can be integrated into diverse model architectures and effectively extended to various complex tasks.

Table 1: Comparisons of detection performance on KITTI-C benchmark in [1] with MonoFlex, regarding AP.

Q2: How does ReCAP handle highly dynamic shifts where local consistency may be entirely lost?

A2: Thank you for your insightful question. Our finite-to-infinite approximation is derived without assuming any consistency condition, ensuring its applicability even when consistency is entirely lost. Based on this foundation, ReCAP employs region-confidence optimization to enhance local consistency, which is crucial for robust adaptation.

Moreover, we evaluate ReCAP in a highly dynamic setting where the data stream undergoes rapid transitions across different domains, including style, corruption, and label shifts (see Appendix B.1). The results demonstrate that ReCAP exhibits strong robustness and achieves SOTA performance, validating its capability to address highly dynamic scenes.

Q3: Quantitative comparison of computational cost reduction.

A: ReCAP reduces the computational cost via feature-level region modeling, eliminating the overhead of image-level region modeling and augmentation. Furthermore, its finite-to-infinite approximation serves as an efficient proxy, removing the need for costly sampling. As shown in Tab. 2, these designs achieve significant runtime reduction. We will incorporate this quantitative comparison into the revised version to enhance clarity on the efficiency of ReCAP.

Table 2: Running time on 50,000 images.

Q4: Clarification on ablation study settings.

A4: In our analysis of the effect of λ in Section 6.1, we fixed τ at 1.2, which aligns with the default value used across all experiments. We will explicitly state it in the revised version.

Q5: Variance + Bias vs. Mutual Information.

A5: Our variance + bias function serves a fundamentally different role from mutual information (MI) in several key aspects:

1. Different Objects: MI is defined between two random variables, measuring their shared information, whereas our variance + bias function is computed over a local region surrounding a single sample x, capturing localized prediction stability.
2. Different Purposes: MI primarily quantifies mutual dependence, while our function serves as prediction confidence and consistency measure within a local feature region, making it more aligned with adaptation objectives.
3. Different Optimization Effects: MI encourages statistical association but does not address prediction probability discrepancies. In contrast, our function directly optimizes both prediction uncertainty (bias) and local discrepancy (variance), enhancing robustness under domain shifts.

Q6: Sensitivity of RaCAP.

A6: We have extensively evaluated ReCAP across diverse datasets (ImageNet-C, VisDA, ImageNet-R), tasks (classification, segmentation, detection), TTA settings (wild, mild, and continual), and hyperparameter configurations. Our results consistently show its robustness and reliability across these various conditions.

Additionally, we notice that the question might miss a word (e.g., choice of L_0 when adapting). If we have misunderstood your concern, please clarify, and we would be happy to provide further insights.

References
[1] Lin, Hongbin, et al. "Monotta: Fully test-time adaptation for monocular object detection." European Conference on Computer Vision, 2024.

Thank you for carefully reviewing our paper and offering a positive assessment. We appreciate your recognition of the contributions made by our work, particularly the idea of the tractable bound on the intractable region confidence and the theoretical results.

Q1: Some of the Empirical Gains are marginal. For example, VitBase in mixed testing domain.

A1: Thank you for your feedback. In highly competitive TTA scenarios, performance gains tend to approach saturation in some cases. However, larger domain gaps, such as more severe corruption or mixed style shifts, still present significant challenges in terms of adaptation efficiency and robustness for TTA methods.

To access the empirical gains under more severe shifts, we increase the severity level from 4 & 5 to levels 6 & 7 (see Tab. 1), and our method achieves significant improvements of +5.7 and +5.2 over DeYO. Furthermore, under complex style shifts, ReCAP achieves average gains of +2.6 on ImageNet-R and +1.7 on VisDA (Appendix B.1). Overall, our method consistently outperforms prior methods across 3 datasets, 3 wild settings, and 2 base models, achieving gains of >+1.5 in the majority of scenarios.

Table 1: Comparisons on ImageNet-C (severity level 6, 7) using VitBase under Mixed Testing Domain.

Q2: Limited applicability of bs=1 setting. Can we not accumulate more examples to increase the batch size?

A2: We appreciate your comment. While bs=1 may seem contrived, some real-world applications (e.g., edge computing) face hardware constraints that necessitate the use of small mini-batches. Following your comment, we evaluate the effect of accumulating examples with varying sizes (See Tab. 2). However, small batch sizes still present a crucial bottleneck, hindering adaptation performance. This underscores the importance of developing robust TTA solutions tailored to such restrictive conditions.

Table 2: Accuracy of Tent on ImageNet-C across different accumulated batch sizes, evaluated on ResNet50.

Q3: Line 235-236: It should be Eq. 5 in place of Eq. 10.

A3: Thank you for pointing out this typo, and we will correct it.

Q4: Additional experiments in the CTTA setting will enhance the contribution of this paper.

A4: Thank you for your constructive suggestion. Following your advice, we evaluate our method in CTTA scenarios for classification (Tab. 3 in this response) and semantic segmentation (Tab. 1 in response to Reviewer ynrW). While ReCAP is not designed for CTTA setup, it shows competitive performance and outperforms several strong baselines. These additional evaluations further highlight the broad applicability of our method across mild, continual, and wild settings.

Table 3: Error rate (%) in CTTA scenario (CIFAR100C) [1], evaluated on ResNeXt-29.

Q5: How is the hyperparameter L_0 in equation 9 tuned for the experiments?

A5: For hyperparameter L_0, we perform a grid search over a range of values on a small validation set, which comprises 10% of the Gaussian-type data from ImageNet-C. Additionally, we conduct a sensitivity analysis (see Appendix C.1) to confirm the robustness of our method to variations in L_0. Further details on this tuning process will be included in the revised version.

Q6: Is the accuracy in Figure 4 measured on the test set itself, on which the final performance is reported? Whether there is a validation split for tuning?

A6: For the sensitivity analysis in Figure 4, we measure accuracy on the entire test set to validate the robustness of our method. For hyperparameter tuning, we use a small validation split (the same set used for L_0). The hyperparameters selected through this process are then validated and shown to be robust in Figure 4. We will provide a clearer explanation of this procedure in the revised version to avoid any confusion.

References
[1] Wang, Qin, et al. "Continual test-time domain adaptation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.