Self-Bootstrapping for Versatile Test-Time Adaptation

Q1. Results on CIFAR-100 and ResNet

In the submission, we compare SPA on ImageNet-C/R/Sketch/A as they are large-scale, more challenging than CIFAR-100, and commonly used. We now report more results on CIFAR100-C below to further validate SPA. Pls see Reviewer 61cM's Q1 for more ResNet results.

Q2. Clarification on "rounds" in segmentation exps

“Round” refers to the number of repeated sequence of datasets to simulate long-term continual TTA (e.g., Fog-Night-Rain-Snow-Fog-…). This is a standard evaluation protocol in continual TTA following CoTTA. We will make this clearer.

Q3. Comparison with more baselines

As you suggested, we compare SPA with more baselines in the table below on classification, showing our efficacy. For detection, we acknowledge that the suggested baselines are designed for 2D tasks, while we focus on more challenging 3D monocular object detection. Due to time limits, it is non-trivial to implement them in our 3D setting. Here, we would like to clarify that MonoTTA (ECCV'24) is tailored for 3D detection and is the current SOTA (more latest than suggested ones), making it more relevant for comparison. Thus, we believe the current thorough comparisons with MonoTTA have fully validated SPA's effectiveness. We will discuss all mentioned methods in the revision.

Table. More results on ImageNet-C with ViT-B

Q4. The method's complexity is limited, ...

SPA's main contribution lies in its overall design for versatile online fully TTA. It comprises an active, deterioration-driven self-bootstrapping scheme and geometry-preserving augmentations inspired by our Fourier analysis of domain shifts.

Unlike prior methods enforcing consistency across different augmentations with a teacher-student scheme, our SPA generates a deteriorated image to create an information gap, enabling self-distillation in a single model from strong to weak predictions. The distilled knowledge is then directly fed back to the strong branch via shared parameters, forming a closed-loop TTA process.

Though simple in form, SPA’s insights and overall design are non-trivial and challenging, reflected in the result of extensive preliminaries where direct consistency learning or traditional augmentations failed to deliver satisfactory performance. We also believe such method simplicity helps enhance both usability and impact.

Q5. The experiments lack a clear focus, ...

Our goal is to design a versatile TTA method, so we evaluate SPA across multiple tasks. Actually, our validations are thorough: we verified our superiority as standalone or as a plug-and-play module on different tasks, e.g., outperforming prior SOTAs like DeYO (ICLR'24) in classification and MonoTTA (ECCV'24) in 3D detection. We also provided extensive ablations and analyses for different tasks. As you suggested, we also compare with more baselines to verify our efficacy, pls see Q3 and Q9.

Q6. The method could benefit from stronger empirical analysis to validate its effectiveness

In Figure 2, we empirically analyze how domain shifts manifest in Fourier domain, and derive insights of which image deteriorations can supply sufficient learning signals for our self-bootstrapping. Extensive experiments (Tables 5,6,7) further validate the conclusions drawn from Figure 2. These analyses are also positively acknowledged by other reviewers, e.g., “analyses…particularly insightful” [DdZn] and “sufficiently supporting the conclusions drawn” [QKc5].

Q7. Adaptation efficiency

Our SPA is much more efficient than augmentation-based CoTTA, which uses 2 or 34 augmentations per sample, while SPA needs only 1–2. It also matches the efficiency of entropy-based SAR. FPS on ImageNet-C with ViT-Base (on A100) are: SPA-I(125)>SAR(102)>SPA(79)>CoTTA(36). Here, SPA-I is a variant that applies two deteriorations within a single image (1 aug) and achieves similar ImageNet-C acc: SPA(70.1%) vs SPA-I(69.0%). Partial results were in Appendix C and we will make this clearer.

Q8. Efficacy on VLM

Our SPA works on VLM, pls see Reviewer DdZn's Q4.

Q9. Results under continual or mixed-domain TTA in classification

Our SPA works under such scenarios, pls see Reviewer DdZn'Q3.

Q10. Computation cost of augmentation

Our augmentations are efficient: 1) Fourier masking uses PyTorch's FFT/iFFT, accelerated via parallelized GPU operations; 2) Noise injection is a trivial element-wise addition. Augmentation takes only a tiny (negligible) fraction of the total adaptation time, and the FPS of SPA is almost the same whether using our augs or MoCo/SimCLR’s.

We deeply appreciate your constructive comments. We sincerely hope our clarifications above have addressed your questions and can improve your opinion of our work.

We deeply appreciate your valuable feedback and your recognition of the novelty and contributions of our work for designing a challenging versatile fully TTA framework. Our SPA incorporates several components, including an active, deterioration-driven self-bootstrapping scheme (distinct from feature-level BYOL), as well as carefully crafted insights into geometry-preserving augmentation strategy design. The overall design of SPA with these innovations is challenging and non-trivial based on our analysis. These innovations collectively ensure the stability and effectiveness of SPA for online fully test-time adaptation. Thank you again for your insightful suggestion. We will make the novelty and contributions of our work clearer in the revised version.

We deeply appreciate your valuable feedback and constructive comments on improving the quality of our paper. We would like to address your questions below.

Q1. Comparison with TTT-series methods.

Thank you for your suggestion. We follow the comparison setting used in Diffusion-TTA and directly compare our results with the TTT methods reported in their work, by implementing our method using their adopted model and experimental environment. This is because TTT-series methods modify the model training process, making them relatively difficult to implement under our diverse environmental setups. As shown in Table A, our SPA, as a fully TTA method, still achieves superior performance, further demonstrating its effectiveness. We will discuss all mentioned TTT-series methods in the revision.

Table A. Comparisons with TTT-series methods on ImageNet-C w.r.t. acc (%).

Q2. Why are conventional augmentations not good?

Conventional augmentations can be broadly categorized into two groups based on whether they preserve geometric information.

• Geometry-preserving augmentations (e.g., grayscale conversion, brightness, contrast, blur) typically adjust the global distribution of images—for instance, changing brightness by adding the same constant across all pixels. Such augmentation patterns are relatively simple and lack diversity or uniqueness for different samples, thus providing limited learning signals for our self-bootstrapping learning pipeline. As in Table 6, none of these augmentations, individually or combined, effectively provide TTA with rich learning signals. These augmentations are also often sensitive to the corruption type and struggle to perform stably across all corruptions, leading to limited overall performance.
• Non-geometry-preserving augmentations (e.g., random resizing, cropping, image masking, SimCLR, MoCo, AugMix) introduce randomness for different samples, which randomly deteriorate the local information of different images, thus can provide more diverse learning signals for SPA. However, these augmentations shall disrupt the image’s geometric structure and directly desert the potentially useful image information. As in Table 7, SPA with these augmentations achieves considerable performance on image classification but performs poorly on finer dense prediction tasks like 3D monocular detection. In contrast, our Fourier augmentations preserve the overall information and geometry to support dense prediction tasks and also introduce sufficient diversity for different samples, supplying rich signals for SPA and achieving improved adaptation performance.

Q3. Results on more realistic distribution shift scenarios.

Thank you for your valuable suggestion. We have conducted additional experiments on ObjectNet in Table B, based on the codebase of VisDA-2021 challenge. Furthermore, we also evaluate SPA under more 'realistic settings', including continual adaptation (Table C) and mixed domain adaptation (Table D). The results across all these tables further validate the effectiveness of SPA, both as a standalone method and as a plug-and-play module to enhance existing approaches.

Table B. Results on ObjectNet with ResNet-50.

Table C. Results of continual adaptation on ImageNet-C (level 5) with ViT-Base. We report avg acc of 15 corruptions.

Table D. Results on mixture of 15 corruptions of ImageNet-C (level 5) with ViT-Base.

Q4. Integration with foundation models.

We briefly test our method on CLIP-ViT model and report results in Table E. With CLIP, our method still works, further suggesting the versatility of SPA.

Table E. Results with CLIP-ViT-B on ImageNet-R.

We thank you for appreciating our contributions. We sincerely hope our clarifications above have addressed your questions and can improve your opinion of our work.

We deeply appreciate your valuable feedback and constructive comments on improving the quality of our paper. We would like to address your questions below.

Q1. More classification results on ResNet-50.

Thanks for your suggestion. We conduct additional experiments on ResNet-50 and compare with more SOTA approaches. As in the table below, the improvements observed on ResNet-50 are consistent with our experiments on ViT-Base. This further underscores SPA's effectiveness both as a standalone method and as a plug-and-play module to enhance existing methods.

Q2. Differences from FDA [A].

Thank you for pointing out this related work. While FDA also utilizes the Fourier augmentation, SPA's primary contributions lie in the overall design of a versatile TTA scheme. SPA comprises several key components, including an active, deterioration-driven self-bootstrapping scheme (distinct from BYOL), and geometry-preserving augmentations inspired by our Fourier analysis across domain shifts. All these innovations are non-trivial and collectively contribute to SPA’s effectiveness and versatility. Compared with FDA, which focuses on unsupervised domain adaptation, our SPA method mainly differs in the following aspects.

• Different Augmentations: FDA augments a source domain image by replacing the low-frequency spectrum of the source image with that from a target image. This augmentation relies on paired source and target data, which is infeasible in TTA as only a single unlabeled target sample is available at test time. Unlike FDA, SPA randomly masks the low-frequency spectrum and injects noise for a single target image, without requiring the source images.
• Different Methods: FDA uses Fourier augmentation to transfer the source image style to the target domain, creating a target-style labeled source dataset $D^{s\rightarrow t}$. It then jointly trains the model offline using both $D^{s\rightarrow t}$ and the original target dataset $D^{t}$ to mitigate domain shifts. In contrast, SPA targets the more general and challenging setting of fully online TTA. It establishes a versatile self-bootstrapping framework that performs active weak-to-strong learning from a deteriorated view to the original image, for each given target test sample.
• Different Design Motivations: The key motivation of FDA is style transfer（$s\rightarrow t$）for cross-domain training, by swapping the spectrum in Fourier domain. Unlike FDA, the online unsupervised learning in SPA is highly unstable, and thus our key motivation is how to design effective deterioration/augmentation strategies to conquer this. To this end, we analyze how information power shifts across frequencies under domain shift and leverage these insights to design augmentations that consistently deliver informative signals across all frequencies for SPA, meanwhile maintaining stability.

We will include the above discussions in the revised paper.

[A] Fda: Fourier domain adaptation for semantic segmentation. CVPR 2020.

Q3. How is the noise factor $\gamma$ hyperparameter tuned?

We did not carefully tune $\gamma$ on each individual test set as at test time we do not have ground truth labels. In main experiments, we briefly set $\gamma$ to 0.4 for all classification datasets, including ImageNet-C (15 corruptions, ImageNet-R/Sketch/A), and $\gamma=0.1$ for all fine-grained tasks of detection and segmentation, including KITTI-C (13 corruptions) and Cityscape-to-ACDC.

We also show the sensitivity of $\gamma$ in Figure 3 on a single corruption (test set) of ImageNet-C(Gaussian) and KITTI-C(Fog). From the results, $\gamma\in[0.1,0.5]$ works stably on classification, while the stable range of $\gamma$ on mono 3D detection is relatively narrower, with performance slightly declining when $\gamma>0.2$—though it still remains significantly better than without adaptation. This difference arises because, for classification, the task is at the image level and does not strictly require content invariance, allowing for a higher $\gamma$ to provide richer learning signals. In contrast, 3D detection involves dense predictions where high noise levels could significantly disrupt the original image content, making it challenging for our self-bootstrapping learning.

We thank you for appreciating our contributions. We sincerely hope our clarifications above have addressed your questions and can improve your opinion of our work.