Persistent Test-time Adaptation in Recurring Testing Scenarios

1. We agree with the reviewer that the real-world scenarios are significantly more complicated. Despite its simplicity, our $\epsilon$-GMMM model can empirically demonstrate the behavior of a collapsing TTA model on a real-world CIFAR-10-C dataset (see the similarity between Fig. 3(a) and Fig. 4(a)). To the best of our knowledge, this is the first attempt to establish a theoretical foundation for studying the collapse of TTA in the simplest case, promoting future research on the theoretical aspects of the collapsing TTA model.

2. Yes, a theoretical study on real images would certainly validate their practical applicability. However, the challenge of modeling real images lies in the difficulty of theoretically analyzing the effect of added noise as it propagates through a highly complex machine-learning model. The idea behind $\epsilon$-GMMM is to simplify this complex process where the update rule at each step is rigorously defined. Nevertheless, the inspiration gained from this simple study facilitates the development of PeTTA, which demonstrated its ability to meet several real-world continual TTA benchmarks.

• We thank the reviewer for the comment on our rebuttal. In this study, and in the area of continual TTA research, the evaluation benchmark using CIFAR-10/100-C or ImageNet-C is the standard benchmarking approach for all the works in this line of research.
• Those datasets are designed to simulate up to 15 conditions that typically appear in real-world conditions (such as snow, fog, and brightness reflecting weather conditions, or motion blur, zoom blur due to camera/hand motions and jpeg, image noises due to capturing condition or image sensor quality), not just lighting factors you mentioned. Please visit [1] for more information. We evaluated our method in the most severe condition.
• Besides image corruptions, we also evaluated on DomainNet126 dataset with 4 domains: clipart, painting, real, and sketch.
• Again, the key focus of this paper is NOT on evaluating the reality of the evaluation protocol, or closing the gap between laboratory experiments and real-world deployment. Here, we point out the risk of the model collapsing, even in the simplest setting that all previous works adopted, with a small extension (longer time horizon) on the current continual TTA evaluation protocol.
• We are thrilled to extend our experiments with your suggestion, but currently, we are not aware of any publicly available dataset, especially with the time constraint of the rebuttal period that matches the criteria you mentioned. We believe that the current evaluations are sufficient to convey our key message. We agree that a more realistic evaluation is necessary and we will keep exploring in the future.

In summary, we acknowledged the gap between real-world deployment and the evaluation of synthetic data, which is commonly used in the test-time adaptation community does exist. However, we provided here some follow-up comments to justify that this setup is sufficient for us to convey the core message of this study. We hope this perspective will be considered in the ongoing discussions and evaluation of our work.

[1] Hendricks et al., Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR’19.

Comments on the weaknesses part:

1. We respectfully disagree with the comment about the novelty since this study is not all about the anchor loss. Indeed, we acknowledged the anchor loss is not a new idea or our novel point on line 751 (Appendix E5) and will include a proper citation to the anchor network in [A] for completeness. The anchor loss alone is insufficient to avoid the model collapsing as we responded to Reviewer rAs4. Nevertheless, the main contributions of this paper are the theoretical analysis and the sensing of divergence for an adaptive model update in PeTTA. We also acknowledged that the model collapsing is observed in previous studies, as an inspiration for improving the performance but to the best of our knowledge, there is no theoretical investigation on this phenomenon. The suggestion of comparing PeTTA and [A] is interesting. We conducted additional experiments and provided the discussion in the following comment.

2. We sincerely appreciate the reviewer for suggesting recent methods such as ROID (WACV'24) [C] and TRIBE (AAAI'24) [A]. We have benchmarked their performance in Table III-V. Even though TRIBE - a SOTA model can provide stronger adaptability, outweighing the PeTTA model, and baseline RoTTA in the first several recurrences, the risk of the model collapsing still presents in TRIBE when increasing the observation period as demonstrated in the case of CIFAR-10-C (Fig. I(b)). Nevertheless, this result underscores the importance of our proposed recurring TTA setting for an extended testing stream evaluation. We also experimented with the class diversity weighting utilized in ROID [C], unfortunately, it cannot handle the temporally correlated testing stream in the recurring/practical TTA as PeTTA, and tend to be collapsed at the beginning. This is consistent with the finding by ROID's authors in Tab. 4 - ablation study of [C]. ROID with class diversity weighting cannot handle the practical TTA scenario and falls behind PeTTA on all benchmarks.

Since PeTTA utilizes RoTTA as a baseline approach, TRIBE - a much better baseline would be interesting for us to combine the amazing adaptability of TRIBE and the collapsing prevention of PeTTA to create a stronger baseline in future work. Overall, further evaluations of PeTTA against the most recent approaches in this rebuttal highlight (1) the value of recurring TTA in spotting the lifelong performance degradation of continual TTA and (2) the novel design of the sensing model divergence and adaptive update in PeTTA, inspired by a theoretical analysis to address the collapsing phenomenon of the TTA model. The revised paper will include a discussion of these methods and an experimental comparison with them as outlined in the rebuttal PDF.

Comments on the questions part:

We appreciate the reviewer's suggestion regarding the feature-based alignment TTA mentioned in [D]. While sharing several similarities in the design, this work does not directly study the performance degradation of continual TTA methods as PeTTA. Nevertheless, evaluating the performance of this approach under our recurring TTA setting is interesting and straightforward due to the simplicity of our setting. We will mention [D] in the revised paper and leave the evaluation for future work due to the length of the rebuttal period and since [D] has just officially appeared on the 8/2024 issue of TPAMI 2024.

Comments on the Weakness Part:

1. We would like to emphasize that recurring TTA serves as a diagnostic tool for catching the lifelong performance degradation of continual TTA, and even in this simplest case, several SOTA continual TTA algorithms fail to preserve their performance. This raises awareness in the community when evaluating their methods (visit Appendix D.2. for further discussion). Extending our recurring TTA to include more challenging scenarios and complex types of shifts is necessary, but should be addressed in the following work.
2. We have provided additional justifications and discussions on the choices of hyper-parameters for consideraiton. See the comments below.
3. See the comments below.

Comments on the Questions Part:

1. Yes, (i) our method can handle non-cyclic domain shift. PeTTA does not make any assumptions or utilize this property of the testing stream. As an example, it achieves good performance on CCC [1] where the corruptions are algorithmically generated, non-cyclic with two or more corruption types can happen simultaneously - see the Appdx. F.4. Regrading (ii), for all experiments, the label distribution is temporally correlated (non-iid) following [2, 3]. The class distribution within each data batch can be highly imbalanced, with some classes dominating the others. The robustness of our PeTTA with label distribution shifts is demonstrated up to this extent.

2. Absolutely, PeTTA is experimented with 40 recurrences in Tab. II and Fig.I (a). The experimental results confirm the persistence of PeTTA. Additionally, the performance of PeTTA over an extended time horizon is presented in Table 13 (Appendix F4). In this case, the model is adapted to over 5.1 million images, which is significantly more than the default 20 recurrences.

3. In PeTTA, $\alpha_0=1e^{-3}$ is the initial learning rate for adaptation. We do not tune this hyper-parameter, and the choice of $\alpha_0$ is universal across all datasets, following the previous works/compared methods (e.g., RoTTA, CoTTA).

Since $\lambda_0$ is more specific to PeTTA, we included a sensitive analysis with different choices $\lambda_0$ on CIFAR-10/100-C and ImageNet-C in Table VI for the sake of time. Overall, the choice of $\lambda_0$ is not extremely sensitive, and while the best value is $1e^1$ on most datasets, other choices such as $5e^0$ or $5e^1$ also produce roughly similar performance. Selecting $\lambda_0$ is intuitive, the larger value of $\lambda_0$ stronger prevents the model from collapsing but also limits its adaptability as a trade-off.

In action, $\lambda_0$ is just an initial value and will be adaptively scaled with the sensing model divergence mechanism in PeTTA, meaning it does not require careful tuning. More generally, the choice of this hyper-parameter can be tuned similarly to the hyper-parameters of other TTA approaches, via the use of an additional validation set, or some accuracy prediction algorithm [4] when labeled data is not available.

4. Except for the recurring testing condition, each recurrence follows the standard continual TTA established in previous studies. Hence, for all compared methods, we use the best parameters provided by the authors. The performance after the first visit is manually verified to ensure the reproducibility of the original work. Noteworthy, the primary observation of this work is to determine how long these approaches can sustain their initial performance.

5. Since the purpose of the anchor loss is to guide the adaptation under the drastic domain shift (Lines 684-691, Appendix E.2), we empirically found out that it is unnecessary to introduce an additional hyper-parameter and let the adaptive regularization term take the leading role of collapse prevention.

6. No, PeTTA requires sampling from the source dataset to perform. Nevertheless, Appdx. E4 demonstrates that only a small number of samples is required for reliably estimating the empirical mean and covariance matrix. PeTTA relies solely on the typical assumptions used in the other methods it is compared against (e.g., EATA, RMT). Please visit Appendix E.4 for our discussions on the feasibility of accessing the source dataset.

[1] Robust Test-Time Adaptation in Dynamic Scenarios, CVPR'23.

[2] RDumb: A simple approach that questions our progress in continual test-time adaptation, NeurIPS'22.

[3] Gong et al., NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation, NeurIPS'22.

[4] Lee et al., AETTA: Label-Free Accuracy Estimation for Test-Time Adaptation, CVPR’24.

Comments on the Limitations Part: We thank the reviewer for the suggestion. Appendix E elaborates some limitations mentioned in Section 6 in detail. The direction of further improvement will be included in the revised paper.

Dear reviewer oiWU,

the authors have posted their rebuttal. Could you please aim to reply to the authors latest till August 12th end of day, in case there are any further comments the authors would like to add/clarify?

Thanks again for your efforts in reviewing!

-AC

Comments on the Weaknesses Part:

1. In Lemma 1, we mathematically showed that under Assumption 1, we have $\lim_{t \rightarrow \tau}\epsilon_t = p_1$. Furthermore, the convergence of $\epsilon_t$ to $p_1$, and the collapsing behavior when $\epsilon_t$ selected following Collary 1 are both empirically validated through a numerical simulation in Sec. 5.1. Since the rate of collapsing depends on various factors, including both data-dependent and algorithm-dependent, the Corollary 1 here holds as a generic condition for model collapsing. Nevertheless, we will further explore specific conditions/settings that can make this statement more rigorous in future work.

2. The motivation, and reasoning for our design choice behind the anchor loss and the regularization term are detailed in lines 684-691 of Appendix E.2. While using anchor loss is beneficial on many benchmarks, it is not sufficient on its own to achieve PeTTA’s performance, as shown in Table I of the rebuttal PDF. In response to reviewer MNYK, we expanded our evaluation to include TRIBE (AAAI'24), a more recent robust TTA algorithm that also utilizes a concept similar to anchor loss. Despite demonstrating better adaptability, this method is still prone to model collapse over a longer time horizon, necessitating the exploration of additional strategies beyond a simple anchor loss.

Comments on the Questions Part:

1. All tables show the performance of MECTA with EATA backbone (line 242). MECTA is an advanced version of EATA, with all the components preserved, and the batch normalization blocks are replaced with MECTA blocks, showing higher efficacy. For completeness, the experiments of a standalone EATA adapter are also included in Table III-V of this rebuttal PDF. In short, EATA still suffers from performance degradation in the recurring TTA setting just like MECTA and other methods.

2. Yes, thank you for catching that. This figure will be updated in the revised paper.

3. In Table I, we provided an additional ablation study where a baseline model is trained with and without the anchor loss (no regularization). The results show that while having initial benefits on some benchmarks, trivially applying the anchor loss alone is incapable of eliminating the lifelong performance degradation in continual TTA.

Thanks for the response. My concerns are addressed and I would raise my score.