Reduce, Reuse, and Recycle: Navigating Test-Time Adaptation with OOD-Contaminated Streams

We would like to first thank Reviewer 8qGX for the thorough feedback. We hope our response below addresses the concerns and questions raised by the reviewer.

W1. Claim of benign OOD contamination

• In our work, we provide two pieces of evidence for benign OOD-contamination. First, we reveal that in certain test streams, OOD data can improve the performance of TENT, which does not employ any filtering scheme. Second, we demonstrate that algorithms with similar OOD detection performance can exhibit wildly different InD adaptation performance, indicating that OOD data have different degrees of helpfulness.
• By default, a test stream contains the same number of InD and OOD samples.
• Because updating the learnable parameters during TTA will likely result in model collapse, it is commonly assumed that only batch norm adaptation is performed at test-time.

W2. Discussion about harmful OOD data

• Generally speaking, if OOD data help adaptation on shifted InD performance, we refer to it as benign OOD data, and vice versa. Therefore, while it is difficult to determine the harmfulness of OOD data without post-adaptation performance, we speculate that OOD data that exist close to InD data in the feature space of the pre-trained model will tend to be benign. This hypothesis is supported through analyses of t-sne plots and 2-Wasserstein distance measurements.

• The performance improvement of R3, compared to other TTA baselines, can be attributed to 1) its increased distinguishability between benign vs. harmful OOD instances and 2) effective utilization of benign OOD instances. R3 only outperforms the clean stream because while benign OOD data exist, harmful OOD data are still quite dominant, and R3, as well as all of the existing baselines, are incapable of perfectly picking out shifted InD and benign OOD data for adaptation. Yet, the significant relative performance improvement of R3 from TTA baselines suggests that R3 does indeed extend the state-of-the-art in this particular TTA scenario.

W3. The cosine similarity is used as an auxiliary metric for filtering OOD instances because we believe that those lie closer to the InD data in the feature space aid better in the domain transfer process.

W4. Without access to explicit test labels, it is impossible to distinguish between the shifted InD and OOD data. Therefore, R3 is bound to be blind to this distinction, leaving it with no choice but to perform a filtering process on both types of data.

W5. The results of ablation studies on the batch size and ratio of InD to OOD data are reported in Tables 4 and 5 of the main paper, respectively.

W6. We were not exactly sure what the reviewer was referring to here by the “CIFAR-10-C+SVHN/DTD/DTD+SVHN and ImageNet-C with OOD data.” If the mentioned quantity refers to the “Clean Stream” accuracy in Tables 1, 2, 3, and 4, it was obtained by performing TENT on clean streams to provide the potential empirical upper bound not by performing inference with a pre-trained model.

We first thank Reviewer szCP for the constructive critique of our paper. With our additional discussion below, we sincerely hope that the reviewer will reconsider the research value of our work.

The term “benign OOD data” in the manuscript refers to OOD data that can improve the adaptation performance on the shifted InD data; any OOD data that does not meet this criterion are considered to be harmful or malignant.

We agree that OOD data have been demonstrated to improve the OOD detection or generalization performance, but this is the first work to study its effect on test-time adaptation. Because intuition that holds in one domain may not always extend to another, it would be a stretch to regard the observed phenomenon of benign OOD-contamination in TTA as simple regurgitation of previously-known facts.

We are deeply disheartened that the reviewer finds the insights from our empirical analyses lacking in research value. While our work may not offer the theoretical depth that the reviewer expects, we kindly ask Reviewer szCP to reconsider the value of our work based on the following grounds:

1. As noted above, we would like to emphasize that this is the first time OOD-contaminated streams have been explored in test-time adaptation. We believe this is a realistic and important setting that significantly broadens the scope and versatility of TTA.
2. We do make a meaningful attempt to explicate the reason behind the existence of benign OOD data through analyses of t-sne plots and 2-Wasserstein distance measurements.
3. We develop a novel algorithm which is grounded on the empirical findings, as well as extending the state-of-the-art across a variety of test scenarios that span combinations of two shifted InD datasets and five OOD datasets. R3 employs conservative dual filtering to preserve benign data, and new R3 loss functions are designed to amplify the effect of OOD data during the adaptation process through mixup and contrastive learning. Moreover, R3 may only lead to marginal performance improvements in certain test scenarios, we would like to bring to the reviewer’s attention the consistent improvement observed across the board.

We would like to thank hkzM for the honest and thoughtful review. In response to the reviewer's comments, we provide additional discussion below.

W1. Additional ablation study to demonstrate the effectiveness of R3: In Table A6 of the revised paper, we report the results of replacing the filtering scheme in EATA and SAR with the proposed “Filter component.” Even with the proposed filtering scheme, abbreviated as R.F., these approaches fall short of R3, indicating that the additional technical components in R3 are integral parts in the success of R3.

Q1. Mixed Setting Implementation Detail As noted in our NeurIPS response, the number of instances in the test set used for the mixed scenario was set to be the same as the number of instances in a single test of shifted InD data in the separate scenario. For instance, for CIFAR-10-c, each shifted test set contains 10,000 samples. To construct the mixed test set for CIFAR-10-c, we sampled 10,000 / 15 number of samples from each test set and shuffled them. Therefore, the test set used for the mixed scenario is different from that used for the separate scenario.

We will incorporate your suggestion to mix all 15 corrupted datasets and include the results in the final version of the paper.

Q2. Number of OOD samples filtered out In Table A4 of Appendix, we compare the OOD detection accuracy on a CIFAR-01-C+DTD stream. The total number of OOD samples in this stream are 5000, so the OOD detection accuracy of 53.38% indicates that 2669 samples are being identified as OOD.

While the three compared approaches perform similarly from the perspective of OOD detection, they yield significantly different results on the InD classification task, further justifying that some OOD data are better preserved than discarded.

Q3. Small batch size We found that running our method with extremely small batch sizes is rather tricky and leads to complete model collapse mostly due to the filtering factor. Oftentimes, the filtering component would leave us with no sample to perform test-time adaptation with.

Q4. Published references We apologize again for not updating the publication statuses of cited references.

Q5. Use of timm model We are deeply sorry that this implementation detail was missing; we will include this detail in the final version.

We would like to thank Reviewer hsXP for the constructive comments. In the author response below, we provide answers to the questions raised by the reviewer. We sincerely hope that our answers will serve to clarify any confusion.

W1. Distinction between Shifted InD and OOD data In the proposed scenario, we assume that two types of out-of-distribution data exist in the test stream: those that share the label space with the in-distribution data but have undergone some type of corruption in the input space (shifted InD data) and those that exist in a completely different label space (OOD data). The objective of test-time adaptation is to adapt the pre-trained model on the shifted InD data, and thus, we consider the latter “OOD-contamination.” As the reviewer correctly notes, “shifted InD data” are commonly considered to be “OOD data,” and we apologize that our wording led to confusion.

W2. Yes, the concept of “benign OOD-contamination” refers to OOD data that could assist in the domain transfer from InD to shifted InD data and consequently lead to the adaptation accuracy on the shifted InD data.

W3. In the reduce part, we utilize both the energy score and the similarity measure to identify and discard OOD instances. In Table A4, we compare the InD adaptation and OOD detection performance of EATA vs. R3. While the two yield a similar detection performance, R3 noticeably outperforms EATA, demonstrating that the filtering schemes in EATA and R3 may appear similar on the surface, the latter is more effective for the proposed scenario.

W4. Yes, the irrelevant OOD data do not share the label space with training data. Because we do not have access to the test labels, when performing adaptation, R3 remains blind to the distinction between the two and applies the filtering scheme to both relevant shifted InD and irrelevant OOD data.

W5. While R3 does lead to latency increase compared to TENT, we would like to argue that such an increase in the computational cost does not completely undermine the practicality of our method, especially because this is the first time OOD-contamination has been studied in TTA. Furthermore, many of previous TTA methods have utilized techniques that can increase the computational cost, and optimizing the computational cost of TTA is a separate research topic on its own.

Thank you for the authors' responses.

Upon careful examination of the rebuttal and the revised manuscript, I have decided to maintain my initial scoring since I still have some concerns:

• Although the authors have clarified the concept of "shifted InD data," I was unable to identify the corresponding modification in the revised manuscript. Additionally, I recommend a rewrite of the introduction section to enhance the clarity of key concepts, as the current version is challenging to comprehend.

• Regarding the distinctions from and advantages over EATA, I encourage the authors to delve into methodological discussions and analyses rather than merely emphasizing empirical results. A more in-depth exploration from a methodological perspective would enhance the overall understanding.

• It appears that the concept of "OOD-contamination" is not original to this work, as EATA has previously highlighted the detrimental effects of high-entropy samples and the potential benefits of relatively low-entropy samples for TTA. I suggest revisiting the novelty claim and acknowledging the existing contributions in this area.

I appreciate the authors' efforts in addressing these concerns and look forward to the continued refinement of the manuscript.