Test Time Augmentations are Worth One Million Images for Out-of-Distribution Detection

Thank you for your attention to detail. We address your comments below:

• Layout Improvements: We apologize for the layout issues. We have carefully revised the manuscript to address the spacing of the above lines (lines 40, 48, 65-68, and 93-107) as well as throughout the article.
• Reference Error in Figure 1: Thank you for your careful review of the manuscript. We have corrected the style of the references and used the correct formatting throughout the manuscript.
• Score in Table 1: The score reported in Table 1 is the AUROC (Area Under the Receiver Operating Characteristic curve). We will clarify this in the table caption and the relevant section of the text to avoid confusion.
• Stability of IDAs Across Datasets: You make an excellent observation about the dataset dependency of LPIPS ordering. We think this variance occurs because of the feature complexity difference between Cifar10 and ImageNet. Despite this difference, it can be observed from Table 2 that the augmentation method with the lowest LPIPS on different datasets is always Mask, and its LPIPS is much smaller than the other augmentations, indicating that Mask is not dataset-dependent. This is why we chose mask as our method in our method design in section 3.
• Rigorous Definition of IDA and OODA: Our definitions of IDA and OODA are derived from [RR1], which distinguish between them by whether they affect the expression of image features. However, it is difficult to formulate a quantitative metric to define IDA and OODA because different datasets have different sensitivities to different data augmentation methods, e.g., MNIST for grey scale changes and Texture for vertical flipping. To provide clearer differentiation, we introduce two methods in the paper to assist in distinguishing between IDA and OODA: one by visualizing the results (Fig. 3 and Fig. 10), and the other by quantifying the effect of enhancement on image features with the help of LPIPS (Table 2). Although LPIPS seems to be a good quantitative metric, Table 2 shows the average LPIPS of 1000 images, which makes it difficult to strictly distinguish between IDA and OODA. We believe that defining a rigorous metric to differentiate between IDA and OODA is a worthwhile future direction to explore.

We appreciate your careful review and believe that addressing these points will improve the presentation and clarity of our manuscript.

[RR1] How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization. ICLR. 2023

Thank you very much for reviewing our paper! We really appreciate your time and effort in providing these helpful comments. We have done our best to address each point you raised and made improvements based on your suggestions.

Below are our responses to your comments. We hope you'll have a chance to take a look!

Thank you for your valuable feedback. We address your concerns below:

Lack of Preliminary Section: We have added a preliminary subsection in the method section to the revised manuscript, which provides the necessary background information on OOD detection, including common definitions and problem settings. We believe this addition will improve the paper's accessibility and context for readers less familiar with the field.

Lack of Algorithm Description: We have added a dedicated algorithm box clearly outlining the steps involved in our TTA-enhanced KNN approach in the Appendix. This will include details on the sequential masking strategy, embedding extraction, KNN search within the augmented set, and the final OOD score calculation.

Unclear OOD Score in Tables 1 and 2: We apologize for the lack of clarity. The OOD score used in Table 1 is AUC and the metric in Table 2 is LPIPS. Table 1 shows the IDA's effectiveness for OOD detection compared to OODA, and Table 2 shows that masking is the mildest augmentation, resulting in the lowest LPIPS score, which inspire the mask-based method design described in Section 3. We have made this clear in the headings of the tables in the latest version.

Adaptability to Other OOD Scores: Our TTA-based OOD detection method relies on the similarity between original and augmented samples within a specific representation space. This makes a straightforward combination with post-hoc scoring methods like MSP challenging. However, as demonstrated in Figure 6, our similarity-based approach remains effective when applied to logits and softmax output. Furthermore, it's compatible and synergistic with post-hoc methods that modify the representation space, such as ReAct, leading to improved performance as shown in Table 5.

Adaptation to Other Architectures and Baselines: We initially compared our method's performance against baselines using the Swin Transformer (Appendix F, Table 13). To provide a more comprehensive evaluation, we expanded these experiments to encompass additional baselines and OOD datasets as follows:

The results demonstrate that our method maintains strong performance with transformer-based architectures like Swin Transformer, generally outperforming most baselines and achieving a performance second only to ASH (consistent with the findings in Table 4). This suggests our method's model-agnostic nature, unlike some baselines (e.g., DICE), as its performance remains robust across different model architectures. The updated results are included in the latest manuscript.

Performance on More Challenging Benchmarks: According to the suggestion in OpenOOD [RR1], we evaluated our approach on the Near-OOD dataset in Table 3 and Table 4, which are Cifar10 for Cifar100 and NINCO and SSB-hard for ImageNet. Our method consistently outperforms baseline approaches on hard-OOD detection tasks, as evidenced by the superior performance on near-OOD datasets like CIFAR-100 (90.79% AUC), NINCO (82.19% AUC) and SSB-hard (71.43% AUC). This advantage can be attributed to the ability of sequential masking to capture subtle feature differences between InD and near-OOD samples. These results suggest that our TTA-based approach is particularly effective for challenging scenarios where traditional methods often struggle to maintain reliable detection performance. For pseudo-relevant benchmarks, we have not found relevant OOD test benchmarks. However, we will discuss this as a valuable direction for future work.

We believe that addressing these points will significantly improve the clarity and completeness of our paper and better demonstrate the effectiveness and generalizability of our proposed method. Thank you again for your insightful comments.

[RR1] Yang J, Wang P, Zou D, et al. Openood: Benchmarking generalized out-of-distribution detection[J]. Advances in Neural Information Processing Systems, 2022, 35: 32598-32611.

Thank you very much for reviewing our paper! We really appreciate your time and effort in providing these helpful comments. We have done our best to address each point you raised and made improvements based on your suggestions.

Below are our responses to your comments. We hope you'll have a chance to take a look!

Thank you for the constructive feedback and valuable suggestions. We address each point below:

Novelty and Baselines: We appreciate the acknowledgment of our method's novelty. According to reviewers’ comments, we add comparisons with more baselines such as GradNorm, GEN, DICE and NAC on both Cifar10 and ImageNet as follows:

As shown in the tables, our method surpassed all baseline approaches on both Cifar10 and ImageNet datasets, with particularly strong performance on Near-OOD data, validating both its effectiveness and reliability. We have updated the relevant data to the latest manuscript.

Additional Architectures: Thank you for this important suggestion. We have briefly compared the performance of our method with the baseline method on the Swin Transformer in Table 13 (Appendix F). To further show the result, we extend the experiments to include more baseline methods and OOD datasets as follows:

It can be seen that our method maintained strong performance on transformer-based architectures, Swin Transformer, where it exceeded most baselines (ranking second only to ASH, aligning with Table 4's findings). This shows that our method is model-agnostic and its performance is not affected by the model architecture as some baseline methods (e.g., DICE) are. We have updated the table in the latest manuscript.

Splitting Table 1: Table 1 shows the performance of IDA and OODA on Cifar10, all tests were performed in the inference phase, independent of training data. The table shows that IDAs can be used for OOD detection and that using multiple IDAs is superior to a single IDA. We think the reviewer's comment about readability is directed to Tables 3 and 4, where we have made relevant notations in the latest manuscript.

Generalizability of TTA: Since our TTA-based method detects OOD by the similarity between the original sample and augmented sample on certain representation space, it cannot be simply combined with other post-hoc scoring methods, such as Energy and GEN (since they are scalar not a vector). However, Figure 6 in the main paper shows that our method is still valid for calculating similarity on logits and softmax. In addition, for some post-hoc methods that rectify the representation space such as ReAct, our approach works well with them and achieves better performance (as shown in Table 5 in the main paper).

We thank you again for your recognition, and valuable comments, and we believe that addressing these issues will improve the presentation and clarity of our manuscripts.

Thank you very much for reviewing our paper! We really appreciate your time and effort in providing these helpful comments. We have done our best to address each point you raised and made improvements based on your suggestions.

Below are our responses to your comments. We hope you'll have a chance to take a look!

Lack of Theoretical Analysis: Our approach builds on findings from [RR4], which demonstrated mild augmentation's superiority over aggressive augmentation for OOD detection during training. Motivated by this finding, We hypothesize that mild augmentation (IDA) is more suitable for OOD detection than aggressive augmentation (OODA) during test time. Our empirical results in Table 1 confirm this hypothesis. In addition, we show in Table 2 that Mask is the mildest data augmentation method through LPIPS, and design a sequential masking TTA method for OOD detection. While lacking formal theoretical proof, our work presents a coherent progression from assumption to empirical validation.

Computational Cost: While our method requires 25 inferences, parallel processing minimizes the additional runtime. Our computational overhead is actually lower than several baselines, such as gradient-based methods (DINO, GradNorm) that require backpropagation, or KNN which must compute distances to the whole reference set. We have evaluated the per-sample runtime of common OOD detection methods on an Nvidia 3090, averaging over 1000 samples for each. Our method demonstrates a favorable balance between performance and runtime.

Generalizability to Different Architectures: Building on our initial Swin Transformer comparison in Table 13 (Appendix F), we've expanded our evaluation to include additional baseline methods and OOD datasets:

The results demonstrate our method's architecture-agnostic nature, performing strongly on Swin Transformer and exceeding most baselines (second only to ASH, consistent with Table 4). Unlike architecture-dependent approaches such as DICE, our method maintains its effectiveness across different model architectures. These results are updated in the latest manuscript.

We believe that addressing these points will significantly strengthen the paper and better highlight its contributions. We appreciate the reviewer's feedback.

[RR4] How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization. ICLR. 2023

We thank the reviewer for their insightful comments and suggestions. We address each point below:

Limited Novelty: We appreciate the reviewer for providing relevant literature. Our method introduces several key innovations:

• Study the Effect of Augmentations on OOD Detection: Our comprehensive investigation into the impact of test-time data augmentation on OOD detection in Section 2 reveals that employing mild augmentation strategies leads to more favorable performance. This focus on mild TTAs and their unique benefits is a novel aspect of our work.
• New Method for OOD Detection: Our key contribution lies in the novel use of test-time augmentations (Sequential Mask) with a KNN-based approach for OOD detection. To the best of our knowledge, this specific combination has not been explored before.
• Method Agnostic Approach: Unlike methods based on training or model output, our method is model-agnostic and can be applied to any pre-trained classifier without requiring additional training or manipulation.

For the references [RR1] and [RR2] given by the reviewer, although they both consider augmentation methods, they require training phases: [RR1] utilizes blurred images during training, while [RR2] trains a Siamese network for tabular data anomaly detection in an unsupervised manner. In contrast, our method innovates by detecting OOD data through a direct comparison of original and augmented samples, eliminating the need for additional training or model modifications—a key distinction from existing augmentation-based OOD detection approaches.

Misunderstanding of InD-Independent Approaches: We apologize for the potential misunderstanding. Possibly due to a clerical error by the reviewer, what we meant in the paper (from line 45-69) is that InD-dependent methods (KNN, ReAct, ViM) require the collection of a large amount of InD data to be used as a reference set, which makes the performance of the method potentially affected by the quantity and quality of the InD data (as shown in Fig. 1). Therefore, we aim to develop an InD-independent method.

Discussion with DICE, ASH, and [RR3]: This is a great suggestion. In response to reviewer feedback, we expanded our evaluation to include additional baseline methods (GradNorm, GEN, DICE, and NAC) on both Cifar10 and ImageNet datasets:

The results show that our approach outperforms all baseline methods across both Cifar10 and ImageNet datasets, notably excelling on Near-OOD data.

For ASH, while it achieves better results on ImageNet, our method surpasses it on Cifar10. In addition, ASH has different optimal strategies on Cifar10 and ImageNet. More importantly, we believe that not only the best-performing method is valuable. Our method provides a new direction for OOD detection and has impressive performance (only weaker than ASH on ImageNet).

These updated results have been incorporated into the latest manuscript.

[RR1] Choi, Sungik, and Sae-Young Chung. "Novelty detection via blurring." ICLR 2019.

[RR2] Cohen, Seffi, Niv Goldshlager, Lior Rokach, and Bracha Shapira. "Boosting anomaly detection using unsupervised diverse test-time augmentation." Information Sciences 626 (2023): 821-836.

[RR3] Liu, Yibing, Chris Xing Tian, Haoliang Li, Lei Ma, and Shiqi Wang. "Neuron activation coverage: Rethinking out-of-distribution detection and generalization." ICLR 2024