Pathologies of Out-of-Distribution Detection

Thank you for your review. We would like to clarify a few important misunderstandings.

While many OOD detection methods are effective for smaller datasets like CIFAR-10, our paper demonstrates why these fundamental limitations prevent robust detection on more challenging problems like ImageNet-1k. Yang et al. [1] benchmark the performance of many SoTA OOD detection methods such as ViM, Max-Cosine, KNN, ASH-B, ReAct, etc, and find that the best of these methods only achieve an AUROC of 0.807 on ImageNet-1k vs ImageNet-OOD, with some methods scoring an AUROC of 0.593. These methods perform even more poorly under covariate shift, often achieving worse than random chance. To demonstrate that OOD detection cannot easily be solved by scaling up model and data size, we show in Figure 4 that even a ViT-G/14 DINOv2 pre-trained on internet-scale data, indicated by the rightmost points, still has significant irreducible error. Therefore, OOD detection methods still have considerable room for improvement, and optimal performance can only be achieved by rethinking the approach to OOD detection and moving away from using supervised models which are mis-specified for the problem.

[1] Yang et al, ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms

We would also like to clarify that our claims in the paper do not rely on empirical examples; instead, we provide a conceptual understanding of the fundamental limitations of how these methods operate, and we use illustrative examples to show how these conceptual pathologies lead to suboptimal performance in practice. Furthermore, we actually do provide extensive benchmarks to support each of our claims:

• To demonstrate the pathologies of feature-based methods (section 4.1), we provide experiment results using four different models: ResNet-18, ResNet-50, ViT-S/16, and ViT-B/16. We use ImageNet-1k as our ID data, and we test on three different OOD datasets: ImageNet-OOD, Textures, and iNaturalist. We also use a ResNet-18 trained on CIFAR-10 for visualization purposes, since smaller feature dimensions and fewer classes lead to clearer 2D plots; furthermore, we also visualize failure modes for ResNet-50 on ImageNet-1k.
• To understand how these feature-based pathologies scale with model and dataset size (Section 5.1 and Figure 4), we benchmark 12 different models of varying sizes and training methods, as listed in Appendix B.3.
• For pathologies of logit-based methods (Section 4.2 and Figure 3), we show results for ResNet-18, ResNet-50, ViT-S/16, ViT-B/16, trained on ImageNet, and ViT-G/14 trained with DINOv2. We also use a LeNet-5 trained on CIFAR-10 for visualization purposes.
• To understand the scaling behaviors of logit-based methods (Section A.4 and Figure A.7), we evaluate 54 models from timm and torchvision, with 9 different architectures and 6 different training/pre-training setups, listed in Appendix B.2.

While these experimental setups were included in the original paper, we have revised the text to highlight the diversity of the models and datasets we benchmarked. We hope this addresses your concerns about generalizability.

We do not propose yet another method to benchmark; instead, our work explicitly identifies the problem misspecification of the existing approaches which utilize supervised models trained on ID data. We do find that this misspecification is slightly alleviated when the model is first pretrained on very diverse datasets, as the model is then incentivized to learn robust distinguishing features. As such, we encourage practitioners to think more deeply about the data that the model has been exposed to, and we recommend using datasets that are as large and diverse as possible for general OOD detection. However, the goals remain fundamentally unaligned and there are still irreducible sources of error from irrelevant features as shown in Figure 4, so we conclude that more work on OOD detection is needed to develop procedures which directly address the problem.

Prescriptive measures are not necessary for a paper to have scientific merit, and we believe that deepening understanding, critically examining limitations, and prompting further research are also important contributions. Many papers of this variety have been very successful at ICLR, ICML, and NeurIPS. To name a few:
“The marginal value of adaptive gradient methods in machine learning” (NeurIPS 2017, 1278 citations)
“Understanding deep learning requires rethinking generalization” (ICLR 2016 best paper)
“How Good is the Bayes Posterior in Deep Neural Networks Really?” (ICML 2020, 373 citations)
“Bayesian Model Selection, the Marginal Likelihood, and Generalization” (ICML 2022 best paper)
“If Influence Functions are the Answer, Then What is the Question?” (NeurIPS 2022)

We would appreciate it if you would consider re-evaluating our paper in light of these clarifications, and we would be happy to answer any further questions.

We hope our response has addressed your questions, and we would really appreciate further engagement with our work. Given the nature of the misunderstandings, we hope you can re-evaluate our paper in light of our corrections.

In response to your feedback, in order to further demonstrate instances where OOD examples have low uncertainty, we now include additional experiments for the rebuttal, shown in lines 294-300 and Appendix A.2. Specifically, we benchmark 14 different models including ResNets, ViTs, and ConvNeXt V2 on the realistic setting where ImageNet-1K is ID, and we evaluate the performance of MSP for three different OOD datasets: Textures, iNaturalist, and ImageNet-OOD. We record the FPR@95, which indicates how many OOD examples are incorrectly classified as ID due to their low uncertainty (false positive), at a threshold where 95% of ID examples are correctly classified (true positive). The average FPR@95 over all models on all three OOD datasets is 66.5%; thus well over half of OOD examples are classified as ID due to their low uncertainty, and other methods such as max logit, energy score, and entropy all have similar FPR@95s over 60%. These experiment results demonstrate that OOD inputs having low uncertainty is a common pathology across many models and datasets.

We would also like to emphasize that the primary contribution of our paper is to deepen the community's understanding of the limitations and pitfalls of current methodologies. While we agree that it is necessary to develop new approaches for OOD detection, resolving these limitations ourselves would require a significantly broader scope than we intend for our work. In fact, comprehensively addressing the identified pathologies would likely require foundational progress to the field of OOD detection itself. Such a contribution would extend well beyond the typical scope of work in OOD, where many papers focus on incremental improvements to existing methods within established families of approaches. Furthermore, it is worth nothing that the current state of OOD detection reflects years of iterative research by the community, and it may be overly ambitious to expect one section of our paper to offer a complete resolution. Instead, we hope our work will inspire future efforts towards more principled approaches.

Please let us know if you have any further questions or comments, and we look forward to your response.

Thank you for your review! We are glad that you recognize our work “goes beyond incremental improvements," and we address your concerns below.

We explore the impacts of scaling model size and pre-training data in Figure 4 and Figure A.7. We see in Figure 4 that even for the rightmost point representing ViT-G/14 DINOv2 pre-trained on internet-scale data, there is still significant irreducible error (the difference between the perfect performance of 1.0 and the purple triangle) for feature-based methods. This irreducible error can be further broken down into “indistinguishable features” (the difference between perfect 1.0 and the green star), as well as “irrelevant features” (the gap between the blue triangle and the purple triangle). These results demonstrate that the crux of the pathology remains unchanged even for SoTA models and pre-training procedures, although we do see that the larger models do have fewer instances of indistinguishable features compared to smaller ones.

While there have been many OOD detection methods which do not purely rely on uncertainty through MSP, we show that logit-based methods also suffer from the pathologies we described. In Figure A.5, we demonstrate that max-logit, energy score, and entropy all fall victim to the same failure mode where the OOD scores for ID and OOD data entirely overlap. Because a supervised model trained only on ID data is not incentivized towards any behavior for OOD data, there is no guarantee that the logits can be used to distinguish ID and OOD.

Furthermore, in Section 5.2, we also explore SoTA methods, like ViM, which combine feature and logit-based methods to compute the OOD score. We find that these hybrid methods also fail to address the fundamental pathologies caused by the model misspecification. For instance, in the cases where ID and OOD features are indistinguishable, hybrid methods would not be able to provide any benefits.

We hope this has addressed your questions, and we are happy to engage with any further comments.

Thank you for your review and references. We have updated the text to fix the minor issues you pointed out, and we address your concerns below.

Our paper focuses on the problem of OOD detection of semantic shift, where new and unseen classes appear at test-time. We specifically address methods which use the features or logits of supervised models to differentiate between ID and OOD data. Therefore, while problems like OOD generalization for covariate shifts addressed by Teney et al [1] are closely related, they are outside the scope of this paper.

[1] Teney et al, ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets

We demonstrate significant research contributions by being the first to explore why entire families of OOD detection approaches, such as logit-based and feature-based methods, are fundamentally flawed. Because there are many works which criticize specific OOD detection procedures, it may seem that our work lacks novelty. However, many SoTA OOD detection methods focus on adaptations within these existing families, often implying the opposing stance that improved methodology can address these innate limitations.

For instance, in the prior work you referenced by Tajwar et al [2], the authors do address limitations of feature and logit-based methods. However, they focus on understanding the tradeoffs between the methods, and they propose a new feature-based method for low-data regimes, suggesting that choosing the best feature or logit-based method is sufficient. In contrast, we discover that even these novel methods, or any other logit-based or feature-based methods, will have fundamental limitations that arise from the misspecification of supervised training. Moreover, their results utilize toy datasets and low-resolution datasets like CIFAR-10, while we demonstrate that these pathologies exist even with sophisticated models and large datasets. Thank you for the reference; we have added this discussion to our paper. Other OOD detection papers [3, 4, 5] attribute the failures of OOD detection to a weakness in the specific method and propose a novel method within the same family of approaches. No prior work has explored the inherent pathologies of OOD detection methods which cannot be mitigated by improved methodology or method selection.

[2] Tajwar et al, No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets
[3] Sun et al, Out-of-Distribution Detection with Deep Nearest Neighbors
[4] Wei et al, Mitigating Neural Network Overconfidence with Logit Normalization
[5] Wang et al, ViM: Out-Of-Distribution with Virtual-logit Matching\

To this extent, we provide many research contributions which demonstrate the limitations of these methods. We demonstrate the existence of irreducible error for all feature-space methods regardless of methodology, which was previously unexplored and unquantified. Furthermore, we illustrate that this error does not disappear even as we increase model size and data size, disapproving the effectiveness of a commonly proposed fix for OOD detection. Similarly, we also identify sources of irreducible error for logit-based methods, even at scale. We further illuminate the pathologies of potential remedies such as outlier exposure, Bayesian inference, introducing an OOD class, and generative modeling. Although our findings may not seem surprising at first glance (e.g. it seems obvious that feature-based methods will not work for indistinguishable features), our paper provides the important contribution of understanding the extent to which these pathologies occur, even for large models trained on internet-scale data.

We do discuss how pre-training helps reduce the model misspecification issues raised in our paper. We also consider effects of combining various procedures, and the different limitations of generative models. Also, given the generality of our paper, a prescription for a principled general resolution to OOD detection is out of scope and not possible as e.g. a section of a paper. Moreover, prescriptive measures are not necessary for a paper to have strong scientific merit, and we believe that deepening understanding, critically examining limitations, and prompting further research are also important contributions. Many papers of this variety have been very successful at ICLR, ICML, and NeurIPS. To name a few:
“The marginal value of adaptive gradient methods in machine learning” (NeurIPS 2017, 1278 citations)
“Understanding deep learning requires rethinking generalization” (ICLR 2016 best paper, 7552 citations)
“How Good is the Bayes Posterior in Deep Neural Networks Really?” (ICML 2020, 373 citations)
“Bayesian Model Selection, the Marginal Likelihood, and Generalization” (ICML 2022 best paper)
“If Influence Functions are the Answer, Then What is the Question?” (NeurIPS 2022)

We hope this has addressed your questions, and we are happy to engage with any further comments.

We hope our response has addressed your questions, and we would really appreciate further engagement with our work. Given the nature of the misunderstandings, we hope you can re-evaluate our paper in light of our corrections.

We would also like to emphasize that the primary contribution of our paper is to deepen the community's understanding of the limitations and pitfalls of current methodologies. While we agree that it is necessary to develop new approaches for OOD detection, resolving these limitations ourselves would require a significantly broader scope than we intend for our work. In fact, comprehensively addressing the identified pathologies would likely require foundational progress to the field of OOD detection itself. Such a contribution would extend well beyond the typical scope of work in OOD, where many papers focus on incremental improvements to existing methods within established families of approaches. Furthermore, it is worth nothing that the current state of OOD detection reflects years of iterative research by the community, and it may be overly ambitious to expect one section of our paper to offer a complete resolution. Instead, we hope our work will inspire future efforts towards more principled approaches.

Please let us know if you have any further questions or comments, and we look forward to your response.

Thank you for your review. We would like to clarify a few important misunderstandings.

Because there are many works which in some way appear critical of OOD detection procedures, it may seem, superficially, as if our work, which is also critical of OOD detection, lacks novelty. But, diving beneath the surface, this characterization could not be further from the truth. These prior works are focused on adaptations of a family of approaches for better benchmark numbers, not on explaining why a paradigm for OOD detection is fundamentally flawed. For instance, there has been no prior analysis on the inevitable existence of indistinguishable and irrelevant features for feature-based methods, and many papers assume OOD examples will be far from the data without further exploration [1, 2, 3]. Therefore, although some of our findings may not seem surprising at first glance (e.g. it seems obvious that feature-based methods will not work for indistinguishable features), our paper provides the important contribution of understanding the extent to which these pathologies occur, even for large models trained on internet-scale data.

[1] Sun et al, Out-of-Distribution Detection with Deep Nearest Neighbors
[2] Ming et al, How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?
[3] Park et al, Understanding the Feature Norm for Out-of-Distribution Detection

Furthermore, our claims in the paper do not rely on empirical examples; instead, we provide a conceptual understanding of the fundamental limitations of how these methods operate, and we use illustrative examples to show how these conceptual pathologies lead to suboptimal performance in practice. For instance, our section about generative models uses a simple example to explicitly demonstrate how the goals of generative modeling are fundamentally misaligned with the goals of OOD detection; therefore, merely improving the quality of the generative model does nothing to solve the pathology. Furthermore, we actually do provide extensive benchmarks to support each of our claims in the paper:

• To demonstrate the pathologies of feature-based methods (Section 4.1 and Figure 2), we provide experiment results using four different models: ResNet-18, ResNet-50, ViT-S/16, and ViT-B/16. We use ImageNet-1k as our ID data, and we test on three different OOD datasets: ImageNet-OOD, Textures, and iNaturalist. We also use a ResNet-18 trained on CIFAR-10 for visualization purposes, since smaller feature dimensions and fewer classes lead to clearer plots when we only have 2D images; however, we also visualize failure modes for ResNet-50 and ImageNet-1k.
• To understand how these feature-based pathologies scale with model and dataset size (Section 5.1 and Figure 4), we benchmark 12 different models of varying sizes and training methods, as listed in Appendix B.3.
• For pathologies of logit-based methods (Section 4.2 and Figure 3), we show results for ResNet-18, ResNet-50, ViT-S/16, ViT-B/16, trained on ImageNet, and ViT-G/14 trained with DINOv2. We also use a LeNet-5 trained on CIFAR-10 for visualization purposes.
• To understand the scaling behaviors of logit-based methods (Section A.4 and Figure A.7), we evaluate 54 models from timm and torchvision, with 9 different architectures and 6 different training/pre-training setups, listed in Appendix B.2.
• For pathologies of generative models (Section 5.6 and Appendix A.5), we use normalizing flows of eight different sizes trained on CelebA. We also use Diffusion Transformers trained on ImageNet-1k to demonstrate the same pathologies.

While these experimental setups were included in the original paper, we have revised the text to highlight the diversity of the models and datasets we benchmarked. We hope this addresses your concerns about scientific rigor.

Although we demonstrate the impacts of model size, architecture, and training method throughout the paper with the experiments listed above, and we explicitly analyze the effect of pre-training in Appendix A.4, one of our key claims is that these pathologies cannot be resolved by simply improving the model or the training procedure. We see in Figure 4 that even for a ViT-G/14 DINOv2 pre-trained on internet-scale data, there is still significant irreducible error (the difference between the perfect performance of 1.0 and the purple triangle) for feature-based methods. This result demonstrates that the crux of the pathology remains unchanged even for SoTA models and pre-training procedures.

We would appreciate it if you would consider re-evaluating our paper in light of these clarifications, and we would be happy to engage with any further questions.

We hope our response has addressed your questions, and we would really appreciate further engagement with our work. Given the nature of the misunderstandings, we hope you can re-evaluate our paper in light of our corrections.

In response to your feedback, in order to further demonstrate instances where OOD examples have low uncertainty, we now include additional experiments for the rebuttal, shown in lines 294-300 and Appendix A.2. Specifically, we benchmark 14 different models including ResNets, ViTs, and ConvNeXt V2s in the realistic setting where ImageNet-1K is ID, and we evaluate the performance of MSP for three different OOD datasets: Textures, iNaturalist, and ImageNet-OOD. We record the FPR@95, which indicates how many OOD examples are incorrectly classified as ID due to their low uncertainty (false positive), at a threshold where 95% of ID examples are correctly classified (true positive). The average FPR@95 over all models on all three OOD datasets is 66.5%; thus well over half of OOD examples are classified as ID due to their low uncertainty, and other methods such as max logit, energy score, and entropy all have similar FPR@95s over 60%. These experiment results demonstrate that OOD inputs having low uncertainty is a common pathology across many models and datasets.

Please let us know if you have any further questions or comments, and we look forward to your response.

I appreciate the authors' responses. I do not believe there is a significant misunderstanding. Major claims proposed in this work are intuitive and already recognized in the literature.

The statement that "prior works are not focused on explaining why a paradigm for OOD detection is fundamentally flawed" appears to be incorrect. This problem has been well-established and rigorously explored. I recommend authors review [1] and many of its follow-up works. Unfortunately, the paper does not mention or discuss this line of research. Specifically, prior works have explored multiple conditions for OOD detection to be successful (e.g., conditions on hypothesis space and ID-OOD separability).

[1] Fang et al., Is Out-of-Distribution Detection Learnable?, NeurIPS 2022 (Outstanding Paper)

Dear Reviewer K5mb,

This paper has polarized opinions among reviewers. The authors have submitted their rebuttal. Could you please take a moment to review their response? Thank you for your efforts and contributions to ICLR'25.

Best regards,

Your Area Chair

I appreciate the authors for the detailed response and clarifications.

However, I am still not convinced that the phenomena described in the paper constitute novel "pathologies" rather than limitations already evident in the literature. The main reason I mention the reference work on ID-OOD learnability is that a study of learnability presents valuable insights for the research community than what is currently provided in the paper.

As most practical OOD detection methods are heuristics-based (e.g., logit or feature-based), the failure modes stated in the paper are well expected. For example, regarding feature-based OOD detection: "If the OOD and ID features are indistinguishable, then no feature-based methods can perform well." Given that no OOD detection methods achieve perfect performance, for any pre-trained model, one can easily find cases where (near or far) OOD samples overlap with ID in terms of learned features.

While the authors claim this is one mode of pathology, I view this as a natural consequence of the heuristic approaches (among various other factors) being used in practice. A positioning paper would benefit from studying the problem more rigorously, beyond a handful of case studies. There are numerous factors in practice, including training data, loss functions, optimization algorithm, and model architectures, that can influence these behaviors and warrant a rigorous examination.

In response to reviewer feedback, we now include additional experiments for the rebuttal in order to further demonstrate instances where OOD examples have low uncertainty. These updates are reflected in lines 294-300 and Appendix A.2. Specifically, we benchmark 14 different models including ResNets, ViTs, and ConvNeXt V2 on the realistic setting where ImageNet-1K is ID, and we evaluate the performance of MSP for three different OOD datasets: Textures, iNaturalist, and ImageNet-OOD. We record the FPR@95, which indicates how many OOD examples are incorrectly classified as ID due to their low uncertainty (false positive), at a threshold where 95% of ID examples are correctly classified (true positive). The average FPR@95 over all models on all three OOD datasets is 66.5%; thus well over half of OOD examples are classified as ID due to their low uncertainty, and other methods such as max logit, energy score, and entropy all have similar FPR@95s over 60%. These experiment results demonstrate that OOD inputs having low uncertainty is a common pathology across many models and datasets.

Please let us know if you have any further questions or comments!