Out-of-Distribution Detection with Hyperspherical Energy

Near-OOD datasets

Thank you for the valuable suggestions regarding the evaluation of Near-OOD benchmarks. The table below presents a performance comparison of various methods on the suggested dataset NINCO (w.r.t. ImageNet-1k). This comparison highlights the efficacy of the Hyperspherical Energy method, which shows an overall improvement in performance compared to the Energy and CIDER methods.

How does the proposed method perform against the common corruption?

Thank you for your suggestion on testing against corrupted OOD data. In response to this, we've conducted extensive evaluations on CIFAR-10 and CIFAR-100 under AugMix and RandAugment augmentation scenarios on OOD datasets, as detailed in the tables below. These results clearly demonstrate the effectiveness of the Hyperspherical Energy method in dealing with harder OOD datasets (from common corruptions) by consistently achieving top performance compared to traditional Energy score and CIDER in various scenarios and datasets.

Under AugMix Augmentation

Under RandAugment Augmentation

Please specify $\mathbf{z}$ before use in the introduction section.

Thank you for pointing that out! We have carefully reviewed the paper again to ensure that we introduce and describe the notations before using them.

We sincerely thank the reviewer for the detailed review and constructive criticism. We also appreciate the reviewer's recognition of our paper's clarity and strong results. We address the question below in detail.

Q1. Why fine-tune the last residual block only?

That's a great question. We performed fine-tuning on the last residual blocks, mostly to lower computational cost due to limited GPU resources. One can easily use our method and fine-tune on the entire backbone, without stability issues. On small-scale datasets such as CIFAR-10 and CIFAR-100, our training is indeed performed from scratch, on the entire network backbone.

Q2. Difference w.r.t. SIREN

We highlight the major differences below:

• SIREN uses maximum class-conditional likelihood (see Table 1), which is mathematically different from the hyperspherical energy score. Unlike SIREN, hyperspherical energy is theoretically sound for OOD detection, due to its log-likelihood interpretation (see Section 3.1 for details). Our OOD scoring function enjoys rigorous theoretical interpretation from a log-likelihood perspective, while SIREN's does not. Our derivation and interpretation of hyperspherical energy provided in Section 3.1 is an entirely new contribution, relative to SIREN.
• SIREN primarily focuses on representation shaping loss during training time. In contrast, we focus on a novel test-time OOD detection scoring function. Our method is the first to establish the connection between the hyperspherical representations and Helmholtz free energy for OOD detection. From a training-time perspective, we also derive new insight into how the learning objective induces lower hyperspherical energy (Section 3.2), which directly connects to our proposed OOD scoring function.

Q3. Performance on vision transformers

Thanks for the suggestion - these new experiments are underway and will be added to our updated manuscript.

Q4. Could the authors provide experimental details to obtain the results in Table 2? Additionally, error bars and the number of tries on the results would be appreciated.

To provide clarity on the results in Table 2, we focused on measuring the time required to calculate the score for each sample. We obtained these results by averaging the time taken to calculate scores for samples in the ID dataset across each column. The updated Table 2 now includes error bars, indicating the variability in results across samples in the dataset.

Q5. Could the authors provide error bars for their results by training on multiple random seeds? I believe this would strengthen their empirical contributions.

Excellent point. Indeed, we have provided error bars in Figure 3, where the results are based on multiple training over 3 random seeds.

Q6. Limitations

A key strength of our method is its grounded interpretation from a likelihood perspective, as outlined in Lemma 3. This approach, however, necessitates training the model under the von Mises-Fisher (vMF) distribution. While our method achieves accuracy comparable to state-of-the-art models in certain scenarios, it's important to note that it is less common than models trained using traditional cross-entropy loss. This may limit its immediate applicability in environments where standard cross-entropy loss models are the norm. However, we believe that the trustworthiness of the models is worth an effort to train them in a certain way.

We thank you for the thorough and encouraging comments! We appreciate that you recognize the novelty and theoretical soundness of our work. We address your comments below in detail:

Distinct advantages of the proposed method compared to recent OOD detection studies

That's an excellent question! The most significant advantage of our method lies in its principled interpretation from a likelihood perspective (see Lemma 3). While related works [1, 2, 3, 4] have shown empirical promise, they lack a rigorous guarantee and cannot be viewed as likielihood-based test --- which is the fundamental crux for out-of-distribution detection.

Comparison with additional baselines

We have additionally compared with these approaches, and provide the results below for your reference (on CIFAR-100 with ResNet-34 model). Our method outperforms the current best post-hoc method ASH by a significant margin, while enjoying the theoretical rigor.

GradOrth [2] has not yet released their codebase. We will include it in our manuscript once it becomes available!

[1] Huang, Rui, Andrew Geng, and Yixuan Li. "On the importance of gradients for detecting distributional shifts in the wild." Advances in Neural Information Processing Systems 34 (2021): 677-689.

[2] Behpour, Sima, et al. "GradOrth: A Simple yet Efficient Out-of-Distribution Detection with Orthogonal Projection of Gradients." arXiv preprint arXiv:2308.00310 (2023).

[3] Conor Igoe, Youngseog Chung, Ian Char, and Jeff Schneider. How useful are gradients for ood detection really? arXiv preprint arXiv:2205.10439, 2022.

[4] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. arXiv preprint arXiv:2209.09858, 2022.

ID-OOD density plot

Great suggestion! We have included such a plot in our revised draft. Please see Appendix F for details.

Performance on different datasets

The high FPR95 for places365 might be due to the artifact in the dataset itself, rather than the methodology. Despite its common use in literature, the test dataset can be flawed since many images can be distributionally close to ID data. For example, recent work [5] pointed out that Places have some overlapping classes with the ImageNet-1k, which may lead to high FPR95 in evaluation.

To investigate this hypothesis for CIFAR-100, we analyze the FPR95 for each category in Places365. We found that more than half of the 365 types of images have FPR95 of at least 79%. This suggests that there’s a lot of overlap between ID and OOD data, which leads to high rates of false positives. On closer inspection, we saw that certain types of images, like natural landscapes and room scenes, are more likely to have this problem. For example, in the places365 dataset, images in the "television_room" category often get mixed up with "television", "couch", or "table" classes from CIFAR-100 because those objects are in the pictures. This confusion leads to FPR95 to be as high as 97%. Another example is the "storage_room" images in places365. They get classified as "table" or "wardrobe" if those items are in the picture, resulting in an FPR95 of 79%.

[5] Bitterwolf et al., In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation, ICML 2023.

We sincerely appreciate your positive feedback and insightful comments. Your acknowledgment of our work's clarity and unique contribution means a lot to us. We address the questions below in detail.

Q1. What can be said of the absolute OOD detection performance of the proposed method?

Thanks for the insightful question. We believe it's meaingful to look at both the absolute and relative performance. In particular, Table 3 is based on a challenging setting of CIFAR-100 as ID. As you can see, early method such as MSP can have average FPR95 as high as 83.90%. This signifies the non-trivialness of the task itself. This can be attributed to two factors. First, the ID accuracy on CIFAR-100 is suboptimal (75.03% as we report in Appendix Table 5), which means the model does not have a strong inherent characterization of the ID data within its representation. Secondly, the ID data and some OOD data can share spurious correlation (e.g., blue sky background), which further exacerbate the detection difficulty.

Noticeably, in this challenging setting, our method can reduce the FPR95 from the earliest 83.9% to now 38.55% on this challenging task, which we believe is significant progress already. Admittedly, there is still large room for improvement in future research. From a practical point of view, we do desire as low absolute FPR95 as possible. That would be the dream of all researchers in this field, as make more progress :)

Q2. Common failure cases

The high FPR95 for places365 might be due to the artifact in the dataset itself, rather than the methodology. Despite its common use in literature, the test dataset can be flawed since many images can be distributionally close to ID data. For example, recent work [1] pointed out that Places have some overlapping classes with the ImageNet-1k data, which may lead to high FPR95 in evaluation.

To investigate this hypothesis for CIFAR-100, we analyze the FPR95 for each category in the places365 dataset. We found that more than half of the 365 types of images have FPR95 of at least 79%. This suggests that there’s a lot of overlap between ID and OOD data, which leads to high rates of false positives. On closer inspection, we saw that certain types of images, like natural landscapes and room scenes, are more likely to have this problem. For example, in the places365 dataset, images in the "television_room" category often get mixed up with "television", "couch", or "table" classes from CIFAR-100 because those objects are in the pictures. This confusion leads to FPR95 to be as high as 97%. Another example is the "storage_room" images in places365. They get classified as "table" or "wardrobe" if those items are in the picture, resulting in an FPR95 of 79%.

Q3. Limitations

A key strength of our method is its grounded interpretation from a likelihood perspective, as outlined in Lemma 3. This approach, however, necessitates training the model under the von Mises-Fisher (vMF) distribution. While our method achieves accuracy comparable to state-of-the-art models in certain scenarios, it's important to note that it is less common than models trained using traditional cross-entropy loss. This may limit its immediate applicability in environments where standard cross-entropy loss models are the norm. However, we believe that the trustworthiness of the models is worth an effort to train them in a certain way.

Q4. Extension to regression

That's a very interesting question! It may be possible to extend from classification to regression setting. For example, previous work [2] showed the feasibility for regression by optimizing outputs as an interpolation between two prototypes on the hypersphere. It's worth noting that the loss function employed in [2] was different from ours, so we need more conclusive investigation in the future to determine the feasibility.

[1] Bitterwolf et al., In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation, ICML 2023. [2] Mettes et al., Hyperspherical Prototype Networks, NeurIPS 2019

Minor suggestions on writing

We have fixed those writing glitches - thank you for the careful read!