Conditional Density Ratio Score for Post Hoc Deep Outlier Detection

We would like to thank the reviewer for taking time to give feedback. We are encouraged by the comments that the proposed CDR method is novel, theoretically driven and versatilile (reviewer 9Qn6, 5AjN and hsYo), that the temperature scaling method is effective without OOD samples (reviewer 9Qn6 and hsYo), and that the proposed method perform well empirically (reviewer 9Qn6, hsYo, GZ1a and 5AjN). We address the comments and questions below.

No Novelty

We respectfully disagree and believe that our contribution has sufficient novelty and value. Below, we outline key points that support the originality of our work:

1. Novel CDR Framework: To the best of our knowledge, this is the first work to propose the general CDR framework for post-hoc density estimation. While the specific implementation of the CDR score leverages previously proposed estimators such as the Mahalanobis and KDE estimators, we believe this only scratches the surface of what the CDR framework can achieve. For instance, instead of using a trained classifier to derive $p(\boldsymbol{z}|y)$ through Mahalanobis or KDE estimators, an independently trained likelihood-based neural network could directly estimate $p(\boldsymbol{x}|y)$. The same CDR framework could then combine $p(\boldsymbol{x}|y)$ and $p(y|\boldsymbol{x})$ to estimate $p(\boldsymbol{x})$. We leave this as a direction for future exploration.
2. Temperature Scaling: The proposed temperature scaling method is a crucial component of the CDR framework. Figure 4 and Appendix E (Table 15) show that our temperature scaling method can significantly improve AUROC scores, increasing performance from 72% to 90% in some scenarios. Unlike prior post-hoc methods (e.g., KNN), which rely on OOD samples for hyperparameter tuning, our temperature scaling approach uses only in-distribution samples, making it more practical.

Lack of Baseline Methods

We respectfully disagree. We thank the reviewer for suggesting additional related works, which we will incorporate into our related works section [1-3]. However, we believe it may not be entirely fair to compare our framework with some of the suggested methods for the following reasons: 1. [1-2] are not post-hoc methods, as far as we know; 2. [2-3] rely on large-scale pre-trained models, whereas our method works with any classifier, including those trained from scratch. Despite these differences, we have included two recently published post-hoc OOD detection methods [5, 6] (CVPR 2023 and ICML 2024) in our evaluation. Preliminary results on CIFAR datasets are provided in the manuscript, and experiments on ImageNet are ongoing. The proposed method performs favorably against the two recently proposed method. We will update the paper with a complete set of results as soon as they are available. Lastly, we do not claim that the proposed method is state-of-the-art in post-hoc OOD detection. However, we believe the findings are worth sharing with the community for their novelty and practical value.

Thank you very much for your response. We sincerely appreciate your time and effort in reviewing our work. Below, we aim to further clarify our problem setup and experimental benchmarks.

Limited Baselines

We would like to reiterate that, with the inclusion of the two additional and most recent post hoc benchmark methods [2-3] according to https://github.com/Jingkang50/OpenOOD mentioned in our last response, we believe our study encompasses a comprehensive comparison within the problem setup of “post-hoc OOD detection. with no extra data.” See [2-5] for several recent/representative works with the post hoc problem setup. Specifically:

1. Problem Setup:Our problem setup focuses on post-hoc OOD detection without the use of additional data or training. This is a common and realistic setup, widely focused on in prior works [1-5] (Please see our paper and [1] for a comprehensive list). Within this setup, we utilize a fixed, pretrained classifier $p(y|x)$ for classification tasks such as CIFAR classification in general, with NO explicit focus on OOD detection and over which we have no control. The objective is to perform OOD detection with this fixed classifier and no additional data. We invite the reviewer to refer to Section 3.3 of the OpenOOD paper [1] for further details on this specific problem setup.
2. Comprehensive Benchmarks:We have included an extensive list of benchmarks relevant to this problem setup. Specifically:
   • As shown in Table 1 of the OpenOOD paper [1], it is standard practice to compare methods within the same experimental setup. Our method belongs to the “OSR & OOD Detection (w/o Extra Data, w/o Training)” category, which includes numerous prior works.
   • Consequently, we respectfully disagree with the comment that “this statement holds true only in relation to the very limited baselines used in the paper.” This problem setup is well-recognized and extensively studied.
   • Additionally, some of our baseline benchmarks, such as the KNN methods, perform competitively even across different problem setups.
3. Fair Comparisons:In line with established practices in post-hoc OOD detection literature [1-5], we compare our method against other post-hoc methods with similar experimental setups. We have not "constructed to exclude competitive methods." Instead, we have made an effort to be as inclusive as possible. However, comparing post-hoc methods with more complex methods that require explicit training or additional datasets for OOD detection would be both unfair and impractical in many real-world scenarios. For example:
   • Methods like [6] rely on “a vast number of negative labels from extensive corpus databases” and vision-language models, which may not be feasible in certain applications.
   • For the above reason, methods such as [6] cannot be easily applied to CIFAR or the NLP experiments in Appendix C of our paper due to their dependence on vision-language models.
4. Novelty and Contribution:While we do not claim SOTA performance across the entire OOD detection literature, we believe our proposed method offers significant value:
   • It achieves competitive performance within the well-established and widely studied post-hoc OOD problem setup we focus on.
   • The method itself is novel and holds potential for application in other domains beyond those explored in our work. We are excited to share these findings with the broader community.

Reminder

We would be happy to include additional experiments from other recent works if the reviewer can justify why [1, 2] are insufficient. However, it may not be feasible to update the paper at this stage, as the deadline to upload a revised PDF is tomorrow. That said, we are happy to provide additional details in the comments for now and incorporate any necessary updates in future versions of the paper if applicable.

Paper Revision

We have updated the manuscript to include the clarifications on our experiments. Please see section 4 for details. We would be happy to make further adjustments to the paper if there are any additional concerns and if time permits.

Lastly, we emphasize that, we are very confident that we have conducted a fair comparison of our method against the most suitable benchmark methods using identical experimental setups for all methods implemented.

[1] Liu, Litian, and Yao Qin. "Fast Decision Boundary based Out-of-Distribution Detector." International Conference on Machine Learning, 2024.

[2] Liu, Xixi, Yaroslava Lochman, and Christopher Zach. "Gen: Pushing the limits of softmax-based out-of-distribution detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[3] Tang, Keke, et al. "CORES: Convolutional Response-based Score for Out-of-distribution Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Sun, Yiyou, et al. "Out-of-distribution detection with deep nearest neighbors." International Conference on Machine Learning. PMLR, 2022.

[5] Sharifi, Sina et al. “Gradient-Regularized Out-of-Distribution Detection.” European Conference on Computer Vision (2024).

[6] Chen, Jiankang, et al. "FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detector." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

Thanks to the Authors for the quick response.

---

Limited Baselines
1.
This is less about the setup itself and more about the questionable nature of such self-imposed limitations to post-hoc methods. A method should be compared under identical conditions (dataset and split) against all methods that can operate under the same conditions. I do not expect comparisons to methods under entirely different settings, but many OOD and anomaly detection (e.g. from ADBench) methods, including those mentioned in the review, could be trained and theoretically and practically compared under the same conditions. Not all of them, but at least the top-performing methods.

The construction of the benchmark should at least be more clearly justified and described in the paper. At a minimum, the explanations provided here should be made more explicit in the paper.

See Q2.

I appreciate the improvements in the new version very much. However, once again, this is not the primary reason for the score.

To clarify again: the issue is not with the experiments that were conducted or the baselines and literature used, but rather with the absence of the extensive, more recent literature and methods (see [1] and references from the review).

---

Lack of Datasets
Once again, I am not criticizing the experiments conducted. The issue is what has been omitted. In my view, there are two ways to establish comparability with the literature: either by using the same setup and datasets or by including the methods from these works. Neither was done here to a sufficient extent.

---

Q2
The original question remains: “In Related Work, Post-Hoc Methods, the overlap with Anomaly Detection is emphasized. Why, then, is KNN the only common anomaly detection method used as a baseline?”
To clarify further: You attempt to justify the exclusion of additional baselines by referencing post-hoc methods. However, KNN is not a post-hoc method, so it seems acceptable to include non-post-hoc methods as baselines. Now, while KNN is not a poor anomaly detection method, it quickly reaches its limits, particularly in high-dimensional problems like this one. There are countless other (and superior) anomaly detection methods that could serve as baselines and would perform better in high-dimensional problems as here. Hence, once more: why was KNN chosen as a baseline method while other, more competitive anomaly detection methods were not?

---

Q3
I recommend this work: [1], [4]. Also [6] is possible to consider (ICLR 24). Even if the focus here is not on new methods for OOD detection, the ideas should be taken into account. Futhermore, this work could be used for identification [2]. Even with your given limits, there are other baselines such as [3] and the baselines used in their paper (the non-overlapping ones). And for different methods [5].(other of many much newer works) I also recommend this git https://github.com/huytransformer/Awesome-Out-Of-Distribution-Detection?tab=readme-ov-file#ood-detection for countless other papers from top conferences, many of them from the years after 2022.

---

Reminder
As I mentioned, despite my remaining concerns and the observed limitations, I will reconsider my score once the new version, including additional baseline methods, is published. Although there are still justified doubts and lingering concerns about completeness, I appreciate the additional experiments and inclusion of further baselines.

---

[1] Sharifi, Sina et al. “Gradient-Regularized Out-of-Distribution Detection.” European Conference on Computer Vision (2024).

[2] Yang, Jingkang, et al. "Generalized out-of-distribution detection: A survey." International Journal of Computer Vision (2024): 1-28.

[3] Tang, Keke, et al. "CORES: Convolutional Response-based Score for Out-of-distribution Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Chen, Jiankang, et al. "FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detector." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

[5] Le Bellier, Georges, and Nicolas Audebert. "Detecting Out-Of-Distribution Earth Observation Images with Diffusion Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[6] Jaini, Priyank, Kevin Clark, and Robert Geirhos. "Intriguing properties of generative classifiers." arXiv preprint arXiv:2309.16779 (2023).

You have had ample time throughout this discussion to incorporate, for instance, the additional experiments mentioned in your initial response into the paper itself. Furthermore, your cited sources confirm the possibility of including numerous other baselines. If, as you claim, these comparisons are valid, then the newer methods presented in these references could also have been included as baselines.

It would have been sufficient if you had incorporated either the two methods outlined in your initial response or any of those mentioned in your references into the results section of the paper as baselines. However, you did not do this. Consequently, none of this is reflected in the paper and cannot, therefore, be considered as part of the evaluation criteria. The baselines did not need to be the ones I originally proposed—just additional and particularly more recent ones (more recent than the newest method you used from 2022, such as those cited in your references (e.g. Liu, Litian, and Yao Qin. "Fast Decision Boundary based Out-of-Distribution Detector." International Conference on Machine Learning, 2024.). This work that you cited is not limited to the scope post-hoc methods you mentioned and compares against different methods, they also do not frame KNN as post-hoc method (see Section 4. Experiments, Baselines, for KNN you can also see Yang, Jingkang, et al. "Generalized out-of-distribution detection: A survey." International Journal of Computer Vision (2024): 1-28). For references from 2022 or earlier, it goes without saying that they cannot use more recent work as a baseline if you refer to it.

The basis for my recommendation must be the final version of the paper. As it stands, none of these additional methods were included in the results section, apart from a few minor textual adjustments. To reiterate clearly: there were no additions to the results section compared to the original version. This, along with the evidence provided in your references, reinforces my assessment, which must be based primarily on the content of the paper itself, not on the discussion.

The discussion cannot address the persistent shortcomings of the paper if the additional results and methods cited here as baselines are not reflected in the paper's practical results (beyond adding references). Apart from a few small textual changes, none of my original concerns have been addressed. Therefore, it is not possible to raise the overall evaluation score.

Additional Post-Hoc OOD Detection Methods

We have implemented two recently proposed post-hoc OOD detection methods for comparison. Preliminary results on CIFAR datasets are provided in the table below. In general, the proposed CDR score performs competitively against these methods. We are still finalizing the experiments and will update the paper with the complete results once they are available.

Lastly, we have updated the manuscript to address all the issues pointed out in the review. We invite the reviewer to examine the revised version and provide any additional feedback. We sincerely appreciate the thoughtful and constructive comments, which have greatly improved the quality of our paper.

[1] Liu, Litian, and Yao Qin. "Fast Decision Boundary based Out-of-Distribution Detector." Forty-first International Conference on Machine Learning, 2024.

[2] Liu, Xixi, Yaroslava Lochman, and Christopher Zach. "Gen: Pushing the limits of softmax-based out-of-distribution detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

We would like to thank the reviewers for taking the time to read our paper and provide valuable feedback. We are encouraged by the comments recognizing that the proposed CDR method is novel, theoretically driven, and versatile (reviewers 9Qn6, 5AjN, and hsYo); that the temperature scaling method is effective without OOD samples (reviewers 9Qn6 and hsYo); and that the proposed method performs well empirically (reviewers 9Qn6, hsYo, GZ1a, and 5AjN). Below, we address the comments, suggestions, and questions in detail.

Unclear Motivation: Advantage of CDR over Existing Post-Hoc Density Estimators

We thank the reviewer for this excellent suggestion. We acknowledge that the advantages of CDR over existing post-hoc density estimators, particularly marginalization-based methods, were not explicitly stated. Below, we summarize the key advantages of the proposed method:

1. Novelty: As an exploratory work, we believe the proposed CDR score itself is a significant contribution. Most existing methods rely on marginalization to estimate the marginal density of a distribution, while the CDR score explores an underexplored direction by avoiding marginalization entirely.

2. Principled Combination of Two Approaches: The CDR score offers a principled method to combine $p(y|\boldsymbol{z})$ and $p(\boldsymbol{z}|y)$, both of which have been independently used for OOD detection. By combining these two approaches, the CDR score achieves better performance than either method used in isolation. Furthermore, beyond the post-hoc OOD detection setup, we believe the CDR score can serve as a general framework for density estimation. For instance, instead of using a trained classifier to estimate $p(\boldsymbol{z}|y)$, an independently trained likelihood-based neural network could estimate $p(\boldsymbol{x}|y)$. The CDR framework could then combine $p(\boldsymbol{x}|y)$ and $p(y|\boldsymbol{x})$ to estimate $p(\boldsymbol{x})$ more effectively. We leave this promising direction for future exploration.

3. Robustness to Partition Function Issues: From a theoretical perspective, the energy score is only a valid density estimator if the partition function for normalization is constant. This assumption often fails in practice, leading to inaccuracies. In contrast, the CDR score does not rely on marginalization, allowing it to function as a density estimator even when the partition function is not constant.

4. Empirical Effectiveness: The proposed method demonstrates strong empirical performance compared to existing marginalization-based methods, as shown in our experiments.

We have added a dedicated subsection in the methods section to clearly discuss these advantages.

Related Work: Need for More Comparison with Existing Post-Hoc Methods

We have revised the related works section to include additional discussions comparing the CDR score with existing post-hoc methods, particularly marginalization-based density estimation methods.

Difficulty in Following the Method and Experiment Sections

We have revised both the method and experiment sections to improve readability. To further facilitate understanding, we provide a brief summary of our contributions here. We propose the CDR score, a novel approach to estimating density scores for OOD detection using a trained classifier. This approach relies on $p(y|\boldsymbol{z})$, directly obtained from the classifier, and $p(\boldsymbol{z}|y)$, which requires additional estimation. To estimate $p(\boldsymbol{z}|y)$, we explore the use of both KDE and Mahalanobis estimators. These estimations are computationally efficient, requiring only a trained feature extractor and a small validation set. We conduct extensive experiments on commonly used benchmark datasets to demonstrate the effectiveness of the proposed method.

Relationship Between Temperature Scaling and Model Calibration

Tuning temperatures is straightforward with some labeled OOD samples. However, in this work, we consider a more challenging yet realistic scenario where no OOD samples are available for hyperparameter tuning.

The guiding principles of our temperature scaling method are twofold:

1. Calibration of $p(y|\boldsymbol{z})$: Neural networks are often overconfident, and tuning the temperature can improve log-likelihood [5], which we hypothesize enhances OOD detection performance. This is confirmed by our experiments in Appendix E.
2. Balancing $p(y|\boldsymbol{z})$ and $p(\boldsymbol{z}|y)$: For CDR to be most effective, the scales of $p(y|\boldsymbol{z})$ and $p(\boldsymbol{z}|y)$ should not differ significantly. Empirically, these scores are often orders of magnitude apart, and summing them naively provides little improvement. To address this, we tune the temperature of $p(\boldsymbol{z}|y)$ while fixing $p(y|\boldsymbol{z})$, and add a regularization term to minimize the $L_1$ difference between their scales.

Motivation for Averaging CDR Scores

Under perfect estimation of $p(\boldsymbol{z}|y)$ and $p(y|\boldsymbol{z})$, all individual CDR scores across classes are identical. In practice, however, taking the average helps reduce estimation variance, leading to a better approximation of $p(\boldsymbol{z})$ and improved OOD detection performance.

Additional Post-Hoc OOD Detection Methods

Lastly, we have implemented two recently proposed post-hoc OOD detection methods for comparison. Preliminary results on CIFAR datasets are provided in the table below. In general, the proposed CDR score performs competitively against these methods. We are still finalizing the experiments and will update the paper with the complete results once they are available.

Please do not hesitate to provide further comments or suggestions. Thank you once again for your valuable feedback!

[1] Sun, Yiyou, et al. "Out-of-distribution detection with deep nearest neighbors." International Conference on Machine Learning. PMLR, 2022.

[2] Liu, Weitang, et al. "Energy-based out-of-distribution detection." Advances in neural information processing systems 33 (2020): 21464-21475.

[3] Morteza, Peyman, and Yixuan Li. "Provable guarantees for understanding out-of-distribution detection." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 7. 2022.

[4] Han, Songqiao, et al. "Adbench: Anomaly detection benchmark." Advances in Neural Information Processing Systems 35 (2022): 32142-32159

[5] Guo, Chuan, et al. "On calibration of modern neural networks." International conference on machine learning. PMLR, 2017.

[6] Liu, Litian, and Yao Qin. "Fast Decision Boundary based Out-of-Distribution Detector." Forty-first International Conference on Machine Learning, 2024.

[7] Liu, Xixi, Yaroslava Lochman, and Christopher Zach. "Gen: Pushing the limits of softmax-based out-of-distribution detection." CVPR 2023.

We would like to thank the reviewers for taking the time to read our paper and provide valuable feedback. We are encouraged by the comments recognizing that the proposed CDR method is novel, theoretically driven, and versatile (reviewers 9Qn6, 5AjN, and hsYo); that the temperature scaling method is effective without OOD samples (reviewers 9Qn6 and hsYo); and that the proposed method performs well empirically (reviewers 9Qn6, hsYo, GZ1a, and 5AjN). Below, we address the comments, suggestions, and questions in detail.

Why is $p(\boldsymbol{z})$ a good approximation of $p(\boldsymbol{x})$?

This is an excellent question. Unfortunately, there is currently no definitive answer to this and remains as an open problem to the best of our knowledge. This assumption is commonly made in the post-hoc density estimation literature for OOD detection, as seen in prior methods such as KNN [1], the energy score [2], and GEM [3]. The intuition is that, for a classifier to perform accurately, $p(\boldsymbol{z})$ should retain the most critical information about $p(\boldsymbol{x})$. However, this assumption may be inaccurate in practice, potentially leading to suboptimal performance.

Beyond the post-hoc OOD detection setup, we believe the CDR score can serve as a general framework for density estimators. For example, instead of relying on a trained classifier to obtain $p(\boldsymbol{z}|y)$, an independently trained likelihood-based neural network could directly estimate $p(\boldsymbol{x}|y)$. In this case, we hypothesize that the CDR framework could effectively combine $p(\boldsymbol{x}|y)$ and $p(y|\boldsymbol{x})$ to estimate $p(\boldsymbol{x})$ more accurately. We leave this for future exploration.

Additional Experiments with Other Application Domains

We thank the reviewer for raising this concern. OOD detection benchmarks and tasks have predominantly focused on computer vision. To further validate the applicability of our method, we included additional experiments with NLP datasets. Details are available in Appendix C (we have added pointers in the main paper to guide readers to this section). For simplicity, the features for these experiments were directly obtained from ADBench [4] without modifications. To our knowledge, this is the first use of these NLP datasets for OOD detection evaluation. As seen from the results, our conclusions remain consistent across domains. We emphasize that we used an identical setup for all experiments to ensure fairness and reproducibility.

Concrete Computational Trade-offs Between KDE and Mahalanobis Estimators

We acknowledge that the KDE estimator is computationally more expensive than the Mahalanobis estimator during inference. Specifically:

1. During training, the Mahalanobis estimator requires an additional step to compute the mean and covariance matrix from the validation dataset.
2. During inference, the computational complexity of KDE scales linearly with the number of samples, while the Mahalanobis estimator has constant complexity with respect to the sample size.

However, the number of samples used is kept very small in our experiments. For example, in the ImageNet experiments, we use only 25 samples per class to compute both the mean and variance for the Mahalanobis estimator and the KDE score. Additionally, since the KDE score is computed in $\boldsymbol{z}$, we precompute the latent representation vectors $\boldsymbol{z}$ for all validation samples. This allows for efficient computation of the CDR score, as it only involves a single forward pass through the classifier and some matrix-vector multiplications.

We have included a discussion on these computational trade-offs in the paper.

Why Does CDR Perform Poorly When the Mahalanobis Estimator Fails?

The proposed CDR score can be understood through its decomposition in Equation 2 of the paper. In this decomposition, the CDR score acts as a correction to the energy score using an "average generative conditional ratio" (GCR) term. Since the GCR term depends on the Mahalanobis estimator, poor performance of the Mahalanobis estimator directly impacts the correction, potentially harming OOD detection performance.

We would like to thank the reviewers for taking the time to read our paper and provide valuable feedback. We are encouraged by the comments recognizing that the proposed CDR method is novel, theoretically driven, and versatile (reviewers 9Qn6, 5AjN, and hsYo); that the temperature scaling method is effective without OOD samples (reviewers 9Qn6 and hsYo); and that the proposed method performs well empirically (reviewers 9Qn6, hsYo, GZ1a, and 5AjN). Below, we address the comments, suggestions, and questions in detail.

Additional Experiments with ADBench

We thank the reviewer for suggesting additional benchmarking experiments using the ADBench dataset. We invite the reviewer to refer to the additional experiments conducted with the NLP datasets, as described in Appendix C (we have added comments in the main paper to guide readers to this section). The features for these experiments were directly obtained from the ADBench datasets without modification. As shown in the results, similar conclusions can be drawn.

While we appreciate the suggestion to incorporate further experiments using ADBench datasets to demonstrate the effectiveness of the proposed method, many datasets in ADBench are not well suited to our problem setup, which involves training a classifier to obtain both $p(y|\boldsymbol{z})$ and $p(\boldsymbol{z}|y)$. Most ADBench datasets are designed for anomaly detection and are inherently unlabeled, containing only inlier and outlier samples.

How to Choose Lambda for the OOD Function

Lambda ($\lambda$) can be selected either by using a small labeled OOD dataset or by setting a threshold based on the inlier sample distribution. For instance, the 0.5% tail of the inlier sample distribution can be used as the cutoff threshold.

We acknowledge that choosing $\lambda$ for OOD detection can be challenging in practice. However, we would like to emphasize that the evaluation metrics we use (FPR95 and AUROC) are independent of the choice of $\lambda$.

Additional Post-Hoc OOD Detection Methods

Lastly, we have implemented two recently proposed post-hoc OOD detection methods for comparison. Preliminary results on CIFAR datasets are provided in the table below. In general, the proposed CDR score performs competitively against these methods. We are still finalizing the experiments and will update the paper with the complete results once they are available.

Please do not hesitate to provide further comments or suggestions. Thank you once again for your valuable feedback and support!

[1] Liu, Litian, and Yao Qin. "Fast Decision Boundary based Out-of-Distribution Detector." Forty-first International Conference on Machine Learning, 2024.

[2] Liu, Xixi, Yaroslava Lochman, and Christopher Zach. "Gen: Pushing the limits of softmax-based out-of-distribution detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.