Normalizing Flows For Out of Distribution Detection via Latent Density Estimation

Thank you for your feedback, we appreciate your attention to detail.

More baselines are needed for comparison.

Thank you for the recommended references. We have updated results to include ODIN as an additional benchmark; please note that additional methods do not outperform the existing best baseline, ReAct, on most evaluations. We have also expanded the related works section to discuss more OOD detection methods, including ASH, as recommended.

Missing Details of the CNF of ResNet activations.

Our methods section (3.1) has been clarified with additional technical details; we hope that the amendments make it straightforward to understand and reproduce our results. Please let us know if you feel additional detail is warranted.

More OOD datasets are needed for the CIFAR10 benchmark.

We have expanded the set of OOD datasets used for CIFAR-10 evaluation as advised, including Places365, CelebA, and Gaussian noise. In line with our existing observations, our method is generally competitive but excels for far-OOD data, giving strong results for Places365 and CelebA and exceeding other baseline methods for Gaussian noise.

Thank you again for your detailed feedback, we hope that our amendments clarify and address your concerns.

Thank you for your review, we appreciate your time and feedback.

The authors compare several discriminative model-based OOD detection methods [Energy, MSP], but omit generative model-based OOD detection methods. For instance: [Provable Guarantees for Understanding Out-of-distribution Detection, AAAI 2022]. Moreover, more advanced OOD detection methods are omit for comparison, such as [ASH,Extremely Simple Activation Shaping for Out-of-Distribution Detection, ICLR 2023].

We have expanded our related works section accordingly, and incorporated more generative model-based OOD detection methods (although we only use flows a discriminative manner in our work).

Contribution is limited. [Why Normalizing Flows Fail to Detect Out-of-Distribution Data, NeurIPS 2020] already claimed that normalizing flow on features are better than normalizing flow on the input image.

Thank you for highlighting Kirichenko et al.’s contribution. They do briefly discuss density estimation in the latent space (Sec 8), but their analysis is extremely limited and their results are not strong (Table 2), underperforming other benchmarks we compare to (for example, they report 73.31% AUROC for CIFAR-10 vs SVHN, which all of our tested baselines exceed by at least 13.8%). Our work carries this investigation much further. We have amended our submission to better credit the important contribution of Kirichenko et al.’s prior work, while clarifying our contributions. Our findings that flow models can provide strong post-hoc OOD detection performance on large datasets with latent normalization and undertraining is novel, and we hope these techniques will be a valuable tool for the community.

I have read the responses and will keep my evaluation unchanged.

Thank you for highlighting the active field of VAEs for OOD detection. We have expanded our related work section accordingly. We appreciate that you highlighted a key strength of our paper: it is a post-hoc method and is intended to be applied to pretrained classifier models.

Exploiting the latent space of deep generative models has many existing works, but this paper lacks a discussion and comparison of them [1, 2, 3, 4, 5].

We note that our explanation of our core contributions was perhaps misunderstood. We investigate density estimation in the latent space of a pretrained classifier model, not the latent space of a generative VAE or flow model. The literature listed does not overlap with our main approach: they do not touch on density estimation in the latent spaces of pretrained classifier models, and are not post-hoc methods, but require training of a VAE or flow on pixel data. Because of this training requirement, it may not be trivial to extend the VAE approaches to larger, more complex datasets like full resolution ImageNet. We believe working with pretrained classifier latents remains an important advantage of our method.

Our findings that flow models can provide strong post-hoc OOD detection performance on large datasets with latent normalization and undertraining is novel, and we hope these techniques will be a valuable tool for the community.

Thank you for your very detailed review.

Namely questions remain over lack of sufficient novelty and contradictions within the crux of the message, in important parts

We have updated our submission to emphasize credit given to prior work (in particular Kirichenko et al. 2020) to better clarify the key aspects of our work which are novel, and cleaned up our writing to clarify apparent contradictions.

The submission reports upon an experimental search for better performance down various well-trodden paths (e.g. PCA, early stopping, normalization), however details of innovation are unclear.

The use of PCA, early stopping, and normalization have not been investigated in depth in the context of using normalizing flows for OOD. Our findings that latent normalization and flow undertraining substantially improve OOD detection is a novel contribution, and we hope the community benefits from these findings. For example, Jiang et al. 2020 applies PCA to reduce their embeddings to 10D for rare example mining; our results show that an unreduced latent space may improve their performance.

Sec 2.3 states both that: 'Normalizing flows [...] have historically performed very poorly for OOD detection in image classification' and also: 'Zhang et al. demonstrate strong OOD detection performance using normalizing flows in image classification' The abstract claims to present a method 'avoiding researcher bias in OOD sample selection' yet Sec. 6.2 strongly recommends that practitioners 'evaluate [...] on both the original validation and a representative OOD dataset'.

Thank you for the suggestions to improve the presentation of contributions in our work. We have clarified that Zhang et al. is not a post-hoc method and has strong but limited results. We have additionally clarified that all our results are produced with exactly one epoch of flow training to ensure consistent evaluations across models and datasets. We have removed the discussion of guiding early stopping using a representative OOD dataset, as although this technique can further improve OOD performance for a specific OOD set, this distracts from the ability to operate without such OOD sample guidance.

Regarding additional recent works in OOD detection: we have expanded our related works section, added ODIN as a comparison method, and added additional datasets for comparison against CIFAR-10.

It has been noted that, in performing OOD detection, the inductive biases of normalizing flows can cause difficulty for image space learning. One interesting question going forward, might be to look at how such biases interact with the noted differences in latent distributions that arise from supervised, unsupervised backbones models (Sec 6.5). Since flows tend to learn representations that achieve high likelihood through local pixel correlations in the image space, rather than discovering semantic structure, can analogous observations be drawn about differences in the 'shapes' of (un)supervised latent spaces?

We appreciate your curiosity into the results of section 6.5: the connection between the structure and shape of ID data and OOD data in latent space and the performance of our method for OOD detection is also intriguing to us but remains an area of active research. We look forward to expanding on the investigation and results of section 6.5 in future work beyond the scope of this paper.

Thank you for your detailed and focused feedback, we hope our amendments address your concerns.

Thank you for your detailed review, we appreciate your feedback and are glad you enjoyed our in-depth discussion.

The core method of modeling flows in the latent space lacks novelty: [Kirichenko et al., 2020], which the authors cite, explicitly mentions that while flows on the input-space don't perform well, latent-space flows perform much better.

Thank you for highlighting this. Kirichenko et al. 2020 does briefly discuss density estimation in the latent space (Sec 8), but their analysis is extremely limited and their results do not exceed other state-of-the-art methods in terms of accuracy. Our work carries this investigation much further and demonstrates compelling results that do present NF-based OOD detection as a competitive method. We have amended our work to clarify and credit their prior discussion of this point.

The authors claim a benefit of their method does not require OOD data. However, in the discussion section, the authors state that representative OOD data is useful to determine when to stop flow training. If early stopping is necessary for good flow performance, the authors should detail how they chose when to stop training in the models used to evaluate the results.

We acknowledge that referring to this as early-stopping was misleading. We have updated our wording to clarify the fixed short-duration training approach and have removed reference to using representative OOD data to determine when to stop flow training. All of our results were produced by training the normalizing flow for exactly one epoch as a matter of convention, as this was found to produce consistently strong results across multiple backbone models and datasets.

The experimental evaluation is limited:

We have updated results to include ODIN, although additional methods will not outperform the existing best baseline, ReAct, on most evaluations. We have also added additional datasets for evaluation against CIFAR-10. Reporting error bars on AUROC is possible but not widespread in the literature, as output variation is both limited and fairly consistent across methods. We are actively researching the connection between OOD detection and compactness of latent representations (as discussed in sec 6.5) but further discussion is beyond the scope of this paper.

Questions: The discussion section of the paper is frank about the practical challenges in getting flow models to work well for OOD detection, and that, in particular, there are considerations on early stopping during training. How did the authors choose when to stop training for the flow models whose OOD detection results are shown in Tables 1 and 2?

As highlighted in section 6.2, peak OOD detection performance is a function of flow training time that is dependent on the dataset and backbone model. To ensure our method is generally applicable and to avoid reliance on OOD data, all of our results are generated by training the flow model for only one epoch on the training dataset. This produces consistently strong, although not always optimal, results across several backbone models and datasets.