On Diffusion Modeling for Anomaly Detection

We thank the reviewer for their careful and thorough review.

The contribution of this paper appears limited. There are several existing works on anomaly detection utilizing diffusion models, such as those mentioned below: [1] Unsupervised Surface Anomaly Detection with Diffusion Probabilistic Model, ICCV23. [2] Feature Prediction Diffusion Model for Video Anomaly Detection, ICCV23. [3] DiffusionAD: Denoising Diffusion for Anomaly Detection, Arxiv. [4] Diffusion models for medical anomaly detection, MCCAI22. Can the authors point out the unique contributions of this paper compared with them?

The main difference between DTE and other applications of diffusion model to anomaly detection, including [1-4] is that DTE does not learn a generative model. We note in section 3 that “we propose a much simpler approach that does not require modeling the reverse diffusion process but instead models the distribution over diffusion time corresponding to noisy input samples.” As a result, not only this is orders of magnitude faster than learning a generative diffusion model (Figure 1 and 11), but it also performs better (Figure 5). Another distinguishing feature is that we are focused on finding anomalous samples rather than identifying localized anomalies within images. We would also like to clarify that papers [1,2] are published after the May 28, 2023, which is ICLR cut-off deadline for citation, and papers [3,4] are already cited and discussed in Section 5. Nevertheless, we have added a discussion of [1] to section 5 and cited both [1,2].

The motivation and rationale for introducing the diffusion model into anomaly detection are somewhat unclear. The authors should emphasize it.

Historically, all generative modelling techniques are used for anomaly detection since they can implicitly identify low-probability samples. Diffusion models are not an exception. As we explain in Section 3, “The reverse diffusion process implicitly learns the score function of the data distribution and can be used for the likelihood-based identification of anomalies.” We then further investigate this application of diffusion modelling for anomaly detection and find an alternative use of diffusion, explained in the previous answer, leading us to DTE.

Authors should compare with some latest AD methods proposed in 2023. Besides, it would be more convincing to include diffusion model-based anomaly detection approaches.

ADBench was used as our benchmark and was published at NeurIPS 2022. As discussed in Section 5, We include all baselines from that benchmark in addition to ICL (ICLR 2022), DROCC (ICML 2020), GOAD (ICLR 2020). We updated the paper using two new 2023 baselines that were published after the cut-off deadline: SLAD [1] and DIF [2]. DTE outperforms both methods overall on ADBench. We found that methods created specifically for image anomaly detection tend to perform poorly on tabular datasets, and most of the recent papers are tailored for images. As discussed in the paper, the latest anomaly detection methods, such as [3] and [4], use variations on vanilla DDPM, where the difference between the original input and its reconstruction is used. We do compare to this baseline and find DTE superior to both in efficiency and accuracy.

[1] Hongzuo Xu, Yijie Wang, Juhui Wei, Songlei Jian, Yizhou Li, and Ning Liu. Fascinating supervisory signals and where to find them: Deep anomaly detection with scale learning. In Proceedings of the 40th International Conference on Machine Learning, ICML’23

[2] Hongzuo Xu, Guansong Pang, Yijie Wang, and Yongjun Wang. Deep isolation forest for anomaly detection. IEEE Transactions on Knowledge and Data Engineering, pp. 1–14, 2023a.

[3] Julia Wolleb, Florentin Bieder, Robin Sandk ̈uhler, and Philippe C Cattin. Diffusion models for medical anomaly detection. In Medical Image Computing and Computer Assisted Intervention– MICCAI 2022: 25th International Conference, Proceedings, Part VIII, pp. 35–45. Springer, 2022.

[4] Julian Wyatt, Adam Leach, Sebastian M. Schmon, and Chris G. Willcocks. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 649–655, 2022.

The reviewer has thoroughly reviewed the experimental results presented in the appendix. The authors have done a comprehensive experiment by including the results from dozens of datasets in this paper. However, the reviewer observed that the proposed method did not consistently outperform other methods on each individual dataset. It achieved the best performance only on a subset of the datasets.

Due to dataset diversity in ADBench, it is indeed challenging for a single method to consistently outperform others across all datasets. Our method shows competitive performance and, in many cases, outperforms other methods on an aggregate score across various datasets while remaining the most efficient in inference time.

We thank the reviewer for their feedback.

The performance in the semi-supervised setting is more competitive compared to the unsupervised setting. This indicates that DTE benefits from labeled data, allowing for a more accurate modeling of the distribution of diffusion time.

How does varying the ratio of labeled data in the semi-supervised setting affect the performance of DTE? Can this method be extend to more challenging tasks, such as the localization of anomaly in data?

Since the question and the point raised under weakness are related, we answer both at the same time: As mentioned in Section 2, the semi-supervised setting used in the paper is also known as one-class classification, where the model only has access to normal samples when training and no labeled data. The ratio is the number of normal samples used for training on the normal samples used for testing. Increasing the ratio simply increases the training data, which should increase the model performance, but can make the test less accurate. This setting is more competitive compared to the unsupervised setting since it does not have any anomalies in its training data, allowing the model to learn the distribution of the normal data more accurately.

We thank the reviewer for their detailed feedback and suggestions.

The performance of the DTE models seems to be upper bounded by the simple kNN-based AD method. There are a number of kNN-based AD methods, including some deep methods like Refs [1]. It would be helpful for the empirical evidence support if these advanced kNN variants are included in the empirical comparison.

Thanks for the pointer; we have now cited this paper and briefly discussed it. We would like to point out that while kNN remains one of the top-performing methods, DTE (parametric) is not upper-bounded by kNN: DTE (categorical, inverse Gamma) outperforms kNN on several datasets, as shown in Tables 13-18. Moreover, as we show, while DTE (parametric) is comparable with kNN/DTE(non-parametric), it has a significant efficiency advantage (Figure 1 and 11).

The referenced paper uses unsupervised embedding followed by kNN in the embedding space. Appendix D presents ablations that compare kNN/DTE (non-parametric) and DTE parametric on embeddings produced through self-supervised learning and pre-trained models. Indeed, many of the datasets in ADBench are embeddings produced by pre-trained models. We generally observe better performance for all methods when using high-quality embeddings.

The models rely on distance in original feature space, and they would fail to work if the data lies in very high-dimensional space, e.g., datasets with hundreds of thousands of features or millions of features.

It is informative to consider this point for diffusion models used for generative modeling: there again, while using latent diffusion can improve results for high-dimensional inputs, diffusion modeling remains competitive in the observation space. Similarly, as also discussed in answer to previous question, using embedding improves the performance of our method. We would also like to note that distance to nearest neighbours (in case of DTE non-parametric) can still be reliable in high-dimensions due to manifold hypothesis (i.e., distance to closes point in high-dimension resembles the distance on data-manifold and therefore the distance in any isometric embedding). Overall, we do observe that for high-dimensional data, performance on learned embeddings is better; see Appendix D.

Since the method is based on generative models, it is important to discuss and compare with other generative model-based AD methods, such as GAN-based methods, to highlight the advantages of the proposed method.

We agree with the reviewer. In addition to the DDPM and the normalizing flows baselines, we have added a comparison on both semi-supervised and unsupervised AD to a GAN-based method from ADBench and VAE-based method (implementation from PyOD). DTE outperforms all these methods in terms of effectiveness and efficiency. We’ve added a sentence identifying the AD baselines that rely on generative models, per the reviewer’s suggestion.

Since the models directly work on tabular datasets, it is misleading to claim that the evaluation is performed on diverse tabular, image, and natural language datasets.

We have reworded this sentence, which now reads “tabular data and embeddings of images and natural language datasets.”

The work may be improved by having more discussion on recent diffusion model-based AD studies, such as [2-4].

Thanks for the pointers; we have cited them and briefly discussed [1,2] in the updated paper. We would also like to note that two of these papers are on video anomaly detection, and all were published after May 28, 2023, which is the cut-off date according to ICLR guidelines.

[1] Learning representations of ultrahigh-dimensional data for random distance-based outlier detection. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2041-2050).

[2] Unsupervised Surface Anomaly Detection with Diffusion Probabilistic Model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6782-6791).

[3] Feature Prediction Diffusion Model for Video Anomaly Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 5527-5537).

[4] Multimodal Motion Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 10318-10329).

We thank the reviewer for their insightful comments.

The advantage of DTE methods hold should be demonstrated quantitatively. As the DTE methods perform worse than kNN in both quality and efficiency, quantitative results are recommended to demonstrate the distinctions in scalability of DTE methods

This is a misunderstanding. As seen in figure 1, DTE (parametric) is significantly more efficient than KNN, which we show is similar to DTE (non-parametric). Exact computation times for training and inference are given in Appendix C.4. In terms of performance, the reviewer is correct that the average performance of DTE (parametric) and KNN is comparable. This is where we believe the efficiency of DTE can facilitate its applications to large datasets.

The presentation in Figure 3 needs optimization. To demonstrate the small difference between non-parametric estimate and analytical posterior, the visualization of residual part seems more straightforward.

Thank you for the suggestion. It makes sense, and we will improve this in the next revision (we are considering several alternatives.)