A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection

The article compares many methods, on many datasets, with many metrics to give a complete assessment. However, the description of these methods, datasets, and metrics is very superficial. The authors should provide a brief overview of the key innovations for each method, summarize the distinguishing characteristics of the datasets, and explain the strengths and limitations of each evaluation metric.

The methods are introduced in one paragraph and then quickly grouped into four broad categories. The article lacks a more detailed description of the methods that are later evaluated. We don't know the order in which these methods were introduced, or what deficiencies they attempted to correct with new design proposals. This information is important for understanding the evaluation tables and for validating or invalidating the claims of the methods. The lack of this information makes the analysis much more difficult for someone without a detailed and up-to-date knowledge of the field. However, this type of benchmark article is also usually intended to serve as a starting point for researchers and engineers entering the field or doing implementations.

The same comment applies to datasets, which are also introduced in a paragraph and a table. Aside from the "Dataset Analysis" paragraph at the end of the article, there is little information about what makes datasets different. We don't know how they work, how diverse they are, or what they actually represent. It would be desirable to have a qualitative comparison of the datasets from the beginning to better understand the evaluation. As mentioned above, Mvtec-AD is simple and can be considered a solved dataset, while Visa and RealIAD are more difficult. As the authors have chosen to present the tables of these datasets in the article, the differences should be presented from the beginning.

Finally, since this paper points out that the methods used are not very informative and suggests the use of others, they should be explained. There are 9 metrics used in this paper, and it would be necessary to explain the advantages of one metric over another. We need to know if they are complementary and what their variance and invariance are, otherwise the tables can't be analyzed in detail.

In the paragraph "Training-free PatchCore" it says: "PatchCore Roth et al. (2022) does not require model training. It extracts all features from the training data, then selects a core subset and stores it in a Memory Bank. During testing, each test image is compared with the Memory Bank to compute an "anomaly score"", which is not exactly the case. Patchcore has a training procedure, which is the extraction and sub-sampling of features from the train base. This procedure is performed only once, and then the resulting memory bank is simply accessed to perform a nearest-neighbor measurement with new features. In fact, there's a problem in Table 1, because Patchcore should have a training time. The authors should revise their description of PatchCore to accurately reflect its feature extraction and sub-sampling process, and they should include the training time for PatchCore in Table 1.

Finally, the authors claim that Patchcore can't run on datasets larger than Mvtec-AD, but this doesn't seem to be the case, as articles [1,2] with Patchcore's AUROC for VISA can be found. It's important to evaluate Patchcore on all datasets, as it's one of the best and one of the only ones that doesn't involve training a network. The authors should either evaluate PatchCore on all datasets or provide a clear explanation for why it was not evaluated on certain datasets, citing any technical limitations or resource constraints they encountered.

[1] Fučka, Matic, Vitjan Zavrtanik, and Danijel Skočaj. "TransFusion–A transparency-based diffusion model for anomaly detection." European Conference on Computer Vision. Springer, Cham, 2025.

[2] Zhang, Hui, et al. "DiffusionAD: Norm-guided one-step denoising diffusion for anomaly detection." arXiv preprint arXiv:2303.08730 (2023).

For reconstruction methods, with the current trend in image generation by diffusion, methods using it have been proposed. One is shown in this article, but others [2] that announce state-of-the-art performance should be included. In general, some methods [2,3,4] that have a high AUROC on VISA should be included, since VISA seems to be the next dataset to be solved after Mvtec for the industrial problem. The authors should consider including these specific diffusion-based methods and other high-performing methods, or they should explain their criteria for method selection if these were intentionally excluded.

[3] Fučka, Matic, Vitjan Zavrtanik, and Danijel Skočaj. "TransFusion–A transparency-based diffusion model for anomaly detection." European Conference on Computer Vision. Springer, Cham, 2025.

[4] Batzner, Kilian, Lars Heckler, and Rebecca König. "Efficientad: Accurate visual anomaly detection at millisecond-level latencies." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

• Reproducibility of experimental errors: The benchmark results are based on single-run outcomes without calculating the standard deviation from repeated runs. What is the rationale behind this approach? Providing an analysis of randomness and reproducibility would enhance the credibility of the paper.

• The meaning of evaluation metrics should be explained in detail.

• Support for Zero-/Few-shot Anomaly Detection (AD) tasks: The authors primarily focus on multi-class AD. Can the proposed method support the recently popular Zero-/Few-shot AD models, such as WinCLIP? This would further expand the applicability of the AD system.

• Some values are missing in Table 6.

• It would be great if conventional class-separate UAD setting (one-class-one-model) is also benchmarked using this framework.

• Minor issues:

  • There are broken links in Line 374 and Line 430.
  • The text in Fig. 1 is too small.

1. The novelty is not at the ICLR level. The primary aim of this paper is to integrate methods, datasets, and evaluation metrics in the field of visual anomaly detection to create a comprehensive modular framework. All the methods, datasets, and evaluation metrics included are based on existing technologies.
2. As the core content of this paper, the details of the ADer framework are not detailed enough. Only Figure 3 briefly describes the core sub-modules of the framework. The main purpose of ADer framework is to integrate the existing VAD algorithms together for convenient and fair comparison and evaluation. However, the process and method of integrating these algorithms are not thoroughly explained, nor are the impacts of integration compared to non-integration detailed or experimentally demonstrated. It is recommended to refer to the comprehensive description of the framework provided in the object detection framework MMDetection [1].
3. As a point mentioned separately, the indicator calculation acceleration algorithm ADEval is not described in detail enough. It is only briefly described in Section 3.6, without any introduction to the iterative-accumulating algorithm it uses. In addition, Section 3.6 only uses an example to describe the improvement of the calculation time of the algorithm, without any supporting experimental data.
4. The primary aim of this paper is to offer a new and fair benchmark for comparing existing visual anomaly detection methods. In the experimental setup, only two parameters—image resolution and training epochs—are fixed, while other parameters remain as defined by the original methods, which does not fully demonstrate ADer’s role and fairness. All experimental results can be compared and verified using the original method's code with the aforementioned parameters standardized.

[1] Kai Chen, Jiaqi Wang, Jiangmiao Pang, et al. MMDetection: Open MMLab Detection Toolbox and Benchmark

1. Alignment with ICLR Scope: While ADer provides a valuable standardized framework, the paper’s relevance to ICLR could be strengthened. Including a discussion of representation learning techniques in AD, or integrating such techniques into the benchmark, could better align the work with ICLR’s focus. Exploring representation learning might also broaden the utility of ADer by connecting it to core themes in machine learning and anomaly detection research.

2. Justification for Multi-class Focus: The paper currently lacks sufficient justification for focusing solely on the multi-class setting. Addressing the rationale behind this choice would improve the discussion. Additionally, several challenges for multi-class AD mentioned in Section 2.2 remain underexplored within ADer. Greater emphasis on how ADer tackles these issues could clarify its contributions to multi-class AD.

3. Paper Structure: Section 3.5 could be more effectively placed within Section 2 to better showcase ADer’s strengths over prior works. This repositioning would improve the flow and help readers appreciate ADer’s advantages earlier in the paper.