MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images

Q5: Could your approach handle a setting when a test dataset lacks ground truth? What is performance when a test dataset lacks ground truth for optimizing the proposed heuristic procedure?

A5: Thanks for your comment. As we mentioned in the response to Q2 of reviewer dCfU, our approach contains three hyperparameters. Our performance is sensitive to the aggregation degree $r$ but insensitive to the other two hyperparameters. In addition, the experimental results on the new BTAD dataset demonstrate that the fixed hyperparameter setting could achieve good experimental results in different datasets. Therefore, we do not need to use the ground truth to optimize our approach. It is clarified that all the steps of our approach do not use the ground truth actually.

Q6: One of the claimed contributions, the mutual scoring mechanism, appears to depend significantly on the relative positions of feature patches. This limits the application of this approach in real settings where orientations and scales are not consistent. What is performance when test images exhibit inconsistent orientations or scales?

A6: Thanks for your comment! In the mutual scoring mechanism, we leverage each image in {$D_{test}$ \ $I_i$} to assign an anomaly score for each aggregated patch token of a test image $I_i$ in stage one as follows，

where $m$ and $n$ are the index of the image patches, $r$ is the aggregation degree, $i$ and $j$ are the index of the unlabeled images, and $l$ indicates the stage $l$ of ViT. We clarify that $m$ is irrelevant to the position of $n$, since Eq.1 first computes the distance between the $m$-th patch (in image $I_i$) and all the patches in image $I_j$, and then performs min operation to select the most similar image patch from image $I_j$. Accordingly, the anomaly score only depends on the aggregated patch token and is irrelevant to the relative positions of feature patches.

In addition, according to your insightful comment, we select the categories with inconsistent orientations or scales from MVTec AD and VisA datasets for the additional experimental evaluation, which is given in Tab. 12 in the paper and also given in the following table. We compare our method with WinCLIP and APRIL-GAN and show the image-AUROC (AC)/pixel-AUROC (AS) in this table. We observe that our approach and WinCLIP are both influenced by inconsistent orientations or scales. The reason is that both of us use the fixed pre-trained vision transformer as the feature extractor. As introduced in [c], during the pre-training process, the vision transformer only performs minor data augmentation on orientations and scales, and hence both of us cannot well address the inconsistent orientations or scales. APRIL-GAN also achieves a decreased AC score, while it obtains a better AS score. We guess that it is due to its additional training set for optimizing AS. In addition, we should emphasize that our approach still outperforms WinCLIP and APRIL-GAN in inconsistent orientations or scale cases.

In our future works, we plan to design rotation and scale invariance features to alleviate such a limitation.

Thanks for your insightful comments and thoughtful questions.

Q1: This paper makes the assumption that access to the entire test dataset is available. For zero-shot settings, such an assumption is too restrictive. Access to test data is mostly very limited in real-world scenarios, especially for zero-shot.

A1: Thanks for your comment! As mentioned by Reviewer H5pV, our approach aligns closely with ACR [a][b], an excellent zero-shot AD approach published in NeurIPS2023, both of us jointly measure the anomaly scores across either the entire dataset or a sub-set of unlabeled test images. Therefore, we hope the comparison with ACR makes our approach more fair.

Q2: The method heavily relies on many heuristic design choices. It consists of a three-step pipeline which depends on many hyperparameters. The paper poorly presents the sensitivity of performance to optimizing all these parameters.

A2: Our method contains three steps, and each step has only one hyperparameter, i.e., the aggregation degree $r$ in LNAMD, Interval Average (IA) on the minimum $X\%$, and window mask size in RsCIN.

The setting of aggregation degree $r$ indeed impacts the result as it is related to the size of anomalies. As shown in Tab. 3, in multiple settings, the model's classification AUROC on the MVTec AD dataset is reduced by up to 3.6%, and segmentation AUROC is reduced by up to 3%. On the VisA dataset, the use of $r$={5} results in a 10.4% reduction in classification results compared to $r$={1,3,5}, due to more small defects on VisA, making it unsuitable to use only the large aggregation degree. For better practicality, we use the same setting {1,3,5} in all the datasets to obtain a good segmentation of anomalies with various sizes.

The $X\%$ in Interval Average is an insensitive hyperparameter. As shown in Fig. 7, we compare the models using different percentage ranges of the Interval Average in the MSM module. The change of this parameter has a maximum effect of 3% on the image-AUROC and 1% on the pixel-AUROC.

The window mask size in RsCIN is also an insensitive hyperparameter, which is proved by the experiments in Fig. 8. Observably, on the MVTec AD data set, F1-max-score fluctuates between 97.1 and 97.5, with the amplitude not exceeding 0.4. On the VisA data set, F1-max-score floats between 87.8 and 89.5, and the amplitude does not exceed 1.7. Furthermore, we also verify the sensitivity of the RsCIN module to different methods, as shown in Tab. 11 of the appendix. All methods maintain the same parameter settings of the RsCIN module as ours. Almost all methods achieve performance improvements after applying the RsCIN module, except DRAEM only. This experiment proves that the window mask size in RsCIN is insensitive for different methods.

In summary, the aggregation degree $r$ is a sensitive parameter, while the other two hyperparameters are insensitive.

In addition, we show the performance of our method on the BTAD dataset, and the parameter settings are consistent with those in our experiments. Specific settings include aggregation degree $r$ as {1, 3, 5},  $X$ as 30, and window mask size as {2, 3}. The following table shows that the classification and segmentation performance of our method on the BTAD dataset still exceeds that of zero/few-shot methods and even some full-shot methods. Such an experimental result shows that although the performance of MuSc is sensitive to the setting of aggregation degree $r$, an appropriate choice of $r$ can achieve good results in different datasets simultaneously, which proves its universality.

Thanks for the insightful comments.

Q1: This paper employs a methodology that utilizes unlabeled test images to collectively measure anomaly scores, differing somewhat from the traditional zero-shot recognition setting. This difference in evaluation methodologies can result in a somewhat unfair comparison.

A1: We agree that our approach is different from several existing approaches, e.g., winCLIP, in using the unlabeled test images, which may lead to a potentially unfair comparison. Therefore we have added the comparison to ACR [b][c] in Tab.1. Similar to our approach, ACR assumes access to a set of unlabeled test images together during inference, the experimental results demonstrate that MuSc outperforms ACR by a large margin. We hope such a comparison is more fair.

Q2: The underlying concept is reminiscent of [a], which presumes homogeneous input texture and identifies image regions that disrupt this homogeneity as anomalies.

A2: Thanks for your valuable suggestion! We have carefully studied [a] and added the discussion in the Related Work section. [a] is based on the assumption that the input texture is homogeneous, and the image regions that break the homogeneity are detected as anomalies. [a] explores the relationship between the patches inside one test texture image, while our method leverages the relationship in a set of unlabeled images.

Q3: The approach of jointly measuring anomaly scores across entire datasets aligns closely with [b]. While not mandatory, it would be beneficial for the authors to discuss or draw comparisons with [b].

A3: Thanks for your valuable suggestion! We have carefully studied ACR [b][c] and added the comparison with it in Tab. 1 and the discussion in the Related Works section. We found that ACR is an excellent zero-shot AD approach, we agree that our approach aligns closely with ACR, since both of us jointly measure the anomaly scores across the entire dataset or a set of unlabeled test images, while we use the unlabeled images differently, another difference is that ACR contains a training process, while our approach does not need any training or prompt.

We give the classification and segmentation performance of our method and [b][c] on the MVTec AD dataset in the following table.

[a] Aota et al. "Zero-shot versus Many-shot: Unsupervised Texture Anomaly Detection." In WACV 2023.

[b] Li et al. "Zero-Shot Batch-Level Anomaly Detection." arXiv, February 2023.

[c] Li et al. "Zero-Shot Anomaly Detection via Batch Normalization." In NeurIPS 2023.

Thanks for your valuable comments.

Q1: It's unclear why the combination of ({3, 5}) performs worse in the anomaly classification (AC) task than using ({3}) alone.

A1: Thanks for your comment. We have explained this phenomenon deeply in the revised Appendix A.2.2. As we written in Section 3.1, aggregation degree ({5}) is suitable for detecting large defects, but it in turn smoothes small defects to a lower score. We show a classic example in Fig. 9 of the revised manuscript. When directly summing the anomaly scores of the two aggregation degrees ({3,5}), the score within the small anomaly region is pulled down by the low score generated by the aggregation degree ({5}), which results in false negative. Consequently, the combination of ({3,5}) performs worse in the anomaly classification (AC) task than using ({3}) alone.

Due to the above reasons, we use the aggregation degrees ({1,3,5}) to take into account various-size anomalies in real industrial scenarios. As displayed in Fig 9, when we add a large anomaly score with aggregation degree ({1}), the anomaly score within the black rectangle in (e) increases and false negatives reduce.

Thanks for your comments, we want to first give the following clarifications.

“It is assumed that a set of images with some form of anomaly is available for inference”.

Here, it needs to be clarified that our approach leverages a set of unlabeled images for anomaly AC/AS but does not assume that the anomaly has the same form.

Q1: The method relies heavily on availability of a set of images in order to compute the anomaly scores. I dont think this emulates the practical scenario of anomaly detection in industry.

A1: Thanks for your comment. Since we have long-term collaboration with several well-known companies, our approach is designed according to the practical industry requirements. Let us take a famous anonymous company as an example. In general, a production line (AMOLED/LCD) has a daily output of 1500 pieces. Typically, a production line is equipped with at least 3 parallel quality inspection equipment, and each furnished with at least 2 cameras. The image resolution of each camera exceeds 4000×5000, and a typical solution is to crop a series of 512×512 images from high-resolution images by sliding windows with 10% overlap. The above production line needs to inspect nearly a million product images per day. As far as we know, such a production quality inspection requirement exists in a large number of companies. This example illustrates that unlabeled images are abundant in real industrial scenarios and easy to collect.

In addition, it needs to be clarified that MuSc did not rely heavily on a large number of unlabeled images, which can be demonstrated by Tab. 7 of the paper. MuSc obtains decent AUROCs under the test sets of different amounts, even when the test set only has 14-56 images. Referring to the example above, we are confident that such a number of images can be easily achieved in practical scenarios.

Q2：The more realistic scenario is a method is required to classify and segment the anomaly given one image. Using test set images to produce the output on the same set of images also does not conform with the scientific procedure. I dont think this is the definition of a zero shot method either.

A2: Thanks for your comment. We agree that it is a widely-used practice to classify and segment anomalies in one image. However, this does not negate the scientific validity of using a set of test images to produce the output on the same set of images. As mentioned by Reviewer H5pV, our approach aligns closely with ACR [b][c], an excellent zero-shot AD approach published in NeurIPS2023. Both of us jointly measure the anomaly scores across either the entire test dataset or a sub-set of unlabeled test images. Therefore, we hope the comparison with ACR makes our approach more fair and dispels your doubts.

Q3: The algorithm appears to be a set of heuristic tricks applied in a sequence. There is not a solid technique that is novel, elegant, theoretically justified and of broad interest.

A3: Thanks for your comment! We clarify our technique novel as follows.

Firstly, we find that the patch-level features extracted by ViT have the limitation of detecting industrial anomalies of varying sizes. Swin transformer has large patches in some stages leading to the patch-level features extracted by it not working well. The proposed LNAMD module addresses this issue in ViT, as far as we know, such a design is a new technique to optimize the patch-level features of ViT in industrial vision.

Secondly, in MSM, the mutual scoring operation in Eq. (1) is inspired by the observation of normal and abnormal patches. It uses the minimum similarity between the patch $m$ of image $I_i$ and all the patches of image $I_j$ to score the patch $m$ of image $I_i$. This is a new technique to score the patch by unlabeled images. The interval average operation in Eq. (2) is based on our observation of the false positives, so we designed this operation to reduce the false positives. To the best of our knowledge, the Mutual Scoring Mechanism is a new mechanism for the zero-shot AC/AS community.

Thirdly, in RsCIN, we observe that due to the high smoothness of $**C**$, the traditional manifold learning approach may be influenced by a large number of images. Therefore, we propose the Multi-window Mask Operation (MMO) to constrain the image number, which makes each image only influenced by a small number of neighborhood images. RsCIN was first proposed in the field of anomaly classification and segmentation. Meanwhile, we experimentally proved that RsCIN can be further employed to improve the AC accuracy of existing approaches in Appendix A.2.4.

Thanks for your insightful comments.

Q1: Even without training or prompts, the method requires long inference times and high memory costs. While the authors provide a solution to increase speed and reduce memory by dividing the test set into subsets, this also reduces performance by a small margin. Are there any other approaches that could have been considered to solve the problem?

A1: Driven by your suggestion, we have examed the MuSc's code implementation thoroughly and found two issues that slowed down the model and increased the number of parameters. For now, both issues have been solved. After the paper is accepted, we will release the source code to prove that both issues are indeed addressed and the parameter amount and speed are optimized. Below, we describe these two issues and the corresponding countermeasures respectively.

First issue: A time-consuming operation in LNAMD. In the original implementation of the LNAMD, we have used the "for" loop operation to process a tensor with dimensions [1369, 1024], i.e., "features", resulting in a lot of time-consuming.

return [x.detach().cpu().numpy() for x in features]

Consequently, this inefficient implementation costs 95.2ms under a specific aggregation degree r, consuming more than 70% of the time in LNAMD which takes 129.9ms only. We have addressed this issue by the following code:

return features.detach().cpu().numpy()

Second issue: Additional GPU memory usage in LNAMD. The original implementation of the LNAMD module unexpectedly loaded the backbone network an extra time, which was neglected when we modified the code structure. For now, we have removed the extra process of loading the backbone, saving 1186MB and 620MB of memory respectively when using ViT-L-14-336 and ViT-B-16-plus-240.

After modifying the code of LNAMD, we have conducted a comprehensive efficiency comparison, which is listed in the following table. Since there is no official source code of WinCLIP, its GPU cost is unknown (marked by “-”). In addition, the author of APRIL-GAN does not provide the pre-trained model using ViT-B-16-plus-240 (our backbone), so its AUROC and AUPRO are unknown (marked by “-”).

In terms of inference time, when using the same backbone as WinCLIP and setting $s$ to 3, our inference time is 238.3ms per image, which is 150.7ms shorter than WinCLIP, and 71.1ms shorter than RegAD. Additionally, APRIL-GAN has the shortest inference time (64.7ms) among these methods. In terms of memory cost, when using the same backbone as WinCLIP, our GPU memory cost is 3002MB, significantly lower than RegAD (8920MB) and 224MB less than APRIL-GAN (3226MB). The detailed comparisons of inference time and memory cost have been added to Appendix A.4.