MetaUAS: Universal Anomaly Segmentation with One-Prompt Meta-Learning

• bbUz-Q1: What are the differences between MetaUAS, MetaUAS*, and MetaUAS*+?

We apologize for the confusion about MetaUAS, MetaUAS*, and MetaUAS*+. In fact, we have given some details of our MetaUAS and its two variants (MetaUAS* and MetaUAS*+) in the Sec. B of Supplementary Materials. Here, we provide some explanations, which might address your concerns.

The MetaUAS is capable of segmenting any anomalies only given just one normal image prompt. Note that the normal image prompt is randomly selected for each testing class in MetaUAS. Different from MetaUAS, the MetaUAS* searches the best-matched normal image as a prompt from the normal training set for each query image. The MetaUAS*+ builds on MetaUAS*, but introduces the visual prior from a CLIP model. The visual prior knowledge is obtained by computing the cosine similarity between the query feature and the corresponding prompt feature extracted from the vision encoder of the CLIP model, which is the same as WinCLIP+ and PromptAD. In addition, the MetaAS is a typo and the correct one should be MetaUAS. We will add these clarifications to the methodology section.

• bbUz-Q2: Why UniAD is excluded from the comparison in Table 1?

UniAD is a powerful unsupervised anomaly detection method, which trains a unified reconstruction model to detect multi-class anomalies using full-shot normal images from the target domain. In contrast to our training-free one-shot method, UniAD requires re-training the model using full-shot normal images. As reported in Table 1, we observe that our one-shot training-free method still exhibits strong performance compared to the full-shot UniAD. We will add a description for UniAD in the caption of Tabel 1.

• bbUz-Q3: Could all change segmentation methods be applied to anomaly segmentation ``off-the-shelf''?

It is challenging to apply existing change segmentation models to anomaly segmentation. Current change segmentation models are primarily focused on two scenarios: remote sensing and street scenes. The main characteristics of these scenarios and models can be summarized in four aspects.

First, the scale of training images in these scenarios is usually small, typically hundreds or thousands of images. Second, the diversity of images is limited because they mainly consist of remote sensing images or street scene images. Third, these models primarily focus on semantic changes (such as buildings) while the background may undergo other changes due to factors like seasons and weather. Fourth, the prompt and query images are generally coarsely aligned in the change segmentation, which is different from anomaly detection scenarios where there are often large geometric variations.

These characteristics limit the development of a general change detection model. Additionally, we evaluated the off-the-shelf ChangerEx with s50 backbone on MVTec, and the corresponding results are reported in the following table. We can see that its generalization performance is poor compared to our method.

• sm3a-Q1: The effects of the number of training data for the anomaly segmentation performance.

Thanks for your interest in the effects of training scale. We are also very concerned about this issue, which has been analyzed in our ablation studies (Sec. 4.3). The corresponding experimental results are reported in Table 3(d), with the related analysis provided in Lines 361-366. The number of synthesized images and the split of training and validation are described in Sec. A of Supplementary Materials. Here, we provide further explanations.

To investigate the influence of training scale on model performance, we conduct experiments with different training subsets where each one is generated by randomly sampling the original training set at various rates, such as {10%, 30%, 50%, 70%, 95%}. In Table 3(d), it can be seen that MetaUAS still works when the number of training images is small scale (e.g., 50%), and the performance can further improve when increasing the number of training samples. Unless otherwise specified, the default number of training images is 95% of the synthesized dataset.

• sm3a-Q2: Compared to other existing approaches, could one afford a larger prompt pool or multiple forward passes of MetaUAS?

We agree with you that using more normal image prompts can potentially enhance the robustness of the model. Additionally, we have observed that the performance gains tend to saturate when the number of normal image prompts increases to a certain level (e.g., 5) (see the response to gb5F-Q3). Assuming that each query image is provided with 5 normal image prompts, this means that the inference cost could increase by up to 5 times. However, considering that we use a lightweight model, even when 32 prompt-query pairs are processed in parallel during forward inference, the required time of each pair is only 3.2ms, as reported in Table 2. Therefore, our method can efficiently handle larger prompt pools compared to existing CLIP-based models.

• sm3a-Q3: Could MetaUAS also be applied to such tasks where the normal ``object'' itself (e.g., the road, SMIYC) could have varying visual appearances as well?

Our method is primarily towards open-world industrial anomaly detection, which has become an attractive yet challenging topic. It may not be suitable to directly apply our method in these scenarios where the normal ``object'' itself or its context exhibits large variations in appearance. This is because the core of our method relies on detecting changes to identify anomalies.

Furthermore, we have carefully reviewed the suggested RoadAnomaly21, which contains only 100 testing images and does not provide normal images. In this dataset, the context changes of the road are more significant than that of the road itself. We believe that one feasible manner could be to first preprocess these images to remove backgrounds (i.e., the context of the road) and then apply our method to detect anomalies on the road surface.

• sm3a-Q4: All prompts must be processed by the encoder to extract their features, which could become expensive as the number of classes increases.

We apologize for any confusion this issue may have caused. Here, we provide some clarification that might address your concerns. For a class-specific query image, we can process prompt-query pairs online to perform anomaly segmentation without extracting offline features for normal image prompts. For a class-agnostic query image, we need to first extract offline features from a prompt pool, and then match the corresponding image prompt for the given query image to perform anomaly segmentation. In fact, in real-world industrial production lines, products are generally produced on a large scale according to specific models, so we could register normal image prompts according to actual needs. In addition, existing language-prompt-based anomaly detection methods (e.g., WinCLIP) also face similar issues, as they require offline computation of normal or anomaly textual features for all objects or textures.

• sm3a-Q5: It is unclear whether a single example can represent all normal examples for a class effectively. Searching for the best matching normal image for each query image is resource-intensive.

As shown in Table 1, our method requires only one normal image prompt to achieve competitive performance when faced with an open-world anomaly detection scenario. This indirectly supports that one normal image can essentially represent the normal pattern for a specific category. Undoubtedly, increasing the number of normal image prompts can further enhance performance, but this involves additional computational costs.

In our MetaUAS*, we have demonstrated that the optimal normal prompt yields better performance compared to a random image prompt, although the matching process does indeed bring computational cost. Exploring other efficient algorithms to obtain a better prompt is also worthwhile, such as using image-matching to pre-align the prompt image with the query image.

• sm3a-Q6: There are a few typos throughout the text, and the figures and tables could be placed differently.

Thank you very much for pointing out the few typos. We will carefully review our manuscript and correct these typos in the revised version.

• DDkp-Q1: How these features are used in the decoder needs to be explained in more detail.

Our method implements the decoder using standard UNet. Furthermore, we also compare the UNet with the FPN decoder in Table 3(c). In our paper, we omit the details of UNet and FPN because they are two popular modules that have been standardized and integrated into the PyTorch library. As we know, the decoder of UNet and FPN is widely used in dense visual prediction tasks, such as semantic segmentation and object detection. They use a top-down structure with lateral connections to fuse multi-scale input features and output a single high-resolution feature map. Considering reproducibility, we will make the codes and models of MetaUAS available.

• DDkp-Q2: It is beneficial to discuss and compare with CLIPSeg.

Thank you for pointing out a related work, CLIPSeg [1]. The CLIPSeg has one-shot semantic segmentation capabilities by providing one support image to the pre-trained CLIP model. However, its better performance still struggles with the powerful CLIP model and textual prompts. Different from the CLIPSeg, our method is compatible with various pre-trained models and does not require any guidance from language, as pointed out by you and Reviewer sm3a.

[1] Image Segmentation Using Text and Image Prompts, CVPR 2022.

• gb5F-Q1: A Transformer backbone in the ablation in Table 3b would be nice.

Thanks for your suggestion. As pointed out by DDkp and sm3a, our method is compatible with various pre-trained models, including Convolutional and Transformer architecture networks. Considering the efficiency, we employ EfficientNet-b4 (E-b4) as our encoder, which is the same as in the previous UniAD.

Following your suggestion, we replace the convolution-based EfficientNet-b4 with the recent EfficientViT [1]. Specifically, three EfficientViTs with different capacities (b1, b2 and b3) are used as our encoders, and the corresponding anomaly detection results on MVTec are reported in the following table. We can see that their performance is still lower than that of EfficientNet-b4 in most metrics. Other more powerful and efficient backbones deserve further exploration. In the revised version, we will add these results.

• gb5F-Q2: How the CLIP's visual prior is introduced into the framework.

We apologize for the confusion about MetaUAS*+. In fact, we have given more details of diverse state-of-the-art methods, our MetaUAS and its variants (MetaUAS* and MetaUAS*+) in the Sec. B of Supplementary Materials. Here, we give some explanations about MetaUAS, MetaUAS* and MetaUAS*+, and hope to address your concerns.

The MetaUAS is capable of segmenting any anomalies only just by giving one normal image prompt. Note that the normal image prompt is randomly selected from a specific normal training set in MetaUAS. Different from MetaUAS, the MetaUAS* searches the best-matched normal image as a vision prompt from the normal training set for each query image. The MetaUAS*+ builds on MetaUAS*, but introduces the visual prior from a CLIP model. The visual prior knowledge is obtained by computing the cosine similarity between the query feature and the corresponding prompt feature extracted from the CLIP vision encoder, which is the same as WinCLIP+ [2] and PromptAD [3].

• gb5F-Q3: Whether the evaluation can be extended from one-shot to few-shot?

Our method can be flexibly extended to a few-shot setting, where a few normal image prompts are provided. However, we seek to push the limits of simple but effective methods for more general and challenging settings of anomaly segmentation. To this end, we explore universal anomaly segmentation from the perspective of a framework paradigm, where synthesized images are considered for one-prompt meta-learning, geometrical variations between prompt and query images can be effectively handled, and the inference is more efficient and simple without any guidance from language or fine-tuning on target datasets.

Here, we present a straightforward manner to extend our approach from one-shot to few-shot. We use an average of all shots’ predictions as the final anomaly segmentation map. In the following table, we can observe a significant performance improvement from 1-shot to 3-shot. However, the performance tends to stabilize when reaching 5-shot. We leave it to future work on how to more elegantly extend MetaUAS from one-shot to few-shot.

• gb5F-Q4: How prompt pooling is performed?

We obtain a feature representation by global average pooling on the highest-level feature from the encoder. For a class-agnostic query image, we match the corresponding normal image prompt using the cosine similarity between the query feature representation and all offline prompt feature representations. The above details have been given in Lines 265-270.

• gb5F-Q6: A further discussion of MetaUAS*+.

Thanks for your suggestion. The MetaUAS*+ can be seen as an ensemble of MetaUAS* and the visual prior knowledge of CLIP. Therefore, the performance improvement is intuitive from an ensemble learning perspective. We will add the discussion to the revised version.

[1] EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction, ICCV 2023.

[2] WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation, CVPR 2023.

[3] PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection, CVPR 2024.