AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection

We thank the reviewer for the positive review and encouraging comments.

Q1: Little/no discussion about the additional cost of training

We have supplemented the analysis about the additional cost of training by measuring the time taken during training (training time per epoch) in Table 8 of Appendix D. For a fair comparison, all experiments are conducted in a single NVIDIA RTX 3090 24GB GPU, and the GPU is kept free before evaluation. In Table 8, AnomalyCLIP takes 13.71 min per epoch on MVTec AD (The total number of samples is 1725) and only requires a total of 15 epochs for the whole fine-tuning. Once AnomalyCLIP finishes fine-tuning, AnomalyCLIP can be applied to different datasets and domains without additional training. We also compare AnomalyCLIP with other baselines that need auxiliary data (i.e., CoOp and VAND). For example, the minimum training time per epoch is 12.25 min of CoOp, and hence, AnomalyCLIP has a similar training computation cost as CoOp.

Q2: No comparison of different ViT backbones of CLIP

We have added the backbone ablation study in Table 13 in Appendix D. We originally used ViT-L-14 as the backbones of CLIP, which is a representative backbone in CLIP. To fully explore the effect of different sizes of backbones, we use ViT-B-16-plus and ViT-H-14, which is a smaller and larger model than ViT-L-14, respectively. Compared to pixel-level performance 90.5% AUROC and 90.8% AUROC from ViT-B-16-plus and ViT-H-14, ViT-L-14 achieves the best performance 91.1% AUROC. The results indicate that AnomalyCLIP maintains promising performance across various ViT backbones. Meanwhile, it also reveals that increasing the model size does not necessarily result in improved performance. The larger model may introduce a strong bias towards object semantics, hindering AnomalyCLIP's ability to learn object-agnostic text prompts capturing generic normality and abnormality in an image. However, a smaller backbone like ViT-B-16-plus could cause AnomalyCLIP to struggle with generalization to unseen anomalies.

Q3: Figure 6 feels confusing and unnecessary

Since CLIP is pre-trained to align the textual semantics and the corresponding object, it harms the representation of local visual semantics which is important to anomaly localization. Therefore, AnomalyCLIP proposes DPAM to improve the local visual semantics. The DPAM strategy has three variants, i.e., $Q$-$Q$, $K$-$K$, and $V$-$V$ attention. To investigate the effect of different variants, we present their respective ZSAD performance in Figure 6. We agree that these results are not as important as the other results in justifying the effectiveness of AnomalyCLIP; they are presented to provide an insight into the effectiveness of the attention mechanism in CLIP on ZSAD since CLIP has emerged to be one key tool to facilitate this task.

Q4: It would be better to provide more intuition why the Focal and Dice loss are necessary

We thank the reviewer for pointing this out. Although local features are refined by DPAM, there still exists misalignment among textual semantic and local visual semantics, especially for unseen objects. Focal and Dice loss play a crucial role in optimizing local context. They are introduced to empower object-agnostic text prompts to focus on fine-grained, local abnormal regions from intermediate layers of the visual encoder. As mentioned in Section 3.2, Focal loss addresses the imbalance between anomaly and normal pixels, typically caused by the smaller size of anomalous regions. Meanwhile, Dice loss aims to precisely constrain the anomaly boundary by measuring the overlap between the predicted segmentation ($S_n$/$S_a$) and the ground truth mask. To provide a more comprehensive analysis, we have included an ablation study on Focal and Dice loss in Table 14 of Appendix D. Compared to scenarios without local context optimization, Dice loss improves the pixel-level and image-level performance from 80.3%AUROC to 87.2%AUROC and 86.6%AUROC to 90.1%AUROC in pixel level on MVTec AD and VisA. Focal loss also brings the performance gain of 10.3%AUROC and 8.3%AUROC. Combining Focal and Dice loss, AnomalyCLIP achieves the best results (i.e., 91.1%AUROC and 95.5%AUROC). Note that the global context optimization is always used during the ablation, since we need at least one loss function to drive the optimization.

Q5: Figure 2 caption should be more explanatory

Thank you for your effort on helping improve the paper. We have added more illustrations in the revised version: "To adapt CLIP to ZSAD, AnomalyCLIP introduces object-agnostic text prompt templates to capture generic normality and abnormality regardless of the object semantics. Then, we introduce glocal context optimization to incorporate global and fine-grained anomaly semantics into object-agnostic textual embedding learning. Finally, textual prompt tuning and DPAM are used to support the learning in the textual and local visual spaces of CLIP."

Thank you for your review and insightful comments.

Q1: The motivation behind DPAM appears to be relatively weak. It is not entirely clear why employing V-V self-attention would preserve more localized information compared to the original Q-K attention.

Thank you for raising this concern. Please refer to our General Response above for an intuitive explanation of DPAM strategy from the attention mechanism.

Q2: Relying solely on a single instance shown in Figure 3 may not furnish sufficient evidence to support this assertion.

We have added experiments to study the effect of DPAM in Table 7 of Appendix D. Without our proposed DPAM module, AnomalyCLIP shows a decrease from 91.1% AUROC to 87.9% AUROC in pixel-level performance and from 91.5% AUROC to 80.7% AUROC in image-level performance on MVTec AD. Additionally, there is a decrease from 95.5% AUROC to 91.9% AUROC in pixel-level performance and from 82.1% AUROC to 73.0% AUROC in image-level performance on VisA. The results demonstrate the superiority of $V$-$V$ attention over $Q$-$K$ attention. However, the performance decreases at the image level is more pronounced than that at the pixel level. This discrepancy is attributed to the fact that the total loss places greater emphasis on local context optimization, driven by a larger local loss compared to the case with DPAM. As a result, AnomalyCLIP lacks sufficient optimization for global alignment, leading to a significant decline in image-level performance. Note that the original CLIP (no DPAM) exploits $Q$-$K$ features and suffers from weak segmentation performance.

Q3: What might be the outcome if we were to combine Q-Q, K-K, V-V, and Q-K features into an ensemble?

Thank you for your helpful comments. Ensembling different features is an excellent idea. We have added two experiments to explore two ensemble methods, namely AnomalyCLIP$\_{ensemble1}$ and AnomalyCLIP$\_{ensemble2}$, involving the ensemble of Q-Q, K-K, and V-V and the ensemble of Q-K, Q-Q, K-K, and V-V, respectively in Table 12 in Appendix D. We average the logit output of different features for the ensemble. In Table 12, AnomalyCLIP$\_{ensemble1}$ shows performance improvement by leveraging the advantages of three DPAM features. However, while AnomalyCLIP$\_{ensemble2}$ outperforms the $Q$-$K$ feature version, it experiences a performance decrease compared to $V$-$V$ from 91.1%AUROC to 90.7%AUROC in pixel level and 91.5%AUROC to 90.8%AUROC in image level on MVTec AD. There is also a decline from 95.5%AUROC to 94.9%AUROC in pixel level and 82.1%AUROC to 80.7%AUROC in image level on VisA. The decline in performance upon adding $Q$-$K$ features to AnomalyCLIP$_{ensemble1}$ suggests that the $Q$-$K$ feature fails to provide valid local visual semantics to facilitate ZSAD. Note that the original CLIP exploits $Q$-$K$ features and gets weak segmentation performance. The good pixel-level performance of $Q$-$K$ in AnomalyCLIP is attributed to local optimization, where the object-agnostic prompt helps alleviate the disrupted local visual semantics of $Q$-$K$.

Thank you for your positive review and the insightful comments.

Q1: The model may misclassify a product with texture patterns similar to scratches as an abnormal one, even though the scratch-like textures are normal texture patterns of the product. It would be better to explicitly discuss the successful cases and the potential failures of the proposed method.

Thank you for raising this concern. We agree that in a minority of cases, AnomalyCLIP exhibits false detection due to the object-agnostic assumption. However, ZSAD requires detection models to detect anomalies without any training sample in a target dataset, and capturing the object-agnostic anomaly patterns is a promising solution because anomalies from different application scenarios typically have substantial variations in their visual appearance, foreground objects, and background features. As suggested by the reviewer, we have supplemented the experiment to explicitly discuss the successful cases and the potential failures in Figure 7 in Appendix D. In Figure 7(a), AnomalyCLIP accurately detects scratch-like patterns on the product, even when they typically appear in the texture. However, false detection occurs when scratch-like patterns are situated in the background, as depicted in Figure 7(b). Meanwhile, we also show the color stain patterns. As shown in Figure 7(c), AnomalyCLIP successfully detects the color stain, which exhibits subtle visual differences from the detected entities. On the other hand, AnomalyCLIP may face challenges when the normal region displays patterns that are almost indistinguishable to the naked eye from anomalies. For instance, in skin cancer detection, the normal regions are falsely detected as anomalies are visually similar to the disease region in Figure 7(d). Also, the stain interference in the background is a problem. These failure cases illustrate that the importance of mitigating background interference and achieving fine-grained discrimination, especially in cases of visually similar abnormalities. Exploring these challenges for enhancing ZSAD is a valuable direction for future research.

Q2: Minor comment: $S_n$ and $S_a$ are first-time used in $L_{local}$ on page 5, but are clearly defined on page 6. Having the definitions and equations of $S_n$ and $S_a$ before $L_{local}$ would be good.

Thank you for your effort on helping improve our paper. We have updated the definitions of $S_n$ and $S_a$ before $L_{local}$.

Q3: The discussion of some related work about zero-shot anomaly detection, e.g., $1,2,3$ , is missing.

We thank the reviewer for pointing us towards these related works. We have updated our related work section and added the works suggested by the reviewer. CLIP-AD $1$ and ZOC $2$ are early works in the utilization of CLIP for anomaly/out-of-distribution detection task, and they primarily focus on anomaly classification tasks, whereas AnomalyCLIP jointly optimizes anomaly detection and segmentation tasks, enabling both tasks within a single forward pass during inference. Despite ACR $3$ addressing detection and segmentation tasks, it requires distinct auxiliary data for specific zero-shot detection tasks. AnomalyCLIP can adapt to various datasets and domains for zero-shot detection by learning object-agnostic prompts only once.

References:

1. Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Klaus Robert Muller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. Transactions on Machine Learning Research, 2022.
2. Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pretrained model clip. In Proceedings of the AAAI conference on artificial intelligence, 2022
3. Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, Stephan Mandt. Zero-Shot Batch-Level Anomaly Detection. Thirty-seventh Conference on Neural Information Processing Systems, 2023.

Q4: Why is learning two text prompts necessary?

We have supplemented experiments to investigate the effect of shared prompts in Table 11 in Appendix D. When sharing the learnable word embeddings of $g_n$ and $g_a$, AnomalyCLIP achieves 90.5% AUROC in pixel level and 90.9% in image-level on MVTec AD and 95.0% AUROC in pixel level and 81.5% AUROC in image level on VisA. The results show that AnomalyCLIP without sharing also works well for ZSAD and the efficiency of our object-agnostic prompt learning. However, the shared prompt performs slightly worse than the unshared prompts (used in the original paper). The performance decrease is 0.6%AUROC and 0.6%AUROC in image level on MVTec AD and Visa, and 0.5%AUROC and 0.6%AUROC in pixel level. We believe that the separate learning for these two prompts helps discriminate the generic normality and abnormality because when we share the parameters of $V_i$ and $W_i$, the learned semantics of normal and anomaly may be confused.

Thank you for your review and the insightful comments.

Q1: The idea and some techniques, like glocal context optimization, are similar to a concurrent work $1$ .

Thank you for pointing us toward this concurrent work. AnomalyGPT $1$ is a great work for anomaly detection. Below we summarize the difference between AnomalyGPT and AnomalyCLIP as follows:

1. The scope is different. AnomalyGPT eliminates the need for manual threshold adjustments via LLM when detecting the seen objects during training. However, AnomalyCLIP aims to improve the model generalization to unseen objects across diverse datasets and domains.
2. We focus on different settings. AnomalyGPT focuses on unsupervised anomaly detection and few-shot anomaly detection. However, our work aims at zero-shot anomaly detection.
3. Although both AnomalyGPT and AnomalyCLIP propose to combine local and global loss, the purpose is different. AnomalyGPT uses global loss to quantify the disparity between the text sequence generated by the model and the target text sequence, ensuring LLM produces corresponding textual descriptions. However, our proposed global context optimization serves to align the global visual semantics for anomaly classification.
4. The supervision information is different. Besides the images and text prompts, AnomalyGPT needs human-crafted text descriptions for different objects to query the LLM during fine-tuning. AnomalyCLIP just uses the images and text prompts.
5. We introduce object-agnostic prompt learning to capture the generic normality and abnormality. AnomalyGPT uses human-crafted text prompts to produce textual embeddings.
6. We propose DPAM to persevere local visual semantics of the visual encoder without additional training. AnomalyGPT introduces a lightweight image decoder to learn the adaption of local features.

In summary, since these two works have different scopes and focus on different aspects of anomaly detection, and the required supervision information is also different. We can not provide a comparison of the quantitative results between AnomalyGPT and AnomalyCLIP. Instead, we include and discuss AnomalyGPT in Related Work in the revised manuscript, which is marked in orange.

Reference:

1. Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anomalygpt: Detecting industrial anomalies using large vision-language models, 2023.

Q2: The DPAM strategy is confusing. These three variants are very similar in the DPAM mechanism. It is better to explain it in the paper and show the results of the original Q-K attention.

Thank you for raising this concern. Please refer to General Response for a detailed explanation of the DPAM strategy from the attention mechanism. We have also explained the reason why the $V$-$V$ achieves the best performance among DPAM variants there. We have added the response in Refinement of the local visual space in Section 3.3 in the updated paper, as well as a more detailed analysis in Appendix C.

We thank the reviewer for the positive and encouraging review. When initially using the term 'glocal' in Section Introduction, we have added the explanation in the revised manuscript: "We then introduce a novel ZSAD approach, called AnomalyCLIP, in which we utilize an object-agnostic prompt template and a glocal abnormality loss function (i.e., a combination of global and local loss) to learn the generic abnormality and normality prompts using auxiliary data."

Thank you very much for your review and the insightful comments.

Q1: The prompt template might be too restrictive. "damaged $cls$ " may not well represent all types of anomalies.

We are not sure about whether the reviewer refers to the object-agnostic text prompt "damaged ** $object$ **" (where object is a fixed token like damaged) or the object-aware text prompt "damaged ** $cls$ **" (where cls is a variable token that will be replaced with a specific class name in the algorithm implementation).In our submission, AnomalyCLIP uses the former one. Nevertheless, we posit that the reviewer's main concern is on the limited scope of the token "damaged" may have for anomaly detection.

Since the pattern of anomaly is typically unknown and diverse, it is practically difficult to list all possible anomaly types. Thus, we agree that the prompt "damaged [object]" can be restrictive. To remedy this problem, during the fine-tuning process, we provide pixel-level anomaly ground truth with diverse anomaly types for training our ZDAD. This way helps substantially extend the scope of the prompt template "damaged [object]". It indicates in the linguistic semantic of the token "damaged", incorporating fine-grained anomaly semantics of diverse anomaly types. This intuition is verified by our ablation study for the prompt template in Table 16 and Table 17 of Appendix D, in which AnomalyCLIP shows stable performance across different datasets when replacing "damaged" with other similar tokens: "anomalous", "flawed", "defective", and "blemished".

As for the reason why such a prompt template can be applied to other domains, we believe that the learnable part of the prompt template is optimized to focus on the anomaly regions regardless of its foreground object, which allows our model to capture generic object-agnostic, or even domain-agnostic normality and abnormality. Further, compared to object-aware text prompt templates (i.e., "damaged [cls]"), excluding the object semantics from text prompt templates allows learnable text prompt templates to focus on capturing the characteristics of anomalies themselves, rather than the objects. Additionally, the strong generalization of the visual encoder is also important because it can project images from different domains into the same embedding space. Therefore, AnomalyCLIP can detect medical datasets even when fine-tuned on the industrial defect dataset, as shown in Table 2.

Q2: Explanations on the effectiveness of DPAM and computation overhead are necessary.

We provide a detailed explanation of DPAM and attempt to analyze the effectiveness of the attention mechanism in General Response. We would like to clarify that AnomalyCLIP does not need to feed the images into two visual encoders. In Figure 2, we plot two paths to intuitively provide the computation difference between global and local visual embedding. Without introducing additional parameters, we just create two paths when computing the attention map (DPAM), and the rest process shares the same visual encoder. As for computation overhead, we have supplemented the assessment of the time taken during training (training time per epoch) and the inference speed (frames per second, FPS). For a fair comparison, all experiments are conducted in a single 3090 NVIDIA RTX 3090 24GB GPU, and the GPU is kept free before evaluation. In Table 8, AnomalyCLIP takes 13.71 min per epoch on MVTec AD (The total number of samples is 1725) and only requires a total of 15 epochs for the whole fine-tuning. Once AnomalyCLIP finishes fine-tuning, AnomalyCLIP can be applied to different datasets and domains without additional training. We also compare AnomalyCLIP with other baselines that need auxiliary data (i.e., CoOp and VAND). The minimum training time per epoch is 12.25 min of CoOp, and hence the training time taken is similar for fine-tuning methods. As for inference speed, CLIP achieves the 13.23 FPS. However, it suffers from weak detection performance. Although WinCLIP achieves better performance, WinCLIP has only 1.2 FPS because it needs multiple forward image patches to derive the segmentation. AnomalyCLIP outperforms WinCLIP and obtains an FPS of 8.92.

We also evaluated the computation overhead of DPAM separately. In Table 8, without DPAM, AnomalyCLIP takes 12.98 min to train per epoch. Compared to the 13.71 min for AnomalyCLIP with DPAM, we observe that introducing DPAM does not significantly increase the time complexity. This is attributed to the fact that DPAM only creates the two paths during the computation of the attention map and is frozen during fine-tuning, thereby avoiding the computationally expensive process of gradient computation. Meanwhile, DPAM also does not result in large computation overhead during inference: AnomalyCLIP w/ DPAM gets 8.92 FPS vs. 10.21 FPS for w/o using DPAM.

We also provide a brief summary of the main changes made to the revised version of the paper in response to the reviews:

• As recommended by Reviewer NCKk, we have provided a more detailed analysis of DPAM and examined the computation overhead.
• To address the concerns raised by Reviewers w1Ea, GzuZ, and vcXy, we have added the suggested module ablation study by systematically removing each module to highlight its effectiveness, particularly in the case of DPAM.
• In response to the feedback from Reviewer GzuZ, we have included an analysis of successful cases and potential failures.
• Ablation studies have been conducted on various aspects, including the text prompt template of CoOp (Reviewer w1Ea), the unshared text prompt (Reviewer GzuZ), local visual features ensemble (Reviewer vcXy), the backbone of CLIP (Reviewer 5wSC), and local loss (Reviewer 5wSC).
• We have also addressed the feedback from Reviewers w1Ea and GzuZ by incorporating important references that were previously missing.

We have uploaded the revised paper which includes additional experiments and various clarifications to address the feedback from the reviewers. For ease of review, we highlight the revised text in orange. For other questions raised by the reviewers, please see our response to individual questions and concerns below each review.

References:

1. Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18082--18091, 2022.
2. Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anomalygpt: Detecting industrial anomalies using large vision-language models, 2023.