ResAD: A Simple Framework for Class Generalizable Anomaly Detection

[To W3]. Below, we present the detailed results on the VisA dataset under the 4-shot setting.

The detailed results show that our method can achieve better results in most classes. In the revision, we will add the above detailed results and also detailed results of other datasets to the Appendix.

[To Q3]. Below, we present the table of computation complexity.

[Limitations]. In Appendix Sec.B, we provide discussions about our method's limitations. In the revision, we will further discuss the limitations of our method more comprehensively and clearly based on the comments of all reviewers.

[To W1 and Q1]. Thanks for your professional review. We think that our work and InCTRL should be concurrent work. We also initially submitted our work to CVPR2024, and received two weak accepts and one reject (you can see the relevant materials in the rebuttal pdf file). Regretfully, our work was rejected at that time. So, we should have proposed the idea of residual learning independently of InCTRL and almost at the same time. But our method has obvious differences with InCTRL in the definition and utilization of residuals. In CVPR2024, our paper was rejected mainly because a reviewer thought our result comparison with unsupervised AD methods was unreasonable and unfair. In this submission, we mainly compare with few-shot AD methods and InCTRL. Please see our response to Reviewer fL7E's Weakness 1, we provide a detailed comparison between our method and InCTRL. Based on the comparison, we think that our method has the following advantages:

(1) One main advantage of our method is that it can achieve image-level anomaly detection and also pixel-level anomaly localization, while InCTRL only achieves image-level anomaly detection (due to the designs in their method).

(2) Compared to residual distance maps (in InCTRL), residual features are easier to integrate into other feature-based AD methods. For example, in Sec.4.4, we combine the residual feature learning with UniAD and RDAD, which can effectively improve the models’ class-generalizable capacity. In InCTRL, the authors devise an anomaly scoring network to learn the residual distance maps. The residual distance maps seem not easy to integrate into other AD methods (based on our analysis in the response to Reviewer fL7E’s Weakness 1).

[To W2]. We checked the results again and found that the results of InCTRL we reported were the same as in its original paper (94.0 and 85.5 under the 2-shot setting, 94.5 and 87.7 under the 4-shot setting). In the InCTRL paper, there are two tables, the AUROC results are in Table 1, and Table 2 are the AUPRC results, which are overall higher than the AUROC results. The original results you mentioned may be from Table 2 of the InCTRL paper. In Table 1 of our paper, we report the AUROC results, the average results on four AD datasets of our method are also better than InCTRL’s.

[To W3]. Thanks for your suggestion. Due to the page limitation and the need to extensively validate the effectiveness of our method on multiple datasets, we adopted to report the dataset-level average results, which was also the way adopted in the InCTRL paper. We present the detailed results on the VisA dataset in the rebuttal pdf file and also the following Comment.

[To Q2]. When writing the paper, we thought the results on these four industrial AD datasets were enough to validate the effectiveness of our method (In Table 1, our method can achieve better average results on these datasets than other methods), so we didn't consider more datasets from other domains. Please see our response to Weakness 1 of Reviewer CQaQ, we further run experiments on a medical dataset and a video AD dataset.

[To Q3]. According to our response to Weakness 2, our method is better than InCTRL and WinCLIP in terms of the average results on four AD datasets. The advantages to InCTRL are discussed in [To W1 and Q1]. Compared to WinCLIP, we think the advantages are:

(1) WinCLIP generates anomaly score maps by directly calculating the similarity between vision and text features. The model is not trained on AD datasets. Thus, WinCLIP is more likely to rely on the visual-language comprehension abilities of CLIP. When text prompts cannot capture the desired anomaly semantics, the results may be poor. In addition, WinCLIP requires text prompts, which can bring extra complexity, while our method does not require any text.

(2) Due to its sliding window mechanism, WinCLIP has low efficiency. In Table 4 in the Appendix, we provide the number of parameters and per-image inference time of our method and other methods (we list the table in the rebuttal pdf file and also the following Comment). The inference speed of WinCLIP is 0.51fps, while our method's is 18.8fps.

[To Q4]. Please see our response to Weakness 3 of Reviewer 226V.

[To Q5]. Yes, in Table 1, our method is mainly compared with these few-shot AD methods. Including the cross-dataset results of UniAD and RDAD is not for comparison, but to demonstrate that conventional one-for-one (RDAD) and also one-for-many (UniAD) AD methods cannot be directly applied to new classes (compared to the results in their original papers). Thus, achieving class-generalizable anomaly detection requires new insights and specific designs. In Table 3, we mainly aim to demonstrate that the residual feature learning can be easily incorporated into conventional feature-based AD methods and can effectively improve their class-generalizable capacity.

We also considered combining our method with WinCLIP and InCTRL but found it not easy to achieve. In WinCLIP, the model is based on the alignment between vision and text features. As the semantics of residual features and initial features are different, converting to residual features can lead to misalignment with text features. In addition, our method is to learn the residual feature distribution, while WinCLIP doesn't have the training stage on AD datasets. In InCTRL, the model is designed based on residual distance maps. InCTRL devises an anomaly scoring network (discriminative model) to learn the residual distance map and convert it to an anomaly score. Our method is designed based on residual features, which utilizes a normalizing flow model (probabilistic generative model) to learn the residual feature distribution. The obvious differences in the definition and utilization of residuals between our method and InCTRL make integrating with InCTRL also not easy.

If you still have any questions, we are very glad to further discuss with you.

Thanks for your further response and provide more empirical results. The response addresses most concerns. However, I noticed that the results were based on new medical datasets. How about the results on the headct and brainmri datasets demonstrated in InCTRL? I am curious whether the proposed ResAD is sensitive to the dataset.

[To W1]. Thanks for your suggestion. We then checked the formula writing in several other papers and found that there is punctuation at the end of a formula. This is a quite good detail suggestion. We will make modifications in the revised version.

[To W2]. Thanks for your suggestion. Using triangles to represent anomalies does look clearer. We will further improve Figure 1 in the revised version.

[To W3, Q1, and Q2]. We greatly appreciate your suggestion. Yes, when the difference between normal images is too large, it may cause the reference feature pool is not representative. For practical applications, this issue should be particularly focused and reasonably addressed. Of course, the simplest resolution is to increase the number of reference samples. This is feasible, as in practical applications, the number of reference samples is usually not as strict as the 2-shot and 4-shot (following previous papers) in our paper. From the perspective of method comparison, we think that random selection is ok, as long as we ensure that all methods use the same reference samples, the result comparison is reasonable. [To Q2] However, in practical applications, we expect that the reference samples can fully represent their class, so it's best to have sufficient differences between the reference samples. Thus, the sample selection strategy cannot be random. [To Q1] A feasible method is to first cluster all available normal samples into different clusters based on a clustering algorithm (e.g., KMeans). Then, based on the number of reference samples, we evenly distribute it to each cluster. When selecting from a cluster, we can prioritize selecting samples closer to the center. During clustering, we think that the FID and LPIPS you mentioned are good ways to calculate the difference between two samples. In addition, when there are a large number of reference samples, we can also use the method in PatchCore to select coreset features as reference features, which will be more efficient and also representative. In the revision, we will add the above discussion on sample selection strategy to the paper.

[To W4 and Q4]. Yes, we train the model using 15 classes in MVTecAD and test it on 12 classes in VisA. For MVTecAD, we train on 12 classes in VisA and test on 15 classes in MVTecAD. Adopting the cross-dataset experimental setup is because we think that it's more challenging than cross-classes within a single dataset, thus can better verify the model's class generalization ability. A dataset may be collected under the same photography condition, variations in other factors besides the object itself may be minimal. For example, all images in MVTecAD only contain a single object, while the images of some classes in VisA have multiple objects. The backgrounds in MVTec3D are all black, which is not the case in other datasets. In addition, we also follow InCTRL to use the cross-dataset experimental setup.

We think that your mentioned experimental setting is also very reasonable, as by varying $n$, we can demonstrate the sensitivity of the model to different numbers of training classes. Under the 4-shot setting, we further run our model on MVTecAD (with $n=5, 10$). Note that different $n$ means the number of test classes $15-n$ is different (this will cause the test results of different n cannot be compared with each other). Thus, we use fixed 5 classes as the test classes, including hazelnut, pill, tile, carpet, and zipper. For $n=5$, the training classes include bottle, cable, capsule, grid, and leather. For $n=10$, the training classes include bottle, cable, capsule, grid, leather, metal nut, screw, toothbrush, transistor, and wood. The results are as follows:

The results demonstrate that cross-dataset is more challenging than cross-class in a single dataset. With more training classes, the results will be better, but the model is not very sensitive. Due to too many experiments, we currently do not have enough time to provide the results of other methods under this setting. In the revision, we will also add this setting to our experimental setup and complete other methods' results.

[To Q3]. For each input image, InCTRL finally only outputs an image-level anomaly score. Thus, InCTRL cannot calculate pixel-level AUROCs. In the revision, we will add a relevant explanation.

If you still have any questions, we are very glad to further discuss with you.

[To W1]. We greatly appreciate your suggestion. Under the 4-shot setting, we further evaluate our method on a medical image dataset, BraTS (for brain tumor segmentation) and a video AD dataset, ShanghaiTech (as our method is image-based, we extract video frames as images for use). The comparison results are as follows:

The results show that when applied to medical images and video scenarios, the cross-domain generalization ability of our method is also good. In the revision, we will include the above results and also results under the 2-shot setting in our paper. We will also attempt more datasets and further discuss our method's generalizability to other domains.

[To W2]. We greatly appreciate your suggestion. Although we only used a single group of few-shot samples, we strictly ensured that all methods used the same few-shot samples. So, when writing the paper, we thought that the result comparison was reasonable and didn't run on other groups. After reading your suggestion, we think that results on multiple groups are necessary. We then randomly select two groups of few-shot samples and obtain the following results:

In the revision, we will include the above results and also results under the 2-shot setting in our paper. Due to too many experiments, we currently do not have enough time to provide the results of other methods on these two groups. We will further supply these results in the revised version.

[To Q1]. We think that it will have an impact on the performance. Because it may cause some abnormal features to match similar abnormal features from the reference samples, making these abnormal residual features hard to distinguish from normal residual features. Since the few-shot samples are used as reference, it should be natural not to contain anomalies. For real-world applications, it's also not very hard for us to ensure that the few-shot reference samples are all normal.

[To Q2]. As UniAD and RDAD are both feature-based AD methods, combining our residual feature learning with them is straightforward. For UniAD, we convert the initial features into residual features and then perform subsequent feature reconstruction. For RDAD, the initial features extracted by the teacher network are converted into residual features as the learning target of the student network. Then, the student network is trained to predict residual representations of the teacher network. In the revision, we will add relevant details to describe clearly how to combine residual feature learning with existing models.

If you still have any questions, we are very glad to further discuss with you.

[To W4]. The window mechanism of WinCLIP is not limited to ViT. In WinCLIP, the window mechanism is mainly used to address the issue that the local patch features extracted by the CLIP image encoder are not aligned with the text features. The window mechanism provides image patches of different scales, which are then sent into the CLIP image encoder to obtain global features that can align with the text features. The window mechanism is intrinsically similar to the classic sliding window in object detection, thus can also be used in CNN networks. We can send image patches provided by the window mechanism into WideResNet50, and also obtain the window embedding maps of different scales as shown in Figure 4 of the WinCLIP paper. However, because the features of WideResNet50 are not aligned with the text features, we remove the language-guided anomaly score map and only generate the vision-based anomaly score map based on the few-shot normal samples (the WinCLIP+ in the WinCLIP paper). In lines 516-518, we have briefly mentioned our adaptation way. In the revision, we will add more details about the experiments in Table 5.

[To Q1]. The goal of our Feature Constraintor (optimized by the OCC loss) is to constrain initial residual features to a spatial hypersphere for further reducing feature variations. After the Feature Constraintor, feature variations can effectively be reduced, but this does not mean that the feature distribution is fixed within the hypersphere. Moreover, even if the feature distribution is fixed, we still need to learn the distribution and then can perform anomaly detection based on the learned distribution. The NF model is used to learn the feature distribution.

[To Q2]. During training, we found that summing up the three loss terms and then backpropagating gradients to optimize the whole model would lead to unstable training. Then, we used the "torch.detach()'' method in the Pytorch library to detach the features after the Feature Constraintor and then sent the detached features into the NF model. This simple way can make the model training more stable. In the revision, we will add this code detail to the paper. Thus, the weight of $L_{occ}$ can be set as 1 (i.e., we can not need to balance $L_{occ}$ with $L_{ml}$ and $L_{bg-spp}$, as the Feature Constraintor and the NF model parts are separated in the gradient graph). When training the NF model, the $L_{ml}$ is the basic loss. So, we keep the weight of $L_{ml}$ as 1 and set a variable $\lambda$ as the weight of $L_{bg-spp}$. By varying different $\lambda$ values, the results (under the 4-shot setting) about the sensitivity are as follows:

Both small and large $\lambda$ values can lead to performance degradation. $L_{bp-spp}$ is to assist model in learning abnormal residual features. Small $\lambda$ may cause the impact of abnormal features on final loss to be relatively small. Large $\lambda$ may lead to overfitting to known anomalies, which is not conducive to generalization.

[To Q3]. In our paper, we mainly propose the Abnormal Invariant OCC (AI-OCC) loss $L_{occ}$. The maximum likelihood loss $L_{ml}$ is the basic loss for training the NF model. The $L_{bg-spp}$ loss is following BGAD to effectively utilize abnormal features. In fact, the framework ablation studies in Table 2(a) also include ablation studies regarding our proposed AI-OCC loss. In Table 2(a), “w/o Feature Constraintor'' means the $L_{occ}$ is not used (as the Feature Constraintor is removed), “w/o Abnormal Invariant OCC loss'' means the $L_{occ}$ only has the first part of E.q.(2), while is not our proposed AI-OCC loss. In lines 293-298, we also provide discussions about the ablation studies of our AI-OCC loss.

[Limitations]. In Appendix Sec.B, we provide discussions about our method's limitations. In the revision, we will further discuss the limitations of our method more comprehensively and clearly based on the comments of all reviewers.

[To W1]. Thanks for your professional review. We think that our work and InCTRL should be concurrent work. We also initially submitted our work to CVPR2024, and received two weak accepts and one reject (you can see the relevant materials in the rebuttal pdf file). Regretfully, our work was rejected at that time. So, we should have proposed the idea of residual learning independently of InCTRL and almost at the same time. But our method has obvious differences with InCTRL in the definition and utilization of residuals. This (i.e., two independent works almost simultaneously proposed the residual learning idea) also demonstrates residual learning is an effective way to achieve class-generalizable anomaly detection. Then, we made comprehensive revisions to our paper based on the reviewers' suggestions. Subsequently, we also noticed the InCTRL paper and felt that our method has advantages compared to it. Thus, we submitted our revised paper to NeurIPS2024. In CVPR2024, our paper was rejected mainly because a reviewer thought our result comparison with unsupervised AD methods was unreasonable and unfair. In this submission, we mainly compare with few-shot AD methods and InCTRL. The main differences between our method and InCTRL are as follows:

(1) The definition of residuals in InCTRL is based on feature distances. The residual map is defined by ((E.q.(1) in the InCTRL paper): $M_x^l(i,j) = 1 - \left<T_x^l(i,j),h(T_x^l(i,j)|\mathcal{P}^\prime)\right>$, where $h(T_x^l(i,j)|\mathcal{P}^\prime)$ returns the embedding of the patch token that is most similar to $T_x^l(i,j)$ among all image patches in $\mathcal{P}^\prime$, and $\left<\cdot\right>$ is the cosine similarity function. Thus, InCTRL is based on residual distance maps, while our method is based on residual features.

By comparison, we think that residual distances in InCTRL can limit the range of residual representation (as the cosine similarity is in [-1,1]). This is not beneficial for distinguishing between normal and abnormal regions, as a position on the residual map is only represented by a residual distance value. Within a limited representation range (1-[-1,1] $\rightarrow$ [0,2]), normal and abnormal residual distance values are more likely to be not strictly separable. Thus, for a position on the residual map, it's hard for us to make decision based on a scalar value. So, InCTRL makes image-level classification based on a whole residual map (see the following (2)). In contrast, our residual features don't limit the range of residual representation and can retain the feature properties. In high-dimensional feature space, we can also establish better decision boundaries between normal and abnormal (a basic idea in machine learning: solving low-dimensional inseparability by converting to high-dimension).

(2) InCTRL devises a holistic anomaly scoring function $\phi$ to learn the residual distance map $M_x = \frac{1}{n}\sum_{l=1}^{n}M_x^l$ and convert it to an anomaly score: $s(x) = \phi(M_x^{+};\Theta_\phi) + \alpha s_p(x)$ (E.q.(8) in the InCTRL paper), where $M_x^{+} = M_x \oplus s_i(x) \oplus s_a(x)$ (E.q.(7)). $s_i(x)$ is an anomaly score based on an image-level residual map $F_x$ (see E.q.(4) in the InCTRL paper) and $s_a(x)$ is a text prompt-based anomaly score. Thus, InCTRL is to train a binary classification network based on residual distance maps. For each input image, InCTRL finally only outputs an image-level anomaly score. Our method is to learn the distribution of residual features, an anomaly score can be estimated for each feature, thus can be used to locate anomalies.

(3) Due to the designs in InCTRL that we mentioned above, one main advantage of our method is that it can achieve image-level anomaly detection and also pixel-level anomaly localization, while InCTRL only achieves image-level anomaly detection functionality.

As for performance, the average results on four AD datasets of our method are better than InCTRL's (please see Table 1 in our paper). In our response to Weakness 1 of Reviewer CQaQ, the cross-domain generalization ability of our method is also better.

[To W2]. We greatly appreciate your suggestion. Under the 4-shot setting, we further reproduce FastRecon and AnomalyGPT by using the same few-shot samples as ours. Based on the official open-source code, we obtain the following results:

Note that the original image encoder in AnomalyGPT is significantly larger than the ViT-B/16+ used in InCTRL and our ResAD$^{†}$. When AnomalyGPT also uses ViT-B/16+ as the image encoder, our method is more superior to it. In the revision, we will include the above results and also results under the 2-shot setting in our paper. We will also cite FastRecon and AnomalyGPT and add discussions with them.

[To W3]. Thanks for your suggestion. We will carefully check and modify obscure expressions and unclear concepts to make our paper easier to follow. We will also seriously consider other reviewers' suggestions to further improve the quality of our paper.

[Others]. For Weakness 4 and questions, please see the rebuttal pdf file or the following Comment.

If you still have any questions, we are very glad to further discuss with you.

The authors' rebuttal has addressed most of my concerns and I believe that the generalization ability of the proposed method is practical and significant, so I decide to raise my score. However, I still have some concerns about the conflicts of OCC and NF. Also, I observe the rebuttal file in the supplementary (Line 42-51) that same problem have mentioned by other comments, so I think it would be a major concern about the method. It would be better If there are some more convicing validation.