MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection

Q1: Motivation

(1) Since the model needs to learn the data distribution among samples with significant differences in a multi-class anomaly detection task, it requires a global modeling capability, such as that provided by Transformers. UniAD, as the first model to propose a multi-class anomaly detection task, uses a Transformer decoder to model global features. However, due to the quadratic computational complexity of Transformers, UniAD only uses Transformers on the smallest scale feature maps, which limits its performance in detecting small defects.

(2) Mamba, a model with linear computational complexity, is appropriately applied to multi-class anomaly detection. Its global modeling capability allows it to better learn the data distribution of different category samples, and its linear computational complexity enables modeling at multiple scales, improving the detection performance for small defects.

In summary, Mamba's linear computational complexity and global modeling capability make it well-suited for addressing multi-class anomaly detection problems. As illustrated in Fig. 1 in the main text, MambaAD enhances its performance in multi-class anomaly detection by integrating CNNs with local modeling capabilities into the Mamba structure of the decoder, thereby improving its ability to learn the distribution of each feature.

Q2: Local/Global Branches

We conducted three ablation experiments to verify the impact of branches in the LSS module. The Local branch represents the use of only the parallel CNN branch, without the Mamba-based HSS branch. The Global branch represents the use of only the Mamba-based HSS branch, without the parallel CNN branch, making the decoder in this structure purely Mamba-based. Finally, Global+Local represents the proposed LSS structure used in MambaAD, which combines the serial Mamba-based HSS with parallel CNN branches of different kernel sizes. The experimental results are shown in the table below. The Local branch, which uses only CNNs, has the lowest parameter count and FLOPs but also the lowest mAD metric, indicating high efficiency but suboptimal accuracy. The Global method, based on the pure Mamba structure, consumes more parameters and FLOPs than the Local method but shows a significant improvement in performance (+2.7%). Finally, the combined Global+Local method, which is the LSS module used in MambaAD, achieves the best performance with a notable improvement of (+1.6%) over the individual methods.

Q3: Evaluation Metrics

Please refer to Q3 to Reviewer gwhg for more quantitative analysis in Tab. 6 of the Rebuttal PDF and qualitative analysis in Fig. 1 of the Rebuttal PDF.

Q4: Mamba Encoder

(1) Existing anomaly detection methods primarily use pre-trained encoders based on CNN backbone networks. For instance, [1,3] utilize WideResNet-50, while [2] employs EfficientNet-b4. Additionally, [4] uses the Transformer-based ViT as the feature extraction backbone. Currently, no pre-trained encoder based on Mamba has been applied to the anomaly detection domain. Therefore, MambaAD also uses widely adopted CNNs as the pre-trained backbone network.

(2) In the main text, Tab. 2 presents ablation experiments on different CNN networks. The results show that ResNet-34 achieves a good balance between efficiency and performance. Although WideResNet-50 can achieve better results, its computational complexity and parameter count are eight to ten times higher than those of ResNet-34. Hence, MambaAD selects ResNet-34 as the backbone network.

(3) To explore the impact of pre-trained Mamba encoders on anomaly detection tasks, we selected two recently popular vision Mamba models as backbone feature extraction networks: VMamba [5] and EfficientVMamba [6]. We replaced the original ResNet-34 backbone with these models, and the results are shown in the table below. The parameter count and FLOPs represent the values for the backbone network only, not the entire model, to facilitate the comparison of different backbone network performances. EfficientVMamba-T has the fewest parameters and FLOPs but performs poorly on the MVTec-AD dataset in terms of the mAD metric. VMamba-T has a parameter count nearly four times that of ResNet-34, but its FLOPs are only 1.5 times higher. However, its mAD results are still inferior to those of ResNet-34.

Currently, existing pre-trained Mamba-based backbone networks are neither lightweight nor effective for feature extraction in anomaly detection. Therefore, we will continue to research lightweight, efficient, and high-accuracy Mamba-based backbone networks and apply them as feature extractors in the anomaly detection domain.

[1] Anomaly detection via reverse distillation from one-class embedding. Deng et al.. CVPR'22.

[2] A unified model for multi-class anomaly detection. You et al.. NeurIPS'22.

[3] Hierarchical vector quantized transformer for multi-class unsupervised anomaly detection. Lu et al.. NeurIPS'23.

[4] Exploring plain vit reconstruction for multi-class unsupervised anomaly detection. Zhang et al.. arXiv'23.

[5] Vmamba: Visual state space model. Liu et al.. arXiv'24.

[6] Efficientvmamba: Atrous selective scan for light weight visual mamba. Pei et al.. arXiv'24.

Q5: Scanning Methods

Please refer to Q1 to Reviewer aMDx and additional ablation experiments in Tab. 5 of the Rebuttal PDF.

Q1: Compared with Transformer-based Method

(1) As shown in Fig. 1 of the main text, although UniAD is a Transformer-based method, it employs a single-scale approach, whereas our MambaAD, based on Mamba, utilizes a multi-scale approach. Specifically, single-scale methods model and reconstruct only the smallest scale feature map during the decoder stage. In contrast, multi-scale methods progressively increase the resolution from the smallest scale feature map during the decoder stage, modeling and reconstructing feature maps at multiple scales. Due to the quadratic computational complexity of Transformers, UniAD opts for a single-scale framework as its primary architecture. Consequently, UniAD, which reconstructs only the smallest scale feature map, naturally has the least parameter count and FLOPs. On the other hand, MambaAD, which models feature maps at three different scale resolutions, has slightly higher parameter counts and FLOPs compared to UniAD.

(2) To ensure a fair comparison, we used the same encoder, ResNet-34, as the feature extraction network. Within the current multi-scale framework of MambaAD, we employed both a pure Transformer-based decoder and a pure Mamba-based decoder to validate the efficiency of the method. The results are shown in the table below. It can be observed that, under the same framework, the pure Mamba-based decoder has significantly fewer parameters -4.6M and FLOPs -0.9G compared to the pure Transformer-based decoder, demonstrating a clear efficiency advantage. Additionally, in terms of performance, the Mamba-based method shows a notable improvement in the mAD metric (+1.9%), achieving a significant advantage in effectiveness as well.

Q2: Different Backbone

(1) The backbone feature extraction network used in MambaAD is ResNet-34. Although WideResNet-50 can achieve slightly better performance (+0.6%), it has ten times the number of parameters and more than eight times the computational complexity. Therefore, we opted for the more lightweight ResNet-34 as the backbone.

(2) To ensure a fairer comparison with UniAD and eliminate the influence of different backbone networks, we replaced the backbone networks in UniAD with three different options: EfficientNet-b4, ResNet-34, and WideResNet-50. Similarly, we replaced the backbone networks in MambaAD with these three options. Originally, UniAD with an EfficientNet-b4 has 3.6G FLOPs and an accuracy of 81.7. When we replaced the MambaAD backbone with EfficientNet-b4, it achieved only 2.7G FLOPs and an accuracy of 84.1, which is +2.4% higher than UniAD. With the same backbone network, MambaAD has fewer parameters and FLOPs and better performance compared to UniAD. Subsequently, replacing the UniAD backbone with ResNet-34 resulted in a performance decline, while further replacing it with WideResNet-50 led to a +0.7% improvement in performance, albeit with doubled parameters and FLOPs.

In conclusion, under the same backbone network, MambaAD consistently outperforms the Transformer-based single-scale UniAD method, demonstrating significant improvements in performance.

Q3: Evaluation Metrics

(1) From the perspective of method categorization, our method and the compared SoTA methods can be distinguished as follows: RD4AD, UniAD, DiAD, and MambaAD are rec.-based anomaly detection methods. DeSTSeg, on the other hand, is a hybrid method that combines data augmentation and reconstruction. During training, DeSTSeg artificially introduces random anomalous regions to construct anomalous data, enabling the segmentation network to learn the precise locations of these regions, thereby reducing the probability of missed and false detections. In contrast, other rec.-based methods only learn the feature distribution of normal samples and perform comparisons at the feature level during testing. Consequently, the hybrid DeSTSeg method may outperform other rec.-based methods in certain anomaly localization metrics. However, MambaAD, with its simpler approach, achieves the majority of SoTA metrics and a few near-optimal results overall.

(2) When comparing rec.-based methods, MambaAD demonstrates its effectiveness and robustness across four anomaly localization evaluation metrics on six different datasets. Recent work [1] has suggested that existing anomaly localization evaluation standards are not specifically designed for segmentation-based anomaly detection tasks. We selected the most commonly used IoU metric to demonstrate the accuracy of anomaly localization which intuitively reflects the accuracy and overlap. We tested the mIoU of rec.-based SoTA methods on six different types of six datasets. The results, as shown in Tab. 6 in Rebuttal PDF, indicate that MambaAD significantly outperforms existing SoTA methods in terms of mIoU values across all six datasets, thereby proving the accuracy and robustness of our method in anomaly localization.

Finally, we conduct a qualitative analysis of the MambaAD method in comparison to RD4AD within the same framework. As visualized in Fig. 1 in Rebuttal PDF, MambaAD demonstrates higher localization accuracy on anomaly images compared to RD4AD. This further validates the accuracy of the method in anomaly localization.

[1] Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark. Zhang et al.. arXiv'24.

Q1: Further Exploration of Scanning Methods and Direction Selection

(1) Scan Directions

Firstly, we further investigate the impact of different scanning directions on Params, FLOPs, training time over 100 epochs, training memory usage, and final performance, as shown in the table below. It is evident that as the number of scanning directions increases, there is a corresponding increase in Params, FLOPs, training time, and training memory usage. Consequently, the performance in terms of mAD also improves. Therefore, for those prioritizing efficiency, scanning with 2 directions offers the lowest Params and FLOPs, while also requiring less training time and memory, yet achieving satisfactory performance. Conversely, for those aiming for optimal performance and willing to accept increased computational load and training time, scanning with 8 directions yields the best results.

(2) Scan Methods

Subsequently, we further explore the impact of different scanning methods and the fusion of multiple scanning methods on the results, as shown in the table below. Different scanning methods exhibit negligible differences in terms of parameters, FLOPs, training time, and memory usage, with variations only observed in the final results. The results indicate that using any of the five different scanning methods individually can achieve satisfactory outcomes. However, combining different scanning methods tends to degrade performance, possibly due to the significant differences between the methods, which may hinder the selective scanning and global modeling convergence of the SSM. Therefore, we ultimately choose to use only the Hilbert scanning method. This method allows for continuous scanning in the central part of the image, which is advantageous for SSM in learning and modeling the distribution of anomaly detection data, as most data in anomaly detection datasets are centered in the image.

Q2: Additional Datasets

In Tab. 1 of the main text, we demonstrate the effectiveness and robustness of our method on three industrial anomaly detection datasets: MVTec-AD, VisA, and Real-IAD. To further validate the effectiveness of our method, we include additional datasets from medical, 3D, and real-world scenes. The Uni-Medical dataset [1] comprises CT scans from three different anatomical locations. The MVTec-3D dataset [2] includes both RGB images and depth maps. Additionally, [3] extends the commonly used COCO dataset for detection and segmentation to anomaly detection, creating a large-scale, general-purpose COCO-AD dataset that encompasses complex scenes and diverse distributions. These datasets exhibit significant distributional differences compared to traditional industrial anomaly detection datasets, providing a robust test for our method. The results are summarized in the table below. For more detailed results, please refer to Appendix C-E. Our MambaAD method shows improvements over existing SoTA methods, with +4.2% on the medical dataset, +3.7% on the 3D dataset, and +2.5% on the COCO-AD dataset. These significant improvements across datasets with different distributions and scenarios validate the effectiveness and robustness of our method.

Q3: Robustness Under Different Environmental Conditions

Currently, the anomaly detection field lacks specialized datasets that account for varying lighting conditions and occlusions. However, the COCO-AD[3] dataset we use is a general-purpose anomaly detection dataset derived from real-world detection and segmentation scenarios. It includes a wide variety of objects and categories from real-world settings. As illustrated in Fig. 1(a) of the [3] paper, the dataset encompasses diverse scenes with significant variations between them. These scenes exhibit different data distributions and include various environmental conditions such as sunny and cloudy days. Additionally, the dataset features occlusions both within the same object and between different objects. In the table above, our method achieves SoTA on the COCO-AD, with an improvement of over +2.5% in the mAD metric. This further validates the robustness of our method in complex and general anomaly detection scenarios.

[1] Exploring plain vit reconstruction for multi-class unsupervised anomaly detection. Zhang et al.. arXiv'23.

[2] The mvtec 3d-ad dataset for unsupervised 3d anomaly detection and localization. Bergmann et al.. arXiv'21.

[3] Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark. Zhang et al.. arXiv'24.

Q4: Limitations and Future Work

The primary limitations lie in high-resolution application scenarios and the Mamba-based encoder. These areas will be the focus of future research. For a more detailed explanation, please refer to Q1 addressed to Reviewer FXx2.

Dear reviewer aMDx:

Thank you for recognizing our work. We commit to incorporating the content you suggested in the revised version. Thank you again for your effort in the review and the discussion!

Best regards!

Authors of MambaAD.

Q1: From a methodological perspective

(1) We adopt a reconstruction-based framework. Anomaly detection methods based on reconstruction can be broadly categorized into image reconstruction and feature reconstruction. Methods such as GAN and diffusion-based approaches primarily fall under image reconstruction, where the similarity between the generated image and the input is calculated. In contrast, feature reconstruction methods compute the similarity between the input features and the reconstructed features. RD4AD is merely a specific instance within the feature reconstruction methods. Additionally, this framework is not referred to as a contribution in our paper.

(2) Mamba, as a novel global modeling structure, has not yet been explored for its potential in anomaly detection (AD). We found that directly applying it to multi-class anomaly detection tasks does not significantly improve performance compared to the original CNN architecture (+0.4%) in Tab. 1. Therefore, we first combined Mamba's global modeling capability with CNN's local modeling capability in Tab. 3, which significantly improved performance (+2.8%), and proposed the AD-adaptive LSS module, corresponding to the contribution in the fusion modeling approach. Secondly, we introduced the AD-adaptive HSS method to specifically enhance the scanning approach (+1.1%) for multi-class anomaly detection data, corresponding to the contribution in the scanning method.

Q2: More key ablations

(1) Decoder Ablations

We conducted experiments on different decoder structures within the same RD4AD framework, and the results are summarized in the table below. The Mamba-based decoder, which uses only Mamba as the component for each decoder module instead of the proposed LSS module that integrates Mamba with CNN structures, shows a decrease in performance compared to the result of 86.0 reported in the paper. However, compared to decoders based purely on CNN or Transformer architectures, the Mamba-based decoder achieves the highest mAD score of 82.1. Additionally, the Mamba-based decoder has fewer parameters and FLOPs compared to the Transformer-based structure.

(2) Components Ablations

The ablation experiments for the proposed components are summarized in the table below. Using the most basic decoder purely based on Mamba and employing only the simplest two-directional sweep scanning method, we achieve an mAD score of 82.1 on the MVTec-AD dataset. Subsequently, by incorporating the proposed LSS module, which integrates the global modeling capabilities of Mamba with the local modeling capabilities of CNNs, the mAD score improves by +2.8%. Finally, replacing the original scanning directions and methods with HSS, which combines features from different scanning directions and employs the Hilbert scanning method, better aligns with the data distribution in most industrial scenarios where objects are centrally located in the image. This results in an additional +1.1% point improvement in the mAD score. Overall, the proposed MambaAD achieves an mAD score of 86.0 on the MVTec-AD dataset, reaching the SoTA performance.

Q3: More Limitations

(1) Although Mamba has linear computational complexity, the current multi-scale reconstruction-based framework (c.f. Fig. 1) is more effective than the single-scale UniAD method (81.7 -> 86.0). However, due to the limitations of the framework, existing methods are still not efficient enough.

(2) Moreover, since most industrial anomaly defects are relatively small, as seen in datasets like VisA and Real-IAD, where defects occupy a very low proportion of the entire image, it is necessary to improve detection resolution to identify small target defects. However, increasing the resolution brings additional computational complexity to the model. Therefore, designing more efficient and lightweight anomaly detection models is urgently needed in industrial scenarios. We extended the MambaAD method to high-resolution tasks, such as $512^2$, $1024^2$, and even higher $2048^2$ input resolutions. At a resolution of $512^2$, the model's FLOPs reach 33.2G with a throughput of only 45.1. At a resolution of $1024^2$, the FLOPs soar to 133G with a throughput of only 10.3. At a resolution of $2048^2$, we could not measure the corresponding FLOPs due to OOM (Out of Memory) errors. In summary, there is still much room for improvement in the lightweight design of current models, especially when applied to high-resolution industrial scenarios. The current models do not yet meet the efficiency and real-time requirements of high-resolution industrial applications.

(3) The existing Mamba-based encoder still has certain deficiencies in terms of efficiency and effectiveness. In the table below, we replaced the current encoder with two different lightweight pre-trained Mamba encoders. The parameters and FLOPs only represent the results of the encoder. Although EfficientVMamba-T has the highest efficiency, its feature extraction capability is lacking. On the other hand, VMamba-T shows significant improvement in effectiveness but still lags behind the current CNN encoder in terms of real-time performance and overall effectiveness. Therefore, designing more lightweight, efficient, and accurate Mamba-based encoders for anomaly detection is crucial and will be the focus of our future research.

Thank you for the clarification. I noticed that the authors referenced [46] for the Uni-Medical dataset. However, the paper in [46] neither includes experiments on medical anomaly detection nor references any medical anomaly dataset. Could the authors please confirm the accuracy of this citation for the Uni-Medical dataset?

Dear reviewers,

Thanks a lot for your effort in reviewing this submission! We have tried our best to address all the mentioned concerns/problems in the rebuttal. Feel free to let us know if there is anything unclear or so. We are happy to clarify them.

Best regards!

Authors of MambaAD.