MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection

We believe the reviewer may have misunderstood the contribution of this paper. To clarify, we would like to restate the key contributions of MMAD as a benchmark. First, this benchmark fills the gap in the application of MLLMs in the industrial domain and sets new challenges for their capabilities. Second, we introduce a novel pipeline for generating semantic annotations for visual anomaly detection data, addressing the issue that MLLMs cannot be directly evaluated on IAD datasets. Finally, we comprehensively evaluate the performance of representative MLLMs on MMAD, highlight the weaknesses of current MLLMs in fine-grained industrial knowledge and multi-image understanding, and provide two general enhancement schemes.

Now, we will address the reviewer’s comments one by one:

1. Weak Contribution Due to Dependence on Existing Datasets

We believe that constructing a new benchmark based on existing datasets is a reasonable approach. On the one hand, industrial anomaly detection datasets have been extensively validated by researchers, providing a certain level of authority and reliability. On the other hand, most benchmarks for MLLMs are built upon existing datasets, provided that the images meet the evaluation criteria. Examples of such benchmarks include:

• MathVista [1]
• Seed-Bench [2]
• CompBench [3]
• MME [4]
• MME-Video [5]
• MMBench [6]

In addition, existing industrial anomaly detection datasets cannot be directly used for evaluating MLLMs. We have designed a new annotation pipeline and incurred the cost of generating semantic annotations for the dataset. Furthermore, we have invested a substantial amount of manpower in filtering the data.

2. "First-ever comprehensive" is kind of misleading

We believe the term "first-ever comprehensive" is not misleading. First, to the best of our knowledge, there is no existing benchmark in the industrial anomaly detection domain specifically designed for MLLMs, so the term "first-ever" is justified. Second, the comprehensiveness of our dataset is reflected not only in the variety of image categories and defect types but also in the breadth of sub-tasks. Existing datasets typically focus only on anomaly detection and localization. In contrast, our benchmark introduces additional tasks such as defect classification, defect analysis, defect description, object classification, and object analysis, resulting in a total of seven key sub-tasks.

3. Two enhancement strategies proposed are fairly standard

The two enhancement strategies were conceived after evaluating MMAD, and they represent two potential avenues for improvement. RAG aims to test whether knowledge-based texts can assist in anomaly detection, while the Agent strategy tests whether existing expert models can help improve the perceptual capabilities of MLLMs. Our intention was not to develop entirely novel methods but rather to conduct experiments by applying existing ideas in a reasonable manner.

However, we have “domain-specific” optimizations while we apply the concepts of RAG and Agent, and their specific implementations are entirely new. Specifically, the database construction in RAG is different from existing general methods. We created a retrieval library by summarizing defect and normal samples using descriptions generated by both experts and models rather than by collecting data from the web. Moreover, anomaly detection models have never been integrated into the Agent framework before, and we are the first to introduce such models. As a result, we also explored novel visualization methods for this approach.

4. Interest for the ICLR audience

We believe that ICLR has a strong sense of inclusivity and a practical focus, which makes it highly relevant to the industrial anomaly detection domain. For example, works like AnomalyCLIP[7] and MuSc[8] have been well-received. Additionally, many other evaluation-focused works in different domains, such as MathVista[1] and ViLMA, have been presented at ICLR. Therefore, we argue that the contribution to ICLR is not solely based on presenting novel methods but also on advancing important application domains and benchmarks.

Thank you for recognizing the contribution of our dataset. We will address your concerns one by one. 

1. Contribution to the General ML Community

We believe that MMAD also makes an important contribution to the general ML community for three main reasons:

I The Need for AGI to Handle Simple Tasks: General Artificial Intelligence (AGI) should possess the ability to handle basic tasks. AGI should not only solve complex logical problems but also take over repetitive and simple tasks, much like the work of industrial quality inspectors. As we mentioned in the paper:

"Nowadays, multimodal large language models, represented by GPT-4, can already do many human jobs, especially high-paying intellectual jobs like programmers, writers, and data analysts. In comparison, the work of quality inspectors is simple, typically not requiring a high level of education but relying heavily on work experience. Therefore, we are greatly interested in the question: How well are current MLLMs performing as industrial quality inspectors?"

However, our testing data reveals that, even for simple tasks, current MLLMs still require significant improvements.

II Broad Relevance to Anomaly Detection: The data we tested uses anomaly detection tasks, which have broad relevance to other fields. For example, anomaly detection in medical imaging, remote sensing, and other areas could benefit from similar approaches and methodologies.

III Revealing Gaps in Current MLLM Capabilities: Our benchmark introduces a new, specific application for MLLMs, while also highlighting areas of capability gaps that are rarely addressed in previous benchmarks. For instance, fine-grained image comparison is crucial in our benchmark, yet existing benchmarks like CompBench, while offering image comparison tasks, rely on synthetic tasks that do not provide realistic or actionable references.

2. Multiple Choice Format

We acknowledge that the multiple-choice format differs from real-world scenarios, but we believe it is currently the most appropriate format for constructing MMAD, for the following three reasons:

I. Multiple-choice questions exist in industrial settings. For instance, when asking whether there is an anomaly, the only options are typically "Yes" or "No". The number of defect categories is also limited, allowing the model to choose from a set of options. Furthermore, the position of the defect and the affected category can be designed in a limited manner, such as dividing the area into a grid (e.g., a 3x3 grid) or categorizing the impact by function.

II. Accuracy in multiple-choice questions reflects model performance. The multiple-choice format is also adopted by many well-known general benchmarks, such as MMBench[1] and MMMU[2]. We provide only four options for the model to select from, which could indeed lead to situations where the model provides the correct answer without fully understanding it. An example of this is provided in Appendix B, Figure 10. However, our benchmark asks multiple questions from different angles about the image. The model must form a correct understanding to answer subsequent questions. Therefore, our benchmark is capable of comprehensively evaluating the model's abilities.

III. Open-text answers in industrial scenarios are difficult to evaluate. In industrial-related question-answering, the correctness of open-text answers is hard to measure. For example, the same defect might be referred to by multiple names, or there may be proprietary terms. Thus, matching words becomes unfeasible, and at present, no suitable text embeddings for industrial domains exist. Additionally, MLLMs generally lack industrial domain knowledge, making it difficult to evaluate open-text answers in this context.

[1] Liu, Yuanzhan et al. "MMBench: Is Your Multi-modal Model an All-around Player?" ECCV 2023.
[2] Yue, Xiang et al. "MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI." CVPR 2024.

3. Report Recall

Given that other sub-tasks have four options and recall cannot be calculated, we assume that your concern is about recall in the anomaly discrimination task with two options. In this task, we use the average accuracy of normal and abnormal samples as a metric, which provides a good measure of the model's performance. Based on your suggestion, we further measured recall for each model in the anomaly discrimination task, as shown in the table below.

From the table, we can see that the recall metric does not show any clear pattern, and it cannot serve as a comparison indicator between different models. However, when combined with accuracy, it can reflect some of the errors made by the models. For example, in the case of LLaVA-1.5, despite its low accuracy, it has a particularly high recall rate, indicating that it classifies almost all samples as abnormal. If you are still interested in such results, we can further test precision and conduct a combined analysis.

4. How RAG Improves the Performance

In our paper, RAG is primarily used to retrieve domain knowledge. This domain knowledge provides descriptions of common defects as well as component descriptions of normal objects, enabling the model to better discern the content of the image based on this knowledge, thereby improving its performance. The improvement is not due to the increase in the context length, but rather due to the presence of key information within the context. We provide two examples of RAG-retrieved content in Figures 15 and 16 of Appendix B.

5. Providing the Expert Information Beyond the Visualizations

Thank you for your suggestion. We conducted additional experiments to validate that text can also serve as a form of expert information. Since there is no expert model available to provide information directly from other modalities, we converted the ground truth (GT) into text descriptions of anomaly locations and sizes and provided this to the MLLM. The results are shown in Table 6.

Some of the data from Table 6 are as follows:

By comparing with other methods of providing GT information, using text further improves anomaly detection and defect localization capabilities. However, the improvement in perception of categories and appearance is not as significant as that achieved with visual cues.

6. Differences between Industrial Anomaly Detection Tasks and General Visual Q&A Tasks

Industrial anomaly detection tasks fundamentally differ from general visual Q&A tasks in several key aspects:

I. Simplicity and Fixed Context: Industrial anomaly detection tasks are relatively simple because the scenarios are highly standardized, with little variation between samples. Models are not required to solve complex logical problems, unlike general visual Q&A tasks, which often involve more dynamic, diverse, and abstract visual reasoning.

II. Focus on Low-level Visual Features: In contrast to general visual Q&A tasks, which typically focus on object-level features (e.g., identifying objects or their relationships), industrial anomaly detection emphasizes low-level visual characteristics such as texture and structure. These features often lack explicit semantic meaning, making it more challenging to generate accurate and meaningful descriptions.

III. Availability of Contrasting Samples: In industrial anomaly detection, obtaining contrasting samples (i.e., normal samples for comparison) is relatively easy and cost-effective. This allows us to provide a template image to the model for comparison, a common practice in industrial inspection. In contrast, general visual Q&A tasks often struggle to define such "standard templates," making them inherently more complex.

3. Multiple Choice Question Format

We appreciate the reviewer's discussion on this issue. Below, we explain our rationale for using four-option multiple-choice questions:

• Open-text responses may not be the way MLLMs are applied in industrial settings. MLLMs are likely to be implemented in industrial scenarios through multiple-choice questions due to their greater stability. Consider that with open-text responses, it would be difficult to constrain the model's output range, making it challenging to determine how to process samples. In actual industrial production, Quality Assurance experts typically provide categories and descriptions for high-frequency defects while categorizing long-tail defects under "other."

• Too many options might prevent accurate evaluation of models. Four-option multiple-choice questions are the most common format used by humans and are what most MLLMs have been trained on. Having too many options would create longer contexts, making it difficult for some models to properly understand each option. MMAD primarily aims to measure different models' defect perception and judgment capabilities, and we want to avoid any unfairness caused by the question format, hence our choice of the four-option format.

• The four-option format can also accommodate scenarios with multiple options. For instance, if there are 16 possible defect categories, we can first divide them into four groups, have the model select the most likely option from each group, and then recombine the selected four options for the model to choose from. Thus, our evaluation method can be extended to more options. In practice, the MMAD dataset includes numerous repeated objects and defects, but each sample has different options, making MMAD's evaluation comprehensive. For example, when questioning about "large broken" defects on a "bottle," the first sample's distractors are "Foggy appearance," "Color fading," and "Scratched surface," while the second sample's distractors are "Smudge," "Discoloration," and "Scratch," with only "Scratch" being repeated.

We will add this discussion to the appendix after the discussion concludes. We also think that open-text responses or additional options also have evaluation value, and we will explore these in future research.

4. Broader Value to the ML Community

We believe this question is a natural extension of our discussion on "industry anomaly detection is a unique challenge." The unique requirements of IAD mean that solving IAD problems can significantly improve the general MLLM capability gap to real-world problems. The reasons are as follows:

• The high requirements of industrial tasks mean that MLLMs solving IAD problems must possess greater stability.
• The focus on low-level visual features contributes to understanding real-world problems.
• The multi-image understanding and image comparison capabilities required in IAD can inspire other real-world domains, such as lesion detection in medical imaging and change detection in remote sensing images.

Again, we thank you for providing valuable feedback and taking the time to share your thoughts. In conclusion, we believe this research work has clear value, and we hope you can help make this work visible to the community to contribute to the development of the field. We are committed to addressing any remaining concerns.

Thank you for your recognition and suggestions. We have made several additions to the paper, and below are our responses to each of your comments:

1. Add human baseline

First, we would like to clarify that humans do not achieve 100% accuracy in the MMAD evaluation. This is because during the benchmark construction, human annotators had access to the original data annotations as references. However, during the human evaluation, evaluators do not have access to this data, which means they might miss subtle flaws or make mistakes for other reasons.

We have added detailed information about the human evaluation in Section 4.2. The updated text is as follows:

Human evaluation. We conduct a preliminary human evaluation using 177 examples randomly sampled from the entire benchmark. Eight evaluators were divided into two groups: 3 industrial anomaly detection researchers as experts and 5 ordinary participants. As shown in Table 2, ordinary participants outperform the best models by 4%, and experts by 12%. Humans excel in anomaly and defect-related tasks, highlighting MMAD's challenges and MLLMs' limitations in anomaly detection. However, in object-related VQA, GPT-4o surpasses humans due to its broader knowledge base, with examples provided in Appendix B.

Some of the data from Table 2 are as follows: | Model | Scale | Anomaly Discrimination | Defect Classification | Defect Localization | Defect Description | Defect Analysis | Object Classification | Object Analysis | Average | |:--------------------------:|:-----:|:--------------:|:--------------:|:------------:|:-----------:|:--------:|:--------------:|:--------:|:-------:| | Human (expert) | - | 95.24 | 75.00 | 92.31 | 94.20 | 83.33 | 86.11 | 80.37 | 86.65 | | Human (ordinary) | - | 87.96 | 66.25 | 85.58 | 81.52 | 71.25 | 90.97 | 69.31 | 78.98 | | claude-3-5-sonnet-20241022 | - | 60.14 | 60.14 | 48.81 | 67.13 | 79.11 | 85.19 | 79.83 | 68.36 | | Gemini-1.5-flash | - | 58.58 | 54.70 | 49.10 | 66.53 | 82.24 | 91.47 | 79.71 | 68.90 | | Gemini-1.5-pro | - | 68.63 | 60.12 | 58.56 | 70.38 | 82.46 | 89.20 | 82.25 | 73.09 | | GPT-4o-mini | - | 64.33 | 48.58 | 38.75 | 63.68 | 80.40 | 88.56 | 79.74 | 66.29 | | GPT-4o | - | 68.63 | 65.80 | 55.62 | 73.21 | 83.41 | 94.98 | 82.80 | 74.92 | 

2. Evaluate Claude Sonnet 3.5

We have tested the latest Claude-3-5-Sonnet-20241022, and the results are also included in Table 2. However, this commercial model performed poorly, with results similar to the much cheaper Gemini-1.5-flash. 

3. Response to Minor Style Suggestions

We appreciate your valuable suggestions and have made the corresponding revisions. Here, we would like to clarify two specific changes:

I. The term "unfair" in Line 79 refers to previous works [1] that forced MLLMs to output anomaly detection results (such as a heatmap), rather than testing their linguistic outputs, which we believe is an unfair comparison.

II. The term "autonomous agent" in Line 346 was indeed not used correctly, and we have changed it to "Tool agent."

4. The Mechanism of Contribution to Manufacturing

This dataset is primarily used to measure the capabilities of MLLMs and promote their advancement in industrial applications. Several sub-tasks mentioned in the paper are of significant interest in the industrial domain. If an MLLM performs well on the current dataset, it could potentially be deployed directly on the production line as a quality control worker. In this case, humans could provide information to the model in various forms, such as a few images, brief textual descriptions, or complex documents. MLLMs can not only perform quality control tasks but also combine other capabilities, such as data summarization, table generation, and more, to bring additional possibilities to the manufacturing industry.

[1] Zhang, J., Chen, X., Xue, Z., Wang, Y., Wang, C., & Liu, Y. (2023). Exploring Grounding Potential of VQA-oriented GPT-4V for Zero-shot Anomaly Detection. ArXiv, abs/2311.02612.

Table 7: Performance comparison of different MLLMs with and without vision.

Figure 10: A case of a question-and-answer result in qualitative analysis. Since MMAD is test- ing with multiple-choice questions, we cannot directly analyze the answers. Instead, we use the chain-of-thought approach to encourage the model to provide its own analysis. In this example, the InternVL-40B model did not notice the damage to the cable, yet it still chose the correct answer from the options. In comparison, GPT-4o can output a proper analysis and letter. Subsequent issues such as anomaly classification, localization, and description can further test whether the model’s cor- rect answers are coincidental. Therefore, a comprehensive evaluation, rather than simple anomaly detection, is essential.

Figure 11: A case of a question-and-answer result in qualitative analysis. A human can quickly notice the defect in the query image, while the model focuses on the number of components. This illusion tends to occur when the defect is minor, and the object’s composition is complex.

Figure 12: A case of a question-and-answer result in qualitative analysis. For object-related questions, GPT-4o possesses extensive knowledge and can analyze object information based on subtle clues. However, most humans do not have such comprehensive knowledge. For instance, non-specialists may not recognize BAT+ and BAT-, let alone the TC4056A IC chip.

We appreciate the reviewer’s thoughtful feedback and constructive suggestions, which have helped us improve our paper. Based on your comments, we have made several revisions, detailed as follows:

1. Details on Human Supervision

We have added a detailed description of the human supervision process for dataset filtering in Appendix A.5, which includes both textual explanations and an illustration of the human filtering tool. The text now reads as follows:

Our textual data is generated by MLLMs, so its accuracy requires human supervision. In our designed process, human supervision is responsible for filtering the final model-generated multiple-choice questions. We first perform preliminary filtering through the program and then enable annotators to conduct an item-by-item review. A total of 26 personnel were involved in the review process, all of whom are researchers in industrial inspection or computer vision and possess a certain level of expertise. However, it should be noted that the industrial inspection field is highly specialized, with significant differences between products. Our annotators' understanding may differ from the professionals' understanding of where the original image data originates. We have developed a tool to enable annotators to filter out problematic multiple-choice questions quickly. As illustrated in Figure 8, we provide the original annotation information through visual and textual means to help annotators accurately identify objects and defects. At the same time, annotators do not need to correct every issue; they simply mark the erroneous questions for exclusion, significantly improving efficiency.

2. Diversity Analysis of Dataset

We agree that a diversity analysis of the dataset is important. While we have already analyzed the diversity of image categories and defect types in the original manuscript, we have expanded the analysis to include semantic diversity in Appendix A.6, as suggested. This section now includes both visual and textual information, with word clouds showing the distribution of question and option terms. It is important to note that in the context of industrial inspection, questions are typically less diverse than in natural scene tasks, as the scenarios are more fixed, and the questions generally revolve around defect detection, categories, and appearance. Therefore, while the diversity of questions in our MMAD benchmark is relatively low, the diversity of options is higher.

The added text reads as follows:

To systematically analyze the diversity of the dataset, we separately calculated the frequency of phrases appearing in questions and options, presenting them in the form of word clouds in Figure 9. In the question text, the word “defect” appeared most frequently, followed by “object,” reflecting that our benchmark is constructed for industrial inspection scenarios, focusing on seven sub-tasks. In addition to common words such as “image,” “type,” and “appearance,” the questions also include diverse expressions with lower frequencies. For example, expressions indicating position such as “relative position,” “arrangement,” and “defect located.” In the text of the options, “Yes” and “No” are the standard answers for anomaly discrimination questions, thus appearing most frequently. Beyond these two words, the diversity of options is more evident, including specific descriptions of various positions or expressions of different types of defects.

3. Comprehensive Analysis

We would like to clarify that we have already included some in-depth analyses in the original manuscript. For instance, Section 4.3 contains analyses on the utilization of template images in MLLMs, the impact of model size, and the effect of the number of template images. Section A.2 discusses in-context learning. After reviewing your comments, we have added more extensive analyses to the appendix. Specifically, we have updated three qualitative analyses in Appendix B, included an analysis of the impact of the Chain of Thought in Section A.3, and added an ablation study of the visual components in Section A.4.

The added text reads as follows:

A.3 ANALYSIS OF CHAIN OF THOUGHT

Chain of Thought (CoT) is a commonly used method to enhance the logical reasoning abilities of MLLMs (Chu et al., 2024). To investigate whether current MLLMs lack reasoning when addressing MMAD problems, we introduced a straightforward CoT approach. The process involves three steps: first, the model identifies objects in the image; second, it compares the differences between the template image and the query image; and finally, it determines whether the identified difference constitutes a defect in the object. We incorporated a set of rules and adjusted the instructions accordingly, dividing the CoT responses into two stages. As shown in the Table 6, InternVL2-76B, the best-performing open-source MLLM we tested, achieved a 1.5% improvement in CoT-based performance. However, its anomaly detection accuracy, which is the most critical metric, showed no improvement. On the other hand, InternVL2-40B experienced a performance decline after introducing CoT, potentially due to insufficient stability in the language model's reasoning capabilities.

Table 6: Performance comparison of different MLLMs with and without CoT.

A.4 ANALYSIS OF VISION DISABLE

Some studies (Tong et al., 2024a; Chen et al., 2024a) have proposed that determining whether a benchmark requires visual input to be solved has been a persistent challenge in vision-language research. To validate MMAD, we masked the visual components and compared the performance of MLLMs with and without visual input. The results, as shown in the Table 7, indicate that most subtasks in MMAD are highly dependent on the visual components. Performance significantly decreases when the visual input is removed.

I thank the authors for providing comprehensive evaluations and clarifying several points. I have increased my soundness score and contribution score accordingly!

Dear Reviewer nkBV,

Thank you for your swift response and the revised evaluation of our paper. We sincerely appreciate the time and effort you have dedicated to reassessing our work. Your initial feedback was instrumental in enhancing our paper.

We believe that an improved rating could help this pioneering work gain the visibility it deserves within the community. We kindly request that you consider raising the rating of our paper. Should there be any additional improvements or unresolved issues that might impact the review, we would be grateful for your suggestions. We are committed to resolving any remaining concerns.

Thank you once again for your support.

Best regards, The Authors