Defect Spectrum: A Granular Look of Large-Scale Defect Datasets with Rich Semantics

Thank you for your thorough and insightful feedback on our paper. Your expertise in unsupervised anomaly detection is evident, and we appreciate the depth of your review. Firstly, we’d like to revisit our work. This paper primarily aims to pose a new question in defect detection research, rather than provide a definitive solution. We observed that current benchmarks in defect inspection research are approaching performance saturation. However, in real-world scenarios, defect inspection remains a challenging problem. This gap between academic research and industry applications prompted us to propose a new, more challenging benchmark. In this paper, we invite the research community to explore algorithms that can effectively address this benchmark. As for your concerns. we fully understand and would like to address them as follows:

Inclusion of Anomaly Detection (AD) Methods: In our benchmark, we focused on evaluating semantic segmentation methods, intentionally excluding unsupervised AD methods. The reason is that current AD methods are not equipped to provide masks for different defect types, which is crucial in our approach. We compute the mean Intersection over Union (mIoU) by averaging across different defect types, a metric where AD methods fall short as they cannot differentiate between these types. Regarding the concern about 'poor detection performance,' our use of mIoU for multi-class segmentation sets a higher standard than the binary classification task's Average Precision at a Recall (AUPR). Although our mIoU values are lower than AUPR scores reported in AD papers, they reflect a more robust defect inspection capability, as demonstrated in Appendix Section. E of our updated paper. Additionally, AD methods struggle with unaligned images, such as those in the VISION dataset, where variance is high.

Utilizing Normal Data: We agree with your observation regarding the utilization of normal data. Normal data are indeed part of our defect spectrum dataset, initially included in our four subsets without requiring additional annotation. To address your concern about performance benchmarking, we have decided to incorporate normal data into our training set and re-evaluate our candidate methods. Due to time constraints in the rebuttal period, we have re-conducted a portion of the experiments from Table 3. Preliminary results shown in the table below (also see Appendix Section. F of our updated paper) demonstrate improved performance with the inclusion of a small amount of normal data by addressing the issue of over-penalizing non-defective areas. We plan to revise Table 3 accordingly, incorporating these findings.

Combining different percentages of normal(defect-free) data. The source indicates the re-fined MVTec training set without any normal data

Differences from Semantic Segmentation: While our task may appear similar to semantic segmentation, there are distinct challenges in handling industrial data. These include issues such as few-shot learning and the small area problem, which differentiate our task from standard semantic segmentation in natural images.

Future Directions and Community Involvement: Our work aims to define a more complex defect inspection task in a practical manner and present augmented datasets for exploration. We leave the design of specific methods to the broader research community. Our paper's goal is to establish a foundation for researchers to adapt and leverage state-of-the-art methods within our proposed benchmark.

Finally, I sincerely appreciate the valuable insights provided by the reviewers and warmly welcome any further criticisms or suggestions. Your timely feedback is crucial for the enhancement of this paper. I look forward to your valuable comments.

Dear reviewer EDJD,

As the window for reviewer-author interaction is closing soon, on November 21st, I wanted to extend my sincerest gratitude for the invaluable time and effort you have dedicated to reviewing our work. To ensure that we have met your expectations, may I kindly ask if you find our responses satisfactory and if there are any remaining issues that need further clarification or improvement?

Thanks to the author for his detailed answer. Although there is a large gap between this work and existing research, I believe it is a valuable attempt. The experimental results that the author added using more normal samples are in line with expectations, but what I want more is some insight into how to better utilize normal samples. Taken together, I think the paper is a meaningful attempt. However, it's might suitable for some dataset track or workshop.

Dear reviewer EDJD,

Thank you for your invaluable comment. I noticed that you change the rating from 6 to 5 and added a comment of “However, it's might suitable for some dataset track or workshop.”

I would like to clarify a key point regarding the submission category of our paper. We have submitted our paper under the "Primary Area: datasets and benchmarks" track. This submission choice aligns with the guidelines of ICLR, which, to our understanding, accepts papers focused on datasets.

Given this alignment with the submission track requirements, I kindly request you to reconsider the rating of our paper. We believe that our submission is well-placed in the datasets and benchmarks track, and we hope this clarification might positively influence your assessment.

Thank you for your insightful observations. We address each of your concerns as follows:

Open Source Issue: We completely understand your concern regarding the open-source of our dataset. We have now made these resources publicly through this anonymous link: https://defect-spectrum-authors.github.io/defect-spectrum/, ensuring that our contributions to the field of defect inspection are substantial and accessible to the research community.

Comparison between Refined and Baseline Dataset: To address your concern about the comparative experiments. Since both the training set and testing set were being refined, it is hard to compare the baseline-trained model quantitatively using the refined testing set or the source testing set. Thus, we qualitatively compared the segmentation model trained on the baseline dataset and our refined dataset, as shown in the Appendix section. D of our updated paper. The results with our annotation demonstrate 1) a more granular segmentation mask; 2) a more comprehensive detection (including those that remain unlabeled in the baseline dataset); 3) captures a wide range of defective types with better granularity while having rich semantics (different defective classes).

Clarification on Table 4 and Synthetic Data Utilization: We appreciate your comments on Table 4 and the need for clarification regarding synthetic data. Regarding the results in Table 4, we would like to clarify that we have selected the best baseline to perform our performance boost experiment. We think although the improvement is comparatively smaller than the increase in the MVTec dataset, it still demonstrates the effectiveness of the synthetic data. We also extended the experiment to compare using three other baselines: M2F, MiT-B0, and DeepLabV3+, the experiments can be found in the table below (also see Appendix Section. A in the updated paper).

Performance (mIoU) comparison between models trained with and without synthetic data. The bolded text indicates results with synthetic data.

Generation Data Ratio Experiment: Further, as shown in Figure. 8, our experiments include a detailed analysis of the generation data ratio, where we examine the impact of varying proportions of synthetic data on model performance. This provides a comprehensive view of how synthetic data supplementation affects defect detection capabilities. We are sorry for the confusion it brought and will adjust it by extending the experiment as we mentioned in the response to reviewer 2(qXne), the results for the extended synthetic data ratio experiment can be found in Appendix Section. B of our updated paper.

Finally, I sincerely appreciate the valuable insights provided by the reviewers and warmly welcome any further criticisms or suggestions. Your timely feedback is crucial for the enhancement of this paper. I look forward to your valuable comments.

Thanks for your response. My concens are not well addressed.

• The publication of captions and synthetic data. The authors only publish the refined annotation about 4 public datasets. The most important captions and synthetic data are totally unavailable. The method for data generation is the most important part for this paper. However, the solution and synthetic data are not well clarified.
• The source code only provides the diffusion generator for configuration file of cotton fabric dataset. The configurations for other datasets are unavailable.
• The important details about auxiliary annotation tool and diffusion generator for defect image generation are not clearly clarified.
• The visualization results cannot effectively support the superority of the proposed dataset. I would like to see more quantitative results.
• I still think the results of VISION dataset in Tab.4 is somewhat trival.

Hence, I keep the previous score.

Dear reviewer VeNd,

As the window for reviewer-author interaction is closing soon, on November 21st, I wanted to extend my sincerest gratitude for the invaluable time and effort you have dedicated to reviewing our work. To ensure that we have met your expectations, may I kindly ask if you find our responses satisfactory and if there are any remaining issues that need further clarification or improvement?

Thank you for your insightful query regarding our paper, we will address your concerns one by one:

Concerns on the quantitative evaluation result: The concern about the similar average performance of various methods across different datasets, despite their varying performance on individual defect image types, is a valid point. However, if we look at the mean IOU report in Table 3, the min and max MIOU reported are 45.40 and 51.58. The segmentation results on the CityScapes dataset across various baselines also demonstrated similar MIOU results between 76.9 to 81.6, except for the traditional UNet(69.1). As for the varying performance on individual classes, this is due to the different granularity that exists in different defect types, making some of the defective regions comparatively smaller than other types.

Extended experiments on other baselines: The decision to select the best baseline aims to demonstrate Defect-Gen's capability in the most demanding scenarios, particularly where baseline performance is already high. This approach highlights the robustness and effectiveness of our method, even under challenging conditions. To further underscore the efficacy of Defect-Gen, we conducted comprehensive experiments using three distinct segmentation models: Mask2Former, MIT-B0, and DeepLabV3+, as detailed in the table below (also see Appendix Section A in the updated paper). The results from these models reinforce our findings, demonstrating that Defect-Gen not only excels with top-performing methods but also significantly enhances the performance of less advanced models. This broad applicability underscores the versatility and practical utility of our approach.

Performance (mIoU) comparison between models trained with and without synthetic data. The bolded text indicates results with synthetic data.

The reason to choose MVTEC, VISION, DAGM, and Cotton datasets: MVTEC Dataset: Renowned for its high quality, the MVTEC dataset is a well-established benchmark in the field. Its high-resolution images and large-scale nature provide a robust testbed for models, enabling detailed defect detection.

VISION Dataset: as the largest scale dataset among those we evaluated, VISION has high variance and high-resolution images. This dataset challenges models with a wide array of defect types and imaging conditions, making it an excellent benchmark for assessing a model's versatility.

DAGM Dataset: Unique in its use of special imaging techniques, DAGM provides images that are far from natural representations. This characteristic is crucial for testing a model's ability to generalize and perform in less conventional or more challenging imaging environments.

Cotton Dataset: As a real-world industry dataset produced directly in factory settings, the Cotton dataset offers high variance and practical relevance. It simulates real-world industrial conditions, thereby providing a realistic assessment of a model's performance in actual deployment scenarios.

Finally, I sincerely appreciate the valuable insights provided by the reviewers and warmly welcome any further criticisms or suggestions. Your timely feedback is crucial for the enhancement of this paper. I look forward to your valuable comments.

Dear reviewer J4Kk,

As the window for reviewer-author interaction is closing soon, on November 21st, I wanted to extend my sincerest gratitude for the invaluable time and effort you have dedicated to reviewing our work. To ensure that we have met your expectations, may I kindly ask if you find our responses satisfactory and if there are any remaining issues that need further clarification or improvement?

Thank you for your insightful observations. We address each of your concerns as follows:

Importance of Image-Text Pair Datasets: Image-text pair datasets are crucial for training Multimodal Large Language Models(MLLMs), which have shown their dominant capabilities in a large range of tasks. Our descriptive captions offer additional contextual information that can be leveraged by these models. We look forward to empowering MLLMs with the ability of defect inspection through this effort.

Extended Synthetic Data Ratio Experiments: To address your concerns, we have extended our experiments to include scenarios with higher ratios of synthetic data, shown in the Appendix Section. B(Please refer to the updated paper). Specifically, we have now included results for systems trained with 200% and 300% synthetic data ratios. When using synthetic data that is 200% of the size of the original training set, there is an enhancement in the performance, but results in greater variance. Additionally, the performance starts to decrease after reaching 300%. This is due to the overly-used synthetic data disrupting the original data distribution.

Sampling Strategy: As an immediate response to your suggestion, we conduct an experiment where we employ a uniform sample strategy – using a fixed number (e.g., 150) of samples in the training set for each object. So those with limited data gain more synthetic data and vice versa. As shown in the table below (also see Appendix Section. C), with this sampling strategy, baselines such as MiT-B0 show a great leap in performance.

Finally, I sincerely appreciate the valuable insights provided by the reviewers and warmly welcome any further criticisms or suggestions. Your timely feedback is crucial for the enhancement of this paper. I look forward to your valuable comments.

Dear reviewer qXne,

As the window for reviewer-author interaction is closing soon, on November 21st, I wanted to extend my sincerest gratitude for the invaluable time and effort you have dedicated to reviewing our work. To ensure that we have met your expectations, may I kindly ask if you find our responses satisfactory and if there are any remaining issues that need further clarification or improvement?