Diffusion-based Layer-wise Semantic Reconstruction for Unsupervised Out-of-Distribution Detection

1. Novelty of this paper

Thanks for the valuable comments. I believe the reviewer may misunderstand our contributions on OOD detection by stating that “It looks like it simply replaces a generative model with diffusion model”. Our main novelty lies in the following three aspects:

Firstly, we are the first to successfully incorporate generative modeling of features within the framework of OOD detection in image classification tasks. This demonstrates that when using diffusion models for OOD detection in such downstream tasks, it is not necessarily required to operate in the pixel space.

Secondly, we devise a multi-layer semantic feature reconstruction mechanism. Performing feature reconstruction on top of the multi-layer semantic features, encourages to restrict the in-distribution latent features distributed more compactly within a certain space, so as to better rebuild in-distribution samples while not reconstructing OOD comparatively. As a result, the projected in-distribution latent feature space should be compressed sufficiently to capture the exclusive characteristics of ID images, while it also provides sufficient reconstruction power for the large-scale ID images of various categories with the high-level semantic features.

Thirdly, the proposed Latent Feature Diffusion Network (LFDN) is built on top of the feature level instead of the traditional pixel level, which could significantly improve the computation efficiency and achieve effective OOD detection.

2. More explanations and theoretical analysis may need.

Thanks for the valuable comments. We will improve these descriptions in the camera-ready version. As illustrated in Figure 1, the proposed framework consists of the following three components:

(1)The multi-layer semantic feature extraction module, which sets up a comprehensive and discriminative feature representation for each image, and could help to better rebuild the samples and encourage the in-distribution features distributed more compactly within a certain space from different semantic layers.

(2) The latent feature diffusion stage, which introduces the DDPM to model the multi-layer semantic features. It builds a latent feature diffusion network to reconstruct the semantic features from their distorted counterparts.

(3) The OOD detection head, which utilizes different evaluation metrics (i.e., MSE, MFsim and LR metric), to measure the reconstruction error. Finally, we can use the reconstruction error to justify whether the input sample belongs to ID or OOD.

The success of our method lies in that, the projected in-distribution latent feature space can be compressed sufficiently to capture the exclusive characteristics of ID images, while it also provides sufficient reconstruction power for the large-scale ID images of various categories.

Besides, to further illustrate the effectiveness of the proposed method, we also compare the MFsim score distributions between the initial model and the final model after training. We can clearly see that the reconstruction errors of the ID data tend to decreasing as the model training, and the final scores can be distinguished from the OOD data.

3. In addition to the AUROC and FPR95, other metrics that could capture the imbalance characteristics should be used.

Besides the commonly used AUROC and FPR95 evaluation metrics, we also adopt the F1-score and AUPRC evaluation metrics which could capture the imbalance characteristics of the dataset, for comprehensive experimental comparison. As shown in the table below, we compare the proposed method with some of the latest representative Classification-based methods under the F1-score and AUPRC evaluation metrics, respectively.

Comparison of F1-Score

We provide the average AUPRC values across six different datasets in Table I of the PDF attached in the global rebuttal.

We can clearly see that our method also obtains the best performances under all the settings.

Besides, to analyze the effect of the imbalanced dataset, we design different ratios of ID and OOD data samples for evaluations. As shown in the table below, the detection accuracy tends to decreasing as the sample ratio (ID: OOD) gets larger.

4. What would happen if the difference between ID and OOD is not so significant?

To analyze the effect of the difference between ID and OOD on the model performance, we also artificially design the following two experiments:

1. The ID dataset is from CIFAR10, and the OOD dataset is CIFAR100 which shares similar distributions with CIFAR10.

The results are in Table III of the PDF.

2. Both the ID and OOD datasets are CIFAR10, while the OOD dataset is CIFAR10 by adding random Gaussian noise.

The results are as follows:

We can clearly see that our method is superior to other methods.

Dear Reviewer Jgzd,

Thank you very much for reviewing our paper and providing valuable feedback. We have made the best effort to address all your questions and comments. In particular, we have further clarified the rationale behind our proposed method and provided additional points to highlight its novelty. We have also introduced several metrics that can capture the characteristics of imbalanced datasets. Additionally, we demonstrated the detection performance of our method in scenarios where the ID and OOD datasets are similar, showcasing the scalability of our approach.

We sincerely hope that our responses can address all your concerns. Is there anything that needs us to further clarify for the given concerns?

Thank you again for your hard work and thoughtful review.

Dear Reviewer Jgzd,

Thank you very much for reviewing our paper and providing valuable feedback. We sincerely hope that our responses can address all the remaining concerns. Thank you again for your great help and many good questions and suggestions, which largely help improve the quality of our paper. We would like to clarify if you have further concerns.

We would like to clarify if you have further concerns. Thanks very much.

Q1. Comparison against other methods using the multi-scale feature encodings as the input.  

A: We have made comparison of our method against AE and VAE using the multi-layer feature encodings as inputs. 

For AE (AutoEncoder), we use the LFDN network without the timestep embedding, i.e., a 16-layer linear network. For VAE, we use a 5-layer linear network as the encoder and an 8-layer linear network as the decoder.  

Compared to AE and VAE, the diffusion model has significant advantages when modeling complex multidimensional distributions.  

Q2. Speed comparison with classification or distance-based methods. 

A: The speed comparison between classification-based and distance-based methods is presented below. All experiments were conducted on an NVIDIA Geforce 4090 GPU. 

The inference speed of our method based on MSE or MFsim is faster than that of distance-based methods SimCLR+Maha and SimCLR+KNN, because the computation of covariance matrix or K nearest neighbors occupies part of time. Our method is also comparable to classifier-based methods including MSP, EBO, DICE and ASH-S. 

Q3. Comparison using the same image encoder as backbone. 

A: The experimental results based on classification and distance methods are taken from the best results in the original papers using their optimal backbones.

We also conducted experiments to compare our method with other methods using the same image encoder. The FPR95 and AUROC metrics for methods using EfficientNet-b4 as the backbone are as follows.

We provide the average FPR95 and AUROC values across six OOD datasets: SVHN, LSUN-c, LSUN-r, iSUN, Textures, and Places365.

The average FPR95 and AUROC metrics of methods using ResNet-50 as the backbone are provided in Table II of the PDF file attached in the global rebuttal.

Q4. Qualitative results. 

A: We have included three types of failure cases in the PDF.

The first type, shown in Figure B, represents ID samples misclassified as OOD. It can be observed that these misclassified samples often have significant shadows and lack semantic information, resulting in high reconstruction errors and being incorrectly classified as OOD samples.

The second type, shown in Figure C, represents OOD samples misclassified as ID. It can be observed that these OOD samples have categories very similar to those of the ID samples (CIFAR-10), such as cars and ships, which are categories present in CIFAR-10.

The third type, shown in Figure D, represents OOD samples with colors very similar to the ID samples, leading to their misclassification as ID.

Q5. Explanation to more compact semantic features.

A: Due to the higher dimensionality and complexity of the feature distribution at the pixel level, our proposed multi-scale feature fusion and compression strategy significantly reduces the feature dimensions and makes the distribution more compact. This allows the sample reconstruction process to focus more on the primary features with inter-class discriminative power, rather than secondary details, thereby achieving the goal of distinguishing between ID and OOD samples.

Q6. AUPRC Measurement 

A:  We use the AUPRC metric to evaluate OOD detection methods based on the EfficientNet-b4 backbone model. We provide the average AUPRC values across six different datasets in Table I of the PDF. Our method achieves better performance than existing methods as well. 

Q7. Experiments on more challenging datasets. 

We provide experiments on more challenging datasets using EfficientNet-b4 as the encoder.

(1) We use CIFAR-10 as in-distribution data and CIFAR-100 as OOD data, which has relatively high similarity with CIFAR-10. The results are shown in Table III of the PDF.

(2) We also test OOD detection methods in experiments where ImageNet100 is regarded as the ID dataset and SUN, iNaturalist, Textures, and Places365 are used as OOD datasets. The results are shown in Table IV of the PDF.

Our method still shows significant advantages over the baselines, which demonstrates the scalability of our approach.

Q8. Potential negative social impact have not been mentioned. 

A: We have discussed potential negative social impact in the appendix of our paper: As with any advanced detection method, there is a risk that the technology could be misused. For instance, surveillance applications, it could be employed to monitor individuals without their consent, leading to privacy violations and ethical concerns.

Q1.at the core, the idea is that reconstruction error points to OOD. This is new, but is in a broader family (including AE, contrastive leraning), thus some limited.

A: Thanks for the insightful comment. We agree that the core idea of using reconstruction error to indicate OOD is indeed related to broader methodologies, including Autoencoders (AEs) and contrastive learning. However, we believe our approach introduces a novel perspective and significant innovations in this field. Our overall framework is innovative in that it applies diffusion models to multi-scale feature layers, establishing reconstruction error from this unique vantage point. This new approach attempts to leverage the strengths of diffusion models in capturing and reconstructing complex data distributions, setting it apart from traditional AE and contrastive learning methods. We appreciate the recognition of the novelty within the broader context and are confident that our contribution offers valuable advancements to the OOD detection domain. 

  Additionally, we compared the performance of AE and Diffusion for multi-scale feature modeling to observe their performance differences. For AE (AutoEncoder), we use the LFDN network without the timestep embedding, i.e., a 16-layer linear network. The results are as follows:

This further demonstrates the advantage of using diffusion for feature modeling.

Q2.the limitation sections points only to the strength of the encoder. However the assumption that OOD samples lead to large reconstruction error is also a prior assumption (although a strong one given the evaluation), but one may identify cases where is not true.

A: Thanks for the valuable feedback. We acknowledge the point regarding the prior assumption that OOD samples lead to large reconstruction errors. While this is generally a prior  assumption supported by our evaluation, there are indeed scenarios where it may not hold true. Some difficult in-distribution (ID) samples may exhibit large reconstruction errors, and certain OOD samples similar to ID samples may have smaller reconstruction errors. The reconstruction errors for ID and OOD samples after training, as shown in Figure 3, also indicate this point. There is a small overlap between the reconstruction errors of some ID and OOD samples.

We also considered the detection performance of our method when the reconstruction errors are quite close. For example, the ID dataset is from CIFAR10, and the OOD dataset is CIFAR100 which shares similar distributions with CIFAR10. Below are our results:

CIFAR-10 as ID and CIFAR-100 as OOD:

CIFAR-10 as ID and noisy CIFAR-10 as OOD:

It can be observed that our method still achieves the best performance in these two specific scenarios. This validates the reasonableness of this prior assumption.

Q1. Writeup revision.  

A: Thank you for your valuable feedback. We appreciate your observation regarding the repetition of the contributions and the main idea of the paper. We will address these issues to ensure clarity and conciseness. 

Q2. Clarification of OOD detection step. 

A: During inference, samples having higher reconstruction errors measured by MSE, MFsim, or LR  are more likely to be OOD samples.  FPR95 indicates the false positive rate when the true positive rate reaches 95%. When calculating FPR95, the threshold is determined by the measurement value where the true positive rate (TPR) reaches 95%. AUROC measures the area under the ROC curve. When calculating this metric, we vary the true positive rate from xxx to xxx in step of xxx to choose the threshold. In practical usage, we can select the threshold value with the OTSU algorithm. We report the F-scores of different methods using OTSU to determine the threshold value as follows: 

Q3. Comparisons with recent generative methods based OOD 

A:  The comparison between our method and DDPM [Graham et al., 2023] can be referred to Table 1 and 2 of our paper. Our method outperforms DDPM consistently on benchmarks using CIFAR10 or CelebA as ID data.  

The comparison between our method and Diffuard [Gao et al., 2023] is provided in the table below. Results of Diffuard are taken from its original paper. Here, CIFAR10 is regarded as ID data, while CIFAR100 or TinyImagenet is regarded as OOD data. Our method based on MFsim achieves overall better performance than ‘Diffuard+Deep Ens’, with 1.55 higher AUROC and 21.77 lower FPR95.

The comparison between our method and LMD [Liu et al., 2023] is shown in the following table. The evaluation metric is AUROC. The average AUROC of our method based on MFsim is 6.94 higher than that of LMD.

Q4. Comparisons with pixel-level denoising approaches.

A: The quantitative comparisons between our method and pixel-level approaches are as follows: 

We provide the distribution differences of the MSE scores at two levels after training, with CIFAR-10 as the ID dataset and other datasets as OOD; The results are shown in Figure A of the PDF attached in the global rebuttal.

It can be observed that at the pixel level, the reconstruction error distributions of ID and OOD samples are very similar. The mixed MSE scores make it very hard to distinguish ID samples from OOD samples. However, at the feature level, the reconstruction score distribution of ID samples shows a clear distinction from that of OOD samples. The reason is that, our feature-level diffusion-based generative model makes the projected in-distribution latent space not only be compressed sufficiently to capture the exclusive characteristics of ID images, but also provide sufficient reconstruction power for the large-scale ID images of various categories.

Q5. Focusing on Classification Problems

A: Thanks for the valuable comment. Our experiments, along with most of the cited works, focus on classification problems. This focus is largely because the field of OOD detection has traditionally concentrated on classification tasks. However, we believe our method has the potential to extend beyond classification. Our approach can serve as an effective data pre-processing step for more complex tasks like semantic segmentation and object detection. By identifying and filtering out OOD inputs, our method can enhance data security and improve the performance of these models. Additionally, our method can be integrated into semantic segmentation and object detection pipelines to judge whether detected or segmented object proposals are out of distribution.

Q6. Effect of generalisation on model performance.

A generalised backbone model is crucial for ensuring the extracted features are comprehensive and representative of the samples. When modeling in multi-scale feature spaces, it's essential that the features extracted are thorough and meaningful. Small-scale models may have shallower layers, which might not fully capture the complexity of a sample. The generalization ability of large models allows them to extract a more complete and representative set of features, which is essential for the effective performance of our OOD detection method. This ensures that the model can adequately generalize across various inputs, leading to a more robust and reliable detection system.