Response to reviewer sEzZ

We thank the reviewer for the feedback and constructive suggestions. Our response to the reviewer’s concerns is below:

Weakness 1: How can we ensure that during the generation from 0 to c, the original ID sample of class y is not mistakenly perturbed into another ID class? Simply controlling the step size α and number of steps c may make the model overly sensitive to hyperparameters.

We appreciate this important inquiry regarding hyperparameters. We would like to emphasize that OOD features are the features that distributed between classes, so they are ambiguous and likely to confuse the classifier. But if BOOD generates features that are distributed in the ID area, the final OOD image will be considered as noisy data and be harmful to the model performance. While we don't have an explicit way to measure whether image features have been mistakenly perturbed into another ID class, by setting relatively small $\alpha$ and $c$ values, we can guarantee that the generated OOD features will located around the decision boundaries. Figure 5 shows that the performance of OOD detection rises firstly and latter decrases as $c$ increases, which illustrates our control of $c$ are effective and most of the generated OOD images are effective. In the next step, we are working on filtering mechanism to rule out the potential mistakenly perturbed features.

Weakness 2: I would like to understand whether the performance improvement over the DreamOOD method arises from generating OOD data with lower error rates, or from a higher number of OOD samples being situated closer to the decision boundary.

Thank you for the interesting point. Would you clarify the meaning of "lower error rates" in the context?

Assume the "lower error rates" indicates the rate of mistakenly generated ID images in the generated OOD data. Since OOD data are the samples that are not distributed in the input data distributions, thus they are hard to be classified by the classifier and difficult to decide whether they are OOD samples. Our method intend to find the boundaries of ID data and perturb ID features which are cloestest to the boundary to the area between classes and close to the decision boundaries in the latent space, thus it's diffiucult to quantify the error rate of the generated OOD data.

We also conduct the experiment comparing the feature's average distance to their closest decision boundaries between DreamOOD[1] and our method:

The results show that BOOD's generated features are closer to the closest decision boundaries, resulting in the improvement of performance in OOD detection task.

Weakness 3: I have noticed that 100,000 generated OOD images were used during training. Would reducing or increasing the number of images have an impact on the ID accuracy or OOD metrics?

Great point! Here we list the performance statistics of BOOD and DreamOOD[1] training on different number of OOD images (use CIFAR-100 as ID dataset):

From the table above we can find that as the number of OOD training images increases, both BOOD and DreamOOD's[1] performance on OOD detection and ID classification increases.

We hope we have responded to all your concerns. If you have further questions, we are pleased to discuss with you. Thank you again for taking the time to read our response and your constructive feedback!

[1] Xuefeng Du, Yiyou Sun, Xiaojin Zhu and Yixuan Li. Dream the impossible-Outlier imagination with diffusion models. NeurIPS, 2023

Response to reviewer Mr32

We appreciate the review for providing valuable advice. Below are our responses:

Claims and Evidence 1: there is no clear evidence showing that the method is truly efficient

We apologize for the inaccurate expression in the abstract: BOOD provides a more training efficient strategy for synthesizing informative OOD features. We will fix this part in the updated manuscript, thank you again for pointinng this out.

Claims and Evidence 2: need to provide a visual example showing that the feature indeed returns to the ID region.

Thank you for your advice. Please check Figure 2 in our paper, which shows the perturbation process of two features from one ID class to another ID class: one from Tiger to Fish, another one from Streetcar to House. We will also include more visual examples illustrating the perturbation process.

Some essential References Not Discussed.

We extend our gratitude to the reviewer for the recommendation of essential references. We will update the corresponding part in the related works of paper in the camera-ready version.

Weakness 1: The third step of the method (the OOD image generation) is based on the approach in Du et al. (2023), while the regularization technique is derived from VOS (Du et al., 2022). These elements are largely borrowed from prior works and integrated into the BOOD framework. This raises concerns about the novelty of the method.

We thank the reviewer for the concern regarding the novelty. We would like to emphasize that our framework mainly focus on the generation of imformative OOD features in the latent space. To guarantee the fairness in comparison with the previous SOTA method DreamOOD in synthesis-based OOD detection frameworks, we follow the same regularization function as them. After generating the OOD dataset, the regularization function can be substitute by other common functions to train the OOD detection model.

Weakness 2: Code not provided.

Thanks for concern regarding the code. We will release our code, data and model once the paper is accepted.

Comments: In the related work section, use ‘citet’ instead of ‘citep’ for “…guidance. (Dunlap et al., 2023) proposes to caption the images…” and “…model. (Li et al., 2024) generated augmented images with the guidance…”.

Thanks for pointing out the minor mistakes, we will update them in the revised manuscript.

Question: Why is regularization used in the OOD detection model in Section 3.3? What is the significance of this regularization, and what is the underlying intuition? It would be helpful to mention this in this paragraph. Moreover, why does setting such a large value of $\beta$ help; what could be a potential rationale?

The significance for regularization the OOD detection model is to keep model's ability of performing visual recognition task while regularizing the model to identify the generated OOD images. Equation (7) shapes the uncertainty surface, which predicts high probability for ID data and low probability for OOD data. Compared to other regularization method such as Energy-bounded[1], Equation (7) is hyperparameter-free so easier to be implemented.

Our analysis suggest setting a relatively larger regularization weighting $\beta$ to ensure better performance (Figure 6 middle). The potential reationale could be: using a relatively larger $\beta$ will help to force the model to push OOD samples away thus enhance the model's ablility for distinguishing ID and OOD input. However, if the $\beta$ is too large, the model will be over-regularized and the performance for OOD detection will decrease.

We sincerely thank you for reviewing our response and your constructive feedback. If any aspects still need clarification, we are happy to discuss them further.

[1] Weitang Liu, Xiaoyun Wang, John Owens and Yixuan Li. Energy-based out-of-distribution detection. NeurIPS, 2020

Response to reviewer Smhu

Essential References Not Discussed.

Thanks for recommendation for essential references. We will add them into our camera-ready paper.

Weakness 1: the process of tuning hyperparameters may be challenging.

We appreciate the reviewer for the meaningful concerning. While fine tuning the parameter $c$ and $\alpha$ may be somehow time-consuming, we would like to emphasize that $r$, $\beta$ and $K$'s effect are not remarkable (Figure 6): the difference in AUROC are within 0.5% when $r < 10$ and within 0.4% for $\beta \in \{1.5, 2, 2.5, 3\}$, the performance will not change if $K > 50$. Thus, our suggestions for tuning the hyperparameters can make the process of tuning less difficult. We are working on a possible automatic adaptive method to adjust the additional pertuabation steps $c$ in the next step, aiming to reduce the tuning time.

Weakness 2: BOOD's performance on ImageNet-200, CIFAR-10 and NearOOD datasets.

Since our baseline method DreamOOD[1] does not include the performance on ImageNet-200 and CIFAR-10, we do not choose them as the ID dataset to guarantee comparison fairness. Due to the limited author response time, we cannot present the result of these two dataset, but we will include them in our camera-ready paper.

We summerize the experiment results on two selected NearOOD datasets NINCO and SSB-hard as below:

From the table above we can see that in the NearOOD dataset, BOOD shows better performance than DreamOOD[1], thus indicating that BOOD can handle more complex OOD datasets.

Question 1: importance of class embedding alignment strategy, capability between feature space and the diffusion model.

We appreciate the reviewer for the reasonable concern. We aim to create a latent space that is compatible for the input space of diffusion model, so the alignment between the image features and their corresponding class token embeddings is necessary. By training an image encoder with Formula (2), we ensure that the image features can be decoded by the diffusion models. Please check Section 3.1 in our paper for detailed explanation.

Question 2: hyperparameter $\alpha$ and $K$'s sensitivity on accuracy of identifying features closest to the decision boundary.

While it's difficult to directly measure the accuracy, from our experiment on ablation of OOD feature synthesizing methodologies (Sec 4.3.1 and Table 3), we can find that perturbing random ID features to the boundaries will significantly decrease the OOD detection performance compared to using the ID features closest to the decision boundary. Therefore, we can measure the accuracy of identifying features closest to the decision boundary by the final OOD detection performance.

Employing a relatively small $\alpha$ facilitates a more nuanced differentiation between samples, and a large $\alpha$ may lead to large discrepancy between each iteration in adversarial perturbation, making the counting of distance to decision boundaries not accurate. Selecting a relatively large max iteration number $K$ can ensure comprehensive boundary crossing for most features. While increased $K$ do affect computational overhead in boundary identification, the impact remains manageable. We provide detailed discussion about the sensitivity of the hyperparameters $\alpha$ and $K$ in Section 4.3.2, Figure 5 (left) and Figure 6 (right).

Question 3: backbone architecture and diffusion model's impact on BOOD's performance.

We provide comparison between BOOD's performance on Resnet-18, Resnet-34 and Resnet-50 using CIFAR-100 as ID dataset:

From the table above we can see that the selection of backbone architecture will not significantly affect the performance of BOOD.

For some datasets that has large domain discrepancy from the diffusion model's training distribution (Textures), BOOD's performance might be affected by the diffusion model's capability. But this constraint is inherent to all methodologies utilizing diffusion-based generative data augmentation. While future developments in generative modeling may address these limitations, we emphasize that our primary goal is to leverage diffusion models to generate informative OOD images thus increasing the OOD detection model's performance.

We believe we have responded to all your concerns. If anything remains unclear, we would be pleased to have further discussions with you.

[1] Xuefeng Du, Yiyou Sun, Xiaojin Zhu and Yixuan Li. Dream the impossible-Outlier imagination with diffusion models. NeurIPS, 2023

Response to reviewer wxMm

We thank the reviewer for the comments.

Weakness 1: The compared methods are 2023 and before, how about the performance regarding to the most recent works?

Thank you for your concern regarding baseline methods. Please note that our work is based on using diffusion models to generate image-level OOD datasets. However, there are few work in this specific area. We provide comparison between BOOD and a SOTA methods FodFom[1] from ACMMM 2024, which also harnesses diffusion model to generate OOD images for enhancing OOD detection model. The results are summarized in the table below:

Compare to FodFom[1], BOOD demonstrates superior performance, illustrating its competitiveness.

Weakness 2: The discussion of OOD methods misses most works on 2024.

Thanks for your question related to the discussion on recent OOD methods. We will include them into our camera-ready version.

Weakness 3: The complexity analysis in appendix should be included in the main text, as the timing of diffusion-based methods is always an interested point for researchers in the community. Besides, the details about the complexity analysis are unclear. For instance, what the dataset used for Table 5, how about the timing when generating samples one by one? Does the memory requirements in Table 6 are for training? How about the inference?

Thanks for pointing out the this. We will include the detailed complexity analysis in the main text in our revised manuscript.

Regarding the dataset used in Table 5, we use CIFAR-100 as the ID dataset. It takes around 4 second to generate a single image in our framework on Stable Diffusion v1.4 with one NIVDIA L40S GPU. And for Table 6, the memory requirements indicates the space needed for storing the generated OOD images used for training the final OOD detection model. There's no extra memory requirements during the inference time.

Weakness 4: There are many hyper-parameters. Although the paper provided curves of using different values, setting them for a new dataset is still not an easy job. This could be a disadvantage of this method for practical applications.

We appreciate the reviewer for the meaningful concerning. While fine tuning the parameter $c$ and $\alpha$ may be somehow time-consuming, we would like to emphasize that $r$, $\beta$ and $K$'s effect are not remarkable (Figure 6): the difference in AUROC are within 0.5% when $r < 10$ and within 0.4% for $\beta \in \{1.5, 2, 2.5, 3\}$, the performance will not change if $K > 50$. Thus, our suggestions for tuning the hyperparameters can make the process of tuning less difficult. We are working on a possible automatic adaptive method to adjust the additional pertuabation steps $c$ in the next step, aiming to reduce the tuning time.

Weakness 5: The quality of figures can be improved. The size of tables in the appendix can be adjusted.

Thank you for your advice, we will polish all the figures and tables in our camera-ready version of paper.

We thank the reviewer for taking the time to read our response and the positive feedback again. If you have any further concerns, we are willing to have discussions with you.

[1] Jiankang Chen, Ling Deng, Zhiyong Gan, Wei-Shi Zheng and Ruixuan Wang. "FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detector." ACMMM 2024.