BOOD: Boundary-based Out-Of-Distribution Data Generation

Question 2: performance gap between NPOS and BOOD on OOD dataset Textures using ImageNet-100 as ID dataset.

Textures is a dataset containing textural images in the wild, which has large discrepancy from the training distribution of Stable Diffusion. NPOS [2] is an OOD detection framework that leverages outlier features synthesized from low-likelihood area in the latent feature space. We do found the performance gap between NPOS [2] and BOOD on Textures while choosing ImageNet-100 as ID dataset. Compared to NPOS, we argue that the performance gap between BOOD and NPOS [2] is because the Stable Diffusion lacks the ability to generate images that located near Textures' distritbution while using ImageNet-100 as ID dataset. But we want to emphasize that: for the area of generative data augmentation, the performance of the frameworks are limited by the capability of diffusion models. In the following table, we select NPOS [2], BOOD and another generative OOD data augmentation framework DreamOOD [1] for comparison. The three methods are all tested on Textures, using ImageNet-100 as ID dataset.

From the table above, we can discovery that both DreamOOD [1] and BOOD have performance gaps between NPOS [2] on Textures, indicating that the performance of generative data augmentation is bounded by the capability of diffusion model. However, BOOD shows superior performance in the average OOD dection results, indicating our framework is promising. This performance gap might be narrowed with the stronger diffusion models in the future study.

Weakness 3: architectures of the image encoder and the OOD classification model.

This is a good point regarding reproducibility. To guarantee the fairness between frameworks for comparison, we choose the same architectures of image encoder and OOD classification model as DreamOOD [1]. We summerize the architectures of the models as below:

• Image encoder. We employed a ResNet-34 architecture for image encoder for both CIFAR-100 and ImageNet-100. Here's the breakdown:

  Input Layer:

  • Initial Conv2d: 3→64 channels, 3×3 kernel, stride=1, padding=1
  • BatchNorm2d
  • ReLU

  Main Blocks (using BasicBlock structure):

  • Layer1: 64→64 channels, 3 blocks
  • Layer2: 64→128 channels, 4 blocks, stride=2 at first block
  • Layer3: 128→256 channels, 6 blocks, stride=2 at first block
  • Layer4: 256→512 channels, 3 blocks, stride=2 at first block

  Final Layers:

  • Adaptive Average Pooling to (1,1)
  • Flatten
  • Linear transformation: 512→768 dimensions (768-dimensions features are aligned with the class token embeddings $\Gamma(y)$)

  Each BasicBlock contains:

  • Conv2d (3×3) → BatchNorm2d → ReLU
  • Conv2d (3×3) → BatchNorm2d
  • Skip connection (with optional 1×1 conv if dimensions change)
  • Final ReLU

• OOD classification model. We also employed a ResNet-34 architecture for OOD classification model. Most part of the architecture are same as the image encoder, except the final layer: the Linear transformation changed from 512→768 to 512→100 (the number of classes equals 100).

We have included the above architecture explanations in our updated version of paper, please check it, thanks!

Weakness 4: error in equation

We apologize for the typo. We have corrected this in our updated version of paper, please check it.

We trust our responses have adequately resolved your concerns. We sincerely thank you for reviewing our rebuttal and your constructive feedback. If any aspects still need clarification, we are happy to discuss them further.

[1] Xuefeng Du, Yiyou Sun, Jerry Zhu and Yixuan Li. "Dream the Impossible: Outlier Imagination with Diffusion Models." NIPS 2024.

[2] Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. "Non-parametric outlier synthesis." ICLR 2023.

We appreciate the review for providing valuable advice. Below are our responses:

Weakness 1: the perturbation strategy may unintenrionally transform the features into in-distribution classes.

Thank you for providing a reasonable concern. To prevent the generated OOD features from transforming into the distribution of other in-distribution classes, we set a relatively small step size $\alpha$ for minor perturbations in each iteration when synthesizing OOD features, which guarantees small deviations and prevents the synthesized features from entering other distributions. We also employ small additional perturbation steps $c$ to guarantee the synthesized OOD features will not step into other distributions after crossing the decision boundaries.

To further provide the empirical evidence to support our theory, we attach the performance of BOOD on CIFAR-100 with a larger range of $\alpha$ and $c$:

From the results above and Figure 4 in the paper, we can conclude that our perturbing strategies are promising. We also include the tables above in our updated version of paper.

Weakness 2: ablation studies on alternative perturbation strategies.

Our proposed strategy perturbs the identified ID boundary features through the direction of gradient ascent, aiming to synthesize OOD features that distributed around the decision boundaries. To gain a deeper insight of the effectiveness of our strategy, we provide additional ablation studies on the different perturbation strategies, including (1) adding Gaussian noises to the latent features, (2) displacing features away from class centriods and (3) BOOD's perturbation strategy. Below are the results:

From the statistics above, we can conclude that our perturbation strategies are solid. We also include this analysis in the updated version of summision. We will discover more perturbation strategies in our camera-ready version of submission.

Question 1: the sensitivity of the method to the hyperparameter r.

This is a great point. For pruning rate $r$, we suggest a relatively mild rate since too small $r$ will not have enough features to generate diverse OOD images, and a large $r$ may lead to selection of ID features that distribute far from the boundaries. Below are BOOD's performance with different $r$ on CIFAR-100:

We also attached the table above in the updated paper, please check it!

Dear Reviewer v1ZR:

The six aspects covered in our previous response directly address the concerns you raised: (1) we provide explanations regarding why our perturbation strategy will not transform the features into ID classes, (2) we provide ablation studies on the different perturbation trategies, (3) we provide sensitivity analysis of hyperparameter $r$, (4) we explain the performance gap between NPOS and BOOD on Textures using ImageNet-100 as ID dataset, (5) we introduce the architectures of the image encoder and OOD classification model, and (6) we fix the error in equation 2.

With the discussion session closing soon, we want to ensure we've adequately addressed all your questions. We truly appreciate your constructive comments throughout this process.

Best,

Authors

We thank the reviewer for the feedback and constructive suggestions. Our response to the reviewer’s concerns is below:

Weakness 1: notation conflict between the additional perturbation steps c and the earlier use of C for the number of classes.

Thank you for pointing out the ambiguous notation. We have changed the notation for the number of classes to $V$. Please check L104 and L107 our updated version of paper.

Weakness 2: including more comparisons between BOOD and SOTA methods from 2024.

We provide comparison between BOOD and a SOTA methods FodFom [1] from ACMMM 2024. FodFom is a framework that utilizing Stable Diffusion to generate outlier images for OOD detection. The results are summarized in the table below:

Compare to FodFom, BOOD demonstrates superior performance, illustrating its competitiveness.

Weakness 3: experimenting with a wider range of hyperparameter values.

We provide analysis on BOOD's performance with a wider range of hyperparameters as below.

From the results, we conclude that our previous choices are optimal. Employing a smaller step size $\alpha$ facilitates nuanced differentiation between samples across different distances, thus resulting in precise boundary identification. Choosing a moderate $c$ guarantees that the synthesized OOD features are adequately distant from the ID boundaries, and prevents the OOD features from distributed in the in-distribution area. We have uploaded the analyses in our new submission of paper.

Weakness 4: additional visualizations to illustrates the improvements.

Good point! We provide more visualizations in the Appendix B Figure 8 in our updated version of paper. Please check them, thanks!

We're confident that our explanations have fully addressed your concerns. We greatly appreciate your time in reviewing our rebuttal and your feedback. Please don't hesitate to raise any lingering concerns for discussion.

[1] Jiankang Chen, Ling Deng, Zhiyong Gan, Wei-Shi Zheng and Ruixuan Wang. "FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detector." ACMMM 2024.

Dear Reviewer dKDk:

We have thoroughly addressed your feedback across four key dimensions in our previous response: (1) we fix the notation conflict for the number of classes, (2) we provide comparison between BOOD and a SOTA method, (3) we provide analysis on BOOD's performance with a wider range of hyperparameters, and (4) we provide additional visualizations in Appendix B in our updated PDF.

As we approach the end of the discussion window, please let us know if our responses have fully addressed your questions or if additional clarification would be helpful. Thank you for your thoughtful feedback.

Best,

Authors

Dear Reviewer dKDk,

As the discussion period draws to a close, we wanted to check if our responses have addressed your concerns adequately. Please let us know if you need any further clarification.

Best regards,

Authors

Question 2: general guidelines for hyperparameter selection.

Our suggested guidelines for hyperparameter selection are summarized as follows:

For pruning rate $r$, we recommend implementing a moderate pruning rate, as insufficient pruning (small $r$) may limit the diversity of generated OOD images (not enough features), while excessive pruning (large $r$) risks selecting ID features proximally distributed to the anchor.

Our analyses suggest selecting a relatively large max iteration number $K$ to ensure comprehensive boundary crossing for most features. While increased iterations do affect computational overhead in boundary identification, the impact remains manageable. We present a detailed computational cost and performance analysis across varying $K$ values:

For regularization weight $\beta$: Empirical evidence suggests optimal performance is achieved with moderate regularization weighting, as excessive OOD regularization can compromise OOD detection efficiency.

For step size $\alpha$: A moderate $\alpha$ value is recommended for boundary features identification and OOD features synthesizing. A large $\alpha$ may lead to large discrepancy between each iteration in adversarial attack, thus making the counting of distance to the decision boundaries not accurate.

For step size $c$: We suggest a moderate $c$ value, since a large $c$ may force the generated OOD features to step into other classes' distributions, and a small $c$ may not guarantee the OOD features that are adequately distant from the ID boundary.

The above statistics illustrates that our suggested guidelines for hyperparameters selection are effective. We have included the new analysis in the Appendix C. Please check our updated summision of paper, thanks!

Question 3: analysis on computation-performance tradeoffs.

This is a reasonable question concerning computational cost. With the increase of number of OOD images per class, the performance of OOD detection becomes better. Below are the performance vs . different number of generated images per class on CIFAR-100:

We also provide the computational cost comparison between BOOD and DreamOOD on CIFAR-100 below. The computational cost are almost no difference.

We have included the comparisons regarding computational cost in Appendix E in our updated version of paper, please check them, thanks!

Weakness 3: several typos in the use of parenthetical vs. textual citations.

Thank you for pointing out the typos! We have already fixed them, please check our updated version of paper.

We feel confident that we have thoroughly addressed the points you raised, and we thank the reviewer for taking the time to read our rebuttal and for your positive feedback again. If you still have concerns, we are willing to have discussion with you.

[1] Hendrycks, Dan, Mantas Mazeika, and Thomas Dietterich. "Deep anomaly detection with outlier exposure." ICLR 2019. [2] Xuefeng Du, Yiyou Sun, Jerry Zhu and Yixuan Li. "Dream the Impossible: Outlier Imagination with Diffusion Models." NIPS 2024.

We thank the reviewer for the thorough review! Our response to the reviewer’s concerns is below:

Weakness 1: The method seems to be bounded by the capability of the stable diffusion model.

We acknowledge this important limitation regarding domains significantly divergent from the diffusion model's training distribution (e.g., SVHN, Textures, and medical imaging). But this constraint is inherent to all methodologies utilizing diffusion-based generative data augmentation. While future developments in generative modeling may address these limitations, we emphasize that our primary goal is to leverage diffusion models to generate informative OOD images, thus increasing the OOD detection model's performance. With two novel strategies: (a) an adversarial perturbation strategy to identify the ID features closest to the decision boundaries precisely, and (b) an OOD feature synthesis strategy to generate outlier features which distributed around the decision bondaries, we successfully achieve the state-of-the-art results on CIFAR-100 and ImageNet-100.

Question 1: Necessity of synthesize OOD data compared to finding publicly available OOD data, and comparison with methods that use real OOD data for augmentation.

Great point! Finding publicly available OOD data as auxiliary OOD detection training data is feasible, but it has two significant drawbacks: (1) It requires significant labor and time cost to labeling and filtering an OOD dataset that is completely not overlapped with the ID data. (2) it's impossible to collect images distributed outside the data distribution boundary, which can not be captured in the real world. BOOD addresses these problems in two aspects: (1) BOOD is an automatically OOD images generation framework, which significantly decrease the human labor involved in creating a new OOD dataset traditionally. (2) BOOD leverages efficient feature perturbation strategies and diffusion model, to generate images that distributed around the decision boundaries, eliminating the issue of not able to collect OOD images that are unreal. To conclude, it's necessary to synthesize OOD data for OOD detection training.

To understand the effectiveness of BOOD, We provide comparison of OOD detection results between BOOD and MSP+OE [1], using CIFAR-100 as ID dataset . Below are the results:

From the table above, we can conclude that our method is more superior than the method using real OOD data for augmentation.

Weakness 2: The performance improvement on ImageNet-100 is only marginal.

We thank the review for providing valuable suggestions. In our research, we found that the performance improvement on ImageNet-100 is not significant as the improvement on CIFAR-100. But our proposed method BOOD still surpasses the state-of-the-art framework DreamOOD [2] by 3.39% (35.37% vs. 38.76%) in FPR95 and 0.42% in AUROC (92.44% vs. 92.02%). We suppose that the performance improvement gap is due to two reasons: (1) the distribution of ImageNet-100 is not optimal as an ID dataset to build a latent feature space, and (2) the number of OOD images we generate for ImageNet-100 is not enough to fit the size of the dataset. We are still discovering the reasons behind this, and we want to express our appreciation to reviewer for giving the useful insight.

Dear Reviewer Fhg5:

Our earlier response systematically addressed your concerns from five different perspectives: (1) we explain the reason of the marginal improvement on ImageNet-100, (2) we explain the reason why the capability bounded by stable diffusion should not be our focus, (3) we explain the necessity of synthesizing OOD data and provide comparison analysis between BOOD and a method using real OOD data for augmentation, (4) we provide general guidelines for hyperparameter selection, and (5) we fix the typo of the use of parenthetical citations.

With the discussion period drawing to a close, we want to ensure our responses have addressed your questions adequately. Thank you again for your valuable feedback.

Best,

Authors

We thank the reviewer for the feedback and constructive suggestions. Our response to the reviewer’s concerns is below:

Weakness 1 and Question 1: BOOD may be time-consuming

We appreciate this important inquiry regarding computational efficiency and resource utilization. It is worth noting that all methodologies in the domain of generative data augmentation utilizing diffusion models necessarily involve image generation processes. Our main research focus is developing automatically utilizing diffusion models to generate OOD dataset and improving the OOD detection performance, where we proposed two key procedures: (a) an adversarial perturbation strategy to identify the ID features closest to the decision boundaries precisely, and (b) an OOD feature synthesis strategy to generate outlier features which distributed around the decision bondaries. Optimizing the generation cost of diffusion models falls out of our research focus.

In our analysis, we conducted a comparative study of computational efficiency between BOOD and DreamOOD [1], which also is a framework regarding generating OOD images through diffusion model. We specifically focus on four key processes: (1) the building of latent space, (2) OOD features synthesizing, (3) the OOD image generation and (4) regularization of OOD detection model. To provide quantitative evidence, we present below a detailed comparison of computational requirements between BOOD and DreamOOD [1] on CIFAR-100:

We also summarize the memory requirements of BOOD and DreamOOD on CIFAR-100 as below:

Our empirical evaluation reveals that the differences between these approaches are not statistically significant. Thus, our proposed framework is not time consuming or has harsh memory requirements.

Weakness 2: need to provide the basis for hyperparameter selection.

We present a detailed analysis of hyperparameter sensitivity, with experiments conducted on the CIFAR-100 dataset. Based on our systematic investigation, we propose the following basis for hyperparameter selection:

For pruning rate $r$, we recommend implementing a moderate pruning rate, as insufficient pruning (small $r$) may limit the diversity of generated OOD images (not enough features), while excessive pruning (large $r$) risks selecting ID features proximally distributed to the anchor.

Our analyses suggest selecting a relatively large max iteration number $K$ to ensure comprehensive boundary crossing for most features. While increased iterations do affect computational overhead in boundary identification, the impact remains manageable. We present a detailed computational cost and performance analysis across varying $K$ values:

For regularization weight $\beta$: Empirical evidence suggests optimal performance is achieved with moderate regularization weighting, as excessive OOD regularization can compromise OOD detection efficiency.

For step size $\alpha$: A moderate $\alpha$ value is recommended for boundary features identification and OOD features synthesizing. A large $\alpha$ may lead to large discrepancy between each iteration in adversarial attack, thus making the counting of distance to the decision boundaries not accurate.

For step size $c$: We suggest a moderate $c$ value, since a large $c$ may force the generated OOD features to step into other classes' distributions, and a small $c$ may not guarantee the OOD features that are adequately distant from the ID boundary.

The above statistics illustrates that our suggested basis for hyperparameters selection are effective. We have included the new analysis in the Appendix C. Please check our updated summision of paper, thanks!

Weakness 3 and Question 2: Further comparative studies on different perturbation strategies.

Our proposed strategy perturbs the identified ID boundary features through the direction of gradient ascent, aiming to synthesize OOD features that distributed around the decision boundaries. To gain a deeper insight of the effectiveness of our strategy, we provide additional ablation studies on the different perturbation strategies, including (1) adding Gaussian noises to the identified latent features, (2) displacing features away from class centroids and (3) BOOD's perturbation strategy. Below are the results:

From the statistics above, we can conclude that our perturbation strategies are solid. We also include this analysis in the updated version of submission. We will discover more perturbation strategies in our camera-ready version of submission.

Weakness 4 and Question 3: Descriptions of the images presented are lacking in the main text.

Thanks, this is a great point. Please check our updated version paper, where we include the descriptions of Figures 2, 3, 4 in the main text in L215, L252, L454 and L461.

We believe that our responses have sufficiently addressed your concerns. If you have further questions, we are pleased to discuss with you. Thank you again for taking the time to read our rebuttal and your constructive feedback!

[1] Xuefeng Du, Yiyou Sun, Jerry Zhu and Yixuan Li. "Dream the Impossible: Outlier Imagination with Diffusion Models." NIPS 2024.

Dear Reviewer YNDo:

In our previous response, we have addressed your concerns in four aspects: (1) we provide computational cost of BOOD to prove that BOOD is not time consuming or requires large memory, (2) we provide bases for hyperparameter selection including $\alpha, \beta, c, K$ and $r$, (3) we provide comparative studies on different perturbation strategies, and (4) we include the descriptions of Figures 2, 3, 4 in the main text of updated PDF.

As the discussion period ends soon, we just wanted to check if the response clarified your questions or needs further discussions. Thanks again for your constructive feedback.

Best,

Authors