Dear Reviewer 8gYg, we are grateful for your detailed feedback. Please see our following point-to-point responses to address your comments.

Presentations of Figure 2. and Figure 3.

Thanks for your valuable suggestions about presenting the results. The reason why we choose to use numbers of OoD samples, instead of InD/OoD ratios, as X-axis is because, in practice, the OoD samples at hand are usually very scarce (ex, unseen manufacturing defects, rarely seen obstacles in autonomous driving), whereas the size of InD training data can grow easily. This makes the ratio metric not as informative as numbers. As for the number of OoD samples that we generate, we generate OoD samples in each iteration of the iterative training process and use those to regularize the model. For detailed information, we kindly refer the reviewer to Appendix D.2.2, where $B_G$ denotes the number of OoD samples generated in each iteration.

Differences between two regimes

We are glad that you point out that OoD data are unlabeled anyway. We totally agree with you. Although the OoD labels are not used in the model training, the inherent differences between OoD samples from different classes can affect the model performance. And there might be a distribution shift between observed and test OoD samples. The idea here is to control what has been observed by our method in the training time. Under the balanced sampling strategy in regime I, the testing OoD samples resemble part of training OoD samples. However, in practice, people only possess very limited knowledge about OoD samples so it is reasonable that some types of OoD samples are not observed. Under the imbalanced sampling strategy in regime II, the test OoD samples may be different from the observed OoD samples.

Further explanation of imbalanced regime

In the imbalanced regime, we deliberately leave one class (i.e., type) of OoD samples unobserved. For example, in the SVHN Within-Dataset experiment, classes 8 and 9 are considered OoD, but during training, only samples from class 8 are observed. We employ this setting to assess the generalizability of our method. For specific details regarding experimental setups and the number of samples included during training, we kindly refer the reviewer to Table 2 and Appendix E.

Evaluation of quality of generated outliers

Thank you for highlighting this crucial aspect. In our paper, Theorem 1 and Theorem 2 enable us to offer theoretical guarantees on the quality of generated outliers. We have demonstrated that, theoretically, the generated outliers will not be part of InD clusters. Regarding the experimental setup, we deliberately refrain from controlling the generation of outliers. This choice aligns with the primary goal and contribution of our method, which is to explore OoD spaces. As long as the generated outliers do not become part of InD, our objective is considered accomplished.

After carefully analyzing authors' responses, I consider that my concerns have been satisfactorily addressed. Therefore, and despite other reviewers' opinions, I have decided to maintain my initial rating. Overall, I consider the paper has a contribution, the proposed approach is supported by a mathematical formalism (including proofs of threorems) and the experimental validation is convincing. I agree that including OOD samples in the training process limits somehow the generalization capability of the method, but it is still considered a valid setting.

Dear Reviewer cBKA, we are grateful for your detailed feedback. Please see our following point-to-point responses to address your comments.

Visualization of OoD samples

Thank you for your question on the interpretability of the generated outliers.

Given that our approach aims at the exploration of OOD spaces, the generated images need not look like observed OOD images. That said, one can add a distance regularization term within our objective to penalize the distance between OoD samples and the generated outliers. With such penalty, the generated images will resemble observed OoD samples. For your interest, we provided two generated images (for the MNIST experiment, where we treat classes 1 and 7 as OoD) with the distance regularization term. We can see meaningful images are generated in these cases, where interpretability is maintained.

• Example 1
• Example 2

However, by imposing the penalty term, the exploration of OoD spaces will be constrained. Indeed, we found that such a constraint will decrease generalization power. This observation also agrees with existing literature showing that such a regularization term led to reduced generalizability (Wang et al. 2021, Liu et al. 2020). Therefore, we decided to remove the penalty term in our final formulation.

Missing AUROC metrics

We kindly refer the reviewer to our general response.

Experiments are limited

We kindly refer the reviewer to our general response. In addition, we would like to communicate the following perspectives with the reviewer.

Are the difficulties of OoD detection tasks determined by the complexities of datasets?

It is critical to point out that the complexity of the InD and OoD datasets is not the only factor that influences the difficulties of OoD detection tasks. According to Liang et al. (2020), Lee et al. (2018), and Liu et al. (2020), one of the central challenges for OoD detection tasks in classification models arises from the fact that DNN models are usually over-confident towards InD samples.

For example, OoD detection on a simple InD dataset like the FashionMNIST Within-Dataset experiment, where the classification accuracy can easily reach 97% (i.e. high predictive confidence for InD data), is not necessarily easier than CIFAR100-Texture Between-Dataset experiments done in Liu et al., 2020, where the classification accuracy is only about 79% (i.e. low predictive confidence). Indeed, this explains why the baseline methods, including OE (Hendrycks et al., 2018), Energy Finetuning (Liu et al. 2020), and VOS (Du et al., 2022) fail to achieve desirable performance on the simpler datasets but can achieve decent performance on those seemingly complex datasets like CIFAR100-Texture according to Liu et al. 2020.

That said, we have included another set of experiments on CIFAR10-Texture experiments to further highlight the benefits of our approach.

Furthermore, from our experiments, we notice that another critical factor influencing the difficulties of OoD detection tasks is the proximity between InD and OoD samples. This observation motivated the Within-Dataset experiments, where OoD samples come from classes within the same dataset. The idea here was to create situations where InD and OoD samples highly resemble each other and highlight the capability of our model to effectively detect such cases. While in practice, this situation may not be universally applicable, it does represent various real-world situations, such as anomaly detection, where anomalies can manifest as small deviations from the InD samples or their features (Sun et al. 2023).

Dear Reviewer BP77, we are grateful for your detailed feedback. Please see our following point-to-point responses to address your comments.

Evaluation Metric

We kindly refer the reviewer to our general response.

Generality of Within-Dataset Detection Tasks

We would like to thank Reviewer BP77 for pointing this out. As you alluded, the primary objective of introducing this new setting is to assess OoD detection accuracy when InD and OoD samples closely resemble each other. The resemblance between InD and OoD samples introduces additional challenges compared to the conventional between-dataset detection setting.

We agree with your comment on the universal applicability of Within-Dataset detection tasks.

However, in various real-world situations, InD and OoD samples may resemble each other. For instance, in anomaly detection, anomalies can manifest as small deviations from the InD samples (Sun et al., 2023). This insight motivated the Within-Dataset experiments, where OoD samples originate from classes within the same dataset. The goal was to create scenarios where InD and OoD samples come from the same dataset and emphasize our model's ability to effectively detect such cases.

That said, it is crucial to clarify that the within-dataset detection setting serves as an additional metric when compared to other methods. We have demonstrated that our method significantly outperforms baselines in the traditional Between-Dataset setting, particularly in terms of generalizability to test OoD samples not encountered during the training process. This also indicates that our method effectively leverages existing limited OoD samples and explores potential OoD spaces more efficiently compared to the baselines.

Scalability Concerns

We kindly refer the reviewer to our general response. Following the settings in Liu et al. (2020) we added one more Between-Dataset experiment, CIFAR10-Texture with more InD classes, and one more baseline OE (Hendrycks et al., 2018), to illustrate the scalability and effectiveness of our method.

In addition, we would like to communicate the following perspective with the reviewer.

Are the difficulties of OoD detection tasks determined by the complexities of datasets?

It is critical to point out that the complexity of the InD and OoD datasets is not the only factor that influences the difficulties of OoD detection tasks. According to Liang et al. (2020), Lee et al. (2018), and Liu et al. (2020), one of the central challenges for OoD detection tasks in classification models arises from the fact that DNN models are usually over-confident towards InD samples.

For example, OoD detection on a simple InD dataset like the FashionMNIST Within-Dataset experiment, where the classification accuracy can easily reach 97% (i.e. high predictive confidence for InD data), is not necessarily easier than CIFAR100-Texture Between-Dataset experiments done in Liu et al., 2020, where the classification accuracy is only about 79% (i.e. low predictive confidence). Indeed, this explains why the baseline methods, including OE (Hendrycks et al., 2018), Energy Finetuning (Liu et al. 2020), and VOS (Du et al., 2022) fail to achieve desirable performance on the simpler datasets but can achieve decent performance on those seemingly complex datasets like CIFAR100-Texture according to Liu et al. 2020.

That said, we have included another set of experiments on CIFAR10-Texture experiments to further highlight the benefits of our approach.

Furthermore, from our experiments, we notice that another critical factor influencing the difficulties of OoD detection tasks is the proximity between InD and OoD samples. This observation motivated the Within-Dataset experiments, where OoD samples come from classes within the same dataset. The idea here was to create situations where InD and OoD samples highly resemble each other and highlight the capability of our model to effectively detect such cases.

While in practice, this situation may not be universally applicable, it does represent various real-world situations, such as anomaly detection, where anomalies can manifest as small deviations from the InD samples or their features (Sun et al. 2023).

Dear Reviewer Qajt, we are grateful for your detailed feedback. Please see our following point-to-point responses to your comments.

Some arguments are impervious or unfaithful

Based on your comments, we added more explanations to clarify our paper's goal and to justify the rationale and advantages of our method. We kindly refer the reviewer to our "general response".

Experiments are limited

1. Datasets are simple

We address this question in our "general response". Specifically, we followed your suggestion and included one more Between-Dataset experiment on the CIFAR10 and Texture dataset introduced in Liu et al. 2020. In addition, we would like to communicate the following perspective with the reviewer.

Are the difficulties of OoD detection tasks determined by the complexities of datasets?

It is critical to point out that the complexity of the InD and OoD datasets is not the only factor that influences the difficulties of OoD detection tasks. According to Liang et al. (2020), Lee et al. (2018), and Liu et al. (2020), one of the central challenges for OoD detection tasks in classification models arises from the fact that DNN models are usually over-confident towards InD samples.

For example, OoD detection on a simple InD dataset like the FashionMNIST Within-Dataset experiment, where the classification accuracy can easily reach 97% (i.e. high predictive confidence for InD data), is not necessarily easier than CIFAR100-Texture Between-Dataset experiments done in Liu et al., 2020, where the classification accuracy is only about 79% (i.e. low predictive confidence). Indeed, this explains why the baseline methods, including OE (Hendrycks et al., 2018), Energy Finetuning (Liu et al. 2020), and VOS (Du et al., 2022) fail to achieve desirable performance on the simpler datasets but can achieve decent performance on those seemingly complex datasets like CIFAR100-Texture according to Liu et al. 2020.

That said, we have included another set of experiments on CIFAR10-Texture experiments to further highlight the benefits of our approach.

Furthermore, from our experiments, we notice that another critical factor influencing the difficulties of OoD detection tasks is the proximity between InD and OoD samples. This observation motivated the Within-Dataset experiments, where OoD samples come from classes within the same dataset. The idea here was to create situations where InD and OoD samples highly resemble each other and highlight the capability of our model to effectively detect such cases.

2. Baselines are missing

We kindly refer the reviewer to our "general response".

3. Recent strong OoD detection methods are not compared

We want to thank the reviewer for pointing out these two excellent papers which came out in May 2023. However, we have to respectfully mention that it is a rare request to conduct a comparison study with the methods that are officially published four months prior to the submission deadline.

4. It is straightforward to introduce OoD samples in the training process

Please refer to our responses in the "general response", where we reiterate the main contributions of this paper. Secondly, we respectfully disagree with Reviewer Qajt's comment suggesting that incorporating OoD samples into the training process is straightforward. It is, instead a very challenging task that needs to be done carefully. There are three primary reasons for this:

• Methods that solely rely on existing OoD samples without exploring OoD spaces, such as Energy Finetuning (Liu et al., 2020), WOOD (Wang et al., 2021), and OE (Hendrycks et al., 2018), fail to detect OoD samples not encountered in the training process, leading to poor generalizability. The main pitch of our method is to effectively use observed OoD samples to perform iterative exploration of OoD spaces. Our experimental results in the imbalanced regime substantiate this claim.
• Existing methods that exploit OoD samples are susceptible to overfitting when OoD samples are scarce in the training dataset, as demonstrated in our experiments. In such cases, augmenting the limited OoD samples is desirable, particularly in the original image space rather than the feature space. This is because feature spaces may exhibit high overlap, even for images that appear vastly different.
• We agree with Reviewer Qajt that many methods that do not use OoD samples have strong modeling. However, their modeling approach is mainly based on operations (including exploration) within InD spaces (Du et al. 2022; Ming et al. 2022; Lee et al. 2017). In these cases, there is no straightforward method to incorporate OoD samples during the training stage. Introducing OoD samples naively using score regularization is likely to give rise to the aforementioned two problems.