How Does Unlabeled Data Provably Help Out-of-Distribution Detection?

We are deeply encouraged that you recognize our method to be novel, significant, and solid in both the algorithm and theory and with remarkable empirical results.

Your summary and comments are insightful and spot-on :)

A1. Verification of the assumptions on additional datasets

Thank you for the suggestion! As suggested, we verified the assumption of distribution discrepancy using CIFAR-10 as ID and five other OOD datasets, i.e., SVHN, PLACES365, LSUN-C, TEXTURES, and LSUN-R. The result is shown as follows and in Appendix Section W, and we can conclude that $\zeta$ can indeed satisfy the regulatory condition in Theorem 2, i.e., $\zeta > 1.011\sqrt{\pi}$.

SVHN

PLACES365

LSUN-C

TEXTURES

LSUN-R

Dear Reviewer dSEG,

Thank you for taking the time to read our response and increasing your score! We are glad to hear that the response solved your concern. We will make sure the discussion and results will be added in the final version!

Thanks,

Authors

We thank you for recognizing our method to be novel with promising results. We thank the reviewer for the thorough comments and suggestions, which we address below:

A1. Baselines

As suggested, we have additionally compared with the two related works (TSL [1] and STEP [2]). To ensure a fair comparison, we strictly follow the experimental setting in TSL [1], and rerun SAL under the identical setup. The comparison on CIFAR-100 is shown as follows. Accordingly, we have also added discussion and proper citations of the mentioned papers in the revised paper (See related work section and Appendix Section U). We thank you for pointing them out!

A2. Discussion on weakly supervised OOD detection

We agree that weakly supervised OOD detection is indeed similar to the problem setting of SAL. We have already updated the related work section of our paper and included more discussions/citations on weakly supervised OOD detection. Thank you for pointing this out!

A3. Near-OOD Scenario

We are glad you bring that up. We have already evaluated the near-OOD detection in Appendix J. Specifically, we use the CIFAR-10 as the in-distribution data and the CIFAR-100 as the OOD data in the wild. During test time, we use the test set of CIFAR-100 as the OOD for evaluation. With a mixing ratio $\pi$ of 0.1, SAL achieves an FPR95 of 24.51% and AUROC of 95.55% compared to 38.92% (FPR95) and 93.27% (AUROC) of WOODS.

In addition, we follow the suggested data setting by the reviewer, i.e., the first 50 classes of CIFAR-100 as ID and the last 50 classes as OOD. The comparison with the most competitive baseline is reported as follows. We have also added the new results to Appendix Section J in the revised manuscript.

We are glad to see that the reviewer finds our work significant and novel from various perspectives. We thank the reviewer for the thorough comments and suggestions. We are happy to clarify as follows:

A1. ID accuracy

Great observation! As explained in paragraph 2 of Section 5.2 in the original submission, the slight discrepancy is due to that our method only observes 25,000 labeled ID samples, whereas baseline methods (without using wild data) utilize the entire CIFAR training data with 50,000 samples. We have used bold fonts to highlight it in the revision.

A2. Additional experiment results with varying $\pi$

Thank you for your suggestion! In our main experiment, we default $\pi$ to be 0.1, which strictly follows the original setting in WOODS [1]. This reflects the practical scenario that the majority of test data may remain ID. Compared to larger $\pi$, our setting with $\pi=0.1$ is also more challenging due to limited information of OOD data.

We would like to point the reviewer to the Appendix Table 6 of the original submission, where we report the OOD detection result and the filtering error on SVHN with different mixing ratios $\pi$. The result aligns well with our observation of the bounds presented in Section 4.1 of the main paper.

During rebuttal, we provide additional results on more OOD datasets with varying $\pi$, i.e., 0.05, 0.2, 0.5, 0.9, and contrast with the baselines, which are added to Appendix Section T, and also shown below (CIFAR-100 as the in-distribution dataset). We found that the advantage of SAL still holds.

[1] Julian Katz-Samuels et al. Training ood detectors in their natural habitats. In International Conference on Machine Learning, 2022.

A3. Additional experiments on multiple principal components

Another great point! In our original submission, we reported results using multiple principal components in Appendix Section K. We observed that using the top 1 singular vector for projection achieves the best performance.

A4. Discussion on using gradient norm

We have already evaluated the GradNorm score as suggested in Table 2, where we replace the filtering score in SAL with the GradNorm score and then train the OOD classifier. The result underperforms SAL, showcasing the effectiveness of our filtering score.

We have also extensively discussed the design rationale of the filtering scores of SAL in Section 3.1, saying that the scores in SAL for ID and OOD data are shown to be provably well-separated (Remark 1) and thus ensure a low filtering error, while the norm of the gradient is not. Both the theoretical result and empirical verification can demonstrate the advantage of SAL compared with GradNorm.

We thank the reviewer for the thorough comments and suggestions. We are encouraged that you recognize our method to be novel and effective, and with robust theoretical analysis. We address your questions below:

A1. Different outlier dataset from the actual test OOD

We are glad you bring that up! In the original submission, we have included the result where the OOD data in the wild is different from the test OOD data (please see Appendix Section I and Table 8). Specifically, we use 300K RANDOM IMAGES from outlier exposure [1] to create the wild OOD training dataset. We evaluate on SVHN, PLACES365, LSUN-C, LSUN-RESIZE, and TEXTURES as the test unseen OOD data. We observe that SAL can perform competitively on unseen OOD datasets as well, compared to the most relevant baseline WOODS.

[1] Hendrycks et al. Deep anomaly detection with outlier exposure. In Proceedings of the International Conference on Learning Representations, 2019.

A2. Quality of the unlabeled data

To address your concern, we have designed the following experiment where the quality of the unlabeled data deteriorates. The results of SAL and competitive baselines are shown in the table below and have also been added to the Appendix Section S.

Specifically, we corrupt the outlier data in the wild with additive Gaussian noise [2]. As such, the filtered candidate outliers will have a much lower quality compared to the outliers in SAL. We use the CIFAR-10 as the in-distribution dataset and keep other configurations the same.

[2] Hendrycks et.al., Benchmarking neural network robustness to common corruptions and surface variations. In Proceedings of the International Conference on Learning Representations, 2019.

A3. ID accuracy

Great observation! As explained in paragraph 2 of Section 5.2 in the original submission, the slight discrepancy is due to that our method only observes 25,000 labeled ID samples, whereas baseline methods (without using wild data) utilize the entire CIFAR training data with 50,000 samples. We have used bold fonts to highlight it in the revision.