Rethinking Out-of-Distribution Detection on Imbalanced Data Distribution

We thank the valuable comments, and we answer them in detail as follows.

Q1: How to deal with the self-imbalanced OOD data?

The self-imbalanced OOD data is indeed a practical problem. As suggested, we now take CIFAR100 as the imbalanced auxiliary OOD data, and train models with CIFAR10 as ID. Then models are evaluated on the remaining 5 subsets of the SCOOD benchmark (CIFAR100 was originally used as test OOD data but is now eliminated). According to the results reported below, our ImOOD consistently outperforms the advanced PASCL method.

Besides, we have also noticed a recent work by BalEnergy $^{[1]}$ that discussed this issue and added class-aware loss regularizations on OOD training data. According to the experiments in Appendix B.3 (due to different experimental settings) of our submitted paper, we equipped our ImOOD with BalEnergy and achieved better OOD detection performance. It indicates our method does not conflict with imbalanced OOD data optimization techniques.

Q2: About the effectiveness of post-hoc normalization.

Thanks for pointing out this. In the manuscript, we only provide the performance gain brought by our post-hoc normalization on CIFAR10-LT benchmark. Here we supplement the results on CIFAR100-LT and ImageNet-LT benchmarks, which shows applying our method leads to consistent improvements on various datasets. The general validation of our post-hoc normalization is verified.

However, as our experiments suggest, the enhancement by post-hoc normalization is relatively limited, while our training-time regularization seems to bring more significant advances. The main reason may be that the estimates of probability distributions are not well-calibrated, especially for the class prior $P(y) \coloneqq \pi_y$ whose numerical instability may severely affect the OOD detection process, as discussed in Appendix 2.3 in the submitted paper. We thus further develop the training-time regularization technique to automatically adjust the estimates, in order to ultimately learn a better OOD detector $g(x)$ close to the ideally balanced $g^{bal}(x)$. Once $g(x)$ is optimized, no estimate is used for detecting OOD samples, and the output of scorer $g(x)$ is more stable and effective in performing OOD detection.

We will add the experiments and discussions.

We thank the valuable comments, and we answer them in detail as follows.

Q1: Comparison and extension to distance-based OOD methods.

Thanks for the thoughtful suggestion. Due to the time limitation, here we mainly compare our method with Mahalanobios distance $^{[1]}$ (denoted as "Maha") on the CIFAR10-LT benchmark. The results show that our post-hoc normalization (denoted as "Maha + infer") brings slight improvements (e.g., 0.3% increase on AUROC), while our training-time regularization (denoted as "Maha + train") significantly boosts the imbalanced OOD detection (e.g., 1.2% increase on AUROC and 3.4% decrease on FPR95).

However, the final results are still worse than advanced models (with 92%+ on AUROC for example). A possible reason might be that the feature-level distance estimation is basically inaccurate, especially for the tail classes with rare data samples to statistic (e.g., computing the mean and variance). On the other hand, our method takes the imbalance problem into consideration, which helps balance the feature space by using the training-time regularization.

Q2: Ablation on synthesized OOD training data.

Thanks for the insightful suggestion. Here we equip with the novel VOS $^{[2]}$ and NPOS $^{[3]}$ to synthesize virtual OOD data in the feature space for training. According to the results shown below, synthesized outliers can also improve the OOD detection performance (e.g., 72% v.s. 75% on AUROC), and our method brings further advance regarding the vanilla VOS and NPOS methods (e.g., 75% v.s. 78% on AUROC).

However, compared to the real OOD training data (e.g., the TinyImages80M used in our original setting), the synthesized outliers seem to be unsatisfactory to help train a SOTA OOD detector. It indicates that the density-based VOS and the distance-based NPOS also suffer from inaccurate feature-level statistics on imbalanced data distribution, and are unable to generate qualified virtual outliers. This is more severe for models training from scratch, as NPOS also discussed, and image-level OOD sample synthesis may help alleviate this problem.

We will add those experiments and discussions.

We thank the valuable comments, and we answer them in detail as follows.

Q1: Ablation on the choice of auxiliary OOD training data.

Thanks for the insightful suggestion. In the main experiments reported in our manuscript, we follow PASCL $^{[1]}$ to adopt the TinyImages80M $^{[2]}$ as auxiliary OOD data for training-time regularization, and the models are tested on the SCOOD benchmark for 6 subsets. We now train the models with CIFAR100 as auxiliary OOD data (CIFAR10-LT is ID), and test them on the remaining 5 subsets of the SCOOD benchmark $^{[3]}$ (CIFAR100 was originally used as test OOD data but is now eliminated). According to the results reported below, our ImOOD consistently outperforms the advanced PASCL method.

However, models trained with CIFAR100 perform slightly worse than models trained with TinyImages80M, indicating that a more generally collected auxiliary OOD dataset also contributes to the final OOD detection performance, which is consistent with common sense. We will add the experiment and discussion.

Q2: Comparison with other representative OSR/OOD detection methods.

Thanks for the thoughtful suggestion. Due to the time limitation, we mainly compare our method with the representative OpenMAX $^{[5]}$ and OLTR $^{[6]}$ on CIFAR100-LT benchmark.

Before the comparison, we want to carefully identify that the classical Open-Set Recognition (OSR) setting is slightly different from the Out-Of-Distribution Detection (OOD) setting $^{[4]}$. In particular, vanilla OSR methods (like OpenMAX and OLTR) take a proportion (e.g., 50%) of CIFAR100-LT classes as ID (or, base) classes, and the remaining as OOD (or, novel) classes. No auxiliary OOD training data is utilized for optimization. To ensure comparability, we follow the OOD setting, that is, taking all 100 classes from CIFAR100-LT as ID classes, and use the corresponding codebase to train the models of OpenMAX and OLTR without auxiliary OOD dataset. OOD test set is the SCOOD benchmark.

The results below show that compared to the MSP baseline, OpenMAX gains 1.4% on AUROC, and OLTR further improves 1.0% on AUROC and 3.5% on AUPR by taking the imbalanced distribution into account. Moreover, with the same model as MSP baseline, Energy shows comparable performance, and our method on post-hoc normalization further brings significant improvement, reaching 66.38% of AUROC and 76.98% of FPR95. Finally, we also report the performance of our method trained with auxiliary OOD data (i.e., TinyImages80M) merely as a reference, to avoid unfair comparison with the above methods without OOD training data.

We thank the valuable comments, and we answer them in detail as follows.

Q1: Can algorithms that merely handle the imbalanced ID classification immediately address the OOD detection issue?

Thanks for pointing out the important preliminary problem. Wang et al.$^{[1]}$ had widely discussed this issue and conducted a series of experiments (part of the results on CIFAR10-LT is displayed below). The results indicated that simply combining the ID classification optimization (e.g., LA $^{[2]}$ or $\tau$-norm $^{[3]}$) with vanilla OOD detection techniques (e.g., OE) cannot address the challenge. We thus have to take imbalanced ID classification and OOD detection into integrated consideration.

Relevant citations and conclusions will be added to the revised manuscript.

Q2: Why previous methods failed at handling the imbalance problem but our ImOOD works?

Thanks for the thoughtful suggestion. We briefly summarize the main drawbacks of prious representative methods (PASCL $^{[1]}$ and ClassPrior $^{[4]}$):

1. Our method concurrently considers the misidentified ID samples from tail-class and OOD data predicted as head-class, as discussed in Section 3.1 in our manuscript.
2. PASCL focuses on improving the separability between tail-class ID samples and OOD data, while ignores the misidentification between OOD data and ID head-class, as illustrated in Figure 1 in our manuscript.
3. ClassPrior takes both the two factors into account, and develops a series of manually-designed post-hoc normalization techniques on models only trained with ID data. However, it makes a strong assumption that the posterior probability of OOD data equals the prior ID probability, that is, $P(y|x \in \mathcal{D}^{out}) = P(y)$ (see its Theorem 3.2). The question is this assumption does not always hold. In our Table 2, we report its "RP+MSP" (RePlacing) strategy on models trained with CIFAR10-LT, which leads to negligible improvements (e.g., 0.3% increase of AUROC). We further test its "RW+Energy" (ReWeighting) strategy on the same model, as shown in the table below, which even causes performance degradation. In contrast, our method does not make any assumption on $P(y|x \in \mathcal{D}^{out})$, and shows strong robustness to the practical scenarios (e.g., 4.3% increase of AUROC). Besides, our ImOOD provides training-time regularization to further boost the imbalanced OOD detection, where ClassPrior is inapplicable since its assumption $P(y|x \in \mathcal{D}^{out}) = P(y)$ does not necessarily hold during training.

We will supplement those discussions to clarify our novelty and contribution.