Rethinking Out-of-Distribution Detection on Imbalanced Data Distribution

Dear reviewer XZSm:

We thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:

W1: The method relies on external OOD data, which is often difficult to obtain in OOD detection settings.

A: We follow PASCL[1]'s setting to evaluate our method for imbalanced OOD detection with auxiliary OOD data during training, which is a commonly-used setting in the literature[2][3][4] because auxiliary OOD data is relatively easy to obtain in practice[4][5].

On the other hand, to further evaluate our method without real OOD training data, we also integrate with feature-level synthesis methods (e.g., VOS[6] and NPOS[7]) to generate pseudo-OOD data for training. In particular, we follow ClassPrior's setting to train a MobileNet model on ImageNet-LT-a8 with VOS for OOD synthesis, and evaluate the performance on four OOD test sets. Our method outperforms the SOTA ClassPrior method by a large margin on all subsets.

W2: The comparisons in CIFAR and ImageNet experiments seem inconsistent, with fewer methods evaluated on ImageNet. Notably, ClassPrior seems to perform better on ImageNet.

A: Thanks for pointing out this. The reason is that ClassPrior uses a totally different setting against the literature[1-4] that we follow, and the differences are detailed as follows:

In Table 1 of our manuscript, we take ClassPrior's results on CIFAR10/100-LT from COCL's paper. Here, we provide a further comparison on ImageNet following ClassPrior's setting, and the results are reported in the above table (in the response to W1), which demonstrates our superiority against ClassPrior and other OOD detection methods.

W3: Some non-long-tail-specific OOD detection methods in the ClassPrior paper also perform well. Comparing and discussing these methods with the proposed approach would provide a more comprehensive evaluation.

A: Thanks for this suggestion. On one hand, after aligning with ClassPrior's setting, we have further illustrated our method's efficacy over other OOD methods.

On the other hand, we have also compared it with several SOTA OOD detection methods published in the most recent top conferences (ICLR'24, CVPR'24, and ICML'24). Due to the space limit, please kindly refer to our response to Reviewer QTK3 (R3)'s Question 2 (Q2).

We will add those experiments and discussions to further illustrate the novelty and contribution of our paper.

Q1: Why do the ID labels in Figure 1a show similar fluctuations across OE, Energy, and PASCL?

A: In Figure 1a, the ID labels actually means label distribution for **wrongly-detected** ID samples, and OE, Energy, and PASCL show similar fluctuations because they face the same problem: wrongly detecting samples from tail ID classes as OOD data, which inspires us to formulate this phenomenon and develop a unified solution.

Q2: I believe the OOD prediction results in Figure 1a do not support the authors' claim. If I am mistaken, please correct me. The distribution of OOD predictions appears consistent with the prior distribution, with no significant deviation in the head region. For OOD samples, since the model hasn't been trained on them, it would predict according to the training sample distribution (prior distribution). I don't see this behavior as problematic.

A: The fact that models tend to predict according to training data distribution is exactly the core problem for imbalanced OOD detection.

First, we may kindly clarify that Figure 1a displays the class predictions for OOD samples that are wrongly-detected as ID data, and all three OOD detectors tend to wrongly detect OOD data as head ID classes. On the other hand, we have shown in our uploaded PDF file that both of the correctly-detected ID and OOD samples are insensitive to the class distribution.

Therefore, we argue that this behavior (the model will predict according to training distribution) is exactly the problem to solve (e.g., wrongly detecting OOD samples as ID data from head class), which is similar to the problem studied by the long-tailed recognition community (wrongly recognizing test samples as head classes). We are thus motivated to formulate the distribution gap on imbalanced OOD detection, and try to regulate the imbalanced detectors towards balanced models using proposed techniques.

We hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.

Best regards,

Authors

---

[1] Wang et al. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. ICML'22.

[2] Cui et al. Class-balanced loss based on effective number of samples. CVPR'19.

[3] Kang et al. Decoupling representation and classifier for long-tailed recognition. ICLR'20.

[4] Menon et al. Long-tail learning via logit adjustment. ICLR'21.

[5] Yang et al. Generalized out-of-distribution detection: A survey. IJCV'24.

[6] Du et al. Vos: Learning what you don't know by virtual outlier synthesis. ICLR'22.

[7] Tao et al. Non-parametric outlier synthesis. ICLR'23.

Dear Reviewers,

Thank you for taking the time to review the paper. The discussion has begun, and active participation is highly appreciated and recommended.

Thanks for your continued efforts and contributions to NeurIPS 2024.

Best regards,

Your Area Chair

Dear Reviewer XZSm,

We thank you again for your time and efforts in evaluating our paper. As the discussion period will end in 24 hours, we are eager to receive your feedback on our responses, which is very important to us. If there are any further questions, we are more than happy to provide more clarifications as needed.

---

Best regards,

Authors

Dear reviewer QTK3:

We thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:

W1: It seems that OOD detection under class imbalance is merely an overlap of the two tasks: class-imbalanced learning and OOD detection. Using techniques from both fields can address this issue quite well.

A: OOD detection and class-imbalanced learning are not seperate tasks. This issue has already been investigated and verified by PASCL[1]. According to the results on the CIFAR10-LT benchmark below, a simple combination of popular long-tailed learning methods (e.g., Reweighting[2], $\tau$-norm[3], LA[4], etc.) does not address well the imbalanced OOD detection problem. Instead, PASCL treats imbalanced ID classification and OOD detection as a joint problem and achieves considerable improvement over the baselines, and our method further boosts imbalanced OOD detection significantly with the help of our theoretical groundedness. We hope the discussion and experiments can help establish the significance and contribution of our paper.

Q1: Why not obtain the class-specific scaling factor using methods from the long-tailed image recognition domain?

A: In our paper, the estimate of class prior $P(y)$ is actually borrowed from the long-tailed recognition literature[4] (using label frequency), while we have not currently[5] seen a reliable method to calculate the class-specific factor over balanced and imbalanced data likelihood $\gamma_y(x) = \frac{1}{K}\frac{P^{bal}(x|y)}{P(x|y)}$. Therefore, we choose to learn it by automatic parametric mapping, which has been proven effective[6] when estimating the data density $P(x)$, and the empirical and statistical results in Table 3 and Figure 2 have further confirmed the effectiveness of our estimated $\gamma_y(x)$. Despite that, we are also glad to integrate our method into reliable estimates of $\gamma_y(x)$ for a further study if applicable.

Q2: The paper compares relatively few OOD detection methods; it is recommended to include more recent OOD detection approaches.

A: Thanks for this suggestion. We have additionally compared with several SOTA OOD detection methods recently published in top conferences (ICLR'24, CVPR'24, and ICML'24):

• NECO[7] studies the neural collapse phenomenon develops a novel post-hoc method to leverage geometric properties and principal component spaces to identify OOD data.
• fDBD[8] investigates model's decision boundaries and proposes to detect OOD using the feature distance to decision boundaries.
• IDLabel[9] theoretically delineates the impact of ID labels on OOD detection, and utilizes ID labels to enhance OOD detection via representation characterizations through spectral decomposition on the graph.
• ID-like[10] leverages the powerful vision-language model CLIP to identify challenging OOD samples/categories from the vicinity space of the ID classes, so as to further facilitate OOD detection.

The experimental results are displayed as follows:

On the CIFAR10-LT benchmark, our method surpasses the recent OOD detectors by a large margin. For imbalanced data distribution, IDLabel[9] struggles to capture the distributional difference across ID classes, while the pure post-hoc methods NECO[7] and fDBD[8] even get worse results because they fail to generalize to the scenario with highly skewed feature spaces and decision boundaries. On the ImageNet-LT benchmark, even the incorporation of the powerful CLIP model cannot well address the imbalance problem for ID-like[10], while our method can specifically and effectively facilitate imbalanced OOD detection with the help of our theoretical groundness.

We will add those experiments and discussions to further illustrate the novelty and contribution of our paper.

Q3: The manuscript also lacks a summary of related works in class-imbalanced learning.

A: Thanks for this suggestion. We will supplement the discussion of current class-imbalanced learning methods[5] (via class re-balancing, information augmentation, module improvement, etc.), and specifically illustrate that the imbalanced ID recognition and OOD detection are a joint problem in the literature (the response to W1) to highlight our contribution.

We hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.

Best regards,

Authors

---

[1] Wang et al. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. ICML'22.

[2] Cui et al. Class-balanced loss based on effective number of samples. CVPR'19.

[3] Kang et al. Decoupling representation and classifier for long-tailed recognition. ICLR'20.

[4] Menon et al. Long-tail learning via logit adjustment. ICLR'21.

[5] Zhang et al. Deep long-tailed learning: A survey. TPAMI'23.

[6] Kumar et al. Normalizing flow based feature synthesis for outlier-aware object detection. CVPR'23.

[7] Ammar et al. NECO: NEural Collapse Based Out-of-distribution Detection. ICLR'24.

[8] Liu et al. Fast Decision Boundary based Out-of-Distribution Detector. ICML'24.

[9] Du et al. When and how does in-distribution label help out-of-distribution detection? ICML'24.

[10] Bai et al. ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection. CVPR'24.

Dear Reviewers,

Thank you for taking the time to review the paper. The discussion has begun, and active participation is highly appreciated and recommended.

Thanks for your continued efforts and contributions to NeurIPS 2024.

Best regards,

Your Area Chair

Dear Reviewer QTK3,

Thank you for your positive feedback and for raising your score! We are glad that our responses addressed your concerns and will ensure to include them in the final version.

---

Best regards,

Authors

Dear Reviewer QTK3,

We thank you again for your time and efforts in evaluating our paper. As the discussion period will end in 24 hours, we are eager to receive your feedback on our responses, which is very important to us. If there are any further questions, we are more than happy to provide more clarifications as needed.

---

Best regards,

Authors

Dear reviewer E388:

We thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:

W1: A minor issue is that, when we know in advance that the data are long-tailed, it is a common practice that we will use the long-tailed learning methods to counteract its impacts. In this case, is the study of long tailed OOD detection an important topic? Or, even with long-tailed learning, the OOD detection therein still be a critical issue?

A: Yes, this issue has already been investigated and verified by PASCL[1]. According to the results on the CIFAR10-LT benchmark below, a simple combination of popular long-tailed learning methods (e.g., Reweighting[2], $\tau$-norm[3], LA[4], etc.) does not address well the imbalanced OOD detection problem. Instead, PASCL treats imbalanced ID classification and OOD detection as a joint problem and achieves considerable improvement over the baselines, and our method further boosts imbalanced OOD detection significantly with the help of our theoretical groundedness.

W2: I am not sure if the prior information can be properly estimated, considering the situations that the adopted models are not well calibrated (otherwise, OOD detection will not be a challenging task).

A: In previous studies, statistics on label frequency is a commonly-adopted estimate for class prior information (i.e., $P(y) := \frac{n_y}{n}$)[4][5], and automatically learning the data density $P(x)$ (e.g., by normalizing flow[6]) is also proved applicable. Therefore, we follow the literature to estimate the class-aware and input-dependent scaling factor $\gamma_y(x) = \frac{1}{K}\frac{P^{bal}(x|y)}{P(x|y)}$ by automatic optimization, and our empirical statistics on Figure 2 has validated the estimated results. On the other hand, we also believe that a more precise estimation will further enhance our methods.

We hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.

Best regards,

Authors

---

[1] Wang et al. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. ICML'22.

[2] Cui et al. Class-balanced loss based on effective number of samples. CVPR'19.

[3] Kang et al. Decoupling representation and classifier for long-tailed recognition. ICLR'20.

[4] Menon et al. Long-tail learning via logit adjustment. ICLR'21.

[5] Jiang et al. Detecting out-of-distribution data through in-distribution class prior. ICML'23.

[6] Kumar et al. Normalizing flow based feature synthesis for outlier-aware object detection. CVPR'23.

Dear Reviewer E388,

We thank you again for your time and efforts in evaluating our paper. As the discussion period will end in 24 hours, we are eager to receive your feedback on our responses, which is very important to us. If there are any further questions, we are more than happy to provide more clarifications as needed.

---

Best regards,

Authors

Dear reviewer 5Lex:

We thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:

W1: Train-time regularization techniques cannot be applied to existing pretrained models, which may limit their adoption.

A: We have also tried to apply our method during pre-trained models' inference. According to our Theorem 3.2 in our manuscript, for an existing OOD detector $P(i|x)$ (e.g., trained with BinDisc), we can calculate the bias term $\beta(x)$ to regulate the vanilla scorer into balanced $P^{bal}(i|x) = \beta(x) \cdot P(i|x)$. However, as $\beta(x) = \sum_y{\gamma_y(x)\frac{P(y|x,i)}{P(y)}}$, the estimation of $\gamma_y(x)$ presents considerable difficulty without training, but we have also tried some trivial approaches to adapting our method to inference time.

First, we simply treat $\gamma_y(x)$ as a constant (e.g., $1$) for arbitrary input $x$ and class $y$ to calculate the bias term (denoted as $\beta_1(x)$), and the results on CIFAR10-LT benchmark immediately witness a performance improvement (e.g., 0.28% increase on AUROC and 1.29% decrease on FPR95) compared to the baseline OOD detector. However, the improvement is relatively insignificant, and the phenomenon is consistent with our ablation studies in Table 3, which demonstrates the importance of learning a class-dependent and input-dependent $\gamma_y(x)$ during training.

Then, inspired by the statistical results in Figure 2, we take a further step to use a polynomial (rank=2) to fit the curve between the predicted class $y$ and $\gamma_y(x)$ learned by another model, and apply the coefficients to estimate a class-dependent $\hat{\gamma}_y$ for the baseline model (denoted as $\hat{\beta}(x)P(i|x)$). This operation receives further enhancement on OOD detection and gets close to our learned model (e.g., 30.80% v.s. 29.95% of FPR95).

In conclusion, our attempts illustrate the potential of applying our method to an existing model without post-training, and we will continue to extend $\hat{\gamma}_y$ to an input-dependent version (say $\hat{\gamma}_y(x)$) in our future work.

W2: The method requires OOD data during training, and results may not generalize to other unseen OOD settings.

A: We follow PASCL[1]'s setting to evaluate our method in imbalanced OOD detection with auxiliary OOD data during training, which is a commonly used setting in the literature[2][3][4] because auxiliary OOD data is relatively easy to obtain and usually generalizes well to unseen OOD samples[2][5].

On the other hand, to further evaluate our method without real OOD training data, we also integrate with feature-level synthesis methods (e.g., VOS[6] and NPOS[7]) to generate pseudo-OOD data for training. In particular, we follow ClassPrior[8]'s setting to train a MobileNet model on ImageNet-LT-a8 with VOS for OOD synthesis, and evaluate the performance on four OOD test sets. Our method outperforms the SOTA ClassPrior method by a large margin on all subsets.

Q1: Have you tested the benefits of your method with varying levels of imbalance? How does the performance change?

A: Yes. In our original manuscript, we mainly take the default imbalance ratio ($\rho=100$), which means the least frequent (tail) class only has $\frac{1}{100}$ of training samples than the most frequent (head) class). Here, we follow PASCL to investigate another imbalance ratio of $\rho=50$ (relatively more balanced than $\rho=100$) on CIFAR10-LT, and the results are presented as follows:

Accordingly, our method consistently surpasses PASCL on various imbalance levels. Specifically, our method gains a larger enhancement (e.g., near 2.0% of AUROC) on the more imbalanced scenario with $\rho=100$, which further demonstrates our effectiveness in mitigating imbalanced OOD detection.

We hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.

Best regards,

Authors

---

[1] Wang et al. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. ICML'22.

[2] Hendrycks et al. Deep Anomaly Detection with Outlier Exposure. ICLR'19.

[3] Wei et al. EAT: Towards Long-Tailed Out-of-Distribution Detection. AAAI'24.

[4] Miao et al. Out-of-distribution detection in long-tailed recognition with calibrated outlier class learning. AAAI'24.

[5] Yang et al. Generalized out-of-distribution detection: A survey. IJCV'24.

[6] Du et al. Vos: Learning what you don't know by virtual outlier synthesis. ICLR'22.

[7] Tao et al. Non-parametric outlier synthesis. ICLR'23.

[8] Jiang et al. Detecting out-of-distribution data through in-distribution class prior. ICML'23.