Long-Tailed Out-of-Distribution Detection via Normalized Outlier Distribution Adaptation

Thank you for your appreciation of the novelty of our approach, empirical justification, problem setting, paper clarity, as your invaluable feedback on further enhancing the work. Please find our one-by-one responses to your concerns below:

W1: It would be comprehensive if the accuracy is reported on ID data

We have reported the ID accuracy of all methods on CIFAR10-LT and CIFAR100-LT in Tab. 2 and on ImageNet-LT in Tab. 4. It is impressive that our AdaptOD achieves not only consistently substantial improvement over current SOTA methods on OOD detection metrics but also consistently better ID classification accuracy on all three widely-used ID classification benchmarks.

W2: I am curious why there is not a weight coefficient before DNE loss

This is mainly because the DNE loss works stably with the cross-entropy loss in Eq. 11. We provide some ablation study results of having a coefficient hyperparameter for regularizing the DNE loss in Tab. G below, from which it is clear that the DNE loss can work well within its importance hyperparameter set in [0.5, 2.0].

Table G: Averaged AUC results of OOD detection on CIFAR10/100-LT for AdaptOD with different importance weights of the DNE loss

W3: It is better to ablate on the number of outlier data used during adaptation

If we understand your question correctly, the results in Fig. 3 should be the results you are looking for, where we report the performance of our proposed DODA and two existing TTA methods with an increasing number of OOD samples. It shows that our AdaptOD can utilize the test-time OOD data to adapt the true OOD distribution much faster and substantially more effective.

Thank you for your constructive feedback and appreciation of the novelty of our work and its empirical justification. We provide our response to your concerns one by one in the following.

W1: The overview of Figure 2 is difficult to understand. Simplifying and highlighting only the necessary information would be beneficial.

Thanks for your advice, we have provided a revised Figure 2 in the attached PDF and simplified the overview of Figure 2 as follows.

• we will rewrite the DODA part as: Each test sample is assigned a global energy-based OOD score $\mathbb G(x)$ to adapt the outlier distribution $\mathcal{P}^{out}$. DODA then uses the adapted outlier distribution $\mathcal{P}^{out}$ to calibrate the global energy score $\mathbb G(x)$, obtaining the calibrated global energy score $\mathbb G^{\mathcal{P}}(x)$ as the OOD score.

• we will also rewrite the DNE part as: For each iteration, DNE first applies Batch Energy Normalization on logit output to obtain the normalized energy, and then utilizes this energy to optimize a dual energy loss function at both the class and sample levels.

W2: While recent methodologies improve long-tailed OOD detection performance using synthetic outliers without real outliers, this approach still relies on auxiliary outliers. This dependency can limit the practical applicability of the method.

As far as we know, existing SOTA long-tailed OOD detection methods all take the outlier exposure (OE) approach, relying on the availability outlier data; non-OE-based methods still cannot work well on long-tailed ID data due to the heavy class imbalance. However, all these existing OE-based methods are challenged by the distribution gap between the distributions of outlier data and true OOD data. The proposed AdaptOD effectively tackles this common, crucial challenge by the novel DODA and DNE modules. We agree that there might be application scenarios where no auxiliary outlier data may be available. We will explore to extend our method to this setting by having the initial outlier distribution trained on synthetic outlier data, or even randomly initializing the outlier distribution. Specifically, we attempt to randomly Gaussian initializing which has each dimension sampled from an isotropic Gaussian distribution as the auxiliary outliers to train the OOD models as shown in Tab. E and Tab. F below.

Table E： Results on CIFAR10-LT using randomly Gaussian initializing as auxiliary outliers

Table F： Results on CIFAR100-LT using randomly Gaussian initializing as auxiliary outliers

W3: The method's effectiveness is questionable when handling distribution shifts between training auxiliary outliers and actual real-world outliers, potentially compromising its performance if the shift is large.

To mitigate the influence of a large distribution shift between training auxiliary outliers and actual real-world outliers during DODA, we utilize the statistic of training ID data to guarantee the effective optimization of outlier distribution in Eq. 3 during the testing phase, which effectively alleviates the reliance on the prior from the training auxiliary outlier data during the adaptation of DODA, thereby reducing the large adverse effects brought by the auxiliary outlier data that is highly different from the true OOD data. As a result, our AdaptOD can achieve SOTA performance over six large diverse OOD test datasets (including both the near-OOD dataset and the far-OOD dataset) and three synthetic OOD datasets as well on the CIFAR10/100-LT-based ID dataset benchmarks.

Thank you for appreciating our contribution and for your thoughtful and detailed feedback. Please find our responses to your concerns below:

W1: The goal of the DNE component is to ``enforce more balanced prediction energy for imbalanced ID samples" as line 88, which is similar to BERL

BERL only balances the prediction energy at the sample level, while our DNE balances not only the sample level but also the class level, with the support of the novel batch energy normalization. This difference allows AdaptOD to achieve a much better energy balance and thus a better OOD detection and ID classification accuracy, as shown in Tabs. 2 and 3. It also provides stable energy margins across different datasets, eliminating the need of manual tuning of these margins as in BERL, helping a better adaptation during DODA.

W2: AdaptOD performs better in the near-OOD dataset CIFAR, which cannot be achieved by previous methods. It would be better to discuss the reason in further detail

Near OOD dataset CIFAR is similar to the training ID data, so previous methods are always confused between them. However, DODA can effectively utilize the test-time OOD data during inference to effectively adapt the outlier distribution and subsequently calibrate the global energy score. As a result, it helps AdaptOD discriminate near OOD samples from the ID samples better than previous methods. This is supported by the results in Fig. 1 in the attached PDF in the general response, where the performance of three TTA methods, including ours, increases significantly with an increasing number of OOD samples from CIFAR involved in the adaptation.

Q1: DNE is designed for more effective DODA, what's the most important factor in applying TTA methods on OOD detection in LTR during the training stage?

Energy regularization is a mainstream method to improve OOD performance, and in long-tailed OOD detection, it is used to alleviate the bias of the global energy toward head samples in existing studies such as BERL. But we observe that there is a class-level bias toward head classes, in addition to the sample-level bias. Thus, one major contribution from DNE is to enable more balanced energy predictions overhead and tail samples due to its sample- and class-level energy debiasing. Another important contribution lies at its Batch Energy Normalization, which addresses a largely ignored issue in long-tailed OOD detection that requires careful manual tuning of energy margins to work well on different ID/OOD datasets. DNE provides stable energy margins for different long-tailed ID/OOD datasets, alleviating the aforementioned biases toward head classes without relying on the data-specific manual margin tuning.

Q2: The updated form of the outlier distribution in Eq.4 is sample-wise, how does it compare to batch-wise updates?

In DODA we utilize the statistic of training ID data to guarantee the optimization of outlier distribution in Eq. 3 during the test phase, and the assigned pseudo label for each unlabeled test sample only depends on this statistic. Therefore, our predicted OOD samples used to adapt the outlier distribution once is sufficient for our method to work well. If batch-wise updates are used, it can help the optimization of outlier distribution become more stable at the beginning of DODA, with only minor difference in detection effectiveness, as shown in Tab. C and Tab. D below. Sample-wise dynamic updates as in Eq. 4 are more desired than batch-wise approaches in the pursuit of instant OOD detection, so we focused on the former one in this work.

Table C: Result of sample-wise and batch-wise update on CIFAR10-LT with AdaptOD.

Table D: Result of sample-wise and batch-wise update on CIFAR100-LT with AdaptOD.

Thank you for recognizing the novelty of our method and its improvement over existing methods. On the other hand, there might be some misunderstanding regarding the setting, which we will clarify in the following responses and further improve our writing.

W1: concern for the setting

• Test-time adaptation (TTA) for OOD detection

TTA is a widely used technique that utilizes test data (without knowing their class labels) to help models improve performance and adapt to real-world scenarios. Its application to OOD detection is relatively less explored, but it can help quickly adapt to new OOD scenarios and improve OOD performance due to the possibility of utilizing real OOD samples. This has been demonstrated in recent studies, AUTO and AdaOOD, and more recently published RTL at CVPR2024 [Ref1]. These justify the importance of designing TTA methods to enhance OOD methods. However, please note that the TTA module is assumed NOT to be able to get access to the ground truth of the test data in all these methods, including ours, meaning that it handles unlabeled test data. Ground truth of test data is used in detection performance evaluation only. Thus, there is no data (label) leakage issue.

• Unfair comparison to methods that could not access the true OOD data

It is true that our method and other TTA-driven methods have the advantages of accessing the samples drawn from true OOD distribution. To ensure a fair, comprehensive comparison, in Tab. 2, we compare our full method AdaptOD with DODA-enabled four SOTA, non-TTA methods. Furthermore, in Tab. 3, we compare our DODA with the two existing TTA methods for OOD detection, where each of them is respectively combined with different current OOD detection methods. Additionally, we further compare our AdaptOD with RTL [Ref1] in the general response. All of these experiments demonstrate the superiority of AdaptOD in long-tailed OOD detection.

• Small AUC improvement on ImageNet-LT compared to SOTA without the DODA module

Both our modules, DODA and DNE, make major contribution to the overall detection performance of our method AdaptOD. We agree that the DNE module leads to relatively small AUC improvement on ImageNet-LT, but it can consistently improve not only all OOD detection metrics but also ID classification accuracy across all three benchmarks. Additionally, DNE also provides stable energy margins on different datasets, eliminating the need of manual tuning of these margins.

[Ref1] Fan, Ke, et al. "Test-Time Linear Out-of-Distribution Detection." Proceedings of CVPR, 2024.

W2: the motivation of DODA

• Why don't we use the true OOD distribution to calibrate the scores?

There may be a setting misunderstanding. Although we can access the test data that contains the true OOD data, we do not know the true OOD distribution since we cannot access the test labels. Therefore, DODA assigns the pseudo labels to test data at first, and then uses the predicted OOD samples (maybe involving false predictions) to gradually estimate the true OOD distribution.

• The curves of the adapted outliers almost coincided with the true OOD

This occurs because of highly effective outlier distribution adaptation in our method. This is also justified by the results in Tab. 5, where AdaptOD has only a small performance gap to the Oracle model that utilizes the ground truth labels of the OOD test data to update the outlier distribution. This indicates that AdaptOD can well approximate the true OOD distribution by the predicted labels of the OOD samples.

W3: the mechanism of DODA. Wrongly predictions may lead to bad optimization of outlier distribution

Yes, it is true that the wrongly predicted OOD samples would lead to bad outlier distribution during TTA, as shown in Fig. 3. Therefore, our DODA designs a Z-score-based method in Eq. 3 to implement an OOD filter based on training ID data, where $\alpha=3$ (corresponding to 99% confidence interval) is set so as to utilize a predicted OOD sample as true OODs with very high confidence. There may be a few wrongly predicted OOD samples, but their influence is limited in DODA as they are often similar to true OOD data, and moreover, the optimization of the outlier distribution would be dominated by the most correctly predicted OOD samples. As a result, our AdaptOD can quickly adapt to the outlier distribution and perform very stably thereafter (see Fig. 3).

W4: the meaning of adapting the outlier distribution to true OOD distribution with specified OOD samples

Yes, OOD samples can be drawn from any unknown distribution in different deployment stages and/or applications. This is exactly the main motivation of our work that we aim to continuously adapt the outlier distribution by utilizing dynamic, unknown OOD samples in the target application scenarios to tackle this challenge. Our comprehensive results with different OOD data across three ID data benchmarks show that AdaptOD can generalize well regardless of the difference in ID or OOD datasets.

Q1: The experimental comparisons for Tab. 1 seem unfair

COCL and EnergyOE are previous SOTA methods, so we directly compare them in Tab. 1 to show the significant improvement of AdaptOD as a whole. For a fairer comparison, in Tab. 2 and Tab. 3, we report the average performance across six OOD datasets of Tab. 1 which combine COCL and EnergyOE with DODA and other TTA methods, so COCL and EnergyOE also can access the test-time true OOD data.

Q2: for Tab. 2 and Tab. 3, the previous methods with DODA modules obeying the OOD setting should be considered

DODA is a TTA method that can be combined with previous OOD methods to utilize the unlabeled test data for tackling the distribution shift problem. For a fair comparison, we have compared AdaptOD with the previous OOD detection methods enabled by DODA in Tab. 2, and we have also compared DODA with other TTA methods in Tab. 3 and the table above with the recent RTL.