AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models

Dear Reviewer eV4A，

We sincerely thank you for the constructive comments and recognition on our work! Please find our responses below.

Q1: Potential Overhead in Memory Management: The implementation of a memory bank for caching features may introduce significant overhead in memory management, especially when dealing with large-scale datasets or high-dimensional feature spaces.

A1: Thanks for the question. Indeed, the introduction of a memory bank introduces additional memory overhead, which we have briefly discussed in Section 5 of the main paper. However, we clarify that our memory overhead does not continuously increase with the scale of the dataset. This is because we drop image features with high prediction entropy when the memory bank is full, as detailed in Line 184 and Appendix A.2 of our submission

For a typical high-dimensional feature of 512 dimensions and a maximum memory length $L$ of 10 for each class, our memory bank occupies a storage space of 214.75MB when using the ImageNet dataset as ID. This storage requirement is negligible compared to the memory consumption of the CLIP model during forward passes.

Q2: Generalization to Other Domains: Although the approach demonstrates promising results on existing public datasets, its effectiveness in other domains or with different types of data remains uncertain and requires further investigation

A2: Thanks for the nice suggestion. We further validate our method on the BIMCV-COVID19+ dataset [a], which includes medical images, following the OpenOOD setup [68, 71]. Specifically, we selected BIMCV as the ID dataset, which includes chest X-ray images CXR (CR, DX) of COVID-19 patients and healthy individuals. For the OOD datasets, we follow the OpenOOD setup and use CT-SCAN and X-Ray-Bone datasets. The CT-SCAN dataset includes computed tomography (CT) images of COVID-19 patients and healthy individuals, while the X-Ray-Bone dataset contains X-ray images of hands. As illustrated in the table below, our AdaNeg method consistently outperforms NegLabel on this medical image dataset. We will add these analyses in the revision.

[a] BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients

Q3: Testing Phase Dependency: It is unclear whether the approach can maintain the same level of reliable performance when only a small number of images are tested in practical applications. This dependency on the number of test images warrants additional examination.

A3: Thank you for your suggestion. We examined the dependency of our approach on the number of test images by evaluating its performance across different scales of test samples. As the number of test samples increases (from 900 to 90K), the cached feature data also increases, leading to an improvement in our method's results, as shown in the table below. Even with a small number of test samples (e.g., 90 and 900), our method significantly reduces FPR95 compared to NegLabel, demonstrating its robustness across different numbers of test images.

Note that with only 90 test images, the task of distinguishing between ID and OOD samples degenerates into a simpler task since the number of test images is even smaller than the number of classes (e.g., 1000 for ImageNet). Consequently, both NegLabel and our method achieve lower FPR95 in such an easier scenario.

We will add these findings to the revised paper.

Table Caption: FPR95 ($\downarrow$) with different numbers of test images, where test samples are randomly sampled from ImageNet (ID) and SUN (OOD) datasets while maintaining their relative proportions.

Dear Reviewer Pmx9，

We sincerely thank you for the constructive comments and recognition on our work! Please find our responses below.

Q1: Analyses on the stability of our method with different ID and OOD sample mixture ratios

A1: Many thanks for the detailed and valuable comments. Following this reviewer's suggestion, we investigated the stability of our method by constructing test sets with various mixture ratios of ID and OOD samples. Specifically, we adopted the 1.28M ImageNet training data as ID and randomly sampled 12.8K and 1.28K instances from the SUN OOD dataset to construct the ID:OOD ratios of 100:1 and 1000:1 settings, respectively. To construct the ID:OOD ratios of 1:100, 1:10, 1:1, and 10:1 settings, we used the full 40K SUN dataset as OOD and randomly sampled 400, 4K, 40K, and 400K instances from the ImageNet training data. We did not validate the setting where the test set contains only ID data, as the absence of OOD data makes it impossible to calculate evaluation metrics (e.g., FPR95 and AUROC).

As shown in the table below, our method outperforms NegLabel across a wide range of mixture ratios (from 1:100 to 100:1), validating the robustness and reliability of our approach. As pointed out by this reviewer, unbalanced mixture ratios do pose a challenge to our method. Our approach performs the best in scenarios with a balanced mixture of ID and OOD samples, reducing the FPR95 by 11.18%. As the mixture ratio becomes increasingly unbalanced, the improvement brought by our method gradually decreases. When the unbalanced ratio reaches 1000:1, our method shows some negative impact. We will include these analyses in the limitation part of our revised manuscript, and attempt to address this challenging setting in future work.

Table Caption: FPR95 ($\downarrow$) with different mixture ratios of ID and OOD samples.

Q2: The necessity of negative labels in the initialization of the memory bank.

A2: Thanks for the insightful questions. Our use of negative labels to extend the memory bank is carefully designed for effective implementation. Initially, our memory bank is empty, containing only zero values. Consequently, the derived negative proxies are also zero vectors. In other words, during the early stages of testing, it is impossible to generate effective negative proxies solely based on the memory bank. To enable our method to be operational from the beginning of the testing phase, we extend the memory bank with negative labels. It is important to note that while negative labels play a crucial role during the early stages, their influence diminishes as the memory bank gradually accumulates data, and the negative proxies will be progressively dominated by cached negative images.

Regarding the reviewer's suggestion to use the MCM score to judge negative samples and organize them into OOD proxies, this is indeed feasible. One could cluster the negative samples selected by the MCM score and use the cluster centers as OOD proxies. However, this method has a significant drawback: it requires detecting a portion of negative samples before applying our method. In other words, this approach cannot be applied online to the initial test data. From this perspective, we utilize the negative labels as predefined initial cluster centers in our AdaNeg. During testing, these cluster centers are gradually refined as more image features are cached into the memory.

Considering that negative labels are words semantically distant from ID labels, if OOD samples are similarly distant from both ID and negative labels, they typically fall into the intermediate hard examples between ID and negative labels. Handling such intermediate hard examples is a long-standing challenge for all methods, including NegLabel and our AdaNeg. Our AdaNeg partially addresses these hard examples via an easy-to-hard strategy. Specifically, our method first caches easy OOD samples, which are closer to negative labels and farther from ID labels, into the memory. These easy OOD samples, being closer to the intermediate hard OOD examples, can serve as bridges to facilitate the detection of hard OOD samples. The effectiveness of our method is validated by the improved OOD detection performance shown in Table 2 of the main paper.

Q3: More detailed experimental settings.

A3: Thanks for the kind suggestion. In our experiments, we maintain an independent memory bank for each OOD dataset (i.e., each ID-OOD pair). In implementation, we clear the memory bank when switching to a new OOD dataset.

We validated performance using a shared memory bank across four OOD datasets: iNaturalist, SUN, Places, and Textures. In this setup, the memory bank retains features from previous OOD datasets when testing a new one. As shown in the table, shared memory banks outperform independent ones, suggesting that cached features aid in recognizing new OOD datasets. However, shared memory banks may leak information between datasets, which can be problematic in practice. Therefore, we use independent memory banks by default.

Table Caption: FPR95 ($\downarrow$) with different types of memory banks.

Q4: Typos and symbol issues

A4: Thank you for the kind correction. We will correct $\tau$ = 0.01 in L247 and unify the subscript NL as $S_{nl}$ in the revision. We will further proofread the manuscript carefully.

Dear Reviewer ACdF，

We sincerely thank you for the constructive comments and recognition on our work! Please find our responses below.

Q1: While the proposed AdaNeg shows clear improvements over training-free baselines, its overall performance on ImageNet still lags behind training-based methods. This raises the question of whether there are opportunities for complementarity between the two approaches.

A1: Thanks for the nice suggestion! We validated the complementarity between our AdaNeg and the existing state-of-the-art method NegPrompt [31], which is characterized by learnable negative prompts. We reproduced the results of NegPrompt with the author released codes. As shown in the table below, our method brings significant performance improvements over NegPrompt, validating the complementarity between our approach and training-based methods. We will include these analyses in the revision.

Table: FPR95 ($\downarrow$) of AdaNeg based on the training-based NegPrompt method.

Q2: Can the dynamic update of the memory bank and refinement of OOD proxies during the testing stage be considered a form of test-time training? The authors are requested to clarify the inherent connections and distinctions, especially from the perspectives of training versus training-free approaches.

A2: Thanks for the suggestions. Our online update of the memory bank and refinement of OOD proxies is a kind of training-free test-time adaptation. Unlike existing training-required test-time adaptation methods [13, 15, 69], which typically require test-time optimization and subsequently slow down the testing process, our approach is optimization-free. It introduces only a lightweight memory interaction operation, enabling rapid and accurate testing, as analyzed in Table 4. We will further clarify this point in the revision.

Q3: If negative proxies can directly identify true out-of-distribution (OOD) test images during the testing phase, is it possible to use the identified OOD samples to update the model parameters online?

A3: Thank you for the comments. We performed experiments to use the identified OOD samples to update the text prompt with cross-entropy loss, similar to the training objective in [1]. However, it is unstable and often leads to model collapse, especially in challenging settings. For example, in the ImageNet (ID) and SSB-hard (OOD) setup, around 44% of OOD samples were misclassified as ID, disrupting model learning.

Our training-free method is more robust to such misclassifications due to the weighted combination of cached features. Additionally, updating model parameters online would significantly slow down the testing process. In contrast, our method ensures rapid and accurate testing with only lightweight memory interaction.

Dear Reviewer yxd9，

We sincerely thank you for the constructive comments! We hope our following responses can address this reviewer's concerns.

Q1: The motivation in this paper is not very clear. Specifically, in Figure 1(a), it is not evident why the newly proposed AdaNeg is better than NegLabel. On the contrary, the distribution of OOD samples seems to be closer to NegLabel.

A1: Thank you for pointing out this issue. In the visualization of our submission, the adaptive negative proxies are not $L_2$ normalized, whereas the features of ID, OOD, and NegLabel are normalized vectors. This makes the visualization not very clear. We have corrected this issue by revising the visualization with normalized adaptive negative proxies. As expected, the distribution of adaptive negative proxies is much closer to the ground truth OOD labels than to the negative labels in NegLabel. We have attached an updated Figure in the PDF of the Global Response and will revise Figure 1(a) in the manuscript.

Q2: The method proposed in this paper is based on the features and results of test samples during testing, which limits the upper bound of the method. In my opinion, the effectiveness of the proposed method relies on the vision-language model's strong inherent OOD detection capability, meaning that most test samples can be correctly processed. Based on these correctly processed samples, the method can further improve the detection accuracy of other samples. However, if in a certain scenario, the model itself cannot correctly estimate most of the samples, this method might actually make the results worse.

A2: Thanks for the comments. We agree that our method leverages the inherent strong capability of vision-language models (VLMs). However, we would like to emphasize that our method is highly robust to misclassified samples.

For instance, we investigate a challenging task setting with ImageNet as the ID dataset and SSB-hard [58] as the OOD dataset, where NegLabel achieves an FPR95 of 77.26%. With a threshold $\gamma = 0.5$ and a gap $g = 0$, about 44% of OOD samples were incorrectly stored in memory slots for ID classes. In this scenario, our method achieves an FPR95 of 72.90%, showing a 4.36% improvement over NegLabel. This robustness comes from the weighted combination of cached features used to form the proxies, which resists interference from a few misclassified features.

In a worst-case scenario, where test samples are randomly cached into ID or OOD memories (50% OOD misclassified), our method achieves an FPR95 of 77.69%, comparable to NegLabel's 77.26%. Such extreme confusion is unlikely in real-world scenarios, where there are usually some clues to distinguish between ID and OOD samples.

Q3: This paper merely performs optimizations based on the NegLabel framework, without many innovative points. The novelty of this improvement is insufficient to support a NeurIPS paper.

A3: We appreciate this reviewer's comments; however, we'd like to argue that our method introduces significant improvements to the NegLabel framework and makes notable contributions to negative proxies guided OOD detection. Our contributions and distinguished features are as follows:

• We identify the label space misalignment between existing negative-label-based proxies and the target OOD distributions. To address this issue, we dynamically generate adaptive negative proxies to align with the OOD label space more effectively. This is a "novel approach" and a "significant advancement," as recognized by Reviewers ACdF and eV4A.
• We construct adaptive negative proxies using a feature memory bank that incorporates carefully designed write and read strategies. Additionally, we propose a novel multi-modal score that combines complementary textual and visual knowledge, which is an "interesting design" as highlighted by Reviewer Pmx9.
• Our method is simple yet effective (cf. Reviewer ACdF), training-free, and annotation-free (cf. Reviewer eV4A). It exhibits good scalability (cf. Reviewers ACdF and eV4A), has been comprehensively evaluated (cf. Reviewers yxd9 and eV4A), and achieves substantial performance improvements (cf. Reviewers yxd9 and eV4A).