Negative Label Guided OOD Detection with Pretrained Vision-Language Models

W1: OOD detection methods have different performance on different datasets. On some datasets, the performance is extremely good. Can we see the difference among these datasets? Why can OOD detection be easily addressed on some datasets?

A1: Thanks for your comments! We think that the performance differences mainly stem from the characteristics of the OOD datasets themselves. In Figure 4 and Figure 5 of the Appendix, we provide a few sample images from the ID and OOD datasets for reference. The ID dataset ImageNet contains a large number of categories related to animals and food, while iNaturalist consists of images of natural plants. SUN and Places datasets contain images of natural landscapes. Therefore, compared to other OOD datasets, iNaturalist has the largest differences in category terms compared to ImageNet, such as animals vs. plants. Therefore, introducing negative labels can significantly improve the OOD detection performance. On the other hand, SUN and Places datasets often contain multiple elements from the natural world beyond their annotated ground truth, making them more prone to confusion with the ID data.

Furthermore, recent research trends have also shown that achieving high performance on iNaturalist is quite common. For example, ASH [1] and NNGuide [2] both achieve over 97% AUROC on iNaturalist, while GEN [3] achieves 99.13% AUROC.

W2: Especially for results based on CLIP-B/16 with various ID datasets, the performance is too high. More explanation is need.

A2: Thanks for your suggestions! We follow MCM to conduct experiments on the large-scale benchmark in Table 1 and small-scale benchmarks in Table 2. The performance of MCM in Table 2 is very high and our NegLabel outperforms MCM. Our opinion about the high performance is consistent to MCM, i.e., the small-scale ID datasets have less challenges for VLMs.

W3: Figures might be better to illustrate the final performance in Section 4.

A3: Thanks for your suggestions! We will replace some tables with figures in Section 4 in the updated paper.

W4: What is the relationship between Bernoulli distribution and binomial distribution? This is a well-known result in statistics?

A4: The Bernoulli distribution is a special case of the binomial distribution, where n = 1. Symbolically, X ~ B(1, p) has the same meaning as X ~ Bernoulli(p). Conversely, any binomial distribution, B(n, p), is the distribution of the sum of n independent Bernoulli trials, Bernoulli(p), each with the same probability p.[4]

W5: Why is B.3 more general than things demonstrated in main context?

A5: Thanks for your comments! In the theoretical analysis of the main text, to simplify the theoretical model and improve the readability of the article, we assume that the probability of a sample $x$ matching with each negative label is $p$. Therefore, for $M$ negative labels, we consider the number of matches between an image $x$ and them to follow a binomial distribution $B(M, p)$. However, due to the large number of negative labels, covering a wide semantic space, their affinity with the sample $x$ is variable.

Specifically, in Appendix B.3, we assume that the probability of an image $x$ matching with a negative label is $p_i$, where different negative labels have different probabilities. This better reflects the real-world scenario, and thus we consider the number of matches between an image $x$ and them to follow a Poisson binomial distribution.

We will provide more details in the main text to help readers better understand the general case in B.3.

W6: Please keep the naming strategy consistent in Algorithm 1 and other algorithms.

A6: Thanks for your suggestions! We will revise the algorithms to keep the naming strategy consistent in the updated paper.

Reference

[1] Djurisic, Andrija, et al. "Extremely Simple Activation Shaping for Out-of-Distribution Detection." The Eleventh International Conference on Learning Representations. 2022.

[2] Park, Jaewoo, Yoon Gyo Jung, and Andrew Beng Jin Teoh. "Nearest Neighbor Guidance for Out-of-Distribution Detection." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Liu, Xixi, Yaroslava Lochman, and Christopher Zach. "GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[4] https://en.wikipedia.org/wiki/Binomial_distribution#Bernoulli_distribution

W1: Figure 1 is clear and important. However, it makes me think about one question: in a statistical view, what should the right subfigure be?

A1: Thanks for the comments! The actual subgraph would be very difficult to visualize, as the x-axis would contain 1,000 (ID labels) + 10,000 (negative labels) elements. We have provided the visualization results of the top 5 ID labels and negative labels similarities in Figures 4 and 5 in the appendix.

W2: In Algorithm 1, the comments are too long. It would be better to say the key functionality of this part.

A2: Thanks for your suggestions! We will revise the comments in Algorithm 1.

W3: Why does grouping strategy appear in Section 3.2 instead of 3.1? It seems that Grouping Strategy is a more advanced way to pre-process negative labels?

A3: Thanks for this question. Here we detail our proposed pipeline of Negative-Label-based OOD detection: corpus -> NegMining -> Grouping -> NegLabel Score. Specifically, our method requires first selecting appropriate negative labels through NegMining (as shown in Section 3.1), and then designing a NegLabel score using these negative labels (as shown in Section 3.2 part 1). The Grouping strategy is an optimization of the NegLabel score. By grouping the selected negative labels and calculating the NegLabel score for each group, the average of all groups is taken as the final OOD score. Therefore, the Grouping Strategy is not a more advanced way to preprocess negative labels. It does not involve the preprocessing process of negative labels. It is suitable to be placed in Section 3.2 part 2 as it optimizes and enhances the OOD score.

To better understand the grouping strategy, we provide a toy example. Let's assume for an ID image, there is a relatively high response in the ID labels, but coincidentally, there is also a relatively high response in the negative labels, with both responses being of comparable magnitude. Without grouping, after softmax function, the probabilities for ID and negative labels might both be around 0.5, resulting in a score of 0.5.

However, if we divide the negative labels into 10 groups, this misclassification would be included in one of the groups at most. The score for this group would be 0.5, while the rest of the groups would be close to 1.0. Thus, the final score would be calculated as 1 * 0.9 + 0.5 * 0.1 = 0.95. This illustrates how the Grouping Strategy helps mitigate the impact of such misclassifications.

W4: Can the authors explain why is the case considered in B.3 more general? More explanation can make readers know your analysis better.

A4: Thanks for your comments! In the theoretical analysis of the main text, to simplify the theoretical model and improve the readability of the article, we assume that the probability of a sample $x$ matching with each negative label is $p$. Therefore, for $M$ negative labels, we consider the number of matches between an image $x$ and them to follow a binomial distribution $B(M, p)$. However, due to the large number of negative labels, covering a wide semantic space, their affinity with the sample $x$ is variable.

Specifically, in Appendix B.3, we assume that the probability of an image $x$ matching with a negative label is $p_i$, where different negative labels have different probabilities. This better reflects the real-world scenario, and thus we consider the number of matches between an image $x$ and them to follow a Poisson binomial distribution.

We will provide more details in the main text to help readers better understand the general case in B.3.

W1: In Eq. (5), the cosine similarity is selected. Although the reason is given in the paper, it is still interesting to see if we can have other choices.

A1: Thanks for your suggestions! We tried using L1 distance and KL divergence to measure the similarity between images and negative labels, and the results are shown below. As the CLIP-like VLM models are trained under cosine similarity supervision, the features are naturally measured in the cosine space.

Observing the experimental results, it can be seen that when using L1 distance and KL divergence as metrics, there is a significant drop in performance on SUN, Places, and Textures datasets, while the impact on iNaturalist dataset is relatively small. This is because our method (based on cosine similarity) achieves a 99.49 AUROC on iNaturalist, almost completely distinguishing between ID and OOD data. Therefore, even when using metrics such as L1 and KL divergence that are not suitable for cosine space, there is still a significant difference between ID samples and OOD samples from iNaturalist. For more discussions on iNaturalist, please refer to our response to reviewer 9gus in Reply A1.

W2: I enjoy reading how you move from basic data modelling to a more realistic data modelling in B.3. However, it would be better to point out why the contents in B.3 is more general, which can increase the interests of readers for understanding the proposed score.

A2: Thanks for your comments! In the theoretical analysis of the main text, to simplify the theoretical model and improve the readability of the article, we assume that the probability of a sample $x$ matching with each negative label is $p$. Therefore, for $M$ negative labels, we consider the number of matches between an image $x$ and them to follow a binomial distribution $B(M, p)$. However, due to the large number of negative labels, covering a wide semantic space, their affinity with the sample $x$ is variable.

Specifically, in Appendix B.3, we assume that the probability of an image $x$ matching with a negative label is $p_i$, where different negative labels have different probabilities. This better reflects the real-world scenario, and thus we consider the number of matches between an image $x$ and them to follow a Poisson binomial distribution.

We will provide more details in the main text to help readers better understand the general case in B.3.

W3: In Table 2, all methods can obtain good results (AUROC>90% or FPR95<10% in the most cases). I actually do not see the necessarity to put that table in the main content. It would be better to move some interesting parts in Appendix to the main content.

A3: Thanks for your suggestions! We will reorganize our paper in the updated version.

W1: Assumption about CLIP's latent space The assumption that ID image and corresponding ID label embeddings are close in CLIP's latent space is actually false. The contrastive learning strategy of CLIP is based on cosine similarity and thus textual and visual embeddings are actually on different manifolds see (Weixin Liang et al. 2022) for more details. Using ID labels as proxies for the ID images may thus produce unexpected results. In my understanding, the negative labels are compared to the images using cosine similarity, and thus having them close to ID textual embeddings still makes sense but the authors should be cautious here.

A1: Thank you for this valuable comment. We apologize for the inappropriate description. As mentioned in the question, images and text belong to different manifolds in the embedding space of CLIP, so they cannot be considered close. We will revise this description in the last paragraph of Section 3.1.

Besides, we would reiterate our OOD detection setting based on CLIP-like VLM models. We only work with a pre-trained CLIP-like model and a specific set of class labels or names for a zero-shot classification task. And the ID images are not available under this zero-shot setting. Hence，using ID labels as the proxies of ID images for negative label selection is intuitive, and it is experimentally verified to be effective in our method. We believe that not relying on the accessibility of ID images allows our method to be used in a wider range of scenarios and is more consistent with the working conditions of CLIP-like models.