Out-of-Distribution Detection with Negative Prompts

We sincerely thank you for your constructive comments! Please find our responses below.

Q1: Meta-Net in Figure 6 is not introduced in the paper.

A1: Thanks for pointing out the problem. Accoridngly, we have added the following explanations in our revision.

The Meta-Net in Figure 6 is a lightweight neural network built with a two-layer bottleneck structure (Linear-ReLU-Linear), with the hidden layer reducing the input dimension by 16×.

Q2: The motivation and model design of this paper are similar to DualCoOp.

A2: Thank for your insightful comments and kind suggestion. Accordingly, we have added the following explanations to clarify the difference bewteen our method and DualCoOp.

• The roles of negative prompts in DualCoOp and LSN are different. In DualCoOp, negative prompts act as a threshold for positive prompts of the corresponding category, whereas in LSN, negative prompts focus on learning generic negative features across all other categories except the corresponding category.
• In DualCoOp, positive and negative prompts are optimized simultaneously with Asymmetric Loss, which make the learning of negative logits intractable, resulting DualCoOp tend to generate false negative responses, which is consistent with the observation in DualCoOp++[1]. In contrast, in LSN, the learning of positive and negative prompts is separated, and the learned negative prompts successfully focus on different region than the positive prompts, as demonstrated in Figure2 in our paper.
• DualCoOp mainly focuses on the multi-label recognition (MLR), which is a closed-set problem. This needs class names of the unseen classes for testing. In contrast, LSN focuses on the OOD detection, which is an open-set problem. This does not need unseen classes.

In response to your constructive comments, we conducted the following experiments on ImageNet-100 to hilight the difference from an empirical perspective. The corresponding results and discussions have been added to the revised paper. These results show that LSN can outperform DualCoOp.

Q3: Some existing OOD detection methods proposed in 2022 and 2023 are not compared or discussed.

A3: Thanks for your valuable comments. Following your kind suggrestion, we conducted comparative experiments with KNN and LINe on raw CLIP model. The results on ImageNet-1k are as follows:

Our experimental results show that LSN outperforms these post-hoc methods on raw CLIP model. And the reason is the same as that in A1 to Reviewer Dyvx, please see that.

As the mentioned DML requires re-training the model, it is challenging to report detailed results. Thus, we will add the results when finishing our experiments. We have added these discussions and outstanding works in our revision.

Q4: It seems that the authors have submitted this paper with an ICLR 2023 template.

A4: Thank you very much for the kind reminder, in our resubmitted version we have fixed this error.

Q5: In equation(9), the positive score and the negative score are obtained independently. If the obtained positive category is different from the negative category, could this lead to mistakenly recognizing a correct positive prediction as out-of-distribution?

A5: We appreciate your careful review and indepth question. What you describe is exactly the reason we get positive score and negative score respectively. It can indeed happen that positive and negative categories are different, especially when the number of class is large. In this case, for ID samples:

$\max _c Softmax( \cos \left(\boldsymbol{f}(\mathbf{x}), \boldsymbol{g}\circ \boldsymbol{V}(l_c) \right) / \tau) - \min _c Softmax( \cos \left(\boldsymbol{f}(\mathbf{x}), \boldsymbol{g}\circ \bar{\boldsymbol{V}}(l_c) \right) / \tau) > \max _c (Softmax( \cos \left(\boldsymbol{f}(\mathbf{x}), \boldsymbol{g}\circ \boldsymbol{V}(l_c) \right) / \tau) - Softmax( \cos \left(\boldsymbol{f}(\mathbf{x}), \boldsymbol{g}\circ \bar{\boldsymbol{V}}(l_c) \right) / \tau))$ Getting positive and negative scores independently can alleviate the problem you describe and this can be seen as an integration of positive classifier and negative classifier.

Reference:

[1]: DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotations. Hu et al.

Dear Reviewer #ayMs,

Thanks very much for your time and valuable comments.

Here is a summary of our response for your convenience:

• (1) Missing information issues: We have added a description of Meta-Net in our revision.
• (2) Novelty issues: DualCoOp differs from LSN in the following three points:

Negative prompts of DualCoOp and LSN serve different purposes;

Negative prompts of DualCoOp and LSN are learned in different ways;

DualCoOp and LSN focus on different tasks.

And in OOD detection task, LSN outperforms DualCoOp.

• (3) Related work issues: Following your constructive comments, we have discussed related works including KNN, LINe and DML.
• (4) Score function issues: Inconsistencies between positive category and negative category can indeed occur. When it happens, the influence on ID data scores can be mitigated by using our scoring function.

We understand you're busy. But as the window for responding and paper revision is closing, would you mind checking our response and confirm whether you have any further questions? We are very glad to provide answers and revision to your further questions.

Best regards and thanks,

Authors of #1826

Dear Reviewer #ayMs,

Thanks a lot for your time in reviewing and insightful comments, according to which we have carefully revised the paper to answer the questions. We sincerely understand you’re busy. But since the discussion due is approaching, would you mind checking the response and revision to confirm where you have any further questions?

We are looking forward to your reply and happy to answer your further questions.

Best regards

Authors of #1826

Dear Reviewer #ayMs,

Thanks a lot for your time in reviewing and insightful comments. We sincerely understand you're busy. Would you mind checking the response and revision to see if you have any further questions?

We are looking forward to your reply.

Best regards,

Authors of #1826

Dear Reviewer #ayMs,

Thanks a lot for your time in reviewing and insightful comments. We sincerely understand you're busy. Would you mind checking the response and revision to see if you have any further questions?

We are looking forward to your reply.

Best regards,

Authors of #1826

We sincerely thank you for your constructive comments! Please find our responses below.

Q1: This paper is related to negative learning or learning with complementary labels. The authors may consider adding some related discussion in the related work section.

A1: Thanks for your constructive suggestion. We have added a section of Complementary Label Learning to the related work in our revised version

Q2: The proposed method may double the training and inference time with the negative prompts.

A2: We agree with your point. Due to the use of both positive and negative prompts, we indeed double our training and inference time. We will add the corresponding overhead analysis into our revision.

Q3: I found a related work that the authors may add discussion in the related work section: " How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models? , IJCV 2023."

A3: Thanks for mentioning the outstanding paper. This paper investigates the effect of fine-tuning on OOD detection in large vision-language models and the maximum concept matching (MCM) score is highlighted as effective. We have added the above discussion to the related work.

Q4: What's the ID dataset in Table 3?

A4: We apologize for the missing information. Accordingly, we have added the following explanations into our revised paper.

For the small-scale dataset, we follow the setting in ZOC[1]. For CIFAR10, 6 classes are selected as ID classes, and the 4 remaining classes are selected as OOD classes. Experiments are conducted on 5 different splits. For CIFAR+10, 4 non-animal classes from CIFAR10 are selected as ID classes, and 10 animal classes from CIFAR100 are selected as OOD classes. For CIFAR+50, 4 non-animal classes from CIFAR10 are selected as ID classes, and all 50 animal classes from CIFAR100 are selected as OOD classes. For CIFAR100, 20 consecutive classes are selected as ID classes. The remaining 80 classes are selected as OOD classes. For TinyImagenet, 20 classes are used as ID classes, and the remaining 180 classes are used as OOD classes. All the split is the same as ZOC. We give detailed split information in Table6.

Q5: Why do CoOp/CoCoOp and CoOp/CoCoOp + LSN achieve the same ID results in Table 1 and Table 2?

A5: Thanks for pointing out this potentially confusing description.

In our experiments we notice that the classification performance of negative prompts is slightly less than that of positive prompts. We are not committed to improving ID classification performance. To avoid further trouble, we only use positive prompts when determining the classes of ID samples. We have clarified this point in our revision.

Thanks again for your careful review.

Q6: The ID results of CoOp/CoCoOp appear to be significantly lower than other baselines such as NPOS in Table 2. Can the authors explain some reasons?

A6: We do not run NPOS on ImageNet-1k because it will consume a lot of resources. We get their experimental results directly from the original NPOS paper. We believe that the gap in ID classification performance comes from the difference in the amount of training data as well as the difference in training methods. NPOS uses the full ImgaNet-1k training set to fine-tune the CLIP model, which allows the fine-tuned model to fit better to the ImgaNet-1k test dataset. In contrast, LSN uses only 64 samples per class on ImageNet-1k and only learns prompts without making changes to the model.

Q7: 1: It is suggested to use \citep rather than \cite in the latex code; 2: Typo: we use we use at the bottom of page 6.

A7: Thank you very much for the kind reminder; in our resubmitted version, we have fixed these errors.

Dear Reviewer #ZZhi,

Thanks very much for your time and valuable comments.

Here is a summary of our response for your convenience:

• (1) Related work issues: In response to your constructive suggestions, we have added a discussion of the work mentioned in your comments and added a section on Complementary label learning to the related work.
• (2) Training and testing time issues: We have added a section of Limitations in the Appendix and discussed this issue.
• (3) Missing information issues: We describe Table3 in detail and give specific split information in the Appendix.
• (4) ID ACC issues: When we determine the class of ID samples, we only use positive prompts, resulting in the same ID results of CoOp/CoCoOp and CoOp/CoCoOp+LSN. And the reason NPOS achieves high ID ACC is that it uses the full ImageNet dataset to fine-tune CLIP.

We understand you're busy. But as the window for responding and paper revision is closing, would you mind checking our response and confirm whether you have any further questions? We are very glad to provide answers and revision to your further questions.

Best regards and thanks,

Authors of #1826

Dear Reviewer #ZZhi,

Thanks a lot for your time in reviewing and insightful comments, according to which we have carefully revised the paper to answer the questions. We sincerely understand you’re busy. But since the discussion due is approaching, would you mind checking the response and revision to confirm where you have any further questions?

We are looking forward to your reply and happy to answer your further questions.

Best regards

Authors of #1826

We sincerely thank you for your constructive comments! Please find our responses below.

Q1: The overall prompt learning approach is still based on CoOp without a lot of innovation.

A1: Thank you for your valuable comment. We agree with your point that LSN is built upon the success of CoOp, while we would like to provide more explanations to highlight the novel design for OOD detection.

Although our approach to learn negative prompts looks basically the same as CoOp, there is a fundamental difference between positive prompt learning and negative prompt learning. In positive prompt learning, using a shared prompt across classes is enough to achieve good performance. This is because in positive prompt learning, the positive features of each class are carried by the class names, and the positive prompt is only used to calibrate these features to the downstream dataset. In negative prompt learning, on the other hand, the situation is completely different. The negative features need to be contained in the learned negative prompts, and the class names appear to be less important. Diversity is the key to the success of negative prompts. We illustrate this in Table 5.

Q2: If the features extracted by CLIP itself for some categories of images are not strongly discriminative, the effect of learning the prompts based on these features may be poor.

A2: We wholeheartedly agree with your observation and acknowledge that this situation is indeed inevitable in the current framework. Inspired by your insightful comments, we will explore the promising direction, like finetuning a "degenerated" CLIP model for promoting OOD detection.

Q3: The way to learn negative prompts is to mine the general negative features that each class of samples does not have but all other classes have, i.e., the negative classifier produces low activation values for that class and high activation values for other classes. Does this result in learning what are actually generic background features rather than general features that all other classes have?

A3: We agree with your indepth point. It is indeed possible for the model to learn generic background features as negative features. However, it is noteworthy that even when background features are learned, they can at times serve as effective negative features. For example, consider the ID classes Horse, Wolf, and Pet Cat, and the OOD class Tiger. Due to similar positive features with pet cats, tigers can easily be mistaken for ID samples. But if we learn further about negative features the situation is different. Since horses and wolves are often found in grassland and pet cats are unlikely to be found in grassland, then the background grassland is likely to be learned as a negative feature for pet cats. The background of tigers is also usually grassland, and then the classifier can easily determine that tigers are not pet cats.

Dear Reviewer #r91y,

Thanks very much for your time and valuable comments.

Here is a summary of our response for your convenience:

• (1) Novelty issues: Positive prompt learning is fundamentally different from negative prompt learning. Negative features need to be included in the learned negative prompts. Class names do not provide valid negative features.
• (2) Validity issues: It does happen that the learned negative features are background features. However, background features are also useful for OOD detection.

We understand you're busy. But as the window for responding and paper revision is closing, would you mind checking our response and confirm whether you have any further questions? We are very glad to provide answers and revision to your further questions.

Best regards and thanks,

Authors of #1826

Dear Reviewer #r91y,

Thanks a lot for your time in reviewing and insightful comments, according to which we have carefully revised the paper to answer the questions. We sincerely understand you’re busy. But since the discussion due is approaching, would you mind checking the response and revision to confirm where you have any further questions?

We are looking forward to your reply and happy to answer your further questions.

Best regards

Authors of #1826

We sincerely thank you for your constructive comments! Please find our responses below.

Q1: Comparison and related works to state-of-the-arts are missing. (e.g., NNGuide, ASH, CLIPN)

A1: Thanks for your contructive comments. In accordance with your suggestion, we conduct comparison experiments with NNGuide and ASH to enrich our analysis. The results on ImageNet-1k are as follows:

Our experimental results show that LSN outperforms these post-hoc methods on raw CLIP model. We believe the reason for the poor performance of these pos-hoc methods is that the the difference in training data. Many pos-hoc methods are designed on ImageNet pre-trained networks, where only ID datas are used during training. In contrast, when training CLIP, both ID datas and OOD datas are used. The difference in training data leads to different activation of OOD data. Another reason is that the pos-hoc method relies heavily on the choice of hyperparameters. The hyperparameters need to be re-selected on different models.

CLIPN also employes CLIP to perform OOD detection, coming up with ideas similar to ours. We compare the best results of CLIPN and LSN on ImageNet-1k with CLIP ViT-B-16 as follows:

• Experimental results show that LSN achieves comparable performance with CLIPN.
• We have to point out that this is not a fair comparison. This is because, to endow OOD detector with the ability to say "no", CLIPN leverages a large-scale dataset [1] (containing 3.3 million image-text pairs) to finetune the text encoder.
• These results highlight the contribution of the proposed approach to learn negative prompts. Specifically, LSN only needs to use few shot samples to learn negative prompts, which can be done in a very short time.

Inspired by your constructive comments, we summarize the differences between LSN and CLIPN as follows.

• LSN only requires learning negative prompts at a small computational cost without training "no" text encoder and achieves performance comparable with CLIPN.
• LSN points out that the diversity of negative prompts is the key to success, and the use of class-specific negative prompts leads to a reduction in FPR95 from 81.27% to 21.94% compared to the use of class-shared negative prompts, please see Table 5.
• LSN focuses on prompt learning and points out how negative prompt learning differs from positive prompt learning, while CLIPN focuses more on learning "no" text encoder.

We will add the above results and discussion into our revision to highlight the difference bewteen our work and the mentioned work. Thanks again for mentioning these outstanding papers.

Q2: The performance is too behind the state-of-the-art.

A2: Thanks for your valuable comments. Accordingly, we have added the following explanations to our revision.

• We are focused on improving the OOD detection performance of pre-trained vision-language models such as CLIP. As can be seen in A1, our method can outperform NNguide and ASH, the recent SOTA pos-hoc methods.
• Moreover, when compared to CLIPN, our method achieves comparable performance without the need for large-scale datasets and significantly computation overhead.

Q3: The performance improvement is very marginal.

A3: As mentioned above, we get the comparable performance at a much smaller cost compared to CLIPN. And compared to standard prompt learning CoOp and CoCoOp, LSN's FPR95 on ImageNet1k decreased by 6.86% and 8.41%, respectively.

Reference: [1] Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. Sharma et al. ACL2018.

Thank you for your prompt responses! Please find our responses below.

Q1: By 'fine-tuning' I mean fine-tuning of the learnable input prompt weights (i.e., 'prompt learning' by LSN). I assume that the set $S$ in the subscript of the expectation in Eqs. (4) and (5) are your training set for prompt learning. As far as I understand, based on your responses comments, this set is not the full ImageNet-1k training set. If you increase the size of $S$ to the full train fold of IN-1k, then would it correspondingly increase the performance, and achieve the SOTA? (By the way, the notations are quite confusing. $S$ is elsewhere defined as the score function while in the Eqns (4) and (5) it seems not the case.)

A3: Thank you for the heads up, we have fixed this error in the revised version. We are afraid that even using the full ImageNet-1k dataset we could not achieve the SOTA you mentioned. We think the main reason is that there are too few parameters that can be learnt in prompt learning compared to fine-tuning networks, and further increasing the number of training samples will not lead to performance improvement. We don't use the full ImageNet-1k dataset to learn the prompt due to the fact that it takes too long even if only the prompts are tuned. We estimate that the program takes 15 hours to run one epoch (batchsize is set to 8), which is intolerable. We provide the effect of different taining sample sizes on the OOD detection performance on ImageNet-100 dataset in the Appendix. You can refer to that. It can be seen that when the sample size is large, the performance gain from further increasing the sample size is not significant.

Q3: Hopefully, the LSN works on BLIP2 as well; I am expecting the result.

A3: We apologize for the problems we have when we are about to validate LSN on BLIP2. The model structure of BLIP2 consists of three types: blip2, blip2_t5 and blip2_opt (Please see details here: BLIP2). When we use the smallest model blip2, we find that the model still can not understand the meaning of "not". Only when using the larger model blip2_t5 the model is able to understand the meaning of "not". However, blip2_t5 is only used in image-to-text generation task and caption generation task. We fail to find a suitable prompt that allows BLIP2 on these tasks to generate a suitable score for testing the model's OOD detection ability. Therefore, we are still only able to use smaller model to perform image-text matching task to get cosine similarity between images and text as the OOD scoring function. In this case, we need to learn negative prompts through prompts learning. To this end, we test the performance of LSN on ImageNet-100 with BLIP. The result is as follows:

Dear reviewer #Dyvx,

Thanks for your swift reply despite such a busy period. We sincerely appreciate that you can raise the score. For the experiments on the BLIP, we believe that the results above are not the best results as we don't explore much due to time constraints. We will do more experiments on BLIP and present the results in the paper. As for Q1, it may be a fact at this stage that when the ID dataset is ImageNet-1k, the OOD detection performance of prompt learning on CLIP may not be as good as the performance of ImageNet-1k pre-trained models. For now, this is indeed a shortcoming of such methods based on CLIP models. We will make this clear in our paper.

Best regards and thanks,

Authors of #1826