Enhancing Near OOD Detection in Prompt Learning: Maximum Gains, Minimal Costs

We would like to clarify the reviewer’s comments.

1. “Lack of novelty: The contribution of the paper is limited. All I could discover is a post-hoc OOD score. More description and deeper analyses are needed to certify its usefulness.”

We respectfully believe that it is incorrect and unfair to consider the contribution of our method limited due to its post-hoc nature. One of our key contributions is a simple post-hoc OOD score that improves near-OOD detection by up to 11.67% for any CLIP-based prompt-learning model, without the need for retraining or model modification, and with minimal additional computation. To our knowledge, no previous works have addressed the problem studied in the paper. The usefulness of our method is clearly demonstrated by the comprehensive experiments with detailed analysis. If the reviewer has more specific comments on this, we are happy to address them.

2. “Incomplete comparison with current methods: The paper mainly compares with few-shot based methods but not OOD-based methods for VLM [1, 2, 3], which should be addressed.”

We would like to emphasise that our focus is improving near OOD detection performance of few-shot CLIP-based prompt learning models without retraining or making any change in the model. As discussed in L514-527, we intentionally excluded OOD methods that require access to OOD samples during training, retraining of prompt learning models, or that are incompatible with fine-tuned CLIP-based prompt-learning models. In other words, these methods do not fit the purpose. Nevertheless, we also have provided comparisons with CLIPN which is one of references the reviewer pointed out, RMD, and LogitNorm in Appendix A.3.6 and A.3.8.

3. “Marginal improvements: The performance gains in the main tables are marginal, even when compared with the basic MaxLogit.”

We could not agree with the reviewer on this comment, as our approach demonstrates consistent improvements across 13 datasets and 8 recent prompt-learning models with a minimal post-hoc process. For example, the average AUROC across datasets for LoCoOp, the state-of-the-art method for far-OOD detection in CLIP-based prompt-learning models, is increased by 4.18%, while its average FPR95 is reduced by 9.13%. We believe that the improvement is significant in the literature.

4. “The writing of the paper should be improved and standardized, including but not limited to the usage of quotation marks in Sec. 2.1 and the usage of superscripts and subscripts in Sec. 3.1, which may confuse readers.”

We have changed the quotation marks to the standard quotation marks in the revised version. Regarding Section 3.1, we believe that superscripts and subscripts have been used consistently. However, if the reviewer could point out any specific inconsistencies, we would be happy to address them.

5. “The comparison is not fair. In Table 2, the paper merely compares with prompt-based methods and achieves seemingly significant improvements. However, as shown in Table 3, the improvement becomes marginal when assessed using different metrics and OOD-based methods.”

We would like to emphasise once again that our work is specifically tailored for fine-tuned CLIP-based prompt-learning methods and is not applicable to other zero-shot CLIP models. Therefore, we believe that comparisons with 8 recent prompt-learning models are appropriate. Additionally, Table 3 demonstrates that our method consistently outperforms MCM in both AUROC and FPR95. For example, the average FPR95 for MaPLe is reduced by 12.32% when comparing MCM to MLS-M.

6. “The writing may be somewhat confusing. For example, in the near OOD setting, which dataset is the ID dataset? I did my best but could not find any description.”

Please refer to L373-377 where we provided details of ID and OOD data settings in our original submission. We used the same base-to-new splits used in prompt learning models (Zhou et al., 2022a; Khattak et al., 2023a; Yao et al., 2023; Zhu et al., 2023; Khattak et al., 2023a;b; Miyai et al., 2023) as near OOD tasks.

7. “Based on the results shown for the given datasets, the effectiveness of the method is not convincing. I suggest that the authors provide more visualizations and additional results to demonstrate its effectiveness and generalization.”

In our original submission, we provided extensive empirical evaluations using 8 recent prompt learning models, 13 main datasets for near OOD detection, 5 few-shot settings (16-, 8-, 4-, 2-, and 1-shot), 3 random seeds, 1 additional dataset for near OOD detection, 4 far OOD datasets, and comparisons with zero-shot CLIP OOD detection, LogitNorm, and RMD, provided in the main text and Appendix. We kindly ask the reviewer to specify which visualisations or additional results the reviewer believes are missing.

I totally understand what the author means by mentioning the definition of the ID and OOD datasets. However, I do not agree with the author's opinion that previous method is inapplicable in existing prompt-based methods. Post-hoc OOD score is algorithm-irrelevant and should not be constrainted within certain methods. I would keep my opinion that comparing with MaxLogit, Energy and MCM is not sufficient enough to demonstrate the usefulness (the most updated algorithm to be 2022)

As for the performance improvement, I do not think the author's explanation is persuasive and would keep my opinion.

Last but not least, I agree with the question about context score raised by 19PE and the counterintuition mitigates the persuasiveness of the paper.

We thank the reviewer for the comments and would like to answer the questions below.

1. “The cause of the phenomenon is not clearly explained.”

We agree with the reviewer that when the class name is present in a prompt, logit-based scores are generally higher for ID images than for OOD images, which is a way to reflect the model’s confidence that an image belongs to an ID label. Model confidence is the key to distinguishing between ID and OOD samples.

In this paper, we propose a new way to better reflect the model’s confidence, which is the gap between the logit-based score and the context score namely Marginal Logit Score (MLS). The reason that MLS is better than the original logit-based score is quite intuitive. Recall that the context score measures the distance between the representation of an image and the representation of the prompt without the CLASS label and it contributes to the logit-based score. We believe there are two main factors that lead to high logit-based scores: 1) the model being genuinely confident in its prediction for an ID image, and 2) the model overly paying attention to the context vectors for an OOD image which leads to high logit-based scores. Relying solely on logit-based scores cannot differentiate between these two scenarios and, therefore, fails to accurately reflect the model’s confidence. This limitation explains why the model struggles to distinguish between ID and OOD images when their scores overlap, as illustrated by the shaded region in Figure 3(a).

In this problematic overlapping region, the context score provides a valuable complement to the logit-based scores where the context score tends to be lower for ID images than for OOD images. This occurs because the model recognises ID images as belonging to ID labels, so the context score is relatively lower for ID images than for OOD images as the context score is computed using a prompt without the ID label. Thus, subtracting the context score from the logit-based scores yields a more distinctive scoring function, as illustrated in Figures 3(b) and 3(c), where the shaded overlapping region is notably reduced.

2. “The discussion is not thorough enough.”

The context score should be used alongside logit-based scores because, on its own, the context score is not an effective scoring function. We emphasise that the context score is meant to complement the logit-based scores, enhancing them rather than serving as a standalone measure. This can be observed by examining the range of x-values in Figure 3(a), which represents the context score. The ranges of x-values for ID and OOD images completely overlap, making it impossible to set a threshold that effectively separates ID and OOD images using only the context score. The synergy and improved distinguishability arise only when the context score is combined with logit-based scores.

As discussed before, distinguishing between ID and OOD images is about capturing model confidence accurately. Using the context score alone might not be a good choice. We further provide empirical results with two datasets comparing between using the context score alone and using MLS to illustrate our claim. It can be seen that using the context score alone results in a poor performance.

Table R1. Near OOD detection results (AUROC) comparing MLS and Context score with Caltech101 (16-shots). \begin{array}{cccc} & \text{MLS-M} & \text{MLS-E} & \text{Context} \\ \hline \text{CoOp} & 92.93 & 92.50 & 49.85 \\ \text{CoCoOp} & 91.63 & 91.06 & 44.46 \\ \text{IVLP} & 94.08 & 93.74 & 35.80 \\ \text{KgCoOp} & 90.11 & 88.86 & 38.66 \\ \text{ProGrad} & 90.24 & 88.95 & 42.94 \\ \text{MaPLe} & 95.00 & 94.76 & 36.55 \\ \text{PromptSRC} & 90.49 & 89.66 & 40.79 \\ \text{LoCoOp} & 89.43 & 88.03 & 42.02 \end{array}

Table R2. Near OOD detection results (AUROC) comparing MLS and Context score with OxfordPets (16-shots). \begin{array}{cccc} & \text{MLS-M} & \text{MLS-E} & \text{Context} \\ \hline \text{CoOp} & 86.80 & 86.66 & 44.12 \\ \text{CoCoOp} & 93.63 & 93.56 & 48.31 \\ \text{IVLP} & 93.12 & 92.99 & 46.12 \\ \text{KgCoOp} & 92.80 & 92.49 & 50.56 \\ \text{ProGrad} & 87.83 & 87.60 & 49.43 \\ \text{MaPLe} & 94.59 & 94.54 & 45.36 \\ \text{PromptSRC} & 94.35 & 94.22 & 45.71 \\ \text{LoCoOp} & 89.68 & 88.95 & 42.89 \end{array}

Thank you for the author's rebuttal. Sorry for the late response. While some concerns have been addressed, they do not fully convince me regarding comments #1 and #2. Therefore, I will maintain my original rating.

Regarding #1, the key should be the difference of logits between ID and OOD images rather than "logit-based scores are generally higher for ID images than for OOD images". The main challenge of OOD detection is to find the difference between ID and OOD samples. Thus, my main question is actually why ID and OOD samples have similar logit-based scores but different context scores?. However, I think that the authors do not address it well. For example, the authors claim "the model recognizes ID images as belonging to ID labels, so the context score is relatively lower for ID images than for OOD images, as the context score is computed using a prompt without the ID label." But why does the change of prompt lead to the change of score? Generally speaking, prompts adjust features through attention (which is why I mentioned ''the attention on prompts'' in the review). The change in features leads to a change in the score. The authors haven't clearly explained how the score changes after the prompt is altered, and why there is a difference between ID and OOD samples.

Regarding #2, the reason I believed that using only the context score was sufficient is due to the example in Figure 2; this example is clear. After reading the rebuttal, I still don't understand why the logit-based score must to be combined. Could you provide an example where using only the context score fails, and explain why, similar to Figure 2? In other words, I appreciate the additional experiments and Figure 3 provided by the authors, but why is it not good to use only the Context score? What are the limitation of the Context score?

Dear Authors,

Thank you for your response; I feel much clearer now.

After reading your response, my understanding is that the context score corresponds to domain-specific features, while the logit score corresponds to both class-specific and domain-specific features. The core function of subtracting the context score is to eliminate the influence of domain common knowledge when distinguishing between ID and near-OOD samples. Intuitively, this makes sense to me, but I believe the authors may need to restructure the entire paper around this point (e.g., more evidences to support it). Therefore, I keep my score now. I encourage the authors to restructure the paper with this focus and consider submitting it to the next conference.

We appreciate the reviewer’s feedback and would like to answer the questions below.

1. More evidence of the relationship between logit-based scores and context score

We acknowledge the reviewer’s concern and have added further evidence of the proportional relationship between logit-based scores (MaxLogit and Energy) and the context score, using all eight prompt-learning models trained with 16-shot settings across 13 datasets (see Appendix A.4 for the added images). This pattern is consistently observed across various models and datasets.

2. More near OOD datasets

We would like to kindly highlight that we have included additional experimental results using classic near-OOD datasets in Appendix A.3.7. These experiments follow the ImageNet Protocol (Palechor et al., 2023; Li et al., 2024), which closely resembles the ImageNet-10 vs. ImageNet-20 setup, featuring categories like dogs vs. wolves.

We appreciate your reviews for our work. We would like to clarify and respond each question below:

1. Impact of the number of shots

We believe this comment can be addressed in the appendix along with our original submission. As outlined in L452–453, we included Appendix A.3 to present the near-OOD detection performance across various few-shot settings. Tables 7–11 display the results for 16-, 8-, 4-, 2-, and 1-shot scenarios with MLS-M, reporting the mean and standard deviation over three random seeds, while Tables 14–18 correspond to the same few-shot scenarios with MLS-E. These results demonstrate that our method consistently improves near-OOD detection performance across different few-shot settings. This also highlights our method’s effectiveness in learning the margin scale, given that the optimal learning scale cannot be directly obtainable.

2. Far-OOD results

This comment can also be answered with our original submission. We provided an in-depth analysis of far-OOD detection performance in L491–512, with results detailed in Appendix A.3.5. For far-OOD detection, logit-based scores perform comparably to MCM, outperforming MCM in half of the evaluations while MCM performs better in the other half. For near-OOD detection, however, our method outperforms MCM in most cases. Consequently, in real-world scenarios where it is unclear whether OOD samples are from far or near OOD distributions, our method can be considered a more reliable method.

The reviewers addressed my concerns and I will stick to my original score.