Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

Q1: Overall, the technical contribution of this paper is relatively incremental, primarily focusing on how to weight the two loss components.

Thanks for the valuable comments! We would like to re-clarify the novelty and insights of our SCT as follows.

Conceptually, the motivation of SCT is to mitigate the problem of spurious OOD features in prompt-tuning based OOD detection methods. Generally, these methods rely on the ID-irrelevant local context extracted by VLMs as the surrogate OOD features to perform regularization, the quality of which is greatly affected by the foreground-background decomposition of VLMs. As shown in Figure 1/4 in the submission, although VLMs can mask out some ID-related regions, large portions of the extracted OOD features(shown as the colored patches of images) obviously belong to ID features.

Empirically, we find that the quality of extracted OOD features is significantly correlated with the uncertainty level of ID data. As illustrated in the left panel of Figure 2 in the submission, the extracted OOD features become more inaccurate as the uncertainty increases. In the right panel of Figure2, we train LoCoOp on multiple data groups with different uncertainty levels and the results demonstrate that the OOD detection performance of LoCoOp can be significantly impacted by the uncertainty level of ID data. Therefore, to mitigate the issue of unreliable OOD features, we propose SCT to calibrate the influence of OOD regularization from different ID samples based on their uncertainty level.

Q2: The effectiveness of the proposed method is quite limited. For example, as shown in Table 2, the improvement in averaged AUROC under the 16-shot scenario is minimal, only around 0.3%, and there is even a slight decrease in results on ID-like data.

Thanks for the comments! As shown in Table 1 in the submission, the space for improvement of AUROC of the prompt-tuning based method is close to saturation, both exceeding 90%. However, the space for improvement of FPR95 is still very large, so these two metrics should not be treated equally. The improvement of SCT on FPR95 (e.g. +5.95% for IDLike and +2.73% for LSN under 16-shot setting) is significant. Nevertheless, we will continue to promote the improvement of AUROC in future work!

Q3: There is a lack of comparison with existing state-of-the-art (SOTA) methods. For instance, the results reported in this paper are not as good as those of NegLabel[1] (AUROC 94.21 > 93.37), which is a zero-shot method that does not require training and training samples. The results for ID-like method reported in this paper are also lower; the official paper reports 94.36 AUROC under 4-shot, while this paper reports 92.14 AUROC under 16-shot.

Thanks for the comment! Zero-shot methods and prompt-tuning based methods are compatible with each other, further boosting the OOD detection performance. We conduct experiments on the compatibility of NegLabel with SCT in Table 5 in the attached PDF and the results show that SCT can be combined with advanced Neg-Label for better OOD detection.

Regarding the results for ID-like method, we strictly follow the official source code [1] and the hyperparameter settings of the official paper on a single A100 GPU. We will add this to our revised version.

Q4: More exploration is needed regarding the settings of function $\phi$ and $\psi$ in Equation 4.

Thanks for the suggestions! We conduct experiments on more instantiations of modulation functions in the following table. The results demonstrate that All the instantiations show significant improvement over LoCoOp, which verifies the effectiveness of the learning framework of SCT.

Q5: The statement in lines 158-159 is somewhat unclear. Should it be that inaccurate OOD features hinder the effective learning of better OOD detection?

Thanks for the comments! What we mean by the original statement is that we aim to design a mechanism to mitigate the issue of unreliable extracted OOD features. We will make this statement more clear in our revision.

Reference: [1] Bai, Han, et al. "ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection." CVPR, 2024

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

W1: Inaccurate motivation verifications

Thanks for the constructive comments! Although true-class probability (TCP) and accuracy are not two identical things, classification accuracy could indicate TCP to a certain degree under the large-scale ImageNet-1k dataset. Since ImageNet-1k has a large number of classes and CLIP models are prompt-tuned with one-hot labels, the prediction probability of CLIP can be overconfident, which means that CLIP models normally output high maximum softmax probability $max_y p(y|x)$. Therefore, when models correctly classify samples, TCP are likely to be very high. Nevertheless, we would acknowledge it has some non-accurate parts.

To make our motivation verification clearer and more accurate, we conduct a new experiment on the correlation between data uncertainty and OOD detection performance. Specifically, we calculate the TCP of all the training samples in a 64-shot set using a CLIP model prompt-tuned with a 4-shot training set which contains no overlapping samples with the 64-shot set. We choose the data with the lowest and highest TCP for every ID class to generate two data groups with different uncertainty levels respectively, and train LoCoOp on these two data groups. As shown in Table 1 in the attached PDF, the OOD detection performance of LoCoOp is significantly impacted by the data uncertainty level, which is consistent with Figure 2 in the submission.

W2: No calibration verifications

Thanks for the valuable suggestions! We add the figures of calibration comparison as Figure 1 in the attached PDF. The illustrations show that our SCT can significantly help with CLIP calibration for more accurate OOD feature extraction. The extracted OOD features (shown as grey patches of images) of SCT models are significantly more accurate and contain less ID-relevant regions than LoCoOp models.

W3,W4

We leave the answers in General Response.

W5: No much AUROC improvement

Thanks for the comments! As shown in Table 1 in the submission, the space for improvement of AUROC of the prompt-tuning based method is close to saturation, both exceeding 90. However, the space for improvement of FPR95 is still very large, so these two metrics should not be treated equally. We will continue to promote the improvement of AUROC in future work!

W6: (Lack of CIFAR results)

Thanks for the valuable suggestions! We conduct experiments on the CIFAR benchmark in Table 4 of the attached PDF and the results show that our SCT can still outperform the baselines under CIFAR benchmark.

W7: Discussions with pre-trained features + post-hoc

Thanks for the comment! First, prompt-tuning based method can leverage the generalization ability of VLMs to better fit the domains of the downstream tasks with relatively low computational cost. Secondly, post-hoc methods need to be built on a well-trained model, the capacity of which greatly affects the OOD detection performance. Thirdly, post-hoc methods and prompt-tuning based methods are compatible with each other, further boosting the OOD detection performance. We conduct experiments on the compatibility of An advanced post-doc method, NegLabel [1], with SCT in Table 5 in the attached PDF. The results show that SCT can be combined with post-doc methods for better OOD detection.

W8: Validation set

Thanks for the question! We tune the hyperparameter $\lambda$ on the dedicated OOD validation set in the OpenOOD v1.5 benchmark[2]. The other hyperparameters are chosen following previous works.

W9: About ablations.

Thanks for the constructive question! In the ablation study of Figure 3(b), we follow OE [3] and Energy-OE [4] to implement prompt-tuning based OOD detection with MSP and Energy regularization functions, respectively. The motivation of conducting this experiment is to verify that SCT can outperform LoCoOp in different regularization functions. Specifically, the energy regularization needs tuning the two energy threshold hyperparameters $m_{in}$ and $m_{out}$, limiting its advantages over other regularization. As shown in Fig. 3(b), we conjecture that directly forcing the probability distribution of OOD features to the uniform distribution(MSP) performs worse than entropy maximization under the setting of prompt tuning.

Regarding OOD feature extraction methods, the probability-based and entropy-based methods both have a threshold hyperparameter to discriminate between ID and OOD features. The sensitivity to hyperparameters of different methods may be the reason behind the different OOD detection performance. For the entropy-based method, the performance is poor since it is challenging to determine the appropriate threshold. [5]

W10: About Table 1

Thanks for the suggestion! We will conduct new experiments of newer post-hoc and fine-tuned methods on CLIP models in our revision. Since ASH and Scale can't apply to CLIP as analyzed in W3, we will include the results of them on conventional CNN models in the appendix for fair comparison.

W11, W12, W13: About presentation and limitations.

Thanks for the suggestion! The original images arrangement is designed to compare the neighboring images with different uncertainty levels. we will make the modifications in our manuscript.

Reference: [1] Jiang, Liu, et al. "Negative Label Guided OOD Detection with Pretrained Vision-Language Models." ICLR, 2024. [2] Jingyang Zhang,Jingkang Yang, et al "Openood v1.5: Enhanced benchmark for out-of-distribution detection." [3] Dan Hendrycks, et al, "Deep Anomaly Detection with Outlier Exposure", ICLR 2019 [4] Weitang Liu, et al, "Energy-based Out-of-distribution Detection", NeurIPS 2020 [5] Miyai1, Yu, et al. "LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning." NeurIPS, 2023

I have carefully read the reviews of the reviewers and the author's detailed rebuttal, and I am sincerely grateful. In particular, the author corrected the experimental verification of TCP motivation and provided the benchmark result of CIFAR. However, in general, I am still worried about:

• If we refer to related methods such as feature representation-based ones, 90 AUROC is not saturated, and their FPR is also relatively low ~20, so the AUROC performance should be improved further.

• The method is a bit incremental and empirical, lacking theory as the author also acknowledges. If I understand correctly, the author wants to achieve the balance and relative weighting of OOD and ID items by modulating the function. Even so, the ablation OOD/ID ratios in the last three rows of Tab. 4 all increase monotonically with p, so either the hyperparameters may not be tuned well or there would be a deeper understanding of the exact reasons that matter.

• For calibration verification, in addition to the recommended qualitative OOD region visualization comparison, you can also consider adding quantitative metrics such as ECE [4].

(Minor)

• Post-hoc studies can be more comprehensive in research, and choosing different K is heuristic and not a very good calibration strategy. The authors could consider temporal scaling [4], etc.

Based on the above reasons and careful consideration, I tend to keep the original score.

[4] A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

W1: A primary concern of the reviewer is the lack of evaluations against the more traditional CIFAR set of benchmarks for OOD detection.

Thanks for the valuable suggestions! We conduct experiments on the CIFAR benchmark in Table 4 in the attached PDF and the results show that our SCT can still outperform the baselines under the CIFAR benchmark.

W2: Additionally, the empirical performance gain of SCT (table 2) in combination with other prompt-tuning-based methods, seems minimal.

Thanks for the comments! As shown in Table 1 in the submission, the space for improvement of AUROC of the prompt-tuning based method is close to saturation, both exceeding 90%. However, the space for improvement of FPR95 is still very large, so these two metrics should not be treated equally. The improvement of SCT on FPR95 (e.g. +5.95% for IDLike and +2.73% for LSN under 16-shot setting) is significant. Nevertheless, we will continue to promote the improvement of AUROC in future work!

Q1: The reviewer would also like to point out some small inconsistencies in the bolding for Table 2 (IDLike+SCT).

Thank you for pointing out this issue! We are sorry for the carelessness and the original bolded data should be 92.44 instead of 91.44. We will make the modification in the revision.

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

Q1: My concern is mainly about the computational cost and training cost of SCT, since it involves operations on dense/local features.

Thanks for your valuable question!

First, SCT doesn't incur any extra computational cost to the LoCoOp due to its simple design. Technically, SCT introduces modulating factors respectively on the two components of the original learning objective. The modulating factors (instantiated as $1-p(y|x)$ for $\phi$ and $p(y|x)$ for $\psi$ in Equation (4) in the submission) only involves the computation of the prediction probability of ground truth classes $p(y|x)$, which can be repeatedly used after the original forward pass of CLIP models.

Secondly, the operation of local features involved in LoCoOp and SCT are also relatively low-cost in terms of computation. The local are generated from the forward pass of the vision encoders of CLIP, which doesn't bring additional computational cost compared to regular training. Regarding the extraction of OOD features, we compute the similarity between local features and text features of all the ID classes and we identify regions that do not include their ground truth class in the top-K predicted classes as ID-irrelevant regions. Empirically, we evaluate the time and memory consumption of SCT compared with other baselines in Table 1 and the results show that SCT is relatively compute-efficient. The evaluation is conducted on a single A100 GPU with the batch size of 32.

Table 1. Evaluation of computational cost of SCT and baselines.

Q2: My second concern is about the rationality of using pre-trained models (CLIPs .etc) to perform OOD detection tasks, because the concepts in both ID and OOD datasets are probably seen during pre-training stage. I want to know the authors' opinions towards the benchmarking and research paradigm.

A2:

Thanks for your valuable comments! The definition of VLM-based OOD detection differs significantly from that of conventional OOD detection. VLM-based OOD detection aims to detect samples that do not belong to any ID class text designated by the downstream task [1]. Therefore, the current benchmarks, such as the large-scale ImageNet-1k benchmark, still apply to VLM-based OOD detection as long as they satisfy the rule that ID and OOD concepts don't overlap. For the better development of this field, future works should be focused on building benchmarks based on realistic datasets and scenarios.

References: [1] Miyai, Yang, et al. "Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey." 2024.

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

W1,Q1: The contribution over LoCoOp is incremental. The only difference is an extra reweighting term based on the current prediction score. And the reweighting mechanism is purely based on heuristics - for example, $1-p(y|x)$ for $L_{ce}$ implicitly enforce hard sample mining. Please clarify the novelty and new insights.

Thanks for the valuable comments! We would like to re-clarify the novelty and insights of our SCT as follows.

Conceptually, the motivation of SCT is to mitigate the problem of unreliable OOD features in prompt-tuning based OOD detection methods. Generally, these methods rely on the ID-irrelevant local context extracted by VLMs as the surrogate OOD features to perform regularization, the quality of which is greatly affected by the inaccurate foreground-background decomposition of VLMs. As shown in Figure 1/4 in the submission, although VLMs can mask out some ID-related regions (shown as the grey patches of images), large portions of the extracted OOD features (shown as the colored patches of images) obviously belong to ID features.

Empirically, we find that the quality of extracted OOD features is significantly correlated with the uncertainty level of ID data. As illustrated in the left panel of Figure 2 in the submission, the extracted OOD features become more inaccurate as the uncertainty increases. In the right panel of Figure 2, we train LoCoOp on multiple data groups with different uncertainty levels, and the results demonstrate that the OOD detection performance of LoCoOp can be significantly impacted by the uncertainty level of ID data. Therefore, to mitigate the issue of unreliable OOD features, we propose SCT to calibrate the influence of OOD regularization from different ID samples based on their uncertainty level.

Technically, despite the simple design, SCT is significantly different from hard sample mining. The latter conducts reweighting directly on the samples based on the classification difficulty during training. The former adaptively adjusts the importance between the two components of the original learning objectives for every single sample. Data with high uncertainty are directly down-weighted in hard sample mining while they are utilized more for OOD regularization in SCT. As shown in Table 4 in the submission, under 16-shot ID data, the OOD detection performance of simply assigning $1-p(y|x)$ to $L_{ce}$ (denoted as $\phi$ $\checkmark$ and $\psi$ $\times$) are significantly inferior to SCT (denoted as $\phi$ $\checkmark$ and $\psi$ $\checkmark$), demonstrating the difference of SCT and hard sample mining.

W2: The intuition in Figure 1/4 is not clear to me. The shown examples validate that LoCoOp can detect and mask-out the inlier-related regions well. Also, the GT label should be annotated.

Thanks for the suggestions! As shown in Figure 1/4, although VLMs can mask out some ID-related regions (shown as the gray patches of images), large portions of the extracted OOD features (shown as the colored patches of images) obviously belong to ID features. We will make the captions and figures clearer as suggested in our revised version.