An Information-theoretical Framework for Understanding Out-of-distribution Detection with Pretrained Vision-Language Models

We thank Reviewer jjLB for valuable comments. As to the questions and suggestions you raise, we took them seriously. Our response is as follows.

1. What is the underlying intuition of the proposed method and why and how the proposed method enhances OOD detection

The core intuition of our method starts from the thorectical analysis that representative post-hoc CLIP-based OOD detectors (MCM and NegLabel) can be understand as stochastic estimators of Pointwise Mutual Information (PMI) estimator. We, motivated by thedivide-and-conquer philosophy, propose to decompose PMI into two subterms, i.e., $\mathcal{W}(\mathbf{x},y)=\mathcal{W}(\tilde{\mathbf{x}},y)+\mathcal{W}(\mathbf{x},y|\tilde{\mathbf{x}})$ so as to transfer the estimation of $\mathcal{W}(\mathbf{x},y)$ into separately estimating $\mathcal{W}(\tilde{\mathbf{x}},y)$ and $\mathcal{W}(\mathbf{x},y|\tilde{\mathbf{x}})$.

The theoretical benefits of our PMI decomposition to OOD detection can be summarized in two aspects:

• Our PMI decompositon reduces the complexity of estimating $\mathcal{W}(\mathbf{x},y)$. This is becuase

a) estimating $\mathcal{W}(\tilde{\mathbf{x}},y)$ involves estimating $p(\tilde{\mathbf{x}},y)$, which is simpler than estimating $p({\mathbf{x}},y)$ as $\tilde{\mathbf{x}}$ is a subview of $\mathbf{x}$ to make the space of possible $\tilde{\mathbf{x}}$ and $y$ lower-dimensional.

b) estimating $\mathcal{W}(\mathbf{x},y|\tilde{\mathbf{x}})$ involves estimating $p(\mathbf{x},y|\tilde{\mathbf{x}})$, which is simpler than estimating $p({\mathbf{x}},y)$ as $\tilde{\mathbf{x}}$ is a subview of $\mathbf{x}$ to make the conditioned space of possible $\mathbf{x}$ and $y$ lower-dimensional.

• We show in Theorem 4 that our PMI decompositon provably increases the upper bound of the the estimated PMI to $\log(K+L)$ from NegLabel’s $\log(K + L/T)$ or MCM's $\log K$, therefore reducing the underestimation bias.

2. Are there any illustrations or pseudocode to help convey the core idea?

The pseudo algorithm of our method is given as follows:

Input: input image $\mathbf{x}$, ID labels $\mathcal{Y}\_{I}$, Negative labels $\mathcal{Y}\_{N}$, pre-trained CLIP model parameters $\theta$, sub-view generator $\mathcal{T}$, pre-defined critics $h\_{\boldsymbol{\theta}}(\mathbf{x},\hat{y})$ and $h\_{\boldsymbol{\theta}}(\mathbf{x},\tilde{\mathbf{x}},\hat{y})$

Output: Our proposed OOD scoring function $S\_{ours}(\mathbf{x})$

Generate a subview via $\tilde{\mathbf{x}}=\mathcal{T}(\mathbf{x})$

Randomly divide $\mathcal{Y}\_{N}$ into $T$ non-overlapping groups, i.e., $\mathcal{Y}\_{N}=\cup_{i=1}^L \mathcal{G}_i$

For each $y$ in $\mathcal{Y}\_{I}$ do

$\hat{\mathcal{W}}(\tilde{\mathbf{x}},y;\boldsymbol{\theta})\leftarrow\log \frac{1}{T}\sum\_{t=1}^T\frac{ \exp h_{\boldsymbol{\theta}}(\mathbf{x},y)}{\underset{{y\_j\in\mathcal{G}\_t\cup\mathcal{Y}\_I}}{\sum}\exp h_{\boldsymbol{\theta}}(\mathbf{x},y_j)}+\log(K+L/T)$

end For

$S\_{ours}(\mathbf{x})\leftarrow\sum\_{y\in\mathcal{Y}\_{I}}\frac{\exp \big [\hat{\mathcal{W}}(\tilde{\mathbf{x}},y;\boldsymbol{\theta})+h\_{\boldsymbol{\theta}}(\mathbf{x},\tilde{\mathbf{x}},y)\big ]}{\underset{{y\_j\in\mathcal{G}\_t\cup\mathcal{Y}\_I}}{\sum} \exp \big [\hat{\mathcal{W}}(\tilde{\mathbf{x}},\hat{y};\boldsymbol{\theta})+h\_{\boldsymbol{\theta}}(\mathbf{x},\tilde{\mathbf{x}},\hat{y})\big ]}$

3. Including real-world benchmarks would further strengthen the results.

Thanks for your constructive. As per your advice, we conduct experiments on the BIMCV-COVID19+ dataset [a], following the same setup as AdaNeg [b]. Specifically, we select BIMCV as the ID dataset, which includes chest X-ray images CXR (CR, DX) of COVID-19 patients and healthy individuals. For the OOD datasets, we, following AdaNeg [b], use CT-SCAN and X-Ray-Bone datasets. The CT-SCAN dataset includes computed tomography (CT) images of COVID-19 patients and healthy individuals, while the X-Ray-Bone dataset contains X-ray images of hands. We report AUROC as follows:

[a] Bimcv covid-19+: a large annotated dataset of rx and ct images from covid-19 patients

[b] AdaNeg: Adaptive Negative Proxy Guided OODDetection with Vision-Language Models

Thank you so much for your recognition of our work and your constructive comment, which is very helpful in improving the quality of our work. We will incorportate these insights and the outcomes of the new experiments into the revised version of the paper.

We appreciate the insightful comments provided by Reviewer X4sW. Please see our responses to your questions below.

1. It would be helpful to provide clarifications on the motivation behind focusing specifically on CLIP-based post-hoc OOD detection

Indeed, we argue that our thorectical framework is CLIP-specific. In the following, we will show this point by proving that a pretrained CLIP is naturally a PMI estimator.

We start from the following following inequality [a]

$F=\mathbb{E}\_{(\mathbf{x},y)\sim\mathbb{P}\_{XY}}[\mathcal{W}(\mathbf{x},y)]=\mathbb{E}\_{(\mathbf{x},y)\sim\mathbb{P}\_{XY}}[\log\frac{p_{Y|X}(y|\mathbf{x})}{p_{Y}(y)}]\geq\mathbb{E}\_{(\mathbf{x},y)\sim\mathbb{P}\_{XY}}[\log\frac{\hat{p}\_{Y|X}(y|\mathbf{x};\boldsymbol{\theta}')}{p\_{Y}(y)}]=F_{\boldsymbol{\theta}'}$

Clearly, the inequality holds with inequality if and only if $\hat{p}\_{Y|X}(y|\mathbf{x};\boldsymbol{\theta}')=p_{Y|X}(y|\mathbf{x})$.

Recalling a CLIP training dataset $\mathcal{D}=${$(\mathbf{x}\_1,y\_1),\ldots,(\mathbf{x}\_N,y\_N)$ } $\sim\mathbb{P}^N\_{XY}$ and CLIP learning objective $L^{CLIP}\_{\boldsymbol{\theta}'}=\frac{1}{N}\sum\_{i=1}^N\log\frac{\exp h\_{\boldsymbol{\theta}}(\mathbf{x}_i,y_i)}{\sum\_{j=1}^N \exp h\_{\boldsymbol{\theta}}(\mathbf{x}\_j,y\_j)}$

For the MCM case, by parameterizing $\hat{p}\_{Y|X}(y|\mathbf{x};\boldsymbol{\theta}')$ via Eq. (7), we have $F_{\boldsymbol{\theta}'}^{MCM}=\mathbb{E}\_{(\mathbf{x},y)\sim\mathbb{P}\_{XY}}\left [\log\frac{\exp h\_{\boldsymbol{\theta}}(\mathbf{x},y)}{\mathbb{E}\_{\hat{y} \sim \mathbb{P}\_{Y}}\left [\exp h\_{\boldsymbol{\theta}}(\mathbf{x},\hat{y})\right ]} \right]$ One can easily check that $L^{CLIP}\_{\boldsymbol{\theta}'}$ is an empricial version of $F_{\boldsymbol{\theta}'}^{MCM}$ (by the law of large number)

For the NegLabel case, by parameterizing $\hat{p}\_{Y|X}(y|\mathbf{x};\boldsymbol{\theta}')$ via Eq. (12) and Eq. (13), we have $F_{\boldsymbol{\theta}'}^{NegLabel }=\mathbb{E}\_{(\mathbf{x},y)\sim\mathbb{P}\_{XY}}\bigg[\mathbb{E}\_{\hat{\mathbf{y}}\_{N-1}\sim\mathbb{P}\_Y^{N-1}}\bigg[\log\frac{\exp{h_{\boldsymbol{\theta}}(\mathbf{x},y)}}{\exp{h_{\boldsymbol{\theta}}(\mathbf{x},y)}+\sum\_{j=1}^{N-1}\exp{ h\_{\boldsymbol{\theta}}(\mathbf{x},\hat{y}\_j)}} \bigg] \bigg]+\log N$ $F\_{\boldsymbol{\theta}'}^{NegLabel }$ is known as InfoNCE [b]. It has been widely acknowledge in the literature that contrastive learning optimizes InfoNCE.

Based on the discussion, one may conclude that maximizing $L^{CLIP}\_{\boldsymbol{\theta}'}$ is equivalent to maximizing $F_{\boldsymbol{\theta}'}$ no matter whether $\hat{p}\_{Y|X}(y|\mathbf{x};\boldsymbol{\theta}')$ is parameterized via Eq.(7) or Eq. (12).

Thanks to the large-scale nature of the CLIP training dataset and the CLIP model itself, it is reasonable to approximate $F$ with $F_{\boldsymbol{\theta}}$ where $\boldsymbol{\theta}=\arg\max_{\boldsymbol{\theta}'}L^{CLIP}\_{\boldsymbol{\theta}'}$.

This implies that we can approximate the PMI $\mathcal{W}(\mathbf{x},y)=\log\frac{{p}\_{Y|X}(y|\mathbf{x})}{p\_{Y}(y)}$ with $\log\frac{\hat{p}\_{Y|X}(y|\mathbf{x};\boldsymbol{\theta})}{p\_{Y}(y)}$

2. It seems that incorporating any kind of non-ID prompt would help over MCM.

Yes. Empirical evidence confirms that any non-ID prompts (including randomly selected negative labels) significantly enhance OOD detection performance compared to MCM [c]. As demonstrated in Table 7 (Appendix A.4) of NegLabel [d], using 10,000 randomly chosen labels from WordNet yields superior results:

Even though, we never deny that the quality of negative labels has a significant impact on performance, which can be attributed to semantic consistency in representation space, i.e., inputs with similar semantics cluster closer in the embedding space. In other words, Introducing lower-discriminability negative labels (i.e., those semantically similar to ID classes) focuses the sampling distribution on a narrower region, therefore resulting in a more biased Monte-Carlo estimation.

3. the theoretical derivation are built upon existing works cited in the paper

While we agree that our theoretical analysis is partially inspired by foundational principles and estimation techniques from prior work, we respectfully argue that our contribution lies in how we extend these ideas to provide novel insights tailored to OOD detection with vision-language models (VLMs), which is not incremental as it solves the lack of theoretical grounding in CLIP-based OOD detection and directly enables our state-of-the-art method.

• To our best knowledge, we are the first to propose a PMI decomposition via sub-views (Theorem 3) for OOD detection. This decomposed formulation is not only theoretically elegant but also leads to provable benefits in bias reduction (Theorem 4).

• We provide the first formal information-theoretic explanation that unifies MCM and NegLabel as stochastic estimators of PMI (Theorems 1 and 2), bridging previously disconnected empirical techniques under a energy-based view (Section 3.1 & 3.2).

• We argue that it is non-trivial to extend energy-based scoring function from single-modal to multi-modal settings. Clearly, a natural approch to define the energy based on the CLIP-based zero-shot classifier logits, i.e., cosine similarity. However, as thorectically justified in [e], it is PMI not cosine similarity that correspondes to the optimal similarity measure in the CLIP's uni-modal embedding space. This is also empirically supported by empirical observations in [d], i.e., Energy (zero-shot) achieves significantly worse performance (79.57 AUROC and 82.21 FPR95 in average) than MCM and NegLabel, both of which, as proved in the manuscript, can be understood as different PMI estimators.

[a] The im algorithm: A variational approach to information maximization

[b] Representation learning with contrastive predictivecoding

[c] Delving into out-of-distribution detection with vision-language representations

[d] Negative label guided ood detection with pretrained vision-language models

[e] Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric, ICLR25

We appreciate the insightful comments provided by Reviewer oce3. Please see our responses to your concerns below.

1. The authors have not clearly distinguished their approach from [a]

We respectfully point out key distinctions that demonstrate the novelty of our work:

• Multi-Modal Energy as PMI Estimation

[a] defines energy over unimodal pre-trained classifier logits. In stark contrast, we derive an information-theoretic energy function directly from the Pointwise Mutual Information (PMI) between images and text embeddings in pre-trained VLMs (e.g., CLIP). This provides the first unified framework explaining established CLIP-based scores (MCM, NegLabel) as PMI estimators, establishing a novel theoretical grounding for post-hoc VLM-based OOD detection and directly motivates our method.

It is non-trivial to extend energy-based scoring function from single-modal to multi-modal settings. Clearly, a natural approch to define the energy based on the CLIP-based zero-shot classifier logits, i.e., cosine similarity. However, as thorectically justified in [e], it is PMI not cosine similarity that correspondes to the optimal similarity measure in the CLIP's uni-modal embedding space. This is also empirically supported by empirical observations in [b], i.e., Energy (zero-shot) achieves significantly worse performance (79.57 AUROC and 82.21 FPR95 in average) than MCM and NegLabel, both of which, as proved in the manuscript, can be understood as different PMI estimators.

• Divide-and-Conquer Chain Rule Decomposition

Rather than estimating PMI in one shot, we decompose it via the chain rule into two simpler terms，this provably raises the estimation upper bound and therefore reduces underestimation bias while keeping the number of selected negative labels unchanged. We note that This decomposition is unique to our framework and not possible in prior energy methods.

Our contributions are:

(i) A new PMI-theoretic foundation for multi-modal energy

(ii) A novel chain-rule decomposition mitigating PMI estimation bias

This represents a clear conceptual and practical advancement beyond [a] and related works. We thank the reviewer for the opportunity to clarify our novel contributions.

2. Lacking an ablation study about the computation cost.

Thanks for your constructive comments. Since, same as NegLabel [b], this paper focuses on post-hoc OOD detection, our method have no computational cost for training. Regarding the computational cost for inference, the table below reports the inference speed, where the inference speed is measure by Frame Per Second (FPS) with a NVIDIA V100 GPU. It can be found that our method achieves the better trade-off between efficiency and effectiveness.

3. Including comparisons with other VLMs would strengthen the validity of the claims.

Thanks your insightful comments. In Appendix H.2, following NegLabel [b], we evalute our method with GroupViT [c]. For your convenvience, the OOD detection results are reported as follows, where our method consistently outperforms NegLabel beyond CLIP-based models .

Besides, we conduct extra experiments using ALIGN [f], following NegLabel [b]. We provide the results in the below, where our method keeps outperforming NegLabel.

[a] Energy-based out-of-distribution detection

[b] Negative label guided ood detection with pretrained vision-language models

[c] Groupvit: Semantic segmentation emerges from text supervision

[d] AdaNeg: Adaptive Negative Proxy Guided OODDetection with Vision-Language Models

[e] Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric, ICLR25

[f] Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisio

We're grateful for your feedback during this busy period. As requested, we have conducted additional experiments to evaluate the Ours+AdaNeg setting. The results below demonstrate that our method combined with AdaNeg achieves a better trade-off between speed and performance. While we admit that our method makes FPS slightly decrease, the significant performance improvement justifies this marginal cost. Please be free to let us know if you have any questions. We are happy to address.

We appreciate the insightful comments provided by Reviewer p925. Please see our responses to your concerns below.

1. Evaluate the effect of different sub-view generation methods

Indeed, we have conducted studies in Appendix H.3 of the manuscript with regards to different sub-view generation methods including cutout and cropping (the our best knowledge, they are the only two options for image data). For your convenvience, we report the detailed results in the below, where it can be found that cutout and cropping achieve comparable OOD detection perfomance while significantly outperforming NegLabel. This implies the flexibility of our method.

2. Test the proposed method on OOD benchmarks from domains other than natural images

In fact, we have conducted experiments on the ImageNet-sketch dataset in Table 2 where ImageNet-sketch is abbreviated as ImageNet-S. For your convenvience, we provide the results in the below.

As per your advice, we additionally conduct experiments on the BIMCV-COVID19+ dataset [a], following the same setup as AdaNeg [b]. Specifically, we select BIMCV as the ID dataset, which includes chest X-ray images CXR (CR, DX) of COVID-19 patients and healthy individuals. For the OOD datasets, we, following AdaNeg [b], use CT-SCAN and X-Ray-Bone datasets. The CT-SCAN dataset includes computed tomography (CT) images of COVID-19 patients and healthy individuals, while the X-Ray-Bone dataset contains X-ray images of hands. We report AUROC as follows:

3. Provide quantitative analysis on inference time

Regarding the computational cost for inference, the table below reports the inference speed, where, same as AdaNeg [b] the inference speed is measure by Frame Per Second (FPS) with a NVIDIA V100 GPU. It can be found that our method achieves the better trade-off between efficiency and effectiveness.

4. Include comparisons with recent OOD detectors

We thank the reviewer for bringing HFTT, NPOS and ZOC into our eyes. Finding that these works are interesting and make a solid contribution to OOD detection, we are greatly glad to compare them.

In particular, while both HFTT, NPOS and ZOC offer creative directions for OOD detection using vision-language models, our work provides a fundamentally deeper contribution by establishing a unified information-theoretical framework that explains and generalizes existing CLIP-based post-hoc methods. Unlike HFTT, which relies on a specific training scheme and synthetic textual data, or ZOC/NPOS, which introduces an additional generation module requiring supervised training, our method is training-free, fully post-hoc, and compatible with any pretrained CLIP-like model. More importantly, we offer provable theoretical insights into why and how exsiting representative CLIP-based OOD scoring function scores can be interpreted as estimators of point-wise mutual information (PMI), and we propose a novel PMI decomposition strategy that improves estimation fidelity without introducing additional negative labels. This level of theoretical rigor and generality not only bridges gaps in the current literature.

The empirical comparsion below shows that our method achives significantly better performance than others, which empirically highlights the superority of our method.

[a] Bimcv covid-19+: a large annotated dataset of rx and ct images from covid-19 patients

[b] AdaNeg: Adaptive Negative Proxy Guided OODDetection with Vision-Language Models

Dear Reviewer p925:

We're grateful for your mandatory acknowledgement during this busy period. As the rebuttal period is ending soon, please be free to let us know whether your concerns have been addressed or not, and if there are any further questions.

Thanks,

Authors.