Text-Image Dual Consistency-Guided OOD Detection with Pretrained Vision-Language Models

Response to Reviewer aTAh
We thank the reviewer aTAh for the valuable feedback. We addressed all the comments. Please find the point-to-point responses below. Any further comments and discussions are welcomed!
W1: The paper assumes that Stable Diffusion can reliably supplement visual information for ID/OOD detection. However, this assumption may not always hold: (1)Generation Quality(2)Distribution Alignment(3) Representation Fidelity
Reply:
R1: We appreciate the suggestion. While synthetic images may contain noise, style variations, or artifacts that could potentially mislead OOD detection models, our score function incorporates multi-level features from pixel to semantic levels. Thus, even when pixel-level features are affected by such deviations, language-level features remain effective. To validate this, we generated oil painting-style images to simulate style shifts. As shown in Table A, our method maintains robust performance despite this variation. Due to space constraints, the complete table is available at:
https://anonymous.4open.science/r/fjutlfy-31D8 Table K.

Table A: Experimental comparison under generated image style shifts. ID dataset: ImageNet-1k.

R2 and R3: We confirm that the synthetic ID/OOD data maintain consistent language-level features with real data, ensuring the robustness of our method (as validated above) . Notably, as demonstrated in Experimrnts (Table A) in R1, we deliberately generated oil-style images that deviate from the ID distribution, yet they still contribute to improved detection performance. This further demonstrates the robustness of language-level features.

W2: The proposed DualCnst score function is a simple weighted combination of textual and visual similarities, but there is no theoretical analysis to justify the chosen weighting strategy.
Reply:
We gratefully acknowledge the reviewer's suggestion. Our supplemented theoretical analysis demonstrates that under a weighted combination strategy of textual and visual similarities, the false positive rate ($\\text{FPR}\_\\lambda$) exhibits a monotonic decrease with increasing multimodal (visual) labels under certain conditions. This result not only validates that incorporating auxiliary visual features enhances OOD detection performance, but also confirms the appropriateness of the weighted combination strategy. Due to space limitations, the detailed theoretical analysis can be found in the response to Reviewer YYQ7-W1.
W3: The paper does not provide a mathematical proof of convergence or generalization for the score function, relying only on empirical validation. A more rigorous theoretical foundation would strengthen the contribution.
Reply:
As noted in our W2 reply, we have discussed the proposed score function's ability to improve the separability between ID and OOD samples in terms of the false positive rate ($\\text{FPR}\_\\lambda$). Specifically, the proposed score function integrates multi-level synthetic image features into the existing text-based labels. Theoretically, we consider a more general scenario—how expanding multimodal labels improves OOD detection. We prove that, under certain conditions, $\\text{FPR}\_\\lambda$ decreases as the number of multimodal labels increases, demonstrating that incorporating additional auxiliary modalities into labels improves OOD detection performance.
W4: DualCnst's synthetic image generation incurs significantly higher costs than baselines, potentially limiting real-world applicability. Urge optimization of computational overhead.
Reply:
Thank you for your comment. Regarding the efficiency of Stable Diffusion generation, we address the issue through two key optimizations.Due to space limitations, Reviewer ViPC raised the same question. Please refer to our detailed response under W5 in the Reviewer ViPC section.
Q: Stable Diffusion performs well on natural image datasets, but is it still effective for OOD detection in domains such as medical imaging or remote sensing? Have the authors considered evaluating its generalization across different types of datasets?
Reply:
We have supplemented experiments on remote sensing data using UCM $1$ as the ID dataset and AID $2$ as the OOD dataset. As shown in Table D, our method achieves superior performance compared to the baseline NegLabel.

$1$ Y. Yang and S. Newsam, "Bag-of-visual-words and spatial extensions for land-use classification "

$2$ Zhong et al. "AID: A benchmark dataset for performance evaluation of aerial scene classification "
Table D: OOD Detection Experiments on Remote Sensing Data.

Response to Reviewer Ci5u
We thank the reviewer Ci5u for the valuable feedback. We addressed all the comments. Please find the point-to-point responses below. Any further comments and discussions are welcomed!
W1: Why ID images are not accessible (fig. 1) ? Please justify. I think most testing benchmarks provide training samples which can be used as "actual ID Image". This scenario should be consider as a baseline.
Reply:
Thank you for your question. However, in the zero-shot OOD detection setting considered in this paper, training samples are not ID images. Specifically, the definitions of ID and OOD labels are as follows. As stated in the Introduction section of ZOC. W2: If negative labels are essential, the method’s novelty over NegLabel is unclear. An ablation study (removing negatives) is needed to isolate the impact of synthetic image features.
Reply:
Our method is not limited to applications with NegLabel. As shown in Table A, the results demonstrate that incorporating synthetic image labels can effectively enhance OOD detection performance even without negative labels/images. Due to space constraints, the complete table is available at:
https://anonymous.4open.science/r/fjutlfy-31D8 Table H.
Table A: Performance comparison of integrating DualCnst with MCM for OOD detection on ImageNet-1k (ID dataset) .

W3: Considering negative images, is the upper bound given by using real OOD samples? The authors are encouraged to include this as one of the ablation studies.
Reply:
Yes, we have conducted ablation studies using real OOD samples (Table B) . The results demonstrate that while the proposed method achieves better performance with real samples compared to synthetic ones, the improvement margin is not substantial, demonstrating synthetic samples' effectiveness. Due to space constraints, the complete table is available at:
https://anonymous.4open.science/r/fjutlfy-31D8 Table I.
Table B: Experimental comparison between real OOD samples and synthetic OOD samples, with ImageNet-1k as the ID dataset.

W4: How to ensure evaluation fairness for models (e. g. , CLIP/Stable Diffusion) trained on large datasets that may include OOD data, particularly when comparing methods with/without auxiliary datasets?
Reply:
Thanks for your question. We would like to clarify our definition of ID classes. Following zero-shot OOD detection $1, 2, 3$ , in our setting, the ID classes are defined based on the classification task of interest rather than the classes used in pre-training. Additionally, we adopt widely recognized zero-shot OOD detection benchmarks, where the label spaces of ID and OOD datasets do not overlap.

$1$ Ming et al. Delving into Out-of-Distribution Detection with Vision-Language Representations.

$2$ Jiang et al. Negative Label Guided OOD Detection with Pretrained Vision-Language Models.

$3$ Esmaeilpour et al. Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP.
W5: Suggest expanding discussion in Sec. 5 to include newer GAN/diffusion-based OOD methods (e. g. , $a$ with GAN, $b-c$ with training-free Stable Diffusion) rather than older VAE-based approaches.
Reply:
Thank you for your suggestions. These three articles will be incorporated into the Related Work section.
W6: I noticed that the image generation time is pretty long in Table 19, which should be emphasized in the main paper as one of the limitations.
Reply:
We acknowledge this as a limitation of our current framework and will include it in the limitations. However, we note that the computational efficiency can be significantly optimized without compromising detection performance.
Due to space limitations, Reviewer ViPC raised the same question. Please refer to our detailed response under W5 in the Reviewer ViPC section.
W7: The authors have talked about different CLIP models but the architectures for diffusion models are similar. The authors are encouraged to consider other models such as DiT-based ones.
Reply:
We appreciate the suggestion. we incorporate synthetic images generated by Hunyuan-DiT $1$ in our experiments, compared to SD1.5, Hunyuan-DiT achieved better results on the FPR95 metric.Due to space constraints, the complete table is available at:
https://anonymous.4open.science/r/fjutlfy-31D8 Table J.

$1$ Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding.
Table E: Performance comparison between SD model and DiT models in the DualCnst method.

Response to Reviewer yyQ7
We thank the reviewer yyQ7 for the valuable feedback. We addressed all the comments. Please find the point-to-point responses below. Any further comments and discussions are welcomed!
W1: While the method’s simplicity is a strength, it risks underselling technical novelty. Expand theoretical guarantees or visual evidence to bolster significance.
Reply:
Thank you for your comment. Yes, our method is simple and effective, but it initially lacked a theoretical explanation. To address this, we have supplemented the theoretical analysis. The proposed method primarily enhances the diversity of the label space by incorporating multi-level synthetic image features into the existing text-based labels. Theoretically, we consider a more general scenario—how expanding multimodal labels improves OOD detection. We prove that, under certain conditions, the false positive rate ($\\text{FPR}\_\\lambda$) decreases as the number of multimodal labels increases, demonstrating that incorporating additional auxiliary modalities into labels enhances OOD detection performance. The specific theoretical details are as follows:

Theoretical Analysis of Multimodal Label Enhancement

Core Contribution
We prove that expanding multimodal labels reduces OOD detection false positive rate ($\text{FPR}_\lambda$). The method enhances label diversity through multi-level synthetic image features, improving separability between ID/OOD samples.

---

Key Theoretical Steps

1. ​Multimodal Label Definition
   Define $N$-modal negative labels $\\widetilde{Y}\_i = \\{\widetilde{y}\_{i,1}, \\dots, \\widetilde{y}\_{i,N} \\}$, where $\\widetilde{y}\_{i,1}$ is the primary modality (text) and $\\widetilde{y}\_{i,2}, \\dots, \\widetilde{y}\_{i,N}$ are auxiliary modalities (synthetic image embeddings).

2. ​Weight Allocation
   Assign non-uniform weights:
   w\_j \= \\begin{cases} \\frac{a}{N}, & j=1 \\quad \\text{(primary modality weight)} \\\\ \\frac{1 \- \\frac{a}{N}}{N \- 1}, & j=2, \\dots, N \\quad \\text{(auxiliary modality weights)} \\end{cases}

3. ​Aggregated Similarity Score
   Compute weighted similarity:
   $s_i = \frac{a}{N} s_{i,1} + \sum_{j=2}^N \frac{1 - \frac{a}{N}}{N - 1} s_{i,j}$

4. ​Statistical Properties
   For i.i.d. $s_{i,j} \sim (\mu, \sigma^2)$:
   $\mathbb{E}[s_i] = \mu, \quad \text{Var}(s_i) = \frac{\sigma^2}{N'(N)}, \quad N'(N) = \frac{N(N-1)}{a^2 + N - 2a}$

5. ​OOD Score Distribution
   Match count $c = \sum_{i=1}^M \mathbb{I}[s_i \geq \psi]$ follows:
   $c_{\text{in}} \sim \mathcal{N}(Mp_1, Mp_1(1-p_1)), \quad c_{\text{out}} \sim \mathcal{N}(Mp_2, Mp_2(1-p_2))$
   where $p_1 = 1-\Phi(k_1\sqrt{N'})$, $p_2 = 1-\Phi(k_2\sqrt{N'})$ with $k_1 = \frac{\psi-\mu_1}{\sigma}$, $k_2 = \frac{\psi-\mu_2}{\sigma}$.

6. ​FPR Analysis
   $\text{FPR}_\lambda = \Phi\left(\frac{\sqrt{M}(p_1 - p_2) + \sqrt{p_1(1-p_1)}\Phi^{-1}(\lambda)}{\sqrt{p_2(1-p_2)}}\right)$

7. ​Critical Result
   $\\frac{\partial \\text{FPR}\_\\lambda}{\\partial N} < 0 \\quad \\text{when} \\quad \\mu_1 + \\frac{\\sigma}{\\sqrt{N'}} > \\psi > \\mu_2$ ​Conclusion: Increasing $N$ strictly reduces $\text{FPR}_\lambda$.

---

Remark on Assumptions

• The i.i.d. assumption on $s_{i,j}$ can be relaxed to dependent variables.
• Variance reduction ($\text{Var}(s_i) \propto 1/N$) remains the driving mechanism.

Theoretical Impact: Provides mathematical guarantee for multimodal label enhancement in OOD detection. A detailed proof will be provided in the paper.

W2: The main contribution of this paper is to enhance the zero-shot OOD detection capability of the CLIP model. Recently, there have been many related papers in this direction, such as $1, 2$ . It is suggested that the authors add discussion and comparison. $1$ LAPt: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models, ECCV2024 $2$ CLIPScope: Enhancing Zero-Shot OOD Detection with Bayesian Scoring
Reply:
We appreciate the reviewer's valuable suggestion. In response, we have incorporated the LAPt method into the main experiments for comparison, as shown in Table A. Although LAPt adapts to ID data by fine-tuning prompt parameters, our method still achieves superior performance on the FPR95 metric.

Table A: Experimental comparisons on the ImageNet-1k task, with iNaturalist, SUN, Places, and Texture used as out-of-distribution (OOD) data for evaluation.

Response to Reviewer ViPC
We thank the reviewer ViPC for the valuable feedback. We addressed all the comments. Please find the point-to-point responses below. Any further comments and discussions are welcomed!
W1: The paper uses OOD-specific α (tuned per OOD dataset) , violating OOD agnosticism.
Reply:
We thank the reviewer for the feedback. Experiments with a fixed α=0.1 achieve performance comparable to our original method while retaining significant advantages over baselines. Due to space constraints, the complete table is available at:
https://anonymous.4open.science/r/fjutlfy-31D8 Table A.
Table A: Comparison with Other Baselines Using Fixed α=0.1.

W2: Sec4's fixed ω contradicts method's varying encoder ω (sum=1). Typo? (e.g., should be r)?
Reply:
Thank you for catching this - we've corrected ω to r in the manuscript.
W3: The current OOD detection framework oversimplifies evaluation. Adopting full-spectrum OOD would enable rigorous validation.
Reply:
Following OpenOOD v1.5 protocols, we now use ImageNet-1k/R as ID with near-OOD benchmarks (SSB-Hard, NINCO). Our method maintains superiority over baselines in this enhanced evaluation.
Due to space constraints, the complete table is available at:
https://anonymous.4open.science/r/fjutlfy-31D8 Table B.
Table B: Robustness experiments on ImageNet-1k and ImageNet-R.

W4: VLM-based OOD methods (e. g. , LoCoOp $1$ /LSA $2$ ) already use visual semantics without costly generators (e. g. , SD) . Comparison needed to justify added complexity.
Reply:
We thank the reviewer for the suggestion. Added comparisons with LoCoOp (Table C) and LSA (Table D) show our method achieves superior performance over few-shot VLM baselines, with added compatibility for text enhancement (e.g., NegLabel). While LSA's official code is not fully released, we implemented it on their original dataset for fair evaluation. Due to space constraints, the complete table is available at:
https://anonymous.4open.science/r/fjutlfy-31D8 Table C & D.
Table C: Experimental comparisons on the ImageNet-1k benchmark.

Table D: Experimental comparisons on the ImageNet-1k task, with near and far datasets evaluated as OOD data.

W5: 1. High time cost of Stable Diffusion image generation undermines real-time deployment claims. 2. Limited methodological novelty: The approach primarily combines NegLabel with text-to-image synthesis
Reply:
R1:
Regarding the efficiency of Stable Diffusion generation, we address the issue through two key optimizations:
First, numerous accelerated versions of Stable Diffusion are now available. SDXL-Turbo achieves 10× faster generation than SD1.5 (reducing time from 55m42s to 4m10s) while maintaining equivalent detection performance (Table E). This acceleration also enhances overall method performance (Table F).
Second, our synthetic images only need to be generated once, eliminating the need for repeated generation. Due to space constraints, please refer to the full table at the link https://anonymous.4open.science/r/fjutlfy-31D8 Table E & F.
Table E: Time comparison for accelerated SD models.

Table F: Performance comparison between SD models in the DualCnst method.

**R2: ** To better understand our approach, we provide deeper theoretical insights into our approach, and have supplemented the theoretical analysis. The proposed method primarily enhances the diversity of the label space by incorporating multi-level synthetic image features into the existing text-based labels. Theoretically, we consider a more general scenario—how expanding multimodal labels improves OOD detection. We prove that, under certain conditions, the false positive rate ($\\text{FPR}\_\\lambda$) decreases as the number of multimodal labels increases, demonstrating that incorporating more auxiliary modalities in labels enhances OOD detection performance. Due to space limitations, the detailed theoretical analysis can be found in the response to Reviewer yyQ7-W1.