Enhancing Zero-shot OOD Detection with Pre-trained Multimodal Foundation Models

We sincerely thank the reviewer for the thoughtful and detailed feedback. Below, we respond to the main concerns:

W1. Novelty and Contribution

We respectfully disagree with the assessment that our work lacks novelty. While we acknowledge that retrieval-augmented generation (RAG) and negative sample generation have been explored in many domains, our contributions are distinct in the context of OOD detection:

• Formulation-level innovation: We introduce a novel formulation of zero-shot OOD detection by explicitly characterizing the types of reference signals that are most effective for distinguishing ID from OOD samples. Specifically, we propose using dynamically generated, multi-modal, and test-aware references, in contrast to prior works that rely on statically selected, single-modal, and ID label-aware references. This formulation provides a principled foundation for zero-shot OOD detection and explains the superior performance observed in our empirical results.
• Method-level Innovation: We propose a Model- and Modality-Agnostic, Training-free, Open-set Aligned, Online pipeline, in contrast to prior methods that are VLM-restricted, require training or fine-tuning when datasets or architectures change, and depend on offline-constructed references. These differences are crucial in practice:
  • Model- and Modality-Agnostic: Refer-OOD works seamlessly with both VLMs and MLLMs without any fine-tuning or architectural changes, making it broadly compatible and future-proof as foundation models evolve.
  • Training-Free and Open-Set Aligned: Our method avoids any supervised training and instead leverages pretrained models in a zero-shot manner. This not only simplifies deployment but also better preserves the open-set nature of OOD detection, reducing the risk of overfitting to specific known ID classes.
  • Online and Adaptive: Unlike methods relying on training on static offline retrieval, our pipeline retrieves references online, allowing dynamic adaptation to current inputs and real-time semantics.

W2. Hallucination

Hallucination is an inherent, likely long-standing issue in MLLMs, yet this should not to be a reason to reject any works built on MLLMs. Our work is hardly affected by hallucination thus does not focus on addressing this problem.

W3. Hyperparameters

There may be a misunderstanding regarding the number of hyperparameters. In practice, only one parameter, $\tau$​ (the filtering threshold), requires tuning, as it depends on the semantic gap between the ID and OOD classes.

• $m$ and $n_i$ simply control the number of generated reference labels and retrieved images, respectively. Both theory and experiments show that larger values consistently improve performance.
• $\beta$ and $\alpha$ are fusion weights, which we empirically observe to be stable across datasets.
• $\lambda$ is a standard threshold in OOD detection. For metrics like AUROC and FPR95, it is automatically swept during evaluation.
• Only $\tau$​ is dataset-dependent and requires tuning based on the semantic similarity gap between ID and OOD.

Hence, our method does not introduce 6 independent, sensitive hyperparameters.

W4&5. Datasets

• We followed the setting of EOE and focused primarily on fine-grained datasets, which pose greater challenges for OOD detection and better reflect real-world ambiguity.

• Regarding ImageNet-1k:

  • Inconsistent dataset splits: Due to its large scale and associated computational costs (especially API usage during retrieval and generation), we report results on a randomly chosen subset.

  • Prompt sensitivity and standardized implementation: Prompt choices can cause up to 10% fluctuation in FPR95 for methods like NegLabel, explaining discrepancies in previous results. We adopt "a photo of a <label>" as the standardized prompt for fair comparison.

Final Remark

We hope our clarifications help highlight the unique contributions and practical impact of Refer-OOD. We are more than willing to provide additional clarification should any concerns remain.

Thanls for the rebuttal. After reading the response, I keep my score unchanged due to the following reasons

1. Since Hallucination is an inherent, likely long-standing issue in MLLMs, it definitely should be taken into consideration when using MLLMs rather than simply assuming the output of MLLMs is reliable.

2. Reporting results on a randomly chosen subset is not justified. Besides, given the optimal prompt of NegLabel is "the nive <label>", using "a photo of a [object Object]" may lead to unfair comparision.

3. As the author said, using MLLMs requires large scale and associated computational costs, which is a significant disadvantage and implies the lack of practical viability. such large-scale computational costs contributes to marginal peformance improvement (1.6 FPR and 0.2 AUROC) on large-scale challenging imagenet (the widely used benchmark in the literature of CLIP-based OOD detection).

4. The authors are expected the compare the memory usage among the proposed, NegLabel and CSP.

5. From my view, the claim 'In practice, only one parameter, $\tau$​ (the filtering threshold), requires tuning' is somewhat confusing. While $\lambda$ is a standard threshold in OOD detection, there is still 5 tunable hyper-parameters essential ot the algorithm designing. Besides, the results of hyper-parameters sensitiveness on $\beta$ and $\alpha$ (Fig 6 and Fig 7) does not support the stability.

6. Regarding the response to W1, how does your method reflect “dynamically generated” and 'online pipeline'. What I mean here is that the authors use highly abstract terms to make the responses not informative. Besides, authors are expected to clarity 1) how the Reference Acquisition Module different from ECE and the mining strategy in NegLabel and 2) Eq. 9 in VLM-based Mapping Module is exactly same as EOE.

7. After checking the manuscript again, what does the term 'optimize ' in lines 173-174 (In the generation phase, we optimize the prompt forms to guide the MLLM toward producing the highest semantically aligned labels) mean.

We thank the reviewer for the time and effort spent reviewing our work. Below, we provide detailed responses to the identified weaknesses and questions.

Summary Misunderstanding

We respectfully clarify a key misunderstanding: our method is not merely a VLM-based framework, but the first to incorporate MLLMs into zero-shot OOD detection. To better distinguish the IDs and OODs, we dynamically construct test-specific multimodal references through generation, filtering, and retrieval.

W3. Novelty

We respectfully disagree with the assessment that our work lacks novelty. While we acknowledge that retrieval-augmented generation (RAG) and negative sample generation have been explored in many domains, our contributions are distinct in the context of OOD detection:

• Formulation-level innovation: We introduce a novel formulation of zero-shot OOD detection by explicitly characterizing the types of reference signals that are most effective for distinguishing ID from OOD samples. Specifically, we propose using dynamically generated, multi-modal, and test-aware references, in contrast to prior works that rely on statically selected, single-modal, and ID label-aware references. This formulation provides a principled foundation for zero-shot OOD detection and explains the superior performance observed in our empirical results.

• Method-level Innovation: We propose a Model- and Modality-Agnostic, Training-free, Open-set Aligned, Online pipeline, in contrast to prior methods that are VLM-restricted, require training or fine-tuning when datasets or architectures change, and depend on offline-constructed references. These differences are crucial in practice:

  • Model- and Modality-Agnostic: Refer-OOD works seamlessly with both VLMs and MLLMs without any fine-tuning or architectural changes, making it broadly compatible and future-proof as foundation models evolve.
  • Training-Free and Open-Set Aligned: Our method avoids any supervised training and instead leverages pretrained models in a zero-shot manner. This not only simplifies deployment but also better preserves the open-set nature of OOD detection, reducing the risk of overfitting to specific known ID classes.
  • Online and Adaptive: Unlike methods relying on training on static offline retrieval, our pipeline retrieves references online, allowing dynamic adaptation to current inputs and real-time semantics.

• While both methods condition on the test sample, our approach is distinct from ZOC (Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP) in several ways:

• While both methods use generation and retrieval, our approach is distinct from LAPT (LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models) in several ways:

W1. Inference cost and latency

As noted in the Appendix (Table 7), we report per-sample latency (generation + retrieval). Calling APIs and performing online retrieval inevitably introduce non-negligible latency.

To improve efficiency and practicality, some strategies can be considered, for example,

• When the distributions of OOD test samples are similar, textual generation can be reused or cached to improve efficiency.
• Retrieval can be partially cached or parallelized, especially for ID samples, to reduce latency without sacrificing performance.

While more efficient system design strategies may be possible, they are beyond the scope of this paper.

W2. Comparison with LAPT

• Model Robustness: We clarify that our concern about generalization risk targets training-based methods. Fine-tuning on limited ID data — especially under a binary OOD formulation — can cause overfitting and reduce the entropy and diversity of representations. This harms the generalization ability gained from large-scale pretraining, which is especially important for the open-set nature of OOD detection.

• Computational cost: While LAPT is inference-efficient compared with our un-cached method, it requires retraining for each new ID dataset, which is costly in dynamic settings. In contrast, our zero-shot, plug-and-play method works directly with both VLMs and MLLMs, offering greater flexibility when ID data changes frequently. Notably, with caching, the inference time of our method becomes comparable to LAPT, with minimal discrepancies.

Q1. Quality of generated OOD labels

The quality of generated OOD labels is carefully examined in our theoretical analysis, ablation studies, and case study. Specifically:

• Theoretically, desirable OOD labels are those that are more diverse (i.e., with a larger $m$), dynamically related to the input image $x$, yet clearly distinct from the ID label — specifically, they should exhibit low $p_{in}$ and high $p_{out}$.
• Empirically, our method outperforms others that also use negative OOD labels, demonstrating the generated OOD labels are more effective and contribute meaningfully to overall performance..
• Qualitatively, Figure 3 presents a detailed case study showing how the generated OOD labels adapt to the test sample and contribute to correct detection.

Q2. Prompt variations

• Prompts for Refer-OOD-VLM:

  • Robust for label generation prompt: We have conducted experiments (AUROC) with alternative label generation prompts.

    Label Generation Prompt	ImageNet->Texture	CUB->CUB	CUB->iNaturalist
    relevant	92.16	84.11	99.97
    similar	92.85	83.02	99.98
    resemble	93.07	83.37	99.97

  • Robust for VLM text encoder prompt: Unlike NegLabel and LAPT, our method, with improved reference quality, needs no prompt adjustments but still yields significant performance gains.

• Prompts for Refer-OOD-MLLM:

  • Robust for label generation prompt: We have conducted experiments (F1) with alternative label generation prompts.

    Label Generation Prompt	ImageNet->Texture	CUB->CUB	CUB->iNaturalist
    relevant	73.51	72.50	92.14
    similar	74.96	72.65	91.86
    resemble	74.28	71.35	91.75

Q3. Availability of online search / fallback strategy

Our current implementation uses Bing Search API and a browser plugin for retrieval. However, this retrieval is fundamentally based on query-document similarity matching, and can easily be substituted by an offline retrieval corpus (e.g., LAION-5B or a local web-scale dataset).

Additional Clarifications

Minor Typos: We appreciate the reviewer’s attention to such details and will correct the identified typos accordingly.

Final Remark

We hope these responses address your concerns and clarify the key strengths and contributions of our work. We are more than willing to provide additional clarification should any concerns remain.

We thank Reviewer Av1q for the thoughtful and constructive feedback. We are encouraged by the positive comments on our method’s simplicity, novelty, and extensive experimentation. Below, we address the raised concerns in detail.

W1. Struggles with Fine-Grained Classes and Domain Limitations

We appreciate the reviewer’s insight into this challenge. While it is true that VLMs/MLLMs may struggle with fine-grained or domain-specific categories, we highlight that our method was designed specifically to mitigate this:

• Our framework outperforms prior methods on fine-grained benchmarks, showing its strength in handling subtle semantic distinctions.
• For specialized domains like medical or aerial imagery, we agree that textual generation may be less reliable. One potential solution is to extend beyond label-level references to:
  • Explicit descriptions (e.g., token-level captions) and/or
  • Implicit image-driven retrieval, leveraging feature similarity or offline domain-specific datasets.

Although no method can fully generalize to all domains, our modular pipeline allows easy adaptation, making it a promising foundation for future extensions.

W2. Writing Clarity and Equation Complexity

We appreciate the suggestion and will polish the writing to improve clarity.

Q1. Disentangling VLM/MLLM vs. Search Engine Contributions

While MLLM-generated text provides semantic anchors, the retrieval module compensates for label ambiguity, especially in fine-grained settings. For example, in the bird dataset CUB (e.g., Brandt Cormorant), VLMs may not capture fine labels well due to limited pretraining coverage. However, retrieved reference images from the web contain rich visual cues, helping align ambiguous classes.

Q2. Using Reverse Retrieval

We further evaluate our method by using the reverse retrieval (test image as a query) for online retrieval and comparing the mean feature of the retrieved samples against that of our generated textual references.

The performance drop with reverse retrieval may be due to its tendency to return visually similar but semantically uninformative samples.

Q3. Using Newer VLM/MLLMs (e.g., Qwen 2.5VL, SigLIPv2)

• SigLIPv2 for Refer-OOD-VLM: We additionally evaluate our method on SigLIPv2 (siglip2-base-patch16-256 ) compared with CLIP as Refer-OOD-VLM encoder using Qwen-based generation.

  Model	CUB	Stanford-Cars	Food	Oxford-Pet
  CLIP	83.08	77.42	92.82	91.00
  AltCLIP	86.76	84.19	85.93	89.60
  SigLIPv2	78.18	83.99	85.20	95.14

  This suggests that SigLIPv2 can complement CLIP but may not universally outperform it across all domains in fine-grained OOD detection.

• Qwen2.5VL: In our experiments, we used the qwen-vl-max-2025-04-02 model, which is an updated version of qwen-vl-max-2025-01-25(upgraded to the Qwen2.5-VL architecture).

Q4. Datasets Settings

• We thank the reviewer for the suggestion. We conducted experiments by swapping the ID and OOD datasets. Note that this setup only changes which classes are labeled (i.e., treated as ID), while the overall task structure and test images remain the same. Therefore, any well-generalized method should perform consistently under this change.

  As shown below, Refer-OOD maintains stable performance across all datasets, confirming that it does not rely on specific ID categories but generalizes effectively to different partitions.

  • Refer-OOD-VLM (AUROC) :

    	CUB	Stanford-Cars	Food	Oxford-Pet
    origin	83.08	77.42	92.82	91.00
    swap	79.01	74.29	94.83	87.12

  • Refer-OOD-MLLM (F1) :

    	CUB	Stanford-Cars	Food	Oxford-Pet
    origin	72.80	77.83	81.67	89.07
    swap	71.91	74.31	84.53	90.62

• OOD detection is harder when ID and OOD data are semantically similar, but till now, there's no standard metric to quantify the difficulty of such dataset settings.

• Since our method performs zero-shot OOD detection with only ID labels and no access to the ID dataset itself, dropping the ID dataset does not affect the results.

Final Remark

We appreciate the reviewer’s relevant and thoughtful comments. We believe the paper’s simplicity, extensibility, and empirical gains make it a valuable contribution to zero-shot OOD detection.

We thank Reviewer muED for the constructive comments and thoughtful questions. We appreciate the positive recognition of our framework, theoretical insights, and extensive experiments. Below, we address each concern in detail.

W3. Novelty Clarification

We respectfully disagree with the assessment that our work lacks novelty. While we acknowledge that retrieval-augmented generation (RAG) and negative sample generation have been explored in many domains, our contributions are distinct in the context of OOD detection:

• Formulation-level innovation: We introduce a novel formulation of zero-shot OOD detection by explicitly characterizing the types of reference signals that are most effective for distinguishing ID from OOD samples. Specifically, we propose using dynamically generated, multi-modal, and test-aware references, in contrast to prior works that rely on statically selected, single-modal, and ID label-aware references. This formulation provides a principled foundation for zero-shot OOD detection and explains the superior performance observed in our empirical results.
• Method-level Innovation: We propose a Model- and Modality-Agnostic, Training-free, Open-set Aligned, Online pipeline, in contrast to prior methods that are VLM-restricted, require training or fine-tuning when datasets or architectures change, and depend on offline-constructed references. These differences are crucial in practice:
  • Model- and Modality-Agnostic: Refer-OOD works seamlessly with both VLMs and MLLMs without any fine-tuning or architectural changes, making it broadly compatible and future-proof as foundation models evolve.
  • Training-Free and Open-Set Aligned: Our method avoids any supervised training and instead leverages pretrained models in a zero-shot manner. This not only simplifies deployment but also better preserves the open-set nature of OOD detection, reducing the risk of overfitting to specific known ID classes.
  • Online and Adaptive: Unlike methods relying on training on static offline retrieval, our pipeline retrieves references online, allowing dynamic adaptation to current inputs and real-time semantics.

W1 / Q1 Computational Cost and Inference Latency

As noted in the Appendix (Table 7), we report per-sample latency (generation + retrieval). Calling APIs and performing online retrieval inevitably introduce non-negligible latency.

To improve efficiency and practicality, some strategies can be considered, for example,

• When the distributions of OOD test samples are similar, textual generation can be reused or cached to improve efficiency.
• Retrieval can be partially cached or parallelized, especially for ID samples, to reduce latency without sacrificing performance.

While more efficient system design strategies may be possible, they are beyond the scope of this paper.

W1&Q2. Theoretical Assumptions and Gaussian Approximation

To support the i.i.d. and Gaussian assumptions, we visualized similarity score distributions for ID and OOD samples, which align well with Gaussian patterns. Due to OpenReview limitations, the plots will be included in the revised paper.

Q2. Online Search and Reproducibility

We acknowledge the concern about reproducibility and API reliance. While our current setup uses Bing Search, our method is agnostic to the retrieval source. Any retrieval engine (e.g., Google, Baidu, or even offline datasets like LAION) can be used.

Q3. Retrieval Robustness to API and Adversarial References

• Retrieval API quality: Table 11 indicates that performance is relatively stable across different retrieval platforms.

• Image quality: We tested the effect of noisy retrieval results by injecting unrelated images into the retrieved set. The experiments are conducted on the dataset Food.

  	0%	5%	10%	25%	50%	75%	100%
  Visual	91.15	90.84	90.96	90.47	89.24	88.87	88.64
  Multi-modal	92.18	92.15	92.16	92.15	91.64	91.51	91.27

  Excessive image noise degrades OOD detection performance, though the impact is negligible at low noise levels, and multi-modal supplementation effectively mitigates such degradation.

Q4. Adaptive Generation vs. EOE Baseline

Yes — as shown in our ablation study (Table 5), even using only the generation module (Dynamic Textual) outperforms EOE (Fixed Textual).

Final Remark

Once again, we sincerely thank the reviewer for the helpful suggestions. We believe our adaptive multimodal framework, validated both theoretically and empirically, makes a solid contribution to zero-shot OOD detection, and we will revise the paper to better emphasize practicality and robustness.