OOD-Barrier: Build a Middle-Barrier for Open-Set Single-Image Test Time Adaptation via Vision Language Models

We sincerely thank you for your time and effort in reviewing our work. We are grateful for your recognition of the novelty of our idea and the robustness of our methodology.

Thank you for your feedback on the experimental section. We believe that your concerns can be addressed through a review of the original details, as well as the additional experiments and analysis provided. To facilitate your reading and help you locate key points in the original text, we have highlighted the important parts in bold.

---

[Weakness 1] Condensed Information in Paper

We sincerely appreciate your recognition of the detailed aspects of our work, as well as your constructive suggestions regarding the overall structure. In our writing, we aimed to convey as much relevant information as possible to facilitate understanding, which may have led to an overly condensed presentation. In future revisions, we will streamline certain parts of the text and reallocate some of the detailed content to the supplementary material for improved clarity and readability.

---

[Weakness 2] Additional Baselines (Tent [2], WATT [3], OSTTA [1])

Thank you for your suggestion. Our work specifically targets single-image-level test-time adaptation (TTA) in open-set recognition. Many classical TTA methods, including Tent, WATT, and OSTTA, were not originally designed for this specific setting. To ensure a fair comparison, we adapted these methods accordingly (Line 660) to align with our evaluation protocol.

Due to space limitations, we have to adopt representative baselines in the main paper. In response to your comments, we have conducted additional experiments incorporating Tent, WATT, and OSTTA. Specifically:

1. For the classical closed-set TTA method Tent, we integrated it with our original Zero-shot Evaluation (ZS-Eval) setup.
2. For WATT and OSTTA, we applied a fair and consistent evaluation strategy as described in Line 660.

We observe that OSTTA shows the most notable improvement due to its alignment with open-set conditions. Tent and WATT primarily focus on test-time adaptation without explicitly modeling the distinction between ID and OOD data. Although OSTTA is more suitable for open-set scenarios, it still falls short in adaptability compared to more recent methods like [5] and our approach with 7.18 average FPR improvements.

We acknowledge the important role these baselines have played in advancing the field, and we believe our work builds upon and extends their foundational contributions in the context of open-set, single-image TTA.

---

[Weakness 3] Does Corruption Dataset Influence Experimental Results?

Thank you for your valuable suggestion. We carefully addressed this concern in our experimental design. In Table 3 (Line 308), we present experiments with uncorrupted datasets, using VisDA as the ID dataset and MNIST and SVHN as OOD datasets—none of which are corrupted. Additionally, we evaluated settings where both ID and OOD data are corrupted, as shown in Table 1 (Line 275) and Table 7 (Line 750), for CIFAR-10C and CIFAR-100C. For example, in the MapLe-based CIFAR-10C to CIFAR-100C experiment, our method outperforms the second-best method by 1.72 in FPR and improves accuracy by 0.41.

Additionally, in response to [Weakness 2], we conducted experiments with clean CIFAR-10 and CIFAR-100 datasets for ID, as well as MNIST, SVHN, and Tiny-ImageNet for OOD. Compared to the full ImageNet, we used Tiny-ImageNet for more efficient validation. These results further show that performance trends between our method and baselines remain consistent, regardless of corruption.

In these settings, our method consistently performs well, demonstrating that its effectiveness does not rely on data corruption. Furthermore, if corruption introduces domain shifts, all methods would be equally affected, ensuring fairness in the evaluation.

From a theoretical perspective, as outlined in Hypothesis 2, our method models the distance between ID, OOD, and the intermediate domain, driven by semantic-level distributional differences, which corruption does not fundamentally alter. For example, even if a cat image is corrupted, its core semantic representation as a "cat" remains intact.

---

[Question 1] On the Choice of Harmonic Accuracy and Related Metrics

Thank you for your insightful comment. We agree that harmonic accuracy alone does not explicitly separate ID classification from ID-OOD discrimination performance, as it reflects only their overall balance.

To address this, we include two additional metrics—FPR@95 (Line 641) and AUROC (Line 627)—to directly evaluate ID-OOD discrimination. Lower FPR@95 and higher AUROC values indicate better separation between ID and OOD samples.

Together, FPR@95 and AUROC capture OOD detection performance, while harmonic accuracy helps reflect ID classification indirectly. This combination provides a more comprehensive assessment of model performance under open-world settings.

---

[Question 2] Potential Impact of Corruption Dataset on Experimental Results

As discussed in [Weakness 3] and supported by the experiments in [Weakness 2], we conclude that our method does not directly benefit from dataset corruption. The observed performance improvements stem from the effective modeling of the distance between ID and OOD distributions, as described in Hypothesis 1 and Hypothesis 2.

---

[Question 3] Final Score Calculation via Equation 3

Thank you for your comment. Indeed, the final ID/OOD score is computed using Equation 3. To clarify, Section 3 is structured as follows: Section 3.1 focuses on the calculation of the ID/OOD score based on Hypothesis 1, while Section 3.2 introduces the concept of the intermediate domain and the test-time adaptation method, which is developed around Hypothesis 2.

---

[Question 4] Additional Experiments (Tent [2], WATT [3], OSTTA [1])

Thank you for your suggestion regarding the experiments. As per your recommendation, we have included the requested experiments with Tent [2], WATT [3], and OSTTA [1] in [Weakness 2]. All experiments were conducted using uncorrupted datasets.

---

Reference

[1] Lee, Jungsoo, et al. "Towards open-set test-time adaptation utilizing the wisdom of crowds in entropy minimization." ICCV . 2023.

[2] Wang, Dequan, et al. "Tent: Fully test-time adaptation by entropy minimization." arXiv preprint arXiv:2006.10726 (2020).

[3] Osowiechi, David, et al. "WATT: Weight average test time adaptation of CLIP." Advances in neural information processing systems 37 (2024): 48015-48044.

[4] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models." NeurIPS 35 (2022): 14274-14289.

[5] Sreenivas, Manogna, and Soma Biswas. "Efficient Open Set Single Image Test Time Adaptation of Vision Language Models." arXiv preprint arXiv:2406.00481 (2024).

Thank you very much for your valuable suggestion and insightful feedback. We do have the standard ID accuracy results, as they are used to compute the harmonic accuracy in our paper. We apologize for not including them during the rebuttal phase.

Below, we provide the complete accuracy results, including the standard ID Accuracy, which clearly reflects whether a method experiences degradation on ID samples under the presence of OOD inputs during test-time. We hope this resolves your concern:

The first table is derived from the results presented in our rebuttal [Weakness 2], while the latter two are reproduced from Table 3 and Table 7 in our main paper. These results illustrate the performance of various methods on the ID dataset in the presence of OOD samples.

From the tables, it is evident that despite the presence of OOD inputs, our method consistently achieves either the best or second-best performance in ID accuracy across multiple tasks. This further supports the effectiveness of our approach in maintaining ID performance while ensuring robust OOD detection.

[1] Lee, Jungsoo, et al. "Towards open-set test-time adaptation utilizing the wisdom of crowds in entropy minimization." ICCV . 2023.

[2] Wang, Dequan, et al. "Tent: Fully test-time adaptation by entropy minimization." arXiv preprint arXiv:2006.10726 (2020).

[3] Osowiechi, David, et al. "WATT: Weight average test time adaptation of CLIP." Advances in neural information processing systems 37 (2024): 48015-48044.

[4] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models." NeurIPS 35 (2022): 14274-14289.

[5] Noisy Test-Time Adaptation in Vision-Language Models, ICLR 2025

[6] Li, Yushu, et al. "On the robustness of open-world test-time training: Self-training with dynamic prototype expansion." ICCV. 2023.

[7] Karmanov, Adilbek, et al. "Efficient test-time adaptation of vision-language models." CVPR. 2024.

[8] Gao, Zhengqing, Xu-Yao Zhang, and Cheng-Lin Liu. "Unified entropy optimization for open-set test-time adaptation." CVPR. 2024.

Thanks to the authors for their efforts in responding to my questions.

In regard of the competitive results in ID accuracy, I believe their contribution is valuable. I think that harmonic accuracy could become an interesting standard metric for the future of OOD TTA, but encourage the authors to include these ID results in a next version of the paper.

In light of the provided evidence and the authors' effort in responding my concerns and those from other reviewers, and as a contribution to OOD TTA (a field that is just starting to be more widely explored in VLMs), I would increase my score to borderline accept.

Dear Reviewer,

We sincerely appreciate your thoughtful question regarding the ID accuracy under the presence of OOD inputs. As a follow-up, we’ve provided detailed results addressing this point, demonstrating that our method maintains strong ID accuracy while achieving robust OOD detection.

Since the discussion phase is approaching its end, we would like to kindly check if there are any further questions or concerns we can help clarify.

Thank you again for your valuable feedback and time.

Thank you for your time and consideration. We appreciate your recognition of the novelty and practicality of our method.

We believe that your concerns can be addressed by providing further clarification of the technical details, allowing us to reach a mutual understanding. We would like to highlight the original details corresponding to the bolded lines in the rebuttal section, as they provide further clarification and location in paper. Your attention to these details would be greatly appreciated.

---

[1-1] Motivation Behind the Method in the Introduction

The motivation behind our Open-IRT method is intuitively presented in Figure 1 and its caption (Line 47) and Hypothesis 1 & 2: the separation of ID and OOD distances through an intermediate domain, with the focus on expanding the distances within the separated regions to achieve a better distinction. Due to space limitations in the Introduction, we were unable to elaborate on the complete motivation for the method or specific terms (such as "polarity"). Therefore, the full motivation for our method is elaborated in the Methodology Section, particularly around Hypothesis 1 (Line 47) and Hypothesis 2 (Line 191).

For Hypothesis 1, we design an ID/OOD scoring system based on the differential representation capabilities of the visual-language model's text encoder for prompts. In Hypothesis 2, we construct an intermediate domain. To substantiate the validity of the intermediate domain, we provide both Theoretical Justification (Line 559) and experimental verification (Line 323) in the appendix, along with a clear visualization of the intermediate domain (Figure 4).

---

[1-2] L144-L148 Motivation of the Method in the Introduction Section

As discussed in [1-1], we provided only a brief overview of the motivation for the method in Figure 1 (Line 47). A more detailed explanation is found in Hypothesis 1 (Line 47), and Hypothesis 2 (Line 191). The theoretical foundations for the method are further analyzed and validated in the supplementary materials (Lines 487-583).

---

[1-3] Explanation of Symbols in Figure 1

Due to the need for clarity and conciseness, we opted for abbreviations in Figure 1. This process corresponds to Equation 3 (Line 164), where:

$sim\_{pos} = sim(f, p\_c)$, $sim\_{neg} = sim(f, p_c')$, $sim\_{diff} = \left| sim(f, p\_c) - sim(f, p\_c') \right|$

These quantities are used to compute scores in Equation 3. In the context of the paper, Prototypes refers to the embedded prompt tensors.

---

[1-4] Definition of Sliding Window Strategy

We adopt the standard definition of Sliding Window in sequence data processing from computer science [1], where, in the context of our paper, it functions equivalently to a FIFO queue.

---

[1-5] $f\_s$ Computation Paradigm

The method for calculating the variance of $f_s$ has been validated for its rationality in [2], where it is computed across spatial dimensions independently for each feature channel. The feasibility of this approach lies in preserving channel-specific style information. The variance for each channel emphasizes fine-grained structural characteristics in that feature dimension, such as color, texture intensity, etc., without mixing with other channels, thereby achieving refined style representation and alignment.

---

[1-6] Differences between Equation 8 and Equation 9

The differences between Equation 8 and Equation 9 arise from the distinction between ID and OOD categories in the standard OOD detection task [3]. The overall setup of the OOD detection task is to first perform ID/OOD binary classification, followed by internal classification within ID data [3], without considering internal classification for OOD data. Therefore, in Equation 8, for ID data, the model needs to learn fine-grained semantic consistency and achieve precision for each class, which is why we set $y^{+} = \hat{y}\_{p}$, where $\hat{y}\_{p}$ is the pseudo-label designed for ID data (Line 239). Similarly, in the experiments, our accuracy metrics are also based on class-level accuracies (Line 262, 655). For OOD samples, since their class labels differ from those of ID samples, including $y^{+} = \hat{y}\_{p}$ in the calculation would render this condition meaningless.

We assume that the distinction between fine-grained classification for ID data and binary classification for OOD data is well-understood in the context of OOD Detection. Therefore, we did not elaborate further in the main text.

---

[1-7] Exceptions for $\text{Conf}(x) > \mu_{I}$ or $\text{Conf}(x) < \mu_{O}$

As shown directly in Equation 13, the loss term $L\_{psd}$ is calculated only when $\text{Conf}(x) > \mu_{I}$ or $\text{Conf}(x) < \mu_{O}$; otherwise, there is no backpropagation of loss, and the sample is directly used for inference (Line 251). The same applies to Equations 7 and 10. This approach avoids introducing noisy information due to ambiguous confidence into the model's learning process. For clarity, we will explicitly state that the loss is zero when the conditions $\text{Conf}(x) \leq \mu_I$ and $\text{Conf}(x) \geq \mu_O$ are met.

---

[2] The Impact of $-L\_{CE}$ on OOD Samples

In fact, for a standard OOD Detection task setup [3], we only need to perform binary classification between ID and OOD, and focus on class-level classification for ID samples without needing to perform fine-grained classification for OOD samples. As shown in Figure 6, existing OOD Detection methods (TPT[4], ROSITA[5], Open-IRT) introduce some distortion in the feature distribution of OOD data compared to direct inference (ZS-Eval). This is a well-known phenomenon in this field and does not negatively impact the evaluation metrics.

Additionally, it can be observed that our $-L\_{CE}$ method pushes samples away from intra-class noise. Consequently, the resulting impact on the distribution of OOD samples is a normal occurrence and does not interfere with the ID-OOD binary classification, nor does it affect the fine-grained classification results within ID classes.

---

[3] Values of $\mu_I$ and $\mu_O$, and Hyperparameters Robustness Experiment

In fact, $\mu_I$ and $\mu_O$ are not hyperparameters. Their definitions are provided in Line 216, where they are dynamically calculated mean values based on the confidence of the samples. By dynamically computing the confidence mean, we are able to adjust what is considered high confidence and low confidence according to different distributions, thus avoiding the limitations of fixed thresholds.

Furthermore, the main performance improvement in our method is driven by the two theoretical mechanisms proposed in Hypotheses 1 & 2. As seen in the ablation study (Line 317), the choice of threshold is merely a technical means in the implementation process and is not the primary contribution.

Additionally, other hyperparameter experiments (B, K, $\alpha$, $\lambda$) can be found in Sections C.2 (Line 770) and C.3 (Line 779) of the supplementary materials. Our method demonstrates robustness with respect to these hyperparameters. For example, the accuracy fluctuation due to $\alpha$ values ranging from 0.1 to 0.5 is less than 0.33%, and for $\lambda$ values between 0.1 and 0.5, the accuracy fluctuation is less than 0.57%.

---

[4] Optimization Goal and Theory for the Distances between $\mathcal{F}_I$, $\mathcal{F}_M$, and $\mathcal{F}_O$

By examining the optimization goals in Equation 8 & 9 (Line 233 & 235), it can be seen that they separately promote the separation between $\mathcal{F}_I$ and $\mathcal{F}_M$, as well as between $\mathcal{F}_O$ and $\mathcal{F}_M$. They also facilitate the separation of $\mathcal{F}_I$ and $\mathcal{F}_O$ through contrastive learning of positive and negative sample pairs. This fundamentally prevents the alignment between $\mathcal{F}_I - \mathcal{F}_M$ and $\mathcal{F}_O - \mathcal{F}_M$. Our method consistently improves FPR metrics, demonstrating better ID and OOD separation (Figure 6). As shown in Table 1 (CIFAR-10C to SVHN with CLIP), it boosts FPR95 by 12.88% over the second-best method.

Based on Line 208, we visualize the result of the intermediate domain in Figure 4. It is clear from the figure that, according to our construction method, the intermediate domain is theoretically situated precisely between ID and OOD, which aligns with the situation set in the theoretical analysis of the supplementary materials (Line 559). This intuitively demonstrates the theoretical foundation of Hypothesis 2.

---

[Additional question] How is the prompt learned?

The definitions of prompts are given in Line 152. We did not design updatable prompts because the core contribution of our paper lies in constructing an intermediate domain from the data perspective, and this design does not conflict with the use of updatable prompts. A detailed discussion of the update module can be found in Line 665 of the supplementary materials.

---

Reference

[1] Ganardi, Moses, Danny Hucke, and Markus Lohrey. "Derandomization for Sliding Window Algorithms with Strict Correctness∗." Theory of Computing Systems 65.3 (2021): 1-18.

[2] Huang, Xun, and Serge Belongie. "Arbitrary style transfer in real-time with adaptive instance normalization." ICCV. 2017.

[3] Yang, Jingkang, et al. "Generalized out-of-distribution detection: A survey." IJCV 132.12 (2024): 5635-5662.

[4] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models." NeurIPS 35 (2022): 14274-14289.

[5] Sreenivas, Manogna, and Soma Biswas. "Efficient Open Set Single Image Test Time Adaptation of Vision Language Models." arXiv preprint arXiv:2406.00481 (2024).

First, we sincerely thank you for recognizing the novelty of our method and the thoroughness of our experiments. We appreciate your detailed comments and understand the concerns regarding clarity and design motivation. In addition to our previous response, we would like to briefly reiterate the key clarifications and confirm that all identified issues can be resolved in the revised version:

• The rationale for using an intermediate domain is to create a separating margin that increases the distance between ID and OOD features, as supported by both theoretical analysis (Appx. Line 559) and visualizations (Figure 4).
• The "polarity" score combines absolute and relative semantic alignments (Equation 3), computed via similarity between image and prompt embeddings (Sim_pos, Sim_neg, Sim_diff).
• The sliding window is implemented as a standard FIFO queue. Variance is computed per channel to retain fine-grained feature structure.
• Thresholds are not hyperparameters, but dynamically derived from sample confidence (Line 216).
• Losses are applied only when specific conditions (Equation 7/10/13) are met; otherwise, no update is performed to avoid noisy gradients.
• Prompts are predefined and not learned in this work, though our framework supports integration with learnable prompt modules (Appx. Line 665).

We are committed to addressing these presentation issues—such as clearer motivation, terminology consistency, and explanation of loss conditions—in the revised manuscript. These improvements are straightforward and will significantly enhance clarity without altering the core method. Thank you again for your constructive feedback.

Dear Reviewer,

We sincerely thank you again for your valuable comments on our paper and your recognition of the novelty of our method and the thoroughness of our experiments. We truly appreciate the time and effort you have devoted to reviewing our work.

We fully understand that you may have a very busy schedule, but we would be extremely grateful if you could kindly take a moment to review our responses to your previous comments. We have carefully addressed your concerns, especially those related to clarity, and have prepared corresponding revisions that we believe significantly improve the manuscript.

Your feedback is very important to us. If you have any further suggestions or could confirm whether our clarifications resolve the issues you raised, it would be immensely helpful in guiding our revision.

Thank you again for your time and consideration. We greatly appreciate your support.

Dear Reviewer,

We sincerely thank you for your feedback throughout the review process. We are truly grateful for the time and effort you devoted to carefully reading our manuscript and thoughtfully engaging with our rebuttal.

Your comments have been instrumental in helping us identify and address areas for improvement. We are committed to fully incorporating the clarifications and enhancements discussed in the rebuttal—particularly in strengthening the introduction, refining terminology, and improving technical presentation—into the revised version.

We deeply appreciate your updated evaluation and your willingness to reconsider your recommendation. We wish you continued success in your research and all the best in your professional endeavors.

Thank you for the valuable time you spent reviewing our manuscript. We appreciate your recognition of the readability of our writing and the thoroughness of our experiments.

Thank you for your insightful comments. In response, we have conducted additional verification experiments. We believe your concerns can be addressed through these experiments and the additional clarifications provided. Key points have been highlighted in bold to help you easily locate the relevant parts in the paper.

---

[Weakness 1] Performance with Small B Values (B<64)

We appreciate your careful reading, including the supplementary materials. Your point is valid, and we have addressed it by including experiments with different values of B = 4, 8, 16, 32, 64, 128, 256, 512, 1024 based on CLIP, specifically for the CIFAR-10C to MNIST task. The results are presented in the table below:

---

The results confirm that extremely small values of B lead to performance degradation due to insufficient data for reliable GMM estimation. However, when B ≥ 16, the accuracy stabilizes, indicating that our method is robust under a wide range of practical settings.

---

[Weakness 2] Effectiveness of using negative prompts

Your question is critical, and we have carefully considered this aspect in both the method design and theoretical analysis. As detailed in Section A.2 of the supplementary materials, we address the theoretical validity of the negative prompt. Specifically, in Line 525, we acknowledge that VLMs face known challenges in implementing human-like negation understanding [3]. However, the effectiveness of the negative polarity prompt in our framework does not entirely depend on the perfect negative logical reasoning ability of the VLM. Instead, it leverages the differential representation capabilities of the text encoder: the vocabulary difference between the positive and negative prompts, $w_c^+$ and $w_c^-$, naturally leads to different embedding vectors $p_c$ and $p_c'$ in the shared embedding space. Our framework utilizes this induced representational difference as a discriminative signal for OOD detection. This idea is reflected in the design of the polarity gap item in Equation 3 (Line 158). It is also intuitively visualized in Figure 3.

---

[Weakness 3] Effectiveness of GMM in the Early Adaptation Phase

Thank you for raising this insightful point. From a technical standpoint, during the early adaptation stage of our method, we deliberately delay the fitting of the GMM and the computation of the corresponding loss until a sufficient number of samples (i.e., bank size $B$) has been accumulated. In other words, to avoid the uncertainty introduced by small sample sizes propagating to the model through loss gradients, we perform direct inference for those samples instead, without engaging in backpropagation. We acknowledge that predictions for these samples may be relatively weak.

Therefore, there are two main motivations behind incorporating the bootstrapped resampling method:

1. OOD detection inherently depends on modeling data distributions, which requires a sufficient number of samples to distinguish ID from OOD instances effectively.

2. GMM fitting relies on the statistical properties of the data; with limited samples, parameter estimates may deviate from the true distribution, degrading performance.

The following table presents the experimental results to support this design:

Based on our experiments under the conditions of $B = 4$ and $B = 8$, we effectively simulate scenarios where the memory bank contains only a small number of samples during the early adaptation stage. The results demonstrate that the bootstrapped resampling strategy successfully mitigates performance degradation caused by insufficient data during GMM fitting.

Moreover, when the bank size $B$ is sufficiently large, the impact of bootstrapped resampling becomes marginal. This is expected, as no loss backpropagation occurs during the early inference stage, and thus, performance is less sensitive to sample size once adequate data is accumulated.

Overall, our experiments suggest that as long as a reasonable $B$ value is chosen—avoiding extremely small settings—the proposed method remains robust and performs consistently across different configurations.

---

[Question 1] Performance under Very Small B

As discussed in [Weakness 1], model performance degrades when $B$ is very small, primarily because GMM fitting requires a minimum number of samples to produce reliable estimates. However, we observe that when $B > 16$, the model performance stabilizes and reaches saturation.

---

[Question 2] Theoretical Validity of Negative Prompts

As discussed in [Weakness 2], we elaborate on the theoretical rationale of using negative prompts in Section A.2 of the supplementary material (Line 525). Our method does not rely on the VLM’s precise understanding of semantic negation. Instead, it leverages the representational difference between positive and negative prompts.

---

[Question 3] Choice of Prompt Format

Yes, we explored the prompt format based on the discussion in CLIPN [1]. In practice, vision-language models (VLMs) are known to be sensitive to the phrasing of input prompts, and carefully crafting prompt formats can indeed improve performance. However, we intentionally adopted a simple and consistent format — “a photo of no <class>” — for two key reasons.

1. We aimed to ensure fairness across all baseline comparisons. All baselines use the prompt “a photo of a <class>” for inference. To maintain consistency and minimize bias, we chose a minimal modification of that format for negative prompts.

2. All methods, including ours and baselines, utilize the same CLIP-based text encoder. Since a well-optimized prompt would benefit all methods equally, we argue that the main performance differences stem from algorithmic and theoretical innovations, rather than from prompt engineering.

Our method focuses on data-level improvements by constructing an intermediate domain. This perspective is theoretically and practically complementary to works based on learnable or tunable prompts [4–6]. As such, our contributions emphasize data-driven design and theoretical insights. We also briefly discuss the learnable module option in the supplementary material (Line 665).

We now provide results on CIFAR-100C to MNIST using negative prompts introduced in CLIPN [1], as follows:

---

From the experimental results, we observe that different prompt formats do slightly impact model performance. This is due to the sensitivity of Equation 3 to the input text prompts, where different textual inputs lead to different computed scores. Notably, some prompts even yield better performance than our original formulation.

Although different negative prompts lead to marginal performance differences (±0.1%), we intentionally avoided prompt optimization to ensure fair comparisons. This further highlights that our gains stem from algorithmic contributions rather than prompt engineering.

---

[Question 4] Effectiveness of GMM in the Early Stage

As discussed in [Weakness 3], we employ bootstrapped resampling to address the challenge of limited samples during the early stages of training. This design is motivated by two key considerations: (1) OOD detection fundamentally relies on characterizing the distributional differences between ID and OOD data, which becomes more effective as the sample size increases; and (2) GMM fitting depends on sufficient data to reliably estimate the underlying statistical properties of the distribution.

As shown in the experiments under [Weakness 3], once a sufficient number of samples is accumulated, the effect of this strategy becomes negligible in terms of performance improvement. From a theoretical perspective, alternative sampling strategies could also serve to increase the diversity of the feature bank and alleviate the sparse-sample issue in early training stages.

---

Reference

[1] Wang, Hualiang, et al. "Clipn for zero-shot ood detection: Teaching clip to say no." ICCV. 2023.

[2] Yang, Jingkang, et al. "Generalized out-of-distribution detection: A survey." IJCV 132.12 (2024): 5635-5662.

[3] Li, Tianqi, et al. "Learning transferable negative prompts for out-of-distribution detection." CVPR. 2024.

[4] Zhou, Kaiyang, et al. "Learning to prompt for vision-language models." IJCV 130.9 (2022): 2337-2348.

[5] Zhou, Kaiyang, et al. "Conditional prompt learning for vision-language models." CVPR . 2022.

[6] Kan, Baoshuo, et al. "Knowledge-aware prompt tuning for generalizable vision-language models." ICCV. 2023.

Thank you for the detailed and thoughtful rebuttal. I appreciate the additional experiments and clarifications provided, which help to address some of the concerns I raised.

That said, I remain somewhat concerned about the method’s reliance on a sufficiently large memory bank for GMM estimation. While the reported results suggest that performance stabilizes for B≥16, this requirement appears to contradict the claim that the method is applicable in a truly single-image test-time scenario. In practice, needing several images beforehand seems to weaken the core promise of single-image adaptability. I would encourage the authors to clarify this nuance more explicitly in the paper to avoid potential overstatement.

Thank you for your valuable time and effort in reviewing our manuscript. We appreciate your positive feedback on our idea, as well as your recognition of the clarity and thoroughness of our writing and experimental design.

We truly appreciate your thoughtful concerns. We believe that your concerns can be addressed through clarifications and further explanations in paper details. Below, we provide our clarification in detail, organized to address each of your concerns. We have highlighted the key points in bold to enhance your reading experience and make it easier to navigate through the important points.

---

[Weakness 1] Additional Experimental Comparison of AdaND

Thank you for your suggestion. We have noted the work by [1]. It introduces a new setting, Zero-shot Noisy Test-time Adaptation (ZS-NTTA) , which is different from ours. It adapt to noisy samples during test-time, focusing on noisy sample detection and classification accuracy.

Our setting does not involve noisy conditions, so we did not run this experiment in the main body. However, we have incorporated this framework into our dataset and conducted additional experiments during the rebuttal phase. Despite improvements in the latest baseline, our method remains superior, such as 7.18 average FPR improvement, because our method excels in traditional single image OOD Detection tasks.

---

---

[Weakness 2] Main Improvements of the Experiments

We observe that in some cases (e.g., Table 1 and Table 3), accuracy improvements are limited. This is due to current baseline methods (e.g., ROSITA [2], OWTTT [3]) already reaching near-saturated performance on specific datasets (CIFAR-10C, VisDA). However, our experiments consistently show superior performance in most scenarios, as evidenced in the main text and supplementary tables (Table 7, Table 10, and Table 11).

Our proposed intermediate domain approach enhances the separation between ID and OOD samples, leading to notable improvements in FPR95. For example, in Table 1 (CIFAR-10C to SVHN with CLIP), our method improves FPR95 by 12.88% compared to the second-best method. This is attributed to Hypothesis 2, where the intermediate domain concept facilitates better ID-OOD distinction.

In Table 2 and Table 3, apart from near-saturated results (with FPR95 < 10), our method also shows significant FPR95 improvements. For instance, in the ImageNet-C to MNIST experiment (Table 2), we observe both an accuracy increase and a 7.49 improvement in FPR95, further validating the effectiveness of Hypothesis 2.

---

[Weakness 3] Structural Complexity and Completeness of the Method

Thank you for considering the complexity of our approach. While Open-IRT involves multiple components, each is essential and optimized to realize our core innovation: The score bank and GMM implement Hypothesis 1, while the KNN search and loss terms realize Hypothesis 2.

Our design achieves significant performance gains with minimal overhead, including a 7.19% average reduction in FPR compared to the second-best method ([Weakness 1]) and an inference latency of just 40.15ms ([Question 3]), demonstrating real-time viability. Ablation studies (Line 311, 317) further confirm the module's effectiveness and necessity.

This is an efficient trade-off: Modularity enables the novel intermediate domain mechanism, while latency remains suitable for single-image deployment.

---

[Question 1] Additional Experimental Comparison and Analysis Regarding AdaND

Please refer to the table provided in [Weak 1] section for the experimental results.

---

[Question 2] Hyperparameter Robustness Experiment

Hyperparameters B and K: We validated the robustness of B and K in Figure 10 and Table 8 (Appendix), with an analysis in Line 779. We tested values of B = 64, 128, 256, 512 and K = 0, 1, 3, 5, and found that accuracy fluctuations were less than 0.70%, indicating that reasonable adjustments to these values do not significantly affect results.

Hyperparameters $\alpha$ and $\lambda$: The robustness analysis for $\alpha$ and $\lambda$ is presented in Table 9. As noted in Line 796, changes in $\alpha$ (0.1 to 0.5) resulted in accuracy fluctuations of less than 0.33%, and for $\lambda$ (0.1 to 0.5), fluctuations were under 0.57%. It is worth noting that some results in Table 9 even slightly outperform those in the main experiments (Table 1, Table 7), as we did not perform deliberate hyperparameter tuning. We firmly believe that a robust method should not rely on selecting the optimal hyperparameter configuration but instead be adaptable to a wide range of real-world deployment scenarios.

Thresholds $\mu_I$ and $\mu_O$: $\mu_I$ and $\mu_O$ are not hyperparameters. Defined in Line 216, they are dynamic thresholds computed from confidence values, adapting to sample distributions, thus avoiding robustness issues related to fixed hyperparameters.

---

[Question 3] Computational Efficiency of the Method

From an implementation efficiency perspective, the experimental results, including both overall dataset samples and single-sample analysis. Our method demonstrates excellent computational efficiency. With a single 3090 GPU, the average execution time is 40.15ms, making it suitable for lightweight deployment. These results are expected to improve further with advancements in computational power and hardware performance.

---

---

Reference

[1] Noisy Test-Time Adaptation in Vision-Language Models, ICLR 2025

[2] Sreenivas, Manogna, and Soma Biswas. "Efficient Open Set Single Image Test Time Adaptation of Vision Language Models." arXiv preprint arXiv:2406.00481 (2024).

[3] Li, Yushu, et al. "On the robustness of open-world test-time training: Self-training with dynamic prototype expansion." Proceedings of the IEEE/CVF ICCV. 2023.

[4] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models." NeurIPS 35 (2022): 14274-14289.

We sincerely thank the reviewer for the thoughtful comments and for giving us the opportunity to address these concerns. Below we respond point-by-point.

[1] On Generalization to Harder OOD Settings:

We agree with the reviewer that using datasets such as MNIST/SVHN makes the OOD detection problem relatively easier due to their large semantic shift from ImageNet-based ID sets. To address this, we conducted additional experiments on a more challenging and realistic setup: ImageNet-R (ID) vs SUN (OOD), following the AdaND [1] protocol. We used 10,000 samples for both ID and OOD and evaluated all methods under the same condition. The results are shown below (with five different random seeds):

As shown, although this setting leads to generally higher FPRs due to increased difficulty, our method still achieves the best overall performance, which demonstrates that the proposed Middle-Domain approach generalizes well to more complex ID-OOD distributions.

We further emphasize that our method performs better in complex scenarios, such as CIFAR-100 → Tiny-ImageNet (Rebuttal Table above), because the semantic distribution shift allows the Middle-Domain to function more effectively. Even when the semantic space is shared and only the style differs (i.e., in Domain Adaptation settings), Middle-Domain representations can still be constructed effectively, as shown in prior work [2]. Thus, as long as there exists any semantic or distributional gap between ID and OOD, our method is applicable, and its advantage becomes even more prominent with increasing difficulty (see Line 295).

---

[2] On Performance Margin:

We understand the reviewer’s concern that the performance gain in Table 1 may appear modest. This is due to the relative simplicity of datasets such as CIFAR-10C, where many methods already achieve strong results. In contrast, on harder datasets (Table 2 and Table 3, e.g., VisDA or ImageNet-C), the improvements become much more substantial:

For example, in the ImageNet-C → SVHN experiment, Open-IRT achieves a +5.12% increase in Acc_HM, -16.86% in FPR95, and +6.55% in AUROC.

This demonstrates that our approach is particularly beneficial when the feature space is complex or less separable, which is where many real-world TTA scenarios lie.

Moreover, even on easier datasets, we still show consistent performance:

In CIFAR-10C (Table 1), across 8 groups, Open-IRT achieves an average AUC improvement of +0.20%, FPR reduction of -1.92%, and HM accuracy gain of +0.44% over the second-best method — without extensive hyperparameter tuning.

Based on our existing experimental results, we further report the variance across five random seeds in Table 1 to demonstrate the stability of our method.

The low variance and consistent improvements across settings further reinforce the robustness of our method.

---

[3] On Computational Complexity:

As reported in the rebuttal, our method is efficient in practice: it takes only 40ms per image on a single 3090 GPU, and scales better with higher-end hardware. Open-IRT is designed as a lightweight feature-level enhancement, and we believe its benefits outweigh its minor overhead, especially in real-world TTA applications.

---

[4] On Complementarity to Existing Methods:

We emphasize that the Middle-Domain theory is orthogonal and complementary to existing OOD-TTA frameworks. It can be integrated into other methods to potentially boost their performance further. We sincerely thank the reviewer again for the insightful suggestions. We will include all additional experiments, variance results, and discussions in the revised paper and supplementary.

---

References

[1] Noisy Test-Time Adaptation in Vision-Language Models, ICLR 2025

[2] Huang et al. "Arbitrary style transfer in real-time with adaptive instance normalization." ICCV 2017

---