Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation

We thank the reviewer for their thoughtful and constructive feedback. We are encouraged that they found our method novel, well-motivated, and clearly presented, and that they appreciated the strong empirical results across diverse datasets, corruption types, and model families. We also thank the reviewer for highlighting the value of our extensive ablations and recognizing that our work addresses an understudied area. Below, we respond to each of the reviewer’s comments in detail.

A1. Improving Figure 1

The variance is computed over all samples in the V20-C benchmark, which includes all corruption types. We appreciate the reviewer’s feedback and will incorporate all these clarifications to improve the clarity and completeness of the figure.

A2. Clarifications on Multi-Prompt Robustness Claims

To illustrate how multiple prompts help prevent degenerate collapse, we conducted an experiment where we ran adaptation using either all seven templates or only a single one and averaged them, using the same 10 images for efficiency. The results are summarized in the table below.

On V20-C, single-template runs often show unstable behavior or steep drops as we can see thanks to the average, in contrast, the multi-template setup maintains high performance, with only a slight decline. These results support our claim that using multiple prompts acts as a regularizer and mitigates collapse during adaptation.

A3. Improving Figure 4

We will update this figure in the final version by adding the performance with no adaptation (the quantitative results are already in the main table).

A4. Missing Citation for GTAV

We will add the appropriate citation for the GTAV dataset in the final version.

A5. Typo in Line 155

Thank you for pointing this out. The reference to Fig. 1b in that sentence is a typo. The intended evidence for the complementary nature of intermediate layers is in Table 1, which reports the ablation on fusion strategies, and in Figure 3, which visualizes the learned layer-wise confidence weights. We will correct the sentence and update the citation accordingly in the final version.

A6. Clarification on Adaptation Loop and the Effect of Batch Size

The reviewer’s understanding of the adaptation/inference loop is correct: we adapt the model using a batch of two test samples for 10 steps, predict the masks for both samples, and then restore the original model parameters.

We chose a batch size of 2 mainly for computational efficiency; it provides roughly 2× faster adaptation compared to using a batch size of 1, which was important given the need to evaluate multiple adaptation methods across several datasets.

While larger batch sizes can be beneficial in classification due to entropy bias correction, this is less relevant in segmentation, where supervision occurs at the pixel/token level. In fact, increasing the batch size may require additional tuning of hyperparameters and can hurt performance due to over-smoothing or weaker adaptation signals.

To address your intuition that a batch size of 1 may work better, we confirm this in Appendix Table 3. As shown below, our method performs well with a batch size of 1, supporting the robustness of the approach across batch configurations:

We will update the paper to make this clearer and explicitly reference the ablation results to improve transparency.

A7. Episodic vs Online Setting

We thank the reviewer for the insightful suggestion regarding an online adaptation setting. We experimented with this setting using the exact same hyperparameters as our main method (i.e., learning rate and adaptation steps), but found that both our method and TENT eventually collapsed: while performance initially improved, it degraded after a few batches and dropped significantly thereafter.

Motivated by the reviewer’s reference to [1], we further explored the online setting by reducing the number of adaptation steps and lowering the learning rate. The results are as follows:

We observe that while the online variant improves over baselines, it still underperforms compared to the reset-based (episodic) setting. We believe this is primarily due to error accumulation and model drift, as documented in prior work showing that some recent classification TTA methods exhibit performance degradation in an online setting—even when the distribution is static [2]. This issue is particularly pronounced in semantic segmentation, where dense, spatial predictions make the model more sensitive to incorrect updates, and entropy minimization can amplify early misclassifications. Over time, this leads the model to drift away from its well-initialized source parameters, reducing its ability to generalize across the target domain.

Episodic (reset) adaptation avoids the runaway effects of carrying over errors, which is why it tends to be safer [2]. In contrast, fully online adaptation—though attractive for its potential to continuously refine the model—must confront these challenges.

We believe that online adaptation in segmentation can be improved through several future directions: (1) introducing occasional resets or using exponential moving average (EMA) of model weights to limit drift and reduce accumulated errors; (2) filtering high-entropy samples or gradients, which is particularly important in segmentation due to its spatial granularity and sensitivity to local noise; and (3) incorporating an auxiliary self-supervised objective, such as rotation prediction [1], to provide a more reliable adaptation signal—though this may come with increased computational cost. Notably, the fact that our method works effectively with very few test samples and does not rely on accumulating state makes it efficient for real-world applications.

We will include a discussion of these findings and potential improvements in the final version.

[1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts, ICML 2020.

[2] On Pitfalls of Test-time Adaptation, ICLR 2023

A8. Insights on the Underperformance of WATT and CLIPArTT

Thank you for the observation. Indeed, as shown in the table below, our own variant of MLMP using a pseudo-labeling loss performs worse than the non-adapted baseline, consistent with the behavior observed for WATT and CLIPArTT. We also show in Table 3 of the paper that parameter averaging, as done in WATT, does not improve performance in our setup either.

We believe this highlights a key difference between segmentation and classification tasks. In segmentation, pseudo-labeling relies on confidently predicting a hard label for every pixel, which can amplify noise and lead to unstable adaptation, especially under distribution shift. In contrast, entropy minimization provides a smoother learning signal by encouraging confident predictions without committing to hard labels, making it more effective in this context.

We will clarify these insights in the paper to better explain why some adaptation strategies from classification may not transfer well to segmentation.

A9. Experience with Longer Prompts

Thank you for the suggestion. We agree that exploring longer prompts, such as those enabled by models like Long-CLIP or TULIP, can be an interesting direction. To investigate this, we generated extended templates based on our 7 class-agnostic prompts using ChatGPT, as well as extended class names, where each class was described in natural language. For example, "a photo of the large {class}." becomes "A photograph of a very large {}, where the size of the object dominates the frame or is shown in contrast to smaller elements to emphasize its scale." and “aeroplane” becomes “aeroplane: a powered flying vehicle with fixed wings and engines, designed to transport people or cargo through the air over long distances”

As shown in the table below, MLMP consistently improves performance over the Long-CLIP baseline across all variants. However, using extended templates or extended class descriptions did not lead to further improvements over the base prompts. This suggests that while MLMP is robust and effective even with longer inputs, simply extending prompt length does not guarantee performance gains, possibly due to the complexity or ambiguity introduced by verbose descriptions.

We will add this analysis and discussion to the paper to provide a clearer picture of MLMP’s behavior under extended prompts.

We thank the reviewer for their thoughtful and constructive feedback. We are pleased that they acknowledged the practical significance of adapting vision-language models during inference, appreciated our theoretical insights on prompt-based variance reduction, and recognized the strength of our experimental results. In the following, we address the reviewer’s comments point by point.

A1. Novelty of Feature Fusion and Image-Level Loss

Thank you for the comment. While feature fusion across layers is indeed common in standard segmentation models (e.g., U-Net, HRNet, FPN), its use in TTA, especially in the context of OVSS, has not been explored to our knowledge. In our method, we go beyond basic fusion by introducing adaptive layer weighting, which allows the model to selectively emphasize features based on their relevance during adaptation. This is a key contribution that distinguishes our approach.

As for the image-level entropy loss, we note that this is not a common technique in segmentation, and especially rare in OVSS. Unlike pixel-level entropy minimization used in methods like TENT, we apply entropy regularization at the image-level representation (CLS token), encouraging more confident global predictions in a zero-shot setting. To our knowledge, this is the first use of such a loss in OVSS TTA.

We believe the overall design brings a novel perspective to this recent problem, a view that is also echoed in the feedback from R-Fuid.

A2. Evaluation Under One-Iterate TTA Protocol

The number of adaptation iterations in TTA varies across the literature. While some works evaluate under a one-iteration (step) protocol, many others use multiple iterations. For example, the original TTT paper (Sun et al., 2020) as well as more recent vision-language TTA methods like CLIPArTT and WATT commonly use 10 adaptation iterations. Our choice to follow this convention is aligned with these prior works. For fairness, we use the same hyperparameters across all experiments and baseline methods.

Nonetheless, we agree it is valuable to assess performance under a single-iteration protocol. As shown in the table below, MLMP still improves over the baseline and outperforms TENT even with just one iteration, which demonstrates the effectiveness of our method even under a stricter setting.

We will clarify this setup in the main paper and include the single-iteration results to make the evaluation more comprehensive.

A3. Open-Vocabulary Capability

We thank the reviewer for this insightful question. The set of concepts C_k ⊂ C mentioned in Line 120 is provided at inference time and can be arbitrary—there is no restriction to a fixed or predefined vocabulary. This flexibility is what enables open-vocabulary segmentation: the model is not trained on specific categories and can adapt to any set of text labels at test time.

In our experiments, we follow the evaluation protocol used in prior OVSS works (e.g., NACLIP [1] ), which adopt a predefined set C purely for benchmarking purposes. However, this set can be changed at any time, and the model is capable of segmenting new, unseen categories that were not present during training.

We will clarify this aspect in the paper to better emphasize that our method fully supports open-vocabulary segmentation by design.

[1] Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation, WACV 2025

A4. Selection of Prompts

Thank you for the observation. We agree that prompt selection is an important aspect in vision-language models. As noted in Appendix Section C, we did not tune prompt templates for performance; instead, we used the general-purpose templates proposed by the original CLIP authors. These prompts are not tailored to specific image content, which helps ensure fairness and broad applicability.

While our multi-prompt strategy helps mitigate sensitivity to any single template, we agree that exploring more effective or adaptive prompt selection strategies could further improve performance. However, such approaches often come with additional computational and memory costs, and we consider them complementary to our method. We view this as a promising direction for future work.

We thank the reviewer for their time and feedback. We are glad that they found the writing clear and appreciated the observations motivating our MLMP design. In the following, we address each of the reviewer’s comments in detail.

A1. General Motivation behind TTA for OVSS

We appreciate the reviewer’s concern and would like to clarify the motivation behind the OVSS-TTA task. The main goal is to evaluate whether test-time adaptation can offer consistent improvements across a broad spectrum of potential shifts. As also noted by Reviewer t4bA, we believe the task is well motivated, and we agree with Reviewer tiCb on its high practical relevance, particularly given the unique challenges of zero-shot OVSS, where models are especially sensitive to distribution shifts due to the lack of supervised adaptation. This sensitivity is evident in Table 4 of the main paper, where the non-adapted model's performance drops from 75.91 mIoU on V20 (original) to 69.01 mIoU on V20-C (common corruptions), underscoring that even robust foundation models remain vulnerable to distribution shifts in the absence of test-time adaptation.

While some of our benchmark shifts are synthetic (e.g., blur, noise), this follows standard practice for robustness evaluation, such as in ImageNet-C, and provides a controlled, reproducible setting to test the effectiveness of adaptation strategies. This is especially valuable in zero-shot scenarios, where the test distribution is inherently unknown.

To demonstrate that our approach goes beyond synthetic corruptions, we include results on more realistic domain shifts. In our response to t4bA-A3, we present evaluations on ACDC, which features real-world weather-related conditions, and GTA5, a distinct synthetic domain with different scene statistics. We also highlight intra-dataset variations within Cityscapes (e.g., lighting, camera viewpoints, and location diversity). Furthermore, our method improves performance not only under distribution shift but also on the original (unshifted) datasets (Table 4), which underscores the general utility of test-time adaptation in OVSS.

In real-world settings, it is unlikely that a model can fully account for all possible unseen target distributions. This is due to several factors: (1) the space of potential corruptions and environmental shifts is vast and essentially unbounded; (2) training on such a wide range of conditions would require extensive labeled and diverse data, which is often costly or impractical to obtain; and (3) the distribution encountered at deployment is typically unknown and may evolve over time, especially in open-world scenarios like OVSS.

These constraints make TTA not only relevant but necessary, as it allows models to adjust to novel domains without requiring retraining or labeled data.

A2. Comparison with Few-Shot

Thank you for raising this point. While FSS methods do allow for adaptation using a small number of labeled examples, we believe they are fundamentally different from the TTA setting we explore in OVSS.

First, most FSS methods assume access to a clean support set with annotated masks at inference time. In contrast, OVSS operates in a fully zero-shot setting, without any class-specific labels or examples at test time. The set of target categories is also unknown and potentially large, making it impossible to construct or assume a meaningful support-query structure.

Moreover, while FSS methods do not require training-time fine-tuning, they still rely on labeled masks at inference, which can be difficult to obtain for open-vocabulary segmentation tasks, especially when dealing with arbitrary or long-tail classes.

For these reasons, we believe that FSS and TTA operate under different assumptions and are not directly comparable. However, we appreciate the suggestion and will include a discussion of this distinction in the revised version of the paper.

A3. Clarification on the Generality of MLMP Across OVSS Frameworks

We would like to clarify that we do evaluate MLMP across multiple similarity-based OVSS frameworks to assess its generalization ability, as mentioned in lines 282–283 of the paper. Specifically, we test MLMP with (1) different OVSS methods (Table 6 in Appendix), and (2) different vision-language models, namely SigLIP-2 (Table 7 in Appendix) or LongCLIP (please see R-Fuid-A9) and (3) different backbones (Tables 4 and 5 in Appendix). These experiments demonstrate that MLMP is robust across architectures, feature extractors, and vision-language model variants within this class of methods.

As stated in line 117, our method is specifically tailored for the OVSS framework based on contrastive VLMs, which covers most existing approaches in open-vocabulary semantic segmentation. We will ensure this point is stated more clearly in the revised manuscript.

We thank the reviewer for their thoughtful evaluation and valuable feedback. We are pleased that they found the work to be well motivated and appreciated both the introduction of our OVSS TTA benchmark and the thorough experimental analysis. In what follows, we address the reviewer’s comments point by point.

A1. Clarifications on Novelty

We appreciate the reviewer’s comment and welcome the opportunity to clarify our technical contributions. We agree that TENT is a well-established approach for test-time adaptation in classification, and we do not consider entropy minimization itself to be a novel contribution, nor is it the core of our method.

Our main contribution lies in test-time adaptation of open-vocabulary vision-language models, a fundamentally different and largely underexplored setting compared to standard classification. Due to the dense, spatial nature of segmentation and the prompt-sensitive behavior of VLMs, directly applying methods like TENT, WATT, or CLIPArTT is non-trivial and often ineffective. We address these challenges by proposing a set of novel and carefully designed components: (1) A multi-layer adaptation strategy that integrates features from intermediate vision-encoder layers to capture complementary, shift-resilient cues (2) An adaptive uncertainty-aware weighting strategy that dynamically upweights more confident layers and downweights noisier ones, allowing the model to emphasize the most reliable feature representations during test time (3) An intuitive multi-prompt aggregation mechanism at the loss level to address increased prompt sensitivity in OVSS. Each of these components is validated through detailed ablations (see Table 1-3), and together they yield consistent, substantial gains across a wide range of scenarios. Our method is also highly efficient, adding only marginal overhead compared to TENT (please see Appendix E).

In addition to our design choices, we evaluate MLMP and adapted baseline methods exhaustively on a comprehensive OVSS TTA benchmark suite that now includes not only clean and synthetically corrupted data, but also realistic domain shifts (please see the t4bA-A3). This suite ensures fair and consistent evaluation, and we hope it will serve as a foundation for future TTA research in this domain.

In summary, we believe that the novel design of MLMP, its plug-and-play and efficient formulation, and its strong empirical gains across diverse and realistic settings represent a meaningful step forward in the emerging and largely underexplored field of test-time adaptation for open-vocabulary segmentation. Moreover, MLMP is modular and can be seamlessly applied to a wide range of contrastive VLM-based OVSS methods, enabling lightweight and effective adaptation without requiring additional supervision. It proves effective not only under synthetic corruptions and real-world domain shifts, but also enhances performance on clean, natural images.

A2. Differences with Multiprompt used in OVSeg

We thank the reviewer for this helpful observation and take the opportunity to clarify key differences between our multi-prompt design and prior approaches such as OVSeg [1]. While OVSeg employs prompt ensembling, it does so during training by simply averaging text embeddings at the text encoder output, collapsing multiple linguistic features into a single representation. This results in a fixed perspective and lacks the diversity necessary for robust adaptation in a TTA setting.

In contrast, our method operates entirely at test time, and for the first time, introduces a loss-level integration of multiple prompts during test time adaptation. We demonstrate that prompt variability induces significantly greater uncertainty in segmentation (Fig. 1a), especially at the patch and intermediate-layer levels. To exploit this signal effectively, we directly incorporate prompt diversity into our optimization objective, treating each template as an independent “critic” and averaging their losses. Using multiple prompts in this way acts as a form of cross-modal regularization, encouraging more stable and generalized learning signals. While different from traditional image augmentation, it can be viewed as a lightweight and safe augmentation technique in the text modality. This design is further supported by Proposition 1, which shows that the resulting gradient is both unbiased and lower-variance, thereby promoting more stable and generalizable adaptation.

We also compare our approach in Table 3 to alternatives, including simple prompt averaging (as in OVSeg [1]) and parameter ensembling (as in WATT [2]), and show that our strategy outperforms both in adaptation effectiveness. While WATT incurs notable computational overhead, our method remains lightweight—even when using multiple templates—making it well-suited for dense prediction tasks such as OVSS.

[1] Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP, CVPR 23

[2] WATT: Weight Average Test-Time Adaptation of CLIP, NeurIPS 24

A3. Including Real Domain Shift

We appreciate the reviewer’s suggestion to include evaluation on realistic, non-synthetic domain shifts. In response, we have conducted additional experiments on the ACDC [3] dataset, which contains real-world variations such as fog, night, rain, and snow. As in our main experiments, we perform test-time adaptation separately on both reference (clean) and shifted images within each domain. It is worth mentioning that in ACDC, half of the adverse-condition images have corresponding clean reference views. While these reference images are captured at approximately the same location as the shifted ones, the scene content may differ (e.g., different vehicles or objects present) due to real-world variability.

As shown in the table below, our method yields consistent improvements over both baselines across all domains. Compared to the non-adapted model, MLMP achieves substantial gains, around 6% mIoU improvement on reference images and over 7% mIoU in the presence of environmental domain shift. These results highlight the robustness of our method in handling real-world domain shifts, demonstrating effectiveness beyond synthetic corruption scenarios. We also note that the Cityscapes dataset, on which we already reported results in the main paper, exhibits natural distributional shifts due to variations in lighting, camera viewpoints, and urban layouts across different cities.

To further address the reviewer’s concern, we also evaluated MLMP on a rendered domain shift, using the GTAV dataset [4]. GTAV consists of photorealistic urban scenes rendered from a game engine and exhibits a distribution shift distinct from real-world imagery seen during CLIP pretraining. As shown in the table below, MLMP achieves consistent improvements over both TENT and the non-adapted baseline. This complements our ACDC results and demonstrates MLMP’s robustness across both real-world and rendered test-time domains.

[3] ACDC: The Adverse Conditions Dataset, ICCV 21

[4] Playing for Data: Ground Truth from Computer Games, ECCV 16

A4. Selection of Prompts

We thank the reviewer for this important question. As discussed in Appendix C, the seven prompt templates used in our method were selected directly from the original CLIP repository, without any tuning or dataset-specific filtering. These prompts are general-purpose and were originally proposed to offer diverse textual views across a wide range of classes—not tailored to specific datasets or domains. We intentionally adopted this fixed, predefined set to avoid any risk of target-data information leakage

While we agree that prompt selection can influence performance, selecting or filtering prompts based on the target domain shift would violate the fundamental assumptions of test-time adaptation, where no access to shift-specific information is allowed. In fact, using domain-aligned prompts (e.g., referencing weather conditions) could introduce bias and unfairly benefit adaptation. Our goal was to evaluate MLMP under a fair and domain-agnostic setting.

A5. Insights into Layer-Wise Confidence Trends in COCO-Stuff

We thank the reviewer for this insightful observation. Indeed, as shown in Fig. 3, the final vision layer receives lower confidence in the COCOStuff dataset compared to Pascal and Cityscapes. We believe this is due to the inherent properties of COCOStuff, which includes 171 classes, many of which are amorphous “stuff” categories (e.g., sky, grass, wall) with weak spatial boundaries and higher semantic ambiguity. In such cases, earlier and intermediate layers—preserving finer texture and edge information—often yield more reliable predictions than the deeper, class-aligned representations of the final layer. A similar trend is observed in the COCOObject dataset as well (please see Fig. 1 in the Appendix H).

This observation further supports the necessity of our proposed uncertainty-aware multi-level fusion strategy, which dynamically reweights intermediate features based on their entropy. Unlike fixed fusion schemes, our approach adapts to the specific input and dataset, allowing the model to emphasize more reliable layers under varying conditions.

Thanks for detailed rebuttal and responses. My concerns are addressed and I have raised my score to 4. While the overall technical novelty seems limited, the task of test time OVSS in general looks new with detailed benchmark and analysis.