OpenDAS: Open-Vocabulary Domain Adaptation for Segmentation

We thank the reviewer for acknowledging the simplicity and effectiveness of our method. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

Q1. In my opinion, task name is missleading. The adaptation procedure actually optimizes segment classification (images containing individual segments), while it assumes there is a mask proposal generator available. More precisely, it assumes perfect mask generator both during training and evaluation. The term "open-vocabulary segmentation" implies both localization (mask proposal generation) and classification. Because of that, I was confused while reading the manuscript all the way until section 5 and the description of evaluation metrics. I think that this difference must be clearly stated and thoroughly described earlier in the manuscript.

The task name is chosen to fit the definition of the task that we introduce in Section 3, independently of the studied method and integrated models which can be considered a first proposal of how to approach the task. Prior works already decoupled the task of OVS into segment proposal and segment classification [1, 2]. Our preliminary analysis shows that the limitation of such decoupled OVS models for the setting of domain adaptation is the segment classification and not mask proposal. Therefore, we focus our proposed method on adapting the segment classification, while showing that integrating the adapted VLM with existing mask-proposal models can indeed improve the more general task of Domain Adaptation for Open-Vocabulary Segmentation. We update the manuscript to better emphasize this (L99-L101).

[1] Liang et al., Open-vocabulary semantic segmentation with mask-adapted clip, CVPR'23.

[2] Takmaz et al., OpenMask3D: Open-Vocabulary 3D Instance Segmentation, NeurIPS'23.

Q2. The manuscript lacks a proper comparison with the related work from open-vocabulary segmentation. Table 3 shows that the adapted model can be used to improve performance of the open-vocabulary segmentation model OVSeg. However, the table shows results only for the ADE20k validation set which contains only in-vocabulary (base) classes, while it is usual to evaluate the performance on datasets with different vocabularies.

Thanks for your observation. Table 3 shows that our method can be used with predicted masks instead of just ground truth ones and can be integrated to existing OVS models for better segment classification. To further strengthen this analysis, we have now added results with FC-CLIP, a more recent model, showing that OpenDAS also effectively integrates with and improves this framework. You can also find the updated table with results on ADE20K-150 below.

Additionally, Table 3 already shows evaluation with predicted masks on novel queries in ScanNet++ Offices, where it improves upon the OpenMask3D framework (see below). However, we agree that extending this evaluation to 2D segmentation for novel queries with predicted masks would further strengthen the study. We are currently preparing these additional results and will update them shortly here in the discussion.

Q3. Furthermore, OVSeg significantly lacks in performance for the current SOTA models (e.g. CatSeg (Cho et al.,2023) or FC-CLIP (Yu et al., 2024)).

Based on the suggestion, we look further into integrating our model into newer OVS methods, FC-CLIP [3] and MAFT+ [4]. We observe that OpenDAS shows consistent improvement over existing methods on ADE20K-150, even when it predicts from masked images. We also added these additonal results to the manuscript (see Appendix H).

[3] Yu et al., Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP, NeurIPS'23.

[4] Jiao et al., Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation, ECCV'24.

In response to Q2, we share additional results for open-vocabulary 2D semantic segmentation on ScanNet++ Offices using OVSeg predicted masks. These results allow us to demonstrate the strength of our method on novel classes, even when the model predicts from the predicted masks. Further results with FC-CLIP and MAFT+ mask proposals will be included in the camera-ready version.

We would like to thank the reviewer for reconsidering their evaluation based on our rebuttal. We recognize the importance of further clarifying the distinction between open-vocabulary domain adaptation (OVDA) and open-vocabulary segmentation (OVS), and we provide additional clarification below.

OVS is designed to be a generalist model and handle domain gaps by definition. In a prior work [1], cross-dataset setting for open-vocabulary segmentation is defined as follows:

Cross-Dataset Setting. In this setting, the model is trained on one dataset and evaluated on another dataset without fine-tuning. This is a more challenging setting than the zero-shot setting, where the model not only deals with the unseen classes, but also has to address the domain gap among different datasets.

However, OVDA for segmentation focuses on adapting OVS models to specialized domains while keeping their ability to respond to unseen queries and aims to infuse domain-specific priors to the existing models (as defined in Sec. 3). These domains could be indoor scenes, like offices and homes, or outdoor scenes like urban driving.

We hope this explanation further clarifies the distinction between OVDA and OVS. We understand that this should be better reflected in Sec. 3, where we define OVDA. This will be included in the final version of the paper.

[1] Xu et al., A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model, ECCV'22.

We would like to sincerely thank the reviewer for the detailed and constructive comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

Q1. The "lack of pixel-aligned training masks for VLMs (which are trained on image-caption pairs)" has been extensively studied in previous work [1, 2]. The authors should analyze and compare with [1, 2], even though they use different optimization strategies

We appreciate the reviewer for highlighting this point. Our model is designed as a specialist model that can complement generalist models effectively. For instance, the concurrent work MAFT+ [2] also focuses on adapting CLIP for open-vocabulary segmentation tasks. Notably, our model can integrate with such approaches, providing additional improvements for domain-specific queries. Below, we present a direct comparison of our model on ADE20K-150 with the generalist models referenced in [1, 2].

Q2. This work introduces the concept of Domain Adaptation in OVS. The authors need to clarify the distinction between Domain Adaptation OVS and standard OVS. OVS itself can segment any text and has a general concept of data domains. In experimental settings, the authors use cross-dataset evaluation, which does not differ from the standard OVS settings.

Thanks for your question. Domain adaptation for OVS is conceptually similar to how domain adaptation for closed-set semantic segmentation relates to closed-set segmentation. It should be considered a sub-problem of OVS, because as in any other learning task dependent on the scenario, there is a practical interest to trade in overall generalization for higher performance on a better defined domain.

For instance, in scenarios such as deployment of a robot or XR headset in a factory, the environment domain is well-defined and the model can adapt to its environment while maintaining the ability to respond to verbal commands from users.

Our work addresses these practical needs by proposing the novel task of open-vocabulary domain adaptation. Through our analysis, we demonstrate that standard OVS methods have inherent generalization limits (as expected), and their performance can be improved by performing domain adaptation. While neither domain adaptation nor open-vocabulary segmentation alone fully addresses these issues, open-vocabulary domain adaptation bridges this gap by combining the strengths of both paradigms.

Q3. In Table 3, the authors only compare with the CVPR23 work, which is insufficient. The latest OVS works [3, 4, 5] should be included for comparison.

We thank the reviewer for their suggestion. For further analysis, we integrate our domain-specific model into FC-CLIP [4] and a concurrent work MAFT+ mentioned above in [2]. We utilize OpenDAS to match segments upon their predicted masks. We observe that even without ground truth masks, our model can build upon generalist models’ prediction and achieve consistent improvement on ADE20K-150.

We would like to sincerely thank the reviewer for the detailed and constructive comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

Q1: The proposed framework is engineering, because the triple loss, visual and textual prompt tuning are common ways in open-vocabulary and domain adaptation communities.

We respectfully disagree with the notion that our framework is solely an engineering contribution. While our approach incorporates established techniques like triplet loss and prompt tuning, the key contribution lies in their integration into a unified framework specifically tailored for the novel task of open-vocabulary domain adaptation.

This task addresses critical real-world challenges that are not adequately addressed by existing paradigms, whether in open-vocabulary segmentation or domain adaptation, do not adequately resolve. By bridging these contexts, our framework introduces a new perspective that goes beyond the scope of traditional methods.

Furthermore, we conduct extensive comparisons with prior visual and textual prompt-tuning methods (see Table 1 and 2), showing a clear and significant advantage of the proposed framework in addressing this problem.

Q2: More recent vision-language models should be compared, like SigLIP, llama, etc.

Thank you for your suggestion. Our paper focuses on the general setting of open-vocabulary segmentation methods. As CLIP is the standard VLM used by most SOTA OVS models [1,2,3,4,5], we prioritize adapting CLIP to ensure easy integration and comparison to existing models in this domain.

We acknowledge the value of exploring more recent vision-language models like SigLIP and LLaMA, and we consider this an exciting direction for future research to further enhance our framework.

[1] Liang et al., Open-vocabulary semantic segmentation with mask-adapted clip, CVPR'23. [2] Xu et al., Side Adapter Network for Open-Vocabulary Semantic Segmentation, CVPR'23. [3] Cho et al., CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation, CVPR'24. [4] Yu et al., Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP, NeurIPS'23. [5] Jiao et al., Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation, ECCV'24.

Q3: Several ablation studies are needed, like the discussion about the margin μ

We thank the reviewer for their suggestion to look further into margin $\mu$. Our initial analysis had shown that $\mu=1.5$ was better than $\mu=1.0$ (default value in Pytorch) for domain adaptation. Now, we further look into the effect of $\mu$ and we add the results to the Appendix F. You can find the Weighted-F1 scores below. We observe that higher margin is only better when we test the model with the base classes. When there are novel classes, the model performance drops heavily after a certain threshold, which is in this case > 1.5.

We are open to further suggestions.

Q4: The writing and structure should be further improved, e.g., the pipeline fig is a bit ambiguous.

We are always eager to further improve our writing and make sure the paper is well-readable to everyone. While we focused our efforts during the discussion phase on the additional results, we would be very thankful for more input on what is currently ambiguous about Figure 2.

We thank the reviewer for acknowledging the clarity of our writing and recognizing the strong experimental results. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

Q1: The paper diverges from the established definition and test setup for open-vocabulary segmentation. While previous studies assume that a segmentation model fine-tuned on one domain should retain its generalizability across other domains, this work fine-tunes on a more general domain and evaluates on a narrower domain. Consequently, the test setup is not directly comparable to previous open-vocabulary approaches, such as those in (Liang et al. 2023)

Thank you for the question. We agree that our setup differs from the established definition and test setup for open-vocabulary segmentation. This difference was intentional and motivated by the limitations observed in existing setups, where generalizability across domains remains a significant challenge (see Fig. 1, Tab. 1 and 2 for no adaptation results).

To address these limitations and better reflect real-world use cases, we introduce a new setup for domain-adaptive open-vocabulary segmentation, where a model adapts to its environment while retaining the ability to respond to any language query from the user. This hybrid scenario lies at the intersection of domain adaptation and open-vocabulary segmentation, addressing gaps that neither field adequately covers.

We understand that this new setup is not directly comparable to prior open-vocabulary approaches. However, we argue that exploring relevant real-world problems, especially ones that have not been previously studied, represents an important scientific contribution rather than a weakness.

Our test setup was specifically designed to address these real-world user needs, which is also why we created our custom split from ScanNet++. The official splits do not sufficiently capture the scenarios we aim to address.

Q2: The fine-tuning approach assumes access to densely labeled, domain-specific images, which differs from previous domain adaptation frameworks that often use self-supervised or student-teacher setups. Here, the fine-tuning process effectively performs domain-specific classification training.

Our approach is motivated by real-world applications where access to densely labeled images in the target domain is often feasible such as deployment of a robot or XR headset in a factory. In these settings the model is required to learn in-domain language queries while answering to novel queries.

Unlike traditional domain adaptation frameworks, which rely on self-supervised or student-teacher setups, our method employs a unique hybrid framework inspired by supervised domain adaptation (SDA) and open-vocabulary segmentation. Specifically, we leverage domain-specific images as in adaptation approaches for foundational models [1], to better align the model with the target domain.

While SDA methods typically utilize small datasets for adaptation [2], we ensure that our dataset remains relatively small compared to the original vision-language model (VLM) training scale. This approach maintains the model's generalization capability while enabling effective domain adaptation. We further clarify our approach in the revised version (see L107-L108).

[1] Ha et al., Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptability, AAAI’24 [2] Wang et al., Deep visual domain adaptation: : A survey, Neurocomputing'18

Q3: What are the relations between ADE20K-150, SN++ Offices and KITTI-360 classes in Table 2?

To test adaptation on both base and novel queries, we use datasets with overlapping but not identical category sets. This approach allows us to evaluate our method across diverse visual domains, while ensuring a fair test of generalization to unseen categories.

• ScanNet++ Offices Split: The best test case for the target scenarios described in the introduction is our custom ScanNet++ Offices split. This curated subset of ScanNet++ ensures a significant number of test categories that never appear in the training set, providing a robust evaluation for novel category segmentation.
• Cross-Dataset Evaluation: To further test adaptation across distinct visual domains and facilitate comparisons on established datasets, we include cross-dataset evaluations using ADE20K, KITTI-360, and ScanNet++ Offices.

ADE20K-150 spans both indoor and urban scenes, creating natural overlaps with KITTI-360 (urban scenes) and ScanNet++ (indoor scenes). Most frequently appearing classes in KITTI-360 and ScanNet++ Offices also overlap with ADE20K-150 categories in different forms:

• 18 out of 37 classes in KITTI-360 also appear in ADE20K-150.
• 47 out of 233 test classes in ScanNet++ offices also appear in ADE20K-150.

We update the manuscript to further clarify this (see L373-L377).

Q4: Were the image segments used for "domain adaptation" obtained from densely labelled images?

Yes, this setup aligns closely with how supervised domain adaptation (SDA) is typically studied [1, 2, 3, 4]. Our work is motivated by scenarios where densely labeled target domain images are available, but the dataset is not large enough to cover all possible user queries. Therefore, we adapt large-scale pre-trained VLMs to the target domain using a relatively small amount of data compared to the original training dataset.

In our custom ScanNet++ offices split, designed specifically for our task, we use fewer labeled images for training and evaluate on more diverse scenes, including those with unseen objects. This setup reflects our task definition, where the dataset cannot fully cover all possible language queries.

We also agree with the reviewer that exploring an unsupervised variant of this problem is a promising direction. This is an area we are actively researching as a follow-up to this work.

[1] Wang et al., Deep Visual Domain Adaptation: A Survey, Neurocomputing'18. [2] Sun et al., Not All Areas Are Equal: Transfer Learning for Semantic Segmentation via Hierarchical Region Selection CVPR'19. [3] Ha et al., Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptability, AAAI'24. [4] Motiian et al., Unified Deep Supervised Domain Adaptation and Generalization, ICCV'17.