DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation

Q4: The experimental section should include comparisons regarding the computational resource consumption of the models, such as memory usage and inference time.

Thank you for raising this question. Our method comprises three main components: the UDA semantic segmentation model (HRDA), SAM, and CLIP. For semantic segmentation on high-resolution images, the memory usage primarily depends on the semantic segmentation model (HRDA), while the inference time is influenced by the reassignment of novel class labels. We provide detailed information on memory usage, inference time, and model parameters in the table below.

The total model parameter count of our method includes the UDA model parameters, plus 308M for SAM (ViT-L) and 351M for CLIP (ConvNeXt-Large). The reported total memory usage corresponds to the memory allocated during the complete inference process for a single image. We anticipate that further memory optimizations could be achieved by refining the code base, particularly by cleaning up intermediate tensor caches.

Overall, our framework requires reasonable computational resources. Moreover, the UDA framework can be flexibly substituted to accommodate memory limitations. As shown in the table, replacing HRDA with DAFormer significantly reduces memory consumption. For efficient inference, a segmentation model can also be trained using pseudo-labels generated by our framework, as described in Section 5.3. For example, a trained Mask2Former model using our inference pseudo labels, indicated as Ours (pseudo-label, Mask2Former), achieves inference in just 0.3 seconds per single image, offering a significant efficiency boost while maintaining high performance.

NOTE: For the inference time, the performance on average of 10 runs on a single NVIDIA A100-SXM4-80GB is reported.

[1] Gong, Rui, et al. Tacs: Taxonomy adaptive cross-domain semantic segmentation. ECCV 2022.

[2] Ren, Tianhe, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024).

[3] Hoyer, Lukas, et al. MIC: Masked image consistency for context-enhanced domain adaptation. CVPR 2023.

[4] Zhang, Chaoning, et al. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289 (2023).

[5] Wang, Feng, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. ECCV 2025.

[6] Touvron, Hugo, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).

Q3: How to better utilize foundation models has been a significant research topic in recent years, and I find the authors' exploration of this subject very meaningful. However, with the increasing number of foundation models available, it is insufficient to merely leverage more or better foundation models to complete tasks. The authors should emphasize the specific benefits of their design rather than simply implementing it in an unsupervised manner. Since the inference of large models is inherently unsupervised, simply using them does not constitute a substantial contribution.

We thank you for your thoughtful comments. We agree that simply leveraging foundation models does not inherently lead to superior performance or constitute a significant contribution in specific tasks. The primary contribution of our work lies in effectively fusing domain-specific knowledge with the prior knowledge of foundation models.

As highlighted in Table 1, domain-specific knowledge remains crucial for achieving robust performance. For example, our results compared to Grounded-SAM [2]—which relies solely on foundation models for open-vocabulary semantic segmentation—demonstrate that incorporating domain-specific knowledge through UDA significantly enhances performance, even when using strong foundation models for open-vocabulary tasks.

While foundation models are inherently unsupervised and designed for general-purpose tasks, the novelty of our approach lies in adapting these models to domain-specific challenges via a modular framework. To validate this, we tested the performance by substituting different modules within our framework and reported the results in the table below. By testing multiple configurations (e.g., HRDA → MIC, SAM → MobileSAM), we show that our framework does not rely on simply using "better" foundation models. Instead, it provides an effective and flexible way to integrate any general foundation models into domain-specific tasks, making the framework in a practical and adaptable way.

To summarize, our work is the first to implement and benchmark unsupervised taxonomy-adaptive domain adaptation, addressing the underexplored intersection of open-vocabulary tasks and domain adaptation. We hope that releasing our code and benchmark setup will serve as a useful resource for the community. Additionally, 'better' foundation models could improve performance in open-vocabulary setups, our results clearly demonstrate that combining these models with domain-specific knowledge significantly improves performance in road-scene segmentation.

Q1: Confusion between UDA and open-vocabulary methods

Thank you for your thoughtful feedback. The contribution of this paper lies exactly in addressing the intersection of UDA and open-vocabulary setups within a single setup. In this work, we try to solve the unsupervised taxonomy adaptive domain adaptation problem with challenges in two key aspects: 1) Unsupervised Domain Adaptation (UDA): While UDA methods address domain shifts, they typically assume consistent label spaces, making them inflexible for handling label-space discrepancies. 2) Taxonomy Discrepancies: Adapting to mismatched label spaces—arising from differences in granularity, definitions, or new categories—remains challenging. Recent open-vocabulary approaches built on foundation models prioritize open-world knowledge but often overlook the integration of domain-specific knowledge critical for specialized tasks. To conclude, most existing research and benchmarks have tackled these challenges separately (e.g., standard UDA assumes consistent label spaces, while taxonomy adaptation often overlooks domain shifts). Consequently, the intersection of these two problems remains a relatively new area with limited prior work. Our study highlights this gap and aims to provide a flexible framework to address both challenges simultaneously for more robust solutions to this real-world problem.

To address the concern about comparisons with UDA methods, we compare our performance with the traditional UDA setting below. As explained in Lines 410-411, Traditional DA methods are limited to fixed label spaces, making them inapplicable to our problem setting. Therefore, we compare the results for known classes of HRDA under the traditional UDA setting, adapted from GTA to the Mapillary dataset. The results show that our framework effectively preserves domain knowledge for the known classes while enriching prior knowledge for the unknown classes.

These results demonstrate that our approach is competitive even under traditional UDA settings while retaining the flexibility to handle open-vocabulary setups. This highlights the advantage of our framework in bridging these two tasks effectively.

Q2: I understand that earlier methods also utilized foundation models but still required target labels for few-shot training. In contrast, this paper employs foundation models in a fully unsupervised manner, correct? Is this the main difference/contribution this paper has compared to previous methods?

Thank you for raising this question. We would like to clarify that 1) the previous taxonomy-adaptive domain adaptation methods do not use foundation models, but few-shot in-domain labeled images for training. For example, TACS [1] uses few-shot images for each novel class to learn the new label distribution. Instead, our method leverages foundation models to incorporate its prior knowledge to replace the in-domain few-shot samples in a fully unsupervised method. 2) recent open-vocabulary methods leverage foundation models' prior knowledge for segmentation, but do not incorporate any in-domain knowledge. This leads to their inferior performance when it comes to specific domains.

To conclude, our main contribution does not solely lying in its unsupervised manner, but also in proposing a new benchmark and approach to leverage the prior knowledge of foundation models to stengthen the in-domain prediction and make it scalable to unknown classes.

I appreciate the authors' efforts in their rebuttal, which addressed most of my concerns, leading me to raise my score. While the rebuttal provides strong insights, these are not sufficiently reflected in the current version of the paper. As such, I am not fully confident in my revised score. If the paper is accepted, significant revisions will be necessary to better integrate the insights presented in the rebuttal.

Q6: The framework seems to rely purely on SAM for pixel grouping, with UDA and CLIP then reclassifying each mask region. Is this my understanding accurate?

Thank you for bringing up this question. We would like to clarify that the pixel grouping knowledge in our framework does not rely solely on SAM. The initial pixel grouping is derived from the in-domain predictions of the UDA model. Building on this, SAM is utilized to identify finer-grained pixel groups.

To integrate in-domain knowledge with prior knowledge, we fuse predictions from the UDA model and CLIP to reclassify each region. To enhance the use of prior knowledge, we propose multi-scale visual feature extraction for improved scene understanding at different scales. Additionally, context-aware text feature extraction is introduced to fully capture the semantics of class labels. When the confidence for the reassigned new class is low (below the 0.5 threshold), we retain the pixel grouping from the UDA model's original predictions. Thus, our framework does not rely solely on SAM for pixel grouping with a reclassification based on CLIP. Instead, it combines the strengths of both UDA and SAM predictions to produce a more robust and accurate final pixel grouping. Furthermore, our approach introduces an effective method to fuse knowledge from different sources, rather than merely combining them.

We hope this clarification addresses your concerns and are happy to provide further explanations if needed.

[1] Hoyer, Lukas, et al. MIC: Masked image consistency for context-enhanced domain adaptation. CVPR 2023.

[2] Zhang, Chaoning, et al. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289 (2023).

[3] Wang, Feng, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. ECCV 2025.

[4] Touvron, Hugo, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).

Q3: Criteria for Label Ambiguity: The paper states, “For labels without ambiguity, e.g., sky, we only use the original label for text feature extraction” (lines 1000-1001). The criteria for “ambiguity” remain unclear, introducing additional heuristics or human intervention.

Thank you for raising this concern. For labels with inherent ambiguity, we refer to classes that are broad or highly context-dependent. For example, the class "other vehicles" may encompass various types of vehicles that are not explicitly defined in the dataset, making precise labeling a challenge. Similarly, the class "ego vehicle" refers specifically to the vehicle equipped with the camera, which often appears only partially in the image. This adds complexity, as the visual cues for identifying this class are limited and heavily reliant on context. Such ambiguous labels are difficult to describe accurately in language because their meaning often depends on the dataset’s specific context or visual representation. This highlights the importance of careful consideration during the mapping process to ensure accurate and meaningful interpretations. In datasets with more domain-specific expert knowledge, these ambiguous classes can be better defined and mapped appropriately.

Q4: Additional Baselines with Foundation Models: Numerous studies have leveraged foundation models for semantic segmentation, yet many of these are omitted in the experimental section. Including models such as [A] for domain-generalized segmentation and [B] for open-vocabulary segmentation would enhance comparability.

We thank you for mentioning the important works for comparison. For work [A] in domain generalized segmentation, as it does not address label-level adaptation, a direct comparison is not feasible. This work [A] leverages the foundational models to mitigate the domain gap on the image level. We thank you for highlighting this correlated work and have updated our paper to include a discussion on this work in the related works section. For work [B] on open-vocabulary segmentation, we are in the process of adapting it to the street scene dataset used in our study and will update the experimental results later.

Q5: Traditional Domain Adaptation Settings: Although the framework targets handling substantial label-space domain gaps for real-world applications, demonstrating its efficacy in traditional domain adaptation settings (e.g., GTA5 to Cityscapes, with a smaller label-space gap) is also important. and Other Label-Space Domain Gaps: The experiments primarily focus on scenarios where the target domain includes more semantic classes than the source (e.g., 45 classes for Mapillary, 24 for IDD, and 19 for GTA). Evaluating scenarios with fewer classes in the target domain (e.g., Mapillary to Cityscapes) would substantiate the framework’s generalizability.

We thank the reviewer for pointing this out. In the first table below, we present results for the following settings: 1) Traditional UDA Setting: In this case, the target label space is identical to the source label space. We adapt from GTA to Cityscapes, both using the same label space. 2) Fine-to-Coarse Setting: Here, the target domain labels are coarser than the source domain labels. For this, we adapt from Mapillary (45 classes) to Cityscapes (19 classes). In both the traditional UDA and fine-to-coarse settings, where sufficient domain knowledge about the label space is available, our framework preserves the in-domain performance and provides slight improvements when adequate in-domain knowledge is already present.

In the second table, we present results under taxonomy-adaptive domain adaptation with a smaller label-space domain gap. Specifically, we adapt from Synthia (16 classes) to Cityscapes (19 classes). These results highlight that our method not only maintains strong performance on known classes but also predicts effectively for unknown classes.

Q1: Inference complexity

We thank you for raising this concern. We provide details on the memory usage, inference time, and model parameters in the table below for your reference. The total model parameter count of our method includes the UDA model parameters, plus 308M for SAM (ViT-L) and 351M for CLIP (ConvNeXt-Large). The reported total memory usage corresponds to the memory allocated during the complete inference process for a single image. We anticipate that further memory optimizations could be achieved by refining the code base, particularly by cleaning up intermediate tensor caches.

Overall, our framework requires reasonable computational resources. Moreover, the UDA framework can be flexibly substituted to accommodate memory limitations. As shown in the table, replacing HRDA with DAFormer significantly reduces memory consumption. For efficient inference, a segmentation model can also be trained using pseudo-labels generated by our framework, as described in Section 5.3. For example, a trained Mask2Former model using our inference pseudo labels, indicated as Ours (pseudo-label, Mask2Former), achieves inference in just 0.3 seconds per single image, offering a significant efficiency boost while maintaining high performance.

NOTE: For the inference time, the performance on average of 10 runs on a single NVIDIA A100-SXM4-80GB is reported.

Regarding the combination of the UDA model, SAM, and CLIP, we would like to emphasize that our framework does not involve a complex integration strategy. Instead, it is modular, allowing straightforward combination and substitution of individual components. As demonstrated in the table below with results on the Mapillary dataset, we substituted specific modules (e.g., replacing HRDA with MIC), and the results highlight the flexibility of our method to integrate effectively with various existing models. Overall, while our framework comprises multiple components, it is not strategically complex. On the contrary, it is flexible and practical across diverse scenarios.

Q2: Semantic Taxonomy Mapping: The framework initially prompts GPT-4 for potential taxonomy mappings, refined through human input. This process adds a human burden due to prompt tuning and taxonomy refinement and is only briefly outlined in the supplementary material (lines 916 & 947), lacking comprehensive detail.

We thank the reviewer for highlighting this important issue. Our framework utilizes GPT-4 to generate an initial proposal for potentially correlated taxonomy mappings between the source and target domains. However, we acknowledge that human intervention is necessary to refine these mappings due to the differing definitions of classes across datasets. For example, in the Mapillary dataset, the term "guard rail" specifically refers to roadside guard rails, whereas in other datasets or in general usage, it may represent a broader guarding fence. These discrepancies underscore the importance of incorporating human understanding to ensure that the mappings are accurate and contextually appropriate.

We recognize that this process adds a degree of human involvement and that the results of the framework can be influenced by the quality of these mappings. However, it is important to emphasize that the involvement of human expertise in this process is minimal. The taxonomy mapping only needs to be defined once per dataset pair and involves verifying and refining the automatically proposed mappings, which is a significantly smaller effort compared to tasks such as pixel-wise manual annotation. This small but essential step ensures consistency and reliability in the results without imposing a substantial burden on practitioners.

Q6: The symbol usage is inconsistent.

Thank you for reading carefully and pointing this out! We have updated the paper to ensure consistent usage of the term multi-scale visual feature. In Figure 3, $F_l$ represents the local feature, and $F_g$ denotes the extracted global feature. The final visual feature $F_V$ is computed as a weighted sum of $F_l$ and $F_g$, as detailed in Formula 6. To improve clarity, we have added more detailed descriptions to the paper. We appreciate your feedback in helping us make this improvement and are open to any further adjustments that are needed!

Q7: In Figure 3, what does lane-marking 0.6 mean?

Referring to Formula (7), we compute the CLIP visual-text similarity between the visual feature and the text features of the candidate classes. The softmax value of these similarities is used as the prediction confidence for each class label. Specifically, the value 0.6 in Figure 3 represents the estimated softmax similarity for the class label "lane-marking", which we use as the confidence score for predicting the mask region as belonging to this class.

Q8: In 5.4 ABLATION STUDIES, should multi-scale vision feature be changed to multi-scale visual feature? Please keep the name consistent in the paper.

Thank you for pointing out this inconsistency. We have updated all occurrences to ensure uniform usage of the term multi-scale visual feature throughout the paper, including in Section 5.4 and the corresponding figure captions.

Q9: Please add more experimental details. For example, the proposed method uses the CLIP backbone, but is it the CLIP ViT-B-16 or CLIP ViT-L-14 backbone or something else?

Thank you for raising this question. As shown in the table in Q2, we use the SAM ViT-L model and the ConvNeXt-Large CLIP model in our experiments. All these implementation details to reproduce our results are included in the instructions and code, which we will make publicly available after publication.

Q1-2: The explanation of figure and method can be clearer

Thank you for your detailed suggestions on improving the clarity of our paper.

Regarding the two CLIPs in our figure, we would like to clarify that the first CLIP represents the vision and text encoder used to extract corresponding vision and text features. The second CLIP represents the process of computing similarity between these features to obtain probabilities. We have provided additional details and add arrows in Figure 3 for better understanding in the rebuttal updated version.

For the two-stage approach, as described in Lines 202–209, we referred to extracting domain-specific knowledge and prior knowledge. In Figure 2, we depicted three components: extracting domain-specific knowledge, extracting prior knowledge, and demonstrating how they are fused.

We acknowledge that the notations in the figure, along with the explanations, may have caused ambiguity. To address this, we have revised the description into a three-stage framework and updated the figure in the revised version of the paper. Thank you again for bringing this issue to our attention.

Q3: Memory usage, training time, and model parameter.

Thank you for raising this question. Our method comprises three main components: the UDA semantic segmentation model (HRDA), SAM, and CLIP. For semantic segmentation on high-resolution images, the memory usage primarily depends on the semantic segmentation model (HRDA), while the inference time is influenced by the reassignment of novel class labels. We provide detailed information on memory usage, inference time, and model parameters in the table below.

The total model parameter count of our method includes the UDA model parameters, plus 308M for SAM (ViT-L) and 351M for CLIP (ConvNeXt-Large). The reported total memory usage corresponds to the memory allocated during the complete inference process for a single image. We anticipate that further memory optimizations could be achieved by refining the code base, particularly by cleaning up intermediate tensor caches.

Overall, our framework requires reasonable computational resources. Moreover, the UDA framework can be flexibly substituted to accommodate memory limitations. As shown in the table, replacing HRDA with DAFormer significantly reduces memory consumption. For efficient inference, a segmentation model can also be trained using pseudo-labels generated by our framework, as described in Section 5.3. For example, a trained Mask2Former model using our inference pseudo labels, indicated as Ours (pseudo-label, Mask2Former), achieves inference in just 0.3 seconds per single image, offering a significant efficiency boost while maintaining high performance.

NOTE: For the inference time, the performance on average of 10 runs on a single NVIDIA A100-SXM4-80GB is reported.

Q4: As a new benchmark, will the proposed method publish the complete training and testing code for subsequent follow-up research?

Thank you for raising this valuable question and recognizing our contribution in proposing a new benchmark! We will release the full training and inference code publicly after publication. We also anticipate that this will encourage and support more follow-up studies in the community.

Q5: How is the context description set generated? In formula 2, on which dimension is the average operation performed?

Thank you for the detailed question! As described in the appendix A.5 CONTEXT NAMES, we generate the context descriptions by prompting GPT-4 with the following instruction: "Generate new class names within the context of street scene semantic segmentation, using the original class name as the head noun. Use synonyms or subcategories of the original class that make sense within this context, and if the class has multiple meanings, add specific context to avoid ambiguity. Please provide the original class names along with the context names." Using this approach, we ensure that the generated context descriptions are both meaningful and specific to the domain of street scene segmentation.

Regarding Formula 2, for each word token in a context description, we compute its CLIP text feature embedding, which has a dimension of [768]. For all context description embeddings, we concatenate them to form a text embedding of shape $[k, 768]$, where $k$ denotes the number of words in the context description. We then average along the first dimension ($k$) to obtain a single averaged representation of shape $[1, 768]$.

Q1: In Lines 80-81, why this problem is significant but not be studied?

Thank you for raising this interesting discussion. We find the unsupervised taxonomy adaptive domain adaptation problem meaningful yet underexplored, primarily due to its complexity in two key aspects: 1) Unsupervised Domain Adaptation (UDA): While UDA methods address domain shifts, they typically assume consistent label spaces, making them inflexible for handling label-space discrepancies.2) Taxonomy Discrepancies: Adapting to mismatched label spaces—arising from differences in granularity, definitions, or new categories—remains challenging. Recent open-vocabulary approaches built on foundation models prioritize open-world knowledge but often overlook the integration of domain-specific knowledge critical for specialized tasks.

To conclude, most existing research and benchmarks have tackled these challenges separately (e.g., standard UDA assumes consistent label spaces, while taxonomy adaptation often overlooks domain shifts). Consequently, the intersection of these two problems remains a relatively new area with limited prior work. Our study highlights this gap and aims to provide a flexible framework to address both challenges simultaneously for more robust solutions to this real-world problem.

Q2: What are the experimental results of existing taxonomy-adaptive domain adaptation and UDA methods on two street scene semantic segmentation benchmarks?

Thank you for raising this question. As explained in Lines 410-411, Traditional DA methods are limited to fixed label spaces, making them inapplicable to our problem setting. Therefore, we compare the results for known classes of HRDA under traditional UDA setting, adapted from GTA to Mapillary dataset. The results show that our framework effectively preserves domain knowledge for the known classes while incorporating rich prior knowledge for the unknown classes.

We identified two existing taxonomy adaptation methods that address the taxonomy adaptation problem in a few-shot setting [1, 2]. Specifically, [1] focuses on medical semantic segmentation, making it not directly applicable to our experimental setting. [2] requires representative few-shot supervision for each unknown class to adapt the original model. For a fair comparison, we present results for the Synthia → Cityscapes task following the same setting as in [2]. The results demonstrate that our method outperforms on unknown classes, even without in-domain supervision. Additionally, we will attempt to adapt [2] for scenarios involving a larger number of unknown classes on the Mapillary and IDD datasets. These results will be incorporated into the paper to provide a more comprehensive and thorough comparison.

Note: Comparison of mIoU between traditional UDA method (HRDA) and Our DynAlign from Cityscapes (19 classes) to Mapillary (45 classes).

Note: Comparison of mIoU between the few-shot taxonomy adaptive DA method (TACS) and Our DynAlign from Synthia (16 classes) to Cityscapes (19 classes).

Q3: How to choose the confidence threshold value in Lines 372 - 373? More parametric experiments about the confidence in Lines 372 - 373 need to be conducted and discussed.

Thank you for raising this important question. In our experiments, we set the confidence threshold to 0.5 by default for reassigning the class label. To address your concerns, we have conducted ablation studies with varying confidence thresholds, and the results are provided in the table below. The results show that while the mAcc improves with a lower confidence threshold for assigning new class labels, the mIoU may decrease correspondingly. Overall, our framework demonstrates robustness and is not very sensitive to the confidence threshold within a certain value range.

[1] Fan, Jianan, et al. Taxonomy adaptive cross-domain adaptation in medical imaging via optimization trajectory distillation. ICCV 2023.

[2] Gong, Rui, et al. Tacs: Taxonomy adaptive cross-domain semantic segmentation. ECCV 2022.

Q1-2: Integration with other methods: it would be better that author provide another examples to show the compatibility of the proposed method (e.g., MIC + DynAlign, or other UDA model + DynAlign) and use different foundation model (e.g., MobileSAM for SAM, Llama for GPT-4, another VLM for CLIP) can also work with the proposed method to show this compatibility can provide more flexible fusion environment.

Thank you for suggesting this meaningful experiment! In the table below, we present results on the Mapillary dataset, where we substitute individual modules in our framework (e.g., replacing HRDA with MIC). The results demonstrate the flexibility of our method to work effectively in combination with various existing models. Due to time constraints, we implemented only one substitution per module for this study. However, we acknowledge the value of a more comprehensive evaluation and will include additional substitution experiments in the final version of the paper.

Q3: Any failure case while fusing UDA and foundation model?

Due to the flexibility of our method, any UDA model or foundation model can be substituted into the corresponding module, as demonstrated in the response above. Potential failure cases may arise when foundation models, such as SAM or CLIP, are applied to domains significantly different from those encountered during their pre-training, such as medical image segmentation. This challenge could potentially be addressed by using foundation models specifically developed for specialized fields [5, 6]. Nevertheless, their performance in this context needs to be thoroughly tested and validated.

[1] Hoyer, Lukas, et al. MIC: Masked image consistency for context-enhanced domain adaptation. CVPR 2023.

[2] Zhang, Chaoning, et al. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289 (2023).

[3] Wang, Feng, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. ECCV 2025.

[4] Touvron, Hugo, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).

[5] Zhang, Yunkun, et al. Text-guided foundation model adaptation for pathological image classification. MICCAI 2023.

[6] Chen, Zeming, et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079 (2023).

Q1: In real-world scenarios, the label structure of the target domain may sometimes be coarser than that of the source domain. How would the proposed method be adapted for such a setting?

We thank the reviewer for pointing this out. To address this concern, we have conducted additional experiments under Fine-to-Coarse Setting: Here, the target domain labels are coarser than the source domain labels. For this, we adapt from Mapillary (45 classes) to Cityscapes (19 classes). The results demonstrate that our framework produces robust predictions under the fine-to-coarse setting, effectively handling scenarios where the label spaces are not identical.

Q2: The contribution towards DG area is limited.

We thank you for your for highlighting areas of concern. While our framework incorporates pre-trained models such as SAM and CLIP, the primary contribution lies in integrating their general knowledge with domain-specific knowledge from UDA. This fusion enables the framework to address real-world scenarios where the generalization capabilities of foundation models alone are insufficient, as demonstrated in Table 1. Therefore, our work emphasizes strengthening UDA in-domain performance through this integration, rather than leveraging or improving the generalization ability of pre-trained models themselves.

We would also like to clarify that the taxonomy mapping serves as one step in our framework to establish the initial label-space correlation. Beyond this, we contribute an effective approach to combine domain knowledge and prior knowledge by fusing their predictions. Additionally, we have compared our method with others, such as Grounded-SAM [1], which maps open-vocabulary labels to SAM-segmented regions. The results validate the effectiveness of our approach in leveraging in-domain knowledge to improve the performance. We thank you again for opening this discussion and are willing to discuss further or conduct additional ablations to compare with more related works.

We acknowledge that this paper places greater emphasis on practical contributions than on theoretical developments. Specifically, we introduce a new benchmark and experimental setup to evaluate taxonomy adaptive domain adaptation under both domain and label-space shifts. Our work is an initial attempt at addressing these challenges through a robust and practical approach, which we believe also contributes to the ICLR community. We are happy to receive suggestions or more discussions on possible theoretical analyses that could be interesting to explore in future in-depth follow-up studies and thank you again for your insightful feedback.

[1] Ren, Tianhe, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024).

Minor issues We thank the reviewer, and have fixed them in the updated main document.