Text-driven Zero-shot Domain Adaptation with Cross-modality Graph Motif Matching

Major：

1. Inferior performance. Comparing Table 1 in this paper with Table 2 in Ref A and Table 3 in Ref B, the performance is significantly lower. For example, in terms of mAcc, this paper reports 66.38 in Table 1, while Ref A reports 80.2 in its Table 2. What could be the potential reasons for the performance gap?

2. No computational cost analysis. The authors claim the existing methods cannot accomplish a proper balance about the computation cost. However, there are no quantitative comparisons provided to demonstrate the superiority of this paper in computational cost. Could you provide specific metrics (e.g., training time, inference time, memory usage) to quantitatively compare the computational efficiency of your method to existing approaches？

3. No ablation analysis of the cosine similarities. The authors claim the cosine similarities of each paired image and text sample can cause the shared feature space to lack diversity. However, there's a lack of relevant ablation analysis. Could you compare your method against other baselines using only cosine similarities and demonstrate how this impacts feature diversity?

4. The related work section needs updating. For instance, there are no papers published in 2024 included in the Vision-Language Models section, and only one in the Unsupervised Domain Adaptation section. Additionally, Ref B is highly relevant but is neither cited nor analyzed.

Ref A: Unified Language-driven Zero-shot Domain Adaptation. In CVPR2024 (pp. 23407-23415).

Ref B: Collaborating Foundation Models for Domain Generalized Semantic Segmentation. In CVPR2024 (pp. 3108-3119).

Minor:

1. PØDA should be cited upon its first mention in this paper.

1. The novelty of the relationship-constraint adaptation is very limited. Directional loss and contrastive loss are present in many existing works, and the ablation experiment (Table 4) shows that these two components are not effective. For example, comparing the last line with the second-to-last line illustrates this point.

2. Regarding Figure 1, the information presented (e.g., star, triangle) is not well explained; however, it is clarified in Figure 2, which does not align with typical reading habits. Additionally, Figure 1 is missing a component between the image encoder and the stylized target feature, which I believe is the PIN component.

3. In Figure 2, the meaning of the red and blue rows is not clear.

4. In motif matching, there appears to be an error in Formula 4, as it is missing a '1 - '.

5. From lines 276 to 285, it is unclear whether motif matching is applied to matched motifs or all motifs.

6. In the experiment (Table 1), it appears that there is only minimal improvement, with some negative results when comparing the proposed method to PØDA.

• The method seems to be computationally more expensive than [1]. What are the additional time and memory constraints that are implied by the ensemble of the proposed components (constructing the hybrid cross-modality graph, calculating extreme features, mining graph motifs, optimizing PIN ...)? The additional losses improve the mIoU in general, however I am wondering how scalable and practically applicable is the approach, especially that the segmentation performance improvement is not very high.
• How are the values of $\lambda_{match}$, $\lambda_{dir}$ and $\lambda_{con}$ selected? Looking at the ablation on these values in the appendix (Table 6), some combinations lead to lower performance than [1], which raises doubts about practical selection of these hyperparameters when no target data is available.
• The authors of [2] propose also an extension of [1]. Is there any reason the authors do not compare with [2]? [2] is advantageous in training a single segmentation network for all the unseen domains. Are the contributions of this paper complementary to those of [2]?
• The authors of [A] use the same architecture (ResNet-50, DeepLabV3+) and less constrained language information (random texts instead of target-specific texts), and show strong results on domain generalization. This calls into question the practical interest of extending PODA [1] in the zero-shot adaptation setting. One interesting way to better assess the merit of the proposed method is to show if it exhibits some complementarity with [A].
• The image encoder being frozen during training, how would the method perform with better segmentation heads like Mask2Former [B] for instance? Freezing the backbone appears to be a better choice with strong segmentation heads [C]. In this case, would using PIN with the same losses have the same effectiveness?
• The qualitative results do not reflect significant improvement, which are hard to see even in the added boxes.
• The authors mention that coarsely aligning compressed visual features leads to the loss of domain-specific information. Any evidence about this? It is true that the alignment is performed in the latent space, however the optimizable variables in PIN [1] are affine transformations of features having spacial dimensions (height and width). This makes the claim a bit strong and not well-supported by evidence.

[2] Yang et al., Unified Language-driven Zero-shot Domain Adaptation. CVPR 2024
[A] Fahes et al., A Simple Recipe for Language-guided Domain Generalized Segmentation. CVPR 2024
[B] Cheng et al., Masked-attention Mask Transformer for Universal Image Segmentation. CVPR 2022
[C] Wei et al., Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation. CVPR 2024