Concept-Based Unsupervised Domain Adaptation

Thank you for your valuable comments. We are glad that you found our framework "novel", our idea "interesting", and our proofs "correct". We address your comments in turn below.

Q1. ... relationship between the target domain error ... CBM (Formula 3) and the original target domain error in feature space ...?

To see the relationship between Formula 3 and the original target domain error in feature space, we start with the target error defined in the feature space (denoted as $\\mathcal{Z}$):

Without concept learning, we have the standard domain adaptation bound as (similar to Formula 1):

$$\\epsilon_{T}^{\\boldsymbol{z}}(h) \\leq \\epsilon^{\\boldsymbol{z}}_{S}(h) + \\frac{1}{2} d _{\\mathcal{H} \Delta \\mathcal{H}}(\\mathcal{D}^{\\boldsymbol{z}} _S, \\mathcal{D}^{\\boldsymbol{z}} _T) + \\eta^{\\boldsymbol{z}},$$

where $\mathcal{D}_S^{\boldsymbol{z}}$ and $\mathcal{D}_T^{\boldsymbol{z}}$ are denoted as the induced marginalized distributions in $\mathcal{Z}$, and all terms are defined in this space.

With concept learning, the bound above can be refined to include complex transformation error terms (resembling Formula 3):

$$\\epsilon\_T^{\\boldsymbol{z}}(h) \\leq \\epsilon^{\\boldsymbol{z}} _S(h) + \\frac{1}{2} d _{\\mathcal{H}\\Delta\\mathcal{H}}(\\mathcal{D}^{\\boldsymbol{z}} _S, \\mathcal{D}^{\\boldsymbol{z}} _T) + ( const \times \\mathbb{E} _S [ || \\hat{\\boldsymbol{c}} - \\boldsymbol{c}|| ] + error _{other}) + \\eta^{\\boldsymbol{z}},$$

where $error _{other}$ contains additional losses during the transformation from $\\mathcal{Z}$ to $\\mathcal{V}$ (e.g., projection errors). If $error _{other}$ is spread and absorbed into $\\epsilon _S^{\\boldsymbol{z}}(h)$, $d _{\\mathcal{H} \\Delta \\mathcal{H}}( \\mathcal{D} _S^{\\boldsymbol{z}}, \\mathcal{D} _T^{\\boldsymbol{z}})$, and $\\eta^{\\boldsymbol{z}}$, the resulting bound becomes structurally equivalent to Formula 3 (copied below):

$$\\epsilon _{T}(h) \\leq \\epsilon _{S}^{\\boldsymbol{c}}(h) + \\frac{1}{2} d _{\\mathcal{H} \\Delta \\mathcal{H}}(\\tilde{\\mathcal{D}}^{\\boldsymbol{c}}_S, \\tilde{\\mathcal{D}} _T) + R \\cdot \\mathbb{E} _{S} [ || \\hat{\\boldsymbol{c}} - \\boldsymbol{c} ||] + \\eta^{\\boldsymbol{c}}.$$

Thus, Formula 3 can be viewed as a refinement of the feature-space bound, making concept prediction error explicit and improving interpretability.

Q2. ... widely used DA benchmarks ... against more recent UDA methods ...

Good question.

Mainstream DA Datasets Are Not Applicable: Mainstream DA datasets for DA such as Office-31, Office-home, VisDA-2017 are not applicable to our setting because they do not contain concept annotations, which are needed to evaluate the correctness of concept prediction (and interpretation) in all methods.

Additional Datasets: As you suggested, we add a new dataset: AwA2-ImageNet, where AwA2 [Liu et al. 2015] is the source domain containing 85 concepts for animals, and ImageNet is the target domain. We match the AwA2 classes with corresponding subsets of ImageNet and filter overlapping data. We further perform style transfer on ImageNet to induce domain shift, making the task more challenging. The table below summarizes the results, which demonstrate our CUDA outperforms existing CBM variants and DA methods even in the case of larger domain gaps and more diverse data.

More Recent UDA methods. We did compare CUDA with GH++, a more recent, state-of-the-art UDA method (PAMI '24) that outperforms DANN and MCD. Our results show CUDA

• enables concept interpretation while GH++ cannot, and
• achieves competitive prediction performance. We also follow your suggestion to include more DA baselines, GVB (CVPR '20) and DWL (CVPR '21); see our response to Q2&W2 for Reviewer tPTT.

Q3. ... why CUDA stood out much more in WB-2 than other WB related datasets?

Poor Performance of DA Methods on WB-2. DA methods, such as GH++, rely heavily on class label information during training, without access to concept annotation. WB-2 has only binary labels, which contain limited information; it therefore leads to worse performance for DA methods.

Large Improvement from CUDA in WB-2. In contrast, CUDA utilizes concept learning and relaxed alignment, effectively leveraging the 112 concepts for binary classification in WB-2.

Smaller Improvement in WB-200. However, when applied to WB-200 or WB-CUB, it is more challenging for 112 concepts to represent 200 classes, leading to CUDA's smaller relative advantage against baselines.

Q4. Code reproducibility.

Thank you for your interest. We have finished cleaning up the source code and will release it for reproducibility.

Thank you for your valuable comments. We are glad that you found our method "novel and impactful", our theoretical guarantees "rigorous and mathematically sound", and the experiments "comprehensive" and "applicable across a wide range of scenarios". Below, we address your comments one by one.

W1. The paper lacks visual or qualitative examples to illustrate how relaxed alignment impacts interpretability.

Thanks for mentioning this. Following your suggestion, we provide a more concrete example to illustrate how relaxed alignment impacts interpretability. We will also include this example into our revised version.

We consider the task of making predictions for a target-domain image of a Black-Footed Albatross. The table below summarizes the predictions of different methods for both concept predictions and class label predictions; note that "CUDA w/o Relax" means "CUDA w/o Relaxed Alignment". Each row corresponds to one method, while the columns represent the predictions for specific concepts and class labels.

• Concept Predictions: The first three columns show predictions for specific concepts (e.g., Seabird Bill, Black Nepe, Solid Breast). The values "0" and "1" indicate whether the prediction is incorrect or correct, respectively. The ground-truth distribution (GT) of each concept (in terms of GT positive rates) in the source and target domains is provided in parentheses ("GT Ratio: Source / Target").

• Class Label Predictions: The last two columns show the predicted and ground-truth labels indices for each method.

We can see that:

• CBM predicts incorrect concepts, resulting in an incorrect class label prediction.
• CEM and CUDA (w/o Relax.) predict the correct class label but fail to predict some concepts correctly, which harms interpretability.
• Our full method (CUDA with Relaxed Alignment) predicts both the concepts and the class label correctly in the target domain, demonstrating that it achieves both interpretability and performance simultaneously.
• Interestingly, the table shows that concepts with clear distribution differences (e.g., "Seabird Bill" and "Black Nepe") lead to incorrect concept predictions in models without the relaxed alignment mechanism (e.g., CUDA w/o Relax. and CBM predict "Seabird Bill" incorrectly). This further supports the importance of our relaxed alignment.

Q1. Can the proposed method scale to larger datasets or more complex domains? Have you tested this?

Yes, our method can scale to larger datasets and more complex domains. To demonstrate this, we followed your suggestion to conduct additional experiments on ImageNet. In particular, we include a new dataset: AwA2-ImageNet [4], where AwA2 is the source domain containing 85 concepts for animals, and ImageNet is the target domain. We match the AwA2 classes with corresponding subsets of ImageNet and filter overlapping data. Furthermore, we perform style transfer on the target domain (ImageNet) to induce domain shift, making the task more challenging. The table below summarizes the results, which

• demonstrate our CUDA can improve performance even in the case of larger domain gaps and more diverse data and
• highlight the scalability of our approach while maintaining its performance and interpretability in larger and more complex domains.

[4] Deep learning face attributes in the wild, CVPR15.

Thank you for your valuable comments. We are glad that you found our theory "extensive"/"correct", our claim "convincing", our experimental designs/analyses "valid", and the paper "well-written". Below, we address your questions one by one.

W1 & Q1. Provide more concrete examples to demonstrate concept distribution differences between source and target domains.

• One example is the concept "Primary Color: Brown", where the positive rates are 19% and 17% for the source and target domains, respectively. Additional examples include "Black Bill" (48% in source / 51% in target) and "All-purpose Bill" (40% in source / 41% in target).

• Concept distribution differences are indeed a common problem in domain adaptation (DA) datasets with concept annotations. By directly counting the dataset statistics, we observe such discrepancies frequently arise across diverse DA scenarios.

W2 & Q2. Include experimental results on more datasets and compare with additional UDA methods (e.g., [1-3]).

This is a good question.

Common Datasets for Concept Bottleneck Models: Since our CUDA belongs to the category of concept bottleneck models (CBMs), we follow the literature (e.g., CBMs [1], CEMs [2], and PCBMs [3]) to use common datasets such as CUB, MNIST, and SkinCON in our experiments, as they provide rich concept annotations essential for evaluating concept-based methods.

Mainstream Datasets for DA Are Not Applicable: Note that mainstream datasets for DA such as Office-31, Office-home, VisDA-2017 are not applicable because they do not contain concept annotations, which are needed to evaluate the correctness of concept prediction in all methods.

[1] Koh et al., ICML'20. [2] Zarlenga et al., NIPS'22. [3] Yuksekgonul et al., ICLR'22.

Additional Datasets: Inspired by your comments, we add a new dataset: AwA2-ImageNet [4], where AwA2 is the source domain containing 85 concepts for animals and ImageNet is the target domain. We match the AwA2 classes with corresponding subsets of ImageNet and filter overlapping data. Furthermore, we perform style transfer on the target domain (ImageNet) to induce domain shift, making the task more challenging. The table below summarizes the results, which demonstrate our CUDA can improve performance even in the case of larger domain gaps and more diverse data.

[4] Deep learning face attributes in the wild, CVPR'15.

Comparison with UDA Methods [1-3]: Following your suggestion, we tested [5] and [6] on our nine datasets, and the results are summarized in the table below. However, [7] relies on a ViT backbone, making it unsuitable for fair comparison with our results, which use ResNet50. We agree that integrating transformer-based methods into our framework is a promising direction for future work. We will include these references in our paper. Thank you for this insightful suggestion!

[5] Gradually vanishing bridge for adversarial domain adaptation, CVPR20.

[6] Dynamic weighted learning for unsupervised domain adaptation, CVPR21.

[7] Safe self-refinement for transformer-based domain adaptation, CVPR22.