Can We Evaluate Domain Adaptation Models Without Target-Domain Labels?

We sincerely thank the reviewer VJy8 for the insightful and constructive comments. We are glad that the reviewer acknowledges that the proposed metric is novel, the experiment is convincing, and the paper is well organized. In the rebuttal, we answer all the questions and add extensive experiments to improve our paper. We hope the responses can address the concerns. In addition, we have submitted a revised manuscript with an appendix where we mark all the suggested tables, figures, and analytics in magenta.

Q1: The experimental results of baseline methods should be provided in the experiment section. There are only Transfer Score results of many dataset.

Answer: Unsupervised model validation methods (DEV, SND) aim to select model parameters with better hyper-parameters (task 2&3) for the same UDA method, e.g., SND, C-entropy, DEV, ATC that have been compared in the experiments. However, they cannot achieve Task 1 which performs comparison on different UDA methods. The reason is that existing metrics are not comparable as they do not have a specific range of value. As far as we know, the transfer score is the first work that performs unsupervised UDA method comparison.

As suggested by the reviewer, task 2 should be compared with the SOTA method SND [1] (CVPR-21). To this end, we add two more experiments on DANN and SAFN using Office-31 (D$\to$A) and use transfer score and SND to select the hyper-parameters in an unsupervised manner. Note that the SND also studied the hyper-parameters of DANN so it is reasonable to make comparisons. As shown in the table below, the SND selects 62.4% while ours selects 67.8% on DANN. However, it is observed that SND fails to select a good hyper-parameter on the norm-based method SAFN. It contradicts the accuracy with a result of 31.3%, while ours still successfully selects 70.6%. Though the unsupervised metrics cannot fully align with the test accuracy, our transfer score can help select relatively high-performing hyper-parameters, outperforming SND significantly on hyper-parameter tuning. As for Task 3, the comparison with many baselines (DEV, SND, ATC, C-Entropy) has been discussed in Table 2 of the manuscript.

We also supplement the relationship between the training curves and other baseline metrics. We supplement the experiment by comparing our metric with two advanced unsupervised validation metrics, C-Entropy and SND, on Office-31 and VisDA-17. The figures have been added to the revised manuscript (Fig.11 and Fig.12). For convenience, we also provide the table below to show the correlation coefficients between the metrics and the model accuracy as follows. For both datasets, our approach shows the best correlation coefficients. Fig.12 shows that for large-scale datasets like VisDA-17, only our metric can reflect the overfitting issue while the other two metrics that keep increasing during training contradict the decreasing accuracies.

Q2: The authors should supplement the ablation study of Table 3.

Answer: We appreciate the suggestion and supplement two new UDA methods (CST [3] and AaD [4]) on a different dataset (VisDA-17) for the ablation study. As shown in the table below, the proposed transfer score selects the best model of 83.8% for CST and 85.9% for AaD. The results align with those of DomainNet. We have added the results to the revised manuscript in Table 3.

Q3: The Eq.1 is incorrect regarding the use of $i$.

Answer: We appreciate the careful check of the reviewer regarding the wrong formulas in Eq.(1). We have revised it in the manuscript.

Q4: Authors should provide the complete results of other tasks.

Answer: As suggested by the reviewer, we have added experiments on all other tasks of Office-31 and Office-Home. For Offce-31, we use all 4 tasks except the W$\to$D and D$\to$W as their transfer accuracy is over 99% which does not show the domain gap for UDA. For Office-Home, we supplement all 12 tasks. The results of Office-31 and Office-Home are shown as follows. In most cases, the transfer score can help select the best-performing model across the candidates on all tasks. Due to the page limit, these results have been summarized in Fig. 8 in the appendix of the revised manuscript.

Q5: What if the transfer score is used as a learning objective for UDA task? I am curious about this.

Answer: We appreciate the interesting suggestion and add the experiment to explore the effectiveness of the three components of the transfer score. The experiments are conducted on Office-31 (A$\to$W) and Office-Home (Ar$\to$Cl) based on ResNet50. We add the three parts of transfer score as an independent regularizer on the target domain. As shown in the table below, it is shown that every component brings some improvement compared to the source-only model. The total transfer score even brings significant improvement by 24.1% for A$\to$W and 12.6% for A$\to$W. The vanilla hopkins statistic is calculated at the batch level so it can be enhanced further as a learning objective for UDA. We think this is an explorable direction as an independent work in the future. The table and analytics have been added to the revised manuscript in Appendix A.10.

Q6+Q7: Some methods, e.g., SHOT, use the mutual information as an objective. Will it cause the proposed method become invalid? Can authors also provide some results on source-free domain adaptation (SFDA) methods (e.g., SHOT)?

Answer: Our method still works when Eq.4 (mutual information) is used as a learning objective. As we know, mutual information regularizer is used in SFDA methods (e.g., SHOT and AaD which uses a similar regularizer of Eq.4). Despite this, our proposed transfer score consists of three components, which is designed to avoid the single evaluation metric in previous unsupervised validation work (SND or C-Entropy). With three components, even though the mutual information is optimized by training, the other two metrics still work well. As suggested by the reviewer, we add experiments using two novel source-free UDA methods, SHOT (TPAMI’21) [2] and AaD (NeurIPS’22) [4], on Office-Home and VisDA-17. The results are shown as follows:

• The first table shows the accuracy and the transfer score on Office-Home. The AaD achieves the best results on Office-Home with the highest transfer score.
• The second table shows the improvement of epoch selection by transfer score on VisDA-17. Our method effectively selects a better epoch of the model for both methods, improving SHOT and AaD by 0.5% and 2.5%, respectively.

Therefore, the results show that the transfer score still works on source-free domain adaptation methods that leverage mutual information as a regularizer. We have added the results to Fig.8. As suggested by other reviewers, we also supplement the results on cross-domain semantic segmentation [1] and CAN [5] to the revised manuscript, which further demonstrates the effectiveness of transfer score on more tasks and totally 11 UDA baseline methods.

References

[1] Yang, Y. and Soatto, S., 2020. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4085-4095).

[2] Liang, J., Hu, D., Wang, Y., He, R. and Feng, J., 2021. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), pp.8602-8617.

[3] Liu, H., Wang, J. and Long, M., 2021. Cycle self-training for domain adaptation. Advances in Neural Information Processing Systems, 34, pp.22968-22981.

[4] Yang, S., Jui, S. and van de Weijer, J., 2022. Attracting and dispersing: A simple approach for source-free domain adaptation. Advances in Neural Information Processing Systems, 35, pp.5802-5815.

[5] Kang, G., Jiang, L., Yang, Y. and Hauptmann, A.G., 2019. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4893-4902).

I appreciate the author's efforts in conducting additional experiments to address my queries. However, I still have some remaining questions:

Q1: I think SND, Entropy and other UDA validation methods can be used for task1. They calculate a score for a specific task (UDA method+data+hyperparameters). The author's claim that these methods cannot be applied to task1 needs clarification. (The author did not provide citation information for SND and it should be a conference paper of ICCV21.)

Q2: About using as an independent regularizer, I'm curious if using it as an evaluation method would be ineffective in this scenario.

If the author can address my confusion, I would increase my score.

Dear Reviewer VJy8,

We have provided new experiments and discussions according to your valuable suggestions, which have been absorbed into the revised manuscript. We believe that the new manuscript is made to be stronger with your suggestions.

As the rebuttal deadline is approaching, we look forward to your reply or any other suggestions. Thanks so much!

Best Regards,

Authors of Submission 7412

Dear Reviewer VJy8,

We appreciate your response and conduct the experiments immediately. Here we provide the new results and analytics. We believe that these results let us have a better understanding of the proposed transfer score, and hope your concerns can be addressed accordingly.

Q1: I think SND, Entropy, and other UDA validation methods can be used for Task 1. They calculate a score for a specific task (UDA method+data+hyperparameters). The author's claim that these methods cannot be applied to Task 1 needs clarification. (The author did not provide citation information for SND and it should be a conference paper of ICCV21.)

We apologize for the unclear claim and our mistake. We agree with the reviewer that the scores of SND [1] and C-entropy [2] can be tested for Task 1 (i.e., UDA method comparison), though they do not consider Task 1 in their original papers. To this end, we conduct the experiments and calculate the SND and C-entropy of 6 UDA methods on three datasets including Office-31 (D$\to$A), Office-Home (Ar$\to$Pr), and VisDA-17. The results are shown in the tables below, where we mark the selected model for each metric with bold font. It is observed that the C-entropy succeeds in selecting the best model (MCC) on Office-31 but fails in Office-Home and VisDA-17, while SND fails all three datasets. In comparison, our proposed metric consistently selects the best UDA method for all three transfer tasks. All the new results and analytics have been added to the appendix A.11 in Fig.13.

Regarding the citation, we apologize for the missing information and have included the correct reference for SND in our revised manuscript.

Office-31 (D$\to$A)

Office-Home (Ar$\to$Pr)

VisDA-17

Q2: About using it as an independent regularizer, I'm curious if using it as an evaluation method would be ineffective in this scenario.

As suggested by the reviewer, we are happy to further explore this interesting problem. To this end, we perform the UDA model comparison on Office-31 (A$\to$W) and Office-Home (Ar$\to$Cl), and the model epoch selection on VisDA-17. Denote our new UDA method as TS-DA (Transfer Score for Domain Adaptation) in the table.

• For the model comparison (in the 1st table), we show the accuracy and the transfer score. We notice that the highest transfer score still indicates the best model on two datasets. This raises a question: if we aim to optimize the TS, the TS of TS-DA should be the largest. Then we look back to the training procedure of TS-DA, and we find that the TS has converged during training. This inspires us that the current challenge of TS-DA lies in the optimization of TS. In future work, a new UDA method could be proposed to bring more improvement by revamping the optimization of TS to get higher TS for TS-DA.
• For the epoch selection (in the 2nd table), our method still improves the vanilla TS-DA by 7.9% on VisDA-17, demonstrating the effectiveness of TS on TS-DA. The reason is that our epoch selection is based on the saturation level (variance) of the TS, which still works for TS-DA.

References

[1] Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. ICCV 2021.

[2] Minimal-entropy correlation alignment for unsupervised deep domain adaptation. ICLR 2018.

Dear Reviewer VJy8,

We have diligently addressed the remaining issues by conducting extensive experiments to support our responses. A revised manuscript has been submitted, accompanied by an appendix that delineates all revisions, highlighted in magenta color for clarity. We believe that the new manuscript is made to be stronger with your valuable suggestions.

As the rebuttal deadline is approaching in several hours, we sincerely look forward to your reply. Thanks so much for your time and effort!

Best Regards,

Authors of Submission 7412

Thanks for author's reply. The experiment of task1 added by the author solved my previous concerns.

I suggest the author update the description of task1 in the submission pdf and use UDA validation methods as some comparable baselines. It seems that task2 is similar.

I decide to improve my rating to 6.

We would like to summarize the difference between our metric and the existing unsupervised evaluation method (SND) in the table below. We have supplemented the experiments and the differences in the revised manuscript.

Q2: Only six UDA methods are evaluated and five candidate hyperparameters are provided. The number of these candidates should be increased.

Answer: We appreciate the suggestion and add extensive experiments to address this issue.

Firstly, we supplement four new UDA methods including SHOT [2] (TPAMI-21), CST [3] (NeurIPS-21), AaD [4] (NeurIPS-22), and CAN [5] (CVPR-19). The experiments are conducted on Office-Home and VisDA-17 and we use their open-source codes as the backbones. The results have been shown as follows:

• The first table shows the accuracy and transfer score. Our metric successfully selects the best-performing UDA method with the highest transfer score for these tasks.
• The second table demonstrates that using transfer score can help select a better epoch on VisDA-17, improving SHOT, CAN, CST and AaD by 0.5%, 0.4%, 11.8%, and 2.5%, respectively. These comparisons have been summarized and updated in Fig.2 in the revised manuscript and the appendix.

In summary, we demonstrate that our method works effectively on four more UDA methods, two of which are source-free domain adaptation methods. Compared to the SOTA method SND, we have 11 UDA baselines to prove the effectiveness while SND only uses 4 UDA baselines.

For the hyper-parameter candidate, we have enriched the candidate to [1, 2, 4, 6, 8, 10, 12, 14, 16] and the results have been shown in the first table in Q1. We demonstrate that our method can effectively select a better hyperparameter than SND.

We sincerely thank the reviewer iLFW for the detailed summary and constructive comments. We are glad that the reviewer acknowledges that the idea is simple and easy to follow. Here we answer all the questions and add extensive experiments for justification. We hope the responses can address the concerns. In addition, we have submitted a revised manuscript with an appendix where we mark all the suggested tables, figures, and analytics in magenta color.

Q1: Task1 and Task2 are actually unsupervised model evaluation problems, all the unsupervised validation methods such as SND can be employed directly, but the authors do not compare their methods.

Answer: We appreciate the valuable question. Unsupervised model evaluation methods (e.g., SND [1] and C-Entropy [7]) aim to select model parameters with better hyper-parameters for a specific UDA method. Though these works do not consider Task 1 in their original papers, as suggested by the reviewer, we find that their scores can still be tested for Task 1. To this end, we conduct the experiments and calculate the SND and C-entropy of 6 UDA methods on three datasets including Office-31 (D$\to$A), Office-Home (Ar$\to$Pr), and VisDA-17. The results are shown in the tables below, where we mark the selected model for each metric with bold font. It is observed that the C-entropy succeeds in selecting the best model (MCC) on Office-31 but fails in Office-Home and VisDA-17, while SND fails all three datasets. In comparison, our proposed metric consistently selects the best UDA method for all three transfer tasks, significantly outperforming existing unsupervised validation methods. All the new results and analytics have been added to the appendix A.11 in Fig.13.

Office-31 (D$\to$A)

Office-Home (Ar$\to$Pr)

VisDA-17

As suggested by the reviewer, Task 2 should be compared with the SOTA method SND [1] (CVPR-21). To this end, we add two more experiments on DANN and SAFN using Office-31 (D$\to$A) and use transfer score and SND to select the hyper-parameters in an unsupervised manner. Note that the SND also studied the hyper-parameters of DANN so it is reasonable to make comparisons. As shown in the table below, the SND selects 62.4% while ours selects 67.8% on DANN. However, it is observed that SND fails to select a good hyper-parameter on the norm-based method SAFN. It contradicts the accuracy with a result of 31.3%, while ours still successfully selects 70.6%. Though the unsupervised metrics cannot fully align with the test accuracy, our transfer score can help select relatively high-performing hyper-parameters, outperforming SND significantly on hyper-parameter tuning.

Q3: How do you select the transfer tasks on the datasets?

Answer: Following SND, we select some tasks that can contain all domains in Office-31 and Office-Home. The two tasks in DomainNet are selected as the evaluation of imbalanced long-tailed tasks. As suggested by the reviewer, we realize that this is not comprehensive. Therefore we have added experiments on all other tasks of Office-31 and Office-Home. For Offce-31, we use all 4 tasks except the W$\to$D and D$\to$W as their transfer accuracy is over 99% which does not show the domain gap for UDA. For Office-Home, we supplement all 12 tasks. The results of Office-31 and Office-Home are shown as follows (format: accuracy / transfer score). On Office-31, our method selects FixMatch, CDAN, and MCC for the best-performing models, which is very accurate. Our method also works effectively on all 12 tasks on Office-Home (whose table is displayed in the next paragraph). The results have been summarized in Fig. 8 in the appendix of the revised manuscript.

Results on other 8 tasks on Office-Home:

Q4: The three tasks in experiments are actually unsupervised validation (UV) problems, and this is an existing research field. Besides, some out-of-distribution generalization prediction methods also solve the model selection problem without labels.

Answer: We apologize for not clearly illustrating the difference between existing unsupervised validation methods and ours. We would like to elaborate on the difference from three perspectives:

1. Existing UV methods (SND, C-entropy) aim at selecting hyper-parameters or epochs for the same UDA method, while our metric can conduct a UDA method comparison (Task 1) for the first time. We demonstrate in Q1 that other metrics cannot achieve the model comparison task. Therefore, we claim that our method is the first work to “perform simultaneous model comparison and selection without target-domain labels”.
2. For the Task 2&3, we demonstrate our method outperforms existing methods on more datasets using more categories of UDA methods. The reason is that our metric considers multiple aspects of the model while others use a single metric. For example, SND only uses the clustering characteristics and C-Entropy only uses the prediction uncertainty of samples.
3. We have already considered the UV methods in the out-of-distribution (OOD) research community. These methods are mostly based on sample-level uncertainty while ours rely on model-level and manifold-level metrics. The experiments have been made between ours and ATC (ICLR’22) [6] in Table 2. Our transfer score outperforms ATC on Office-Home and VisDA-17 based on two novel UDA baselines.

References

[1] Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. ICCV 2021.

[2] Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer, T-PAMI 2021

[3] Cycle self-training for domain adaptation, NeurIPS 2021

[4] Yang S, Jui S, van de Weijer J. Attracting and dispersing: A simple approach for source-free domain adaptation. NeurIPS 2022

[5] Kang G, Jiang L, Yang Y, et al. Contrastive adaptation network for unsupervised domain adaptation, CVPR 2019

[6] Leveraging unlabeled data to predict out-of-distribution performance. ICLR 2022.

[7] Minimal-entropy correlation alignment for unsupervised deep domain adaptation. ICLR 2018.

I appreciate your experiments, which solve all my concerns and I will improve my rating.

Dear Reviewer iLFW,

We have diligently addressed all the issues by conducting extensive experiments to support our responses. A revised manuscript has been submitted, accompanied by an appendix that delineates all revisions, highlighted in magenta color for clarity. We believe that the new manuscript is made to be stronger with your valuable suggestions.

As the rebuttal deadline is approaching, we sincerely look forward to your reply. Thanks so much for your time and effort!

Best Regards,

Authors of Submission 7412

We sincerely appreciate the reviewer eAAs for the insightful and constructive comments. We are glad that the reviewer acknowledges that the proposed metric is meaningful and intuitive. Here we answer all the questions and add extensive experiments for justification. We hope the responses can address the concerns. In addition, we have submitted a revised manuscript with an appendix where we mark all the suggested tables, figures, and analytics in magenta color.

Q1: How can we prove that “prevailing UDA methods could lead to model degeneration and negative transfer”? Can the proposed metric be applied to them for performance improvement in regular UDA settings?

Answer: There are two probable situations when UDA methods lead to negative transfer:

1. inappropriate hyper-parameters: Since we do not have labeled target-domain samples for hyper-parameter tuning, the inappropriate hyper-parameters could lead to deteriorating convergence. As shown in Fig.3, for three UDA methods (DANN, SAFN, MDD), wrong hyper-parameters lead to even worse performances than the source-only model. This is usually caused by a greater weight of the transfer loss that dominates the training.
2. Overfitting to source domain: For large-scale UDA datasets, it is observed that the model is prone to overfitting to the source domain after a few epochs of training, leading to negative transfer. As shown in Fig.4, we see the overfitting happens on VisDA-17 and DomainNet. The SAFN even drops from 41% to 25% on DomainNet.

The issue of negative transfer is also studied in some literature [7] that analyzes the reasons behind it. Our metric can effectively deal with the issues of negative transfer for regular UDA settings. In practice, as we do not have any target-domain labels, we usually use hyper-parameters and a fixed number of epochs sourced from the paper. However, the UDA method performs distinctly on different data, so the negative transfer may happen due to the aforementioned reasons. Our method is designed to work effectively in regular UDA settings. The task 2 and 3 in our work aim to overcome the issue by unsupervised validation on the target domain. The results in Table 1 and Fig.3 show the improvement of various UDA methods on different public datasets.

Q2: Some existing UDA methods also adopt a metric for guiding the model learning [1]. A comparison should be made.

Answer: We appreciate the suggestion and read the paper carefully. This paper proposes an active domain adaptation approach (CNTGE) based on a transferability metric. We would like to illustrate their differences regarding the tasks, method, and experiments.

In the table, we see that the proposed metric is different from the sample-level transfer score in CNTGE. The transfer score proposed by CNTGE aims at selecting uncertain samples for active domain adaptation. It is a simple but effective metric for sample selection during UDA training, while our transfer score is a generic metric for unsupervised validation. Our metric measures the model-level transferability of the uniformity of model weights, the mutual information of features, and the clustering tendency. We have supplemented the contents in the manuscript.

Q3: Figure 1 is only conducted in one UDA task and model. More analysis should be conducted.

Answer: We appreciate this suggestion. The Fig.1 is originally regarded as an intuitive figure that shows the difference between existing metrics and our metric by a preliminary experiment. As suggested by the reviewer, we supplement the experiment by comparing our metric with two advanced unsupervised validation metrics, C-Entropy [8] and SND [9], on Office-31 (A$\to$W) and VisDA-17. Six figures have been added to the revised manuscript (in the appendix, Fig.11 and Fig.12). For convenience, we also provide the table below to show the correlation coefficient between the metrics and the model accuracy as follows. For both datasets, our approach shows the best correlation coefficients. Fig.12 shows that for large-scale datasets like VisDA-17, only our metric can reflect the overfitting issue while the other two metrics that keep increasing during training contradict the decreasing accuracies.

Q4: In definition 2, how the authors design the score should be justified.

Answer: We appreciate the suggestions and apologize for the lack of justification. In definition 2, we aim to use a larger transfer score to indicate better transferability. To this end, as the greater uniformity indicates a larger bias and lower transferability, we use the negative uniformity. In contrast, we use the Hopkins statistic and the absolute value of mutual information as they are positively correlated to the transferability. The mutual information is especially normalized because it does not have a fixed range as the uniformity and Hopkins statistic does. We have included the justification of the design in the revised manuscript.

Q5: Some SOTA UDA approaches should be employed for experiments.

Answer: As suggested by the reviewer, we supplement four new UDA methods including SHOT [2] (TPAMI-21), CST [3] (NeurIPS-21), AAD [4] (NeurIPS-22), and CAN [5] (CVPR-19). The experiments are conducted on Office-Home and VisDA-17 and we use their open-source codes as the backbones. The results have been shown as follows:

• The first table shows the accuracy / transfer score on Office-Home. Our metric successfully indicates the best-performing method with the highest transfer score for these tasks.
• The second table demonstrates that using transfer score can help select a better epoch on VisDA-17, improving SHOT, CAN, CST, and AaD by 0.5%, 0.4%, 11.8%, and 2.5%, respectively. These comparisons have been summarized and updated in Fig.2 in the revised manuscript and the appendix.

In summary, we demonstrate that our method works effectively on four more UDA methods, two of which are even source-free domain adaptation methods. Note that we cannot reproduce Fixbi [6] since its open-source codes do not release its specially pre-trained DANN model parameters, which have been an issue raised and unsolved in Github repository. To ensure fair comparison we therefore decided not to use Fixbi.

Q6: In Table 1, why are the improvements of the last four methods on DomainNet (p$\to$c) so small?

Answer: In Table 1, the improvement of our method is caused by selecting a better epoch after training. Hence, the margin of improvement is limited by the range of varying accuracy during training using a specific UDA backbone. For example, on DomainNet (p->c), we find that DANN shown in Fig.4(e) has a decreasing accuracy from 40.2% to 34.5%, so when our metric selects the best epoch, the improvement is 5.7%. However, some methods like MDD and MCC show stable training curves w.r.t. the accuracy. Even though our method selects a better epoch, the improvement is marginal. The improvements are closely dependent on the specific training curve based on the datasets and the UDA model.

References

[1] Active universal domain adaptation, ICCV 2021

[2] Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer, T-PAMI 2021

[3] Cycle self-training for domain adaptation, NeurIPS 2021

[4] Yang S, Jui S, van de Weijer J. Attracting and dispersing: A simple approach for source-free domain adaptation. NeurIPS 2022

[5] Kang G, Jiang L, Yang Y, et al. Contrastive adaptation network for unsupervised domain adaptation, CVPR 2019

[6] Fixbi: Bridging domain spaces for unsupervised domain adaptation, CVPR 2021

[7] Characterizing and avoiding negative transfer. CVPR 2019

[8] Minimal-entropy correlation alignment for unsupervised deep domain adaptation. arXiv preprint arXiv:1711.10288.

[9] Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. CVPR 2021

Dear Reviewer eAAs,

We have provided new experiments and discussions according to your valuable suggestions, which have been absorbed into the revised manuscript. We hope that the new manuscript is made to be stronger with your suggestions.

As the rebuttal deadline is approaching, we look forward to your reply or any other suggestions. Thanks so much!

Best Regards,

Authors of Submission 7412

Dear reviewer,

Thank you for your response. Regarding the new issues, we would like to address these issues by pointing out potential misunderstandings and misconceptions.

(1) Lack of comparison with different types of ``transfer score''. How can we ensure the score proposed by the authors is more effective?

We have already compared our method with four types of “transfer score”: (1) entropy-based score, C-entropy, (2) geometry-based score, SND, (3) out-of-distribution method, ATC, and (4) DEV for cross-validation. We perform the comparison based on two UDA baselines and three UDA tasks on Office-31, Office-Home, and VisDA-17. Please refer to the results in Table 2 which demonstrates that our method outperforms these existing “transfer score” for model selection. Furthermore, as C-entropy [1] and SND [2] are also metric-based methods, we can intuitively draw the relationship between these metrics and the test accuracy, as shown in Fig.11 and Fig.12, which shows that our metric has the strongest correlation with the target-domain accuracy during training. In summary, we argue that sufficient empirical studies have been made to show the superiority of the proposed metric against different types of "transfer score".

As suggested by your 1st review (Q2), to further demonstrate the effectiveness accordingly, we again add a new experiment using the sample-level “transfer score” (TS) [6] which has been discussed precisely in our 1st response. The sample-level metric cannot be directly used for model selection, so we use its average score across the whole target domain. The experiments are conducted on three challenging transfer tasks using DANN. Our metric outperforms the sample-level TS by 1.3%, 1.7% and 6.1% on VisDA-17, DomainNet (c$\to$p), and DomainNet (p$\to$c), respectively.

(2) Lack of theoretical analysis for the designed metric, which makes me think the proposed metric is still a heuristic one.

We acknowledge that the current paper focuses more on empirical validation and practical utility in realistic domain adaptation applications, rather than an in-depth theoretical exposition of the metric. The basis of our metric stems from three perspectives: (1) the uniformity of the classifier weights, (2) the clustering tendency, and (3) the mutual information. We empirically show the strong capacity of our transfer score on model comparison and validation using many UDA approaches, outperforming existing unsupervised validation approaches. We argue that it doesn't necessarily mean these heuristic metrics (including our metric, C-entropy [2], and SND [1]) lack novelty, but rather their theoretical foundations have not been thoroughly explored in the learning theory of domain adaptation [3][4]. We have also demonstrated that the metrics from the learning theory (i.e., the A-distance and maximum mean discrepancy [5]) cannot work for unsupervised model validation, as shown in Fig.1. In summary, we argue that the heuristic metrics have been commonly developed in unsupervised validation community, and should not be denoted as a shortcoming of our work.

We sincerely appreciate the suggestions and positively supplement massive experiments to enhance the demonstration. We believe that our paper has been significantly improved after addressing the issues.

References

[1] Minimal-entropy correlation alignment for unsupervised deep domain adaptation. ICLR 2018.

[2] Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. CVPR 2021.

[3] Analysis of representations for domain adaptation. NeurIPS 2006.

[4] Redko, I., Morvant, E., Habrard, A., Sebban, M., & Bennani, Y. (2019). Advances in domain adaptation theory. Elsevier.

[5] A kernel two-sample test. The Journal of Machine Learning Research, 13(1), 723-773.

[6] Active universal domain adaptation, ICCV 2021

We sincerely thank the reviewer hZ7S for the insightful and constructive comments. We are glad that the reviewer acknowledges that the problem is important and the method is effective on three tasks. Here we answer all the questions and add extensive experiments for justification. We hope the responses can address the concerns. In addition, we have submitted a revised manuscript with an appendix where we mark all the suggested tables, figures, and analytics in magenta color.

Q1: “Transfer Score” uses clustering and class balance as the measurement criteria. Does the transfer score apply to the imbalanced UDA task, such as the cross-domain classification and semantic segmentation?

Answer: Yes, the proposed transfer score works effectively on imbalanced UDA tasks. As illustrated in Section 5.5, to validate the effectiveness of handling imbalanced datasets, we have conducted an experiment on DomainNet (two transfer tasks) where severe imbalanced class distribution exists as shown in Fig.5(a). Our method still shows good results as shown in Table 1, Fig.5(b) and Fig.6. According to the reviewer’s suggestion, to better evaluate the imbalanced situation, we add a new experiment on semantic segmentation. We use a classic cross-domain segmentation approach FDA [1] as a baseline on GTA-5 -> Cityscapes. The mIoU results are shown below. It is shown that our metric can choose a better epoch of the model with an average mIoU of 39.7% while the last epoch of the model only achieves 37.7%. This demonstrates that our metric still works effectively on the imbalanced cross-domain semantic segmentation task.

Intuitively, the clustering metric (Hopkins statistic) measures the clustering tendency, which is irrelevant to class balance. The uniformity measures the classifier bias, i.e., how uniformly the feature space is divided by a classifier. It is directly calculated on model weights and does not measure the class balance. For UDA, the better uniformity of a classifier indicates better generalization ability. For example, if a classifier overfits the source domain, it should fail to perform well on the target domain with a different class distribution, and such a classifier has a low level of uniformity.

Q2: Does the transfer score work in some methods (SHOT, AAD) that directly use Eq.1 and Eq.4 as learning objectives?

Answer: Yes. Our method still works when Eq.4 (mutual information) is used as a learning objective. As mentioned by the reviewer, mutual information regularizer is very commonly used in source-free UDA methods (e.g., SHOT and AaD which uses a similar regularizer of Eq.4). Whereas, our proposed transfer score consists of three components, which is designed to avoid the single evaluation metric in previous unsupervised validation work. With three components, even though the mutual information is optimized by training, the other two metrics still work well. To prove it, we supplement the experiments using SHOT [2] and AaD [4] on Office-Home and VisDA-17. The results are shown as follows:

• The first table shows the accuracy and transfer score on Office-Home. The AaD achieves the best results on Office-Home with the highest transfer score.
• The second table shows the improvement of epoch selection by transfer score on VisDA-17. Our method effectively selects a better epoch of the model for both methods, improving SHOT and AaD by 0.5% and 2.5%, respectively. Therefore, the results show that the transfer score still works when Eq.4 is used as a learning objective. Eq.1 is proposed by us in this paper, and we have not found any methods using it as a learning objective.

Q3: The authors should verify the effectiveness of the “Transfer Score” on two advanced UDA methods (AAD, CAN).

As suggested by the reviewer, we have added the experiments on AaD [4] and CAN [5]. We also included an advanced UDA method CST [3] advised by another reviewer. The experiments are conducted on Office-Home and VisDA-17 and we use their open-source codes as the backbones. The results are shown as follows. The first table shows that the transfer score successfully indicates the best-performing method for these tasks. The second table demonstrates that using transfer score can help select a better epoch, improving CAN, CST, and AaD by 0.4%, 11.8%, and 2.5%, respectively. These comparisons have been updated in the revised manuscript in Fig.2 and the appendix.

References

[1] Yang, Y. and Soatto, S., 2020. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4085-4095).

[2] Liang, J., Hu, D., Wang, Y., He, R. and Feng, J., 2021. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), pp.8602-8617.

[3] Liu, H., Wang, J. and Long, M., 2021. Cycle self-training for domain adaptation. Advances in Neural Information Processing Systems, 34, pp.22968-22981.

[4] Yang, S., Jui, S. and van de Weijer, J., 2022. Attracting and dispersing: A simple approach for source-free domain adaptation. Advances in Neural Information Processing Systems, 35, pp.5802-5815.

[5] Kang, G., Jiang, L., Yang, Y. and Hauptmann, A.G., 2019. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4893-4902).

Dear Reviewer hZ7S,

We have provided new experiments and discussions according to your valuable suggestions, which have been absorbed into the revised manuscript. We hope that the new manuscript is made to be stronger with your suggestions.

As the rebuttal deadline is approaching, we look forward to your reply or any other suggestions. Thanks so much!

Best Regards,

Authors of Submission 7412