Advancing Open-Set Domain Generalization Using Evidential Bi-Level Hardest Domain Scheduler

We appreciate the review effort from Reviewer tsHR and provide the point-to-point responses as follows. All the explanations will be included into our final paper.

W1: Training order scheduler is very important in deep learning for some specific techniques, e.g., meta learning and curriculum learning. In curriculum learning, researchers propose the instance scheduler to decide the next training samples to pursue a better optimization of the performance of the deep learning model, where the samples do not contain temporal correlation, as demonstrated in works, [a] and [b]. Meta-learning aims at reserving different training tasks for model optimization, where the training order can guide the model to different optimization situations. Choosing different tasks/samples for training at different stages can result in obvious performance differences in the models. MEDIC and MLDG propose to use meta learning techniques for OSDG, where meta learning shows strong OSDG capability when the meta training and meta testing are conducted on different source domains. However, both of them rely on sequential source domain order when setting up the meta tasks. Our experiments turn out that sequential domain scheduler is not the optimal way for domain scheduling in the meta learning of the OSDG task. In this work, we for the first time propose an adaptive domain scheduler in a bi-level optimization manner, which shows strong OSDG capability across different backbones and datasets.

We conducted the ablation in Table 6 of our main paper to demonstrate the performance gains brought by our proposed adaptive domain scheduler, when we compare our proposed domain scheduler method with random domain scheduler and sequential domain scheduler.

[a] Yulin Wang, Yang Yue, Rui Lu, Tianjiao Liu, Zhao Zhong, Shiji Song, and Gao Huang. Efficienttrain: Exploring generalized curriculum learning for training visual backbones. In ICCV, 2023

[b] Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk. Generalization on the unseen, logic reasoning and degree curriculum. In ICML, 2023

W2: In our paper we for the first time propose an adaptive evidential bi-level optimized domain scheduler for the meta learning used in the OSDG task, where the existing works, i.e., MLDG and MEDIC, rely on sequential domain scheduler. Our approaches achieve consistent state-of-the-art performances across 3 different DG benchmarks where 4 different domain generalization settings are explored. In the response to the Reviewer QWTs, we provide more ablations on larger backbones, i.e., ResNet152 and ViT-base, where obvious OSCR gains are observed. All of these experimental results showcase the strong capability of our proposed method towards open-set domain generalization challenge.

In Table 9 of the appendix, we conducted ablation experiments to justify the effectiveness of our model with more variants of domain scheduler for meta learning on OSDG task.

W3: We clarify that the adaptive domain scheduler is our major contribution, which is proposed by us in this work to estimate the uncertainty level of different source domains during the meta-learning procedure using an evidentially bi-level optimized follower network. Unlike curriculum learning (e.g., [a] and [b]), which focuses on scheduling the training order of different samples or batches, we for the first time propose the adaptive domain scheduler to achieve adaptive reservation of different source domains for meta-learning approaches within the realm of OSDG task. We demonstrate that the proposed domain scheduler is much more powerful compared with other curriculum learning strategies as proved by the experimental results in Table 10 in the appendix.

W4: Domain reliability is based on the statistic mean of the uncertainty estimation of the samples from one specific domain, estimated by the follower network. The prediction generated by the bi-level optimized follower network is used as uncertainty quantification of the corresponding input sample, since this follower network is optimized by the evidentially learned uncertainty score from the main network. From our experiments, we observe that the data partition based on the most unreliable domain, where the uncertainty estimation of this source domain from follower network is the highest, benefit the OSDG performances. In Table 9 of the appendix, we provide further ablation between the domain scheduler achieved by bi-level optimized follower and the uncertainty estimation of the main network. Our method shows better performances.

W5: Evidential learning is a well-established technique to pursue better uncertainty estimation of the deep learning methods. However, as mentioned in related work [c], evidential learning faces with challenge of overfitting. In this work, we propose to use max rebiased discrepancy to force the two rebiased layers to learn different cues to overcome this challenge, where the max rebiased discrepancy serves as additional regularization for the evidential learning of the main network. After the supervision of the max rebiased discrepancy evidential loss, the main network can supervise the follower network well by providing more reliable estimated uncertainty.

In Table 7 in the appendix, we conducted experiments to illustrate the benefits brought by the max rebiased discrepancy regularization term, where the discussion of this ablation is demonstrated in Section B.1 in our appendix. Our observation turns out that by using the proposed max rebiased discrepancy regularization, the follower network can be optimized better and more reliable in the domain reliability quantification, thereby the proposed method can achieve higher OSDG performances.

[c] Danruo Deng, Guangyong Chen, Yang Yu, Furui Liu, and Pheng-Ann Heng. Uncertainty estimation by fisher information-based evidential deep learning. In ICML, 2023.

W6: The corresponding typos will be changed in our final version, thank you.

Thank you for recognizing the motivation and contribution of the proposed method.

W1: The anonymous link to the source code has been submitted in the official comment to AC which can be helpful for the results reproduction. The source code will be released publicly in the final version.

W2: We have conducted additional 24 experiments on the ViT-base model according to your suggestion in Table A of this response. The results can be found below, where our proposed method is compared with the MEDIC and other challenging baselines. We can observe that our proposed method achieves state-of-the-art performances when we use ViT-base model as the feature extraction backbone.

Compared with the MEDIC method, our EBil-HaDS achieves 1.69% and 2.05% performance improvements in terms of OSCR for standard classification head and binary classification head, demonstrating the capability of the proposed adaptive domain scheduler on the generalization to different feature extraction backbones. The corresponding experiments and discussion will be added to the final version of our paper.

Table A: Ablation for ViT base backbone

Dear Reviewer euSj,

Thank you very much for your comments and your contribution to the review procedure.

The authors would like to ask if you have any question on the rebuttal, since the discussion period between the authors and the reviewers will end soon.

We provided the source code link and the additional 24 ablation experiments on ViT-Base according to your instruction in the rebuttal. Please let us know if you have any further questions.

Thank you.

Best,

The authors.

Thank you for recognizing the motivation and contribution of our proposed method. We have conducted an additional 48 experiments using the ResNet152 and ViT base models. The results, detailed below, compare our proposed method with MEDIC and other challenging baselines such as ARPL, MLDG, SWAD, and ODG-Net. From Tables 1 and 2 in our main paper, we observe a significant OSDG performance improvement when increasing the complexity of the feature extraction backbone. However, when comparing Table A (rebuttal) with Table 2 (main paper), some baseline approaches trained with the ResNet152 backbone show performance decay on the major evaluation metric, OSCR.

This observation indicates that most OSDG methods face overfitting issues with larger backbones, which is particularly problematic for recognizing samples from unseen categories in unseen domains. Specifically, the MEDIC-bcls approach shows a 6.48% performance degradation on OSCR when switching from ResNet50 to ResNet152. In contrast, using the proposed BHiL-HaDS for more reasonable task reservation in the meta-learning procedure, BHiL-HaDS-bcls achieves 86.25% OSCR with the ResNet152 backbone, compared to 86.12% OSCR with the ResNet50 backbone. This demonstrates that our adaptive domain scheduler can effectively optimize meta-learning for large complex models by reserving appropriate tasks for model optimization.

Table A: Ablation for ResNet152 backbone

Additional experiments on the ViT-base model (patch size 16 and window size 224) are presented in Table B of this response. We observe that our proposed method consistently delivers performance gains on the transformer architecture. These ablation experiments and discussions will be included in our final paper.

Table B: Ablation for ViT base backbone

Thank you for acknowledging the motivation behind our work and the effort put into the review. We will address the mentioned formatting and citation issues in our final version submission. The source code has been shared in the official comment to the AC, including a sample for result reproduction via an anonymous GitHub link. We hope the provided source code will enhance the contribution of our paper. The included model weight is for the PACS dataset, with the cartoon domain as the target and ResNet18 as the feature extraction backbone. The code will be made publicly available in the final version.