Towards Unsupervised Model Selection for Domain Adaptive Object Detection

Thank the Reviewer XtC2 for your constructive comments and valuable feedback. We appreciate the recognition of our work on motivation and perspective. Now we answer the question raised by the reviewer as follows.

Q1: It seems that the proposed method is only tested on the Faster RCNN baseline.

A1: We thank the reviewer for the valuable comments. To further verify the effectiveness of the proposed DAS method on other detectors, we conduct experiments on SIGMA++[4] (FCOS-based) and AQT[2] (DETR-based). The results on Cityscapes to Foggy Cityscapes are shown below table. It is shown that the proposed method also works with the FCOS-based, and DETR-based methods. In particular, our method chooses a SIGMA++ checkpoint with 41.7% map while the last checkpoint is 39.5% mAP, demonstrating the adaptability of the proposed method on other detectors. Although DAS gives a moderate gain compared with the last checkpoint of AQT, our method selected the optimal checkpoint. We will evaluate our method on other detectors in future research.

Q2: Could the authors explain why DAS does not give obvious gains in some settings? For example, DAS gives no gains for CMT on Real-to-Art Adaptation in Table 1.

A2: The main reason lies in that the performance gap between the last checkpoint and Oracle results is small, so give our method relatively small improvement space. For example, the average performance gap of AT over three settings is $8.68$, while that of MT over three settings is $1.28$. Nevertheless, the proposed method can select better checkpoints in most cases and at least the last checkpoints in very few cases.

Q3: Can the method be used for fine-grained tasks, such as segmentation?

A3: Yes, our method can be used for fine-grained tasks. To verify this, we conduct the experiment on the semantic segmentation. In particular, we evaluate the well-known domain adaptive semantic segmentation approach DAFormer[5]. The qualitative results on GTA5->Cityscapes are shown below. It showes that our DAS method can choose a better checkpoint of the model with an mIoU of 64.2% while the last checkpoint of the model only achieves 60.9%. This demonstrates that our DAS method still works effectively on domain adaptive semantic segmentation tasks.

Q4: Consider giving a more comprehensive review of some advanced cross-domain object detection works [1-4].

A4: Thanks for the valuable feedback. We will enhance the final paper by providing a more comprehensive review of some advanced cross-domain object detection works.

[1] Deng, J., Xu, D., Li, W., & Duan, L. (2023). Harmonious teacher for cross-domain object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23829-23838).

[2] Huang, W. J., Lu, Y. L., Lin, S. Y., Xie, Y., & Lin, Y. Y. (2022, July). AQT: Adversarial Query Transformers for Domain Adaptive Object Detection. In IJCAI (pp. 972-979).

[3] Zhou, W., Fan, H., Luo, T., & Zhang, L. (2023). Unsupervised domain adaptive detection with network stability analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6986-6995).

[4] Li, W., Liu, X., & Yuan, Y. (2023). Sigma++: Improved semantic-complete graph matching for domain adaptive object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7), 9022-9040.

[5] Hoyer L, Dai D, Van Gool L. Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 9924-9935.

Thank Reviewer 9Lzv for the constructive comments. We appreciate the positive comments including clear presentation, rational strategy, and effective experiments. Following are the responses to the reviewer's concerns.

Q1: This paper may neglect a fact that in real application, target labels are often known for testing the provided model, and in fact the training labels may not be accessible. So, I do not think this strategy is really necessary.

A1: The goal of unsupervised domain adaptation (UDA) is to solve the scenario where the target domain labels are inaccessible, which are aligned with the real-world application. However, how to evaluate the UDA model is a long-standing problem in the UDA research community. The researcher in the UDA community uses the target domain label to select models, which is inevitable to overfit the test set. Meanwhile, it violates the assumption of unavailability of the target domain labels. The unsupervised model selection of UDA is also discussed in many previous works [16,51]. In our paper, we are the first attempt to address the unsupervised model selection for domain adaptive object detection. Moreover, because the annotation of object detection is hard to obtain, our method is meaningful for the real-world application. We believe our work would inspire researchers in the community and push the real-world application of DAOD models forward.

Q2: This paper is essentially a DG domain generalization task, where target domain is unseen. So we can only select (train) a better model by using existing source domain data. Flat minima has also been used in DG method to instruct how to train a domain generalization model.

A2: Our paper is intrinsically different from the DG task. Domain generalization addresses the issue where the target domain is not accessible, while our paper addresses the model selection method for DAOD. The flat minima is a general concept in machine learning and has been used in many fields. Different fields have different estimation methods. In domain generalization, flat minima are estimated using the source domain labels. However, the target domain labels are not available for domain adaptation. Therefore, we propose a novel method via a generalization bound to obtain the flat minima without using target labels to address the unsupervised model selection problem for DAOD. This is essentially different from the DG task.

Q3: Experiments on other related tasks.

A3: We thank the reviewer for the valuable feedback. To further verify the effectiveness of the proposed method, we conduct the experiments on the semantic segmentation. In particular, we evaluate the well-known domain adaptive semantic segmentation approach DAFormer. The qualitative results are shown below. It is shown that our DAS method can choose a better checkpoint of the model with an mIoU of 64.2% while the last checkpoint of the model only achieves 60.9%. This demonstrates that our DAS method still works effectively on domain adaptive semantic segmentation task. We will evaluate our method on more tasks in future research.

Q4: In experimental comparison, the authors compare the last checkpoint, which may not be fair.

A4: Due to the unavailability of target labels, choosing the last checkpoint without a specifically designed model selection is a good choice. To this end, we choose the last checkpoint as a baseline. We also compare with other model selection methods in Table 2 in the manuscript to demonstrate the effectiveness of the proposed method.

Q5: Lacking comparisons to many SOTA DAOD models.

A5: Thank the reviewer for the comments. Our method is orthogonal to existing DAOD models. To evaluate the effectiveness of the proposed method, we evaluate our method on classic DAOD methods. To further verify the generalization of our method on SOTA DAOD models, we evaluate the model on SIGMA++. The results are shown in the below table. We can see that the proposed method also works with SIGMA++ where we choose a checkpoint with $41.7$ AP, while the last checkpoint is $39.5$. We will include the new results in our final paper. We will evaluate our method on more SOTA DAOD models in future research.

Q6: What is the computational overhead?

A6: In the implementation, the calculation of DAS needs only two inferences for each target domain sample and one inference for each source domain sample. Thus one checkpoint requires just a few minutes to obtain its DAS. For example, we evaluate one model like AT using 2 minutes (on RTX 3090) while the model training time is usually around 30 hours. Therefore, the computational overhead is small compared with the model training. Once a model is selected by our method, the computational overhead is the same as the original detector.

(1) In real application, for selecting or testing a model, some labeled target data from one or more target domains should be tested to guarantee the reliability of the deployed model. From this view, the proposed work may be useless.

I agree with you that target domain labels are necessary to give any kind of guarantees on the target domain. However, this might simply not be feasible, for example if the target domain is continuously shifting faster than labels can reasonably be obtained. In this setting, this method would select a best-effort model without any kind of guarantees, which is still useful for many real applications. I don't feel like the authors overclaim by suggesting guaranteed reliability on the target dataset.

(2) From another view, if you think in real application the selected model "must be" reliable without need of any labeled target domain data, I think the current evaluation experiments are very insufficient. At least, you should test your selected model on more target domain data to make sure if your selected model is really reliable in real application even without using any labeled target data.

I'm not sure I understand how one would test the selected model on more target domain data. I feel like evaluating mAP on the target datasets, like the authors did, is the best one can do given the limited scope of existing UDA benchmark datasets.

Thank the Reviewer jxDH for your insightful comments. We appreciate the positive comments regarding the writing, method, and performance. Now we answer the question raised by the reviewer as follows.

Q1: Does the proposed method work well in other UDA tasks as well?

A1 Yes. Our method can also work in other UDA tasks. To verify this, we conduct experiments on semantic segmentation and evaluate the well-known DAFormer method on GTA-5 to Cityscapes. The experimental results in terms of mIoU are shown below. It is shown that our DAS score can choose a better checkpoint of the model with an average mIoU of 64.2% while the last checkpoint of the model only achieves 60.9%. This demonstrates that our DAS method can also work effectively on other UDA tasks.

Q2: Is it possible to derive the mathematical relationship between the difference in risks and the variance in outputs?

A2: The variance in outputs serves as an upper bound for the difference in risks. To establish this relationship, let us define the error as $\mathcal{E} _ {h}(o _ {h}, g) = \left| o_ {h}-g \right|$, where $o _ {h}$ is the output of the network $h$ and $g$ is the ground truth. Similarly, error of network $h'$ is denoted as $\mathcal{E} _ {h'}(o _ {h'}, g) = \left| o_ {h'}-g \right|$. By applying the triangle inequality, we derive:

$$\left|\mathcal{E} _ {h}(o_{h}, g) - \mathcal{E} _ {h'}(o _ {h'}, g) \right| = \left|\left| o_{h} - g \right| + \left| o _ {h'} - g \right| \right| \leq \left| o _ {h} - o _ {h'} \right|$$

Consequently, minimizing the variance in outputs effectively constrains the difference in risks, providing a mechanism to bound the variability in predictions across networks.

We appreciate your insightful comments and will delve further into these mathematical underpinnings in our revised manuscript to illustrate the implications of variance in outputs on risk difference.

Q3：Why do better transferability and discriminability lead to flat minima?

A3: This is a good question. I would like to gently point out that variance in outputs primarily indicates flatness. If the labels are available, we can use the labels to calculate the error on the target domain so as to ensure the property of minima. Without using target labels, we estimate the generalization error on the target domain through a domain adaptation theory, where the target error is bounded by source error, domain distance, and a constant term. The transferability and discriminability are widely used to improve the performance on the target domain. With the both properties of flatness and minima on the target domain, we can obtain the flat minima without using target labels. Thanks for your valuable feedback, we will add more explanations to make it clearer in the final paper.

Q4: The correlations between each of FIS and PDR and flatness with GT labels.

A4: Thank the reviewer for pointing this out. We provide an experiment to verify the positive correlation between our FIS and PDR and flatness with GT labels. In particular, we show the correlation coefficient between them on AT weather adaptation. It shows that the proposed FIS is well correlated with the GT flatness with a coefficient of 0.64, demonstrating that our DAS score can be an appropriate estimation for flatness with GT labels. However, PDR represents the target error by assessing the transferability and discriminability thus it doesn't well correlate with the flatness of GT labels, as the coefficient is 0.45. It is by combining FIS and PDR that we could find the flat minima without accessing target labels while achieving accurate model selection for DAOD models.

Dear Reviewer jxDH,

We hope this message finds you well. As the discussion period is ending soon, we would like to bring to your attention that we have submitted our response to your questions. We carefully consider your valuable feedback and suggestions, and hope the response well addressed your concerns.

We are truly thankful for the comments on our work. If you find our responses to be satisfactory, we would greatly appreciate it if you could take this into account when considering the final score.

Best regards,

Authors of Submission 7161

Thank Reviewer 4YNt for constructive comments and valuable feedback. We appreciate the recognition of our work including the support for the novelty, structure, and evaluation. Now, we address the reviewer's concerns as follows.

Q1: The writing quality should be improved.

A1:We sincerely thank your constructive comments. We have corrected the grammar issues, typos, and inconsistent terminology. We also carefully proofread the paper.

Q2: Apart from writing quality, my only significant concern is the apparent invariance of DAS performance towards lambda, which raises the question of whether the flatness index score and prototypical distance ratio measure different aspects in practice and the degree to which they are complementary.

A2: From the generalization bound theory for flat minima present in the paper, the FIS and PDR measure the flatness and the minimum, thus they are theoretically evaluating the models in two aspects. In Table 5, the selected model mAP doesn't vary too much with different $\lambda$ setting. This might have contributed to when the model convergences to some degree, the two terms FIS and PDR represent have been both optimized well. Thus they both show relatively stable results on the mAP. This is why we use PCC between DAS and the model to measure the subtle differences. On the other hand, the PCC has obvious changes with the variation of $\lambda$. This indicates the $\lambda$ indeed changes the DAS and influences the correlation between the proposed DAS and ground truth performance. To examine the complementary, we have conducted an ablation study in Table 4 from the main paper. It is shown that FIS only achieves $0.48$ PCC. With the help of PDR, our DAS improved the PCC from $0.48$ to $0.67$, demonstrating the complementary between FIS and PDR.

Q3: How is PDR or d_inter defined for single-class benchmarks such as Sim10k?

A3: Although Sim10k only has a single class, the object detection model also considers the background class, thus the PDR or d_inter can be calculated. We will add more descriptions in the final paper to reduce the confusion.

Q4: DAS seems to work consistently very well for AT, while performance with other frameworks is less consistent (very apparent in Figure 4). Do you know why this might be the case?

A4: The main reason lies in that the performance gap of other frameworks between the last checkpoint and Oracle results is narrow, so it gives our method a relatively small improvement space. For example, the average performance gap of AT over three settings is 8.68%, while that of MT over three settings is 1.28%. In particular, a narrow performance gap gives less tolerance for potential uncertainty of the DAS estimation. Therefore, the improvements with other frameworks show less consistent improvement like AT. Nevertheless, the proposed method can still select better checkpoints in most cases and checkpoints that are at least comparable with last checkpoints in very few cases.

Q5: It might make sense to include the Oracle checkpoint in the main evaluation, especially tables 1-3. This would make it easier to see how much of the gap to the oracle is closed.

A5: We thank your constructive comments. We will add the orcale checkpoint in tables 1-3. We present Tables A1 and A2 in the attached PDF in the rebuttal to show the added version of Tables 1 and 2 in the main manuscript.

Q6: Please clarify what is meant by "ES" in table 2.

A6: ES[40] indicates the entropy score, i.e., using the entropy of the prediction from the classification branch to select the checkpoints. We have clarified this in the experiment section.

Q7: It would be interesting to see analogues for Figures 2 and 5 for Cityscapes and Sim10k in the appendix.

A7: We provide more examples on Sim10k to Cityscapes and Cityscapes to Foggy Citysacpes adaptation in Figures A1 and A2 in the attached PDF. Similar to Figure 2 in the main paper, the proposed DAS score correlates well with the performance of DAOD checkpoints compared with other baselines.

We thank Reviewer ood2 for the valuable feedback and insightful comments. We appreciate the reviewer's positive comments regarding motivation and method. We now clarify the reviewer's concerns as follows.

Q1: Explain why these methods fail to evaluate the object detection model in line49.

A1: The methods referenced in line 49 rely on classifier-specific properties like predicted confidence and entropy, which are tailored for classification tasks. In object detection, however, the evaluation involves not only classification but also the precise localization of objects within an image. This crucial distinction renders these methods ineffective in fully assessing an OD model. By incorporating your suggestion into our paper, we will explicitly clarify this difference between classification-centric evaluation methods and the multifaceted demands of OD evaluation.

Q2: In Eq.2, since KL term and IoU term are two metrics to evaluate the matching costs of two different tasks, object classification and box regression, why there is no a balance coefficient, such as d=KL-\lambda IoU. If \lambda=1 is the best?

A2: Thank you for regarding the potential absence of a balance coefficient in Eq. (2) to weigh the KL divergence and IoU metrics. We do not deliberately add a balance coefficient following the previous works (e.g., DETR) to combine the classification and localization cost. As suggested by the reviewer, we conducted experiments on AT weather adaptation to assess the impact of tuning the balance coefficient. The results are shown in the table below. Our findings suggest that the parameter adjustment can actually influence performance, while setting the balance coefficient to 1 is a fairly good choice, indicating a balanced consideration of classification and localization within our framework.

Q3: The proposed FIS is easy coming to mind, which uses parameter perturbation to evaluate flatness minima without using target labels. And the proposed PDR to use prototype-based domain alignment is also used as a training method in the existing prototype-based domain alignment for DAOD, such as [1].

A3: We would like to highlight our paper is essentially different from previous works. We address the unsupervised model selection (UMS) problem for DAOD. Previous works [1] design effective methods for addressing the domain gap for object detection, while we address the UMS problem for DAOD through a flat minima principle, i.e., models that locate the flat minima region in the parameter space usually exhibit excellent generalization ability. Different from the traditional flat minima methods that require labeled data to evaluate how well a model is located at the flat minima region, we show via the generalization bound that the flatness can be deemed as model variance, while the minima depends on the domain distribution distance for the DAOD task. To this end, although some strategies in our paper may seem to be similar to existing works, the task and motivation are distinct from the existing works.

Q4: If these kinds of methods are used for unsupervised training, can they be used as unsupervised evaluation methods as the same time? For example, if the benchmark DAOD framework is a prototype-based domain alignment method, can PDR, a prototype-based domain alignment evaluation method, serves as an unsupervised evaluation?

A4: To effectively evaluate the DAOD models, we designed a DAS including FIS and PDR, which is derived from a generalization bounding for flat minima. Thus, they evaluate the DAOD models in different aspects. Additionally, existing work [1] use the prototype-based alignment method to minimize the domain gap between domains. The prototype estimation in existing work only utilizes the samples in a mini-batch during the model training. In contrast, we use the entire dataset to estimate the prototypes, which is more robust and has better generalization ability. To verify this, we conduct experiment results on Sim10k to Cityscapes (S2C) for [1]. The experimental results are presented below. It is shown that the proposed method still works when DAOD frameworks also optimize the prototype-based distance, i.e., choosing a checkpoint with 44.8% mAP (our PDR) while the last checkpoint is 43.1% mAP. Moreover, our DAS with FIS and PDR can further improve the performance, demonstrating the effectiveness of the FIS. The FIS is proposed by us in this paper, and we have not found any DAOD methods using it as a training method.

Q5: As I can see in line 261, this paper only uses adversarial training or self-training methods for unsupervised domain adaptive training?

A5: We evaluate DAOD methods with adversarial training or self-training because they are classical paradigms and achieve promising results in DAOD scenarios. To demonstrate the generalization of the proposed DAS method to other DAOD frameworks, we evaluate our method on GPA[1] and SIGMA++[2] which minimize the domain gap by prototype-based domain alignment and graph matching, respectively. The results are shown in the below table, we can see that our method is able to select better checkpoints for those two DAOD methods (2.7% and 2.2% improvements, respectively) beyond adversarial learning and self-training paradigms.

• We attempted but could not reproduce the GPA results on C2F using the released code. This problem is found by other researchers in the issue of the repo. Thus we choose the setting of S2C.

[1] Cross-domain detection via graph-induced prototype alignment. CVPR 2020.

[2] SIGMA++: Improved semantic-complete graph matching for domain adaptive object detection. TPAMI 2023.

Dear Reviewer ood2,

We hope this message finds you well. As the discussion period is ending soon, we would like to bring to your attention that we have submitted our response to your questions. We carefully consider your valuable feedback and suggestions, and hope the response well addressed your concerns.

We are truly thankful for the comments on our work. If you find our responses to be satisfactory, we would greatly appreciate it if you could take this into account when considering the final score.

Best regards,

Authors of Submission 7161