Seeking Flat Minima with Mean Teacher on Semi- and Weakly-Supervised Domain Generalization for Object Detection

Limited novelty

The response to this point has been addressed in our reply to Reviewer LbNL. We kindly ask Reviewer Poom to refer to that section.

The authors could provide statistical significance tests for the performance differences and discuss potential reasons why their method outperforms DGOD and Oracle

Because it takes nearly one day with four A100GPUs to train the detector, it is not feasible to perform a statistically significant number of trials for model training. Instead, in this rebuttal, we trained the models with three different random seeds, as shown in Table below. Note that we fixed the random seed to seed1 throughout the previous experiments in our paper. GaussianFRCNN+EMA+PL+Regul. slightly outperforms DGOD and Oracle across all random seeds.

The potential reason is that DGOD and Oracle were trapped in sharp local minima due to simple supervised learning (i.e., ERM), while GaussianFRCNN+EMA+PL+Regul. reached flat minima. It has been shown that even when both the train and test sets are from the same domain, there is a slight shift between the train loss and test loss, and falling into a sharp valley can decrease performance [A]. Fig. 9 in Appendix A.5 of the revised manuscript shows the comparison of the flatness similar to that in Sec. 7.5. We observe that GaussianFRCNN+EMA+PL+Regul. on WS-DGOD achieved a flatter solution than DGOD and Oracle.

[A] Averaging Weights Leads to Wider Optima and Better Generalization, Izmailov et al., UAI, 2018.

The SSDG setting of this paper is more similar to [1]

We agree with this comment. As discussed in Appendix D.1, there are two types of settings in semi-supervised domain generalization. The first setting assumes that only a portion of the samples in each domain are labeled, similar to the previous works listed in Section 3.3. The second setting assumes that only a portion of the source domains are labeled, as in [1]. In this paper, we followed the previous SS-DOGD work (i.e., CDDMSL) and tackled the second setting. We have added the citation of [1] and clarified this point in Sec. 2 in the revised manuscript. Tackling the first setting is part of our future work.

The interpretations of model generalization from the perspective of flat minima had appeared in many previous works, such as [2,3], to name a few.

As pointed out, the flat minima theory that explains the model generalization from the perspective of flat minima had appeared in previous works. However, different from the flat minima theory itself, we provide novel findings that the Mean Teacher learning leads to flat minima and the interpretation of the reasoning behind it, which are novel and our main contribution. On the basis of the flat minima theory and our interpretations, we explained the reason of the detectors trained with the Mean Teacher learning framework achieve robustness to unseen test domains.

Had you ever tried other existing tricks to improve flat minima, such as Sharpness-Aware Minimization (SAM) regularization?

In this rebuttal, we conducted the experiments to compare with SAM. The results are shown in Table 12 in Appendix A.9 of the revised manuscript. By comparing GaussianFRCNN+EMA+PL+SAM and GaussianFRCNN+EMA+PL+Regul., our regularization outperforms the SAM. In addition, since the SAM is an optimizer and can be used instead of SGD, it is compatible with our regularization. We can see that GaussianFRCNN+EMA+PL+Regul.+SAM achieved the best performance.

Had you ever tried any other Semi-Supervised or Weakly-Supervised learning methods in the proposed two settings, SSDGOD and WSDGOD?

Although there are previous semi-supervised domain generalization methods proposed for image classification, extending them to object detection is non-trivial and they cannot be directly applied to SS-DGOD. Therefore, we compared our method with CDDMSL in our experiments, which is the previous work specifically tailored for SS-DGOD.

Regarding weakly-supervised domain generalization, there are no previous methods, even for the image classification task, because weak labels cannot be defined in image classification

The proposed method lacks significant novelty

As written in Sec. 1, it is noteworthy that our aim is not to propose an entirely new method or surpass the state-of-the-art methods. Instead, our contributions are summarized as follows:

• We show that object detectors can be effectively trained on the SS-DGOD and WS-DGOD settings with the same Mean Teacher learning framework.
• We provide interpretations of why the detectors trained with the Mean Teacher learning framework achieve robustness to unseen test domains in terms of the flatness of minima in parameter space, based on our novel finding that the Mean Teacher learning framework leads to flat minima.
• On the basis of the interpretations, we introduce a simple regularization method into the Mean Teacher learning framework to achieve flatter minima.
• We are the first to tackle the WS-DGOD setting.

As Reviewer LbNL pointed out, regularizing the two networks to produce more similar outputs is a widely used framework. However, we have revealed why this framework works well and improves performance based on our novel interpretations, which is our main contribution. Since this framework is widely used, our novel interpretations are likely to have a broad impact.

As Reviewer poom pointed out, each of the Mean Teacher and flat minima theory (the relationship between the model generalization ability and flatness of the solution in loss landscapes) are already established concepts, separately. Different from them, our main contribution lies in the novel finding that the Mean Teacher framework leads to the flat minima and the interpretation of the reasoning behind it. On the basis of the flat minima theory and our interpretations, we explained the reason of the detectors trained with the Mean Teacher learning framework achieve robustness to unseen test domains. Because the Mean Teacher has been used across various tasks, our novel interpretation is likely to have a broad impact across a wide range of tasks.

What is the difference between recent mean-teacher based semi-supervised methods and the proposed method?

Although the technical novelty of the regularization is not our main contribution, there are technical differences from recent Mean Teacher-based methods. For example, unlike Harmonious Teacher [Deng, CVPR 2023], which regularizes the consistency between the classification and localization scores, our regularization encourages consistency between the raw outputs from the teacher and student. Unlike SSDA-YOLO [Zhou, CVIU 2023], we use the raw output without post-processing to produce more similar outputs. More importantly, our contribution lies in providing a new interpretation of why such consistency enhances robustness to unseen domains.

Thank you for your response. Despite the clarifications and additional explanations provided by the authors, I still believe that the major issues with the paper have not been adequately addressed, and therefore, I am maintaining my original score of 5 at this point. Below, I provide a detailed explanation of my reasoning.

• Lack of Technical Novelty: The authors have emphasized that their goal is not to introduce a new method or surpass state-of-the-art results but rather to offer an interpretation of the effectiveness of the Mean Teacher learning framework in SS-DGOD and WS-DGOD settings. However, I remain unconvinced that this interpretation provides a sufficiently novel contribution. The conclusion that "finding flat minima is advantageous for domain generalization" is already well-documented in the literature, and the connection between flat minima and generalization is well-established. While I understand the authors' intent to explore why Mean Teacher leads to flat minima, I am skeptical about whether this provides substantial new insight beyond the existing theoretical framework. The contribution seems largely limited to showing that this known effect also holds in the specific context of SS-DGOD and WS-DGOD, which may not be broadly impactful.

• Limited Differentiation from Existing Methods: The authors have explained the technical differences between their approach and existing Mean Teacher-based methods, such as Harmonious Teacher and SSDA-YOLO. However, I still find these differences to be incremental rather than novel. For example, the modification of "omitting post-processing and using raw outputs" is not convincingly shown to be a decisive factor in improving performance. The proposed regularization strategy does not seem significantly distinct from existing consistency strategies to be considered an innovative advancement. I acknowledge that the authors argue that their primary contribution is the interpretation rather than the technique, but this makes the technical novelty less compelling as an independent contribution.

If there are any key points I may have misunderstood or misrepresented about your paper or your argument, please feel free to let me know.

Comparison with CDDMSL+EMA+PL

Thank you for the insightful comment. However, integrating the Mean Teacher framework into CDDMSL to train "CDDMSL+EMA+PL" is challenging due to CDDMSL's already complex network structure. It includes a detection backbone, a RegionCLIP backbone, a ClipCap-based vision-to-language module, and a detection head with an RPN, along with four types of loss functions: detection loss (classification and regression for the detection head and RPN), distillation loss via the v2l module, and instance- and image-level contrastive losses through the projection head and v2l module. Adding the Mean Teacher framework requires duplicating parts of these networks, at least the detection backbone and head, and deciding which parts to duplicate is not straightforward. Additionally, we need to introduce an unsupervised loss and fine-tune the hyperparameters for Mean Teacher. This complexity makes integrating the Mean Teacher framework into CDDMSL difficult.

It is important to reiterate that our goal is not to propose a new method that outperforms state-of-the-art methods. Therefore, instead of comparing with CDDMSL+EMA+PL, we validated the generalization ability of the regularization by conducting experiments with another detector that has a significantly different network design. In these experiments, we used a Transformer-based backbone (Swin-T) with the feature pyramid network, although the detection head was not changed. The results are shown in Table 11 in Appendix A.8 of the revised manuscript. We can see that the regularization improves the performance, which validates the generalization ability.

Does the regularization make it difficult or slow to train the model?

We have added Fig. 8 in Appendix A.4, which shows the mAP on the validation set during training. The regularization does not make the training process more difficult or slower. On the contrary, the regularization helps alleviate overfitting (i.e., less decrease in the validation mAP), stabilizing the training.

We would like to thank Reviewer ePD2 for the constructive comment. The remaining concern of Reviewer ePD2, regarding the limited improvement with the Swin-T-based detector, is due to the fact that the hyperparameters were not tuned. For the experiments in Table 11, we used the same hyperparameters as the ResNet101-based detector and did not tune them at all. As a result, the performance of the baseline detector was inherently limited, which led to the limited improvement from the regularization.

To validate this claim, we conducted additional experiments with tuned hyperparameters. Specifically, we increased the batch size from 16 to 32, as we found that this change improves the baseline performance. Additionally, we used dropout instead of drop path in the SwinTransformer blocks to stabilize the training. The results are shown below. We can see that this hyperparameter change significantly improved the baseline performance (53.0 -> 53.9), and consequently, the regularization further boosted the performance (53.9 -> 55.0). This (+1.1 mAP50) improvement demonstrates the generalization of the regularization across different types of detectors. Although this improvement is a little smaller than that in Table 2 (+1.6 mAP50), we believe that further hyperparameter tuning will lead to even greater performance gains, as we only adjusted two hyperparameters in this experiment.

Before hyperparameter-tuning (Table 11)

After hyperparameter-tuning