AudoFormer: An Efficient Transformer with Consistent Auxiliary Domain for Source-free Domain Adaptation

We apologize for any confusion caused. 1). About novelty. First, most current frameworks of SFDA are mainly based on the traditional architecture, i.e., Resnet series, and very few studies use the Transformer as the backbone for the research. We propose a variant of Transformer with an auxiliary domain that alleviates the dilemma of SFDA without a source domain to some extent. We solve this task from a different aspect by domain consistency which is first proposed in our work and is different from other approaches. Besides, we exploit the MMD with conditional guided information (i.e., CMK-MMD) to improve the effect of the alignment. By Tab 5 in Appendix B, CMK-MMD can further improve the effect of the experiment. 2). About the ablation experiment. The ablation studies about the effect of the three modules are offered in Appendix B. Reviewers can check them in our attachments.

Dear reviewer. I kindly inquire whether our response has successfully addressed your uncertainties. We remain dedicated to addressing concerns you may possess with utmost eagerness. Additionally, we would be encouraged if the reviewer considers raising the score.

Domain consistency is essentially the use of multi-domain consistency to judge invariance features, and our initial inspiration comes from the domain adaptation (DA) accessible to the source domain. Therefore, we construct the source domain in the way of an auxiliary domain and hope that it can play the role of source domain. Compared to the traditional approaches, our method has more feature information to use, and the category centroid with consistency discrimination has higher confidence, while those methods that rely solely on entropy levels are more susceptible to noise.

About related work, we thank the reviewer for the valuable feedback and thoughtful comments. In the original version, our paper provided an introduction to related work. But we have made some selections in main paper due to space constraints. In the revised version, we will attach a comprehensive introduction of the relevant work.

Thanks to the author's detailed comments, for "category centroid" and "category dynamic centroid", we will reset the colors in our revised version. Besides, the colors of data flow and loss are actually different, but for clarity, we will also make further changes. For the other typos in our paper, we will carefully review and revise in the revision. Thanks again to the reviewer.

Regarding the proof of the effectiveness of the individual strategies presented in the paper, we have given in Appendix B. Reviewer can check our supplementary material. In addition, to further validate the effect of our proposed dynamic centroids (Dyn. centroids) and the conventional static centroids (Sta. centroids). We also added the following experiments:

As can be seen from the Tab., we consider that Dyn. centroid can evaluate more reasonable center of the samples, and thus achieve better results on three datasets.

At the time we conducted such a study, the most recent work on 2023 was not available yet. We thank the reviewers for the reminder and we will add relevant comparisons in revised versions. In our Appendix, we offered the analysis of the ADM and other three proposed components, we will further add some evaluation. We are in the process of adding related experiments about Domain-Net. Since it takes some time, once the results are available we will provide them to the reviewers.

Dear reviewer. I kindly inquire whether our response has successfully addressed your uncertainties. We remain dedicated to addressing concerns you may possess with utmost eagerness. Additionally, we would be encouraged if the reviewer considers raising the score.

In Appendix B of supplementary material, we provide ablation studies about the performance of ADM component on source domains. With Tab 4, we can find that AudoFormer can effectively enhance supervised learning and domain adaptation on source domains. For main strategies proposed in our paper, we show its effect in Tab 5.

Thank you for your comments, if I understand you correctly, 2.3.2 and 2.3.3 should be in the same section, cause no section 2.2.3 in our article. In 2.3, we exploit multiple consistency strategies, i.e., first dynamically evaluating the centroids of categories, second evaluating the easy samples by consistent labels, then evaluating the hard samples by consistent neighbors, and at last, training the model by consistent self-supervision. Thus, these strategies are juxtaposed in importance and progressive in process.

MMD can align the feature embedding of hard samples to easy samples in the reproducing kernel Hilbert space (RKHS), which is a different way from supervised learning of pseudo-labels in common space. According to the experimental results in Table 5 of Appendix B, CMK-MMD can further improve the effect of the experiment. Therefore, a combination of these two approaches is beneficial in enhancing the effectiveness of the model.

Dear reviewer. I kindly inquire whether our response has successfully addressed your uncertainties. We remain dedicated to addressing concerns you may possess with utmost eagerness. Additionally, we would be encouraged if the reviewer considers raising the score.

First, we solve the SFDA problem by exploiting a variant of Transformer with an auxiliary domain that alleviates the dilemma of SFDA without a source domain to some extent. Second, we solve the invariable feature representation from a new perspective by domain consistency which hasn't been studied by anyone else. Besides, we exploit the MMD with conditional guided information (i.e., CMK-MMD) to further improve the effect of the alignment.

In [1], DaC mainly exploits contrastive learning to generate robust class centroids, which is different from the way of our domain consistency. Consistent neighbors in our approach are just a strategy that is used to identify difficult samples, while in [2], it is the key approach to evaluate the centroids.

We apologize for any confusion caused. About the sentence of ("Intuitively, if ...... layer features.") , what we're trying to convey is this current mainstream approach is based on invariant features, however, these invariant features are relatively difficult to build in the SFDA. The "Intuitively" is way of our hypotheses and judgment, which prompts the presentation of the methodology and subsequent experimental verification. We will carefully scrutinize our sentences in the revised version.

First, SFDA-DE is not open source, we have tried to reproduce it, but the performance is not as good as announced, probably due to some unknown settings. Therefore, for fairness, our AudoFormer does not reimplement and compare with it. Second, TransDA itself is already based on the Transformer architecture, so there is no need to reuse our method to implement it. Besides, about DomainNet, similar to the setting of [1, 3, 4]，we also select four domains (i.e., Real, Clipart, Painting and Sketch) with 126 classes as the single-source unsupervised domain adaptation benchmark and construct seven single-source adaptation scenarios from the selected four domains. As can been from the following table, our approach is still able to achieve advantages on the large dataset DomainNet.

[1]. Zhang Z, Chen W, Cheng H, et al. Divide and contrast: Source-free domain adaptation via adaptive contrastive learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 5137-5149.

[3]. Saito K, Kim D, Sclaroff S, et al. Semi-supervised domain adaptation via minimax entropy[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 8050-8058.

[4]. Li J, Li G, Shi Y, et al. Cross-domain adaptive clustering for semi-supervised domain adaptation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 2505-2514.

First, MMD is widely used in DA tasks and has been proven to work well. Second, most approaches ignore conditional information which can also be used to guide learning. Thus, we proposed to exploit CMK-MMD such a variant. Besides, MMD is commonly used to align source and target domains in scenarios where the source domain is accessible. In the SFDA scenario, it is difficult to use MMD for alignment when the source domain is inaccessible. We can exploit this approach due to our auxiliary domain module can evaluate source-like samples. We are adding some comparative experiments that will be given soon. Thanks for your reminder, we will further add descriptions of CMK-MMD in the revised version.

Dear reviewer. I kindly inquire whether our response has successfully addressed your uncertainties. We remain dedicated to addressing concerns you may possess with utmost eagerness. Additionally, we would be encouraged if the reviewer considers raising the score.