Neighborhood-Informed Diffusion Model for Source-Free Domain Adaptation: Retrieving Source Ground Truth from Target Query's Neighbors

We appreciate the reviewer's diligent effort in reviewing our manuscript.

Weakness 1

Response to Reviewer’s Concern on Technical Contribution:

We appreciate the reviewer's acknowledgment of the novelty in our application of diffusion models in SFDA. We would like to clarify that our aim is not to propose new diffusion models but to explore their use as domain shift correctors for aligning feature distributions across domains without source data during target adaptation.

Existing SFDA methods typically focus on enhancing unsupervised clustering (like NRC, AaD, SHOT, and G-SFDA) or using generative models for data augmentation (DaC), but they do not address quantifying domain shifts when source data is unavailable during target adaptation.

Our main contribution lies in applying neighborhood searching to parameterize the prior distribution in diffusion models. This approach aligns feature distributions across domains during adaptation without source data, ensuring a correspondence between the source knowledge retrieved from diffusion and encoded target features, a novel advancement in SFDA. Our diffusion model, distinct from its usual role in image generation, refines target features using source knowledge under latent geometry guidance, differing from typical diffusion model applications.

In summary, we have innovatively adapted diffusion models to address a fundamental problem in SFDA, effectively quantifying domain shift without source data. This unique application explores new ground in SFDA, representing a significant contribution to the field.

Weakness 2 and Question 2

Advantages Compared to Other Generative Models:

Our diffusion model, unlike traditional generative models, acts as a domain shift corrector, refining target features with source knowledge and aligning feature distributions during target adaptation, without needing source data. It does not generate new images.

Our diffusion model's key benefit is its ability to quantify domain shift during target adaptation using latent geometry, a feature not seen in other generative models where the prior distribution plays a different role. By integrating semi-supervised learning's neighborhood searching, we ensure that the target adaptation is guided by the latent geometries' similarity between the two domains, ensuring correspondence between transformed and encoded target features.

Our novel application of diffusion models not only quantifies domain shift but also aligns domains effectively when source data is absent. This significant impact of the prior distribution on the diffusion process is unique to our model and is a primary reason for choosing diffusion models over others like GANs in our research.

Comparison with The Work Listed by The Reviewer: We have added these to Related Work section of our revised manuscript, particularly focusing on the works you mentioned. This expanded section, highlighted in blue under source-free domain adaptation using generative models, offers a thorough discussion. Additionally, we have detailed the differences between our method and the referenced works below.

Comparison with [1]: [1]'s Contrastive Prototype Generation and Adaptation (CPGA) method generates class-specific avatar feature prototypes using a prototype generator trained with contrastive learning, informed by a pre-trained source model's classification boundaries. It aligns pseudo-labeled target features with these source prototypes for class-wise domain alignment in target domains without source data. Our method, in contrast, uses diffusion models as domain shift correctors to directly transform target features into source-resembling ones, aligning domain-specific feature distributions without source data. Unlike CPGA's focus on prototype and pseudo-label generation, we use neighborhood searching for parameterizing the diffusion process, allowing feature distribution alignment without source data. This fundamental difference from CPGA's prototype-based approach marks our method's distinct approach to domain adaptation.

Comparison with [2]: We summarize the 3C-GAN [2], which combines a discriminator, a generator for target-style samples based on random labels, and a pre-trained predictor. The generator and predictor improve performance during training, using generated data to enhance target domain accuracy. However, 3C-GAN requires careful regularization to balance generative and classification objectives. In contrast, our method uses diffusion models and neighborhood searching to transform target features into source-like features. Our diffusion model serves as a domain shift corrector, not an image generator, which obviates the need for complex regularization strategies. By focusing on parameterizing the diffusion model's prior density for feature transformation, our approach significantly diverges from existing methods, representing a novel contribution to domain adaptation.

Weakness 4 and Question 3

Response to Reviewer’s Questions about Encoder G and Performance Comparisons

We understand the reviewers' concern about using ResNet-101 as the encoder for all datasets, unlike other SFDA methods that typically use ResNet-50 for Office-31 and Office-Home. Our experiments showed that both ResNet-50 and ResNet-101 have their merits, with no definitive superior choice for Office-31 and Office-Home. We chose ResNet-101 due to its consistent performance across different datasets and to standardize the backbone architecture in our experiments.

As suggested, we have re-conducted our experiments on Office-31 and Office-Home using ResNet-50 and have also re-run the baseline methods using ResNet-101. The results and further details in response to this concern are included in our official comment to all reviewers in section C2.

Weakness 5 and Question 1

Additional Ablation Studies

Thank you for suggesting additional ablation studies, which have greatly improved our research. We have conducted the suggested ablation study on the large-scale VisDA-2017 dataset. The results and a more detailed response can be found in our official comment to all reviewers in section C3.

Weakness 6

Response to Typos

We respectfully request the reviewer to re-examine the noted typos. We believe our use of English, specifically the phrase "to transition from," is correct, as "transition" can function as a verb in standard English writing. Additionally, regarding the use of articles with abbreviations, the choice between "a" and "an" depends on the starting sound of the acronym. For SGD, which begins with a consonant sound when spoken, the use of “an” is appropriate rather than “a”. Furthermore, we would like to clarify that using ‖⋅‖ is a standard notation for expressing the L2 norm, particularly in the context of cosine similarity. Our use of this notation aligns with the conventional representation of the L2 norm in related literature on cosine similarity. Regardless, we will continue to diligently review our manuscript for any other typos or grammatical errors.

Weakness 3

To address the reviewer’s concerns, we approach it from two angles. Firstly, we ensure that our DND method adheres to all SFDA settings and principles. Secondly, we address the concern regarding the use of source data to train generative models.

Concerning the SFDA Problem Setting

In response to the first aspect of your concern, we direct the reviewer to our official comment in section C1, which is detailed in our response to all reviewers.

Existing Source-Free Models Do Not Train Generative Model Using Source Data

Our DND, integrating a diffusion model with the ResNet backbone, conforms to the SFDA principle of using source data only for initial pre-training. Unlike typical SFDA methods, we utilize the diffusion model for feature refinement, not data generation, aligning with SFDA guidelines and avoiding data privacy issues. Our method does not directly use or store source data for target adaptation but relies on latent geometry informed by k-NNs, ensuring influence from source domain knowledge without direct access to source data. While SFDA methods typically refrain from using source data to train generative models, we see no violation or drawback in training our diffusion module with source data, given its role in enhancing target feature refinement.

This implementation of the diffusion model in DND significantly diverges from conventional SFDA approaches focused on data augmentation. By employing diffusion models for domain-specific feature alignment, our method demonstrates their potential in SFDA, extending beyond traditional generative uses.

We hope that our comprehensive explanation adequately addresses the reviewer's concern regarding our adherence to SFDA settings.

Weakness 4

Response to Reviewer’s Questions about Encoder G and Performance Comparisons

We understand the reviewers' concern about using ResNet-101 as the encoder for all datasets, unlike other SFDA methods that typically use ResNet-50 for Office-31 and Office-Home. Our experiments showed that both ResNet-50 and ResNet-101 have their advantages, with no definitive superior choice for Office-31 and Office-Home. We chose ResNet-101 due to its consistent performance across different datasets and to standardize the backbone architecture in our experiments.

As suggested, we have re-conducted our experiments on Office-31 and Office-Home using ResNet-50 and have also re-run the baseline methods using ResNet-101. The results and further details in response to this concern are included in our official comment to all reviewers in C2.

Response to the Additional Gain of Employing Our Diffusion Model for Testing in the Source Domain

We appreciate the reviewer's acknowledgment of our diffusion model's potential to improve classification performance in general. In our original manuscript, we included an analysis, found in Appendices D and E, on using our DND-generated features for source domain testing. This comprehensive experimentation demonstrated that our DND enhances supervised classification as a latent augmentation module. Our testing extended beyond domain adaptation datasets to datasets for standard classification like CIFAR-10, CIFAR-100, and ImageNet. The results confirm our DND's versatility, effective both as a domain shift corrector in domain adaptation and as a latent augmentation technique in supervised classification.

Reference:

[1] Cho, Junhyeong, Gilhyun Nam, Sungyeon Kim, Hunmin Yang, and Suha Kwak. PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pp. 15702-15712. 2023.

[2] Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. Ttn: A domain-shift aware batch normalization in test-time adaptation. International Conference on Learning Representations (ICLR), 2023.

[3] AlecRadford, JongWook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML), 2021.

[4] Yangjun Ruan, Yann Dubois, and Chris J. Maddison. Optimal Representations for Covariate Shift. In International Conference on Learning Representations (ICLR), 2022.

[5] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), pp. 6028–6039. PMLR, 2020.

[6] Zhang, Marvin, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems (NeurIPS) (2022): 38629-38642.

[7] Nie, Weili, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460 (2022).

[8] Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona. Importance weighted adversarial nets for partial domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8156–8164, 2018.

[9] Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Michael I Jordan. Partial transfer learning with selective adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2724–2732, 2018.

[10] Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1426–1435, 2019.

Weakness 3 (continued)

Partial-Set Domain Adaptation Experiments

As suggested, we have expanded our research to include experiments in partial-set domain adaptation (PDA). PDA involves adapting models from a source domain with a wide range of classes to a target domain with fewer classes, focusing on common classes and disregarding source-exclusive ones. Our DND, influenced by the target domain's latent geometry, aligns source-like features with the target's feature space during adaptation. This approach minimizes the generation of features for source-exclusive classes, addressing class mismatches between domains. Our rationale is that source labels not present in the target domain will have distinct latent geometry, leading our DND to be less likely to generate irrelevant source-like features, staying within the target domain's label space.

We validated our claim using the Office-Home dataset. For PDA, following [9], we chose the first six classes alphabetically as target categories and used their samples for both adaptation and testing. Due to time constraints, we used the SHOT codebase for implementation without changing any hyperparameters. The results displayed below demonstrate that our DND effectively addresses class mismatches during the adaptation process. This highlights our method's capability to retrieve relevant source ground truth based on target latent geometry, thus minimizing negative transfer by avoiding the use of non-relevant source information in target adaptation.

We have included the aforementioned discussion and experimental results in Appendix M of the revised manuscript, highlighted in blue for easy reference.

Table. Partial-set domain adaptation on Office-Home (ResNet-50).

Weakness 3

We appreciate the reviewer's suggestion to broaden our work's impact but wish to clarify that our research is focused on SFDA, not domain generalization or partial-set domain adaptation. Our goal is to demonstrate how diffusion models can assist in the discriminative process, retrieving ground truth knowledge without source data, thus extending their use beyond traditional generative roles. We would like to clarify that in the domain adaptation community, VisDA-2017 is consistently considered a large-scale dataset, comprising approximately 280,000 images. It is also recognized as one of the most challenging datasets in domain adaptation, primarily due to its significant domain shift. Additionally, DomainNet is more frequently employed in domain generalization studies, rather than in source-free domain adaptation (SFDA).

DomainNet Experiments

Since other reviewers mentioned the possibility of applying our DND to domain generalization, as suggested by the reviewer, we have included implementation details and experimental results of using our DND in source-free domain generalization on the large-scale DomainNet dataset.

Further Experiments on DomainNet Dataset:

We have completed the experiments on DomainNet, and the details of our experimental settings and results are provided below. We have added the experiments on domain generalization to Appendix J of the revised manuscript, with the relevant sections highlighted in blue for easy reference.

In applying our method to domain generalization, we focused on the target inference stage, keeping the source pre-training stage consistent with our approach for SFDA. This approach aligns with the popular source-free domain generalization setting discussed in [1].

Inference for Domain Generalization: Contrary to SFDA, where the model will be fine-tuned on target domain data, source-free domain generalization involves no training during target testing. Our SiLGA method is used for latent augmentation, specifically to transform encoded target features during the target testing phase, without changing any model parameters:

$\mathbf{\hat{z}}^{t,i} := \frac{(\mathbf{z}_{1}^{t,i}+\mathbb{N}_k (\mathbf{z}^{t,i}))}{k+1}$.

Here, $\mathbf{\hat{z}}^{t,i}$ are the transformed target features, $\mathbf{z}_{1}^{t,i}$ are DND-generated features guided by the prior density parameterized by target latent geometry, and \mathbb{N}_k (\mathbf{z}^{t,i})) is the centroid of the latent $k$-NNs of the target features.

Domain Generalization Experimental Setup: Our method utilizes a single hyperparameter, $k$, which is used for both exploring latent geometry and parameterizing the Gaussian prior, and is set to a value of 5. SiLGA's effectiveness was demonstrated on the large-scale DomainNet for domain generalization, suggesting potential for further exploring our DND in this area. Time constraints during the author's response period limited deeper investigation into domain generalization techniques like batch normalization averaging [2] or additional regularization and augmentation. Our approach primarily involved using DND for feature transformation during target testing.

Table. Domain generalization experiments on DomainNet (ResNet-50).

We appreciate the reviewer's diligent effort in reviewing our manuscript. Below, we provide our detailed response to your concerns.

Weakness 1:

Based on our understanding of the reviewer's concern, we have split it into two parts and addressed each in our official comment to all reviewers:

Response to Concerns about Violation of SFDA Problem Settings

In response to this concern, we direct the reviewer to our official comment in section C1, which is detailed in our response to all reviewers.

Response to Concerns about Additional Model and Time Complexity

In response to this concern, we direct the reviewer to our official comment in section C4, which is detailed in our response to all reviewers.

Weakness 2:

Experiments on Single-Epoch Target Adaptation:

Thank you for the suggestion. We would like to clarify that the focus of our paper is source-free domain adaptation instead of test-time adaptation. However, as suggested by the reviewer, our method can also be seamlessly applied to single-epoch target adaptation. We would like to thank the reviewer for the important suggestion to broaden the impact of our work.

First, we would like to differentiate our method from the one in [a], where the diffusion model is used to project target domain data back to source domain data during target adaptation. This method relies on generating data density. A key limitation of [a] is the unclear correspondence between the generated target data and the original source data. We address this in our approach by integrating latent geometry into the diffusion process to guide this correspondence. Crucially, instead of generating data, our diffusion model operates at the feature level, infusing source-like traits into the encoded target features. This novel application of diffusion models represents a significant shift from traditional approaches, offering a unique contribution to the field of domain adaptation and filling a previously unexplored gap in the community.

Next, we would like to discuss the specifics of implementing our method for single-epoch target adaptation. It is important to note that the source pre-training stage adheres to our approach for SFDA. This experimental framework aligns with the one employed in [a]:

Single-Epoch Target Domain Adaptation: Our method performs one epoch target adaptation using contrastive learning. It utilizes our DND-generated positive key to align target features more closely with the source feature space, focusing on domain-specific feature alignment and domain shift correction. We use ResNet-50 as the backbone, consistent with baseline methods, and set $k$ for neighborhood searching to 5 with 16 diffusion steps. The one-epoch adaptation uses an SGD optimizer, a learning rate of $5e^{-3}$, and a batch size of $128$. Due to time constraints, we conducted experiments only on the large-scale ImageNet-C dataset, with the results displayed below.

We have incorporated the aforementioned discussion and experiments into Appendix K of the revised manuscript and highlighted them in blue.

Table. One-Epoch Target Adaptation Experiments on ImageNet-C (ResNet-50).

We would like to greatly appreciate the reviewer’s positive and constructive feedback on our manuscript. Your recognition of our work and insightful suggestions have been invaluable in enhancing the quality and impact of our work. Thank you again for the supportive and encouraging words, and for contributing significantly to our research journey.

Suggesting Investigation of DND for Domain Generalization Applications:

Thank you for your insightful suggestion to conduct domain generalization experiments, which has helped broaden the impact of our method. We have completed these experiments, and the details of our experimental settings and results are provided below. We have added the experiments on domain generalization to Appendix J of the revised manuscript, with the relevant sections highlighted in blue for easy reference.

First, we would like to clarify that source-free domain adaptation (SFDA) and source-free domain generalization (SFDG) have distinct objectives. SFDA is about adapting a model initially trained on a source domain to a different target domain, and this adaptation occurs without access to the original source data. On the other hand, SFDG involves training a model in one or multiple source domains with the goal of effective generalization across various unknown and unseen target domains, without requiring any specific adaptation training for these target domains. Therefore, our method, which is tailored for SFDA, cannot be directly applied to SFDG without modifying its framework to fulfill the more expansive generalization objectives of SFDG.

In applying our method to domain generalization, we focused on the target inference stage, keeping the source pre-training stage consistent with our approach for SFDA. This approach aligns with the popular source-free domain generalization setting discussed in [1].

Inference for Domain Generalization: Contrary to SFDA, where the model will be fine-tuned on target domain data, source-free domain generalization involves no training during target testing. Our SiLGA method is used for latent augmentation, specifically to transform encoded target features during the target testing phase, without changing any model parameters:

$\mathbf{\hat{z}}^{t,i}=\frac{\mathbf{z}_{1}^{t,i}+\mathbb{N}_k (\mathbf{z}^{t,i})}{k+1}$

Here, $\mathbf{\hat{z}}^{t,i}$ are the transformed target features, $\mathbf{z}_{1}^{t,i}$ are DND-generated features guided by the prior density parameterized by target latent geometry, and $\mathbb{N}_k (\mathbf{z}^{t,i})$ are the latent $k$-NNs of the target features.

Domain Generalization Experimental Setup: Our method utilizes a single hyperparameter, $k$, which is used for both exploring latent geometry and parameterizing the Gaussian prior, and is set to a value of 5. SiLGA's effectiveness was demonstrated on the large-scale DomainNet for domain generalization, suggesting potential for further exploring our DND in this area. Time constraints during the author's response period limited deeper investigation into domain generalization techniques like batch normalization averaging [2] or additional regularization and augmentation. Our approach primarily involved using DND for feature transformation during target testing.

Table. Domain generalization experiments on DomainNet (ResNet-50).

Reference:

[1] Cho, Junhyeong, Gilhyun Nam, Sungyeon Kim, Hunmin Yang, and Suha Kwak. PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pp. 15702-15712. 2023.

[2] Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. Ttn: A domain-shift aware batch normalization in test-time adaptation. International Conference on Learning Representations (ICLR), 2023.

[3] AlecRadford, JongWook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML), 2021.

[4] Yangjun Ruan, Yann Dubois, and Chris J. Maddison. Optimal Representations for Covariate Shift. In International Conference on Learning Representations (ICLR), 2022.

We appreciate the reviewer's diligent effort in reviewing our manuscript. Below, we provide our detailed response to your concerns.

Response to Concerns about Violation of SFDA Problem Settings (weakness 1 and question 1)

In response to this concern, we direct the reviewer to our official comment in section C1, which is detailed in our response to all reviewers.

Clarity and Structure of the Paper (weakness 2):

Thank you for raising concerns about the clarity of our writing, particularly in the introduction and methods sections. As suggested, we have revised these sections and highlighted the major changes in blue in the revised paper for clarity and conciseness (please download the updated PDF file. It now contains our revised manuscript).

The revised Introduction concisely presents our research problem and approach, and the Methods section now focuses on our DND method's novelty, reducing the extent of the existing work explanation. We have limited the discussion about IADB to only the key details that are important for understanding our DND. You can see these updates marked in blue in our revised manuscript.

Hyperparameter Sensitivity Analysis (weakness 3):

Thank you for suggesting a hyperparameter sensitivity analysis for our DND and raising concerns about the challenge of selecting suitable parameters. We would like to clarify that we were aware of the potential complexities in diffusion models while developing our DND, which led us to focus on designing a lightweight model with few hyperparameters. Our primary goal is to show how diffusion models can be used as domain shift correctors for SFDA challenges, beyond their traditional role in data density generation, rather than just maximizing performance. We selected hyperparameters to keep the model light, and consequently, our diffusion model implementation involves just three hyperparameters: (1) the number of diffusion steps; (2) the number of neighbors ($k_s$) for source pre-training; and (3) the number of neighbors ($k_t$) for target adaptation.

As suggested, we have conducted a detailed sensitivity analysis of the three hyperparameters to evaluate their impact on target-domain classification accuracy. This analysis is described in Appendix H of our revised manuscript, with the relevant parts marked in blue for ease of reference. Our results suggest that changes in hyperparameter values have a minimal effect on performance. We hope this explanation and our hyperparameter sensitivity analysis address the concerns about the complexity and choice of hyperparameters in our DND.

In each experiment, we controlled variables as follows: we set $k_s$ to 5 while evaluating $k_t$ and the diffusion steps; $k_t$ was fixed at 5 during the evaluation of $k_s$ and the diffusion steps; and the number of diffusion steps was maintained at 16 when assessing $k_s$ and $k_t$. The analysis results are shown below.

Table. sensitivity analysis for $k_s$ and $k_t$ on VisDA-2017 (ResNet-101).

Table. sensitivity analysis for the number of diffusion steps on VisDA-2017 (ResNet-101).

Ablation Study on A More Complex Dataset (weakness 4):

Thank you for recommending the inclusion of ablation studies on a more complex dataset. As suggested, we have now added new ablation studies on the VisDA-C dataset, known for its large scale and significant domain shifts, to our updated manuscript. The original Office-31 studies have been moved to Appendix G.

Table. Ablation studies on VisDA-2017 (ResNet-101).

Response to Reviewer’s Questions about Encoder G and Performance Comparisons (question 2)

We understand the reviewers' concern about using ResNet-101 as the encoder for all datasets, unlike other SFDA methods that typically use ResNet-50 for Office-31 and Office-Home. Our experiments showed that both ResNet-50 and ResNet-101 have their merits, with no definitive superior choice for Office-31 and Office-Home. We chose ResNet-101 due to its consistent performance across different datasets and to standardize the backbone architecture in our experiments.

As suggested, we have re-conducted our experiments on Office-31 and Office-Home using ResNet-50 and have also re-run the baseline methods using ResNet-101. The results and further details in response to this concern are included in our official comment to all reviewers in C2.

Response to Concerns about Misreported Results for NRC++ (question 2)

We thank the reviewer for pointing out the mismatch in reporting NRC++ results, which were incorrectly taken from the NRC results. We have corrected our tables to accurately reflect NRC++ results and thoroughly reviewed all tables to ensure precise reporting and consistency between cited methods and their results.

Response to Concerns about Time Complexity (question 3)

In response to this concern, we direct the reviewer to our official comment in section C4, which is detailed in our response to all reviewers.

Figure 1 Typo (question 4):

Huge thanks for the detailed examination by the reviewer. As suggested, we have updated Figure 1 in our revised manuscript. Now, diamonds represent class 0 and inverted triangles represent class 1. We have highlighted Figure 1 in blue for easier reference in the revised paper.

Algorithm 1 Typo (question 5):

We appreciate the reviewer's careful examination. In response to the suggestion, we have updated Algorithm 1 in the revised manuscript, with the changes highlighted in blue for clarity. For your reference, we have updated Line 2 of Algorithm 1 in the revised manuscript, changing the ending from T to T-1.

Reference:

[1] Ziyi Zhang, Weikai Chen, Hui Cheng, Zhen Li, Siyuan Li, Liang Lin, and Guanbin Li. Divide and contrast: Source-free domain adaptation via adaptive contrastive learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022.

[2] Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, Shangling Jui, and Jian Yang. Trust your good friends: Source-free domain adaptation by reciprocal neighborhood clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

We appreciate your prompt response.

While it is true that a source provider needs to perform the additional task of training the DND module, whether this is practical or not can vary depending on perspective.

1. If, as per your assumption, there is a separate provider for the classifier, and they wish for their classifier to adapt well to different target domains, they can simply attach and train the DND module before providing it. Our DND functions as a supplementary module that is integrated with the main classification model. While this requires additional work, it does not significantly add to the overall effort. For instance, when using ResNet-101 as the backbone on the largest benchmark dataset, VisDA-2017, the time needed for the source diffusion process is only about 3672.4 seconds for 15 epochs (15 epochs is sufficient for its convergence) ($\approx$ 1 hour).

2. Additionally, as highlighted in Appendix E, incorporating our DND module also improves performance in the source domain. Thus, from the source provider’s standpoint, adding DND could be considered beneficial not just for target domain adaptation but also for enhancing source domain performance.

Our understanding of the definition of source-free domain adaptation (SFDA) is to adapt a model trained on one source domain to a different target domain, without having access to the original source data. This problem focuses on addressing the privacy concern of releasing training data to the public. We would be grateful if the reviewer could share their understanding of SFDA and kindly point out any specific aspects where our paper might diverge from the typical SFDA settings.

C3: Rerunning Ablation Study on Large-Scale Dataset and Including Ablation on k-NN Alone:

Dataset Complexity: As suggested, we have now added new ablation studies on the VisDA-C dataset, known for its large scale and significant domain shifts, to our updated manuscript. The original Office-31 studies have been moved to Appendix G.

Additional Ablation on Using k-NN Features Alone: As suggested by reviewers, we have performed the suggested ablation study, using the target’s k-NN features alone for generating positive keys, on the large-scale VisDA-2017 dataset. These additional studies are highlighted in blue in the revised manuscript. For your convenience, we have also provided the results of these new ablation experiments below.

Table. Ablation studies on VisDA-2017 (ResNet-101).

C4: Response to Concerns about Additional Model and Time Complexity:

We understand the concern that our diffusion model's iterative sampling might add time complexity compared to other SFDA methods. However, we would like to clarify that our model is tailored to transform target features, not to generate images. Thus, our diffusion model requires fewer model parameters and results in quicker inference compared to typical diffusion models. For efficiency, we have limited diffusion steps to 16 in all experiments. Moreover, our model does not require estimating a covariance matrix for state transitions, which further reduces its complexity.

As suggested, we have conducted a runtime analysis of our DND and compared it with other SFDA methods. We selected the large-scale VisDA-2017 dataset to ensure the effectiveness of the analysis. The experiment was conducted on a machine equipped with an Nvidia V100 GPU.

The table below indicates that our DND and NRC++ have similar target adaptation times per epoch, while DaC takes substantially longer due to its adaptive contrastive process and self-training steps, including pseudo label generation and re-training. Our DND and NRC++ typically converge in about 10 epochs, whereas DaC requires around 20 epochs. The total convergence time for target adaptation is also provided.

Despite incorporating a diffusion process, the deterministic sampling process of our diffusion model and the limited number of steps help prevent significant increases in computational complexity. This is crucial, considering the substantial performance improvement our method offers.

A comprehensive runtime analysis is included in Appendix L of our revised manuscript. We hope this clarification and analysis address concerns regarding the time complexity of our method.

Table. Time Analysis of One-Epoch Target Adaptation on VisDA-2017 Dataset (ResNet-101).

Table. Convergence Time Analysis for Target Adaptation on VisDA-2017 Dataset (ResNet-101).

Reference

[1] Ziyi Zhang, Weikai Chen, Hui Cheng, Zhen Li, Siyuan Li, Liang Lin, and Guanbin Li. Divide and contrast: Source-free domain adaptation via adaptive contrastive learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022.

[2] Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, Shangling Jui, and Jian Yang. Trust your good friends: Source-free domain adaptation by reciprocal neighborhood clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

C2: Response to Concerns about Encoder Choice and Performance Comparisons:

Our rationale for using ResNet-101 was to ensure adequate performance on the most challenging dataset, VisDA2017, among the three we used: Office-31, Office-Home, and VisDA2017. We applied this backbone uniformly across all datasets. For Office-31 and Office-Home, ResNet-50 would have been sufficient. We also concluded that there was not a clear winner between ResNet-50 and ResNet-101 for these datasets. Nonetheless, we understand the reviewers' concerns and, in response, have conducted additional experiments on the Office datasets using ResNet-50. These results show that our DND outperforms the baseline methods regardless of the backbone choice.

In addition to this, we have also retested all baseline methods with ResNet-101 for a more comprehensive analysis.

Experiments on Office-31 and Office-Home with ResNet-50:

Table. Comparison of the SFDA methods on Office-31 (ResNet-50).

Table. Comparison of the SFDA methods on Office-Home (ResNet-50).

We have also conducted experiments for all baseline methods using ResNet-101:

Table. Comparison of the SFDA methods on Office-31 (ResNet-101).

Table. Comparison of the SFDA methods on Office-Home (ResNet-101).