Diffusion-Driven Progressive Target Manipulation for Source-Free Domain Adaptation

Q1: Limited contribution to the community.

First of all, we have to emphasize that we provide in this paper a novel paradigm that leverages diffusion-based generation to directly approximate the target domain itself. This paradigm significantly differs from existing methods that rely heavily on pseudo-labeling or attempt to generate pseudo-source data and overcomes the intrinsic domain gap by nature. Experimental results also demonstrate incredibly superior performance as strong support of our paradigm.

In the field of domain adaptation, SFDA usually suffers from degraded performance caused by the lack of source data, which has not yet been addressed by existing methods. In this paper, we provide the key insight of leveraging diffusion-based generation to directly approximate the target domain rather than being restricted by making up the missing source data. We then realize this insight in a robust and generalizable manner and achieve strong empirical performance across diverse datasets. More importantly, this insight itself may open up new research directions for DA beyond SFDA, including broader scenarios such as UDA or even settings where the source model is entirely inaccessible. We have also made a detailed explanation on the practicality of our method in Response to Q1 by Reviewer mAks.

Q2: Similarity to Existing Work.

In SFDA, generating extra samples is a popular research paradigm due to the inherent data scarcity. Nevertheless, the fact that our method follows this paradigm should not compromise its novelty. The two works cited by the reviewer are fundamentally different from our method.

• DM-SFDA: This method follows a pseudo-source data generation paradigm. We have already discussed the key differences between our approach and pseudo-source-based SFDA methods in Section 1. In terms of implementation, DM-SFDA relies heavily on complex fine-tuning of a diffusion model, which is totally different from our mechanism. Moreover, its performance is significantly lower than that of our method, as reported in Response to Q3 by Reviewer fShy.

• DAD: This work addresses UDA with source data, rather than SFDA without source data. In fact, it focuses on leveraging source domain data effectively, while our method operates under a source-free setting where the source data is completely inaccessible, and we do not attempt to generate pseudo-source data.

To the best of our knowledge, there are no existing SFDA methods that adopt the same design or similar specific mechanisms as ours.

Q3: Lack of Visualization Results.

Thanks for the suggestion, we first conduct Grad-CAM visualization using our SFDA model, and compare the results with those obtained by the source model. The results demonstrate that the attention of our model focuses on the target object, instead of other unrelated regions. Besides, we conduct t-SNE analysis on target domain feature distribution, and our method exhibits the least category aliasing.

Q4: Using an image encoder to replace FFT.

We use FFT to obtain a pseudo-image in the Target-guided Initialization stage and a desirable latent in both Semantic Feature Injection and Domain-specific Feature Preservation. In this case, using an image encoder introduces several challenges:

• First, the extracted features often have altered dimensions, making it difficult to reconstruct the original pseudo-image or latent representation without an accompanying decoder.
• Second, this necessitates pretraining both the image encoder and decoder to extract and reconstruct frequency components. However, it is hard to guarantee effective pretraining using the unlabeled target data, especially on small-scale datasets. This can significantly degrade the performance of our method, while adding extra complexity and computational costs.

In contrast, our experiments demonstrate that FFT is simple yet effective. Even in real-world scenarios, it introduces negligible overhead and does not hinder the overall performance of the algorithm. We have evaluated the performance of our methods on real-world scenarios VLCS, using unsupervised model selection without using target labels, as elaborated in Response to Q1 by Reviewer mAks.

Q5: Generation-based methods listed in this paper do not generate samples.

We list 4 generation-based methods, including CPGA, ASOGE, ISFDA, and PS. Actually, two of them, i.e., ISFDA and PS, generate samples.

Q6: Similar colors are used in Figure 1.

We will correct the confusion in Figure 1.

New Q1: Target-guided Initilization

Reviewer comment

I am wondering the motivation of the design of Target-guided Initilization in Figure 1. The authors argue that Target-guided Initialization is designed to obtain an effective sampling starting point. Because such design can initialize a noise maintaining the semantics of target samples? Would it misalign the theory of a standard denoising diffusion model?

Response

Motivation of Target-guided Initialization

We design Target-guided Initialization to construct a semantically neutral yet target-domain-aware sampling starting point. The motivation is based on recent findings about the importance of the sampling starting point and the signal-leak bias in diffusion models. We elaborate on this motivation in the following steps:

1. Sampling starting point matters:

For diffusion models, prior research has shown that the sampling starting point can have a significant impact on the final generation results, and that carefully designing this starting point can lead to improved performance [1][2][3][4].

2. Signal-leak bias in diffusion models:

Specifically, [1] investigates the relationship between the generated image and the sampling starting point of the diffusion model, and identifies a phenomenon termed signal-leak bias:

• During the training process, the sampling starting point is obtained by corrupting an image with $T$ steps of Gaussian noise. Even when $T$ is large, the low-frequency components of the image are not entirely destroyed. As a result, the sampling starting point retains part of the original image's low-frequency information. During training, diffusion models learn to exploit this low-frequency signal for the sake of minimizing the training loss. Consequently, the generated images tend to preserve low-frequency information that is present in the sampling starting point.
• During the inference process, the diffusion model continues to utilize the low-frequency components of the sampling starting point to guide the generation process, which is referred to as signal-leak bias.

3. Leveraging signal-leak bias for SFDA:

We observe that the signal-leak bias can be effectively leveraged to benefit the SFDA task. In domain adaptation and generalization, low-frequency components of images are considered to partly contribute to domain-specific features [5][6]. For each non-trust sample, since it belongs to the target domain, its low-frequency components can be considered to partly carry certain target-domain-specific features. Therefore, our method extracts the low-frequency components from the non-trust sample and uses them to form the sampling starting point of the diffusion model, allowing the generated image to inherit part of the target domain features via signal-leak bias, which facilitates adaptation.

4. Semantic neutrality via high-frequency noise:

Besides, it has been shown that in diffusion models, high-frequency components are closely related to semantic detail patterns[7], such as object contours and textures. Since the real class label of the non-trust sample is unknown, we randomly assign a class label and aim to transform the non-trust sample into the assigned class via the generation process. As this transformation involves a semantic transform process, we choose to extract high-frequency components from a semantically neutral random Gaussian noise, ensuring that no class-specific information is inadvertently introduced at the sampling starting point.

5. Combining both components to construct the sampling starting point:

We combine the aforementioned extracted low-frequency and high-frequency components, and perform IFFT to produce a semantically neutral pseudo-image carrying target domain features. This pseudo-image is then encoded into a latent using the VAE encoder of the diffusion model. The obtained latent is denoted as $\hat{\mathbf{z}}_{0}$ in the manuscript.

Finally, we perform the DDPM forward process on $\hat{\mathbf{z}}_{0}$ by adding $T$ steps of Gaussian noise, resulting in a noisy latent representation $\mathbf{z}_T$. This process corresponds to Equation (5) in the manuscript:

$\mathbf{z}_T=\sqrt{\alpha_T} \hat{\mathbf{z}}_0+\sqrt{1-\alpha_T} \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) .$

The resulting $\mathbf{z}_T$ is employed as the sampling starting point of the diffusion model. According to the aforementioned signal-leak bias, the DDPM forward process preserves low-frequency components while corrupting high-frequency components. This allows the sampling starting point $\mathbf{z}_T$ to retain target domain features while becoming more semantically neutral.

We sincerely thank the reviewer for the response. We will respond to these questions in detail below. To better address the concerns, we first provide a more in-depth analysis of the Target-guided Initialization, and then provide a further explanation of the effectiveness of the FFT component.

Q1: Hyperparameter selection procedure and real-world scenarios performance.

We thank the reviewer’s insightful comments regarding the practical applicability of our method in real-world scenarios, particularly in the hyperparameter selection procedure. We respond to this concern from two aspects:

1. Hyperparameter determination.

We would like to clarify the determination of hyperparameters with a more detailed description. In fact, we did not tune hyperparameters separately for each SFDA task. The guidance scales and FFT-related hyperparameters are fixed on all the datasets, and we also use $E=0.01$ and $R=10$ for all datasets. In lines 254-255, we intend to mean that we only explore different values of $E$ and $R$ once on Office-Home and fix $E$ to 0.01 and $R$ to 10 for all the experiments (except for ablation studies). Thus, the hyperparameter selection is practical for applications in real-world scenarios. We will further discuss the practicality of this hyperparameter below and validate it on VLCS.

2. Recommendations for real-world usage.

We fully agree that hyperparameter determination can be hard in real-world scenarios. Thus, we provide practical recommendations for hyperparameter selection.

• For default settings, we suggest using the same hyperparameters as we do.
• For adjustment strategy, if the model does not perform well, we first recommend moderately increasing both $E$ and $R$ (e.g., $E=0.1$ and $R=20$). Increasing $E$ and $R$ is probably effective in most cases, according to Section 4.3.
• For more complex scenarios, we can also explore other hyperparameter values. In such cases, unsupervised model selection becomes necessary. Inspired by the reviewer, we employ an unsupervised model selection strategy using nuclear norm. We extract the softmax output matrix for all target samples and calculate its nuclear norms. Then， we select the model with the largest nuclear norm [1][2]. As suggested, we train models with $E\in[0.01,0.005,0.001]$ and $R$ from 1 to 10 on real-world benchmark VLCS and apply this model selection metric. In the table below, comparison results are from [3], where UMS is unsupervised model selection for short. Note that all the methods for comparison report their best target accuracy (according to the labels), while ours performs model selection without knowing target accuracy.

[1]Cui S et al. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations, CVPR 2020. [2]Cui S et al. Fast batch nuclear-norm maximization and minimization for robust domain adaptation. arXiv:2107.06154. [3]Lee S et al. Few-shot fine-tuning is all you need for source-free domain adaptation. arXiv:2304.00792.

Q2：$E$ sensitivity.

As mentioned above, we uniformly set $E=0.01$ for all experiments across datasets. The ablation results with $E=0.001$ are only provided to demonstrate that a higher $E$ yields better performance, and this setting ($E=0.001$) was never actually used in Tables 1–3.

Q3: Selectively chosen of baselines in the performance comparisons.

In the ProDe and DIFO papers, ProDe-V and DIFO-C-B32 indicate direct use of CLIP-ViT as the backbone for the SFDA model. Nevertheless, most SFDA methods, including ours, use vanilla ResNet as the backbone. For fair comparison, we did not include ProDe-V and DIFO-C-B32. We have clearly specified the ResNet backbone in the captions of Tables 1-3.

Q4: Quantitative component-wise analysis.

Ablation studies are performed on each component in Response to Q1 by Reviewer fShy, and demonstrate the effectiveness of each component.

Q5: Low-frequency information remains in diffusion latents.

Yes. Prior studies [4][5][6] have demonstrated this conclusion, and they also operated frequency decomposition in the diffusion latent space.

Q6: Reuse already manipulated samples.

The target model determines which samples are included in the non-trust set, and usually selects those it struggles to classify. After training on their manipulated version, some non-trust samples become well-understood, as implied by the growing trust set. Therefore, retaining their manipulated versions is unnecessary and introduces two issues:

• It creates an ever-growing synthetic domain which is much larger than the real target domain, causing the model to adapt to the synthetic domain rather than the real target domain.
• It significantly increases training costs. We offer a similar experimental analysis in Response to Q4 by Reviewer pUPg.

[4]Everaert et al. Exploiting the signal-leak bias in diffusion models. WACV 2024.

[5]Wu et al. Freeinit: Bridging initialization gap in video diffusion models. ECCV 2024.

[6]Koo G et al. Flexiedit: Frequency-aware latent refinement for enhanced non-rigid editing. ECCV 2024.

New results 2: Component-wise ablation study results

We further conduct ablation studies on the independent usage of each individual component. For clarity in the tables, we refer to Target-guided Initialization, Semantic Feature Injection, and Domain-specific Feature Preservation as TGI, SFI, and DFP, respectively. We present comprehensive component-wise ablation results, including the performance of the model with only one component enabled and with each component individually removed.

These results further demonstrate the effectiveness of our method. We provide a detailed analysis as below:

For Target-guided Initialization (TGI for short):

• The comparison of TGI✗/SFI✗/DFP✗ and TGI✓/SFI✗/DFP✗ demonstrates that TGI brings improvements to the overall performance, raising the average accuracy from 62.2% to 77.9%, even when used alone. This shows the crucial role of TGI in our framework.
• The comparison of TGI✗/SFI✓/DFP✗ and TGI✓/SFI✓/DFP✗ demonstrates that adding TGI on top of SFI further improves the performance, boosting the average accuracy from 74.4% to 82.2%.
• The comparison of TGI✗/SFI✗/DFP✓ and TGI✓/SFI✗/DFP✓ demonstrates that TGI can also further enhance when combined with DFP, enhancing the average accuracy from 76.8% to 79.8%.
• The comparison of TGI✗/SFI✓/DFP✓ and TGI✓/SFI✓/DFP✓ demonstrates that introducing TGI on the basis of SFI and DFP further enhances the performance from 82.5% to 91.2%.

Similarly, for Semantic Feature Injection and Domain-specific Feature Preservation, the results can also demonstrate the effectiveness and necessity of each component in our framework. We will include a comprehensive analysis in the revised version.

We sincerely thank the reviewer for the constructive and insightful feedback. We fully agree with the importance of both aspects raised, and we will incorporate these points into the revised version of our manuscript. In addition, we have further extended our evaluation along both aspects and present the corresponding results below.

New results 1: Unsupervised model selection

The reviewer previously suggested evaluating the performance of our method on real-world scenarios, including VLCS and TerraIncognita, to further validate the practicality of our method. In our initial response, due to time constraints, we only conducted experiments on the VLCS dataset.

We have now additionally conducted unsupervised model selection experiments on the TerraIncognita dataset to further evaluate the practicality of our method. The experimental setting follows that of the VLCS dataset, and we also use the nuclear norm to select the best model without access to target domain labels. The comparison results are also taken from [1].

The results demonstrate that, similar to VLCS, our method significantly outperforms all compared methods even when using nuclear norm as the criterion for unsupervised model selection, achieving an average accuracy improvement of 12.4% over the best-performing comparative method. Notably, in some challenging scenarios such as L100→L43 and L38→L43, where existing methods generally perform poorly, our method achieves substantial gains of 34.8% and 36.5%, respectively, even under unsupervised model selection. It is worth noting that all compared methods were evaluated using their best-performing models selected based on target domain accuracy. These results further demonstrate the effectiveness and practicality of our method.

We also acknowledge that there may exist better criteria beyond the nuclear norm. Exploring more effective model selection metrics will be an important direction for our future work, as we believe that a more suitable criterion could further unleash the potential of our method and enhance its practical applicability. We sincerely thank the reviewer for the valuable insights.

[1] Lee S et al. Few-shot fine-tuning is all you need for source-free domain adaptation. arXiv:2304.00792.

Q1: Unfair comparison

• A prevalent paradigm involves employing generative models to synthesize data, as target domain data scarcity is a fundamental challenge in SFDA settings. Consistent with other generation-based SFDA methods, our method strictly adheres to SFDA settings by completely avoiding any use of source domain data during the generation process.
• The use of external models for adaptation is common in SFDA, as demonstrated by established methods like DIFO and ProDe in Section 4.1.
• We only utilize the diffusion model for adaptation training. In the inference stage, we do not use the diffusion model.

Q2: Cost analysis

Please refer to the Response to Q3 by Reviewer Rmwo for detailed cost analysis.

Q3: Reliability of entropy-based trust and non-trust partition

Entropy-based selection of reliable pseudo-labels of target samples is commonly used in SFDA, and its effectiveness is demonstrated in prior studies [1]. To further evaluate its reliability, we report the trust set accuracy evolving with $r$ on the Office-Home dataset as below ($r$ denotes the $r$-th refinement iteration, where $r=1,2,...R$, and in our experiments we set $R=10$). Figure 2(a) shows that the size of the trust set grows as $r$ increases. Remarkably, the trust accuracy remains consistently high across all tasks using a total of 10 refinement iterations. Besides, when $r$ is small, the trust set accuracy may not be high, but will increase to a high value as $r$ grows. For example, in tasks like Pr→Ar and Pr→Cl, trust accuracy is below 90% when $r=2$ but reaches higher than 97% when $r=10$. These results validate the effectiveness of our method that could correct errors in the trust set with the growth of $r$. Note that the non-trust set accuracy does not affect the performance of our method, as we completely discard the original pseudo-labels of non-trust samples.

[1]Liang J et al. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. TPAMI 2021.

Q4: Set all target samples to non-trust set.

The high performance gain of our method stems from two key mechanisms: i) progressively expanding the trust set with high-accuracy pseudo-labels to allow the SFDA model to learn real target domain features, and ii) progressively reducing the manipulated non-trust set. Mechanism ii) is critical to prevent features from the synthetic domain becoming dominant, since there is an inherent gap that persists between the synthetic and real target domains (even though we employ alignments to bridge the gap). This phenomenon fundamentally stems from the inherent domain gap between synthetic and real data, a well-documented challenge that has been rigorously demonstrated in prior work [2].

Therefore, canceling the trust set could cause degraded performance, since we could only obtain features from the synthetic domain. We validate it with an empirical study on the first 4 Office-Home tasks (Ar→Cl, Ar→Pr, Ar→Rw, and Cl→Ar), where we cancel the trust set and set all target samples to the non-trust set. $r$ denotes the $r$-th refinement iteration, and we report results of $r=1,2,...,5$. As shown below, when using only non-trust samples, the performance is not improved as $r$ grows.

[2] Akagic A et al. Exploring the Impact of Real and Synthetic Data in Image Classification: A Comprehensive Investigation Using CIFAKE Dataset. CoDIT 2024.

Q5: Appendix.

We will remove repeating experimental results and add more generated samples.

Q6: The size of the target SFDA model.

Table A.1 shows results on VisDA with ResNet-101. Below, we provide extra results with ResNet-50 to further demonstrate the superior performance of our method.

Q7: Description and classification of the types of benchmark methods.

We will further classify and reorganize benchmark methods.

Q8: Quantitatively measure how well the samples generated by the diffusion model represent the target domain.

We conduct cross-domain validation by training models on real target domain data and evaluating them on i) our generated target domain (1st row) and ii) Other Office-Home domains for reference (2nd row). For example, the model trained on real Art domain data (first column) achieves 84.2% accuracy on our synthetic Art samples, and 64.1% average accuracy on Clipart, Product, and RealWorld domains. The results indicate that our generated samples closely approximate the real target domain, and the domain gap between synthetic and real data is significantly smaller than the natural gap between Office-Home domains.

Q9: How specifically was the grid search performed to find threshold $E$?

We use fixed values that $E=0.01$ and $R=10$ for all the results in Table 1-3 without any change. We only perform grid search once on Office-Home to explore $E\in[0.01, 0.007, 0.005, 0.001]$ and $R\in[1,...,10]$ and then fix the values $E=0.01$ and $R=10$ for all the datasets. We offer a detailed explanation in the Response to Q1 by Reviewer mAks, and will clarify it in the final version.

Q10: What does ‘we do not specifically optimize the search space’ mean?

We intend to mean we do not exhaustively search all the datasets for optimal values of $E$ and $R$, as specified in response to Q9. In fact, using larger $E$ or $R$ (e.g., $E>0.01$ or $R>10$) could obtain better results.

The responses were generally insincere and did not fully address my questions. Below are some details.

Q1: Unfair comparison: The existing methods mentioned by the authors did not use genertive models like diffusion model for generating samples in the target domain. The proposed method utilizes a diffusion model capable of generating the samples in the target domain. Please tell us about the fairness of the comparison from this perspective.

Q2: Cost analysis: There’s no answer for my question in Response to Q3 by Reviewer Rmwo. Maybe response to Q4 is correct. This response simply mentions the computational complexity of the proposed method, and the analysis is very low quality and insincere. A table should be provided comparing the training time, inference time, and memory [GB] usage compared to benchmark methods.

Where’s the response about the performance comparison with the model that is trained with all samples from the target domain? In response to Q6, I don't understand why the authors gave me this answer. What does the Method in each row mean? Where’s Table A? Please review the first question again.

Q1: Comprehensive ablation studies

We have made ablation studies to evaluate each component on the Office-Home dataset, as reported below. The results demonstrate that all the components contribute significantly to the overall performance by obtaining average gains of 8.7%, 11.4%, and 9.0%, respectively.

Q2: Diversity vs. target alignment trade-off

We focus more on target alignment. To explore the diversity vs. target alignment trade-off, we directly use text-to-image generation (which focuses more on image diversity) to replace our generation mechanism of non-trust set, and perform the same iteration refinement. Due to time limits for rebuttal, we perform experiments on the first 4 Office-Home tasks (Ar→Cl, Ar→Pr, Ar→Rw, and Cl→Ar). As shown below, our method prioritizing target alignment is superior to the method prioritizing image diversity.

Q3: Incomplete baseline comparisons

It is not feasible to directly compare our method with DATUM for two reasons. First, DATUM addresses UDA where source data is accessible, while we focus on SFDA without access to source data. Second, DATUM is for segmentation, while we specialize in classification.

To address the concern, we include DM-SFDA [1] for comparison. As far as we know, DM-SFDA is the only existing generation-based SFDA method. It focuses on pseudo-source generation employing multiple stages of finetuning and generation. Since the source code of DM-SFDA is not available, we compare it with our methods on Office-31 (ResNet-50), Office-Home (ResNet-50), and VisDa (ResNet-101) by directly citing their results reported in [1]. The results demonstrate that our method significantly outperforms DM-SFDA on the majority of all SFDA tasks.

[1] Chopra S et al. Source-free domain adaptation with diffusion-guided source data generation. arXiv:2402.04929.

Q4: Limited qualitative analysis.

We conduct extra qualitative analysis in three parts: more generated samples, Grad-CAM of the SFDA model, and t-SNE for the SFDA model features. But according to the requirements of rebuttal, we will add them to the later version.

New Question: Comparison with DATUM.

Response

We thank the reviewer for pointing out this issue. Upon further examination of DATUM, we agree that its generation module is also applicable to the SFDA setting and is task-agnostic. We apologize for the oversight in our initial response in not including a comparison with DATUM. We have now added a comparison between our method and DATUM as follows.

We compare our method with DATUM on the Office-Home dataset. According to the DATUM paper, the method consists of three stages: (1) Employing training of DreamBooth to personalize the diffusion model by associating a unique token $V_∗$ with the appearance of the target domain. (2) Using the personalized diffusion model to generate a pseudo-target domain. (3) Training an existing UDA framework on the labeled source data and the generated unlabeled pseudo-target data. To align DATUM with the SFDA setting, we modify only the third stage. Specifically, we first train a source model using labeled source data, and then adapt the model to the target domain using the pseudo-target data generated by DATUM. The results are shown below. For reference, we also include the performance of the source model as a baseline for comparison.

The results demonstrate that:

(1) Compared with the source model, DATUM achieves better adaptation performance, demonstrating its effectiveness in the SFDA setting. This result suggests that diffusion-based domain adaptation methods like DATUM are capable of significantly improving adaptation performance. We consider DATUM a valuable and insightful work, as its design showcases the potential of leveraging diffusion models to generate pseudo-target data for SFDA.

(2) Compared with our method, the results demonstrate that DATUM performs significantly worse than our method. This may be due to the following two reasons:

• A key factor contributing to the superior performance of our method is the use of our Progressive Refinement Mechanism, which enables the SFDA model to progressively improve its performance through multiple iterations. In contrast, DATUM lacks such a dynamic update mechanism, which may limit its adaptation ability.

• We provide the detailed performance trajectory of our method as $r$ increases from 1 to 10. Notably, when $r = 1$, our method still outperforms DATUM. This demonstrates that even without the Progressive Refinement Mechanism, our method remains more effective than DATUM.

(3) The aforementioned results further indicate that the pseudo-target domain generated by our method is better aligned with the real target domain compared to that generated by DATUM. We attribute this to the following reason. DATUM relies on DreamBooth to learn the appearance of the target domain and map it to a unique token $V_∗$. However, for classification tasks, this process becomes challenging when the true class labels of the target data are unknown. According to the paper of DATUM, during DreamBooth training, the lack of class labels forces the use of vague prompts such as "a photo of a $V_∗$ object". This may cause the learned token to not only capture the domain-specific appearance of the target data, but also absorb semantic information related to the object class and even irrelevant background features. As a result, the generated pseudo-target images may exhibit limited alignment with the true target distribution.

Q1: Practicality, sensitivity to hyperparameters, and performance in critical systems

We appreciate the reviewer's insightful comment. In fact, we did not tune hyperparameters separately for each SFDA task. In lines 254-255, we intend to mean that we only explore different values of $E$ and $R$ once on Office-Home and fix $E$ to 0.01 and $R$ to 10 for all the experiments (except for ablation studies). The performance on benchmark datasets demonstrates that these hyperparameter values work for most cases. To validate the practicality of our methods in critical systems, we also tested the performance of our methods on real-world scenarios VLCS using unsupervised model selection without using target labels. We offer a detailed explanation in Response to Q1 by Reviewer mAks.

Q2: Minor redactional issues

We will correct these issues.

Q3: Class condition

Yes. We use the class name as the text prompt and use the pretrained Stable Diffusion without any finetuning.

Q4: Time cost and memory needs

We present a detailed analysis of the time cost and memory needs.

1. Time cost: The time cost can be mainly divided into two parts:

• SFDA model training. For any $r$-th refinement iteration with $r=1,2,...,R$, we train the SFDA model using the trust set and the manipulated non-trust set. The total number of samples of these two sets equals that of the target domain. We use supervised training here. The time cost mainly depends on the scale of the target domain, and remains constant as $r$ grows. We denote it as $t_{SFDA}$.

• Sample generation. For any $r$-th iteration refinement with $r=1,2,...,R$, the generation time for each sample remains constant, at about 1 second on a single NVIDIA Tesla V100. The generation time depends on the number of samples of the non-trust set, which is a subset of the target domain. Note that, with the growth of $r$, the size of the non-trust set declines rapidly, as shown in Figure 2. Thus, the total generation time for $R$ iterations equals the total number of generated samples $N_{total}$.

  The total training time $t$ can be estimated by $t= R*t_{SFDA}+N_{total}$. We take the 12 tasks in the Office-Home dataset shown in Table 1 as examples. We set $E=0.01$ and $R=10$ and run the tasks on a single NVIDIA Tesla V100. It takes about 7.4 hours for SFDA model training and 3.8 hours for generation to complete the total algorithm on average for each task, demonstrating an acceptable time cost. Note that our model also allows parallel training on multiple GPUs to further reduce the time cost.

2. Memory needs: The memory needs can be mainly divided into two parts:

• SFDA Model Training. During any $r$-th refinement iteration, the memory needs remain constant and approximately equal to those of standard supervised learning on the target domain, and will not accumulate as $r$ grows.
• Sample Generation. Our approach generates samples sequentially (one sample at a time). Thus, this part only requires sufficient memory to run the Stable Diffusion model. In conclusion, our method does not require high total memory needs, and can be run comfortably on a single GPU with 24GB of memory.

Q5: Random assignment of labels for non-trust samples.

From a performance perspective, we employ random label assignment, instead of using the original pseudo-labels of non-trust samples given by the SFDA model, for two key reasons. First, there are no obvious patterns between the original pseudo-labels and ground truth labels of non-trust samples, and the original pseudo-labels cannot provide valid semantic priors. Second, if we use the diffusion model to semantically transform non-trust samples to their original pseudo-labels, this approach could inevitably introduce the class imbalance issue in the manipulated non-trust set. Therefore, we adopt a uniform random label reassignment strategy as formalized in Equation (2). To validate this claim, we perform comparative experiments using the original pseudo-labels for generation. Due to time limits, the experiments were performed on the first 4 Office-Home tasks (Ar→Cl, Ar→Pr, Ar→Rw, and Cl→Ar). The results presented below demonstrate the validity of our choice.

From the perspective of reproducibility and stability, the results are fully reproducible as long as the random seed is fixed. In this case, the division between trust and non-trust sets becomes repeatable, and the sample ordering in the non-trust set's data loader is reproducible. Consequently, the new labels assigned to non-trust samples via Equation (2) are perfectly reproducible.

We sincerely thank the reviewer for the valuable insights. We will incorporate the analysis and results into the revised version of the manuscript.