Towards Debiased Source-Free Domain Adaptation

1. The transition between the second and third paragraphs of the Introduction part seems confusing. In Line046, the authors claimed that "The use of such spurious correlations may cause samples of different classes to have the same prediction, as they are erroneously considered among the nearest neighbors". Then, in the third paragraph of the introduction, the authors stated some experimental observations w.r.t. the confident correct samples (i.e., easy split) and confident wrong samples (i.e., hard split). However, I did not see a clear connection between the motivation and the observations of such experiments. I understand that Appendix A.3 provided an explanation, but I believe that the introduction part needs to be further improved, especially needs to give straightforward support to your claim at Line046.

2. About the relationship between the class spaces of SFDA (i.e., within S/T domains) and reference pre-training datasets (e.g., ImageNet). Between Line103 and Line104, the authors claimed that "(b) classes in T can be considered novel to $f_r$, as it is pre-trained on a different dataset". However, in Lines 210-212, the authors also stated that "2) Its pre-training dataset should be diverse to learn general knowledge, while also related to the SFDA task to allow knowledge transfer". The above two expressions seem contradictory. I wonder what kind of datasets that we should expect to pre-train the reference model $f_r$. In this manuscript, the reference model is pre-trained on ImageNet dataset, whose label space has some overlaps with that of Office-Home, VisDA and DomainNet datasets. How about using a pre-training dataset that has no overlap with these test sets? Could the author provide more ablation studies w.r.t. different pre-training datasets? I believe this is also crucial to understand the proposed method.

3. Based on the above point, we can consider a more general question: how to guarantee the debiasing extent of the reference model such that it is debiased enough to be leveraged by the source model? I believe that the reference model may also be biased if we cannot choose the right one. However, this point has not been discussed in the current manuscript.

4. I noticed that only two methods (i.e., AaD and SF(DA)^2) were integrated with the proposed debiasing algorithm. Could the authors introduce and discuss more methods to support the effectiveness of their algorithm?

5. Some notations used in this manuscript are confusing:

   • $L_{aad}$ in Eq. (1) and $L_{AaD}$ in Eq.(4) are not consistent;
   • What is $f_g$ in Line268? Do you mean $f_g=g \circ f_r$?
   • Between Line277 and Line282, I wonder if $g(\mathbf{x})$ here indicates $g(f_r(\mathbf{x}))$?

• Regarding H-score: While the H-score computes the probability mass on incorrect classes. As the predictions become more underconfident, the H-score can still decrease. However, the entropy on the test domain with further adaptation epochs will increase or plateau if accuracy is not improving with DeCo as indicated in the benchmarks. Authors should include entropy curves also in their analysis.
• It would be interesting to have a detailed ablation of how much the accuracy of the large-scale pre-trained model on the test domain affects DeCo. In a hypothetical setup, is DeCo able to match the performance of the open-source model?
• Does DeCo debias the source model to leave only invariant features? It would interesting to compare DeCo to an invariant risk minimizer across domains.

Questions

1. On page 5, line 225, "model" is misspelled as "mdoel".
2. I can understand that freezing the reference model and training the clustering head from scratch, which is independent of the source domain model, can prevent the clustering model from acquiring incorrect knowledge from the source domain model. However, when the reference model is poor, clustering will produce many indistinguishable samples, and simply matching the classification boundaries of the source domain model may introduce errors. In particular, the selection of positive and negative pairs in the contrastive loss completely relies on the clustering model, which may further exacerbate the mistakes.
3. In the standard InfoNCE method, the denominator of the contrastive loss includes positive sample pairs. Note that the denominator in Eq. (3) only includes negative sample pairs. What is the advantage of this design?
4. On page 5, line 268, the symbol $f_g$ appears without explanation.
5. The authors split the dataset based on hardness scores, generating a series of sample sets with varying degrees of domain bias to assess the domain adaptation performance under different biases. However, the authors' claim on page 3, line 114, that they propose "a novel HAcc metric to assess source bias" is inaccurate. HAcc is used to evaluate the average effect of domain adaptation methods under different domain bias scenarios, rather than "assess source bias". This representation is worth adjusting.
6. As one of the contributions highlighted by the authors in the introduction, the manuscript's description of H-score and HAcc is insufficiently detailed. The authors mention on page 7, line 367, "Additional details are provided in the Appendix", but the description in the appendix is still unclear. Moreover, note that most experiments on HAcc are conducted on the VisDA dataset, and more experiments on other datasets may help evaluate the generality of this metric.
7. Figure 6 does not explicitly label $\beta_1$ and $\beta_2$. I suggest modifying the figure to make it more intuitive.

• Methodology and Experiment: In Eq. (3), can the alignment term be combined with the AaD method, potentially replacing the infoNCE loss with the dot product in AaD? Additionally, what are the final distributions of positive and negative neighbor counts across different datasets?

• Expression:

  • The definitions of H-scores and HAcc, along with Figure 4 and the debiasing analysis, are somewhat confusing. For example, in lines 323 and 364, it is stated that the subsequent split contains fewer samples with higher H-scores, where higher H-scores represent samples that are more distinct or challenging compared to the source domain. Shouldn't the earlier splits be more difficult? Furthermore, in the right panel of Figure 4, what does "h" represent? Does it refer to HAcc (defined as mean-class accuracy in line 365) or the average H-score for a specific split? These details make the debiasing section somewhat unclear.

  • Motivation: Figure 1 illustrates the varying performances of different SFDA methods on easy and hard samples, defined by the similarity between target data points and the source model’s average representation. This performance gap suggests the presence of distribution shift, but how does it relate to spurious correlation or source bias? Please correct me if I’ve misunderstood.