SF(DA)$^2$: Source-free Domain Adaptation Through the Lens of Data Augmentation

Thanks for the effort to address my concerns. I improve the score from 5 to 6.

Q5. In Fig. 3, with only FD or IFA on SNC will deteriorate the performance. The authors posit some hypothesis for explanation, while I think there may need more results on several other datasets to prove the assumption. By the way, I think there is no information we could get from Fig. 3(b), since it is neither not consistent with the corresponding accuracy, nor cable to guide the hyperparameter tuning.

Response: Thanks for your insightful question! IFA identifies directions in the feature space that encodes class semantic information and augments features along those directions. If the feature space is entangled (meaning different directions in the space do not clearly correspond to distinct class semantics), performing feature augmentation in such an entangled feature space may not preserve class information, bringing harm to performance. Applying only FD loss introduces additional constraints to the model by disentangling the feature space. Therefore, the absence of a term utilizing the disentangled space could result in a performance drop.

The VisDA-RSUT dataset used for the ablation study has highly imbalanced classes, resulting in an extremely entangled feature space where using only FD or IFA degrades performance. In contrast, in the VisDA dataset with balanced classes, there was no significant performance drop, with results showing SNC: 87.9%, SNC+IFA: 88.0%, and SNC+FD: 87.9%.

In our analysis presented in Section 3.4, we found that the denominator of the IFA loss encourages the kernel space of the model's Jacobian matrix to align with the principal direction of the tangent space of the augmented data manifold, which is in accordance with a finding on data augmentation [2]. Therefore, by comparing SNC (red) and SNC+IFA (yellow) in Fig. 3(b), we can observe a decrease in cosine similarity due to implicit feature augmentation. Moreover, comparing SNC+IFA (yellow) with SNC+IFA+FD (blue) shows that the addition of the FD loss further reduces cosine similarity, indicating that the FD loss facilitates more effective feature augmentation. While SNC+FD (green) might seem redundant for the explanation, it was included to align with Fig. 3(a).

Q6. There are two extra hyperparameters for FD and IFA losses, and there seems no ablation study for those hyperparameters. In the paper, it mentioned that a1 and a2 are set to different values for different dataset. I conjecture they may be sensitive on some datasets and hard to tune in the unsupervised way.

Response: Thanks for the suggestion! To report the optimal performance for each dataset, we conducted slight hyperparameter tuning. However, in six datasets, we experimented with only two settings for each hyperparameter ($\alpha_1$=1e-4, 1e-3) ($\alpha_2$=1, 10), achieving state-of-the-art performance. The model's performance is not sensitive to hyperparameters $\alpha_1$ and $\alpha_2$. As you suggested, we newly conducted an ablation study on $\alpha_1$ and $\alpha_2$, and the results on the VisDA dataset are presented in the following tables:

We added the additional ablation studies in Tables 10 and 11 in the revision.

We hope that our response has addressed your concerns. Thank you very much!

[1] The effects of regularization and data augmentation are class dependent, NeurIPS 2022.

[2] A Data-Augmentation Is Worth A Thousand Samples: Analytical Moments And Sampling-Free Training, NeurIPS 2022.

We thank you for the insightful comments and suggestions. We have addressed each of your questions below.

Q1. The proposal of SNC is not new. SFDA method AaD already introduced quite similar method (in both motivation and final objective form). More specifically, the proposed SNC just takes [1, Spectral CL] into the SFDA task.

Response: AaD and our method have distinct motivations. We define an augmentation graph based on the given target features. Performing spectral clustering on this augmentation graph identifies meaningful partitions or clusters in the prediction space, which is equivalent to minimizing the SNC loss. While our method and AaD share similar forms for their final loss functions, as you pointed out, our research introduces an innovative perspective on SFDA and significantly expands the scope of research in this area, as acknowledged by Reviewer 44EQ.

Additionally, while we were inspired by the spectral contrastive loss (Spectral CL) to find partitions of our augmentation graph, there are distinct differences between Spectral CL and our SNC loss. Spectral CL utilizes augmented data as positive pairs, and data augmentation requires prior knowledge about the dataset. For instance, transforming the color of a lemon to green would turn it into a lime, thus changing the class of the data. Such augmentation that fails to preserve class information can introduce bias to the model, potentially bringing harm to the model's performance [1]. Hence, while Spectral CL requires class-preserving augmentations based on prior knowledge, our SNC loss determines positive pairs without needing explicit augmentation and prior knowledge.

Q2. Several results for other methods in the main tables are not identical to the results in their original papers. It is not good to report the reproduced results of own without mentioning it in the paper (though there are always software environment difference).

Response: For the VisDA dataset, we reproduced SHOT, NRC, CoWA-JMDS, AaD using their official codes and the pretrained models released by SHOT, and already mentioned it in Section 4.1 of our original manuscript (highlighted in red).

Q3. The introductions of IFA and FD are the main contribution of the paper. In my understanding, IFA loss plays similar role as the first term in SNC loss, while FD loss plays the similar role as the second term of SNC loss (here FD loss may be better as it quantifies the similarity degrees between different categories). I am just wondering what if only using FD and IFA losses for adaptation.

Response: The IFA loss is derived as the upper bound of the expectation of the first term of the SNC loss. However, the FD loss is not directly related to the second term of the SNC loss. Therefore, we can expect that using only IFA and FD losses may not correctly identify the partition of the augmentation graph, leading to inadequate adaptation. This is confirmed through experiments on the VisDA dataset, which demonstrate an accuracy of 51.4%, in line with our expectations.

Q4. In Fig. 2, the results show that the proposed method can not improve SHOT, while in AaD it indicates that SHOT is actually conducting similar operation as NRC and AaD. Do authors have clear explanation about the phenomenon, is it due to the pseudo labeling used in SHOT?

Response: Thank you for the good question. IFA and FD losses utilize local neighbor information in the feature space, while NRC and AaD share a similar philosophy. In contrast, SHOT aligns the target features with global source hypotheses for each class. In the AaD paper, the following is mentioned: "In a nutshell, compared to SHOT and BNM, our method first considers the local feature structure to cluster target features." Moreover, IFA and FD losses make use of class-wise covariance matrices, while SHOT does not. This difference in approach between SHOT and IFA can contribute to a decrease in performance.

Q4. In VisDA, Domain-Net, and PointDA-10 datasets, IFA loss has a weight of 1e-4, while FD loss has a weight of 10, resulting in a limited impact of IFA loss on the training process compared to FD loss. Can the inclusion of the other two datasets, apart from the VisDA-Rust task, demonstrate the significance of this new data augmentation technique?

Response: Since the two losses serve as different types of regularizers, it's difficult to claim that IFA's impact is small solely due to the difference in weight. (For instance, the hyperparameter for weight decay commonly used in neural networks typically has a scale around 1e-3, whereas the hyperparameter for gradient penalty used in GANs often uses a scale of about 10. However, employing both doesn't necessarily mean that the impact of the gradient penalty is substantially stronger.) As depicted in Figure 3(a), even adding just 1e-3 of IFA results in a performance increase of over 5%p. While consistent performance improvements were observed in other datasets (from 87.9% to 88.1% in VisDA and from 68.0% to 68.3% in DomainNet), the enhancement is particularly significant in the highly imbalanced dataset. This notable performance boost in imbalanced datasets is believed to be due to the FD loss enabling distinct covariance matrices for each class and utilizing this for feature augmentation, thereby allowing minority classes to learn in distinct directions within the feature space.

Q5. Why are there no comparisons of NRC and AaD methods on the DomainNet dataset? Similarly, why are the results of AaD not compared on the PointDA-10 dataset?

Response: We appreciate your suggestion. Following your suggestion, we newly compared our method with NRC and AaD on the DomainNet and PointDA-10 datasets and the results are presented in the following tables:

We added the additional ablation studies in Tables 2 and 4 in the revision.

Q6. In the Introduction, the authors mention that recent studies have used data augmentation techniques to improve adaptation performance. However, there is a lack of more relevant work cited, such as [2] and [3].

Response: Thank you for suggesting additional related work. In the revision, we included these related works in Appendix D.3 (highlighted in red).

We hope that our response has addressed your concerns. Thank you very much!

We thank you for the insightful comments and suggestions. We have addressed each of your questions below.

Q1. What are the differences between the proposed SNC loss and the loss function in the existing AaD [1]? As far as I know, the purpose of the loss in AaD is to encourage similar neighbors to be close to each other and dissimilar neighbors to be far apart within a batch. This is similar to the SNC loss proposed in this paper. The author needs to clarify the differences between these two loss functions.

Response: AaD and our method have distinct motivations. We define an augmentation graph based on the given target features. Performing spectral clustering on this augmentation graph identifies meaningful partitions or clusters in the prediction space, which is equivalent to minimizing the SNC loss. While our method and AaD share similar forms for their final loss functions, as you pointed out, our research introduces an innovative perspective on SFDA and significantly expands the scope of research in this area, as acknowledged by Reviewer 44EQ.

Q2. Are the estimated covariance matrices accurate in the IFA and FD losses, given the domain shift between the source and target domains? The estimation of these matrices relies on the output probabilities of the target data. However, this estimation process may introduce biases, especially when there are significant dissimilarities between the classes in the target domain and those in the source domain. As a result, the probabilities associated with these classes in the target domain, as generated by the source domain model, may be greatly reduced. Consequently, there is a potential risk of losing the capability to effectively learn these specific classes during the subsequent learning process, presenting a challenge to rectify the situation.

Response: Thank you for the good question. The IFA loss estimates the covariance matrix using the given target features and pseudo-labels for implicit feature augmentation. However, during the initial adaptation phase, the feature space tends to become entangled due to domain shift, which may lead to an inaccurate covariance matrix and potentially degrade model performance. Therefore, we designed the influence of the estimated covariance matrix to be small during early training and gradually increase as training progresses (as indicated by \lambda in Equation 5). Additionally, to further disentangle the feature space, we introduced the FD loss and regularized the features to distribute in different directions for each class.

If a pretrained model has been trained well, each direction in the feature space is likely to contain valuable class semantic information, and one might expect the cosine distances between groups of features belonging to the same ground-truth class to be close. Even in cases where domain shift may lead to incorrect assignments, the FD loss can be expected to reduce the cosine distance between groups of features from the same class during the process of disentangling the feature space.

However, for two groups of features belonging to the same ground-truth category to be assigned to different partitions, one group of features must possess features that are more similar to those of other classes than to its ground-truth class. In such scenarios, it is challenging to achieve unsupervised domain adaptation without supervision.

Q3. IFA operations have shown promise as a universally applicable data augmentation technique in unsupervised tasks. Can IFA be applied to other existing unsupervised tasks? Can you provide examples to validate its effectiveness?

Response: We appreciate your suggestion. The IFA loss is derived from an augmentation graph in the feature space, taking into account a pretrained model and domain shift. Applying this to different unsupervised tasks would require a redefinition of the augmentation graph and the derivation of a new loss tailored to the specific task. Therefore, we consider this to fall beyond the scope of this paper. Nonetheless, we appreciate your suggestion and believe it could be an interesting avenue for future work.

Dear Reviewer s4wJ,

Thank you again for the insightful comments and suggestions! Given the limited time remaining, we eagerly anticipate your subsequent feedback. It would be our pleasure to offer more responses to further demonstrate the effectiveness of our methodology.

In our previous response, we have thoroughly reviewed your comments and provided responses summarized as follows:

• Explained the differences between SNC loss and AaD loss. SNC loss involves spectral clustering on the augmentation graph based on target features, offering a new perspective on SFDA.
• Explained how our method mitigates bias in the estimated covariance matrices, enhancing accuracy in domain adaptation.
• Conducted the new ablation study on the IFA loss and highlighted the notable performance improvement in the imbalanced dataset, facilitated by learning distinct feature directions.
• Conducted the new comparisons of NRC and AaD methods on the DomainNet and PointDA-10 datasets, showing the superior performance of our method.
• Included more related works on data augmentation techniques for domain adaptation.

We hope that the provided new experiments and the additional explanation have convinced you of the merits of this paper. If there are additional questions, please feel free to let us know.

Additionally, we wish to express our gratitude once again to you for your insightful feedback. Incorporating your suggestions has undoubtedly enhanced the clarity and robustness of our work.

We deeply appreciate your time and effort!

Best regards, Authors

Thanks for the effort to address my concerns. I would like to improve the score.

Dear reviewer s4wJ

The interactive discussion phase will end in few hours, and we cannot have discussions with you anymore after the deadline. We wish that our response has addressed your concerns, and turns your assessment to a more positive side. Please let us know if there are any other things that we need to clarify.

We thank you so much for your helpful and insightful suggestion.

Best, Authors

We thank you for the insightful comments and suggestions. We have addressed each of your questions below.

Q1. A more detailed description of the relationship between the augmentation graph and section 3.4 is warranted, as there appears to be no explicit utilization of G_hat in section 3.4.

Response: G_hat is an instance of the augmentation graph defined from the given target features, and obtaining the partition of G_hat on the prediction space leads to the derivation of the SNC loss. Implicit feature augmentation (section 3.4) goes beyond the given target features to approximate the population augmentation graph (G), and derive the IFA loss by obtaining a closed-form upper bound for the first term of the expected SNC loss in a tractable way from G. In other words, G_hat is defined from a finite number of given target features, while IFA has the effect of augmenting an infinite number of target features.

Q2. How does equation (4) influence G_hat, and what is the impact of equation (4) and G_hat on the overall optimization process, given that equation (4) only considers cosine-similarity neighborhoods and mini-batch statistics?

Response: Thank you for the good question! G_hat is an instance of the augmentation graph defined using cosine distances between the given target features. Performing spectral clustering on this augmentation graph finds meaningful partitions or clusters in the prediction space, which is equivalent to minimizing the SNC loss (Equation 4). In other words, minimizing the SNC loss allows us to discover meaningful clusters of target features in the prediction space, and an optimal solution would result in one-hot vectors for each class. Therefore, successfully leveraging the relationships between target features enables effective source-free domain adaptation.

Q3. Within spectral clustering methods, could the scenario wherein two groups of features belonging to the same ground truth category are erroneously assigned to separate partitions ultimately lead to a degradation in model performance, as they may be pushed apart within the feature space?

Response: If a pretrained model has been trained well, each direction in the feature space is likely to contain valuable class semantic information, and one might expect the cosine distances between groups of features belonging to the same ground-truth class to be close. Even in cases where domain shift may lead to incorrect assignments, the FD loss can be expected to reduce the cosine distance between groups of features from the same class during the process of disentangling the feature space.

However, for two groups of features belonging to the same ground-truth category to be assigned to different partitions, one group of features must possess features that are more similar to those of other classes than to its ground-truth class. In such scenarios, it is challenging to achieve unsupervised domain adaptation without supervision.

Q4. It would be much appreciated if some DA surveys and reviews are included to better show the background and related work, such as "A Comprehensive Survey on Source-free Domain Adaptation", "A Review of Single-Source Deep Unsupervised Visual Domain Adaptation".

Response: Thank you for suggesting additional related work. In the revision, we included these related works in Appendix D.3 (highlighted in red).

Q5. The compared baselines are obviously insufficient. For source-free DA, some typical and latest methods are not compared, such as SHOT++. For source-available DA, the compared baselines are too old. Some more baselines published in 2022 and 2023 are required but missing.

Response: We already compared our method with SHOT++ in Table 3 of our manuscript and our method outperforms SHOT++ by a large margin (2.3%p). We also compared our method with the latest methods up to 2022, both in the main text and the appendix. Additionally, we newly compared our method with NRC and AaD on the DomainNet and PointDA-10 datasets and the results are presented in the following tables:

We added the additional ablation studies in Tables 2 and 4 in the revision.

We hope that our response has addressed your concerns. Thank you very much!

Dear Reviewer 44EQ,

Thank you again for the insightful comments and suggestions! Given the limited time remaining, we eagerly anticipate your subsequent feedback. It would be our pleasure to offer more responses to further demonstrate the effectiveness of our methodology.

In our previous response, we have thoroughly reviewed your comments and provided responses summarized as follows:

• Explained the relationship between the augmentation graph and implicit feature augmentation.
• Explained the relationship between the augmentation graph and SNC loss, and their impact on the overall optimization.
• Explained how our method mitigates incorrect partition assignments, enhancing accuracy in domain adaptation.
• Included the suggested review and survey on domain adaptation.
• Conducted the new comparisons of recent methods (NRC and AaD) on the DomainNet and PointDA-10 datasets, showing the superior performance of our method.

We hope that the provided new experiments and the additional explanation have convinced you of the merits of this paper. If there are additional questions, please feel free to let us know.

Additionally, we wish to express our gratitude once again to you for your insightful feedback. Incorporating your suggestions has undoubtedly enhanced the clarity and robustness of our work.

We deeply appreciate your time and effort!

Best regards, Authors

We thank you for the insightful comments and suggestions. We have addressed each of your questions below.

Q1. The ablation study: why do two regularizations need to be applied simultaneously?

Response: Thanks for your insightful question! Data augmentation that fails to preserve class information can introduce bias to the model. For instance, transforming the color of a lemon to green would turn it into a lime, potentially bringing harm to the model's performance [1].

IFA identifies directions in the feature space that encodes class semantic information and augments features along those directions. If the feature space is entangled (meaning different directions in the space do not clearly correspond to distinct class semantics), performing feature augmentation in such an entangled feature space may not preserve class information, bringing harm to performance. Applying only FD loss introduces additional constraints to the model by disentangling the feature space. Therefore, the absence of a term utilizing the disentangled space could result in a performance drop.

However, a synergistic effect arises when IFA loss and FD loss are jointly employed. The disentangled feature space obtained via FD loss allows the estimated covariance matrix to provide directions for class-preserving feature augmentation. Subsequently, the IFA loss performs the feature augmentation, leading to a notable performance improvement.

Q2. the confusion matrix is estimated in an online way. How to guarantee its quality/accuracy?

Response: The IFA loss estimates the covariance matrix using the given target features and pseudo-labels for implicit feature augmentation. However, during the initial adaptation phase, the feature space tends to become entangled due to domain shift, which may lead to an inaccurate covariance matrix and potentially degrade model performance. Therefore, we designed the influence of the estimated covariance matrix to be small during early training and gradually increase as training progresses (as indicated by \lambda in Equation 5). Additionally, to further disentangle the feature space, we introduced the FD loss and regularized the features to distribute in different directions for each class.

We hope that our response has addressed your concerns. Thank you very much!

[1] The Effects of Regularization and Data Augmentation are Class Dependent, NeurIPS 2022.

Dear Reviewer 7eAE,

Thank you again for the insightful comments and suggestions! Given the limited time remaining, we eagerly anticipate your subsequent feedback. It would be our pleasure to offer more responses to further demonstrate the effectiveness of our methodology.

In our previous response, we have thoroughly reviewed your comments and provided responses summarized as follows:

• Explained the necessity of simultaneous application of two regularizers (IFA and FD losses).
• Explained how our method ensures the quality of the estimated covariance matrix, enhancing accuracy in domain adaptation.

We hope that the provided new experiments and the additional explanation have convinced you of the merits of this paper. If there are additional questions, please feel free to let us know.

Additionally, we wish to express our gratitude once again to you for your insightful feedback. Incorporating your suggestions has undoubtedly enhanced the clarity and robustness of our work.

We deeply appreciate your time and effort!

Best regards, Authors