Enhanced Label Propagation through Affinity Matrix Fusion for Source-Free Domain Adaptation

Thank you for your time and suggestive comments. We will improve the paper to include the discussions below.

W1. We have incorporated the suggested papers into the discussion of the related works in Section A.5. We want to highlight that A$^2$LP relies on both labeled source data and unlabeled target data for label propagation. In the context of SFDA, where only target data is available, the affinity matrix is inherently noisier due to the model's lack of adaptation to the target domain. We address this challenge by exploiting both current and historical models to construct a more precise affinity matrix—a crucial factor for the success of label propagation, especially under SFDA conditions.

On the other hand, CtO [b] used a local clustering approach instead of label propagation (even though the title contains label propagation), which is similar to NRC and AaD. However, as noted in the main paper, refining pseudo-labels solely based on nearest neighbors fails to effectively exploit the global structure of target samples, resulting in suboptimal outcomes.

W2. Thank you for the suggestion. We have provided the results for Baseline + HAF without strong data augmentation in the ablation study, and the detailed outcomes are presented in Table 4.

W3. Our approach has integrated several established techniques, including BN adaptation, uncertainty weight, and diversity loss. While these techniques are not the primary focus of this paper, we acknowledge the importance of understanding their impact. We have included an ablation study in Appendix A.5 to address this concern.

In addition, we also check the effectiveness of BN adaptation when integrated with existing methods, as summarized in the table below. Please note that the code for some baseline models, such as CoT and C-SFDA, is not publicly available. Instead, we provided the results for AaD and NRC, which are the best-performing models with accessible code. Note that AaD and NRC do not benefit significantly from BN, and our method maintains strong performance even without BN. The integration of BN adaptation is particularly synergistic with our approach, as it helps generate more accurate similarity measures and affinity matrices. Since our label propagation process relies on the affinity matrix for pseudo-label refinement, the improved accuracy facilitated by BN adaptation significantly improves the effectiveness of our method. In addition, incorporating BN adaptation into SFDA is also a novel contribution in our paper.

W4. We have included an ablation study on the impact of using a single strong augmentation in Table 4 in the main paper. Additionally, the outcomes of incorporating three augmentations are included in the table below. The results indicate that integrating more strong augmentations during the self-training stage yields only marginal improvements. For efficiency considerations, we opt to use two strong augmentations.

W5. The details regarding hyperparameters are outlined in Section A.6. Notably, incorporating diversity loss for VisDA adversely affects model performance. Additionally, the removal of diversity loss has a more pronounced impact on the Office-Home dataset while causing only minor changes in performance for Office31.

Q2. Thank you for the suggestion! Indeed, the self-training framework in our approach shares some similarities with BYOL. As you mentioned, there is some distinction as we focus on the prediction space. We will investigate this approach in our future study on SFDA.

Thank you for your time and suggestive comments. We have addressed the questions below.

W1. Thank you for pointing out the need for a more detailed discussion of related works. We have added the discussion of relevant works in Section A5.

In addition, we agree that the self-training framework is a prevalent choice in SFDA and many existing SFDA methods adopt a self-training framework. We emphasize that our distinctive contribution lies in the innovative combination of Label Propagation with the self-training framework, resulting in enhanced adaptation performance.

Concerning CtO[1], it used a local clustering approach instead of label propagation (even though the title contains label propagation). This approach shares similarities with NRC and AaD as they all aim to adjust model predictions based on their immediate neighbors, emphasizing local structure. However, our method employs label propagation, which exploits both global and local structures of the target samples. Another significant contribution involves integrating the historical model into the label propagation process. While leveraging historical models is not novel in semi-supervised learning, direct ensembling of historical models for pseudo-label refinement in SFDA often yields suboptimal results due to the lack of adaptation of historical models to the target domain. Motivated by the observation that similar samples form clusters in the feature space using both current and historical models, we leverage a combination of both to construct a more accurate affinity matrix. This integration is essential for enhancing the effectiveness of label propagation and, consequently, improving overall performance.

Q1. For historical models, there is no necessity to store the neighbor banks; only the affinity matrix needs to be preserved. The affinity matrix itself is highly sparse, occupying approximately 6MB for VisDA and 2MB for Office-Home. In an effort to reduce both storage consumption and input/output operations (IO), we conducted experiments by storing the affinity matrix every two epochs and fusing the matrices from time step $\lfloor(t-m)/2\rfloor * 2$, resulting in equivalent outcomes.

Q2. Indeed, this trend is expected. As the training advances, the model converges gradually, leading to minimal disparities between the current and historical models. Once label propagation and model fine-tuning reach an equilibrium, neither label propagation nor historical affinity fusion will further enhance the model.

Q3. Sorry for the confusion. This discrepancy arises from the design of our implementation of historical affinity fusion (HAF). For time step $t$, where $t<m$, we fuse the affinity matrix with the initial affinity matrix obtained at epoch 0. Thus, for epoch 0, the affinity matrix is fused with itself. The figure plots the ratio/percentage of connected samples, weighted by similarity, sharing the same label. Consequently, the fused affinity matrix at epoch 0 assigns higher weights to neighbors with greater similarity compared to the vanilla affinity.

Thank you for your comments. We have addressed the questions below.

W1. Thank you for your valuable feedback. We have already cited these two works in our original submission. In addition, we have added a detailed discussion of these relevant works in Section A.5. While acknowledging the similarities between HCL and our proposed method, it's essential to highlight two fundamental distinctions: 1) HCL relies on the agreement between current and historical models to access the confidence of the pseudo-label without involving a specific refinement process. In contrast, our approach adopts label propagation for pseudo-label refinement, an important step that significantly improves the accuracy of the adaptation process. 2) Our method goes beyond a simple reliance on historical models; we specifically leverage them to construct a more precise nearest-neighbor graph. This strategic incorporation is integral to the effectiveness of label propagation and underscores the nuanced ways in which historical models contribute to the method's performance.

W2. The approach outlined in "Temporal Ensembling for Semi-Supervised Learning (Laine, S. and Aila, T., 2016)" ensembles predictions from current and historical models and uses the ensembled prediction as the target or pseudo-labels. However, in the context of SFDA, where the model is not fully adapted to the target domain, directly using the ensembled source model prediction as pseudo-labels is not viable. In this scenario, label propagation becomes crucial for refining pseudo-labels. As articulated in the main paper, we refine the pseudo-labels using the equation $Q^{\ast} = (I - \alpha S) ^{-1} P$, where $S$ is the normalized affinity matrix, $Q^{\ast}$ and $P$ represent the final refined pseudo-label matrix and the raw prediction matrix respectively.

To explore various ways of integrating historical models, we compare our baseline model without Historical Affinity Fusion with the following two approaches: (1) Following the methodology of Laine and Aila (2016), we use the moving average of predictions from the current and historical models as entries for $P$ (Baseline+MAP). (2) We utilize both the current and historical models to build the affinity matrix (Baseline+HAF), which is our adopted final solution. Intriguingly, compared to our baseline, approach (1) may even degrade model performance, while approach (2) consistently improves it. We conjecture that in the source-free domain adaptation scenario, historical models are inadequately adapted to the target domain, leading to noisy predictions and potential confirmation bias, making direct use of the ensembled predictions for $P$ less effective. On the other hand, historical models prove beneficial for improving the affinity matrix, as similar samples tend to form cohesive clusters in the feature space using either source or target models.

W3. Sorry for the confusion. The baseline in Table 4 represents our baseline approach with vanilla label propagation for pseudo-label refinement, without incorporating historical affinity fusion or employing strong augmentation in the self-training stage (outlined in the caption of Table 4). It's expected that our baseline model should improve over the source model by a large margin.

Thank you for your time and suggestive comments. I have addressed the questions below.

W1. Thank you for your valuable suggestions. We have opted to utilize only the previous model $h_{t-m}$ for both efficiency and effectiveness considerations. The model from the more immediate history tends to be more similar to the current model. We have conducted experiments that fuse the current model with two historical models, including the model at time $t-m$ and the model at time $t-\lfloor m/2 \rfloor$. The corresponding results are presented in the table below, from which we did not observe a significant improvement by integrating more historical models.

W2. The use of the negative dot product is inspired by previous works NRC and AaD, where the dot product loss has proven effective in the context of SFDA. Additional rationale for the use of the negative dot product loss is detailed in Appendix A.2. Particularly, it is a combination of squared error loss and a component that encourages prediction discriminability. Below, we have compared the average accuracy of our approach with both negative dot product loss and cross-entropy loss across all datasets, which proves the effectiveness of negative dot product loss in the context of SFDA.