Reducing Bias in Feature Extractors for Extreme Universal Domain Adaptation

We thank the reviewer for the detailed and constructive feedback. We address the comments below.

1. Novelty

Thank you for pointing out related works to better clarify our novelty and contribution. Please refer to General Response for details.

2. Use of old baselines

We appreciate the reviewers for suggesting up-to-date baselines. We have included the latest method, MLNet, in our experiments. Please see the General Response for details.

For [4] and [5], we believe that their settings and methods differ from the traditional UniDA task we focus on. Specifically, [4] explores generalized UniDA with baselines primarily incorporating active learning modules, while [5] addresses source-free UniDA, where source data is unavailable during adaptation. Nonetheless, we recognize that these settings align with our overarching aim to investigate challenging or less-explored scenarios in current research. Therefore, we have included discussions of these works in the Related Works section. For details, please refer to the updated PDF, where changes are marked in blue. For [3], we did not include it in the current version due to differences in architectures and time constraints. However, we believe that including the latest method, MLNet [7], is sufficient to demonstrate the effectiveness of our approach.

3. Experiments about other settings (e.g., PDA, OSDA, SDA)

UniDA is the generalized version of the mentioned settings, which provides a comprehensive evaluation framework. Existing works [3, 8] also evaluate only on the UniDA setting. However, we agree that focusing on a specific setting can help identify the bottlenecks of different methods. In fact, our experiments across varying SPCR rates effectively represent different settings: a low SPCR closely resembles OSDA, while an SPCR greater than 1 is similar to evaluating PDA. Therefore, we think our experiments on a wide range of SPCR are enough to show the robustness of the mentioned settings.

4. Ablation study on different SSL methods

We agree that the ablation study on different SSL methods is essential to demonstrate SSL’s effectiveness. We are in the process of running these experiments and will update you once finished.

[7] Lu et al. MLNet: Mutual Learning Network with Neighborhood Invariance for Universal Domain Adaptation. AAAI 2024.

[8] Chang et al. Unified Optimal Transport Framework for Universal Domain Adaptation. NeurIPS 2022.

We thank the reviewer for the detailed and constructive feedback. We address the comments below.

1. Clarity on experimental details in the figures

In Figure 1, the adversarial-based method corresponds to UAN, while the OT-based method corresponds to UniOT. The dataset used is Office31 (averaged over across tasks). In section 2.2, we report results on OfficeHome. We have updated the image and caption (marked in red) in the latest PDF.

In Figure 2, we have provided a clearer description in the updated version. Specifically, we aim to convey the following: (1) the distortion of target representations is significantly more pronounced in extreme UniDA (SPCR=4) compared to unsupervised DA (SPCR=0), and (2) incorporating SSL can effectively mitigate this distortion. Consequently, SSL proves to be more effective in extreme UniDA scenarios. Please see the updated PDF for details.

For $e_1$ and $e_2$, it means the basis of $\mathbb{R}^2$, i.e., [1,0] and [0,1]. For more information, please refer to [1], which is now cited in Section 3.3.

2. Incorporation of SSL to other methods

We have included the latest method, MLNet (Lu et al., 2024), in our latest results. Please refer to General Response for details.

3. Typos

Thank you for pointing out typos in our paper! We have addressed them in our latest version.

[1] Liu et al. Self-supervised Learning is More Robust to Dataset Imbalance. ICLR 2022.

We thank the reviewer for the detailed and constructive feedback. We address the comments below.

1. Balance between supervised and self-supervised loss

In Section 4.3, we discuss the sensitivity of hyperparameters for self-supervised loss. The results indicate stability within the range of 0.3 to 0.7. Therefore, we think that the two losses can be effectively balanced within a reasonable range.

2. Use of old baselines

We appreciate the reviewers for suggesting up-to-date baselines. We have included the latest method, MLNet, in our experiments. Please see the General Response for details.

3. Generalizability of SSL

The goal of UniDA is to achieve robust performance across arbitrary label-set distributions. While existing works perform well in general UniDA, they struggle in extreme UniDA. Our objective is to highlight the limitations of current methods in extreme UniDA and propose solutions to achieve robust performance across all label-set distributions. Therefore, we respectfully disagree with the suggestion that our work lacks novelty. Instead, we argue that our study offers a novel observation: SSL primarily addresses source-private bias, which has been insufficiently explored in UniDA literature.

We thank the reviewer for the detailed and constructive feedback. We address the comments below.

1. Clarity on experimental details in the figures

Thank you for pointing out several unclear parts in our paper. We have made the following improvements in the updated version:

• Figure 1: We have added details about the methods, dataset, and metrics used.
• Figures 2 and 3 (now merged into Figure 2): The descriptions and figures have been clarified to emphasize two key points:
  • The distortion of target representations is significantly more pronounced in extreme UniDA scenarios (SPCR=4) compared to unsupervised DA (SPCR=0).
  • Incorporating SSL effectively mitigates this distortion, making SSL particularly effective in extreme UniDA scenarios. Please refer to the updated PDF for further details.
• Figure 4 (now Figure 3): The results are based on the OfficeHome dataset with 65 classes.

2. Use of old baselines

We appreciate the reviewers for suggesting up-to-date baselines. We have included the latest method, MLNet, in our experiments. Please see the General Response for details.

Also, we did not include [1] in our experiments as the code is not publicly available.

3. Difference from prior works on applying SSL to DA

Please refer to General Response for details.

4. Consistency across settings

In Figure 2 (now Figures 2.a, 2.b, and 2.c), our goal is to focus on the target representation after training on source data. Therefore, target-private classes can be omitted for better visualization. However, we emphasize that the work remains within the scope of UniDA problems, as applying SSL to target-private classes can introduce noise—a challenge discussed in Section 3.2. In Figure 3 (now Figures 2.d, 2.e, and 2.f), we present the DA setting to show the distortion degree of target representations under low source-private classes. We have updated the descriptions in Figure 2 for better presentation.

5. Misleading terminology

Our general idea aligns with the reviewer, however, our terminology may have been misleading and caused confusion. Our intention was to emphasize that the predictions are biased toward source-private classes, which aligns with the reviewer’s point about the increased statistical likelihood of misclassification, rather than suggesting an inherent bias in the model itself. We have made the following adjustments in our paper.

• The title has been revised from “Reducing Bias in Feature Extractors” to “Reducing Source-Private Bias”.
• We have clarified the paragraph in Section 2.1 (marked in orange).

6. Difference between adversarial-based and OT-based method

Our hypothesis is that UniOT involves multiple complex modules, which may not be fully compatible. In contrast, adversarial-based methods, which rely solely on an adversarial loss design, are simpler to integrate. We leave a more comprehensive explanation for future research.

We thank the reviewer for the positive feedback. We address the questions below.

1. Difference from existing SSL methods on domain adaptation

Please refer to General Response.

2. Runs for the reported experiments

We follow most prior works (e.g., [1], [2]) on universal domain adaptation to conduct one run per experiment. Given the number of datasets and tasks in each dataset, we respectfully think it is sufficient to show the superiority and stability of our method.

Furthermore, we report the results with the error bar on Office31 with three random seeds in Section A.2. We can see that the standard deviation is relatively minor compared to the performance gain of our method.

[1] Lu et al. MLNet: Mutual Learning Network with Neighborhood Invariance for Universal Domain Adaptation. AAAI 2024.

[2] Chang et al. Unified Optimal Transport Framework for Universal Domain Adaptation. NeurIPS 2022.

We thank the reviewers for their constructive feedback on our experiments. In response, we have completed the remaining experiments on MLNet and included additional analyses to further support the findings from the toy experiments:

1. We ran MLNet with SSL on all four datasets, and the results consistently align with our previous findings, demonstrating that SSL reliably enhances performance (see Tables 1 and 2). We respectfully believe that the observed improvements across three different paradigms effectively demonstrate the robustness and effectiveness of SSL.
2. Additionally, we conducted a singular value spectrum analysis on a real dataset to further validate our hypothesis that SSL is particularly effective in extreme cases. These findings corroborate the results from our toy experiments (see Section 4.3 and Figure 6).