Principled Federated Domain Adaptation: Gradient Projection and Auto-Weighting

”What if one negates the direction of the projections of negative source gradients (e.g., the projections of g_s3 and g_s4 in Figure 1)?”

• Thanks for the interesting idea! We performed the following experiments and the results show that negating the direction does not impact the performance much.

On Fashion-mnist with label shifts

On Fashion-mnist with noisy features

On Cifar-10 with noisy features

”The behaviors of FedDA in Figure 2(a) and Figure 2(c) seem contradict to each other. Is it trivial? Or may I have a clarification?”

• Thanks for the question and we appreciate the chance for clarification. The reason behind the distinct behaviors of FedDA in Figure 2(a) and 2(c) is due to the experiment settings. For 2(a), we have noisy features, so when the shifts are larger FedDA’s performance becomes worse. In this setting, the variances on the target domain do not change much. However, for 2(c), the target variance decreases as the domain shift (eta -> 0) grows bigger (easier to predict with fewer classes), FedGP and auto methods will benefit from this with higher performance when the shifts are large. However, FedDA will have a very low performance with large shifts (eta closer to 0), which makes it have a different behavior than Figure 2(a).

Thank you for reviewing our manuscript and we appreciate your feedback for refining the final version. We address your concern and question in the following.

”…, federatively speaking, paying the whole attention to ONE target client might not align with the original intention of studying federated learning’’

• Thank you for this comment. Our view is that the setting remains federated in the sense that there is a central server coordinating device toward a shared objective. However, we are most interested in the applied problem setting. An example that motivates our federated domain adaptation perspective is that: as a new client with limited local data (and with domain shift) shows up, the goal for the whole federated system is coming up with a model for the new client using federated data across all the clients.

  In general, the settings considered in the paper are different from the original FL setting as the target client has a limited amount of data - which means if we do normal FL, it will gain fewer benefits from the federation. As we show in the paper, personalized baselines do not work well in these settings. However, these settings still require federated collaboration. For the target client with limited data, our framework shows how other clients with sufficient data can collaborate and help it train a better model. Also, our work focuses on Domain Adaptation, which usually has one target domain to be adapted to. Further, we acknowledge the possibility of extending the current framework to multiple targets, which serves as a promising yet non-trivial future work.

”Given the federated learning nature, multiple target clients seem more practical. How would the proposed method scale with the number of target clients?”

• Thanks for the great question! We can potentially have several copies of the global model for each target client in the system, where each target client learns their best-personalized model simultaneously from other source clients in the system. The direct extension of the method scales linearly with the number of target clients (in memory, central computation, and communication). Finding more efficient approaches for multi-target settings is intriguing, and left for future work.

”What are the potential and challenges to extending the proposed method to a scenario where every client learns and transfers simultaneously?”

• Generally speaking, we can separate the learning and transferring stages for this more challenging scenario: where clients are first trained using normal FL methods (e.g., FedAvg) without transferring. At some point when its model is comparatively well-trained, we can start to use it as source clients and perform transferring. However, we see the challenges are: (1) when we should stop the personalization and start to perform adaptation, (2) clients may have different best aggregation rules for transferring individually. However, one can potentially leverage our theoretical framework to decide which aggregation rule is the best to apply.

Thank you for reviewing our manuscript and we appreciate your feedback for refining the final version. We address your concern and question in the following.

”While the authors do provide enough information in the appendix, the main paper does not fully describes the methodology the authors employed in consolidating the results of Table 1. For instance, how are the labeled data points chosen for the experiments? How does the performance change w.r.t. to the choice of data points (i.e. standard deviation of the accuracy on target domain)? These questions are answered in the appendix, but they should be clear in the main paper.”

• Thank you for pointing out the insufficient description of the real-world experiments. The labeled data are sampled uniformly at random, and the performance of the FedGP, FedDA_Auto and FedGP_Autp are robust w.r.t. different levels of target domain data scarcity. We have revised our manuscript and made sure the information was clear in the updated main paper.

”I would like to raise that, while this is a Federated DA paper, the authors assume access to a (small) set of labeled data in the target domain. This somewhat breaks the rules of Unsupervised DA, and may bias performance towards methods that use target labeled data when comparing with UDA algorithms such as KD3A. This remains somewhat true even when supposing a small amount of labeled target data, depending on the degree of distributional shift. Nonetheless, I do think the authors use labeled data in a clearly motivated and justified way.”

• Thanks for highlighting the difference between our setting and the UDA setting. We note that we included comparisons with UDA algorithms for the sake of completeness and to provide a broader perspective on the performance of our method. Indeed, there is a distinction between the setting of Federated DA and Unsupervised DA.

Concerning FedDA and FedGP aggregation schemes, in order to have a convex combination, shouldn't (sum of beta = 1)? Is this constraint enforced? For instance, in Figure 9 (appendix), the sum of betas exceeds 1.

• Thanks for the question and we appreciate the chance to clarify it. For each $\beta_i$, it is the combination factor between one source domain $S_i$ and the target domain $T$. In other words, we use $\beta_i$ and $(1-\beta_i)$ to convexly combine the target domain model update and one source domain $S_i$'s model update. For example, for the non-auto-weighting FedDA and FedGP, we set $\beta_i = 0.5$ for all source domains. Therefore, the sum of beta may not be 1. Then, among different source domains, we use the convention of $w_i = \frac{n_i}{n}$ ($n_i$ being the number of data points at source domain $S_i$, and $n$ being the total number of source data points) for weighting each (source, target) pair, which has a sum of 1.

Thank you for reviewing our manuscript and we appreciate your feedback for refining the final version. We address your concerns as follows.

”I loved the paper till I came to the experiments section. In this day and age, should we still be doing experiments with ColoredMNIST, VLCS, CIFAR-10 and TerraIncognita? ColoredMNIST, VLCS and TerraIngocnita are from 2019, 2013 and 2018 respectively! This raises the questions whether the proposed solutions will scale to larger and difficult datasets.”

• We thank the reviewer for this constructive suggestion to make our experimental analysis more comprehensive. We conducted experiments on the PACS, Office-Home, and DomainNet datasets. The results are detailed in the updated version of our paper (appendix C.3); and are summarized as follows.

  • We see the proposed methods outperform other Domain Generalization (DG) methods with significant margins on PACS, Office-Home and DomainNet.
  • Despite its simplicity and efficiency, FedGP consistently performs well across most of the cases.
  • We found that the auto-weighting versions of FedDA and FedGP generally outperform their fixed-weight counterparts. An interesting exception is observed with FedDA on the OfficeHome dataset, where the fixed weight choice of $\beta=0.5$ is surprisingly good. We observe that on Office-Home potentially the source-only baseline sometimes surpasses the Oracle performance on the target domain. Thus, we conjecture that the fixed weight $\beta=0.5$ happens to be a good choice for FedDA on Office-Home, while the noisy target domain data interferes the auto-weighting mechanism. Nevertheless, the auto-weighted FedGP still shows improvement over its fixed-weight version.

• PACS (with 15% of target domain data)

• Office-Home (with 15% of target domain data)

• DomainNet (with 15% of target domain data)

Thank you for reviewing our manuscript and we appreciate your feedback for refining the final version. We address your concerns and questions as follows.

”It seems like it would be better to compare the algorithms presented in this paper to domain generalization algorithms, such as the ones shown in DomainBed's GitHub repo.”

• We thank the reviewer for this constructive suggestion to make our experimental analysis more comprehensive. We have conducted the suggested experiment and the results have been detailed in the updated version of the paper. As summarized below, the proposed method outperforms those reported in the DomainBed's GitHub repo for the Domain Generalization (DG) methods (we report the highest average DG performance vs. our methods).

”The methods presented make sense mostly in the cross-silo setting, as mentioned in the paper, which limits its applicability to general federated learning problems with a relatively larger number of clients that can benefit a lot from methods for generalizing to new clients.“

• In our work, we focus on the cross-silo setting, but it is possible to extend our ideas to the cross-device setting. For example, we can potentially first perform clustering before applying our methods, to find a subset of clients to collaborate for better target performance. We believe our theoretical and empirical findings showcase good insights for extending current work to this different setting.

”It is mentioned multiple times that data scarcity is the setting of interest, in which FedDA and FedGP are supposed to perform more favorably. However, we do not see experiments showing the effect of data scarcity on the robustness of the performance of FedGP vs. FedAvg, for example.”

• The experiment showing the effect of data scarcity on the robustness is included in Appendix C.8 where it shows the proposed methods are robust to data scarcity. In addition, we appreciate your suggestion of comparing with FedAvg as FedAvg serves as a baseline for the effect of data scarcity. As shown below (detailed in appendix C.8 of the updated paper), we compare to FedAvg showing the effect of data scarcity. Our FedDA_auto consistently outperforms FedAvg with large margins.