Benchmarking Algorithms for Federated Domain Generalization

We appreciate the reviewer's careful reading and recognition of this work. Especially for the suggestions on expanding the data partition method on other scenarios. We next provide a detailed reply as below.

My only critique is a (minor) lack of clarity. In section 4., the term ERM is never defined, and the empirical risk minimization method is never properly described. While, for knowledgeable audiences, this does not hurt the understanding of the paper in general, this makes the paper harder to read for beginners. I suggest authors add a description of the ERM in page 3, on the Federated DG paragraph.

Thanks for pointing this out! We have revised manuscript on page 3: “The most common objective in Federated learning is empirical risk minimization (ERM), which minimizes the average loss over the given dataset.”

Since Domain Adaptation is a related problem to Domain Generalization, is it possible to apply the proposed partitioning scheme to Federated Domain Adaptation?

Thank you for your insightful question. Yes, our partition method is general for partitioning of any D distributions onto C clients; the distributions could be defined based on:

• Domain labels as in DG in our paper or domain adaptation as reviewer mentioned.
• Class labels as in class-heterogeneous FL.
• Sensitive attributes in FL fairness [1,2].

We have added the discussion in the main paper at the start of section 3 heterogeneous partitioning method.

Modification on page 3: “Generally, our partition method effectively handles partitioning $D$ types of integer-numbered objects into $C$ groups. It's broadly applicable, suitable for domain adaptation, ensuring fairness in Federated Learning (FL), and managing non-iid FL regimes. ”

[1] Mohri, Mehryar, Gary Sivek, and Ananda Theertha Suresh. "Agnostic federated learning." International Conference on Machine Learning. PMLR, 2019.

[2] Du, Wei, et al. "Fairness-aware agnostic federated learning." Proceedings of the 2021 SIAM International Conference on Data Mining (SDM). Society for Industrial and Applied Mathematics, 2021.

We appreciate the reviewer for the positive feedback and construcive suggestions. We responded to the questions and suggestions you made individually below.

In the context of domain generalization, we are particularly concerned with whether models trained on limited source domains can generalize to new target domains. This paper also uses held-out domains (in Appendix C.2). I believe that this aspect should be more explicitly explained in the main body of the text...

Good point. We agree that our setup of held-out validation domains is of great importance and need better visibility. We have added the corresponding part to the main paper section 4.3, page 7 per the reviewer’s suggestion, where we emphasize that we are using validation domain dataset: “In DG task, we cannot access the test domain data. However, we are particularly concerned about the model performance outside the training domains, thus we preserve a small portion of the domains we can access as held-out validation domains, and the held-out validation domains are used for model selection and early stopping. ”

Error bars are not provided for experiment results, the conclusion may be influenced by random fluctuation.

Great suggestion. Following the Reviewer's suggestion, we have included in the revised manuscript error bar in Section 4.3. In particular, we have included the standard deviation of the repeated experiments on PACS, CelebA, Cameylon17. Due to the computational challenges of training many different models with different settings, we keep working on the results on Py150, CivilComments, IWildCam and FEMNIST datasets, and we will include them in the final version.

Page5 line 12: homogeneous -> homogeneity

Thanks, fixed.

When considering the second kind of DG, it is relevant to “performance fairness” in FL, which encourages a uniform distribution of accuracy across clients. Although works in this direction might emphasize more on participating clients, I believe at least the AFL algorithm [1] can be a good supplement to the benchmark.

We appreciate the reviewer's mention of this work. In a nutshell, the Reviewer is right: Agnostic Federated Learning (AFL) shares similarities with Domain Generalization in a Federated context. This is evident as both approaches address scenarios where the test distribution diverges from the training distribution. Thus we agree that AFL is a good method to evaluate especially when tackling subpopulation shift tasks. We have added a discussion in the revised paper page 17. Furthermore, we are currently working on implementing AFL in our benchmark, especially for CelebA, CivilComments, which are designed for the sub-population shift task. Our benchmark could also help the evaluation of the future work in this line of research.

Modification on paper page 17: “Agnostic Federated Learning (AFL) [1,2] share similarities with Domain Generalization in a Federated context. This is evident as both approaches address scenarios where the test distribution diverges from the training distribution. AFL constructs a framework that naturally yields a notion of fairness, where the centralized model is optimized for any target distribution formed by a mixture of the client distributions. Thus, AFL is a good method to evaluate especially when tackling subpopulation shift tasks.”

In Eq. (1), two kinds of DG are mentioned. Which kind of DG is mainly used in your benchmark?

Great question. We used both formulations. We elaborate this both in terms of methods and datasets.

Regarding methods, since test domains are inaccessible for direct optimization, DG methods typically start from one of the two theoretical formulations and seek to find an approximate objective on training domains to work with. For instance, Fish starts with the average-case objective, and IRM, GroupDRO starts with the worst-case objective. We included diverse methods from both formulations.

Regarding datasets, PACS, FEMNIST, Camelyon17, and IWildCam focus on averaging test domain performance, aligning with the first DG objective for evaluation. Among them, IWildCam uses the F1 Score, which is a weighted average, placing greater emphasis on rarer species. In contrast, CelebA and CivilComments prioritize worst domain accuracy, aligning with the second DG objective. Further, Py150 is unique in targeting a specific sub-population accuracy, termed 'method-class accuracy.' This is particularly relevant for Py150's application in code completion, where the accuracy of predicting method and class names is critical.

[1] Mohri, Mehryar, Gary Sivek, and Ananda Theertha Suresh. "Agnostic federated learning." International Conference on Machine Learning. PMLR, 2019.

[2] Du, Wei, et al. "Fairness-aware agnostic federated learning." Proceedings of the 2021 SIAM International Conference on Data Mining (SDM). Society for Industrial and Applied Mathematics, 2021.

The performance for some algorithms, for example, FedSR, is very low, and consistently lower than FedAvg. Could you explain the reason behind?

Thank you for your question. First, we would like to point out that we replicated the results in the experimental setting of the FedSR paper, where their evaluation on the PACS dataset is only based on 3 clients. In our benchmark, we observed two new phenomena for FedSR:

• the degradation in performance when the number of clients is large (increase up to $100$)
• the hyperparameters are also more sensitive compared to their 3 clients setting.

As suggested by reviewer AnGM, we conducted repeated experiments (Sec. 4.3), where we provided each method with a larger hyperparameter tuning budget, thus, FedSR achieves better results than in the original experiments. Even with an increased budget, we observed that FedSR is still significantly worse than FedAvg ($40\\%$ vs $90\\%$ on PACS). Thus, it does not alter the conclusion of the original manuscript.

A potential explanation of the phenomena

The convergence analysis is well established for FedAvg in the literature, see [3] and subsequent works. However, there’s no theoretical guarantees for FedSR, where the objective function is constructed by the summation of local data loss and local non-smooth penalty. We suspect that the extra heterogeneous local nonsmooth penalty terms introduced in FedSR potentially diverts each client's learning trajectory, leading to harder convergence when client number increases.

[3] Stich, Sebastian U. "Local SGD converges fast and communicates little." arXiv preprint arXiv:1805.09767 (2018).

The paper mainly summarizes the experimental observations, but does not offer much theoretical analysis or explanation for why some methods perform better than others in certain scenarios, which would be helpful to gain more insights into the FDG problem and the design of effective methods.

We appreciate the reviewer for the comments and feedback. While we cannot fully explain all performance differences, we suggest a few possible explanations and general insights that may help in designing more effective federated DG methods.

Mismatch between method’s (implicit) assumptions vs real-world datasets One potential issue is that each method may implicitly or explicitly assume certain conditions about the data generating process. For instance, IRM assumes the special structural equation model and FedDG assumes that the amplitude spectrum mainly consists of style information and the phase spectrum mainly consists of object information. Yet, the underlying data generating process may not match these assumptions. This mismatch may lead to varying relative performance across datasets. More careful analysis of the implicit or explicit assumptions of each method compared to real-world datasets could be a fruitful future research direction.

Privacy Constraint Centralized domain generalization methods usually assume that the data from each domain are accessible. Yet, in practice, each client may only have access to one domain (our domain separation case). Thus, centralized methods are not directly applicable in the domain separation case. A potential research direction for this could be designing a DG objective to be decomposable across domains so that it is amenable for FL.

Convergence in FL setting While in a centralized setting, it is often assumed that the method will naturally converge using an appropriate learning rate, the speed of convergence and solution at convergence may have a much higher impact on DG in the FL setting. For example, FedSR and FedGMA may generalize well but the convergence is very slow when the number of clients is large. On the other hand, the convergence analyses are well established for FedAvg, FedProx, Scaffold in the literature, see [2] and subsequent works. However, there’s no theoretical guarantees for newly designed methods for Federated domain generalization. Because FedSR’s objective function is constructed by the summation of local data loss and local non-smooth penalty, we suspect that the heterogeneous local nonsmooth penalty terms introduced in FedSR potentially diverts each client's learning trajectory, leading to slow convergence when client number increases. For other methods that are designed to converge quickly, like FedProx and Scaffold, the solution they converge to does not generalize well. This is because they were designed to address client heterogeneity but were not designed for train-test heterogeneity (i.e., distribution shifts). Therefore, it is unsurprising that they do not perform well on DG tasks. Given all of this, a potential research direction would be to combine the strengths of provably convergent FL algorithms with DG objectives or regularizations. It seems that for good federated DG performance, both aspects need to be solved simultaneously. Thus, how to design a method generalize better than simple FedAVG and maintaining the convergence property of FedAvg is of great importance. This is the message we wish to convey in the benchmark. We will incorporate those explanations in the paper to provide intuition on solving domain generalization in the Federated context.

[1] Koh, Pang Wei, et al. "Wilds: A benchmark of in-the-wild distribution shifts." International Conference on Machine Learning. PMLR, 2021.

[2] Stich, Sebastian U. "Local SGD converges fast and communicates little." arXiv preprint arXiv:1805.09767 (2018).