Enhancing Clustered Federated Learning: Integration of Strategies and Improved Methodologies

Dear reviewer KWQ4,

Thank you for your thorough review! Below are our responses to the questions you raised.

For clustered based FL algorithm, there is a recent work [1] to conduct clustering based on the inferred label distributions. The authors are suggested to discuss about this clustering strategy.

Thank you for bringing this to our attention. After reviewing the paper, we have included a discussion on this topic in the revised version (lines 178-179). We believe that inferring label distributions to calculate client distances could be a valuable enhancement to Tier 2 of the HCFL framework.

More Experiments

Thank you for the suggestion! We have conducted new experiments on CIFAR10 dataset as per your recommendation. For baseline algorithms, we choose the setting of ICFL and stoCFL which perform the best in Table 1 of our submission.

• We add experiments on $\beta = 0.8$, and $\beta = {0.2, 0.4}$ settings are included in the original submission. Moreover, we added C=2 and C=4 settings. Following [1], only label shifts are considered here.
• The results demonstrate that HCFL+ consistently outperforms the baseline algorithms across all settings, highlighting its reliable performance gain.

Dear reviewer NXiS,

Thank you for your thorough review. Below, we provide our responses to the questions you raised.

Clarification on contribution

Thank you for the suggestion! We would like to clarify that

• To the best of our knowledge, the HCFL framework provides the first unified framework for supervised clustered FL, incorporating (1) a unified clustering objective function that handles both soft and hard clustering and (2) a unified, four-tier clustering procedure paradigm. We believe that the HCFL framework holds significant importance in illustrating the recent progress in this field.
• The HCFL framework allows for free combination of existing techniques within each layer, enabling new benefits beyond simply recovering traditional methods. For instance,
  • FedRC exhibits good generalization performance but cannot automatically determine the number of clusters, while CFL excels in determining the number of clusters but lacks generalization. The HCFL framework enables the combination of the benefits of FedRC and CFL, achieving both good generalization and personalization performance (Table 1).
  • By allowing both cluster removing and cluster addition, HCFL+ achieves comprable or even better performance to baseline algorithms with significantly less number of clusters.
• Based on the HCFL framework, we identified several potential improvements to current CFL methods and proposed HCFL+ as a solution. Results demonstrate that HCFL+ achieves superior performance compared to baselines.

A closer look at the literature on learning with drift in the incremental setup.

We would like to clarify that our work focuses on the supervised clustered FL scenario, where clients possess data with varying distributions. To address the issue of drifts, studies employ various techniques to group clients into different clusters. These techniques include:

1. Grouping clients based on their local loss values [1]. By examining the loss functions of clients, we can identify those with similar performance characteristics and group them accordingly.
2. Grouping clients based on the distance between their model parameters [2,3]. This approach leverages the similarity of model updates to group clients with similar data distributions.
3. Grouping clients based on feature norm [4]. This method involves identifying representative samples for each client and grouping clients with similar feature norms.
4. Grouping clients by solving bi-level optimization problems [5,6]. This approach involves formulating a nested optimization problem where the outer problem optimizes the clustering of clients, while the inner problem optimizes the model parameters for each cluster.

In this paper, we use feature prototypes, feature means, and local loss values (see Section 4.4 and Appendix D) in conjunction with a bi-level optimization method (Equations 2-3) for adaptive client grouping. The results presented in Table 2 demonstrate that this approach can lead to enhanced performance.

main take aways of the appendix.

Thank you for your suggestions! Below are the key takeaways from the appendix, which we have now included at the beginning of the revised paper.

• Appendix A: We provide a proof showing how Eq. 4–7 are derived, demonstrating how they solve the EM-like objective function (Eq. 2–3) in practice.
• Appendix B: We analyze the HCFL framework in the context of linear representation learning, without assuming an equal number of samples across clusters. Our findings indicate that an imbalance in sample sizes and a higher level of drift can slow down convergence.
• Appendix C: A more detailed discussion of related works.
• Appendix D: We explain the rationale behind the design and practical implementation of the distance metric. For each pair of clients, we compute two types of distances: first, the distance between their local class prototypes for each class, and second, the distance between their feature means. The final distance metric for the client pair is then determined by selecting the maximum of these two distances and multiplying it by the local loss values.
• Appendix D: A detailed version of the HCFL+ algorithm is provided.
• Appendix E: We report the experimental settings used in our study in detail.
• Appendix E: Ablation studies on the impact of hyperparameters and algorithmic components show that HCFL+ is robust to variations in hyperparameters, and that all proposed components provide individual performance gains.

the overall objective of FL

In this paper, we provide two objectives:

• the mean performance/generalization error on clients’ local distribution, evaluating the personalization performance of models.
• the performance/generalization error unseen global distribution, evaluating the generalization performance of models.

These two objectives are evaluated as “Val” and “Test”, correspondingly.

What would be naturally occurring drift/shift in such scenarios and how does this match with the shift modelled in experiments?

In FL scenarios, three types of naturally occurring shifts can arise [7]:

• Label Distribution Shifts: The label distributions P(y) differ among clients.
• Feature Distribution Shifts: The feature distributions P(x) differ among clients.
• Concept Shifts: The conditional distributions P(y|x) differ among clients.

In our experiments, clients may experience all three types of shifts relative to each other. Further details are provided in Appendix E.1.

• Label Distribution Shifts: To model this, we use Latent Dirichlet Allocation (LDA) with α = 1.0 to partition the entire dataset into 100 clients, ensuring each client has a distinct label distribution P(y).
• Feature Distribution Shifts: We introduce random augmentations to client samples, with each client applying the same augmentation type consistently. This results in differing feature distributions P(x) across clients.
• Concept Shifts: These shifts are introduced by swapping the labels of certain classes, causing the conditional distributions P(y|x) to differ among clients.

What are results if the clustering itself is evaluated (eg having ground truth on the data distributions)?

Thank you for posing such a valuable question. The fundamental truth behind clustering depends heavily on the definition and principles of clustering that are being utilized.

• In some studies [1,2,3,4,5], clustering methods are typically designed to group clients with similar data distributions into the same clusters. However, it is important to recognize that in FL settings, all clients' data distributions can be varied, and assigning each client to a separate cluster may not be a practical or effective approach. To address this challenge, clustered FL methods incorporate additional thresholds to determine the similarity of clients within the same cluster, as illustrated in Table 1.
  • To assess the effectiveness of the clustering in this case, we evaluate the model's performance on both local and global data distributions, as shown in Table 1. Generally speaking, better local (val) and global (test) accuracy serves as an indicator of better clustering.
• In certain scenarios, as outlined in [6], the authors take into account diverse distribution shifts and propose grouping clients without concept shifts into the same clusters. This approach aims to enhance generalization performance by maintaining clients with similar concepts within the same cluster. In such cases, the optimal number of clusters corresponds to the number of unique concepts (in our experiments, this number is 3). Here, the global (test) accuracy becomes particularly crucial in evaluating the clustering effectiveness.

How personalized are the models?

We would like to provide further clarification regarding our submission:

• We have reported the personalization performance using the Val metric, which is calculated by averaging the local accuracy on all clients’ local test datasets. The results presented in Table 1 and 3 demonstrate that HCFL+ can achieve comparable or superior performance to the baseline methods, even when using a smaller number of clusters. This underscores the efficiency and efficacy of our proposed method in personalizing the model.
• The personalization performance of HCFL+ is influenced by several hyper-parameters, namely tol for CFL , $\alpha^{*}(0)$ for ICFL, $\tau$ for stoCFL, and $\rho$ for HCFL+. It is important to note that there is often a trade-off between personalization and generalization, and adjusting these hyper-parameters allows us to fine-tune the balance between the two, according to the specific requirements of the application.

We hope our responses have sufficiently addressed your concerns. Please do not hesitate to reach out if you have any further questions. Thank you again for your time and effort in reviewing our paper.

[1] An efficient framework for clustered federated learning. NeurIPS 2020.

[2] Multi-center federated learning: clients clustering for better personalization. WWW 2023.

[3] On the byzantine robustness of clustered federated learning. ICASSP 2020.

[4] Edge devices clustering for federated visual classification: A feature norm based framework. TIP 2023.

[5] Federated multi-task learning under a mixture of distributions. NeurIPS 2021.

[6] FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering. ICML 2024.

[7] Advances and open problems in federated learning. Foundations and Trends in Machine Learning.

Dear reviewer hh5r,

Thank you for your detailed review! Below are our responses to the questions you raised.

This work focuses on using clustering techniques to enhance the classification accuracy of FL. The difference between this type of research and the fully unsupervised federated clustering should be discussed to avoid potential misunderstandings.

Thank you for your suggestion! This work following the line of supervised clustered FL, and having the clear distinction to unsupervised clustered FL. We have revised our paper (footnote of page 1) by adding more clarification and related works [1,2]:

In this study, we address the issue of supervised clustered FL, which is differ from the unsupervised clustered FL examined by [1,2].

The efficiency issue is listed as one of the challenges in Section 4.1. But only the final number of clusters is reported accordingly. More discussions about the time and space complexity of this work, or even corresponding evaluation results are preferable.

Thank you for your suggestion! We have conducted additional experiments on the CIFAR10 dataset, with each client containing data from 2 classes (See results below). The hyper-parameters used were chosen based on the optimal configurations outlined in Table 1 of the submission.

• In addition to the final cluster number, we also report the memory usage and total simulation time.
• Notably, HCFL+(FedEM) achieved superior performance with a significantly reduced number of clusters, which result in reduced simulation time and memory usage.

The source code is not opened in the current version.

We have provided the code in the Supplementary Material.

[1] Ding, Shifei, et al. "Horizontal Federated Density Peaks Clustering." IEEE Transactions on Neural Networks and Learning Systems (2023).

[2] Qiao, Dong, Chris Ding, and Jicong Fan. "Federated spectral clustering via secure similarity reconstruction." Advances in Neural Information Processing Systems 36 (2024).