FLAG: Clustered Federated Learning Combining Data and Gradient Information in Heterogeneous Settings

1 The methodology takes one-shot clustering, which might be inconsistent for gradient similarity during the iterative federated process.

2 The authors can enhance the discussion and survey about related works, e.g., FedAC[1] and FedRC[2].

3 It lacks deeper theoretical analysis from three aspects, (1) privacy leakage of sharing both data and gradient knowledge to server, (2) the communication convergence based on hierarchical clustering, and (3) the communication and computation burden for FLAG.

4 The experiments are not in accordance with the motivation of FLAG. (1) some data heterogeneity types are not evaluated, e.g., concept drift and concept shift. (2) The vital hyper-parameter sensitivities are overlooked, e..g., the sample ratio of clients, and the total number of clients. (3) some crucially related works, e.g., FedAC[1] and FedRC[2].

[1] Zhang Y, Chen H, Lin Z, et al. FedAC: A Adaptive Clustered Federated Learning Framework for Heterogeneous Data[J]. arXiv preprint arXiv:2403.16460, 2024. [2] Guo Y, Tang X, Lin T. FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering[C]//Forty-first International Conference on Machine Learning.

1. The authors do not discuss whether FLAG can prevent privacy leakage, a critical concern in federated learning (FL). Specifically, they propose using truncated Singular Value Decomposition (SVD) to decompose raw data into singular values, which, along with gradients, are then sent to the server for clustering. However, previous research [1] has demonstrated that gradients can be exploited in gradient leakage attacks to reveal clients' local data. While this paper is not primarily focused on security, my concern is that the inclusion of singular values may inadvertently facilitate such attacks. Could the authors discuss the privacy implications of FLAG, particularly regarding the transmission of singular values and gradients? Also, the authors could compare the privacy risks of FLAG to existing methods or propose potential mitigations.
2. The paper does not provide analyses of the time and space complexity for the three algorithms. Additionally, truncated SVD is known to be challenging for parallel processing, and FLAG necessitates running truncated SVD on every class of data, which could impact the practical applicability of FLAG. Could the authors provide time and space complexity analyses for three algorithms, and to discuss how the use of truncated SVD on every class might affect scalability and practical implementation?
3. FLAG performs one-shot clustering during the first iteration. Previous research has demonstrated that, during training, clients' gradients are nearly aligned (i.e., point in the same direction) in the early stages and diverge later. Despite this, Table 4 indicates that one-shot clustering based solely on gradients in the first iteration is effective. Could the authors provide similarity heat maps of G, D and G+D in different iterations during training or explain why using only gradients in the first iteration is sufficient for clustering?
4. The authors claim that the "predefined cluster numbers" is a limitation of existing clustered FL methods. However, the proposed FLAG method also needs to search for an optimal clustering threshold to achieve the best performance. In the paper, the elbow method is used for determining this threshold, similar to how many current clustered FL methods [2] determine the optimal cluster numbers. Therefore, it is unclear how FLAG effectively addresses the "predefined cluster numbers" limitation. Could the authors clarify how the approach to determining the optimal clustering threshold in FLAG differs from or improves upon existing methods for determining cluster numbers?
5. cluster FL -> clustered FL?

I will surely increase my rating if my concerns are well addressed.

[1] Zhu, L., Liu, Z., & Han, S. (2019). Deep leakage from gradients. Advances in neural information processing systems, 32.

[2] Zhou, Y., Shi, M., Tian, Y., Li, Y., Ye, Q., & Lv, J. (2024, April). Federated CINN Clustering for Accurate Clustered Federated Learning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5590-5594). IEEE.

The paper presents an approach for clustered federated learning (FL) with data and gradient similarity. While this is an interesting concept, I have several concerns. See my comments below:

See my comments below

• The abstract and introduction mention addressing limitations of prior clustered FL methods under complex data heterogeneity scenarios, such as concept shift and drift. However, these claims are not supported by experiments or analyses specifically targeting these scenarios.
• The paper contains several statements that are incorrect or need justifications: 1) IFCA is inaccurately characterized as gradient-based clustering. 2) The claim that using only gradient or data similarity independently is insufficient (lines 78-81) requires justification. In particular, gradient similarity may not capture underlying data or task similarity if clients have different model weights or objectives. 3) The explanation of PACFL’s approach (lines 88-91) does not align with its actual methodology, which measures subspace angles to assess data similarity.
• Table 1 presents performance comparisons without sufficient context on the FL setup, making it challenging to interpret the results.
• The discussion on related work in clustered FL lacks depth and omits several state-of-the-art approaches. A comprehensive literature review section, addressing the broader landscape of clustered FL methods is required.
• The paper does not clarify how gradients are communicated within the FL framework. If the approach involves FedSGD, it would be helpful to specify whether gradients or local updates are exchanged in each communication round.
• The proposed algorithm for determining the optimal clustering is a brute-force approach and is unlikely to be communication-efficient. Furthermore, the algorithm lacks any novel aspect.
• The HC clustering threshold depends on the similarity matrix values (lines 280-281). It is important to clarify whether the similarity matrix is normalized, as this impacts the thresholding and clustering results.
• The experimental section overall is weak and does not provide details about the actual FL setup. For example, in Table 2, it is unclear whether different data heterogeneity settings are mixed.

I thank the authors for submitting this interesting work proposing the new FLAG method. Despite this work having some promising aspects, I found a number of weaknesses that, I believe, could be easily addressed to improve the paper. I hope the authors will critically consider these and find them useful for improving their work.

W1. Despite FLAG combining a novel approach, the algorithms used by the entire method are not novel, so FLAG can be considered an incremental contribution.

W2. The federated learning (FL) setting is not fully characterized, making it difficult to locate this work in the literature and to assess its impact and potential limitations. Describing fully the federated setting, with assumptions and examples, could give the readers more grounds to understand FLAG. Also, it might suggest to the FLAG’s authors means of improving evaluation.

W3. The method introduces several hyperparameters that are notably challenging to tune in FL. The experimental section doesn’t fully address practitioners' concerns about the sensitivity of FLAG’s performance to these hyperparameters. $\beta$, introduced at line 276, is the only one that is investigated even though it’s in the context of the ablation study for gradient and data-based proximity matrix. Hyperparameters such as $\delta$ (line 243), number of local steps at line 261 (Alg.1 - line8), the interval size for the $\alpha$ at line 281 (Alg.3 - line 6), the number of cluster FL rounds $t\prime$ (Alg.3 - line 9), number of sampled clients to obtain the optimal clustering $m\prime$ (Alg.3 - line 2). (I am sorry to point out both algorithms and text, but, unfortunately, there is no complete overlap)

W4. At the intersection between the FL setting and the method, this work fails to address the privacy concerns relative to sharing the server information regarding the local data distribution. The authors must discuss the privacy implications of their method, particularly the one-shot clustering step, which involves sharing information strongly related to the clients' private data.

W5. At the intersection between the FL setting and the method, this work fails to address the scalability concerns of FLAG when applied to large-scale federated populations. The authors must discuss the scalability properties of their method and potential downsides related to trading-off between ML efficiency and scalability, particularly for the one-shot clustering step.

W6. Despite an extensive literature review in the Introduction (Sec.1), a complete and detailed comparison between the algorithms used by FLAG and the proposals in previous works, such as PACFL [1], CFL [2], and IFCA [3], deserves a more explicit and evident discussion.

W7. The experimental setting appears to rely extensively on the partitioning methodology used to obtain federated client datasets from a “centralized” dataset. Unfortunately, it seems to be chosen specifically for this method, and the literature does not sufficiently support it. A reader may wonder what the results would be when adopting other, more standard partitioning approaches.

W8. Some sentences throughout the paper, particularly in the evaluation section, sound very speculative and should be modified by strong theoretical/empirical support to justify them.

W9. The evaluations section appears very limited in length instead of the very long and verbose “Introduction” and central section (Section 3). Supporting the claims of this work necessitates a more extensive evaluation. In particular, some of the claims are not well supported or explained, such as “efficiency” in third contribution in introduction, “optimality” of the clustering obtained (is it empirical or theoretical?), “capturing data heterogeneity” in second contribution in introduction, “scalability improvements” in conclusions, or just speculative, such as lines 420-422, lines 271-272 (clustering inaccuracies), lines 274-275 (combining leads to more accurate clusters),

W10. Most researchers nowadays tend to separate “Introduction” from “Related Works” and “Background”. This helps a lot with the narrative flow and conveys the paper's message. I strongly recommend to modify Section 1 to reflect such practices. As it stands, the narrative breaks several times in the introduction, which feels excessively long. “Related Works” can be put before Conclusions.

W11. Similar to the above, the abstract seems to be verbose in introducing the problem and describing the research gap. As a result, the number of lines reserved for the paper’s proposal is very minimal. Also, I suggest adding some numerical results to the abstract to convey the impact of FLAG more directly.

W12. I couldn’t help noticing that the baselines used in this work are relatively old. Comparing with the latest works would strengthen the evaluation and impact of this paper.

[1] Saeed Vahidian, Mahdi Morafah, Weijia Wang, Vyacheslav Kungurtsev, Chen Chen, Mubarak Shah, and Bill Lin. Efficient distribution similarity identification in clustered federated learning via principal angles between client data subspaces. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 10043–10052, 2023.

[2] Felix Sattler, Klaus-Robert Muller, and Wojciech Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 32(8):3710–3722, 2020.

[3] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586–19597, 2020.