We are very pleased and thankful for Reviewer a16f for their thorough review. In particular we are thankful that the reviewer found our method novel and interesting, and the soundness, presentation, contribution and experimental setup of our paper good. In the following, we will answer the reviewer’s remarks and questions in more detail:

Reviewer’s remark: “Some details of the algorithm should be better explained. Some baselines seem to be missing.” Author’s answer: We will explain the algorithm in more detail in the camera ready version. Which parts should be explained better in particular? We report on new comparisons with 5 other global AND local FL baselines. As shown in Figures 1 and 2 in the rebuttal pdf and Tables 1-4 in the global response, our FLOCO method outperforms all the tested baselines (the results were averaged over 5 different random seeds). We implemented our FLOCO in the FL Bench [1] framework, and compared it with most of the baselines except SuPerFed, another state-of-the-art FL method that is not supported by FL Bench. SuPerFed is implemented in our Flower-based framework, and compared with our original implementation of FLOCO. Note that FLOCO run on the FL Bench framework and that run on our Flower-based framework do not exactly match because of the mismatch in the aggregation order of clients. Due to the time limitation, we could have run experiments in the FL base framework only on CIFAR10 Dir(0.5) on a SimpleCNN. For the camera-ready we will run these comparisons on all dataset split-model combinations.

Reviewer’s question 1: Could you clarify how simplex learning works in practice? Do you compute the loss and the gradients on each of the endpoints or do you sample one point in the simplex and backpropagate to update the endpoints? On which weights do you compute the loss? I think this is unclear from the paper and it would be beneficial to add a section “Simplex Learning” for the reader. Author’s answer: We sample one point in the simplex and backpropagate to update the endpoints. The loss to be minimized in the simplex learning (where alpha is drawn from the uniform distribution) is given in Eq.(7) in the original submission. Since FLOCO applies simplex learning only to the last layer, the weights in the other layers are point-estimated, which amounts to tying all end-points to a single point. We will extend Section 2.2 with these details.

Reviewer’s question 2: From my understanding, the idea of using a shared simplex and then dividing it into subregions is to have both global and local classifiers and to incentivize knowledge transfer and robustness. I would also see two other baselines/ablations that the authors might want to try to corroborate that this is actually beneficial. First, FedAvg simplex learning (no subregions, check global accuracy) and then local fine-tuning for personalization (compute local accuracy) personalization can be done with local simplex learning or standard learning. I guess this could be a way to study the impact of simplex learning by itself and the importance of subregions. Author’s answer: Thank you for your suggestion. We show these results in Figs. 3 and 4 in the rebuttal pdf, and will show them in the camera ready.

Reviewer’s question 3: As the authors mentioned in the related work, there are other personalized FL methods that apply simplex learning. Why did the authors not report their results as baseline? E.g. references 29 and 19. Author’s answer: The other simplex learning approach for federated learning, i.e. SuPerFed[1], is now included in Table 2 in the global response, which shows that FLOCO outperforms SuPerFed. Moreover, we have benchmarked FLOCO against many other baselines for the SimpleCNN - Dit(0.5) setting and the results are shown in Table 1, where our method outperforms the other methods. For the camera-ready version, we will show complete comparisons on all dataset-model combinations

Reviewer’s question 4: I’m not sure I fully buy the claims and observations in the paragraph on variance reduction (lines 255-272). The main issue with heterogeneity is convergence speed, so to claim some effect on this the authors should show faster convergence on the global accuracy. There is evidence that the effect of the subregions impacts local accuracy, but this is expected as there is better specialization - as it would be with a local fine-tuning stage. Author’s answer: FLOCO has M times more parameters to be trained because it learns the end-points of the solution simplex. For this reason, the simplex learning is slower than the baselines until the 200th round, where FLOCO projects the clients onto the solution simplex and thus the variance of gradients gets significantly reduced. After the 200th round, the consistent gradient signals of FLOCO accelerate the training and thus FLOCO converges to the model with higher global test accuracy. We can observe in Fig.2 (left) in the original submission steeper slopes in the global test accuracy after the 200th round, compared to FedAve and DITTO, although the difference is not as drastic as in the local accuracy.

Reviewer’s question 5: I suggest the authors also have a look at other more recent methods for global learning such as scaffold, feddyn, mime that could help with heterogeneity. They could also be used in conjunction with simplex learning. Author’s answer: We have been focusing on comparison with many state-of-the-art baselines in the rebuttal, and unfortunately did not have time to evaluate the suggested baselines.

Reviewer’s question 6: I found it odd that the authors decided to show the plots in Figure 2 using the simpleCNN and FEMNIST while from table 1 the cifar10 experiments show more edge of FLOCO over the baselines. Author’s answer: We chose simmpleCNN-FEMNIST because it shows a typical behavior.

Thank you for your rebuttal. I appreciate your effort and will take into account all your responses in my final score which I want to emphasize is positive.

• In Q2 I've suggested more than one option for personalization. Could you clarify how you implemented the experiments for this ablation? I don't understand if the simplex baseline is implying some personalization (local fine-tuning) or not.
• Regarding Q4, I understand what you mean by variance reduction now, but in the context of FL and optimization, this has a specific meaning (see SCAFFOLD). By checking again table 1 I see the improvement over simple baselines such as fedavg also in global accuracy. However, since the baselines for global models are not the most recent ones (besides the newly tested feddyn, which is generally broken) I suggest the authors tone down their claims in the contributions regarding "FLOCO outperforms state-of-the-art approaches in both global and local". It is however interesting the effect of their method on global accuracy hence reducing the effect of heterogeneity.
• One last comment is about scalability. Could you comment on the possibility of making your method work in more realistic settings with thousands of clients (e.g. inaturalist, gldv2) ? What are the challenges you foresee? this could be expanded in the limitations section.

Dear Reviewer a16f,

Thank you very much for your additional comments. Below we give temporal answers, and we will report on additional experiments with the suggested baselines by the discussion deadline.

Best, Authors

Reviewer’s question: In Q2 I've suggested more than one option for personalization. Could you clarify how you implemented the experiments for this ablation? I don't understand if the simplex baseline implies some personalization (local fine-tuning).

Author’s answer: We realized that we misunderstood your suggestions when submitting our rebuttal. The curves labeled as FedAvg-Simplex Learning in Figs. 3 and 4 in the rebuttal pdf are the global (Fig.3) and the average local (Fig.4) test accuracy of simplex learning without client projection and WITHOUT personalization. We apologize for our confusion. We are now evaluating the following methods and will report on the average local test accuracy by the discussion deadline:

• Baseline 1: We run FedAvg with simplex learning, and then, apply the plain DITTO personalization to the midpoint of the simplex solution. Namely, no local simplex is used.
• Baseline 2: We run FedAvg with simplex learning, and then, apply the DITTO personalization with local simplex learning.
• FLOCO+: We run FLOCO (with client projection after 100 communication rounds), and apply the DITTO personalization with local simplex learning.

Unfortunately, the cluster server in my group is under maintenance after the Neurips rebuttal phase ended, and we are running the additional experiments on local computers. To make the experiments feasible, we have to down scale the experiment. Namely, our simplex learning is with a (M=2)-simplex (instead of M=6 in the submitted paper) for a SimpleCNN on CIFAR10 with a Dirichlet(0.5) split. To make a fair comparison with our FLOCO+, we will also report on FLOCO+ results with M=2-simplex. We estimate that all experiments will finish by tomorrow (Tuesday) night.

Reviewer’s question: Regarding Q4, I understand what you mean by variance reduction now, but in the context of FL and optimization, this has a specific meaning (see SCAFFOLD). By checking again table 1 I see the improvement over simple baselines such as fedavg also in global accuracy. However, since the baselines for global models are not the most recent ones (besides the newly tested feddyn, which is generally broken) I suggest the authors tone down their claims in the contributions regarding "FLOCO outperforms state-of-the-art approaches in both global and local". It is however interesting the effect of their method on global accuracy hence reducing the effect of heterogeneity.

Author’s answer: We agree that our original claim of outperforming state-of-the-art global FL approaches is not supported by our experiments. Following the reviewer’s suggestion, we will tone down our claim.

Reviewer’s question: One last comment is about scalability. Could you comment on the possibility of making your method work in more realistic settings with thousands of clients (e.g. inaturalist, gldv2) ? What are the challenges you foresee? this could be expanded in the limitations section.

Author’s answer: We assume that by scalability the reviewer means the situation where many clients do not observe sufficient numbers of data samples. If we would have thousands of clients, all of which observe sufficiently many data samples, we do not foresee critical problems because the projection matrix can be computed with subsamples, and the other computations are linear operations. However, if we assume the generic FL setting, we cannot reasonably assign subregions to the low data clients, because the server does not get sufficient information about those clients. Following the suggestions by Reviewer a16f, as well as Reviewer 44Bh, we will revise the paper so that our target is not the generic FL setting, but the cross-silo FL setting, where each client observes sufficiently many data samples. We appreciate the reviewer’s constructive suggestions.

We are very pleased and thankful for Reviewer 5sib for their thorough review. In particular we are thankful that the reviewer found our application of linear mode connectivity to the FL setting interesting and the soundness, presentation, contribution and experimental setup of our paper good. In the following, we will answer the reviewer’s remarks and questions in detail:

Reviewer’s remark: “To strengthen their empirical findings, the authors could benefit from leveraging the bilevel optimization literature for analyzing the convergence of their algorithm.” Author’s answer: Thank you for your suggestion. We will consider theoretical analysis for future work.

Reviewer’s question 1: “Choosing an appropriate experimental setup is often overlooked in the pFL setting. Note that the following two scenarios are inappropriate for pFL experiments: (1) Clients have enough data to train their local models independently, eliminating the need to participate in an FL system. (2) The data distribution of the clients is identical or nearly identical. This typically results in high accuracy for FedAvg, making it challenging to justify using pFL methods. It would be informative to evaluate your method in experimental setups where local training and FedAvg fail to provide good results and compare your method's performance with SOTA pFL methods, especially methods with decoupled parameter space such as FedPer [1] and FedRep [2] and similarity-based methods such as pFedSim [3].” Author’s answer: We agree that the originally chosen baselines are not sufficient, and thus we report on new comparisons with 5 other global AND local FL baselines. As shown in Figures 1 and 2 in the rebuttal pdf and Tables 1-4 in the global response, our FLOCO method outperforms all the tested baselines (the results were averaged over 5 different random seeds). We implemented our FLOCO in the FL Bench [1] framework, and compared it with most of the baselines except SuPerFed, another state-of-the-art FL method that is not supported by FL Bench. SuPerFed is implemented in our Flower-based framework, and compared with our original implementation of FLOCO. Note that FLOCO run on the FL Bench framework and that run on our Flower-based framework do not exactly match because of the mismatch in the aggregation order of clients. Due to the time limitation, we could have run experiments in the FL base framework only on CIFAR10 Dir(0.5) on a SimpleCNN. For the camera-ready we will run these comparisons on all dataset split-model combinations.

Reviewer’s question 2: As previously mentioned, evaluating the performance of local models on their respective local datasets is crucial in personalized federated learning (pFL). Including the performance metrics of these models (average performance when trained solely on local training data) would provide valuable insights into the experimental results. Author’s answer: Since we have been focusing on additional comparisons with many state-of-the-art baselines, we could not have made it for comparison with this basic baseline. We expect that the local training is at least much slower than the federated learning for the clients that only get a small number of samples. We will conduct this basic experiment in the camera ready.

Reviewer’s limitation remark: The discussion of the limitations of the proposed method is framed within the context of future directions. It is suggested to address these limitations separately, focusing on theoretical analysis and validation using real-world federated learning datasets suited for personalized FL (pFL). Author’s answer: We will follow your suggestion.

[1] FL Bench: (https://github.com/KarhouTam/FL-bench)

Dear Reviewer bjGi,

Thank you again for your constructive comments. We will do our best to improve the paper, according to the reviewer’s comments.

Best, Authors

We are deeply thankful for Reviewer bjGi for their thorough review. We are pleased to read that the reviewer appreciates the richness of our experiments, the soundness and presentation of our work. In the In the following, we answer the reviewer’s remarks and questions in more detail:

Reviewer’s remark: “The proposed methodology lacks innovation.” Author’s answer: We argue that our paper has significant novelty and contributions: Our way of using solution simplexes for federated learning is unique; we employ a novel projection method that spreads FL clients well across the simplex; and we show in many different experiments that FLOCO performs well for global AND personalized FL, which are usually problems tackled separately. Thus we would like to ask for clarification by the reviewer about why our method lacks innovation.

Reviewer’s remark: “The experimental baseline is too old (2021) and there are many recent and similar clustered federated algorithms that lack comparison.” Author’s answer: We agree that the original baselines are old, and thus we report on new comparisons with 5 other global AND local FL baselines. As shown in Figures 1 and 2 in the rebuttal pdf and Tables 1-4 in the global response, our FLOCO method outperforms all the tested baselines (the results were averaged over 5 different random seeds). We implemented our FLOCO in the FL Bench [1] framework, and compared it with most of the baselines except SuPerFed, another state-of-the-art FL method that is not supported by FL Bench. SuPerFed is implemented in our Flower-based framework, and compared with our original implementation of FLOCO. Note that FLOCO run on the FL Bench framework and that run on our Flower-based framework do not exactly match because of the mismatch in the aggregation order of clients. Due to the time limitation, we could have run experiments in the FL base framework only on CIFAR10 Dir(0.5) on a SimpleCNN. For the camera-ready we will run these comparisons on all dataset split-model combinations.

Reviewer’s question 1: Can the authors clarify the difference between the proposed method and other clustering federated learning algorithms? Author’s answer: Our method does not cluster anything but project clients onto the solution simplex, so that the degrees of freedom within the simplex is used to capture characteristics of each client.

Reviewer’s question 2: Can you specifically describe the implementation details of the algorithm? Or improve the description. For example, there is a lack of more detailed description for the generation of R. Author’s answer: R is simply a L1-ball around the projected point of each client. For drawing samples from the uniform distribution over R, we perform the following: Generate random points on the surface of the L1 unit ball by normalizing the absolute values of a randomly generated vector. Scale these points to fit within the desired radius. Translate the points by the given center. We will add further algorithm details in Appendix.

Reviewer’s question 3: In the algorithm section, can the specific algorithm be explained more clearly? Author’s answer: We will add further algorithm details in Appendix. We are happy to clarify its algorithm details if the reviewer specifies on which algorithm our description is unclear, other than the generation of samples on R.

Reviewer’s limitation remark: The authors discuss the reasons for the poor experimental performance of the proposed methods Author’s answer: We do not understand this comment. We do not show poor performance of the proposed method, and thus do not discuss the reason. Could you please elaborate more?

[1] FL Bench: (https://github.com/KarhouTam/FL-bench)

Thank you for the rebuttal. After reading the rebuttal and reviews of other reviewers, I still maintain that this is a borderline paper. I will maintain my original rate.

Dear Reviewer bjGi,

Since the discussion deadline is approaching, we would kindly ask Reviewer bjGi to respond to our rebuttal comments. We see that our rebuttal comments addressed most of the criticisms by the reviewer, and would like to know the reasons why the reviewer still keeps the clear rejection score 3 after our rebuttal, which is a clear outlier among the other reviewers’ scores. We look forward to your response.

Best, Authors

We would like to express our gratitude to Reviewer 44Bh for their detailed review. We are happy to read that the reviewer acknowledges the novelty of our method that exploits recent findings in mode connectivity to train a simplex in the FL setting, which improves local and global performance. Moreover, we appreciate that the reviewer found our proposed Riesz-energy-based Euclidean projection onto the unit simplex novel and convincing. Below, we will answer each of the reviewer’s weakness remark, question and limitation.

Reviewer’s remark: Reviewer points out typos, missing citations, naming conventions, e.g. weight parameter -> parameter, and style changes, e.g. inclusion of equations numbers. Author’s answer: We have fixed all pointed out remarks above.

Reviewer’s remark: Lines 189-190: the baseline methods are not convincing. Since the proposed method explicitly induces the mode connectivity (i.e., line 25 per se) for the improved FL performance, it should have been compared to methods with similar motivations, e.g. [1], which was cited in the Related Works section (line 286) but not directly compared. Plus, authors should have also considered adding more personalization methods (e.g., [2], which shares the same strategy, i.e., only exchanging parameters of classification layer, for communication-efficient personalization in FL) as the authors aim to prove the effectiveness of FLOCO for improved personalization performances. Author’s answer: We agree that the originally chosen baselines are not sufficient, and thus we report on new comparisons with 5 other global AND local FL baselines. As shown in Figures 1 and 2 in the rebuttal pdf and Tables 1-4 in the global response, our FLOCO method outperforms all the tested baselines (the results were averaged over 5 different random seeds). We implemented our FLOCO in the FL Bench [1] framework, and compared it with most of the baselines except SuPerFed, another state-of-the-art FL method that is not supported by FL Bench. SuPerFed is implemented in our Flower-based framework, and compared with our original implementation of FLOCO. Note that FLOCO run on the FL Bench framework and that run on our Flower-based framework do not exactly match because of the mismatch in the aggregation order of clients. Due to the time limitation, we could have run experiments in the FL base framework only on CIFAR10 Dir(0.5) on a SimpleCNN. For the camera-ready we will run these comparisons on all dataset split-model combinations.

Reviewer’s remark: Line 212: please also report standard deviations on 5 different runs. Author’s answer: In Table 1-4 in the global response, we report on standard deviations over the 5 different seeded runs for each experiment.

Reviewer’s remark: Lines 461-466: the communication rounds (100 and 500), the number of clients (100), and the number of per-round clients in CIFAR-10 (which is 30 out of 100) are somewhat deviated from the practical FL settings, which raises doubts about the scalability of the proposed method. It is more acceptable if the proposed method is specifically designed for the cross-silo FL settings, which usually assume a small number of stateful clients and (optionally) full participation across rounds. Author’s answer: We have tested our method on the FEMNIST dataset, a practical FL data set where different clients have different numbers of data points.

Reviewer’s question 1: Line 255: in Figure 2, the convergence speed of the global test accuracy & the average local test accuracy of the proposed method is slower than the baseline methods (when seeing round ~50), which is somewhat counterintuitive provided that the total gradient variance remains the smallest across all FL training rounds (which authors stated that it leads to the better performance, i.e., line 265-266). Could the authors please provide an explanation on this? Author’s answer: The slow convergence of FLOCO in the beginning comes from the larger degrees of freedom (we train M end-points thus have M times more parameters to be trained). After the 200th round where we project the clients onto the solution simplex and thus the variance of gradient is significantly reduced, the global training is accelerated, as seen in Fig.2 (left) in the original submission (not as drastic as the local test accuracy, but still a larger slope than FedAve and DITTO can be observed).

Reviewer’s question 2: Line 255: same in Figure 2 (and similarly in Figure 3), the average local test accuracy is suddenly surged, starting from the 200th round. This is quite intriguing, and I also expect an explanation on this result. Author’s answer: As mentioned in the caption of Fig. 2 in the submission, FLOCO performs the simplex learning (with alpha drawn from uniform distribution over the whole standard simplex) until the 200th round. At the 200th round, FLOCO performs the proposed projection of the clients onto the simplex, based on the update signals, and starts using the solution simplex for personalization. We will make this point clearer in the main text.

Reviewer’s limitation remark: The authors do not evaluate the proposed method out of vision classification task. Author’s answer: We will extend our experimental setup to the google speech dataset for the camera-ready version.

[1] Connecting low-loss subspace for personalized federated learning (2022) [2] Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020) [3] FL Bench: (https://github.com/KarhouTam/FL-bench)

Thanks for the authors' time and effort for my suggestions and questions.
Especially, I really appreciate additional experiments that authors have committed to improve their initial manuscript.

I have only one remaining concern that needs to be clarified further.

The question about the practicality of the current experimental setting & the authors' answer about the FEMNIST baseline being used.

While it is acceptable that FEMNIST dataset[1] if $`Floco`$ is designed for real-world cross-device federated setting, it seems that the authors did not use the full dataset as stated in Appendix B, line 464.
In other words, the total number of clients in FEMNIST dataset is originally 3,597, but only 100 clients are sampled and used in this work.
That is why I recommended authors to tone down and specify the scope of this work into cross-silo FL setting, rather than sticking to an algorithm for generic FL setting.

This can be further justified due to the design choice of the proposed method:

• The $\mathcal{R}_k$ should be kept and tracked by the server for each client $k\in[K]$ till the end of the federation round.
• This means that the algorithm requires stateful clients, which does not hold in the practical cross-device setting. (please see Table 1 of [3])
• Moreover, this statefulness inevitably forces the algorithm to have a moderate number of clients, $K$, which is consistent with the current choice of experimental settings.

If I have misunerstood and if the proposed method is even scalable to the empirical settings with massive number of clients (e.g., original FEMNIST setting with $K=3,597$, StackOverflow dataset[2] with $K=342,477$, and iNaturalist dataset[4] with $K=9,275$), please enlighten me.

[1] (2019) LEAF: A Benchmark for Federated Settings
[2] (2019) https://www.tensorflow.org/federated/api_docs/python/tff/simulation/datasets/stackoverflow/load_data
[3] (2019) Advances and Open Problems in Federated Learning
[4] (2020) Federated Visual Classification with Real-World Data Distribution

I sincerely appreciate the authors' positive answers.
Someone might think that narrowing the scope of the work limits the scalability of one's proposed method, but I humbly believe that clarifying what can and can't be done in one's work adds more value to the research community as well as clarifies its practical implications.
Thus, please reflect what the authors have agreed upon during discussion periods in their final manuscript.
In conclusion, I decide to raise my score, and here are reasons.

Why not a lower score

• The authors made enough efforts in rebuttals to complement lacking and outdated baselines in their initial manuscript, and the results are convincing.
• After the discussion, the authors decide to adjust the main scope of their work in a more acceptable direction.
• The observation of the mitigating gradient variance in the proposed method (i.e., Fig. 2) is also intriguing and noteworthy.

Why not a higher score

• The scope (and possibly the title) of the manuscript should be refined according to the authors' acceptance of my suggestions (i.e., specifying the scope of the algorithm in a cross-silo FL setting), which will require some changes (e.g., re-writing of overall contents) to the status quo.
• While I recognize and accept that the mode connectivity --- a main motivation and an ingredient of the proposed method --- is fairly an empirical observation, a theoretical or in-depth empirical analysis (e.g., generalization guarantee, or convergence analysis) that incorporates the mode connectivity for federated settings is missing. The authors could complement this perspective by building on the recent analysis. (e.g., [1,2])
• (minor) Overall, the notations are difficult to parse and look similar to each other at first glance.

[1] Proving Linear Mode Connectivity of Neural Networks via Optimal Transport (AISTATS'24)
[2] Linear Connectivity Reveals Generalization Strategies (ICLR'23)