Improving Group Connectivity for Generalization of Federated Deep Learning

I am particularly concerned with communication costs and computational overhead of the method when applied to large-scale FL scenarios. Below are few weaknesses:

1. Regarding the communication costs, I agree that there are no additional costs compared to FedAvg, but this remains an issue. There have been many advancements in communication efficiency over FedAvg in recent years. To mention a few recent works, DisPFL[1] and SSFL[2] have demonstrated better performance than FedAvg, even at very high sparsity levels. I believe the baselines are insufficient and suggest comparing the method with at least these sparse baselines on a few vision datasets to evaluate whether the performance gains justify the communication cost to validate its worth.

2. The authors did not consider incorporating techniques to prune or compress unimportant weights, which could reduce communication overhead without sacrificing performance. While the core idea of the paper is appreciated, there is room for optimization and further improvements in terms of communication efficiency.

3. Although the idea is novel, it’s a lot of computations and could pose practical challenge in large-scale FL applications. Authors should think of addressing this in the future works so that this method can be practically applied.

References:

[1] https://doi.org/10.48550/arXiv.2206.00187

[2] https://doi.org/10.48550/arXiv.2405.09037

1. Lemma 3.3 statement and implication

• Firstly, I don't understand what is random about D_\\epsilon(w_{anc}^*) defined in Lemma 3.3. If $w_{anc}^*$ is a deterministic anchor model (as defined later in Theorem 3.5 and Theorem 3.8) then D_\\epsilon(w_{anc}^*) is also deterministic so the bound on the probability of D_\\epsilon(w_{anc}^*) does not make sense to me. Intuitively, I believe Lemma 3.3 is trying to bound the probability of LMC between a deterministic anchor model $w_{anc}^*$ and a random model $w$ such that $||w-w_{anc}^*|| \leq d/2$. So the probability here is over some uniform distribution of $w$ and not $w_{anc}^*$. If so, Lemma 3.3 should be re-written to clarify this.

• There is no understanding of how large $d_{\epsilon}$ can be. For instance if $d_{\epsilon} \geq d$ then the probability bound is just vacuous. Therefore authors need to either assume with some justification or prove that $d_\epsilon < d$ for the bound to make sense.

2. Theorem 3.5 and 3.8 statement and implications.

• It is not clear to me why we need randomness in these theorem statements. Suppose we are given three models $w_{anc}, w_1, w_2$ such that i) $w_{anc}$ and $w_1$ are LMC ii) $w_{anc}$ and $w_2$ are LMC and iii)$||w_{anc} - w_1|| \leq d$ and $||w_{anc} - w_2|| \leq d$. Why can't we use just these assumptions to bound the loss barrier between $w_1$ and $w_2$? In other words why do we need the additional assumption that $w_1$ and $w_2$ are sampled from the uniform distribution?

• Following up on my point in Weakness 1), authors need to show that $d_\epsilon$ is bounded for the bound in Eq. (6) and Eq. (10) to not be vacuous. In addition, authors also need to show that $\delta > 0$ that this there is a non-zero probability that $w_1$ and $w_2$ are sampled from the uniform distribution and are also LMC with $w_{anc}$.

• Currently I don't see a dependence on $d$ in the bounds in Eq. (6) and Eq. (10) which is a bit surprising to me since $d$ is defined in the Theorem statement.

3. Inconsistent experimental results and performance of baselines

• It appears to me that the authors have either not implemented FedProx correctly or have not tuned the regularization parameter $\mu$ in FedProx correctly. If we set $\mu \rightarrow 0$ then FedProx just becomes same as FedAvg so the performance of FedProx should at least be as good as FedAvg. Therefore it is surprising to see the consistently poor performance of FedProx across all the experiments.

• In Table 2, for the Tiny ImageNet column with Non-IID hyper. being 100, the performance of SCAFFOLD seems inconsistent with previous results.

• In Table 3, for the STS-B column, the performance of FedDyn seems inconsistent with previous results.

• Authors should provided information on how they tuned hyperparameters for the experiments and also provide graphs showing test accuracy vs number of rounds for all the baselines. Currently only the final accuracy numbers is reported for each of the experiments

• Why is FedDyn and SCAFFOLD not compared against in Table 4 and Table 5?

W1. The comparison with prior work is limited

W2. The proposed algorithms, including FedGuCci, do not have theoretical convergence guarantees.

W3. The comparisons and conclusions may not have sufficient evidence.

W4. The evaluation needs to be enhanced.