Q5: On the experimental setting Local steps Regarding local steps, for the controlled scenarios we fix to one local epoch, that equals $J=8$ for CIFAR-10/100, $J=20$ for Shakespeare and $J=27$ for StackOverflow. Similarly, for large scale vision datasets, following [1] we run the algorithms for five local epochs (all these details are reported in Table 6)

Number of rounds We added a new section C.10 with plots over fewer rounds ($T=4000$) in our revision. However, we point out that we reported the results over $T=10.000$ rounds because algorithms present very different convergence speed depending on heterogeneity. Restricting a low number of rounds, e.g. $1000$, would cause evaluating the algorithms when they are too far from their convergence point in our most difficult setting ($\alpha=0$).

Other papers only evaluate under milder heterogeneity (e.g. $\alpha \in \{0.1, 0.3, 0.6\}$) and the difference with $\alpha=0$ in convergence speed of FL algorithms is quite substantial (e.g. see our Figure 6-right).

Let us also note that evaluating over more rounds is an advantage for competitor algorithms: since our algorithm is by far the fastest (see Table 4), our practice favors them to reach their best performance (of course there always is a maximum time/resource budget for the experiments).

Methods in Table 3 In Table 3 we extend the experimentation only for the best performing methods, that is we excluded methods that present failure cases or did not improve FedAvg on the controlled scenario. In particular, with reference to settings in Table 3, FedDyn and AdaBest did not converge in any of them, FedProx has lower performance than FedAvg and Mime/MimeLiteMom perform worse than the reported MimeMom (which is expected to be better than both).
FedCM instead has been implicitly considered in the grid search for GHBM, since it is equivalent to GHBM with $\tau=1$. For transparency, we report below the results of GHBM-$\tau=1$ compared to the best tuned GHBM:

Q6: Selection of $\beta$ Good observation, indeed the optimal value for $\beta$ depends on $J, T$, similarly to the current best known proof for momentum-based FL algorithms [1], which ours is based on. Indeed, our bound on $\beta$ is very similar to theirs (see their eq. (5)).

[1] Cheng et al., Momentum Benefits Non-IID Federated Learning Simply and Provably, ICLR 2024

[2] Cho et al., On the Convergence of Federated Averaging with Cyclic Client Participation, ICML 2023

We thank the reviewer for the time spent reviewing our manuscript. In the following, we respond pointwise to the reported weaknesses (W) and questions (Q).

W1: On the novelty and relationship with prior momentum-based approaches The idea of averaging the momentum over $\tau$ rounds is built from a theoretical and simple intuition on why classical momentum in FL may not work well under high heterogeneity and partial participation. In particular, the update rule in eq. (4) is derived from the modification that is needed in theory to extend the guarantees on heterogeneity proved by [1] from full to partial participation (eq. (3)). In practice, even if the algorithmic modification is simple, this stems from fundamental theoretical considerations, and then it is shown to be exactly what is needed to achieve in partial participation a similar update of the momentum term as FedCM in full participation (see Figure 1), unlocking superior performance both in theory and in simulations. The above factors account for the significant contributions we bring with this work, which indeed is not a mere extension of FedCM and FedADC.

W5-W2: Why cyclic participation is a legitimate assumption for GHBM Thanks for pointing out this concern, we will add a discussion on why the cyclic participation assumption is needed for our conclusions, and clarify that our result is non-trivial and follows from the algorithmic design of GHBM rather than from the additional assumption itself.

The FedCM cannot reach the same conclusion as GHBM To this end, the first point we want to highlight is that, while it is needed for simplifying our analysis, cyclic participation is not sufficient by itself to prove the same result for FedCM.

To clearly see this fact, we propose a theoretical experiment where a federation of $K=10$ clients, $K\cdot C=2$ cyclically selected at each round, collaboratively learns a quadratic function, with non-iid local datasets. If the two algorithms are equivalent under the assumption of cyclic participation, they should converge in the same way We report the experiment results in Fig. 9 of our revised manuscript (section C.9), showing that FedCM in cyclic partial participation does not have the same behavior as GHBM. Indeed, the plots also show that GHBM with optimal $\tau$ approximates in (cyclic) partial participation the same momentum update rule of FedCM in the full participation setting, confirming the validity of our analysis in section 4.2.

To further dispel any doubts, we additionally show that this finding also holds in deep learning experiments: in the table below we compare GHBM and FedCM in cyclic partial participation, on ResNet20 with CIFAR-100, $\alpha=0$ and $C=0.1$. It shows that the performance of GHBM under random and uniform client sampling is similar to the ones in cyclic participation and that, even enforcing cyclic participation, the gap between GHBM and FedCM does not decrease.

This confirms that the advantage of GHBM does not come as a consequence of the additional cyclic participation assumption but from the algorithm itself.

The advantages of cyclic participation in literature The best-known analysis of FedAvg under cyclic participation is provided by [2], while it proves that in certain situations (e.g. clients run GD instead of SGD) there can be an asymptotic advantage in the case we prospect with assumption 4.3, their results still rely on forms of bounded heterogeneity, and with this respect the results presented in this work are novel and advance the state of the art.

Based on the above premises, cyclic participation is clearly a legitimate assumption for GHBM and does not affect the validity of the results we present.

W3-Q3: Clarifications about lemma 4.4 and corollary 4.5 Lemma 4.5 and corollary 4.6 show how the error induced by heterogeneity decreases as $\tau$ increases, and for general $\tau$ it does use the bounded heterogeneity assumption. However, for the special case of $\tau=\frac{1}{C}$ the assumption is not needed, since the term in the LHS of the equations is zero by definition. This is already specified in lines 356-358 of the main paper and as the first step in the proof of lemma 4.5, lines 1024-1025.

Additionally, are the inferences from Equations (8) and (9) still valid for the later conclusions?

Consequently, the inferences of equations (8)-(9) are valid for the rest of the proofs, as theorem 4.7 explicitly imposes $\tau=\frac{1}{C}$.

W4: On the (b) term of lemma 4.6 The term $\mathcal{U}_k \forall k \in [T]$ is constant w.r.t. $\tau$. To see this fact it is necessary to combine lemma B.6 with lemmas B.7-B.5 and apply the upper bound on $\eta_l$ in eq. (122). Then, since the (b) term in lemma 4.6 is an average of the last $\tau$ of $\mathcal{U}_k$ terms, increasing $\tau$ does not linearly increase this term. We realize that inferring such dependence is not trivial, even for the attentive reader. We will be more detailed in the explanation of the lemma.

Q1: Analysis of gradients over heterogeneous scenarios This reviewer’s concern is caused by a typo in the caption of figure 1, which should be “... in non-iid ($\alpha=0$, left) and iid ($\alpha=10k$, right) …”, i.e. $\alpha=0$ corresponds to non-iid and $\alpha=10k$ to iid. $\alpha$ is the parameter of the Dirichlet distribution controlling the heterogeneity in splitting the dataset among clients, the procedure is the standard one in FL works and it is detailed in section C.2. Low values of $\alpha$ indicate more heterogeneous splits, while higher values indicate lower heterogeneity. The reviewer’s intuition is correct - there is higher variance in the non-iid scenario than iid - and indeed it does align with the presented results. Apologies for the confusion, we fixed the typo.

Q2: Explanation of why eq. (3) is the desired momentum update rule

The reason why eq. (3) is the ideal momentum update rule is indeed linked to parallelism the reviewer presented in W2. Essentially, the more clients we use to calculate a single gradient over the whole local dataset, the more the resulting gradient is the true gradient (by linearity of differentiation), i.e the momentum is updated as if we do GD instead of SGD. In heterogeneous FL this has a big impact, because the gradient over a small subset of clients (e.g. the ones selected at the current round) can be very different from the true gradient, leading to a biased update of the momentum. By using cyclic participation, it is possible to exactly determine how many clients we considered in averaging the gradients of the last $\tau$ rounds, hence the optimal $\tau$.

This idea, while it is simple and builds on fundamentals, is exactly what is missing from FedCM to offer strong guarantees against heterogeneity even in partial participation. Indeed, a careful analysis of the best proof of FedCM presented in [1] reveals that, in order to reach convergence under arbitrary heterogeneity, the momentum term must be updated with the average gradient over all the clients. FedCM can achieve this only by enforcing full participation, as in other cases there will always be a residual heterogeneity. GHBM instead changes the design of the momentum rule, and attains similar results by using past gradients (i.e. uses eq. (4) instead of (3)).

Q4: General form of theorem 4.7 When $\tau$ is not properly set the algorithm will still depend on bounded heterogeneity, since an additional term on the order of $\mathcal{O}(\frac{G^2}{\sqrt{T}})$ appears in the rhs of the convergence rate. We will add this result to the revision later, due to the short time of the rebuttal process. However, convergence guarantees should be stated for the optimal choice of parameters, and for this reason our Theorem 4.7 is stated for $\tau=\frac{1}{C}$.

We thank the reviewer for the high responsiveness, we would like to address the remaining concerns.

On the claim that FedCM cannot achieve the benefits

Currently, I believe the authors should remove the conclusion that methods like FedCM cannot achieve certain benefits, as the paper does not provide accurate theoretical proofs to support the correctness of this claim. [...] I understand that this is a writing approach, but stating conclusions without strict proof is not appropriate.

We recognize that, at this stage, our paper suggests this imprecisely, and that the suggestion of the reviewer is the correct thing to do. We want to assure that it is an imprecision due to writing and not intended to overstate claims. When talking about theoretical guarantees, we always mean that the available guarantees for other methods are worse than the ones we provide for GHBM.

We will correct lines 408-409, and will state that “the current best convergence guarantees of FedCM do not prove an advantage of momentum in (cyclic) partial participation”, which is accurate and reflects the importance of solving this open question the reviewer highlighted. We will make sure that from the paper it is clear that we leave that interesting theoretical question open. Finally, we believe we can extend theorem B.9 to prove a lower bound for FedCM (not just for FedAvg) and we are working on it.

Connection with Incremental GD. We appreciate the suggestion of the reviewer regarding drawing connection with existing literature in optimization outside FL. To the best of the author's knowledge, the term incremental gradient method is used to refer to a broad family of algorithms in the context of finite-sum optimization of $n$ functions (similar to the FL setting we are in, where functions are clients). They aim at matching or surpassing the worst case performance of GD without requiring to evaluate the gradient at all the $n$ functions at each iteration, but only one for iteration. Incremental Gradient Descent (IGD) evaluates each of the $n$ functions cyclically and uses only the last gradient for updating the model. In practice, the difference with SGD is the deterministic order the $n$ functions are traversed.

Similarly to IGD, Incremental Aggregated Gradient (IAG) uses the cyclic order when traversing the $n$ functions and evaluates only one gradient at each iteration, but also maintains the average of the last $n$ gradients.

In this panorama, our GHBM is based on the similar intuition of IAG (i.e. approximating the full gradient via old gradients of each function), and connects to it in that, under proper choice of $\tau$, the momentum is updated considering all the client's gradients. The scenario is however fundamentally different, as we use this intuition to converge under unbounded heterogeneity in FL, where the analysis must take into consideration additional errors (e.g. the client drift). The use of established optimization concepts in centralized optimization to build powerful FL algorithms is not uncommon, as two of the classical FL algorithms, SCAFFOLD and FedDyn, respectively build on SAGA and ADMM.

This is because Incremental GD does not strictly require a complete pass through the dataset in one round, yet it can still achieve the elimination of randomness.

IGD eliminates the randomness due to sampling the functions, but there is still an error due to the fact that the gradient used at each iteration has “incomplete” information w.r.t to the true gradient. Proofs of IGD assume this error can be bounded by a constant, and such assumption exactly corresponds to the bounded heterogeneity assumption commonly used in FL. As is, just enforcing a client sampling order, while removing the stochasticity due to sampling, does not remove the error that is due to having (highly) dissimilar functions (see section 3.2.1 of [5], in particular assumption 3.4).

This motivates the need of a scheme like IAG, where we explicitly build an estimate of the global gradient and use it as opposed to using only the last one, when it is unreasonable to assume that the dissimilarity among the functions is not upper bounded.

We hope this provides further insight of why just enforcing cyclic partial participation in FL is likely to be not enough to remove the need for the unbounded gradient dissimilarity assumption.

If there are other approaches we should consider, please provide us some reference, we will be happy to add a discussion in the final revision of our manuscript.

[5] Gurbuzbalaban Et Al., Convergence Rate Of Incremental Gradient And Incremental Newton Methods, SIAM Journal on Optimization 2022

We thank the reviewer for the time spent reviewing our manuscript. Although we acknowledge that the reviewer admitted this submission may be outside their area of expertise, we hope our answer can provide the reviewer with a better understanding of our work.

On the use cyclic partial participation assumption Assumption 4.3 simply states that there is a fixed order in which clients appear across rounds in the training, i.e. each client is sampled every $k=\frac{1}{C}$ rounds. Previous works have shown that if client availability is correlated with geographical location (e.g. mobile phones charging at night), then client availability follows a cyclic pattern [3,4]. The best-known analysis of FedAvg under cyclic participation is provided by [1], which proves that in certain situations (e.g. clients run GD instead of SGD) there can be an asymptotic advantage in the case we prospect with assumption 4.3. However, it is important to notice that all the results presented in [1] rely on forms of bounded heterogeneity, and with this respect, the results presented in this work are novel and advance the state of the art.

We hope our answer clarifies the reviewer's concern; if not, please let us know, and we are available for further discussion.

[1] Cho et al., On the Convergence of Federated Averaging with Cyclic Client Participation, ICML 2023

[2] Cheng et al., Momentum Benefits Non-IID Federated Learning Simply and Provably, ICLR 2024

[3] Zhu et al., Diurnal Or Nocturnal? Federated Learning Of Multi-Branch Networks From Periodically Shifting Distributions, ICLR 2022

[4] Crawshaw et al., Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysis, NeurIPS 2024

We thank the reviewer for the time spent reviewing our manuscript. In the following, we respond pointwise to the reported weaknesses (W) and questions (Q).

W1: On the novelty and relationship with [1] The work of [1] proposes a new theoretical analysis of existing momentum-based approaches (FedCM) and their integration with SCAFFOLD. Their core contribution is proving that FedCM (or as called in [1], FedAvg-M), under full participation, converges under arbitrary heterogeneity; their work itself does not propose a new algorithm, but the integration of momentum with SCAFFOLD and other forms of variance reduction. Conversely, our work proposes the novel GHBM formulation, whose motivations exactly lie in the limitations of FedCM, namely that the classical momentum itself is not enough to converge under arbitrary heterogeneity in partial participation. The core difference with classical momentum is explained from the ground up, precisely tackling the theoretical reason why FedCM does not achieve the same result in partial participation, as its momentum term is updated with a biased estimate of the global direction (section 3.3). We then provide a theoretical analysis that builds upon the result of [1] and extends it by considering our momentum formulation instead of the classical one. With these premises, the fact that we build on existing knowledge and extend previous theories does not affect the novelty of our work. On the contrary, by employing similar theoretical tools as [1], we clearly demonstrate the advancements of our new algorithm and highlight the specific improvements achieved, by enabling a result that is unattainable by FedCM.

W2-Q1: On the comparison with SCAFFOLD-M

Our algorithm is the first that can use only momentum to converge under arbitrary heterogeneity in partial participation. Indeed, proving that such a result is possible by using only momentum is the main point of our theory. We did not compare with SCAFFOLD-M because, as SCAFFOLD, SCAFFOLD-M is based on variance reduction to overcome the dependence on bounded heterogeneity, i.e. SCAFFOLD-M converges under arbitrary heterogeneity thanks to SCAFFOLD’s variance reduction, not because it uses momentum. As is, our claim that “existing momentum-based FL algorithms still rely on bounded heterogeneity in partial participation…” refers to momentum-only FL algorithms. We acknowledge that we can be more precise on that point, so we added this discussion to our revision.

From the practical perspective, there are several reasons why GHBM is preferable to SCAFFOLD-M, especially in settings with very low participation. The first one is the algorithmic structure: SCAFFOLD-M requires clients to keep local client states across rounds, a limitation that is known for making the algorithm not well-suited for cross-device FL [2]. The second one is related to actual empirical performance: there is extensive evidence that, despite the strong theoretical properties, variance reduction does not perform well in deep learning [3]. Indeed, performance issues of SCAFFOLD have been noticed in previous work [2] as well as in ours. In particular, it does not work in deep learning settings with both high heterogeneity and low participation (see Table 3). As is, building on the effectiveness of momentum in deep learning, our work significantly improves on the use of momentum in FL, showing that it can both give similar guarantees to SCAFFOLD(-M) and finally improve by a large margin the performance on realistic scenarios.

However, we do agree that an experimental comparison with SCAFFOLD-M could be useful to the reviewer to evaluate the improvement we bring with this work. In the following, we simulate the same setting of Table 2 for CIFAR-100 w/ Resnet20, $C=10\\%$ and $\alpha \in \\{0, 0.1, 0.3\\}$. As it is possible to notice, SCAFFOLD-M does not significantly improve SCAFFOLD, and most notably has much lower performance than GHBM.

W3-Q2: On why our GHBM largely outperforms the state of the art We have explained in section 3.3 that the key intuition behind GHBM is updating the momentum term, at each round, as it would be updated in full participation, that is with the average gradient over all clients. To do so in a way compliant with partial participation, GHBM uses past server pseudo-gradients (eq. 4) instead of (unavailable) gradients at current parameters (eq. 3). This is why we are able to obtain the same theoretical result [1] gets for FedCM under full participation, as GHBM approximates the same kind of update. Indeed, our analysis in Figure 1 demonstrates that this strategy is very effective since it drastically reduces the estimation error for the update of the momentum term. Our theory keeps track of the error induced by using past gradients in the momentum updates, and this aspect is different and expands upon the one proposed in [1].

The empirical improvements we get in large-scale scenarios relate to the theoretical convergence rate established because those considered settings are characterized by (i) high heterogeneity and (ii) very low client participation (indicated directly in Table 3), which are exactly the aspects the algorithm is expected to improve upon by looking at the theory.

We agree that understanding where our improvement comes from is crucial to correctly evaluate the contribution of our work, so please let us know if there are any more concerns on this matter.

Q2: Is GHBM just faster or does it also learn better solutions? This is a very interesting question. In all our experiments we noticed particularly faster convergence over other methods, not only in non-iid settings where the effect is most noticeable but also in iid settings. The theory we propose does explain that speedup, even in the iid case, where we expect the (local) stochastic noise $\sigma$ to affect the optimization. In all our experiments we also notice that the model trained with GHBM reaches at convergence higher accuracy w.r.t other methods, e.g. LocalGHBM (as GHBM) reaches 62.0% accuracy in Table 2 (ResNet20, iid) instead of the 60.5% of MimeMom (best competitor) despite both having long reached their convergence point. Based on these results, we conjecture that it is possible that GBHM promotes learning better solutions, but we do not currently have any theoretical indications of why it should happen, so we leave this question open for future work.

[1] Cheng et al., Momentum Benefits Non-IID Federated Learning Simply and Provably, ICLR 2024

[2] Reddi et al., Federated Adaptive Optimization, ICLR 2021

[3] Defazio et al., On the Ineffectiveness of Variance Reduced Optimization for Deep Learning, NeurIPS 2019

We thank the reviewer for the time spent reviewing our manuscript. In the following, we respond pointwise to the reported weaknesses (W) and questions (Q).

W1-Q3: On literature overview and missing comparison with CE-LSGD Thanks for pointing out those related works, due to space constraints we added a discussion on them in section A.1, but will integrate it into the main paper after the discussion phase. The CE-LSGD and Mime are in the "streaming" or "stochastic" setting where there are potentially an infinite number of clients. In our work, we only focus on settings with a finite number of clients (similar to Scaffold and ScaffNew). This is why the lower bound results of Patel et al. 2022 do not apply to us. We also note that, unlike Scaffold or Scaffnew, our algorithm does not require the clients to store any state, meaning it is easier to implement in practice.

W2: On enhancing presentation We appreciate the reviewer’s feedback on how to improve the readability. We fixed all the equation links and reported in the main text the definitions in eq. (11) and (12) to avoid requiring the reader to go back and forth to grasp the meaning of lemma 4.6. The symbol $G$ has been introduced in the caption of Table 1, but we do agree with the reviewers and will report the definition also in Lemma 4.4

Intuitive explanation of Assumption 4.3 Thanks for the suggestion, assumption 4.3 simply states that there is a fixed order in which clients appear across rounds in the training. For example, if $K=10$ clients $i \in [1,K]$ are present in the system and $C=20\%$ of them participate at each round, a sequence like $\{(1,2), (3,4), (5,6), (7,8), (9,10), (1,2), (3,4), …\}$ satisfies the assumption. It is easy to notice that the period between two subsequent sampling of any client is fixed and it is equal to $\frac{1}{C}$, i.e. the inverse of the participation frequency. We will add a graph in the supplementary for an immediate visual understanding of this pattern.

W3-Q1: On the FL settings our algorithms tackle Thanks for the question, as it allows us to clarify that the general algorithm proposed in this work (GHBM) is designed for settings closer to cross-silo than to cross-device FL. However, we need to distinguish between what we can offer as theoretical guarantees and the expected empirical performance of our methods in realistic scenarios.

From the theoretical standpoint, strong guarantees on statistical heterogeneity in partial participation come with the constraint that we can see each client multiple times (i.e. the $\tau=\frac{1}{C}$ constraint in theorem 4.7), which makes it an algorithm more suitable for settings with limited number of clients that participate often, as it is in cross-silo FL. In this scenario, GHBM is the sole momentum-only FL algorithm that can achieve this result.

In practice, GHBM can also work in settings more akin to cross-device FL (e.g. the one considered in Table 3) since it is a stateless algorithm and the momentum calculation (i.e. the $\tau$ hyperparameter) can be controlled server-side. Indeed, from the empirical perspective, we observed that GHBM is currently the best algorithm in realistic scenarios akin to cross-device FL (e.g. the one considered in Table 3) with even extremely low participation (e.g. $C<1\%$ and the total number of clients on the order of $10^4-10^5$).

LocalGHBM is proposed as a more communication-efficient variant of GHBM, that leverages stateful clients to avoid exchanging additional data (i.e. the momentum term). As such, is not intended and it is not well-suited for settings akin to cross-device, as the reviewer correctly notices.

We will be more clear on these aspects in our revision.

Q2: Clarification about Figure 1 Thank you for pointing this out. The caption of Figure 1 contains a typo that is at the base of your (correct) concern: the caption should be “... in non-iid ($\alpha=0$, left) and iid ($\alpha=10k$, right) …”, i.e. $\alpha=0$ corresponds to non-iid and $\alpha=10k$ to iid, since low values of $\alpha$ indicate more heterogeneous splits, while higher values indicate lower heterogeneity (see section C.2 for additional details). Therefore, the presented results align with the reviewer’s observation - there is a higher variance in the non-iid scenario than iid. Apologies for the confusion, we fixed the typo.

Q4: About our use of cyclic participation assumption 4.3 The cyclic participation assumption (4.3) is used to determine how many different clients are being selected in $\tau$ subsequent rounds and, consequently, the gradients averaged when updating the momentum with the GHBM rule. It is used essentially to ease the proof because when sampling at random it can happen that the same clients are selected more than once in a given window of $\tau=\frac{1}{C}$ rounds, while some others may not be sampled. This is related to our main theoretical contribution in that, for achieving convergence under arbitrary heterogeneity, the proof technique we adopted requires that the momentum is updated with a gradient over all the clients, as it happens in full participation. We want to highlight that the assumption alone is not sufficient to enhance the guarantees of FedCM in partial participation: the reason behind the better result is the structure of GHBM update.

Let us also note that the assumption impacts only the theoretical analysis; in practice, our experiments employ random sampling, and the results are nearly identical to those obtained under cyclic participation. Extending the analysis to account for random client sampling would likely require the development of new theoretical tools, which we leave as a direction for future work.

Q5: About convergence rate and local steps As the reviewer correctly notes, our analysis does not show a benefit in the convergence rate from the local steps $J$, as ScaffNew does. In this regard, our proof shares the same result as the currently best-known theory for momentum-based FL methods [1]. The main goal of our analysis is to show that GHBM can converge under arbitrary heterogeneity and with partial participation, a result previously achievable by FedCM only with full client participation. Improving the analysis with respect to the speedup of local steps is an interesting direction we leave as future work.

Q6: GHBM and server momentum Thanks for the interesting question! GHBM uses (global) momentum on the client side and does not apply any server-side momentum. This is in line with recent momentum-based FL algorithms, such as FedCM [2] and MimeLiteMom [3], and previous work proving that it is important to incorporate client-drift correction locally at clients [3,4], especially in highly heterogeneous scenarios.

However, to precisely answer the reviewer, we added section C.8 in the supplementary material with initial experiments not originally included in the submission, showing that GHBM used at the client level always outperforms its server-side variant.

[1] Cheng et al., Momentum Benefits Non-IID Federated Learning Simply and Provably, ICLR 2024

[2] Xu et al., FedCM: Federated Learning with Client-level Momentum, arXiv 2021

[3] Karimireddy et al., Breaking the centralized barrier for cross-device federated learning, NeurIPS 2021

[4] Karimireddy et al., SCAFFOLD: Stochastic Controlled Averaging for Federated Learning, ICML 2019

[5] Hsu et al., Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification