Communication-Efficient Federated Learning via Model Update Distillation

• While using synthetic samples is original, I disagree with the authors about their novelty claim for getting use of the prior knowledge the server has. There are a number of papers exploiting this prior knowledge (or side information) to reduce the communication cost: [1, 2].

• I think the writing could be improved. Especially having an additional section between sections 3.2 and 4 leaves reader in confusion about how (2) and (3) are being used for too long.

• The framework requires generating $m$ synthetic samples both at the server and the clients every round by solving (2). I am not sure if this is a computationally reasonable choice. Also, there is no theoretical guarantee that the generated samples will minimize (2) sufficiently -- leading to an irrecoverable error.

• The main contribution of the paper is sending $m$ synthetic examples instead of model updates in every round to reduce the communication cost. But how do the authors make sure that the size of the synthetic samples is smaller than the size of the model. What is the typical range of $m$ (number of samples)? How is it tuned for different datasets and models?

• It would be good to see a plot that shows the final model performance as $m$ increases.

• "FedPAQ (Reisizadeh et al., 2020) is the first federated learning quantization scheme". I am not sure what the authors meant by this sentence but FedPAQ is definitely not the first quantization scheme for federated learning, e.g., QSGD (2016) and SignSGD (2018). Please correct this in the next revision.

[1] Adaptive Compression in Federated Learning via Side Information (AISTATS 2024)

[2] Wyner-Ziv Estimators for Distributed Mean Estimation with Side Information and Optimization (AISTATS 2021)

• One of my main concerns is the privacy protection of the proposed FedMUD. The authors mentioned that “MUD's output bears a resemblance to data distillation in terms of its form.” This raises a natural concern: if clients generate synthetic samples and send them to the server, it could potentially reveal the local data distributions on the clients which compromising privacy protection in FL.

• As shown in Figure 3 and discussed in Section 5.2, the proposed method requires significant time for local computation. This raises the question of whether, when using larger models with more trainable parameters, the benefits of employing FedMUD as a communication reduction strategy might become marginal due to the increased local computation time.

• Why did the authors choose LBFGS as the default optimizer for FedMUD? Additionally, the comparison in the experiments might be somewhat unfair if FedMUD uses a different local optimizer than the other baselines. My concern is that the superior empirical results could be partially attributed to the benefits of using the LBFGS optimizer.

1. Limited theoretical analysis: Although some analysis is present, it has several drawbacks (compared to the analysis of standard FedAvg [1]):

   a) Restrictive theoretical assumptions: The paper's convergence analysis relies on a questionable Assumption 3.4 (line 730) about bounded error:

   $\mathbb{E}_{\xi\sim S_i} \| \omega_i^{(t)} - \omega_{i,o}^{(t)} \| \le \Delta^2, \quad \forall i \in [N], \forall x \in \mathbb{R}^d$

   This assumption requires that the error between the reconstructed local model ($\omega_i^{(t)}$) and the original model ($\omega_{i,o}^{(t)}$) is bounded by some constant $\Delta^2$. This is a strong assumption because:

   • $\Delta$ could be arbitrarily large in practice
   • The value of $\Delta$ is unknown before deploying the method
   • There's no practical way to verify or enforce this bound during training
   • The paper doesn't provide empirical evidence that this assumption holds in real scenarios

   b) In line 863, the authors establish a convergence rate $O\left(\frac{f_{\max }}{\sqrt{N T} \gamma}+\frac{N}{T} L^2\left(\gamma \mathbf{G}_{\max }+(T-1) \Delta\right)^2+\gamma \frac{L \sigma^2}{\sqrt{N T}}\right)$ that contains a term $O\left(NT L^2\Delta^2\right)$ not affected by the choice of stepsizes $\eta$ and $\gamma$. This leads to divergence when $T \rightarrow+\infty$ if $\Delta^2$ is fixed. While the authors claim:

   If the approximate error $\Delta$ decays over time $T$ at a rate of $\Delta_t=\frac{\Delta}{T^k}$, where $k>1$, then our method can achieve the same convergence rate as FedAvg, which is $O(1 / \sqrt{N T})$.
   

   The claimed decay of $\Delta$ over time $T$ is not demonstrated in the experiments.

2. Technical documentation issues: a) Numerous typos throughout the text:

   • Lines 702, 704, 728: "Aussumption" instead of "Assumption"
   • Line 754: "eta" instead of $\eta$

   b) Confusing notation:

   • Line 730: expectation is taken over some $S_i$ that is never defined

   c) Incomplete mathematical expressions:

   • Line 730: mathematical expression (11) lacks a quantifier $\forall t > 0$

3. Experimental evaluation limitations: a) Limited scalability: The paper lacks investigation of scalability to larger models (>100M parameters) and larger numbers of clients, which is crucial for real-world federated learning deployments [2,3].

   b) Missing experimental setup details:

   • No information about stepsize parameter tuning for baselines
   • Missing details about batch size B and initial rounds M
   • Only learning rate of 0.01 for FedMUD is specified

   c) Limited baseline comparison: Authors claim "We evaluate FedMUD against five state-of-the-art methods" but cite papers from 2017 [4] and 2020 [5] with outdated or unsuitable methods. More relevant state-of-the-art methods can be found in [6]. Moreover, FedPAQ [5] only compresses uplink communication, making it an unsuitable baseline for this case. Additionally, "TopK" typically refers to the compressor [7,8,9] rather than the entire method, making "SGD with TopK" more conventional terminology.

   d) Insufficient evaluation of partial participation: While partial participation is fundamental in real-world federated learning systems [2,3], where devices may be unavailable due to various constraints, the paper only presents a basic comparison between FedMUD and FedAvg under this setting.

4. Privacy implications: The paper doesn't analyze potential privacy implications of the synthetic tensor sequence, which might leak information about local updates [10].

References:

[1] Khaled et al., "Tighter Theory for Local SGD on Identical and Heterogeneous Data," arXiv 2019

[2] Bonawitz et al., "Towards Federated Learning at Scale: System Design," MLSys 2019

[3] Kairouz et al., "Advances and Open Problems in Federated Learning," 2021

[4] Aji et al., "Sparse communication for distributed gradient descent," arXiv:1704.05021, 2017

[5] Reisizadeh et al., "Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization," AISTATS 2020

[6] Cheng et al., "Communication-Efficient Distributed Learning with Local Immediate Error Compensation," arXiv 2024

[7] Huang et al., "Lower bounds and nearly optimal algorithms in distributed learning with communication compression," NeurIPS 2022

[8] Stich, "On communication compression for distributed optimization on heterogeneous data," arXiv 2020

[9] Beznosikov et al., "On biased compression for distributed learning," JMLR 2023

[10] Zhu et al., "Deep Leakage from Gradients," NeurIPS 2019

Very limited experiments - only on cifar10 and CRCSlide

Try more models ? why just resnet18 and LeNet -- at least a ViT-16 / RN 50 would be more representative beyond toy example

If we use Error Feedback and w a more light-weight compression operator like top-k we might get better results ?

EF + Compression as a baseline

In the exp could you explain the ( comm + Local up dat comm + MUD compute ) times = > also why is local update comm time different -- isn't it the time for local SGD steps ?

while you did mention in the limitations -> it would be nice to have a better sense of the costly updates with large param sizes => may be run some experiments w increasing param sizes => compare the update xomputation complexities