Communication Efficient Federated Learning via Model-Agnostic Projection Adaptation

1. Architecture-independency

You are correct. Our approach flattens the gradients and factorizes them in a matrix form rather than directly factorizing the parameters. This gradient-based factorization is independent of specific architectural details, making it applicable to any model.

2. Centralized setting

We discussed this issue in response 5 to reviewer NJUL and presented the experimental results in response 1 to reviewer FKCh.

3. Layer-wise factorization vs. Entire model factorization

While layer-wise factorization is possible, it introduces following practical issues:

1. Memory Overhead: For reducing model communication $k$-fold, global factorization on $W^d$ results in $W^{k * d/k} = A^{k * 1} B^{1 * d/k}$, while layer-wise factorization of an $n$-layer model, given layer-$i$'s parameters as $W_i^{d_i}$ will be $W_i^{k * d_i/k} = A_i^{k * 1} B_i^{1 * d_i/k}$ requires storing $n$ separate $A_i$, increasing memory overhead $n$ times.

2. Architecture Constraints: In global model factorization, we can essentially decide on any arbitrary rate of compression. In contrast, layer-wise factorization faces limitations, as choosing $k$ higher than the layer size is impossible. As shown experimentally in response 3 of reviewer FKCh, global factorization enables effective fine-tuning even at 10k-fold compression for a model with 357M parameters. Meanwhile, individual compression of a 1024-parameter layer at 10k-fold is not possible.

3. Suboptimal Performance: Lastly, global factorization outperforms layer-wise factorization since layer-wise allocates relatively equal expression budgets $d_i/k$ across layers, regardless of their gradient magnitudes. This leads to suboptimal communication budgeting, unlike global factorization, which allocates a higher expressivity budget to high magnitude gradients and higher compression to less informative gradients.

Additional experiments on QNLI and SST2 illustrate this suboptimality:

4. Backbone is updated

You are correct. Unlike PEFT methods such as LoRA, MAPA does not inherently reduce the number of model parameters, as the full model backbone is updated at each communication round. Instead, MAPA primarily reduces gradient communication overhead in FL, as minimizing communication overhead is typically more critical than reducing parameter storage in this setting.

5. Comparison with more recent baselines

Many FL LoRA studies address broader challenges than communication. We used FA-LoRA as a central baseline due to its communication focus and conceptual similarity. Per your suggestion, we added LoRA and SA-LoRA to our baselines for LLM fine-tuning comparisons. (see response 3 to reviewer FKCh).

6. Forward pass cost

LoRA adds computational overhead in the forward pass by including additional low-rank adaptation layers. The computation $y = Wx + BAx$ incurs complexity:

• Frozen parameters: $O(d^2)$
• LoRA layers: Two multiplications $Ax$ and $B(Ax)$: $O(2dq)$

Thus, LoRA’s total forward pass complexity is $O(d^2 + 2dq)$. In contrast, MAPA applies low-rank factorization only during the backward pass, leaving the forward pass complexity unchanged at $O(d^2)$.

7. Why is it centralized in Fig. 2?

In Fig. 2, we used a single-client setup to isolate the intrinsic performance impact of gradient compression, independent of data heterogeneity or client sampling in FL. Thus, any differences reflect compression effectiveness alone.

8. Typo

Thank you; we fixed it.

9. As B is not the optimal point, can having more parameters in A lead to better convergence?

In MAPA, updates follow $W_t = W_{t-1} + AB$, where the $A$ is fixed within each round. Therefore, gradient $B$ computation is just the gradient $\nabla W$ projection linearly onto the subspace spanned by $A$, and SGD can reliably solve linear projection to find a near-optimal $B$.

10.Theorem 4.3

We thank the reviewer for this insightful point. The JL lemma provides a probabilistic guarantee of low-distortion embeddings. As such, to be entirely rigorous, our convergence statement can be made a high-probability result by union-bounding the (small) failure probability $\delta$ over the $T$ rounds:

• At each round, distortion $\le\epsilon$ holds with probability $\ge 1-\delta$.
• By union bound, for all $T$ rounds simultaneously, the probability is $\ge 1 - T\delta$.
• Conditional on that event, the same inequalities apply, and the convergence proof remains identical. We fully agree with you and will strengthen the theorem statement via the presented arguments.

11. Pre-trained Models

Original experiments were trained from scratch. For LLM fine-tuning, we used the HuggingFace FacebookAI/roberta-large checkpoint.

1. Key contributions

Thank you for raising this important point. To clarify, there are two fundamentally different strategies for leveraging low-rank structures in optimization:

1. Low-Rank Parameterization
2. Low-Rank Gradient Projection

MAPA explicitly utilizes the latter strategy, whereas LoRA and its variants follow the first approach.

Why LoRA should be applied layer-wise? Low-rank parameterization methods, like LoRA, inherently depend on layer-wise decomposition, as reparameterization must preserve input/output dimensions for each layer to maintain forward pass compatibility. LoRA decomposes each layer’s weight matrix $W$ individually as $h = (W+BA)x=Wx + BAx$. Treating the entire model parameters as a single matrix violates compatibility due to nonlinear activations and differing layer dimensions. LoRA paper acknowledges this constraint (Hu et al., LoRA [Page 4, Section 4]).

Given a model-level LoRA factorization of $W^{I *O}$ into $A^{I * r}$ and $B^{r * O}$, where $I$ and $O$ are input and output dimensions of the model and $r$ is the factorization rank, the forward pass will be reduced to $y = BA(x)$, which does not express any nonlinearity of the network.

In contrast, MAPA employs gradient factorization rather than parameter factorization. By applying low-rank constraints directly on the gradient instead of the parameters, MAPA reduces the gradient size while fully preserving model capacity. This approach is not constrained by layer-wise decomposition or model architecture since factorization occurs after computing gradients via standard forward/backward passes. Further discussion on gradient factorization literature appears in response 5 to reviewer NJUL.

2. Theoretical contribution

Our convergence analysis extends standard federated SGD proofs [3,4] by incorporating a random projection (via the JL lemma) that introduces distortion $\epsilon$. This affects both descent direction and update variance. Unlike works that assume a fixed subspace or no gradient projection, we rigorously track how random, time-varying subspaces influence FL convergence. When $\epsilon=0$, MAPA becomes FedAvg. For $\epsilon>0$, we add a factor $(\epsilon + \beta + \epsilon\beta)$ but retain the same $\mathcal{O}(1/\sqrt{T})$ rate.

We will revise the manuscript to emphasize this distinction in our convergence analysis.

3. Fine-tuning of LLMs on larger datasets

Based on your feedback, we conducted fine-tuning experiments of RoBERTa-large on five large datasets of GLUE tasks. We evaluated MAPA, alongside LoRA, FA-LoRA, and SA-LoRA [2].

The 1st Table below compares the number of trainable parameters and communication load per round for each baseline.

The 2nd Table summarizes the results of fine-tuning, in which communication efficiency is evaluated by the number of rounds and the total communication needed to reach 80% accuracy, and the 3rd Table presents the results for centralized LLM fine-tuning.

The experiments used base code from [12], following the experimental setup and parameters from [2] for 300 FL rounds,

References are located in response 4 to reviewer Wq8k.

---

1st Table:

---

2nd Table, FL fine-tuning:

---

3rd Table, centralized:

Overall, it can be seen that MAPA has the potential to enhance fine-tuning performance in centralized training too.

1. Only updating B

Thank you for highlighting this concern. Indeed, relying solely on $B$ limits subspace exploration, as seen in FA-LoRA’s performance decline, SA-LoRA [2], and Figure 7. MAPA addresses this by randomizing $A$ each round, promoting diverse subspaces. Figure 7 shows that fixing $A$ at low ranks severely degrades accuracy, whereas randomizing $A$ maintains performance. We further verified this advantage in response 2, reviewer NJUL.

2. Comparison with [1]

We appreciate your mentioning Factorized-FL. Below are key distinctions alongside comparative experiments under the same setup:

The key difference is that Factorized-FL applies factorization on layer-wise parameters, whereas MAPA factorizes the gradient of the entire model. In response to similar questions, we previously elaborated on why gradient (response 1, reviewer FKCh) and model-level factorization (response 3, reviewer JjTg) can outperform layer-wise parameter factorization.

Although both methods use rank-1 factorization, in Factorized-FL, rank-1 is a hyperparameter needing tuning or increasing for larger models to avoid limiting representation capacity. In MAPA, rank-1 is inherent and does not restrict model capacity; instead, the reshaping factor $k$ determines the compression rate. Consequently, Factorized-FL's communication per round is constrained by model dimensions, while MAPA can compress gradients to arbitrary degrees independent of architecture dimensions.

Factorized-FL is similar to a rank-1 LoRA architecture, with a sparse bias matrix replacing LoRA’s frozen fine-tuned parameter, initialized as zero. LoRA imposes strict regularization to preserve pre-trained parameters, whereas Factorized-FL employs softer regularization, allowing updates when necessary.

Factorized-FL emphasizes personalized FL by sharing one vector globally and keeping the other client-specific. To directly compare factorization effectiveness with MAPA, one could share both vectors globally. However, as noted by the authors [1] (Page 6, Personalized Weight Averaging), sharing both vectors significantly increases communication load, adversely affecting efficiency. Below, we highlight this fact by comparing global model training on CIFAR-10 and SVHN under IID and non-IID splits. “Com@X%” indicates total communication needed to reach X% of FedAvg’s final accuracy:

3. Theorem 4.3

We apologize for the confusion regarding our convergence result. Our convergence bound matches FedAvg and recovers it as a special case: when reconstruction error is zero ($\epsilon = 0$), MAPA reduces exactly to FedAvg with the tightest convergence bound. For $\epsilon \neq 0$, the bound introduces a modest constant factor $(\epsilon + \beta + \epsilon\beta)$ due to compressed update distortion. Nevertheless, MAPA maintains the same asymptotic rate $\mathcal{O}(1/\sqrt{T})$ as FedAvg under standard assumptions (smoothness, bounded variance) [3,4]. Practically, this means MAPA might require slightly more rounds at higher compression, yet the total communication cost to achieve target accuracy significantly decreases, allowing training with substantially reduced overhead.

4. References

[1] Jeong, W. and Hwang, S.J. "Factorized-FL: Personalized Federated Learning with Parameter Factorization & Similarity Matching."
[2] Guo, P. et al. "Selective Aggregation for Low-Rank Adaptation in Federated Learning."
[3] Yu, H. et al. "Parallel Restarted SGD with Faster Convergence and Less Communication."
[4] Kim, D.-Y. et al. "Achieving Lossless Gradient Sparsification via Mapping to Alternative Space in Federated Learning."
[5] Denil, M. et al. "Predicting Parameters in Deep Learning."
[6] Li, C. et al. "Measuring the Intrinsic Dimension of Objective Landscapes."
[7] Gressmann, F. et al. "Improving Neural Network Training in Low Dimensional Random Bases."
[8] Aghajanyan, A. et al. "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning."
[9] Hameed, M.G.A. et al. "ROSA: Random Subspace Adaptation for Efficient Fine-Tuning."
[10] Zhao, J. et al. "Galore: Memory-Efficient LLM Training by Gradient Low-Rank Projection."
[11] Zhao, H. et al. "SEPARATE: A Simple Low-Rank Projection for Gradient Compression."
[12] Kuang, W. et al. "Federatedscope-LLM: A Comprehensive Package for Fine-Tuning LLMs in Federated Learning."

1. MAPA does not consistently outperform certain baselines

Thank you for your careful observation. We want to emphasize that all the results provided in Table 2 and additional experiments during this rebuttal show that MAPA consistently outperforms in communication and performance.

The results shown in Figure 5, in the top row, do not consider communication load. We are just comparing in terms of global rounds. As we take communication load into account, as shown in Table 2, we always perform better than baselines in performance per communication.

 

2. The paper lacks ablation studies

Thank you for your constructive feedback. We initially provided our ablation studies on the effect of matrix rank on training (Figure 7) and the importance of fixed vs. fresh matrix A (Figure 6). Considering your comments, we additionally extended our studies on:

1. Fixed vs. fresh (randomization) of matrix A

To elaborate on the effectiveness of randomization, additional experiments regarding MNIST and CIFAR10 are presented here, showcasing the accuracy across various ranks from $2^0$ to $2^{13}$, which clearly highlights the advantage of randomization, especially at lower ranks. Moreover, a discussion on the importance of randomization in training is located in response 1 to reviewer Wq8k.

2. Effect of rank in LLM fine-tuning

We study the effect of MAPA rank on four different orders of magnitude alongside LoRA's baselines in communication-efficient LLM fine-tuning. The results are presented in the 2nd Table of response 3 for reviewer FKCh.

3. Experiments on layer-wise vs. model-level factorization

Additionally, during the rebuttal, we conducted further experiments on LLM fine-tuning across various MAPA ranks, clarifying the trade-off between communication and performance, and additionally conducted an ablation study on layer-wise vs model-level factorization. (See response 3 to reviewer JjTg)

If concerns remain, please specify any additional ablation studies you recommend. We remain committed to conducting further experiments.

 

3. The comparison between FA-LoRA and MAPA

Methodologically, MAPA:

1. Factorizes gradients, not parameters (response 1, reviewer FKCh).
2. Uses a randomized $A$ instead of a fixed $A$, as shown in our ablations and response 1 to reviewer Wq8k.
3. Operates at the model level rather than layer by layer (response 3, reviewer JjTg).

We further validated these claims via additional GLUE fine-tuning experiments against FA-LoRA and other baselines (response 3, reviewer FKCh).

 

4. Centralized fine-tuning

Following your advice, we tested MAPA in a centralized setup and observed substantial gains over other baselines (3rd table in response 3, reviewer FKCh).

 

5. Have similar ideas been explored in centralized ML?

The literature on low-rank gradient factorization in deep learning can start from:

• [5] shows the inherent low-rank structure of gradients.
• [6] examined intrinsic dimensionality by identifying the lowest-dimensional fixed random subspace enabling model convergence. Subsequent works [7–11] expanded on these concepts by training NN within randomly generated gradient subspaces.

Although these approaches shows the efficacy of low-rank gradient factorization, they suffer from extensive memory overhead, as they represent the gradient as a single vector $G^d$, where $d$ is the number of model parameters, resulting considerable memory usage to construct the random transformation $A^{d \times m}$.

MAPA significantly differs from prior approaches by reshaping gradients before factorization. This simple yet effective modification achieves roughly $k$-fold reduction in computation and $k^2$-fold lower memory usage without compromising performance, supported by our theoretical and empirical analyses (Appendix H and C.5). Additional discussion comparing gradient vs parameter factorization appears in response 1 to Reviewer FKCh.

References are located in response 4 to reviewer Wq8k.

 

6. What makes this method particularly beneficial in FL?

A primary challenge in FL is mitigating communication overhead. Our MAPA directly addresses this via low-rank gradient factorization integrated with efficient communication. While highly beneficial in FL, gradient reductions offer limited advantages in centralized settings, where gradient communication isn't required.