Communication-Efficient Federated Learning via Model-Agnostic Projection Adaptation

6. Baselines:

Besides, the selection of baseline algorithms in the experimental evaluation is inadequate.

We added quantization as a new baseline for communication efficiency for all experiments. We want to emphasize that this paper's main objective and scope is to provide a communication-efficient algorithm for FL, so we evaluated baselines in the scope of communication efficiency and not regarding other aspects of FL.

As most communication-efficient techniques can be combined and SOTA approaches offer a hybrid solution of sparsification and quantization, we believe it can be satisfactory to compare these fundamental ideas. In practice, one can effortlessly combine MAPA with quantization and sparsification or a hybrid of different techniques to achieve better performance.

Regarding communication-efficient baselines, we currently provide sparsification and quantization as the most common fundamental approaches, alongside EvoFed and FFA-LoRA SOTA, which have similar methodologies to our work to highlight our method's advantages.

---

7. MAPAX performance:

Although MAPAX is proposed to address the computational and memory burdens introduced by MAPA, the paper does not present any experimental results for it.

We understand the importance of investigating MAPAX performance and representing the results more clearly. Based on all reviewers' comments, we dedicated Figure 6 to explicitly presenting MAPAX performance and analysis.

Overall, Figure 6, Table 1, and Appendix F demonstrate that MAPAX can perform similarly to MAPA while using significantly less memory. Compared to MAPA, MAPAX's performance difference is less than 0.3%, and the additional communication burden does not exceed 1.0%.

1. Computational and memory concerns:

The method requires generating and storing a matrix ... with entries, where ... represents the number of parameters in the entire model, which is typically very large.

We appreciate the reviewer's observation and acknowledge the concerns regarding the large reconstruction matrix. First, we want to note that MAPA's memory complexity (although very high) aligns with many prior works in this area and is not unusual.
For instance, [4,5] store random matrices, and [6,7,8] store previous gradient signals that have a similar shape to MAPA's $A$.

Based on your concern, we updated the manuscript to emphasize that MAPAX demonstrates performance comparable to MAPA, as supported by additional experimental results and theoretical justification provided in the revised manuscript. Therefore, our partitioning approach (MAPAX) significantly reduces memory usage while maintaining similar performance, making MAPA accessible to low-resource clients.

Please refer to the updated Table 1, Figures 6, and Appendix F for empirical results comparing MAPAX and Proposition 3 and Appendix E for the theoretical analysis.

---

2. Random matrix generation:

The matrix is generated randomly, similar to the approach in FAA-LoRA [1]. Recent studies [2] have indicated that random generation of such matrices may lead to suboptimal performance.

Thanks for pointing this out. We also acknowledged that the FFA-LoRA frozen random matrix is suboptimal and provided evidence in Appendix B. Regarding your comment, we have updated the text to emphasize that our matrix generation differs significantly from FFA-LoRA. Specifically, we generate a new matrix at each iteration, whereas FFA-LoRA uses a fixed and frozen matrix $A$ throughout the entire training process.

This limitation in FFA-LoRA arises from the architectural differences compared to MAPA. In MAPA, $W$ is not inherently fixed during training; after obtaining the optimal $B$, we update $W_t = W_{t-1} + AB$, which allows us to generate a new matrix $A$ and reset $B$ to zero at each iteration. In contrast, FFA-LoRA techniques keep $W$ fixed, necessitating that matrix $A$ remain unchanged to preserve training stability and use matrix $B$ to accumulate training adaptation. Consequently, our approach allows for continuous change in $A$, which prevents suboptimal performance, as also demonstrated in Appendix B of the initial submission.

Regarding the [2] solution, while this paper provides valuable insights into the problem, it is not directly applicable to communication-efficient training in federated learning since they train and communicate the matrix $A$, undermining the ability to achieve communication efficiency. We have updated the manuscript (Lines 171–181) to acknowledge the findings of [2] while presenting our proposed solution and details in Appendix B.

---

3. Representation capacity rate:

The ‘representation capacity rate’ introduction in Definition 2 lacks clarity and justification.

We agree with you on this point and have clarified our intentions for this definition. We provided a new definition of representation certainty, which revolves around error rate and variance, and updated the propositions accordingly. Importantly, this update does not impact our methodology, results, convergence analysis, motivation, or the intuition behind our solution; it only revises the theoretical explanation of MAPA and MAPAX.
We would be happy to know if you find the current version of the theoretical analysis satisfactory.

---

4. Convergence analysis and Assumption 4:

The convergence analysis is built upon Assumption 4, which appears too restrictive and not reflective of practical scenarios.

We appreciate your constructive feedback. We relaxed Assumption 4 to have an unbiased estimator instead of a bounded error. We addressed concerns using the Johnson–Lindenstrauss lemma to justify Assumption 4. Additional details are provided in Appendix C.

References:
[4] Shi, Zai, and Atilla Eryilmaz. "Communication-efficient Subspace Methods for High-dimensional Federated Learning." 2021 17th International Conference on Mobility, Sensing and Networking (MSN). IEEE, 2021.
[5] Mahdi Rahimi, Mohammad, et al. "EvoFed: Leveraging Evolutionary Strategies for Communication-Efficient Federated Learning." arXiv e-prints (2023): arXiv-2311.
[6] Azam, Sheikh Shams, et al. "Recycling model updates in federated learning: Are gradient subspaces low-rank?." International Conference on Learning Representations. 2021.
[7] Oh, Yongjeong, et al. "Vector quantized compressed sensing for communication-efficient federated learning." 2022 IEEE Globecom Workshops (GC Wkshps). IEEE, 2022.
[8] Park, Sangjun, and Wan Choi. "Regulated subspace projection based local model update compression for communication-efficient federated learning." IEEE Journal on Selected Areas in Communications 41.4 (2023): 964-976.

Thank you very much for your prompt response. We would greatly appreciate it if you could specify which concerns remain particularly relevant and which ones have been resolved. This will help us focus on addressing the remaining points effectively.

Your timely feedback is invaluable and provides us with an opportunity to reflect on and refine our work. Thank you again for your support!

1. Lack of diverse model architectures:

Given the author's claim that MAPA is model-agnostic, the paper could strengthen this claim by combining MAPA with more FL models to demonstrate its versatility beyond the current implementations.

We extended our experiments on CIFAR-100 using a ResNet architecture with 5.5 million parameters. Our study now includes three distinct architectures, ranging from 11,000 to 1.4 million and 5.5 million parameters. The experimental results demonstrate that MAPA scales effectively to larger tasks and diverse architectures and performs close to FedAvg, regardless of significantly less communication cost.

The CIFAR-100 experimental results show that MAPA and MAPAX can achieve around 93.3% of the FedAvg performance while using nearly 1.0% of the communication. The full results can be found in Table 1 and Table 6 of Appendix F.

---

2. Missing MAPAX results:

The paper lacks comprehensive experimental results for MAPAX, specifically missing accuracy and minimum cost outcomes on MNIST and CIFAR-10.

Initially, Figure 6 was intended to illustrate various factorizations, including no factorization (FedAvg), LoRA, MAPA, and MAPAX. However, based on your comments and other reviewers' requests for a more in-depth analysis of MAPAX, we revised Figure 6 to focus exclusively on MAPAX results for FMNIST, providing a more detailed evaluation. Results for MNIST and CIFAR-10 datasets are now added to Figure 6.

Figure 6, Table 1, and Appendix F demonstrate that MAPAX can perform similarly to MAPA while using significantly less memory. Compared to MAPA, MAPAX's performance difference is less than 0.3%, and the additional communication burden does not exceed 1.0%.

---

3. IID and sampled-client scenarios:

I am curious whether the proposed method can be applied to the IID setting or the scenario where clients are sampled in each communication round.

We conducted additional experiments on IID settings and client sampling, where 10% of clients are sampled per communication round. The results are in Appendix F, confirming MAPA's performance under these conditions. We appreciate your feedback, as these experiments can provide valuable insights into why MAPA outperforms other baselines.

The IID and client sampling results can be summarized as follows:

Table a. Performance summary in the IID setting for all clients.

Table b. Performance summary in the IID setting for 10% of clients.

These findings demonstrate that MAPA's strong performance in non-IID cases does not come only from capturing more comprehensive gradient information but also from better adaptation. We conclude this from the fact that various baselines perform much closer to MAPA in IID settings but fail to adapt as well as MAPA in non-IID settings.

We can look at non-IID FL settings as continuously adaptive learning, which suggests that methods from adaptive learning, such as low-rank approximation, can be more effective.

---

4. Presentation issues:

The text is too small to read in Fig. 4; misalignment between L. 517-524 and Fig. 6.

We appreciate your attention to these details; we have improved the readability of Figure 4 and corrected the misalignment issues. Additionally, as you suggested, Figure 6 has been revised to focus exclusively on the MAPAX evaluation, providing a more thorough and detailed analysis to address reviewer concerns.

1. Model-agnostic claim:

Although the method is described as 'model-agnostic,' it currently uses the same CNN model and identical initial weights across all clients in the first round. This setup could contradict the claim of model agnosticism...

We appreciate the reviewer's observation and would like to clarify that our intention for the term "model-agnostic" in our work refers explicitly to the flexibility of the factorization framework, which operates on the parameter space as a whole, independent of the network's specific architecture or layer structure. This independence allows MAPA to generalize to various model types without requiring specialized adaptations for each architecture.

Our work does not address heterogeneity in model architectures or initializations across clients, as this is orthogonal to MAPA's contributions. Instead, the framework focuses on enabling efficient communication through black-box factorization, which remains valid regardless of the specific client setup.

This is due to the fact that we primarily focus on minimizing the reconstruction error as $\|\Delta W - AB\|_2^2$, which, given $A$ drawn from a random Gaussian distribution, spans the sub-space uniformly and ignores existing structure or correlation in $\Delta W$. This makes it a general black-box factorization technique.

---

2. Additional experiments on model architectures:

Additional experiments with different model architectures are recommended to showcase the method's versatility and robustness.

We extended our experiments on CIFAR-100 using a ResNet architecture with 5.5 million parameters.
Our study now includes three distinct architectures, ranging from 11,000 to 1.4 million and 5.5 million parameters. The experimental results demonstrate that MAPA scales effectively to larger tasks and diverse architectures and performs close to FedAvg, regardless of significantly less communication cost.

The CIFAR-100 experimental results show that MAPA and MAPAX can achieve around 93.3% of the FedAvg performance while using only nearly 1.0% of the communication. The full results can be found in Table 1 and Table 6 of Appendix F.

---

3. Dataset scalability:

Further testing on more complex datasets, such as ImageNet, would help demonstrate MAPA's scalability.

To demonstrate scalability, we performed additional experiments on CIFAR-100. The results in Table 1 and Appendix F highlight MAPA's effectiveness in handling complex datasets, achieving 93.3% of FedAvg performance while using nearly 1.0% of communication. While conducting extensive experiments on ImageNet would be ideal, given the short review timeline, it is beyond our available resources. If the current results are deemed satisfactory, we are committed to adding additional dataset experiments for the camera-ready version.