FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation

Dear Reviewer 4qUr, thanks for your comments, which helped us improve our paper. The answers to all your questions are as follows:

Q1: The method relies on multiple layers of approximation - empirical Fisher to approximate the Hessian, block-diagonal Fisher instead of full, and approximating global model parameters through optimization. Each approximation introduces some error. These compounding approximations could potentially lead to suboptimal global models. However, the paper's empirical results suggest that in many practical scenarios, these approximations still lead to good performance. Nonetheless, a more thorough theoretical analysis of these approximation errors and their impact would strengthen the paper.

Answer: In our paper, note that we have three approximations listed as follows:

(1) empirical Fisher to approximate the Hessian

In our Appendix E, we show the theoretical analysis of this approximation error.

(2) block-diagonal Fisher instead of full Fisher

Paper [1], which we cited in our paper, provides a detailed evaluation and testing of using block-diagonal Fisher to approximate the full one. Firstly, Chapter 6.3.1, "Interpretations of this approximation" in the paper [2] indicates that using a block-wise Kronecker-factored Fisher closely approximates the full Fisher. Although there is a bias term (due to the approximation in our Appendix Equation 30), this term approximates zero when there are sufficient samples. Furthermore, the paper examines the approximation quality of block-diagonal Fisher compared with the true Fisher and suggests that block-diagonal Fisher captures the main correlations, while the remaining correlations have a minimal impact on the experimental results.

(3) approximating global model parameters through optimization

In our Appendix J, we show the convergence analysis of our method.

In summary, the approximations have a negligible effect on the final test accuracy. Some experiment results are shown in Table 22 of Appendix M.12.

We will summarize these in the appendix once accepted.

[1] Ritter, Hippolyt, Aleksandar Botev, and David Barber. "A scalable laplace approximation for neural networks." 6th international conference on learning representations, ICLR 2018-conference track proceedings.

[2] Martens, James. Second-order optimization for neural networks. University of Toronto (Canada), 2016.

Q2: Although more efficient than some baselines, FedLPA still requires more computation than simpler methods like FedAvg.

Answer: Our method FedLPA only introduces 30% more computation time compared to the simple FedAvg while increasing the test accuracy by up to 35% in some settings, as shown in Table 1 and Table 4. As shown in Table 17, the state-of-the-art Dense will use 6.15x times the computation time of our FedLPA, while our method performs better in most cases considering test accuracy. Co-Boosting, another distillation method, will use 10.77x times the computation time of our FedLPA. Our method is also faster than FedProx and FedOV. Our method is compatible with FedOV and Co-Boosting and performs much better than FedProx.

Our FedLPA strikes a balance between the computation overhead and the performances.

Q3: Storing and transmitting the block-diagonal Fisher matrices for each layer increases memory usage and communication costs compared to methods that only share model weights. For very large models or with many clients, the increased communication overhead from sharing Fisher matrices could become significant.

Answer: Our FedLPA strikes a balance between the communication overhead and the performances. As shown in Table 4, FedLPA also has a communication overhead of only 2x times that of FedAvg. The communication overhead is similar between our FedLPA and the popular SCAFFOLD. In some settings, FedLPA could increase the performance up to 35% compared to FedAvg and SCAFFOLD.

In our paper, we also give detailed examples of the communication overhead of our FedLPA in Appendix M.8.

For all the datasets, as the number of clients increases, the communication overhead will also increase linearly. Intuitively, for very large models or with many clients, our communication overhead is solely about two times of FedAvg. Compared with the computation overhead in the one-shot setting, this is acceptable.

Dear Reviewers 4qUr,

We sincerely appreciate your time and efforts in reviewing our manuscript and offering valuable suggestions. Note that there will be no second stage of author-reviewer discussions. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues.

If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best regards,

Authors of Submission #8916

Dear Reviewers 4qUr,

We sincerely appreciate your time and efforts in reviewing our manuscript and offering valuable suggestions.

We understand you are busy viewing multiple papers. As the rebuttal deadline is approaching, we are slightly nervous and look forward to your reply or suggestions. We would be more than grateful if you took some time to confirm whether our responses have effectively addressed your concerns and increased the evaluation of our paper.

Please do not hesitate to contact us if you require further clarification or have any additional concerns. We are more than willing to continue our communication with you.

Thanks so much!

Best regards,

Authors of Submission #8916

Dear Reviewer XECp, thanks for your comments, which helped us improve our paper. The answers to all your questions are as follows:

Q1: Implementation Complexity: The use of layer-wise posterior aggregation and the empirical Fisher information matrix introduces significant complexity. The practicality of implementing FedLPA in real-world settings could be challenging without detailed guidelines or simplifications.

And

Complexity and Practicality: The complexity of the FedLPA framework may hinder practical implementation. Providing more detailed guidelines or potential simplifications could make the approach more accessible for real-world applications. Additionally, discussing potential trade-offs between complexity and performance would be beneficial.

Answer：The theoretical analysis of the layer-wise posterior aggregation may be complex, however, the practical implementation is not complicated. Besides, we have provided several functions with APIs to ensure non-experts can adopt our FedFPA framework within a few lines of code. In detail, in our submitted source code zip file, we provide the APIs for the users to use our method in line 541 of experients_our.py. We also include the artifact details in our paper and provide all the script files to reproduce the results of our paper.

In our paper, we also show that our FedLPA strikes a balance between the computation overhead and the performances and incurs solely 30% more computation time compared to the simple FedAvg method while increasing the test accuracy by up to 35% in some settings, as shown in Table 1 and Table 4.

Q2: Scalability: The paper does not adequately address the scalability of the proposed method to large datasets or a high number of clients. Evaluating the computational and communication overheads in such scenarios is necessary to understand the practical feasibility of FedLPA.

Answer：In our paper, our FedLPA shows good performances on MNIST, FMNIST, CIFAR-10, CIFAR-100, SVHN, EMNIST datasets. We further add the experiments on the same experiment setting in the paper using ResNet-18 on Tiny-ImageNet. The results show that our method has the potential to deal with large datasets.

We also add the experiments with more clients in the same experiment setting of the paper using FMNIST datasets. The results show that our method could support up to 200 clients.

The computational and communication overheads are shown in Table 4 in our paper. As the number of clients increases, the computational and communication overheads increase linearly.

Q3: Privacy Considerations: While the method claims to preserve data privacy, the paper lacks a detailed discussion on potential privacy risks and mitigation strategies, which is crucial for federated learning applications.

Answer：In Appendix L, we give a detailed discussion about privacy concerns for our FedLPA. We show that our method has the same privacy level as the baselines, counteractting the iDLG attack. Our method could also be compatible with existing privacy-preserving technologies (i.e., DP). At the end of Appendix L, we also provide a concrete example of privacy attacks to show how FedLPA preserves data privacy.

Q4: Experimental Scope: The experiments, although comprehensive, are limited to a few datasets and simple neural network structures. Additional validation on larger and more complex datasets, as well as a variety of neural network architectures, would provide stronger empirical support for the method's generalizability.

And

Experimental Validation: While the experiments are comprehensive, further validation on larger-scale datasets and more diverse GNN architectures would provide stronger empirical support for the proposed methods. Expanding the experimental scope would help demonstrate the applicability of the methods in various real-world scenarios.

Answer：In our paper, our FedLPA shows good performances on MNIST, FMNIST, CIFAR-10, CIFAR-100, SVHN, EMNIST datasets. We further did the experiments with Tiny-ImageNet, as shown above.

In Appendix M.10, we also show the results using more complex network structures such as ResNet-18.

In this paper, we conduct the experiments on CV tasks without focusing on the graph neural network.

Q5: Parameter Sensitivity: The performance of FedLPA may be sensitive to the choice of parameters, such as the noise levels for the empirical Fisher information matrix. A detailed analysis of parameter sensitivity and guidelines for selecting appropriate parameter values would enhance the robustness of the method.

Answer: In our paper, we do not have the noise level settings for the empirical Fisher information matrix. We only have one hyper-parameter, $\lambda$ from Eq. 33, which controls variances of a priori normal distribution and guarantees $A_k$ and $B_k$ are positive semi-definite.

All other Laplace Approximations are sensitive to the hyper-parameter $\lambda$ based on their experimental results, but Table 3 shows that our approach is relatively robust.

Dear Reviewers XECp,

We sincerely appreciate your time and efforts in reviewing our manuscript and offering valuable suggestions. Note that there will be no second stage of author-reviewer discussions. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues.

If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best regards,

Authors of Submission #8916

Dear Reviewers XECp,

Thanks for replying to us while you are busy reviewing multiple papers. Although you maintained your rating after careful consideration, we, feeling a little bit of pity, still appreciate your time and efforts in reviewing our manuscript and offering valuable suggestions. We would be grateful if you could increase the rating of our paper during the following discussions among the reviewers and AC.

Again, thanks for your time and reviews. If you have any further concerns, we will be more than happy to continue our communication with you.

Best regards,

Authors of Submission #8916

Dear Reviewer F77H, thanks for your comments, which helped us improve our paper. The answers to all your questions are as follows:

Q1: There are some typo problems.

Answer: Thanks for pointing that out and thanks for your efforts in reviewing our paper. We found the following typos and will revise if we find more.

Sec3.7 We->we

Sec4.1 benchmark->benchmarks

Appendix F in one-shot -> in the one-shot

Appendix M.13 proportion -> proportions

Q2: Authors should compare the proposed method with differential privacy (DP) FL or prototype-based methods rather than conventional FL methods. All these approaches address communication security but from different perspectives.

And

As I mentioned in the weakness, the authors should compare the proposed method with DP FL and prototype-based FL rather than conventional FL methods. I believe the experiments in the paper do not provide a fair comparison.

Answer: Beyond the privacy protection and the reduction on the attack surface, the one-shot FL framework also tries to deal with the substantial communication overhead and higher demand for fault tolerance throughout the rounds. In Appendix L.1, we show the experiment of DP-FedLPA and DP-FedAvg in the one-shot setting. Specifically, since the sensitivity of the data sample distribution after the normalization is 1, we add Laplacian noises with $\lambda=\frac{1}{\epsilon}$. We set $\epsilon=\{3,5,8\}$ that provides modest privacy guarantees since normally $\epsilon \in (1,10)$ is viewed as a suitable choice.

Besides, we have added the experiments using the same experiment setting in the paper on the FMNIST dataset to compare DP-FedAvg (multiple-round) as Appendix L.1 with our FedLPA (one-shot), show the round results of how many rounds DP-FedAvg needs to achieve the same test performance.

The results show that DP-FedAvg needs about 10 rounds of communication to achieve the same test performance, compared to our one-round FedLPA. Combined with our previous results in Table 4 and Table 7, our FedLPA could save the communication and computation overhead and combine with the DP method to mitigate the potential privacy leakage. Based on the above settings, DP-FedAvg needs at least 3x communication overhead and 5x computation overhead. While DP-FedAvg needs multiple rounds to get similar accuracy, DP-FedAvg maybe vulnerable to more privacy attack methods due to the multiple queries, such as curvature-based privacy attacks.

Q3: Would the proposed method also show superior performance on more challenging datasets, like tinyimagnet and office-home?

Answer: In our paper, our FedLPA shows good performances on MNIST, FMNIST, CIFAR-10, CIFAR-100, SVHN, EMNIST datasets. We further add the experiments on the same experiment setting in the paper using ResNet-18 on Tiny-ImageNet. The results show that our method has the potential to deal with large datasets.

Due to the rebuttal time limit, we leave the implementation on office-home dataset as future work. However, we believe that the FedLPA performs well on office-home dataset, since it shows the potential to deal with challenging data on Tiny-ImageNet dataset.

Q4: Any proof to show that "computing the co-relations between different layers brings slight improvement"?

Answer: Paper [1] we cited in our paper provides a detailed evaluation and testing of using block-diagonal Fisher to approximate the full one. Firstly, Chapter 6.3.1, "Interpretations of this approximation" in the paper [2] indicates that using a block-wise Kronecker-factored Fisher closely approximates the full Fisher. Although there is a bias term (due to the approximation in our Appendix Equation 30), this term approximates zero when there are sufficient samples. Furthermore, the paper examines the approximation quality of block-diagonal Fisher compared with the true Fisher and suggests that block-diagonal Fisher captures the main correlations, while the remaining correlations have a minimal impact on the experimental results.

We will add the above analysis in the appendix on the camera-ready version.

[1] Ritter, Hippolyt, Aleksandar Botev, and David Barber. "A scalable laplace approximation for neural networks." 6th international conference on learning representations.

[2] Martens, James. Second-order optimization for neural networks. University of Toronto, 2016.

Q5: According to the authors in Supplementary Sec.L, the security level of the proposed method is similar to that of FedAVG, meaning it is also vulnerable to attacks targeting FedAVG. Does the only advantage of the proposed method is that attackers have fewer opportunities to strike since they only communicate with the server once?

Answer: In fact, our approach can reduce the attack surface as attackers have fewer opportunities since they only communicate with the server once. Beyond that, in the one-shot setting, the FedLPA could also deal with the substantial communication overhead and higher demand for fault tolerance throughout the rounds.

FedAvg is vulnerable to privacy attacks with huge communication overhead. However, FedLPA is compatible with existing preserving approaches (i.e., DP) to achieve an even higher privacy level and strike a balance between computation, communication overhead, and performance. Note that our paper mainly focuses on the efficiency perspective to improve the performance of one-shot FL with inferior consideration of the security perspective.

Dear Reviewer F77H,

We sincerely appreciate your constructive comments and prompt responses, which help us improve our paper. It is our pleasure to address your concerns during the discussion. Again, thanks for your time and reviews. Thanks for evaluating the value of our work!

Warm regards,

Authors of Submission #8916