All for One and One for All: A Collaborative FL Framework for Generic Federated Learning with Personalized Plug-ins

Q2: comparison with FedRoD

The discussion with the prior work (Chen & Chao, 2021) is not clear to me. In (Chen & Chao, 2021), the authors are also designing a generic feature extractor, and two plug-in heads: one is used for the generic scenario and the second is used for the personalized scenario. If the authors aim to do both generic and personalized testing during inference, one may simply choose between the generic plug-in module and the personalized plug-in module developed in (Chen & Chao, 2021), instead of selecting the plug-in modules from all clients in the system (this can significantly improve resource consumptions while achieving the similar performance). (a) What is the advantage/contribution of the authors work compared to these methods/concepts? I'm not sure whether the authors' approach will provide any advantages compared to this method. (b)The authors should provide comprehensive experiments as well as discussions regarding this. Related to this comment, $c$ probably the authors' scheme can have advantage compared to (Chen & Chao, 2021) when considering mixed dataset of personalized and generic dataset similar to as done in [7]: Each client's test dataset has more local classes when the distribution shift is small, while having more classes outside of the client's local dataset when the distribution shift is large.

Authors' Response: (a) While FedRoD [4] can handle both GFL-GM and PFL-PM by two plug-in heads, our proposed setting GFL-PM is different from GFL-GM [5] and PFL-PM [6], because in our setting distribution of local test data may not the same as that of training data, in this situation, a single personalized classifier is not enough for the diversified test data. (b) Our experiment also shows that FedRoD's local classifiers trained on local training data can't handle test data in different distributions, leading to poor performance in our proposed GFL-PM setting. It fails to provide a more fine-grained solution compared with our methods. As shown in Table 2 of origin main text, even using the global classifier to infer on every client in the GFL-PM setting (it is the same as the performance in the GFL-GM setting as the model in every client is identical), our method shows a significant performance gain compared with FedRoD in most cases. $c$ Our proposed set-up GFL-PM is totally different from the setting in [7], which is an interpolated dataset of local test data and global test data. In our set-up, clients may meet with local test data from both other clients and themselves. Since the problem we are trying to solve is quite divergent, we are not sure if our proposed method HPFL is able to perform well in setting in [7], as we assign a single plug-in from other clients or the client itself to the client, instead of interpolating the local head and the global head.

Reference

[1] Hoi, Steven CH, Doyen Sahoo, Jing Lu, and Peilin Zhao. "Online learning: A comprehensive survey." Neurocomputing, 2021.

[2] Jiang, Liangze, and Tao Lin. "Test-Time Robust Personalization for Federated Learning." In ICLR, 2022.

[3] Tan, Yue, Chen Chen, Weiming Zhuang, Xin Dong, Lingjuan Lyu, and Guodong Long. "Is Heterogeneity Notorious? Taming Heterogeneity to Handle Test-Time Shift in Federated Learning." In NeruIPS, 2023.

[4] Chen, Hong-You, and Wei-Lun Chao. "On Bridging Generic and Personalized Federated Learning for Image Classification." In ICLR, 2021.

[5] Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017.

[6] Arivazhagan, Manoj Ghuhan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. "Federated learning with personalization layers." arXiv, 2019.

[7] SplitGP: Achieving both generalization and personalization in federated learning, arXiv, 2022.

We truly thank the reviewer for taking the time to review. We appreciate that you find our proposed problem interesting and important, the result of our method promising, and the paper is easy to follow. According to your valuable comments, we provide detailed feedback. Due to similar concerns of different reviewers on the scalability and privacy safety of our method, which are all solvable by uploading the noise test features to the server for selection plug-ins, we have conducted experiments to testify the feasibility of this slight modification against communication and storage burden of HPFL in FL systems with plenty of clients. The results are in the Solution toward shared scalability and privacy concern. Please find special responses to your comments below.

Q1.1: scalability problem

Whenever a test sample appears, the scheme requires the client to download all the features from the server (which is same as the number of clients in the system), and compute (8). This can incur communication, computation, and latency issues.

Authors' Response: Thanks for pointing out potential problems when HPFL meets with an FL system with a great number of clients. As the number of clients grows to a large number, clients can choose to add noise to their local test features and send the noised test features to the server to select the plug-ins. In this way, each client can get the required plug-in without the need to download all the plug-ins and the noised training features, which cause a great communication cost with a great number of clients in the FL. We conduct experiments to testify the feasibility of this method against the communication burden of HPFL in FL systems with plenty of clients. Please find them in the Solution toward shared scalability and privacy concern.

Q1.2: Online test-time adaption

Moreover, each client may need to download multiple plug-in modules during inference, which will again incur additional communication and latency. For example, suppose there are three clients, each having classes (1, 2), (3, 4), (5, 6) in its local dataset, and produce plug-in modules 1, 2, 3, respectively. Now for an arbitrary client X, when the first test sample belongs to class 1, client X will download the plug-in module 1. Then, if the next test sample belongs to class 2, client X will download the plug-in module 2. Finally, it will also download plug-in module 3 whenever classes 5 or 6 appears in the test set. Is this really happening in the proposed method?

Authors' Response: HPFL is designed to infer in batches, online inference where test samples arrive one by one is not our main application scenario [1, 2, 3]. Selecting plug-ins for every sample may incur expensive computation costs. However, when met with a similar situation and the computation costs are unavoidable, clients can give up downloading plug-ins. Similar to method stated in Solution toward shared scalability and privacy concern, following instructions free clients from the storage burden brought by downloading plug-ins and corresponding training features for every sample: simply send the noised test feature to the server; the server select the appropriate plug-in; then infer at the server side; and finally return the inference result back to the client. In this way, communication and latency issues with online inference can be solved with slight modifications in HPFL. We have added Appendix F.2.2 in our revision to discuss how HPFL can adjust to special circumstances like online inference.

Q1.3: clarification on details of our method

Finally, it is not clear how the server should generate $h_m$ tailored to a specific client. For example, again consider an example with three clients having classes (1, 2), (3, 4), (5, 6). If class 1 appears as a test sample in client 1, what should the server send to this client? In other words, what are $h_1$, $h_2$, $h_3$ in this case?

Authors' Response: We appreciate your effort to understand our method in detail. However, $h_m$ is not tailored to a specific client during the inference period, and it is uploaded by clients with the training data used to finetune plug-ins. In the case mentioned by the reviewer, $h1, h2, h3$ will respectively be the feature of training data in client 1,2,3 and have nothing to do with test samples. Suppose $\xi_m \sim \mathcal{D}_m$ is local training data sampled from the local distribution of client m, $h_m=g(\xi_m)$ where $g$ is the global shared feature extractor. In other words, $h_m$ is the feature of client m's local training data, obtained with a single forward pass through the global feature extractor. We will be grateful if you can point out the presentation in our paper that mislead you to the misunderstanding of $h_m$.

Dear reviewer 3cLo,

We appreciate your time for reviewing and helpful comments, according to which we devote maximum effort to addressing the concerns and enriching the discussions in our paper accordingly. We summarize our response as follows:

• (1.1) Scalability problem: To address the potential problems when HPFL meets with an FL system with a great number of clients, we propose a simple but effective scheme to choose plug-ins with the noised test features, and test its performance to the availability of this scheme enhancing scalability.
• (1.2) Online test-time adaption: We claim that original purpose of HPFL is inferring in batches; however, HPFL also provides a solution in online inference settings with several modifications in the process of HPFL with inference on the server, without the communication and latency issues.
• (1.3) Clarification on details of $h_m$: In response to the reviewer's confusion about the details of training data features $h_m$, we explain the definition of $h_m$ and how to get $h_m$ in detail.
• (2) Comparison with FedRoD: According to the reviewer's questions about the discussion with FedRoD, we highlight the difference between HPFL and FedRoD as limited adaptation to test data of FedRoD's local classifiers , which lead to poor performance of FedRoD in GFL-PM. In response to the reviewer's suggestion, we also analyze the difference between our proposed GFL-PM setting and the setting mentioned by the reviewer.

Hopefully, your questions have been well addressed with our responses. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

Best regards

Authors of #2503

Thanks for the authors for providing the responses. I still have concerns on the scalability issue and the practicability of the algorithm:

1. Regarding the solution toward scalability, do you mean that clients should send the features of all test samples to the server? This will cause significant communication burden at the clients. Note that other baselines do not require any communication during inference. The privacy guarantee is also not sure even when the noise is added. A more detailed discussion is needed in terms of differential privacy if the authors choose this way of inference.

2. Regarding the on-line inference, if the authors are making inference per batch, are the accuracies getting averaged over multiple batches when reporting the results? I also have a concern regarding this batch-wise inference: Again, suppose there are three clients, each having classes (1, 2), (3, 4), (5, 6) in its local dataset, and produce plug-in modules 1, 2, 3, respectively. Now suppose that a test batch of a specific client has classes 1, 3, 5. Which modules should the client download? All modules will cause some problems in this case. How can the current solution handle this issue when making inference in batch-wise?

Thanks for the clarification. I'm still not convinced especially with the second response. The authors' main focus is on data distributional difference between training and testing. It is very natural to have a new data distribution during testing that is different from the clients' training data distribution in the system. In cross-silo setting, this will be a huge problem. Even in cross-device setting, I don't think simply saying that there will be no issues due to a large number of clients is not enough. There could still be new clients when the task is complicated. I will keep my original score.

Q4: experimental results of PerFedMask

The performance of PerFedMask (Table 1 of origin main text) in the personalized setting is much lower than other baselines. I wonder if there was an error in implementing this method, or if its hyper-parameters were not appropriately fine-tuned?

Authors' Response: Our results of PerFedMask is presented in Table 2 of origin main text instead of Table 1 of origin main text. We followed the open-source implementation of PerFedMask [6]. To fairly compare it with other baselines and HPFL, we only change the data partition method, the learning rate scheduler, and the communication round to align with other methods. To testify the scalability of methods, we increase the number of clients, and raise batch size from 50 to 128 as the value we use in other methods for better convergence. All other hyperparameters, including the initial learning rate, stay the same as the origin implementation of PerFedMask. We have added these information about hyperparameters to Appendix C.4 in our revision.

Q5: experiments with different levels of noise

About the experiments with added noise (Table 4 of origin main text). Why do experiments with different levels of noise produce nearly the same results? This appears somewhat counterintuitive. Can the authors provide a more detailed analysis?

Authors' Response: For this counterintuitive phenomenon, we provide our explaination in Section 5.3 of origin main text. Since our selection method only chooses plug-ins with selection scores, if the changes in selection scores are not significant enough to affect the final choices of plug-ins, which can be observed in the MMD scores heatmaps in Figure 3 of origin main text, the performance of HPFL will remain the same, which adds robustness towards noise to HPFL without significant loss on accuracy performance. We have added a detailed discussion about this phenomenon in Section 5.3 of our revision.

Reference

[1] Marfoq, Othmane, et al. "Personalized federated learning through local memorization." In ICML, 2022.

[2] Oh, Jaehoon, SangMook Kim, and Se-Young Yun. "FedBABU: Toward Enhanced Representation for Federated Image Classification." In ICLR, 2021.

[3] Collins, Liam, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. "Exploiting shared representations for personalized federated learning." In ICML, 2021.

[4] Arivazhagan, Manoj Ghuhan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. "Federated learning with personalization layers." arXiv, 2019.

[5] Liang, Paul Pu, Terrance Liu, Liu Ziyin, Nicholas B. Allen, Randy P. Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. "Think locally, act globally: Federated learning with local and global representations." arXiv, 2020.

[6] Setayesh, Mehdi, Xiaoxiao Li, and Vincent WS Wong. "PerFedMask: Personalized Federated Learning with Optimized Masking Vectors." In ICLR, 2022.

[7] Qu, Zhe, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. "Generalized federated learning via sharpness aware minimization." In ICML, 2022.

[8] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In ICLR, 2020.

[9] Luo, Mi, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. "No fear of heterogeneity: Classifier calibration for federated learning with non-iid data." In NeruIPS, 2021.

[10] Ren, Jie, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. "Likelihood ratios for out-of-distribution detection." In NeruIPS, 2019.

We would like to express our gratitude for the time the reviewer spent reviewing. We appreciate that you find our method easy to follow, the application of our method interesting, and the finding in the experiments insightful. According to your valuable comments, we provide detailed feedback.

Q1: comparison with KNN-Per FedBABU

The combination of using a separate backbone and classifier in PFL for improvement, along with the idea of customize the classifier on local client, is not novel. The former has been widely validated for its effectiveness in FedPer and FedRep, while the latter's idea is highly relevant to that of KNN-Per (a) [1] and FedBABU (b) [2]. However, the authors do not provide relevant discussion or experimental comparisons of the latter ones.

Authors' Response: Thanks for providing us with two excellent methods in the PFL setting for comparing our methods with. KNN-Per (a) [1] and FedBABU (b) [2] share the similar problem of classic PFL algorithms [3-6] in our GFL-PM setting.

• KNN-Per (a) build a local datastore $S_m$ with local training data and query $S_m$ to assist inference using a knn method. Imagine a circumstance where some classes in local test data are not presented in local training data, in this situation, the test samples of those classes are likely to retrieve a wrong label from $S_m$ and mislead the final decision.
• In FedBABU (b), classifiers of local clients are randomly initialized and never updated during federated training, and for personalization during the evaluation process, FedBABU fine-tuned its classifiers on local training data. Here, FedBABU also requires local test data to be in the same distribution as local training data to fine-tune the classifiers on local training data to perform well on local test data.

These two methods only customize the classifier to fit static local test data with the same distribution as its training data. Instead, our proposed setting GFL-PM is faced with the problem of changing local test data distributions, whose distribution is different from local training data, and HPFL is proposed to solve this problem. These two methods don't propose the adaption mechanism to test data, which is implemented with plug-in selection in HPFL, thus probably face with similar difficulty of classic PFL algorithms like FedPer and FedRep when being exploited in GFL-PM setting.

Q2: GFL baseline

Since the authors compared the performance of both GFL and PFL, the experiments should include more commonly used GFL-related baselines.

Authors' Response: We'd like to thank you for pointing out the potential deficiency in GFL baselines. For comparison in the GFL setting in our experiments, FedAvg is a classic and most commonly used GFL algorithm, and FedRod can simultaneously achieve leading generic and personalized performance. For more comprehensive comparison in the GFL setting, we prepare to compare HPFL with a more recent SOTA GFL method FedSAM [7]. However, due to the time needed to carry out the experiments, we may need several days to report the result of FedSAM and add the result to our revision.

Q3: privacy risk of sharing the noised features

There is a missing detailed analysis of potential privacy risks in uploading and sharing local features.

Authors' Response: Due to the page limit, in Appendix E.2, we discussed potential privacy risks and conducted experiments of model inversion attack [8] to testify the privacy safety of sharing noisy features obtained with our method. As the experiments show, after adding noise in a way as stated in Section 4.3 of origin main text, even an attacker with the global model can hardly recover the original image, while with raw feature and the model, an attacker can easily recover the raw image, which shows the effectiveness of our privacy protection method. We also discuss other possible methods to protect privacy, like averaging the raw features [9] or choosing plug-ins with methods without the requirements of sharing direct information about raw data or features like OOD detection [10]. To increase the visibility of our discussions about privacy in Appendix E.2, we have directed the reader to them in Section 4.3 of main text in our revision.

Dear reviewer TW8h,

Thanks for reviewing and constructive comments, from which we put forth our best effort to answer the questions and enrich the content of our paper accordingly. A summary of our responses for your convenience is as follows:

• (1) comparison with KNN-Per & FedBABU: To answer your question on the difference of HPFL compared with KNN-Per and FedBABU, we review the main steps of both KNN-Per and FedBABU, and analyze their limitation when met with GFL-PM. Then we conclude that the lack of the adaption mechanism to test data put obstacles in applying KNN-Per and FedBABU in GFL-PM setting.
• (2) GFL baseline: In response to the reviewer's advice to add more GFL baselines, we point out the existing GFL baselines FedAvg and FedRoD, and compare HPFL with a more recent SOTA GFL method FedSAM.
• (3) Privacy risk of sharing the noised features: In response to the reviewer's concerns about sharing the noised features, we directed the reviewer to Appendix E.2, where we discussed potential privacy risks and privacy protection methods, and conducted experiments of model inversion attack using the shared features and the global backbone in HPFL.
• (4) Experimental results of PerFedMask: We explain the missing details of PerFedMask in our responses and add these details in Appendix C.4 in our revision.
• (5) Experiments with different levels of noise: We direct the reviewer to the explanation of the counterintuitive phenomenon that experiments with different levels of noise produce nearly the same results in Section 5.3 of origin main text, and further discuss the possible cause of this phenomenon in our responses.

Hopefully, your questions have been well addressed with our responses. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

Best regards

Authors of #2503

We show the results of the new GFL baseline FedSAM in Table 3 below, and compare FedSAM and HPFL, as suggested in the reviewer's comments. Unless otherwise stated, the setting remains the same as the main experiments in Table 2 of origin main text, exclusive hyper-parameters for FedSAM is specified in Appendix C.4:

Table 3

The bold numbers mean they are the best GFL results compared with other methods in Table 3. From the results, we observe that though FedSAM achieves excellent GFL-GM performance in some settings, HPFL outperforms FedSAM in all GFL settings in the experiment.

Q2: experimental result of FedRod

Considering FedRoD is the sota in this topic evidenced by Table 2 of origin main text, can the authors provide a more fine-grained comparison? (a) Specifically, FedRod (also followed by FedTHE) proposes to train the general model with a class-balanced loss; will this improve HPFL as well? FedRoD also fine-tunes the whole general model for each client with local training; (b) how many personalized epochs are trained like the reported HPFL (e.g., E_p=1 or 10)?

Authors' Response: Yes, and thank the reviewer for the suggestions about a more elaborated discussion on FedRoD. (a) From the results of HPFL(FedRoD) given in Response to Q1, we observe an interesting fact that even if FedRoD shows excellent performance in GFL-GM (surpasses FedAvg in many settings), fine-tuning the backbone trained with it is not advantageous as shown in PFL-PM (only comparable with fine-tuning on the backbone trained with FedAvg). From this phenomenon, we presume the advantage of FedRoD in GFL-GM should mainly be attributed to its global head trained with a class-balanced loss instead of its backbone. (b) As is stated in Section 5.1 of origin main text, the local training epoch of all methods is $1$. To avoid the possible misunderstanding, we would like to clarify that the local training epoch is different from the $E_p$ in HPFL, because the backbone will not be updated when fine-tuning the plug-ins in HPFL. Fine-tuning of HPFL happens after the GFL backbone training and is completely done in local clients. We apologize for not clearly indicating personalized epochs of other baseline methods with personalized epochs like PerFedMask. We have specified the value of personalized epochs for those methods with personalized epochs in Table 2 of main text in our revision.

Q3: presentation

There are many typos and inconsistent wording

Authors' Response: We apologize for the typos and inconsistent wording. We will check the whole paper thoroughly and carefully, and revise mistakes in it.

Q4: How HPFL tackles very different distributions unseen in the training

(a) Can the authors provide a discussion on the data/client assumptions on when should HPFL work or not work? (b) Suppose the new client has very different distributions unseen in the training, how possible will HPFL work well?

Authors' Response: Sure. (a) HPFL works well in situations where the distribution of local training data and test data is quite different, but its test data distribution is similar to the distribution of other clients' training data, HPFL can help the client by borrowing knowledge from other clients, i.e., utilize plug-ins trained by other clients. (b) For the clients that meet with totally different test data distributions that haven't been seen in the training data of all clients (include those clients themselves; if those clients have similar distribution on training data and test data, they can choose the plug-in trained on their own local data). However, if there is really no similar data pattern even in the whole FL system, HPFL can only attempt to choose the most appropriate plug-in, though in this situation, even if the most appropriate plug-in may not perform well on the totally strange test data due to not identical training and test distribution [8]. But it is worth noting that with the increasing number of clients in the FL system, this situation happens significantly less with the abundance of clients' training data distribution.

Reference

[1] Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017.

[2] Li, Tian, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. "Federated optimization in heterogeneous networks." In MLSys, 2020.

[3] Wang, Jianyu, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. "Tackling the objective inconsistency problem in heterogeneous federated optimization." In NeurIPS, 2020.

[4] Karimireddy, Sai Praneeth, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. "Scaffold: Stochastic controlled averaging for federated learning." In ICML, 2020.

[5] Durmus, Alp Emre, Zhao Yue, Matas Ramon, Mattina Matthew, Whatmough Paul, and Saligrama Venkatesh. "Federated Learning Based on Dynamic Regularization." In ICLR, 2021.

[6] Chen, Hong-You, and Wei-Lun Chao. "On Bridging Generic and Personalized Federated Learning for Image Classification." In ICLR, 2021.

[7] Ren, Jie, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. "Likelihood ratios for out-of-distribution detection." In NeruIPS, 2019.

[8] Quinonero-Candela, Joaquin, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence, eds. Dataset shift in machine learning. Mit Press, 2008.

We genuinely appreciate the reviewer's efforts. We are delighted to know that you find our proposed framework sound, the extension of our method interesting, and the discussions and solutions on the practical drawbacks make sense. According to your valuable comments, we provide detailed feedback.

Q1: Ablation study on backbone training and selection methods

Some parts of the design choices of the proposed selective FL could be elaborated more to give the paper a broader scope. For example, the part of the general feature extractors can be trained with any GFL algorithm. (a) Considering there are many GFL methods proposed, what are their performances in comparison? Are they compatible with selective FL? (b) For the selection methods in section 4.3, could the author provide more insights and discussion besides the used distance-based methods? Any other potential replacements?

Authors' Response: Thanks for the advice on improving the ablation study of our method. (a) As long as the used GFL methods are able to train a strong general feature extractor, HPFL is able to utilize the feature extractor to train the personalized plug-ins and extract features. Due to the great availability of GFL methods used for training backbone [1-5], we didn't carry out experiments to see how different GFL backbone training methods affect the overall performance of HPFL. We add experiments using FedRoD [6] to testify HPFL's compatibility with other GFL methods. The results are given in Table 2 below. Number of clients equal $10$, local fine-tuning epoch $E_p=10$, local datasets are partitioned in $Dir(0.1)$. Other settings remain the same as the main experiments in Table 2 of origin main text:

Table 2

From the overall performance of HPFL(FedRoD), we can see that HPFL using FedRoD as its backbone training method is comparable to that using FedAvg, which confirms HPFL is compatible with the GFL methods other than FedAvg. Following your valuable comments, in Section 5.2 of main text of our revision, we have directed readers to results of HPFL(FedRoD) shown in Appendix D.6 in our revision to discuss the ablation study of GFL backbone training methods. (b) As for selection methods other than distance-based methods, due to the page limit, we choose to introduce other selection methods in the main body. In the Appendix D.2, we elaborately discuss an OOD-Detection (Out of Distribution Detection)-based method [7] to select suitable plug-ins by judging whether the test data is OOD for plug-ins and choose the plug-in with the lowest OOD score. Aside from OOD-Detection-based methods, if a method can be used to judge how appropriate plug-ins are for certain test data, it is available for selection. Even a supervised classifier trained with the noised training features as data and corresponding client indexes as labels may be able to select plug-ins.

Dear reviewer ENEM,

Thanks for reviewing and constructive comments, from which we put forth our best effort to answer the questions and enrich the content of our paper accordingly. A summary of our responses for your convenience is followed:

• (1) Ablation study on backbone training and selection methods: According to the appreciated suggestion of the reviewer, to testify the compatiablity of HPFL, we discuss the possibility of HPFL backbone training with FedRoD and provide the result of HPFL(FedRoD). As for selection methods, we direct the reviewer to the Appendix D.2 of main text for other selection methods and introduce a simple supervised selection method.
• (2) Experimental result of FedRod: We direct the reviewer to the discussion about HPFL using FedRoD for backbone training and avoid possible misunderstanding by clarifying training details of FedRoD.
• (3) Presentation: In response to the reviewer's concerns on paper presentation, we have checked the whole paper thoroughly and carefully, and revise its mistakes in our revision.
• (4) HPFL's intuition dealing unseen distributions: To answer the possible confusion of the reviewer on HPFL's ability to unseen distributions in local training data, we dicuss HPFL under different circumstances: (i) HPFL when test data distribution is similar to the distribution of other clients’ training data; (ii) and HPFL with totally different test data distributions that haven’t been seen in the training data of all clients (include those clients themselves).

We hope your questions have been well addressed with our responses. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

Best regards

Authors of #2503

Dear Reviewer ENEM,

Thanks very much for your time and valuable comments. We understand you’re busy. But as the window for responsing and paper revision is closing, would you mind checking our response (a brief summary, and details) and confirm whether you have any further questions? We are very glad to provide answers and revision to your further questions.

Best regards and thanks,

Authors of #2503

Dear reviewer ENEM,

Thanks for your great efforts in reviewing our paper and your support for our work. We appreciate that you find our proposed problem interesting; with discussions and solutions on the practical drawbacks of naive solution for selective FL; interesting and appreciated applications; and proper discussion of limitations, impact, and related works.

Your valuable comments have greatly helped us improve the presentation of our paper, the integrity of the ablation study, and the clarity of HPFL’s intuition. It is greatly appreciated that the reviewer provide insightful comments about the backbone trained with different GFL methods.

If you have no further questions/concerns, would you mind raising the score?

Best regards and thanks,

Authors of #2503

We are grateful for the time the reviewer spent on reviewing. It is pleasant to learn that you find our proposed setting novel, our solution intuitive, the performance gains significant, and the evaluation method appropriate. According to your valuable comments, we provide detailed feedback. Due to similar concerns of different reviewers on the scalability and privacy safety of our method, which are all solvable by uploading the noise test features to the server for selection plug-ins, we have conducted experiments to testify the feasibility of this slight modification against communication and storage burden of HPFL in FL systems with plenty of clients. The results are in the Solution toward shared scalability and privacy concern. Please find special responses to your comments below.

Q1: presentation of paper

The main weakness of the paper is its presentation. In particular, the manuscript is not explicit about the form of the test distribution that the clients will encounter. I was confused about this, and could only infer it until I reached Section 3.3 of origin main text and I finally understood the problem's objective.

Authors' Response: We are sorry that we failed to highlight the test scenarios of our problem, which is briefly illustrated in the 3rd paragraph in the introduction of origin main text (Section 1): "i.e., clients equally meet with the test data from all clients." We chose to elaborate more in our theory part for this problem setting. We have added a detailed explanation, "In GFL-PM setting, the test set every client encounters comes from the local test data of other clients, whose distribution is the same as that of clients' local training data." in the 3rd paragraph in the introduction of our revision to help readers to more easily understand our proposed problem.

Q2: scalability problem

Similarly, it is not clear to me whether the method is proposed for cross-device or cross-silo setting. Although the motivating example in Section 1 of origin main text is cross-device, the experiments are performed with a limited number of clients (<= 100). More concerning, in the proposed algorithm, the server stores all the training clients' plug-ins. In cross-device scenarios with a massive number of clients, this would not scale. There is no discussion of this in the paper.

Authors' Response: Thanks for pointing out the potential scalability problem of HPFL. In fact, HPFL can be applied to both cross-device and cross-silo setting, with a slight modification in cross-device setting where the number of clients is overwhelmingly large. We introduce the methods to enhance the scalability problem of HPFL as follows:

• To handle the massive plug-ins needed to be stored in the server, the server can cluster the plug-ins with client-cluster methods in a similar way as done in IFCA [1], CFL [2], FL+HC [3], and so on. Then the server aggregates the plug-ins in the same clusters to keep a controllable number of plug-ins, like in $O(1)$ or $O(log(number \ of \ clients))$. The server can significantly reduce the number of plug-ins in this way, thus increase the scalability of our method.
• A simpler method is enough for solving the issue of the massive plug-ins: as our selection method doesn't require the presence of plug-ins, clients may not upload the plug-ins after training. Instead, the server can request the appropriate plug-in from the corresponding client after calculating the selection scores. Considering the common issue in cross-client setting, FL systems may encounter client dropout [4, 5, 6]. In a situation where the client with the most appropriate plug-in is out of connection, the server may attempt to request plug-ins one by one with the selection score.
• To avoid downloading all plug-ins and training features to clients, if the number of clients grows to a large number, clients can choose to add noise to their local test features and send the noised test features to the server to select the plug-ins. In this way, each client can get the exact plug-in they need without the need to download all the plug-ins and the noised training features, which will cause a great communication cost with a great number of clients in the FL. We conduct experiments to test the feasibility of this method against the communication and storage burden of HPFL in FL systems with plenty of clients, please find them in the Solution toward shared scalability and privacy concern.

We have added the discussions of those scalability problems of HPFL in the Appendix F.2.1 of our revision.

Q3: privacy concern of sharing plug-ins

Storing all the clients' plug-ins may also be a privacy risk, as there is not any aggregation that protects the clients' privacy from a malicious server. The threat model is not discussed by the paper.

Authors' Response: Due to page limit, we discussed the potential privacy risk of sharing plug-ins in Appendix E.1.

• As stated in Section 4.3 of origin main text, sharing part of models (i.e. plug-ins) is exposed to actually no more privacy risk than sharing the parameter gradients or the complete global model as adopted in classic FL algorithms like FedAvg [7], which in practice is often protected with regular privacy protect techniques like differential privacy (DP) [8].
• As clients in HPFL don't upload the gradient during plug-ins training, no gradient of models is accessible for the server, so there is no need to worry about gradient inversion attacks [9-12] towards plug-ins.
• We also carry out experiments to testify the safety of our method toward the model inversion attack [13,14,15] when with both the noise features and the model trained on raw features in Appendix E.2.
• Even extreme concern on potential privacy risk of keeping plug-ins in the server can be solved by only requesting plug-ins after selection as described in Response to Q2, clients providing the plug-ins can ask the server to delete the plug-ins after sending the plug-ins to the clients in need. To increase the visibility of our discussions about the privacy problem of sharing plug-ins in Appendix E.1, we have directed the reader to them in Section 4.2 of main text in our revision.

Q4: the importance of proposed set-up

Although the given set-up is novel, I am still not convinced of its importance, at least not justify the drawbacks of the proposed solution. I recommend the manuscript update its introduction to include more motivating examples that are explicitly related to their problem formulation.

Authors' Response: We appreciate the suggestion given by the reviewer to justify the motivation of our proposed set-up. We give some examples exhibiting the significance of our proposed set-up below:

• Case 1: Some clients may have insufficient computing resources or local training data to fine-tune a deep learning model in a cross-device setting. In these situations, training distribution can be regarded as an empty set $\emptyset$. In this way, the client cannot get a personalized model by locally fine-tuning the global model. In traditional GFL and PFL settings, the client has no choice but to adopt the global model and endure the lack of personalization. This problem is caused by the mismatch of training data distribution and test data distribution, as assumed in our proposed set-up, and is solvable with our proposed method HPFL by exploiting personalized plug-ins from other clients.
• Case 2: A car with a personalized automated driving system (ADS) has driven out of the previous city it used to be. It requires to personalize on geometric data from the present city it is now in for improving the performance of the ADS in this new city. Classic GFL and PFL in this situation leave the ADS no option but to collect the geometric data and personalize on it after the collection completes, and accept the temporary performance loss using the previous personalized model before finishing the new personalization, since the distribution of test data has greatly changed. It's another example where the discrepancy between training data (geometric data from the previous city) and test data (geometric data from the present city) threatens the availability of FL systems. While with our proposed method designed to solve the problem, the ADS can attempt to access the plug-ins from car owners living in the present city.
• Case 3: Imagine a person is traveling from a high latitude area to an equatorial region, and the recommender system on their phone is supported by federated learning. If the recommender system uses the personalized model trained when in the high latitude area, it will continue to prompt thick down jackets for the person, which is clearly an unexpected and unreasonable recommendation. With our method, one can get the same recommendation as the local people with plug-ins on their phones without time to fine-tune the model again.

Following your suggestion, we have added the examples listed above to the Appendix F.1 of our revision due to the page limit.

Q5: evaluation framework

Although the presented evaluation framework is good for the presented problem, a more exhaustive framework could be formulated to rigorously test the approach, e.g., one in which train client $m$ is given test data $n$, and we check if it indeed downloads plug-in $n$.

Authors' Response: For our proposed methods HPFL, because all clients have the same selection method $C$. So the selection method is only determined by the test data $n$ and not affected by which client carries out the inference as stated in Equation 15 in origin main text $C_i(\xi_m,i)=argmax_n \space g\left (\xi_n, \xi_m\right)=C(\xi_m)$ in Appendix C.3. In this situation, the evaluation method the reviewer have suggested is formularized as following equation: $\text{Accuracy} \left(\Omega, \theta _1, \ldots, \theta _M\right)=\frac{1}{M} \sum _{i=1}^M \sum _{m=1}^M p _m \mathbb{E} _{\xi _m \sim \mathcal{D} _m} \mathcal{T}\left(f\left(\theta _{C _i(\xi _m,i)}, \xi _m\right), \xi _m\right)$ is the plug-in n downloaded by client m; and the simplified evaluation method used in our implementation is accuracy of PM selected by client i on its own data: $\text{Accuracy} _{simplified}\left(\Omega, \theta _1, \ldots, \theta _M\right)=\sum _{i=1}^M p _i \mathbb{E} _{\xi _i \sim \mathcal{D} _i}\mathcal{T}(f(\theta _{C(\xi _i)} ^{pfl}, \xi _j), \xi _j)$. We have strictly proven the equivalence of our simplified evaluation framework and the origin evaluation metrics in Appendix C.3.

Reference

[1] Ghosh, Avishek, Jichan Chung, Dong Yin, and Kannan Ramchandran. "An efficient framework for clustered federated learning." In NeurIPS, 2020.

[2] Sattler, Felix, Klaus-Robert Müller, and Wojciech Samek. "Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints." IEEE transactions on neural networks and learning systems, 2020.

[3] Briggs, Christopher, Zhong Fan, and Peter Andras. "Federated learning with hierarchical clustering of local updates to improve training on non-IID data." In IJCNN, 2020.

[4] Li, Tian, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. "Federated learning: Challenges, methods, and future directions." IEEE signal processing magazine, 2020.

[5] Kairouz, Peter, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz et al. "Advances and open problems in federated learning." Foundations and Trends® in Machine Learning 14, 2021.

[6] Tan, Alysa Ziying, Han Yu, Lizhen Cui, and Qiang Yang. "Towards personalized federated learning." IEEE Transactions on Neural Networks and Learning Systems, 2022.

[7] Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017.

[8] Wei, Kang, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H. Vincent Poor. "Federated learning with differential privacy: Algorithms and performance analysis." IEEE Transactions on Information Forensics and Security, 2020.

[9] Ligeng Zhu, Zhijian Liu, and Song Han. "Deep leakage from gradients." In NeurIPS, 2019.

[10] Zhao, Bo, Konda Reddy Mopuri, and Hakan Bilen. "idlg: Improved deep leakage from gradients." arXiv, 2020.

[11] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients–how easy is it to break privacy in federated learning? In NeurIPS, 2020.

[12] Yin, Hongxu, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, and Pavlo Molchanov. "See through gradients: Image batch recovery via gradinversion." In CVPR, 2021.

[13] Hitaj, Briland, Giuseppe Ateniese, and Fernando Perez-Cruz. "Deep models under the GAN: information leakage from collaborative deep learning." In CCS, 2017.

[14] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In ICLR, 2020.

[15] Lyu, Lingjuan, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and S. Yu Philip. "Privacy and robustness in federated learning: Attacks and defenses." IEEE transactions on neural networks and learning systems, 2022.

Dear reviewer nQmW,

Thanks for sparing your valuable time to review and offer insightful comments, from which we have exerted maximum effort to answer the questions and thoroughly improve our paper. We summarize our responses for your convenience:

• (1) Presentation of paper: According to the appreciated suggestion of the reviewer, we provide a more detailed explanation for the form of the test distribution and revise our paper to improve the readability.
• (2) Scalability issues: We show that slight modifications on implementation boost the scalability of HPFL, which the reviewer kindly reminded us to discuss. We demonstrate the schemes against the scalability issues from several perspectives: (i) a cluster-based method against massive plug-ins stored in the server; (ii) a quicker and easier scheme against the aforementioned issue; (iii) massive plug-ins and training features downloaded to clients.
• (3) Privacy issues: In response to the reviewer's concerns on sharing plug-ins, we analyze the privacy issue of HPFL and find there are no more privacy risks than GFL methods like FedAvg. We also direct the reviewer for supportive experiment results on model inversion attack against HPFL. We additionally discuss an approach to accommodate extreme privacy concerns of sharing plug-ins.
• (4) Importance of proposed set-up: We list several real-world examples to convince the commonness and importance of the proposed set-up GFL-PM, and show the difficulty classic GFL and PFL algorithm met in those situations.
• (5) evaluation framework: To better illustrate the equivalence of our implementation and the evaluation method the reviewer suggests in HPFL, we list the formulas of both the evaluation method the reviewer suggests and our implementation and show that they are the same when all clients use the same selection method as in HPFL.

Hopefully, our responses have addressed your concerns well. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

Best regards

Authors of #2503

Dear Reviewer nQmW,

Thanks a lot for your efforts in reviewing paper and your constructive comments. We understand you’re busy. Thanks very much for your valuable comments. We sincerely understand you’re busy. But as the window for responsing and paper revision is closing, would you mind checking our response (a brief summary, and details) and confirm whether you have any further questions? We look forward to answering more questions from you.

Best regards and thanks,

Authors of #2503

Reference

[1] Arivazhagan, Manoj Ghuhan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. "Federated learning with personalization layers." arXiv, 2019.

[2] Liang, Paul Pu, Terrance Liu, Liu Ziyin, Nicholas B. Allen, Randy P. Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. "Think locally, act globally: Federated learning with local and global representations." arXiv, 2020.

[3] Collins, Liam, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. "Exploiting shared representations for personalized federated learning." In ICML, 2021.

[4] Chen, Hong-You, and Wei-Lun Chao. "On Bridging Generic and Personalized Federated Learning for Image Classification." In ICLR, 2021.

[5] Jiang, Liangze, and Tao Lin. "Test-Time Robust Personalization for Federated Learning." In ICLR, 2022.

[6] Setayesh, Mehdi, Xiaoxiao Li, and Vincent WS Wong. "PerFedMask: Personalized Federated Learning with Optimized Masking Vectors." In ICLR, 2022.

[7] Li, Xiaoxiao, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. "FedBN: Federated Learning on Non-IID Features via Local Batch Normalization." In ICLR, 2020.

We sincerely thank the reviewer for taking the time to review. We appreciate that you find our proposed problem novel, our paper well-written, and the experiments comprehensive. Based on your valuable comments, we provide detailed feedbacks as follows.

Question: novelty

The paper has low novelty. Federated learning and fine-tuning the sub-modules are really common and baseline personalized FL methods.

We not only propose a novel problem setting but also introduce a novel mechanism to solve this problem. We propose a novel problem named GFL-PM and reduce the problem into Selective FL (SFL), and finally solve the SFL problem with our proposed method HPFL, which introduces a novel plug-in mechanism. Our difference with other previous personalized FL methods, which carry out GFL and fine-tuning the sub-modules, is listed below:

From the table above, we can clearly see most PFL algorithms lack the ability to adapt to test data. FedTHE utilizes an interpolation of global prediction and personalized prediction, whose coefficient is trained on test data. While our proposed method HPFL obtains similar ability with selection method and plug-ins from other clients, which enables the exploitation of personalized knowledge from other clients simultaneously. Therefore, the novelty of HPFL does not lie in how the backbone is trained or how the plug-ins are personalized, which is the key point of most previous methods, but in its overall framework and plug-ins selection mechanism.

Dear reviewer MHGx,

Thanks for your time in reviewing our paper. According to your comments, we have tried our best to convince the reviewer of the novelty of our paper. Here is a summary of our response for your convenience:

• Novelty issues: We clarify the novelty of our proposed problem setting GFL-PM; reduction of GFL-PM into Selective FL; and solution to the SFL problem HPFL, with a novel plug-in mechanism. To further explain the novelty of our proposed method HPFL, We provide a table comparing different methods with our proposed method HPFL, demonstrating that the ability to adapt to test data and share personalized knowledge in an FL system are the key differences between HPFL and those classic PFL algorithms.

We humbly hope our response has addressed your concerns about the novelty of our paper. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

Best regards

Authors of #2503

Dear Reviewer MHGx,

Thanks a lot for your time in reviewing, and we appreciated your precious suggestions for improving our work. We sincerely understand you’re busy. But as the window for responsing and paper revision is closing, would you mind checking our response (a brief summary, and details) and confirm whether you have any further questions? We look forward to answering more questions from you.

Best regards and thanks,

Authors of #2503

Dear reviewer MHGx,

Thanks for your great efforts in reviewing our paper. We appreciate that you find our proposed problem novel and interesting, our paper is well-written with comprehensive experimental results.

Your constructive comments have greatly helped us improve the statement of our novelty. Especially, the suggestion of providing a more detailed comparation of personalized FL methods with federated learning and fine-tuning the sub-modules is critical.

If you have no further questions/concerns, would you mind raising the score?

Best regards and thanks,

Authors of #2503