Federated Learning Empowered by Generative Content

Thank you for your time and comments.

General Response: We think that the main reason of the reviewer's rejection is the seemingly simplicity of our method. However, we would like to justify for ourselves.

1. We effectively mitigate the issue of one of the most critical issuses in FL: data heterogeneity. Our method is simple yet effective as verified by the consistent and significant performance improvement, which should not be overlooked.

2. Simplicity does not equal no novelty. Our proposed FedGC framework for the first time training private heterogeneous data mixed with synthetic data generated by the powerful advanced generative models. Our work points out its potential and we believe that it could inspire more explorations in this topic to better address or even fully address the issue of data heterogeneity.

3. We intentionally to make our framework simple and general. As the first exploration on this topic, rather than designing seemingly complicated algorithm, we decided to implement a general framework to conduct a systematic empirical study to provide more fundamental insights. Besides, thanks to this simplicity, our method is naturally compatible with conventional FL communication protocols such as secure aggregation and differential privacy.

---

W2: The paper solely focuses on data generation for local training. However, there is nothing in the communication among the clients that carries any information about data generation from one client to another, other than the number of samples to be generated for each client. In other words, the use of federated learning and data generation appear to be unrelated. I wonder if the whole work could have been better presented in a non-federated setting.

Response: Focusing on data heterogeneity, we propose a new method that can effectively alleviate its negative effects. The generative data can significantlly mitigate the level of data heterogeneity and the issue of overfitting, promoting the performance of FL. Thus, it is not 'unrelated'.

Data heterogeneity is a representative and common issue in federated setting while in a non-federated setting there is no defination of data heterogeneity. Thus, the idea of using generative content to address data heterogeneity is quite unique and suited for FL but not non-FL setting. From the following table, we can see that local training (without FL) with generative content performs significantly worse.

[Table R1. Performance comparison between local training with generative content and our FedGC.]

To make the design 'more like' FL, there could be many methods following FedGC framework. For example, to mitigate the potential negative effects of dissimilarity between generative and real data, we can use soft-label supervision to train local model on generative data uwhile hard-label supervision for real data, where the soft labels come from the FL global model. We leave such methods to future works.

---

W1/W3: There is no theorem, no proposal. Not a single equation is found in the paper. I think this part of the paper is good from a practical point of view. However, it appear to be dominantly engineering contributions, which does not seem suitable for ICLR, a conference about learning representations.

Response:

First, federated learning itself is a practical setting and we propose a practical solution, which naturally fits the overall objective.

Second, this judgment varies from person to person. We kindly request the reviewer to refer to the following ICLR papers, where they might seem to be 'unrelated' and 'engineering' for the reviewer.

[1] Chen, Hong-You, et al. "On the importance and applicability of pre-training for federated learning." The Eleventh International Conference on Learning Representations. 2023.

[2] Nguyen, John, et al. "Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning." The Eleventh International Conference on Learning Representations. 2023.

[3] Oh, Jaehoon, SangMook Kim, and Se-Young Yun. "FedBABU: Toward Enhanced Representation for Federated Image Classification." International Conference on Learning Representations. 2022.

W4: The dataset for experimenting was made up by merging from a few existing known datasets which were not designed for federated learning?

Response:

Please note that most of the used datasets are quite common in federated learning literature and we follow exactly the same setting. Here are some examples.

• Sentiment140, which is from FL LEAF benchmark [1] (1052 citations).
• CIFAR10, which is commonly used in most of FL papers [2,3,4].
• HAM10000 [5,6,7], PACS[8,9,10].

[1] Caldas, Sebastian, et al. "Leaf: A benchmark for federated settings." arXiv preprint arXiv:1812.01097 (2018).

[2] Chen, Hong-You, et al. "On the importance and applicability of pre-training for federated learning." The Eleventh International Conference on Learning Representations. 2023.

[3] Nguyen, John, et al. "Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning." The Eleventh International Conference on Learning Representations. 2023.

[4] Oh, Jaehoon, SangMook Kim, and Se-Young Yun. "FedBABU: Toward Enhanced Representation for Federated Image Classification." International Conference on Learning Representations. 2022.

[5] Thapa, Chandra, et al. "Splitfed: When federated learning meets split learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 8. 2022.

[6] Yang, Fu-En, Chien-Yi Wang, and Yu-Chiang Frank Wang. "Efficient model personalization in federated learning via client-specific prompt generation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[7] Pennisi, Matteo, et al. "Experience Replay as an Effective Strategy for Optimizing Decentralized Federated Learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[8] Zhang, Ruipeng, et al. "Federated domain generalization with generalization adjustment." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[9] Nguyen, A. Tuan, Philip Torr, and Ser Nam Lim. "Fedsr: A simple and effective domain generalization method for federated learning." Advances in Neural Information Processing Systems 35 (2022): 38831-38843.

[10] Chen, Haokun, et al. "Fraug: Tackling federated learning with non-iid features via representation augmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

---

Q: Doesn't that mean a related generative model has to exist for a given problem? What if no generative model exists for a given problem?

Response:

Currently, there has been many generative models for many domains, which is why we think it is interesting and timely to explore this topic.

Besides, the generative model does not neccessarily need to fully resemble real data. Here are our evidence.

1. The experiments on medical dataset HAM10000 in Table 4. Since stable diffusion can not generate medical with good quality, we choose this experiment to alleviate the concern of contamination. Please note that the generated images are completely different from the real iamges and we decided not to posting them to avoid causing discomfort. Here, we put the results below for convinience. Despite the dissimilarity, our method still brings significant gain.

[Table R2. Results on medical dataset.]

2. The experiments of training on generated data only in Table 6. Here, we put the results below for convinience. From the table, we see that merely using generative data to train a model fail to perform well, indicating that there is a huge gap between generative data and real data. Generated data can only exhibit its effectiveness when used in conjunction with real data in our proposed FedGC.

[Table R3. Comparisons with training on genrative data only. Training on generative data only achieves low performance.]

3. We compare real data and generative data in Figure 8 via visualization, where we see that there is a large amount of generative data with incorrect concept and can not fully represent real data.

---

Overall, we thank the reviewer for the time for reviewing. We hope that our responses can fully address your concerns. We also kindly request the reviewer for reconsideration and look forward to your feedback to improve our work.

W1: I am glad that you acknowledge that federated learning itself is a practical setting. While you have shown that in your datasets, the experimental results when additional generated content was used increase the accuracy, the problem I see is that it only says that for these datasets FedGC works. Without some sort of theoretical statements about the generalisation of FedGC to other datasets, it is hard to understand how well FedGC would work in a new setting.

W3: You are right. I have overlooked. I will increase my rating.

I thank the authors for their comments. They largely address my concerns. Unfortunately due to time constraints I will not be able to discuss all points at once. I will try to tackle one by one.

Regarding your response to W2, maybe I was not clear in explaining my concern. The problem of data heterogeneity is very unique to FL. This is a major problem. I am aware of that. But it is not what I was concerned. I was concerned with the proposed solution where the data generation process is almost independent from one client to another. I was hoping that FedGC would do something smarter than existing FL approaches such that it can somehow pass some knowledge related to its local data distribution or the knowledge of how the local generated data were generated to other clients in the hope that they can use that knowledge to learn their own classification problem better. But it would not. Hence, it appears the federated setting here is unnecessary to the idea that the paper is promoting, which is to use generative models to generated data, other than to bring the data heterogeneity problem into the context.

In the new Table R1 you provided, do you have a row representing the results of local training without using generated content? How do they compare to Local+GC results? I suspect Local+GC should outperform Local (without GC). Going from Local to Local+GC can tell us how much the gain comes from generated content alone, and then from Local+GC to FedGC can tell us further how much we get when the gradients are communicated.

Thank you for your time for reviewing and your comments. Here are our detailed responses.

---

W1: On the one hand, the authors show that increasing the amount of generative data always increases the learning performance. On the other hand, the authors also show that FL on only generative contents performs poorly in Table 5. This seems counterintuitive. Could the authors explain why?

Response:

That is a good point. Acturally, they are not counterintuitive. Please note that in Table 2, the performance of using 50000 samples is worse than performance of using 20000 samples.

Here is the rationale. (1) First, the results showing that only generative contents performs poorly indicates that the generative data cannot fully represent real data. (2) Second, when generative data is combined with real heterogeneous data, the generative data serves to mitigate data heterogeneity and overfitting issue. Therefore, the performance is increased after adding generative data.

However, when the number of generative samples is too large, the issue of dissimilarity between generative and real data tends to bring negative effects.

[Table R1. Performance under different number of generative samples.]

---

W2: If we allow each user to train a local model on its private data combined with generative contents, would the performance be comparable to FL training on private data combined with newly generated data?

Response:

Thanks for this insightful comment. Following your advice, we have conducted the following experiments. From the table, we see that there is still a huge gap between local training and FedGC, thus incentivizing clients to participate federated learning.

[Table R2. Performance comparison between local training with generative content and our FedGC.]

---

Q: Additionally, could authors discuss the possible dataset contamination? That is, whether the benchmark dataset is already used in training dataset for the generative model.

Response: Sure, we also take this factor seriously and that is why we have shown the following results in the submission.

1. The experiments on medical dataset HAM10000 in Table 4. Since stable diffusion can not generate medical with good quality, we choose this experiment to alleviate the concern of contamination. Please note that the generated images are completely different from the real iamges and we decided not to posting them to avoid causing discomfort. Here, we put the results below for convinience. Despite the dissimilarity, our method still brings significant gain.

[Table R1. Results on medical dataset.]

2. The experiments of training on generated data only in Table 6. Here, we put the results below for convinience. From the table, we see that merely using generative data to train a model fail to perform well, indicating that there is a huge gap between generative data and real data. Generated data can only exhibit its effectiveness when used in conjunction with real data in our proposed FedGC.

[Table R2. Comparisons with training on genrative data only. Training on generative data only achieves low performance.]

3. We compare real data and generative data in Figure 8 via visualization, where we see that there is a large amount of generative data with incorrect concept and can not fully represent real data.

---

Overall, we hope that our responses can fully address your concerns and will be grateful for any feedback.

Dear Reviewer,

We have carefully considered your comments and provided further clarifications and explanations in detail. We have:

• Shown clear evidence that the results in Table 2 and Table 6 are not counterintuitive.
• Provided experimental results to verify sufficient incentive for clients to participate FL.
• Provided our detailed discussion on dataset contamination.

As Discussion phase will end in 16 hours, we would be grateful if you could check our responses and reconsider your rating.

Best regards,

Authors

Thank you for your time and comments. Here are our responses in detail.

---

W1: The author does not consider the generation cost in the paper. The Stable Diffusion model needs at least 4.2GB space to deploy locally, and the memory consumption of generation is huge for the IoT or cross-device FL setup. For the black-box foundational models such as ChatGPT, the prompts directly leak the data privacy to the server of ChatGPT. The method does not fit the FL setups?

Response:

Foremost, we want to reiterate the emphasis of our paper: (1) the idea of leveraging advanced generative models to assist training on private data, and (2) the general framework with diverse solutions for each sub-step.

Regarding the concern of generation cost, it can be easily alleviated by generating data using online interfaces (rather than generating locally), which are commonly available right now (e.g., huggingface) [1]. And we can generate generate data without real-data guidance for such cases since our proposed FedGC provides diverse solutions for each generation step. In this case, both the issues of generation burden and privacy issue are alleviated.

Besides, we acturally have recognized such potential concern so that we do not use real-data-guidance for all the experiments except Table 4. However, since our focus is to propose a general and multifunctional framework, we still decided to include this because such strategy can bring additional benefit if the computational resource is available.

[1] https://huggingface.co/runwayml/stable-diffusion-v1-5

---

W2: The mixed-up training of synthetic and real private data directly increases the computational burden for the local devices?

Response: There could be some misunderstanding. Mixed training does not increase the computational burden.

Please note that throughout our experiments, for each round, we are running with the sample number of SGD iterations for all clients and methods. That is, after mixing data, the used batch size and the number of SGD iterations are exactly the sample as baselines.

---

W3: I am concerned about the in-domain generation problem. We could not distinguish whether the performance boost-up is coming from the FedGC or the in-domain generation of the foundational models.

Response: Acturally, we have put efforts to alleviate such concerns from three aspects.

1. The experiments on medical dataset HAM10000 in Table 4. Since stable diffusion can not generate medical with good quality, we choose this experiment to alleviate the concern of in-domain generation. Please note that the generated images are completely different from the real iamges and we decided not to posting them to avoid causing discomfort. Here, we put the results below for convinience. Despite the dissimilarity, our method still brings significant gain.

[Table R1. Results on medical dataset.]

2. The experiments of training on generated data only in Table 6. Here, we put the results below for convinience. From the table, we see that merely using generative data to train a model fail to perform well, indicating that there is a huge gap between generative data and real data. Generated data can only exhibit its effectiveness when used in conjunction with real data in our proposed FedGC.

[Table R2. Comparisons with training on genrative data only. Training on generative data only achieves low performance.]

3. We compare real data and generative data in Figure 8 via visualization, where we see that there is a large amount of generative data with incorrect concept and can not fully represent real data.

---

Q1: How many synthetic data samples does the local client generate for the experiment in Table 1?

Response: The total amount for all clients is 20% of the original dataset. For example, CIFAR-10 has 50000 samples and then we will generate 10000 samples in total. Since there is 10 clients, each client will have 1000 samples.

---

Q2: During local data generation, does the local client only generate the data for the label it holds or generate the data for the whole label space among all participants?

Response: Local client generates data for the whole label space.

---

Overall, we are so regret to see that the reviewer gave such a rating for our new exploration in federated learning. We hope that our responses can fully address the reviewer's concerns and would be grateful to get any useful feedback to improve our work.

Thanks for the further comments and here are our responses.

---

W1: If the generation could be done on the cloud side, the communication cost would be very high for the client?

Response: Even in such cases, the communication cost is not HIGH as stated by the reviewer. On the contrary, the communication cost is quite LOW.

1. Even in such cases, the additional communication cost is acturally minor and the the enhanced performance is significant. Here, we provide a detailed example on launching FedGC on SCAFFOLD on CIFAR10 in the following table. From the table, we can see that FedGC can achieve significantly higher performance than the baseline while introducing minor additional communication cost. Besides, please note that we only introduce some downlink cost rather than uplink cost, and it is commonly known that the uplink is slower at least five times than the downlink [1,2]. Specifically, FedGC can achieve 5.07% absolute accuracy improvement while only introducing 0.007% additional communication cost!

 

[Table R3. Communication cost per client and accuracy in cases where we use cloud generation.]

 

2. To further alleviate the reviewer's concern, we provide the following table where we keep the communication cost less than baselines by reducing the communication rounds (i.e., 1-2 rounds reduction) for FedGC. From the table, we see that even with less communication cost, FedGC still significantly outperforms the baseline!

 

[Table R4. Accuracy comparison between FedGC and SCAFFOLD when keeping FedGC with less communication cost.]

 

[1] Yi, Liping, Wang Gang, and Liu Xiaoguang. "QSFL: A two-level uplink communication optimization framework for federated learning." ICML 2022.

[2] Konečný, Jakub, et al. "Federated learning: Strategies for improving communication efficiency." arXiv preprint arXiv:1610.05492 (2016).

Dear Reviewer,

We have carefully considered your comments and provided further clarifications and explanations in detail. We hope that we have well address them through the provided methodological and empirical evidence. As Discussion phase will end in 16 hours, we would be grateful if you could check our responses and reconsider your rating.

Best regards,

Authors