Rethinking the Starting Point: Enhancing Performance and Fairness of Federated Learning via Collaborative Pre-Training

One might then ask what is the motivation of clients participating in pre-training. First, they can benefit from the pre-trained model for any of their future tasks that share the same task domain. Secondly, the service provider will provide incentives to the clients to promote their participation. The service provider is willing to do this since once a generalized pre-trained model is constructed, the service provider can sell/provide this pre-trained model to any group of clients or customers that share the same task domain. Thanks to the reviewer’s comment, we have made the above points clearer in our revised manuscript. Please let us know if the reviewer has any unclear points related to our response.

 

To model the above practical scenarios with unseen classes during our experiments, we have split the dataset into 80 classes and 20 classes for pre-training and downstream, respectively. Note that this splitting is not done within the client, but naturally happens due to new clients joining the system and time-varying environments. However, as the reviewer pointed out, even in these scenarios, the classes observed during downstream FL might not always be the new, i.e., clients might still be interested in classifying some of the seen classes. To see the performance of different schemes under this scenario, we conducted additional experiments in which clients in the downstream FL task hold 'seen classes'. To achieve this, we sampled 10 classes from the pre-training dataset and 10 classes from our CIFAR-100 downstream dataset, resulting in 20 classes with 10 seen classes and 10 unseen classes. Each time, we randomly selected 5 classes to conduct non-IID downstream FL, and we repeated the experiment 10 times. The pre-trained model was solely trained on the 80 classes of CIFAR-100 originally presented in our paper. The table below shows the results. Each FL pre-training method is trained on a non-IID data distribution and initializes the non-IID downstream FedAvg tasks using the pre-trained model. The results are consistent with the ones with only unseen classes, confirming the advantage of the proposed pre-training algorithm. These new results are now included in Section C.6 of Appendix.

• Experiments with mixed seen/unseen classes during downstream FL:
  |Pre-training method (Scenario I) |Acc | Var | worst 10% | worst 20% | worst 30%
  |--|--|--|--|--|--|
  | Centralized | 82.63 | 63.57 | 67.35 | 69.22 | 70.38
  |FedAvg | 80.19 | 51.35 | 68.72 | 70.15 | 72.33
  | FedMeta | 83.14 | 39.85 | 67.29 | 71.35 | 73.81
  | q-FFL | 81.34 | 47.98 | 69.22 | 70.35 | 72.37
  | CoPreFL | 84.79 | 30.51 | 70.83 | 72.66 | 74.1

Side effects of centralized pre-training

The centralized dataset is constructed by aggregating datasets from all clients during the pre-training phase. As evidenced in Tables 5, 38, 39, 40, and additional experiments provided below, centralized pre-training can introduce side effects such as performance biases and large variance. This phenomenon arises from the lack of generalizability in the model's design. When a model undergoes pre-training in a centralized manner based on SGD, it becomes rigidly bound to the knowledge in the pre-training dataset. This fixity poses a challenge in adapting the model to the diverse clients that may contain new or unseen data in downstream tasks, which arise due to the time-varying environment or new clients as described in our first response above. As mentioned, the key clarification is that the datasets used in the pre-training and downstream FL phases are disjoint, and the set of clients involved in these two phases is also not the same.

 

Additionally, since our downstream tasks involve FL, the absence of learning dynamic distributed setups during the pre-training phase can further magnify this issue. Consequently, the model exhibits performance inconsistencies across different client datasets, leading to heightened performance variance. Our CoPreFL tackles these challenges by meta-updating the temporary global model to mimic FL scenarios that clients may encounter during downstream tasks. This enables our initial model to have generalization capability to arbitrary downstream FL scenarios. We have made these points clearer in Section I of the revised manuscript.

We appreciate the reviewer for the time and efforts, and providing constructive comments. Our responses are given below.

Comments regarding dataset splitting and unseen classes:

We appreciate the reviewer for this comment. We would like to first make the following two points clearer:

1. In our setting, clients do not intentionally split their local dataset into pre-training dataset and downstream dataset: The clients participating during pre-training actually use all their local data samples to construct a pre-trained model that can adapt to any downstream FL tasks. This pre-training stage can be viewed as a “preparation phase” where clients use all their current data to get prepared for various circumstances they may encounter in the future during the downstream task. From the service provider’s perspective, as depicted in Fig. 1 of the manuscript, the pre-training stage can be viewed as a process to prepare a good product (i.e., a generalized initial model) that can be provided to any potential customers in the future (i.e., any group of clients in future downstream tasks).

2. The pre-training stage and the downstream stage share the same task domain (e.g., image classification for self-driving cars, disease predictions for patients), but their specific objective/task could be potentially different due to the time-varying environment or new clients (that share the same task domain but with different objectives) joining the system, e.g., classifying different objects depending on the self-driving car’s environment, classifying different diseases depending on the patient’s interest, or face/speech recognition for new phone users. Here, the clients participating during the downstream task could be either the one who participated during pre-training or the one who didn’t attend the pre-training stage but have become interested in the related task. For example, a self-driving car who participated during pre-training might become interested in another classification task depending on their location, weather condition, or other time-varying environments, which will make them to get interested in unseen classes. Here, even when clients observe unseen classes, they can still benefit from the pre-trained model that is pre-trained with the same task domain, rather than starting from a randomly initialized model. As another example, a new patient who didn’t participate during pre-training may want to take advantage of the pre-trained model for solving its own task with new diseases, instead of starting from a randomly initialized model.

Under this scenario, our main question was: Can we design a pre-training methodology that can generalize well to any downstream FL tasks? Here, we would like to highlight that the downstream tasks are unknown during pre-training, and the downstream tasks may contain unseen classes, due to the time-varying scenarios and new clients joining the system as mentioned above. Note that if a specific downstream task is known during pre-training (i.e., if the set of classes for the downstream task is known), and if all those classes are available during pre-training, it is definitely possible to construct a specific pre-trained model tailored to that task by only utilizing the data samples that belong to those classes. However, this (i) requires the service provider to prepare different pre-trained models for every downstream task, which could be potentially infinite, and also (ii) requires the clients in the pre-training stage to contain all classes that will appear in any future downstream tasks, which is again impractical.

 

Due to these challenges, given the current set of clients, the goal of the service provider would be to construct a “generalized pre-trained model” that can quickly adapt to any, and possibly unseen future downstream tasks. This is the goal of this paper, and we propose a meta-learning-based pre-training algorithm to mimic and generalize well to any unseen future downstream tasks. During the meta-learning process of our scheme, each client splits its local dataset into support set and query set to “mimic” downstream scenarios with potentially unseen classes, but we note that the clients are still taking advantage of all data samples for pre-training: The support sets are used to construct a temporary global model while the query sets are utilized for meta-update, which enables the clients to obtain a generalized initial model when pre-training is completed. (to be continued in the next response)

Public dataset

Thank you for pointing this out. In response to the reviewer's suggestion, we have included an additional experiment considering a scenario where a public dataset is available. We conduct pre-training using the ImageNet_1K dataset following Nguyen et al.'s idea. We sample 200 images for each of the 1000 classes in ImageNet_1K. During the pre-training phase, we trained the model using the SGD optimizer with a learning rate of 1e-3 for 50 epochs, and considered this centralized training result as a baseline.

 

Here, we would like to emphasize that, as stated in Remark 2 of our manuscript, even when provided with a public dataset, we can intentionally split the dataset and apply our scheme (using multiple computing units available at the server) to mimic the distributed scenario of downstream federated learning and to capture generalization, which can handle the limitation of the naive centralized pre-training as mentioned above. Hence, we also implemented our scheme using the ImageNet_1K dataset. For our method, we distribute all the data across 100 nodes, sampling 20 nodes in each round, and conducted our CoPreFL for 50 rounds. For the downstream FL phase, we apply non-IID FedAvg to the 20-classes downstream dataset we used in CIFAR-100 dataset. As a result, all classes observed during downstream tasks are the seen classes that have already appeared during pre-training.

• Experiments with public dataset for pre-training (downstream on CIFAR-100): |Dataset & Pre-training Method | Acc | Var| |--|--|--| |CIFAR100 pretrained – Centralized| 81.30 | 69.44 | |CIFAR100 pretrained – FedAvg| 78.96 | 64.80 | | CIFRA100 pretrained – FedMeta | 82.45 | 48.72 | CIFAR100 pretrained – qFFL | 80.01 | 88.92
  | CIFAR100 pretrained – CoPreFL | 83.29 | 34.69 | ImageNet1K pretrained - Centralized | 86.75 | 67.34 | ImageNet1K pretrained - CoPreFL | 87.96 | 30.79

As we can expect, models pre-trained on ImageNet1K achieve higher accuracy compared to those pre-trained on CIFAR-100 since there is no unseen classes when pre-trained on ImageNet. More importantly, as mentioned, we can still apply our method by intentionally splitting the dataset and mimicking the distributed nature of downstream federated learning to achieve further performance improvements: The centrally pre-trained model on ImageNet achieves lower average performance and higher variance compared to our CoPreFL. This advantage of CoPreFL is achieved by initializing the model to get higher accuracy and better fairness in federated settings based on meta-learning. The overall results further confirm the advantage and applicability of our approach. These new results are now included in Section C.4 and Table 41 of Appendix.

Better results in Non-IID scenarios

Intriguingly, we observed this notable phenomenon in our experiments, when applying initializations pre-trained on different data distribution, the non-IID downstream tasks exhibited higher accuracy than the IID setting in some cases. This results prompted us to delve deeper into the implications of using an initialization pre-trained on a diverse distribution. One plausible explanation for the higher accuracy in the non-IID setting could be attributed to the fact that pre-training in a non-IID manner allows the model to capture heterogeneity information within the data. This learned heterogeneity information may exhibit a certain degree of similarity with the distribution of the downstream tasks during the subsequent phases, thereby contributing to the improved performance observed in some non-IID scenarios. This finding raises interesting questions about the interplay between pre-training strategies and downstream task distributions, suggesting that non-IID pre-training might facilitate better adaptation to certain non-IID downstream tasks.

Comparison with other FL method for pre-training

Based on the reviewer's comment we have conducted additional experiments by using FedDyn to pre-train the model on CIFAR-100. We consider IID and non-IID pre-training for FedDyn with $\alpha$ parameter of FedDyn is selected as 0.01. The downstream measurement is set to consist of 10 non-IID FedAvg tasks.The results show that our method still provides better initialization by taking both performance and variance into account during the pre-training phase. We have included these results in Section C.4 and Table 42 of Appendix.

• Experiments with other non-IID related method for pre-training : |Pre-training method (Scenario I) |Acc | Var | worst 10% | worst 20% | worst 30% |--|--|--|--|--|--| | IID-FedDyn | 83.15 | 40.13 | 69.55 | 71.32 | 73.95 | IID-CoPreFL | 84.36 | 38.56 | 73.66 | 75.63 | 77.40 |--|--|--|--|--|--| | NonIID-FedDyn | 81.23 | 53.17 | 68.39 | 70.23 | 72.46 | NonIID-CoPreFL | 83.29 | 34.69 | 71.58 | 73.20 | 74.59

Again, thank you for your time and efforts. Your comments are clear to the point and made us think carefully, and we feel we have managed to clarify all the issues raised: we tried to make our setting and goal clearer, justified why unseen classes can appear during downstream tasks with several examples, conducted additional experiments with mixed seen/unseen classes in downstream tasks, added new results using the public dataset, and provided further intuitions on centralized training and the performance on non-IID scenarios. In case there are remaining questions/concerns, we hope to be able to have an opportunity to further answer them.

We appreciate the reviewer for the response. We apologize that we have missed some of your comments.

Weakness 2: Pre-training in non-IID setups

In our CoPreFL, each client’s local dataset does not need to necessarily represent the global distribution. As the reviewer understands correctly, our key idea is to use meta-learning to tackle downstream FL. Since the clients conduct meta-update on the global model using their own local dataset (which could be non-IID) and the server aggregates all the meta-updated models, our model is actually learning to perform well across all clients’ local datasets in the system. This enables our pre-trained model to become a good initialization for downstream tasks; the constructed global model (starting from our initial model) performs well across all clients in the system during downstream. This is also confirmed via our extensive experimental results throughout the manuscript.

Weakness 5: Communication cost

Thanks for this comment. As mentioned in our response to the “public dataset above”, even when the public dataset is given, the central server can still adopt our CoPreFL to further enhance the performance (ImageNet1K - Centralized vs. ImageNet1K - CoPreFL). Since the public dataset is centralized, CoPreFL can be also conducted at the server using multiple machines. The communication load between these machines in the server is the cost for achieving a better performance compared to centralized SGD.

Conducting pre-training in a FL setup would become more important for the tasks in which the centralized public dataset is not large enough due to privacy issues. In healthcare applications, e.g., disease diagnosis, the available public datasets are not large enough to construct a strong pre-trained model that can generalize well, due to the privacy issues of collecting clients’ datasets. In such applications, different patients or hospitals can collaboratively pre-train the model based on their local datasets via FL, to construct a more robust, generalized initial model that can quickly adapt to any unseen downstream FL tasks. The communication cost in this process is a must to construct a strong pre-trained model.


Comments regarding dataset splitting and unseen classes

We also appreciate this comment. We agree with the reviewer that from the client’s viewpoint, the main goal would be to prepare a model for current classes, rather than "future classes". Our problem setup is more important from the service-provider’s perspective. From the service provider's viewpoint, once a robust pre-trained model is constructed, the service provider can sell/provide this model as an initial model to any group of clients that aims to construct a global model via FL, in future downstream tasks (Fig. 1 in the manuscript). Our main question was: Can we construct this kind of robust pre-trained model using the currently available clients? The service provider will use the current clients as a tool to construct a pre-trained model, and will give incentives to the clients to participate in the pre-training stage.

The service provider is willing to prepare this especially considering new clients joining the system (e.g., new self-driving cars joining a specific region [Badue et al]), which can introduce unseen classes. Moreover, the clients’ needs can also vary over time (e.g, as in continual FL settings [ICML’21], [CVPR’22]), and the clients may want to construct a global FL model at a specific timestep. These scenarios can be viewed as our key applications, and our work is suggesting the clients in these future downstream tasks to conduct FL starting from our pre-trained model, instead of starting from a randomly initialized model or other pre-trained models. Overall, the ultimate goal of the service provider is to construct a strong pre-trained model that can adapt to any of these kinds of future downstream scenarios. We have added the service provider's viewpoint in Section I of the manuscript.

[Badue et al.,] ”Self-driving cars: A survey." Expert Systems with Applications 165 (2021): 113816.

[ICML’21] Federated Continual Learning with Weighted Inter-client Transfer

[CVPR’22] Federated Class-Incremental Learning

Side-effect of centralized pre-training

Simply conducting centralized training via SGD could potentially restrict the knowledge to the pre-training dataset, which degrades the performance when the dataset during downstream FL is different from the pre-training stage. Although our scheme uses the same dataset as centralized pre-training, the key difference of our CoPreFL is the use of meta-learning. Instead of simply biasing the knowledge to a specific dataset, CoPreFL learns how to adapt well to downstream FL scenarios by meta-updating the global model, which leads to significant performance improvements.

Weakness 4: Contribution over other meta-learning based methods

Some personalized FL works (e.g., FedMeta) also adopt meta-learning. We clearly differentiate our work with existing strategies in Sections I and II of our manuscript. Here, we provide a more detailed response.

The distinction between our CoPreFL and meta-learning based FL works (e.g., FedMeta) lies in the application, which requires us to conduct meta-update in a totally different way. FedMeta is designed to offer personalization to individual clients. In contrast, our emphasis is on creating an initial model that can construct a good “global model” during downstream instead of “personalized models”. This is the reason why we need to update the temporary global model instead of the local models, which is the key technical difference with FedMeta. Another technical difference is the linear combination between fairness and average performance term, as the reviewer understands correctly. These two key techniques enables CoPreFL to construct a robust initial model that can quickly adapt to “any group of clients” (instead of individual clients) to construct a global model during downstream tasks. The advantages of CoPreFL can be also confirmed via extensive experiments throughout the manuscript.

Again, thank you for your time and efforts. We would appreciate further opportunities to answer any remaining concerns you might have.

Best, Authors

Number of tasks during downstream in CoPreFL

We appreciate the reviewer for pointing this out. Yes, we considered 5-way classification during downstream FL to model various unseen downstream scenarios. As mentioned in our second response above, our goal is to design a “generalized initial model” that can adapt to arbitrary downstream tasks that potentially contain unseen classes. To model unseen downstream scenarios, during experiments, we split the dataset into a pre-training dataset and a downstream dataset, allocating 80 and 20 classes for CIFAR-100 dataset, respectively. Here, to prove that our scheme works well for “arbitrary” downstream scenarios, we had to generate different FL scenarios during downstream (i.e., by adopting 5-way classification) and measure the averaged performance over different downstream scenarios. This allows us to assess whether the pre-trained model, generated using CoPreFL or other baselines, can offer effective initialization for a diverse range of downstream tasks. The goal is to evaluate the versatility of the pre-trained model in providing a robust starting point for various scenarios. However, this does not indicate that the number of downstream should be large in CoPreFL. To prove this, we now consider a “single downstream scenario“ with 20-way classification using all downstream classes (that have not appeared during pre-training). The results are provided below, indicating that CoPreFL still performs better than the baselines:

• Performance in a single-downstream scenario (Pre-training: Non-IID; Downstream: 20-way classification Non-IID FedAvg): |Pre-training Method | Acc | Var | worst 10% | worst 20% | worst 30% |--|--|--|--|--|--| |FedAvg | 81.35 | 59.69| 70.35 | 70.91 | 71.63 |FedMeta| 82.69|46.38 |72.74|73.51|74.39 |q-FFL| 83.27|51.48|73.01|73.82|75.02 |CoPreFL | 84.71|39.26|73.28|74.99|75.35

Here, it is also important to mention that CoPreFL targets scenarios where the target downstream tasks are “unknown” during pre-training. If we have prior information on the downstream task (100-way CIFAR-100 classification) during pre-training, and if these classes are available during pre-training, there is no reason to design the initial model to perform well on arbitrary downstream tasks by adopting meta-learning. In such cases, it might be more beneficial to adopt simple FedAvg during pre-training to capture a good data representation for downstream FL, as pre-training and downstream tasks share the same classes. The strength of our CoPreFL lies in the robustness to capture arbitrary and potentially unseen downstream scenarios (as depicted in Fig. 1 of the manuscript) which happen due to the time-varying local datasets of clients or new clients (that share the same task domain but with different objectives) joining the system.

Regarding baselines FedAvg and q-FFL

The comparison aims to evaluate the performance of downstream FL tasks initialized by pre-trained models trained using different FL methods. FedAvg and q-FFL do not involve a meta-learning design, so they perform their algorithms using selected clients during the pre-training phase. After obtaining the final global model for each method, we use it to initialize a fixed downstream FL task. We have included these schemes as a baseline since were not many baselines we could use for pre-training. We explored various federated learning algorithms (including those utilizing meta-learning differently from ours) and centralized training for pre-training baselines.

We appreciate the reviewer for the time and efforts, and providing valuable comments. Our responses to the comments raised by the reviewer are given below.

Comparison with other federated meta-learning algorithms

Yes, as the reviewer pointed out, it possible to replace the personalization stage (of existing federated meta learning methods) with a federated global model training and adopt as a baseline. We have done this for FedMeta in our original submission. Based on the reviewer’s comment, during the rebuttal period, we have further implemented another personalized FL method (Fallah et al.). The final global model from this approach is employed as an initialization in our downstream scheme. Specifically, we implement the personalized baseline under a non-IID pre-training data distribution and subsequently use the global model of this method to initialize the downstream non-IID FL tasks. The experiment is conducted on CIFAR-100 dataset, following the pre-training/downstream partition in our paper. The results are provided below:

• Comparison with modification of PerFedAvg (Fallah et al.): |Pre-training Method (Scenario I) | Acc | Var | worst 10% | worst 20% | worst 30% |--|--|--|--|--|--| NonIID-PerFedAvg | 81.58 |49.73 |67.21|71.05|72.39 NonIID-CoPreFL | 83.29 | 34.69 | 71.58 | 73.20 | 74.59

The result further confirms the superiority of our method that meta-updates the temporary global model by balancing average performance and fairness. This table is now included in Section C.4 of the Appendix of the revised paper.

The utilization of server’s data to mimic unseen scenario, and partitioning server’s data

We appreciate this comment. We would like to first highlight that we are not aiming to capture representations of datasets in downstream FL. This is especially impossible when arbitrary and potentially new classes appear during downstream FL, which is the setup that we consider. In our setup, the pre-training stage and the downstream stage share the same task domain (e.g., image classification for self-driving cars, disease predictions for patients), but their specific objective/task could be potentially different due to the time-varying environment or new clients (that share the same task domain but with different objectives) joining the system, e.g., classifying different objects depending on the self-driving car’s environment, classifying different diseases depending on the patient’s interest, or face/speech recognition for new phone users. Here, the clients participating during the downstream task could be either the one who participated during pre-training or the one who didn’t attend the pre-training stage but have become interested in the related task. For example, a self-driving car who participated during pre-training might become interested in another classification task depending on their location, weather condition, or other time-varying environments, which will make them to get interested in unseen classes. Here, even when clients observe unseen classes, they can still benefit from the pre-trained model that is pre-trained with the same task domain, rather than starting from a randomly initialized model. To summarize, arbitrary “new classes” that have not appeared during pre-training may appear in downstream scenarios.

 

Due to the above setting with arbitrary and potentially unseen classes, even the client’s local dataset may not be a good representation of the datasets of clients in downstream tasks. Meta-learning plays a key role here to tackle this challenge. Instead of directly capturing data representations for downstream, our initial model aims to learn “how to quickly adapt” to arbitrary unseen downstream FL scenarios, which is achieved by distinguishing between the support and query sets during pre-training. To facilitate this in Scenario II, we aimed to construct |m| different query sets based on the server’s data to “mimic downstream FL scenarios”, which is done by arbitrary partitioning. This is a natural choice since we are not able to know any patterns/information of downstream FL. What we have shown in Scenario II is that, even this arbitrary |m| partitioning of the server’s data can lead to significant performance improvements by mimicking a scenario that clients have different objectives/datasets in downstream scenarios. This can even improve the performance of the model compared to the one in Scenario I, as the clients can utilize all of its local dataset as the support in Scenario II. These results further confirm the applicability and effectiveness of the proposed CoPreFL in Scenario II.

I would like to thank the authors for the detailed rebuttal, addressing my questions one by one. However, I am still not completely convinced by the usefulness of Scenario II, and the use of the server data by equal partitions in the proposed algorithm. Given that the paper is rather empirical in nature (without theoretical guarantees) while the experiments are rather weak due to the comparison to baselines (FedAvg and q-FFL) that are not designed for pre-training or meta-training, I cannot raise the score further for this paper.

Thank you for your response and keeping the positive score. We would like to briefly make the point regarding equal partition clearer. Instead of equal partitioning, one can also divide the server-side dataset into partitions with unequal sizes and use them for meta-update. However, during pre-training, the server does not know the dataset sizes of clients in future downstream tasks. In this case, one intuitive way is to treat all clients equally/fairly, by meta-updating the model with the same query set sizes. We show that this equal partitioning provides significant performance improvements as can be seen in our experiments.

Regarding the baselines, there were not many baselines we could consider due to the lack of works on pre-training for/in FL, and thus we tried our best to include most of the variations we can think of (e.g., FedMeta, q-FFL, centralized pre-training). Thanks again for your time and efforts.

Best, Authors

We thank the reviewer for the comments and valuable feedback. Our responses to the comments raised by the reviewer are given below.

Key applications and the use of large-scale public dataset

We appreciate the comment. Our responses can be summarized as follows:

1. Pre-training in a FL setup would be especially important for the tasks in which the centralized public dataset is not large enough due to privacy issues: In healthcare applications, e.g., disease diagnosis, the available public datasets are not large enough to construct a strong pre-trained model that can generalize well, due to the privacy issues of collecting clients’ datasets. In such applications, different patients or hospitals can collaboratively pre-train the model based on their local datasets via FL, to construct a more robust, generalized initial model that can quickly adapt to any unseen downstream FL tasks.

2. More importantly, as mentioned in Remark 2 of our manuscript, our scheme is directly applicable to centralized public datasets to further improve the performance: As mentioned in Remark 2, the server can intentionally split the centralized dataset and apply our scheme (using multiple computing units available at the server) to obtain a pre-trained model. Here the question is, what is the advantage of doing this compared to simple centralized training? As can be seen from Table 5, 38, 39, 40, centralized pre-training can introduce side effects such as performance biases and large variance. This phenomenon arises from the lack of generalizability in the model's design. When a model undergoes pre-training in a centralized manner based on SGD, it becomes rigidly bound to the knowledge in the pre-training dataset. This fixity poses a challenge in adapting the model to the diverse clients that may contain new or unseen data in downstream tasks, which arise due to the time-varying environment or new clients joining the system, e.g., classifying different objects depending on the self-driving car’s environment, classifying different diseases depending on the patient’s interest, or face/speech recognition for new phone users. Additionally, since our downstream tasks involve FL, the absence of learning dynamic distributed setups during the pre-training phase can further magnify this issue. Consequently, the model exhibits performance inconsistencies across different client datasets, leading to heightened performance variance. Our CoPreFL tackles these challenges by meta-updating the temporary global model to mimic FL scenarios that clients may encounter during downstream tasks. This enables our initial model to have generalization capability to arbitrary downstream FL scenarios. We have made these points clearer in Section I of the revised manuscript.

To confirm the advantages of our CoPreFL over centralized pre-training in the large-scale public dataset, we have included an additional experiment considering a scenario where a public dataset is available. We conduct pre-training using the ImageNet_1K dataset following Nguyen et al.'s idea. We sample 200 images for each of the 1000 classes in ImageNet_1K. During the pre-training phase, we trained the model using the SGD optimizer with a learning rate of 1e-3 for 50 epochs, and considered this centralized training result as a baseline. For our method, we distribute all the data across 100 nodes, sampling 20 nodes in each round, and conducted non-IID CoPreFL for 50 rounds. For the downstream FL phase, we apply ten non-IID FedAvg tasks to the 20-classes downstream dataset we used in CIFAR-100 dataset. As a result, all classes observed during downstream tasks are the seen classes that have already appeared during pre-training.

• Results using public large-scale dataset for pre-training (downstream on CIFAR-100): |Dataset & Pre-training Method | Acc | Var| |--|--|--| |CIFAR100 – Centralized| 81.30 | 69.44 | |CIFAR100 – FedAvg| 78.96 | 64.80 | | CIFAR100 – FedMeta | 82.45 | 48.72 | CIFAR100 – qFFL | 80.01 | 88.92 | CIFAR100 – CoPreFL | 83.29 | 34.69 | ImageNet1K - Centralized | 86.75 | 67.34 | ImageNet1K - CoPreFL| 87.96 | 30.79

As we can expect, models pre-trained on ImageNet1K achieve higher accuracy compared to those pre-trained on CIFAR-100 since there is no unseen classes when pre-trained on ImageNet. More importantly, as mentioned, we can still apply our method by intentionally splitting the dataset and mimicking the distributed nature of downstream FL to achieve further performance improvements: The centrally pre-trained model on ImageNet achieves lower accuracy and higher variance compared to our CoPreFL. This advantage of CoPreFL is achieved by initializing the model to get higher accuracy and better fairness in federated settings based on meta-learning. The overall results further confirm the advantage and applicability of our approach. These new results are now included in Section C.4 and Table 41 of Appendix.

Additional comparisons with other FL algorithm in fine-tuning stage

We appreciate this comment. Based on the reviewer’s comment, we additionally consider FedProx, a more advanced FL algorithm that addresses heterogeneity compared to FedAvg, to examine the robustness and generalizability of our pre-trained method. The results are provided in the table below. The additional experiments are conducted using the CIFAR-100 dataset. The FL methods in the pre-training phase are implemented under a non-IID data distribution, and we carry out 10 non-IID FedProx tasks for downstream evaluation. Here, we note that not only centralized pre-training but also other pre-training methods (including ours) can take advantage of FedProx during downstream tasks to further improve the performance. The results demonstrate that our pre-trained method maintains superiority in downstream FedProx compared to other pre-training methods. We would also like to note that the choice of FedAvg as our downstream task is made to minimize the varying impact introduced by other FL algorithms. Comparing the pre-training + downstream pairs, the improvement of CorPreFL+FedAvg over Centralized+FedProx shows that a better initialization, which considers the distributed scenario and balances fairness/performance in the pre-training phase, could potentially benefit the inferior downstream FL algorithm. These new results are now included in Appendix C.7.

• Experiments with applying FedProx in downstream FL: |Pre-training Method + Fien-tuning Method | Acc | Var | worst 10% | worst 20% | worst 30% |--|--|--|--|--|--| Centralized + FedProx | 82.39 | 51.46 | 70.33 | 71.28 | 73.52 FedAvg + FedProx | 79.53 | 46.15 | 63.17 | 69.74 | 71.59 FedMeta + FedProx| 81.77 | 63.12 | 63.58 | 68.19 | 70.28 q-FFL + FedProx | 83.19 | 52.12 | 67.41 | 70.59 | 72.33 CoPreFL + FedProx | 84.31 | 30.55 | 70.19 | 73.88 | 75.13 CoPreFL + FedAvg | 83.29 | 34.69 | 71.58 | 73.20 | 74.59

Defination of $p(\mathcal{G})$

In Eqs. (1) and (2), $p(\mathcal{G})$ represents the distribution of all possible sets of clients that appear in downstream. Hence, $\mathcal{G}$ is a random variable while G is a specific downstream task sampled from distribution $p(\mathcal{G})$. Based on these definitions, equations (1) and (2) can be also clearly defined without any issue.

Gradient computation in Algorithm 1 Line 17

The gradient is computed on the client side. The communication process involves the server acquiring a temporary aggregated model from the local model trained using the support set (line 10). The server then broadcasts this model to each client for meta-evaluation, where gradients are computed locally on each client (lines 11-14). Subsequently, clients send these gradients back to the server, which utilizes them to update the temporary aggregated model (line 17). We have made these points clearer in Section 3.2 of the revised manuscript.

Best-performing value selection

We report the results of our method, which achieved the best accuracy among all $\gamma$-configurations. Moreover, for fair selection/comparison, we also report the results of other baselines with their own best accuracy when searching for hyperparameters.

Again, we appreciate the reviewer for the time and effort. Your concerns are clear, and we feel we have successfully addressed all the issues you raised. Please let us know in case there are remaining questions/concerns; we hope to be able to have an opportunity to further answer them.

We appreciate your response. We pre-trained the model for “ImageNet1K - Centralized” by ourselves to make a fair comparison with “ImageNet1K - CoPreFL” under the same setting. The sentence in the Appendix needs to be removed. We have corrected it accordingly. Our goal here is to highlight that even when the public dataset is given, we can always apply our CoPreFL to mimic downstream FL scenarios, to construct a more robust initial model that can quickly adapt to arbitrary and possibly unseen downstream tasks. If a large public dataset is not available due to privacy issues (e.g., medical applications) as mentioned above, we must rely on pre-training in distributed settings.

Please let us know if you have any further questions. Thank you.

Best, Authors

Performance Guarantees and Baselines

We agree with the reviewer that a formal analysis would make the paper stronger. We actually found this extremely challenging since it requires us to formally show “how the initial model constructed by each pre-trained model” affects the performance of the “global model constructed during downstream FL”. The authors of [1], which studies centralized pre-training for downstream FL, show that pre-trained model can help since it can potentially reduce the initial loss value in the beginning of downstream FL. However, when considering a general scenario involving arbitrary unseen downstream tasks, this might not always be the truth. Due to the challenge of providing exact theory, we aimed to consider as many baselines as possible to strengthen the empirical results. Since there were not many baselines we could use for pre-training, we explored various federated learning algorithms (including those utilizing meta-learning differently from ours) and centralized training for pre-training baselines. We evaluated the performance of these schemes over different data distribution setups for pre-training and downstream, and validated the superiority of our scheme. Nevertheless, we can still explain the superior performance of our scheme as follows: (i) We are adopting meta-learning, which is known to effectively handle unseen tasks, and (ii) we are taking advantage of meta-learning tailored to downstream FL by meta-updating the temporary global model considering both average performance and fairness, which enables us to construct a generalized initial model that leads to a strong global model. We believe that further investigating FL pertaining with theoretical analysis is a promising future research direction.

 

[1] Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning, Nguyen et al.

Parameter selection detail and q-value for qFFL

For all FL pre-training methods, we consistently use 50 global rounds with each round of local training comprising 5 iterations per participant. This configuration has proven to be stable in our setup and is consistent with findings in prior work [1]. Regarding the choice of the q value in q-FFL, we selected values from the range [1, 3, 5] based on previous research findings [2]. The authors have demonstrated that smaller q values (q > 0, i.e., 0.1 or 1) tend to result in better performance. On the contrary, higher q values might compromise performance and slow down the convergence speed. The selected range aimed to strike a balance between fairness considerations and optimization for better overall performance.

 

[1] On the Importance and Applicability of Pre-Training for Federated Learning, Chen et al., ICLR ‘22.

[2] Fair Resource Allocation in Federated Learning, Li et al., ICLR ‘20.

Better results in Non-IID scenarios

Intriguingly, we observed this notable phenomenon in our experiments, when applying initializations pre-trained on different data distribution, the non-IID downstream tasks exhibited higher accuracy than the IID setting in some cases. This results prompted us to delve deeper into the implications of using an initialization pre-trained on a diverse distribution. One plausible explanation for the higher accuracy in the non-IID setting could be attributed to the fact that pre-training in a non-IID manner allows the model to capture heterogeneity information within the data. This learned heterogeneity information may exhibit a certain degree of similarity with the distribution of the downstream tasks during the subsequent phases, thereby contributing to the improved performance observed in some non-IID scenarios. This finding raises interesting questions about the interplay between pre-training strategies and downstream task distributions, suggesting that non-IID pre-training might facilitate better adaptation to certain non-IID downstream tasks.

Response for other questions

• How should the support and query sets be selected in practice by each client?

For each client, we randomly select 80% of their local data as support set and the remaining 20% as query.

 

• Does the phrase "All of these schemes are adopted during the pre-training phase to construct initial models." in "Baselines for pre-training" mean that qFFL and FedAvg algorithms were modified to consider pertaining?

Yes, the phrase means that we consider different baselines, including FedAvg and q-FFL, during the pre-training phase using the pre-training dataset. The goal is to compare their ability to initialize the downstream FL tasks, which is the main focus of this work. We have included these schemes as baselines to strengthen the empirical results as there were not many baselines available for pre-training for FL.

 

• What is the difference between the performance variance across clients during pretraining and downstream tasks for high gamma in the noniid settings?

Performance variance is assessed similarly during both pretraining and downstream tasks, involving the evaluation of the global model's performance on each participant's query/testing samples. However, the context in which variance is employed differs. During pretraining, performance variance serves as a factor to control the pretraining objective, utilizing a parameter, $\gamma$, to modulate its influence. A higher $\gamma$ encourages the pre-trained model to lean towards achieving better averaged accuracy performance, prioritizing accuracy over variance. Conversely, in downstream tasks, performance variance is merely a measure of the quality of the tasks and is independent of IIDness.

 

• Other suggestions & typos

All typos have been corrected in our revised paper.

We again thank the reviewer for providing very helpful comments. We would love to answer any remaining questions the reviewer might have.

We thank the reviewer for the positive comments and valuable feedback. We appreciate the reviewer’s acknowledgment on the importance of our work. Our responses to the comments raised by the reviewer are given below.

Practical applications and differences with FedMeta

Key applications: Consider a healthcare application where each client, such as a hospital or an individual patient, aims to build a comprehensive global model capable of classifying a wide range of diseases. However, individual clients may possess limited types of diseases in their local datasets – for instance, one client may have data on diseases A and B but lacks information on diseases C and D. In this context, federated learning becomes essential. Clients need to collaborate to construct a global model that not only reflects the diseases available locally but also incorporates information about diseases not present in their individual datasets, ensuring a more robust and universally applicable healthcare model. Similarly, in the domain of autonomous vehicles, each self-driving car may strive to develop a global model for scenario detection in various weather conditions. However, individual cars might encounter limited weather scenarios locally – one car might navigate through a desert environment, while another faces challenges in a snowy storm. Through federated learning, these cars can collectively construct a global model that accounts for a broad spectrum of weather conditions, ensuring robust scenario detection capabilities for all vehicles involved. The wide availability of datasets open to public also makes our Scenario II practical, as the server can hold these public datasets and meta-update the global model learned by clients.

 

Contributions beyond FedMeta: The distinction between our CoPreFL and FedMeta lies in the application, which requires us to conduct meta-update in a totally different way. FedMeta is designed to offer personalization to individual clients, e.g., when a specific client is interested in predicting only diseases A and B, or when a specific self-driving car is interested in the model tailored to a specific weather. In contrast, our emphasis is on creating an initial model that can construct a good “global model” during downstream instead of “personalized models”, targeting the aforementioned applications. This is the reason why we need to update the temporary global model instead of the local models, which is the key technical difference with FedMeta. Another technical difference is the linear combination between fairness and average performance term, as the reviewer understands correctly. These two key techniques enables CoPreFL to construct a robust initial model that can quickly adapt to “any group of clients” (instead of individual clients) to construct a global model during downstream tasks. The advantages of CoPreFL can be also confirmed via extensive experiments throughout the manuscript.

 

We have clarified the distinction with FedMeta in the related work section of the revised manuscript.

Thank you very much for acknowledging our efforts and raising your score.

Best, Authors