Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuning

Thank you very much for taking the time to review our paper and providing your valuable comments. Please find below our responses to your comments and questions.

We will revise the presentation of the contributions section in the introduction by breaking down the last paragraph into itemized points.

Baselines in Table 1

For the results in Table 1, we used online kernel-based models to evaluate the performance of Fed-POE in a convex setting. We believe that we compared the performance of Fed-POE against all state-of-the-art online federated kernel learning algorithms which provide rigorous theoretical guarantees. However, all these works were for the year 2022 and before. Moreover, if one employs Fed-DS, which is a baseline from 2023, for the online federated kernel learning problem investigated in Table 1, it is equivalent to Fed-OMD in this specific case. Therefore, our results in Table 1 are relevant to the baselines from 2023.

Improvements on Air and FMNIST Datasets

We believe the performance of the proposed Fed-POE on the Air and FMNIST datasets demonstrates its advantage in real-time prediction tasks. The main challenge in real-time prediction arises when there is no prior information about the streaming data, making it difficult to evaluate model performance before the task begins. As discussed in Section 3.2 of the paper, it is theoretically unclear whether federated or local models perform better under such conditions. The performance largely depends on the dataset.

Results presented in Section 5 confirm that the relative performance of algorithms varies with the dataset. Tables 1 and 2 show that for the CIFAR-10 dataset, federated models outperform local models, while for the Air dataset, local models perform better than federated models. Furthermore, for WEC and FMNIST, both local models and personalized federated learning models outperform non-personalized federated models. This variability complicates the decision between local and federated models. However, the results in Tables 1 and 2 indicate that Fed-POE consistently outperforms all baselines, albeit marginally on some datasets. This suggests that Fed-POE's performance is robust across different datasets, making it a reliable choice for real-time prediction tasks in the absence of prior information.

Complexity

We would like to clarify that at each time step, Fed-POE fine-tunes only one federated model while making inferences with multiple federated models. Given that the computational complexity of making inferences is usually considerably less than that of model fine-tuning, we believe Fed-POE does not introduce significant additional computational overhead. Furthermore, the number of federated models used by clients can be adjusted to ensure that the required computation and memory costs remain manageable for the clients. We analyze the computational complexity of Fed-POE.

Let $C_F$ denote the number of computations required to fine-tune the model $f$, and let $C_I$ represent the number of computations required to make an inference with model $f$. Assume that the complexity of model selection in Algorithm 1 is negligible compared to fine-tuning and making inferences with model $f$. According to Algorithm 2, each client performs $2C_F + (M+2)C_I$ computations per time step. Therefore, the computational complexity of Fed-POE for each client is $\mathcal{O}(C_F + MC_I)$. Typically, fine-tuning deep neural networks with backpropagation requires significantly more computations than making inferences with them. Thus, if the model $f$ is a deep neural network, $C_I$ is negligible compared to $C_F$. In this case, the computational complexity for each client using Fed-POE is $\mathcal{O}(C_F)$, which is comparable to most state-of-the-art federated learning methods.

Increase in the Number of Clients

An increase in the number of clients does not pose scalability issues for Fed-POE compared to other state-of-the-art federated learning methods. As with most other federated learning algorithms, each client using Fed-POE only fine-tunes one model at each step and sends the update to the server.

Thank you very much for taking the time to review our paper and providing your valuable comments. Please find below our responses to your comments and questions.

We would like to briefly review the main contributions of this paper, which we believe are of interest and utility to the community. We hope the respected reviewer will consider these contributions.

Section 3 of the paper highlights the challenge of choosing between local models and federated models in online federated learning when there is no prior information about data distribution. In Section 4.1, the paper proposes an ensemble federated learning algorithm that effectively leverages both local and federated models to provide more reliable performance for clients. Moreover, Section 4.2 demonstrates how federated learning can be employed to address catastrophic forgetting in real-time decision making.

Performance Gain

We believe the performance of the proposed Fed-POE in Section 5 demonstrates its advantage in real-time prediction tasks. The main challenge in real-time prediction arises when there is no prior information about the streaming data, making it difficult to evaluate model performance before the task begins. As discussed in Section 3.2 of the paper, it is theoretically unclear whether federated or local models perform better under such conditions. The performance largely depends on the dataset.

Results presented in Section 5 confirm that the relative performance of algorithms varies with the dataset. Tables 1 and 2 show that for the CIFAR-10 dataset, federated models outperform local models, while for the Air dataset, local models perform better than federated models. Furthermore, for WEC and FMNIST, both local models and personalized federated learning models outperform non-personalized federated models. This variability complicates the decision between local and federated models. However, the results in Tables 1 and 2 indicate that Fed-POE consistently outperforms all baselines, albeit marginally on some datasets. This suggests that Fed-POE's performance is robust across different datasets, making it a reliable choice for real-time prediction tasks in the absence of prior information.

Further Ablation Study

To address your concerns, we performed additional ablation studies to analyze the effect of batch size $b$ on the Fed-POE performance. We conducted experiments on the CIFAR-10 dataset, varying the batch size $b$ and the number of models $M$ selected by each client to construct the ensemble model. The table below illustrates the results. As observed, the batch size $b=1$ results in the worst accuracy, mainly due to the forgetting process where models overfit to the most recently observed data. However, increasing the batch size from $b=10$ or $b=20$ to $b=30$ does not significantly improve the accuracy. Larger batch sizes may lead the model to perform better on older data, as the model is trained on older data over more iterations. Therefore, from this study, we conclude that a moderate batch size is optimal, considering that an increase in batch size leads to an increase in computational complexity. Based on these findings, we choose $b=10$. Furthermore, in Figure 1 in Appendix D, we illustrate the regret of Fed-POE and other baselines over time for the CIFAR-10 and WEC datasets.

Complexity

Similar to other federated learning algorithms, using Fed-POE each client only updates one federated model per step. Therefore, we believe that using Fed-POE does not impose significant additional updating complexity compared to other federated learning counterparts. Please find below the computational complexity analysis of Fed-POE.

Let $C_F$ denote the number of computations required to fine-tune the model $f$, and let $C_I$ represent the number of computations required to make an inference with model $f$. Assume that the complexity of model selection in Algorithm 1 is negligible compared to fine-tuning and making inferences with model $f$. According to Algorithm 2, each client performs $2C_F + (M+2)C_I$ computations per time step. Therefore, the computational complexity of Fed-POE for each client is $\mathcal{O}(C_F + MC_I)$. Typically, fine-tuning deep neural networks with backpropagation requires significantly more computations than making inferences with them. Thus, if the model $f$ is a deep neural network, $C_I$ is negligible compared to $C_F$. In this case, the computational complexity for each client using Fed-POE is $\mathcal{O}(C_F)$, which is comparable to most state-of-the-art federated learning methods.

Thank you very much for taking the time to review our paper and providing your valuable comments. Please find below our responses to your comments and questions.

Relations between Federated Models and Local Models

The federated model can differ significantly from the local models, especially when the data distribution among clients is heterogeneous. The degree of data heterogeneity influences the similarity between the federated model and the local models. Although the federated model is trained on local models, if the gradient trajectories of other clients differ significantly from those of a particular client, it is expected that the federated model will be substantially different from the local model of that outlier client. We can add this to the paper.

Learning Rates in the Long Run

If the time horizon $T$ is unknown which may often be the case in the long run, the doubling trick technique (see e.g., [R1]) can be effectively used to set the learning rates $\eta$ and $\eta_c$ while maintaining theoretical guarantees. The doubling trick is a well-known technique in online learning that adaptively sets the learning rates without knowing the time horizon. We add a note about this to the paper.

[R1] N. Alon, N. Cesa-Bianchi, C. Gentile, S. Mannor, Y. Mansour, and O. Shamir, “Nonstochastic multi-armed bandits with graph-structured feedback,” SIAM J. Comput., vol. 46, no. 6, pp. 1785–1826, 2017.

Complexity

Let $C_F$ denote the number of computations required to fine-tune the model $f$, and let $C_I$ represent the number of computations required to make an inference with model $f$. Assume that the complexity of model selection in Algorithm 1 is negligible compared to fine-tuning and making inferences with model $f$. According to Algorithm 2, each client performs $2C_F + (M+2)C_I$ computations per time step. Therefore, the computational complexity of Fed-POE for each client is $\mathcal{O}(C_F + MC_I)$. Typically, fine-tuning deep neural networks with backpropagation requires significantly more computations than making inferences with them. Thus, if the model $f$ is a deep neural network, $C_I$ is negligible compared to $C_F$. In this case, the computational complexity for each client using Fed-POE is $\mathcal{O}(C_F)$, which is comparable to most state-of-the-art federated learning methods.

Moreover, at each time step, each client sends one updated model to the server. Let $P_f$ denote the number of parameters in model $f$. Therefore, the amount of information that needs to be transmitted to the server is $\mathcal{O}(P_f)$, which is the same as most state-of-the-art federated learning methods.

Thank you very much for taking the time to review our paper and providing your valuable comments. Please find below our response to your review.

The main advantage of the proposed Fed-POE compared to the straightforward online gradient descent approach is its ability to provide sublinear regret upper bounds for both global and personalized regret. The conventional method, as presented in Theorem 1, cannot guarantee a sublinear regret upper bound for personalized regret. The conventional online gradient descent approach only guarantees global regret. Fed-POE achieves sublinear regret upper bounds for personalized regret, as shown in Equations (12) and (20). Moreover, global regret upper bounds for Fed-POE are presented in Equations (11) and (19).