Variational Federated Continual Learning

Comparison of computational speed.

We added the comparison of training and testing time overheads of different methods in Appendix F.4. For the same number of parameters, the training and inference speeds of VFCL are almost the same as those of the other compared methods.

It is worth noting that, in inference, VFCL does not perform Monte Carlo sampling, which uses the mean of the learned parameter distributions as the model parameters, and thus its inference efficiency is the same as that of a plain model.

Would our method still be better than existing approaches on real FCL datasets ?

To the best of our knowledge, existing FCL methods all utilize existing datasets to simulate FCL scenarios, the datasets used in the comparison methods are listed in the following table:

In addition, due to the privacy issues, it is very difficult to collect an FCL dataset of real scenarios, and to the best of our knowledge, there are no available FCL datasets from real scenarios, which is a research area need to be filled.

To further validate the effectiveness of our method, we added the comparison experiments on Overlapped CIFAR-100 and NonIID-50, which are benchmark datasets introduced by "Federated Continual Learning with Weighted Inter-client Transfer". Detailed results are reported in Appendix F.1, and our method still outperforms the comparison method on both datasets.

We consider constructing FCL datasets of real scenarios in our future work to better validate the effectiveness of the FCL methods.

How does Monte Carlo sampling affect training speed ?

We compare the training time for the number of Monte Carlo sampling $n$ from 1 to 10 in Fig. 8 of Appendix F.3. The results show that the training time is linearly correlated with $n$, the training time increases as $n$ increases. It is worth noting that $n$ does not affect the inference time, and if the training budget is sufficient, a larger $n$ can be used for better performance.

What range should be recommended for $n$ in real applications ?

We compared model accuracies for $n$ from 1 to 10 in Fig. 8 of Appendix F.3. We find that when $n$ is greater than 4, an increase in $n$ will not result in more performance gains, so we recommend setting $n$ no greater than 4.

Analytical experiments on varying $\lambda_k$.

$\lambda_k$ controls the relative weight between global and historical knowledge. We compare the accuracy and forgetting under different $\lambda_k$ in Table 7 in Appendix F.2. We found that both global and historical knowledge play important roles in model performance, and that missing either one leads to a decrease in accuracy. In addition, we found that the absence of historical knowledge causes severe forgetting.

Comparison with "Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach" (FedEP).

We compare FedEP with ours in terms of problem setting and method.

Problem setting: FedEP studies the classical federated learning problem, where all clients learn a task together, where all clients jointly learn a task, and the task definition is static and predefined. In our problem setting, each client learns sequentially on a private sequence of tasks, and clients learn different tasks. Our problem is more challenging, and the challenges include catastrophic forgetting due to continual learning, as well as heterogeneity across client tasks.

Method: FedEP is a centralized federated learning approach, which output is a single global model. Our approach is personalized, and it learns a personalized model for each client. The same point is that both FedEP and ours use variational inference to solve the approximate posterior.

Thanks for providing this work, and we have added it to the related work.

In global posterior aggregation, Eq. (11) does not seem related to Eq. (10).

The method described in Eq. (10) is a straightforward way to obtain a global posterior, however, it is intractable due to the non-i.i.d. client data. Although it is possible to approximate the solution using Monte Carlo sampling, the huge computational overhead and the need for a training dataset make it infeasible in practice.

Therefore, we introduce the conflation, which fuses multiple distributions into a single distribution, the aggregated distribution is with density proportional to the product of the densities of input distribution, and Eq. (11) describes how the parameters in the product of Gaussian distributions are calculated, i.e., the parameters of the final aggregated distribution.

We have revised this section to provide a more smooth transition and added a detailed explanation of global posterior aggregation in Appendix E.

What benefits does theoretical analysis bring to the paper ?

As a reminder, the key contribution of our approach is to integrate historical and global knowledge into the mixture prior and migrate the knowledge to the current model, mitigating local overfitting and catastrophic forgetting.

This theoretical analysis proves that the mixture prior can reduce the upper bound of generalization error (as $\lambda_{p}$ increases, the overall error decreases), and clarifies the effectiveness of the mixture prior, which is the core contribution of our work.

Further, we also experimentally (see Table 1) verify the positive effect of the mixed prior on the model performance, which is consistent with the findings of the theoretical analysis. In addition, we also find that too strong prior constraints, while reducing model forgetting, also reduce the model's adaptability to new tasks, ultimately undermining the model's learning ability.

We have revised this section of the article to provide a clearer explanation of the utility of theoretical analysis for method.

The experiments need to be run several times to validate the effectiveness of the method.

Our comparison experiments were run 3 times. We modified tables 1,2, and added the deviation bar (due to space constraints, we only added it on "Avg.", which represents the average of the results of the rows in the table.).

Why are the first 2 tasks called 'warm-up tasks' ?

In incremental learning, when the model learns the first few tasks, the model generalization performance will be poor due to less exposure to data, and the adaptation to new tasks is slow, which is reflected in the low accuracy of the new task learning, so we call the first few tasks of incremental learning 'warm-up tasks' .

What are the evaluation metrics in Tables 1 and 2 ?

The metrics under 1-10 in the table are the average accuracies of all the tasks that have been learned, e.g. after learning the 3rd task, the accuracy under 3 is the average of the model's accuracies on tasks 1,2,3.

"Avg." represents the average of the results of the rows in the table, which is the same as in "Federated Class-Incremental Learning".

We revised the caption of the tables and added a detailed description of the evaluation metrics in Appendix A.

Why we say previous methods do not combat local overfitting ?

Although existing FCL works have utilized the global model as the initialization for local training, the local model will gradually diverge from the initialized global model after starting local training, and the limited local data of the client will lead to local overfitting, which reduces the generalization ability and convergence rate. Existing methods focus on solving the catastrophic forgetting without explicitly preventing the divergence of the client local learning process, so we argue that these works do not address the overfitting problem.

Our method uses a personalized model on the client, and instead of replacing the local model with the weights of the global model, global knowledge is fused into the mixture prior, preventing local overfitting through regularization during the entire local training process, which can also be interpreted as passing global knowledge to the local model.

We have revised the introduction to provide a clearer explanation of why previous work ignored the overfitting problem.

The confusion caused by the Gaussian notation in Fig. 1.

We revised Figure 1 to clarify the meaning of the symbols.

Dear reviewer Htsy:

We greatly thank for your comments and suggestions on our submission. We have tried our best to address all the concerns in the response, including:

• Questions on global posterior aggregation.
• Questions about the contribution of theoretical analysis to our method.
• Suggestions and questions about the experiment.

Due to the coming deadline, we sincerely hope to know your thoughts about our responses. If you have further questions, please feel free to let us know. Thanks again.

Best regards,

Do the empirical setting ignore the catastrophic forgetting brought by client interference (a key challenge of FCL) ?

We agree that the catastrophic forgetting caused by client interference is a key challenge of FCL. However, we are afraid that the reviewer misunderstood our experimental setup. We would like to clarify that the client task sequences in our experiments are different across clients. In addition, we conducted experiments on the benchmark datasets according to the reviewer's suggestions, and the results validated the effectiveness of our method (see later answer for more details).

We provide a detailed explanation of how the client private task sequences in our experiments were constructed in Appendix A. First, the original dataset is split into several tasks that contain no overlapping classes. Second, the split tasks are randomly selected as the client's task and and the classes of tasks included are also randomly selected. Private task sequences from different clients may have similar (i.e., overlapping classes), unrelated, or interfering tasks. Clients utilize the knowledge of other clients in the learning process to improve the performance of the local model while preventing interference from the knowledge of other clients.

In FedWeIT, client interference is addressed through parameter decoupling and selective knowledge migration, while our approach addresses this challenge through personalization and client selection. We describe the client personalization in detail in Appendix A.2, the weights of the backbone as well as the structure of the classifiers are personalized for different clients, which allows VFCL to learn on personalized client task sequences. Client selection is described in Algorithm 1, where the server coordinates the learning of clients with similar tasks in a synchronized manner, which speeds up the convergence rate and reduces client interference.

Limited experimental setting compared with FedWeIT.

We have restated our experimental setup in the first reply, our setup is equally challenging with FedWeIT, the client task sequences are heterogeneous, capable of responding the client interference.

To further validate the effectiveness of our method, we adopted the reviewer's suggestion to rerun the experiments on benchmark datasets Overlapped-CIFAR100 and NonIID-50 (repeated 3 times to obtain the mean and deviation bar), and added the detailed results to Appendix F.1, where the experimental results show that our method outperforms the others.

Experimental results that lack deviation bar are not conclusive.

All of our comparison experiments were repeated 3 times and averaged, and we revised Tables 1,2 to add deviation bar to the results. Due to space constraints, we only added deviation bar on "Avg.", and "Avg." denotes the average of the results in a single row of the table.

Relationship between theoretical analysis and method ?

In FCL, multiple tasks are learned sequentially, and each task is learned by a separate federated learning (FL) process. Since all FL processes are isomorphic, we analyze the upper bound of the generalization error of a single FL process to analyze the overall FCL process.

The theoretical analysis guided our approach as follows: First, the upper bound of the generalization error of our method is affected by sample size, model capacity, and mixture prior (which is the core innovation of our work). Second, mixture prior is theoretically shown to reduce the upper bound on generalization error. Third, we obtain results in our experiments that are consistent with the theoretical analysis, verifying the positive effect of the mixture prior on the model performance.

Lack of background on conflation.

The goal of global aggregation is to find a distribution which can integrate the posterior distributions of all clients. Conflation can fuse information from multiple distributions into a single distribution, and the aggregated distribution is with density proportional to the product of the densities of input distribution. Eq. (11), which is the calculation of the parameters of the global posterior distribution, replaces the optimization process in Eq. (10).

We revised this section and added a detailed description of conflation in Appendix E.

The FL characterization in Eq. (1) is strange.

The client models in our approach are personalized, so the final result obtained in Eq. (1) is all personalized models $\{\boldsymbol\theta_c\}_{c=1}^{C}$. In contrast, traditional centralized federated learning outputs only one global model, we thought this distinction might bother reviewers.

Are the mixing weights in Eq. (5) learnable ?

The mixing weights in Eq. (5) are not learnable, they can be set manually (in our paper) or calculated using existing client evaluation methods.

Dear reviewer uvyG:

We sincerely appreciate your kind efforts in reviewing our paper. Now, we tried our best to clarify all the concerns in our response, including major concerns:

• Concerns about the experimental setup.
• Suggestions for re-running the experiment on the benchmark datasets introduced in FedWeIT.
• Questions on the relationship between theoretical analysis and method.

Due to the coming deadline, please feel free to let us know your thoughts, if you have further questions. Great thanks.

Regards,

---

Dear Authors,

Thank you for the detailed update. I need more time to go over all the new results you have provided.

For now, I have one question:

In Algorithm 1 (see page 15), the server workflow will loop over each task & for each task, it selects clients who will be assigned to the task. Is it supposed to know each client's task sequence in advance?

If that is the case, how will this algorithm be used in practice where the client's task sequence is not known in advance?

--

Furthermore, assuming that we know the task sequence in advance is the same as assuming away the client's interference.

The pseudocode clearly suggests that the clients will not be trained in the order of the tasks they received. Instead, the server dictates which client solves each task at which step. This is no longer continual learning where you cannot manipulate the order in which tasks are presented at will.

--

Otherwise, assuming the server only selects clients that are assigned the task in the current round, doesn't it mean that it already forgot about the other clients' solution model of this task who were constructed in previous rounds?