Rehearsal-Free Continual Federated Learning with Synergistic Regularization

Thanks for helpful comments from the esteemed reviewer! As for the revised manuscript, we will highlight them in purple or blue font in both the main text and the appendix for your convenience.

W1&W2. The novelty and detailed analysis of FedSSI. (blue in Line 300 and 337)

R1: Thank you for this valuable comment. The unique challenges in this paper are more comprehensive yet practical than existing CFL works. In CFL, there are only a handful of works that successfully tackle both catastrophic forgetting and data heterogeneity simultaneously. Moreover, these existing works often rely on substantial resource expenditures and risk privacy breaches, employing techniques such as generative replay, memory buffers, and dynamic networks—primarily traditional centralized methods that overlook the resource constraints of client devices.

We believe that a good paper often starts from practical applications to explore the underlying issues. In our article, we have explored the contradiction between the resource constraints of clients and the generally higher resource costs in continuous learning through empirical experiments. Our paper takes a different approach, starting by considering the resource limitations of client devices in FL and leveraging appropriate technologies. From this foundation, we aim to solve both catastrophic forgetting and data heterogeneity challenges concurrently. We hope our work will propel CFL research towards more practical and real-world scenarios.

Detailed Analysis: Unlike SI algorithms in centralized environments that only consider the data distribution of a single environment, FedSSI introduces a Personalized Surrogate Model (PSM) to balance data heterogeneity across clients. Here, the PSM is not used as the target classification model; it is solely employed to calculate the contribution of parameters. Before clients receive new tasks, a PSM will be trained along with the global model on the current local task. Since this is purely local training assisted by an already converged global model, the training of the PSM is very fast (accounting for only 1/40 of the training cost per task and requiring no communication). We calculate and save the parameter contributions during the local convergence process of the PSM, which can then be locally discarded after its contribution has been computed. Then, each client trains on the new task with the local model and paprameter contribution scores.

At the technical level, FedSSI's PSM is not a simple combination of Personalized Federated Learning (PFL) and Continual Learning (CL) techniques. PFL aims to train a personalized model for each client, which is the target classification model. Here, PSM serves as a computational auxiliary model for parameter contribution, skillfully balancing data heterogeneity while functioning as the model for calculating parameter contributions in SI. The use of this auxiliary model differs from the goals of PFL, exhibiting unique characteristics of CFL.

W3. Concerns about the dependency of FedSSI on SI.

R3: Thank you for raising this concern. Although the FedSSI algorithm uses PSM to improve the SI algorithm, the selection of the SI algorithm itself is not random. In our empirical experiments, we analyzed a vast number of existing CFL methods and techniques, as well as traditional CL techniques. The SI algorithm was chosen as the appropriate one among them, and it has been widely recognized for its efficiency and feasibility in most scenarios. We believe that this assumption is acceptable, similar to how many FL studies are based on CNN and ResNet series networks without considering the feasibility of techniques on a single Linear network.

W4. Exploration of new model mixing methods for CFL.

R4: Thank you for your helpful comment. Similar to the response in R1, if it is merely a simple model mixing method, there have already been numerous studies on this in PFL, rendering it of little research value and difficult to achieve significant innovation. However, at the technical level, FedSSI's PSM is not a simple combination of Personalized Federated Learning (PFL) and Continual Learning (CL) techniques. PFL aims to train a personalized model for each client, which is the target classification model. Here, PSM serves as a computational auxiliary model for parameter contribution, skillfully balancing data heterogeneity while functioning as the model for calculating parameter contributions in SI. The use of this auxiliary model differs from the goals of PFL, exhibiting unique characteristics of CFL and better suited to the characteristics of CFL.

Thank you for your clear response. You have addressed all my concerns. Therefore, I decide to increase my score.

Thanks for helpful comments from the esteemed reviewer! As for the revised manuscript, we will highlight them in green or blue font in both the main text and the appendix for your convenience.

W1. Concerns about the rehearsal-free claim. (green in Line 309)

R1: Thank you for raising this concern. I apologize for the ambiguity in our expression that caused your misunderstanding. In FedSSI, the PSM is not used as the target classification model; it is solely employed to calculate the contribution of parameters. Before clients receive new tasks, a PSM will be trained along with the global model on the current local task. Since this is purely local training assisted by an already converged global model, the training of the PSM is very fast (accounting for only 1/40 of the training cost per task and requiring no communication). We calculate and save the parameter contributions during the local convergence process of the PSM, which can then be locally discarded after its contribution has been computed. Then, each client trains on the new task with the local model and paprameter contribution scores.

In Equation.5, we used "previous task" to distinguish it from a "new task," but in reality, the "new task" had not yet arrived, and the "previous task" in the original text actually referred to the current task. We have revised the ambiguous text.

W2. Confusing definitions about the class-incremental learning. (green in Line 172-175)

R2: Thank you for this constructive comment. We are pleasantly surprised to receive such constructive suggestions, which is unprecedented. We have revised the original description and decided to abandon the statement that class-incremental tasks completely share the same domain. Instead, we have modified it to state that the main challenge in this scenario stems from changes in the number of classes, which we believe is a more precise formulation (we hope to obtain your confirmation).

W3. Lack of in-depth analysis about FedSSI.(blue in Line 300 and 337)

R3: Thank you for raising this concern. The unique challenges in this paper are more comprehensive yet practical than existing CFL works. In CFL, there are only a handful of works that successfully tackle both catastrophic forgetting and data heterogeneity simultaneously. Moreover, these existing works often rely on substantial resource expenditures and risk privacy breaches, employing techniques such as generative replay, memory buffers, and dynamic networks—primarily traditional centralized methods that overlook the resource constraints of client devices.

We believe that a good paper often starts from practical applications to explore the underlying issues. In our article, we have explored the contradiction between the resource constraints of clients and the generally higher resource costs in continuous learning through empirical experiments. Our paper takes a different approach, starting by considering the resource limitations of client devices in FL and leveraging appropriate technologies. From this foundation, we aim to solve both catastrophic forgetting and data heterogeneity challenges concurrently. We hope our work will propel CFL research towards more practical and real-world scenarios.

At the technical level, FedSSI's PSM is not a simple combination of Personalized Federated Learning (PFL) and Continual Learning (CL) techniques. PFL aims to train a personalized model for each client, which is the target classification model. Here, PSM serves as a computational auxiliary model for parameter contribution, skillfully balancing data heterogeneity while functioning as the model for calculating parameter contributions in SI. The use of this auxiliary model differs from the goals of PFL, exhibiting unique characteristics of CFL.

Thanks for helpful comments from the esteemed reviewer! As for the revised manuscript, we will highlight them in red or blue font in both the main text and the appendix for your convenience.

W1. Concerns about novelty and unique challenges. (blue in Line 300 and 337)

R1: Thank you so much for raising this concern. Novelty: This paper originates from an analysis of existing Continual Federated Learning (CFL) methods, focusing on the issue of resource constraints at the client side. Such empirical analysis is forward-looking and significant within the field of CFL, reflecting the advanced nature of our motivation. However, under the condition of considering resource constraints, CFL methods fail to effectively address both catastrophic forgetting and data heterogeneity, which are core issues in continual learning and federated learning respectively. FedSSI first empirically explores and demonstrates the superiority of the Synaptic Intelligence (SI) algorithm in mitigating catastrophic forgetting and reducing resource consumption.

Building on this, it employs a Parameter Sharing Mechanism (PSM) to further address the issue of data heterogeneity. At the technical level, FedSSI's PSM is not a simple combination of Personalized Federated Learning (PFL) and Continual Learning (CL) techniques. PFL aims to train a personalized model for each client, which is the target classification model. Here, PSM serves as a computational auxiliary model for parameter contribution, skillfully balancing data heterogeneity while functioning as the model for calculating parameter contributions in SI. The use of this auxiliary model differs from the goals of PFL, exhibiting unique characteristics of CFL.

The unique challenges in this paper are more comprehensive yet practical than existing CFL works. In CFL, there are only a handful of works that successfully tackle both catastrophic forgetting and data heterogeneity simultaneously. Moreover, these existing works often rely on substantial resource expenditures and risk privacy breaches, employing techniques such as generative replay, memory buffers, and dynamic networks—primarily traditional centralized methods that overlook the resource constraints of client devices. We believe that a good paper often starts from practical applications to explore the underlying issues. In our article, we have explored the contradiction between the resource constraints of clients and the generally higher resource costs in continuous learning through empirical experiments. Our paper takes a different approach, starting by considering the resource limitations of clients in FL and leveraging appropriate technologies. From this foundation, we aim to solve both catastrophic forgetting and data heterogeneity challenges concurrently. We hope our work will propel CFL research towards more practical and real-world scenarios.

W2. Scalability of FedSSI and Configurations. (red in Line 873-895)

R2: Thanks a lot for your comment. At line 374, we described the experimental Configuration and noted that the remaining details are provided in Table 6 (Appendix). We set up different experimental environments based on the size of the datasets (for example, CIFAR10 used 20 clients with a client selection rate of 40%, i.e., 8 clients per round of communication), which is a common FL setting. In response to the scalability concern raised by the esteemed reviewer, we conducted further experiments by increasing the number of clients to 100 while reducing the client selection rate to 10%. We performed some related experiments on CIFAR10 and Digit10 ($\alpha=10.0$), with the results as follows:

Since the dataset needs to be divided into different numbers of tasks, an excessive number of clients can lead to a very small number of samples per client, making model training difficult. However, FedSSI still maintains a leading position. To better address this issue, we will conduct subsequent experiments on larger datasets such as DomainNet to ensure the accuracy of cross-silo validation. (Due to the longer training time, we regret that we cannot provide experimental results on large-scale datasets during the rebuttal period.)

Thanks for helpful comments from the esteemed reviewer! As for the revised manuscript, we will highlight them in blue font in both the main text and the appendix for your convenience.

W1. Hard to understand why FedSSI can address data heterogeneity. (blue in Line 300 and 322)

R1: Thanks a lot for raising this concern. FedSSI can theoretically be proven to address the data heterogeneity. Considering that our writing might lead to misunderstandings on the part of the esteemed reviewer, I will briefly reintroduce the core of our method here:

Unlike SI algorithms in centralized environments that only consider the data distribution of a single environment, FedSSI introduces a Personalized Surrogate Model (PSM) to balance data heterogeneity across clients. Here, the PSM is not used as the target classification model; it is solely employed to calculate the contribution of parameters. Before clients receive new tasks, a PSM will be trained along with the global model on the current local task. Since this is purely local training assisted by an already converged global model, the training of the PSM is very fast (accounting for only 1/40 of the training cost per task and requiring no communication). We calculate and save the parameter contributions during the local convergence process of the PSM, which can then be locally discarded after its contribution has been computed. Then, each client trains on the new task with the local model and paprameter contribution scores. We have supplemented the algorithm in Q1 for your better understanding.

The training of the PSM is defined in Equation 5. Denote that α refers to the degree of data heterogeneity, while λ is a control coefficient in the training process of PSM. In Proposition 1, we show that adjusting λ can control whether the proportion of knowledge in PSM leans towards the local distribution or the global distribution (i.e., it is related to α). When α has a higher value, indicating a trend towards homogeneity in distribution, clients need to focus more on local knowledge. This means that by setting a larger λ value, PSM can rely more on local knowledge. Thus, we can empirically and theoretically judge there exists a positive correlation between α and λ, which is supported by Proposition 1 and extensive experiments conducted in Table 3.

W2. Concerns about the novelty and unique mechanism of FedSSI. (blue in Line 337)

R2: Thank you for raising such an insightful comment. This paper originates from an analysis of existing Continual Federated Learning (CFL) methods, focusing on the issue of resource constraints at the client side. We believe that a good paper often starts from practical applications to explore the underlying issues. In our article, we have explored the contradiction between the resource constraints of clients and the generally higher resource costs in CL through empirical experiments. Such empirical analysis is forward-looking and significant within the field of CFL, reflecting the advanced nature of our motivation.

However, under the condition of considering resource constraints, CFL methods fail to effectively address both catastrophic forgetting and data heterogeneity, which are core issues in continual learning and federated learning respectively. FedSSI first empirically explores and demonstrates the superiority of the Synaptic Intelligence (SI) algorithm in mitigating catastrophic forgetting and reducing resource consumption. Building on this, it employs a Parameter Sharing Mechanism (PSM) to further address the issue of data heterogeneity.

At the technical level, FedSSI's PSM is not a simple combination of Personalized Federated Learning (PFL) and Continual Learning (CL) techniques. PFL aims to train a personalized model for each client, which is the target classification model. Here, PSM serves as a computational auxiliary model for parameter contribution, skillfully balancing data heterogeneity while functioning as the model for calculating parameter contributions in SI. The use of this auxiliary model differs from the goals of PFL, exhibiting unique characteristics of CFL.

W3 & Q1. Reference issues and algorithm. (blue in Line 32 and 270)

R3: Thanks for this constructive suggestion. We have provided an algorithm to illustrate the procedure of FedSSI. Regarding the issue of references, we hope to receive guidance from the esteemed reviewer, but we have revised a mistake in Introduction section and will thoroughly examine this matter later and make corrections in subsquent versions.

Dear reviewer y9b8, during this rebuttal period, we have:

• Explained how our proposed approach addresses the issue of data heterogeneity with both theoretical and empirical analysis.
• Clarified the unique challenge and novelty of our method
• Revised reference issues and provided the algorithm description.

In this rebuttal, all the main concerns the esteemed reviewers raised are similar, and our responses have received acknowledgment and score improvements from all the remaining reviewers. We would love to hear your feedback on our updates and look forward to discussing any remaining concerns you may have. Thank you for your time and consideration.

Dear Reviewer y9b8,

I hope this comment finds you well. During the rebuttal and discussion phrase, all the main concerns the esteemed reviewers raised are similar, and our responses have received acknowledgment and score improvements from all other three reviewers. Since there are only a few hours left until the end of the discussion,we would love to hear your feedback on our updates and look forward to discussing any remaining concerns you may have. Thank you for your time and consideration.

Authors,

Thank you once again to all the reviewers for your diligent efforts during the review and discussion! As the discussion phase draws to a close, considering the new insights into the novelty of FedSSI and the improved presentation in our revised manuscript and rebuttal, we are summarizing here to facilitate the understanding by the esteemed reviewers:

Motivation: In CFL, only a limited number of works have successfully addressed both catastrophic forgetting and data heterogeneity simultaneously. Furthermore, these existing works often rely on significant resource overheads and pose risks of privacy breaches, utilizing techniques such as generative replay, memory buffers, and dynamic networks—primarily traditional centralized methods that neglect the resource constraints of client devices. Our paper adopts a different approach, starting by acknowledging the resource limitations of client devices in FL and leveraging suitable technologies. Based on this foundation, we aim to tackle both catastrophic forgetting and data heterogeneity challenges concurrently. We hope our work will advance CFL research towards more practical and real-world applications.

Technique Improvement: Our improvement is not trivial but rather simple and efficient. We have incorporated the PSM module with SI, which is not merely a straightforward combination of PFL and CL techniques. PFL aims to train a personalized model for each client, which serves as the target classification model. In this context, PSM functions as a computational auxiliary model for parameter contribution, skillfully balancing data heterogeneity while also acting as the model for calculating parameter contributions in SI process. The utilization of this auxiliary model diverges from the objectives of PFL, demonstrating the unique characteristics of CFL.