Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator

We appreciate the comments and suggestions from the reviewer. The following is our response to the raised concerns.

Q1. Explaining Figure 1

Thank you for your observation. We agree that Figure 1 could be made more intuitive. Our approach is inspired by foundational works such as TARGET [17] and MFCL [19], which utilize generative models in federated class-incremental learning (FCIL). To clarify, Figure 1 illustrates that using only a data generator leads to bias towards recent classes, as shown in Figure 2 of the paper. To address this issue, we introduced a feature generator (Section 3.2.2), which enhances the balance between stability (retaining past knowledge) and plasticity (learning new tasks).

We have updated Figure 1 to include clearer labels, step-by-step explanations, visual cues to highlight the interplay between the two generators and the additional loss terms that mitigate these challenges.

Q2. Previous classes accuracy concern

Thank you for pointing this out. The imbalance between learning new tasks and retaining old knowledge is a well-recognized issue in continual learning, referred to as the stability-plasticity dilemma. To address this, we introduced a feature generator that improves the model’s ability to retain knowledge from earlier tasks, thereby increasing the testing accuracy in previous classes.

Our experimental results in Table 1 demonstrate that FedGTG achieves the lowest Average Forgetting (AF) among the evaluated FCIL methods, showcasing its ability to effectively preserve knowledge from past tasks. This result directly supports our claim in the abstract. We have revised the manuscript to highlight this evidence more clearly to ensure alignment with the claims made in the abstract.

Q3. Backbone evaluation

Thank you for raising this concern. In response, we conducted additional experiments using ResNet34 and ResNet50 backbones to further validate the generality and robustness of our approach. Below are the updated results for all methods across various datasets under the Non-IID scenario:

Table R1: Performance of various FCIL algorithms using ResNet34.

Table R1: Performance of various FCIL algorithms using ResNet50.

As shown in Table R1 and Table R2, FedGTG consistently outperforms other FCIL algorithms across datasets, reinforcing its adaptability and effectiveness.

Q4. Typo in line 83, should be “methods”.

Thank you for pointing out the typo. We have corrected it in the revised manuscript.

Q5. Does this work focus on cross-silo or cross-device FL? What is the number of clients evaluated?

Thank you for your question. Our approach is flexible and can be applied to both cross-silo and cross-device FL scenarios. In our experiments, we set the default number of clients to 50 to simulate a realistic FL environment.

To evaluate the robustness of FedGTG across varying client sizes, we performed additional analyses, as presented in Figures 5e and 5f of the paper. These figures demonstrate that FedGTG maintains its superior performance across a range of client sizes, showcasing its adaptability to different FL setups. We have clarified this in the revised manuscript to better highlight the applicability of our work.

Q6. What is the meaning of “stability-plasticity”?

In continual learning, the stability-plasticity dilemma refers to the challenge of achieving a balance where a model remains stable enough to retain previously learned knowledge (stability) while also being adaptable to learn new information (plasticity).

An ideal continual learning system must minimize catastrophic forgetting (loss of old knowledge) while maintaining sufficient flexibility to acquire new information. In our work, this balance is achieved through the dual-generator structure, where the feature generator improves stability and the data generator ensures sufficient plasticity for learning new tasks.

We want to thank you for providing valuable comments and questions. Below is our response.

Q1. Detailed setup

Thank you for your detailed comments on the experimental setup. In our experiments, the batch size of synthetic images and features per epoch is set equal to the batch size of real images. This ensures a fair comparison with prior methods and maintains consistency across training cycles. We will make these details explicit in the revised manuscript to improve clarity.

In the class-incremental setup, we conducted experiments using three widely recognized benchmarks for class-incremental learning as follows:

• Sequential F-CIFAR-10: The CIFAR-10 dataset consists of 60,000 32 × 32 color images in 10 classes, with 6,000 images per class. There are 50,000 training images and 10,000 test images. We split the training set into five disjoint subsets corresponding to 5 tasks.
• Sequential F-CIFAR-100. Sequential F-CIFAR-100 is constructed by dividing the original CIFAR-100 dataset, which contains 50,000 images belonging to 100 classes, into ten disjoint subsets corresponding to 10 tasks. In this way, each task has 5,000 images from 10 distinct categories, and each class has 500 images.
• Sequential F-tiny-ImageNet. Tiny-ImageNet is a subset of ImageNet, containing 100,000 images of 200 real objects. We follow settings in the work of Babakniya et al. [1] to form the Sequential F-tiny-ImageNet. In particular, we split the original dataset into ten nonoverlapping subsets. We consider each subset as a task whose images are labeled by 20 different classes, and each class has 500 samples.

These experimental setups were detailed in the Appendix, but we agree they are critical for understanding and will include them in the main text upon acceptance.

Q2. Experiments on challenging dataset

Thank you for your valuable suggestion to evaluate our approach on high-resolution datasets such as ImageNet. We agree that testing on ImageNet would provide stronger validation of the scalability and effectiveness of our method in handling complex and diverse real-world scenarios. Therefore, following the same setting from MFCL [1], we conducted experiments on the protocol version of ImageNet, named SuperImageNet where a dataset created by superclassing the ImageNet dataset, thus greatly increasing the number of available samples for each class. Below are the results of various FCIL algorithms on this dataset:

We can see that FedGTG still outperforms other FCIL methods in this dataset, showing its ability in the field.

Q3. Contribution concerns

Thank you for your feedback regarding the contributions. While incorporating a feature GAN might seem similar to traditional regularization methods, our novelty lies in the dual-generator architecture, which includes both data and feature generators. This dual approach addresses two critical challenges in federated class-incremental learning:

• Catastrophic Forgetting: The feature generator effectively preserves knowledge across tasks by balancing stability and plasticity, reducing bias toward recent classes.
• Bias Correction: The data generator complements the feature generator by ensuring robustness and better generalization across tasks, mitigating class imbalance.

These contributions are supported by extensive analyses and real-world experimental results, which demonstrate practical advantages over existing techniques. Additionally, our method introduces new insights into combining generative models and regularization for improved knowledge retention, which we believe distinguishes it from prior work.

Q4. Generative model concerns

Thank you for pointing out this important question. Our approach employs a data-free training mechanism for the generative model [2, 3], guided by the global model. Specifically, the generative model produces synthetic data based on the knowledge encoded in the global model after each task. This process does not require independent training for each task, thereby avoiding excessive storage or communication costs.

We acknowledge the potential issue of catastrophic forgetting within the generative model itself if trained incrementally. To address this, we focus on ensuring the global model effectively preserves existing knowledge and adapts to new tasks. This foundation improves the quality of data generated by the GAN. Preventing catastrophic forgetting in the generative model remains an open challenge and will be an area of focus for future studies.

Thank you for taking the time and effort to review our paper. We appreciate your valuable feedbacks and comments. We would like to address your concerns as follows:

Q1. Contribution concerns

Thank you for raising this concern. The key contribution of our work lies in the novel integration of dual generators (a data generator and a feature generator) collaboratively address catastrophic forgetting and mitigate the bias problem towards recent classes in federated class-incremental learning (FCIL). While the two-generator structure is an important aspect, the direction-controlling objective is equally significant, as it regulates the alignment of learned features across tasks, thereby enhancing model generalization.

To improve clarity, we have revised the manuscript to emphasize the role of the direction-controlling objective more explicitly. For instance, Figure 1 has been updated to better illustrate how the objective interacts with the dual generators and its impact on the learning process. This combination of mechanisms provides practical advantages, which are further validated through comprehensive experiments that demonstrate superior performance compared to existing techniques.

Q2. Training time comparison

We appreciate this valuable suggestion. To address this, we have included a detailed analysis of training time and computational costs in the revised manuscript. The table below summarizes the training time per round for all methods across various benchmarks:

As shown above, the increase in training time for FedGTG is comparable to MFCL and TARGET, with a moderate overhead. However, this cost is justified by the significant performance gains achieved, as demonstrated in Table 1 and Figures 4, 5, and 6 of the main paper. These results validate the efficiency and effectiveness of our approach despite the additional parameters required. We believe this balance of cost and performance underscores the practical value of FedGTG.

Q3. Missing related works

Thank you for pointing out this oversight. Yes, our method is exemplar-free, as it avoids storing past data while effectively tackling class-incremental learning challenges. We have revised the literature review section to include key works in exemplar-free continual learning, including the analytic continual learning branch [1-3] and prototypes-based CIL. These additions ensure a more comprehensive contextualization of our approach within the broader field.

Thank you for your positive feedback and constructive comments about our contributions. We have address your questions and comments below. Please let us know if you have any remaining questions.

Q1. Novelty concerns

Thank you for your feedback regarding novelty. While we acknowledge that the individual loss functions used in FedGTG are not novel, our contribution lies in the integration of dual generators (a data generator and a feature generator) into a unified framework. This approach specifically addresses the challenge of catastrophic forgetting in federated class-incremental learning (FCIL) and mitigates the bias towards recent classes. Additionally, our extensive analysis validates the effectiveness of this dual-generator approach across various real-world scenarios, offering practical insights and demonstrating significant improvements over existing techniques. We believe this innovative integration and its application to FL highlight the novelty of our work.

Q2. Hyperparameter selection

We appreciate your concern about the number of parameters. While it is true that each loss term in FedGTG has an associated hyperparameter, these parameters are carefully designed to allow fine-tuning for optimal balance between knowledge retention and adaptation to new classes. We offer basic hyperparameter settings based on extensive experimental results to help users First, note that GANs are sensitive to hyperparameters, we set the generative model's hyperparameters to the same values as MFCL [1] for a fiar comparison. Second, we modify one of the local side hyperparameters to see a difference in accuracy. Our FedGTG results from testing several hyperparameter settings on the CIFAR-100 dataset are shown below:

Table H1: Performance of FedGTG through various $\lambda_{\text{FT}}$.

Table H2: Performance of FedGTG through various $\lambda_{\text{logits}}$.

Table H3: Performance of FedGTG through various $\lambda_{\text{EFM }}$.

Table H1-3 shows that FedGTG is insensitive to hyperparameter settings, making it simple for users to select the ideal set of hyperparameters for their tasks. We have added these results to the revised manuscript to provide a more comprehensive evaluation.

Q3. Baselines comparison

In our manuscript, we compare FedGTG with a regularizer-based approach named FLwF-2T. For more baselines comparison, we conducted experiments on the federated class-incremental version of two regularized-based methods, FedWeIT [4] and FedEWC [5]. Below are the results for all methods across various datasets under the Non-IID scenario ($\alpha=0.5$) using ResNet18:

Table M1: Performance of various FCIL algorithms using ResNet18.

As shown in Table M1, the performance of FedGTG still outperforms other FCIL methods, fostering its effectiveness in the FCIL setting.

Q4. Architecture design

Thank you for your question regarding the architecture design. The architecture for the data generation model was adopted from [1] to maintain consistency with existing benchmarks, enabling a fair comparison with prior techniques. For the feature generation model, we utilized fully connected layers, as they are well-suited for processing flattened feature vectors, ensuring computational efficiency and scalability. The design choice aligns with our goal of balancing effective knowledge generation and resource constraints inherent to FL environments. We believe these architectural decisions provide a strong foundation for evaluating FedGTG while addressing the unique requirements of FCIL tasks.