We thank the reviewer for their thoughtful feedback. Below, we address each comment in detail:

---

What Would Happen if the Encoder Itself Experiences Forgetting?

In our proposed approach, the encoder is safeguarded from forgetting through a dual replay mechanism that ensures both the encoder and the decoder are updated to retain past knowledge. This is achieved by minimizing a loss function that explicitly regularizes their behavior during incremental updates. As described in lines 18–22 of Algorithm 2, the loss minimization simultaneously updates the encoder and the decoder.

To further clarify this, in the revised manuscript, we will:

• Include a detailed figure illustrating how the autoencoder (encoder + decoder) operates within our framework.

---

Distribution Inconsistency in the Latent Space

Our proposed approach addresses the potential distribution shift in the latent space through two mechanisms:

1. Knowledge Distillation: As outlined in lines 20 and 21 of Algorithm 2, knowledge distillation ensures that the updated encoder aligns with the previously stored features, reducing distribution inconsistency.
2. Recalculation of Latent Representations: After learning a new task, the latent representations are recalculated using the updated autoencoder, ensuring consistency between stored features and the current encoder (following iCaRL [1]).

In the revised manuscript, we will reemphasize these mechanisms to provide a clearer explanation of how distribution inconsistency is effectively handled.

---

Generation of Synthetic Data via Class Centroid Embeddings

In our proposed approach, synthetic data generation depends on whether the client has previously encountered the class associated with the received centroid embeddings. The two scenarios are as follows:

1. Client Has Seen the Class:

   • If the client has previously encountered the class, it already possesses the corresponding latent representations. In this case, the client decodes these stored latent representations to generate synthetic data, as outlined in lines 6 and 7 of Algorithm 2. The decoded data are then used for replay.

2. Client Has Not Seen the Class:

   • If the received class centroid embeddings $\{p_{ij}\}$ correspond to a class that the client has not seen (the scenario highlighted by the reviewer), the client follows the routine of Variational Autoencoders (VAEs). Specifically, the client constructs synthetic data by combining the received mean with Gaussian noise vectors, as described in line 9 of Algorithm 2.

We invite the reviewer to refer to lines 3–11 of Algorithm 2 for a detailed description of these processes.

In the revised manuscript, we will include a paragraph explicitly discussing these two scenarios for data generation, ensuring clarity and addressing potential concerns about this aspect of the approach.

---

Addressing the Bias Problem Caused by Multiple Clients

The bias problem, which arises when multiple clients each learn only a subset of categories, can be resolved by ensuring that each client constructs a representative (unbiased) minibatch during autoencoder training. This, as discussed above, is achieved through two mechanisms:

1. Decoding Latent Representations: Clients can use stored latent representations from previously learned tasks to construct synthetic data for a balanced training minibatch (the preferred mechanism).
2. Using Received Class Centroid Embeddings: Clients can generate synthetic data for unseen classes using the received class centroid embeddings $\{p_{ij}\}$ from the server, as described earlier.

This representative minibatch ensures that the training process accounts for all clients, tasks, and classes, mitigating bias, which is critical to addressing intra-client and inter-client task confusion according to the FCIL optimization problem (Eqs. 7 and 8).

In the revised manuscript, we will elaborate on this mechanism and its connection to the optimization framework to clarify how the bias problem is addressed.

---

The end

We appreciate the reviewer’s comments and are committed to addressing these points in the revised version. By incorporating the proposed clarifications, additional experiments, and visualizations, we aim to enhance the clarity and impact of our work (upon comprehensive discussions with you and other reviewers, we plan to include a figure and extensive experiments). We believe our contributions, both the theoretical underpinnings and the proposed approaches, will be valuable to the FCIL community and kindly request the reviewer to reconsider their initial score in light of these improvements. If there are more questions, we are eager to respond.

---

[1] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl:Incremental classifier and representation learning.

We thank the reviewer for their thoughtful feedback. Below, we address each comment in detail:

---

The presentation and Fig. 1

To address the comment on the presentation and Fig. 1, we will include a detailed figure in the revised paper illustrating how the autoencoder operates within our framework and how the autoencoder addresses the challenges our mathematical framework formalizes.

To summarize briefly:

• Our proposed FCIL method consists of three key components: encoder, decoder, and memory.
  • The encoder maps input samples to a latent space, which serves as a compact representation for both classification and memory storage.
  • The memory stores these latent representations for previous classes.
  • The decoder reconstructs raw samples from the stored latent representations when replay is required.
• Our approach addresses local forgetting by enabling replay of previously learned classes from their stored latent representations.
• For global forgetting this approach relies on synthetic data generation of other clients which is performed as described in lines 3–9 of Algorithm 2.

Fig. 1 aims to illustrate the implications of Eqs. 7 & 8, as discussed in the last two paragraphs of Section 3. In the revised paper, we will enhance the caption of Fig. 1 for clarity. The updated caption will read as follows:

This figure demonstrates two critical challenges in FCIL: intra-client task confusion and inter-client task confusion. Intra-client task confusion arises when a client’s model struggles to distinguish between tasks learned sequentially, as it has not encountered them simultaneously. This issue is addressed by replaying previous tasks within the client, as guided by the second term in Eq. 8. Inter-client task confusion occurs when the tasks from different clients are not seen together and hence not learned. This challenge is mitigated by replaying tasks from other clients, as guided by Eq. 7. Addressing intra-client task confusion reduces local forgetting, while resolving inter-client task confusion mitigates global forgetting.

We invite the reviewer to study Section 3 of the paper (to fully understand Fig. 1) where we formalize and discuss intra-client task confusion, inter-client task confusion, local forgetting, and global forgetting.

---

The Mathematical Formulation of FCIL and the Proposed Approach

The proposed approach, HR, is explicitly designed to solve the optimization problem presented in Section 3, as described in Eqs. 7 & 8. The connection between the mathematical framework of FCIL and the proposed approach is elaborated in the last two paragraphs of Section 4. We kindly invite the reviewer to refer to these paragraphs for further details.

The primary aim of the HR approach is to address two critical challenges in FCIL:

1. Intra-client task confusion, which arises when a client struggles to differentiate between sequentially learned tasks due to the lack of simultaneous exposure.
2. Inter-client task confusion, which occurs when task representations across different clients interfere with each other.

These issues are directly linked to local and global forgetting, as explained in Section 3. We are willing to respond to any specific or short question by the reviewer on the link between our proposed mathematical framework and the proposed architecture and algorithms. However, the best comprehensive & compelling explanation without sacrificing rigor is in the paper in Sections 3 & 4 and it is not reducible to a shorter length.

---

Insufficient Experiments

We appreciate the reviewer's suggestion for additional experiments. We have two pages available in the main paper and will utilize this space to include extended empirical results. Specifically, we will present experiments under varying skewness conditions, such as alpha = 0.1 and alpha = 0.5, to evaluate the robustness of the HR approach across diverse data distributions.

Also, in the revised paper, as Reviewer Wd4J requested, we will:

• Include visualizations showing accuracy trends on old tasks after completing each new task.
• Incorporate Backward Transfer (BWT) calculations to provide a comprehensive analysis of forgetting.

We hope by including these experiments in our paper the paper will have strong numerical support on top of its strong theoretical foundation.

---

Analysis of memory footprint

In the following Table (presented in the next comment section), we provide the performances for fixed memory (both the decoder and exemplars) and compute budgets. Furthermore, in the revised paper, we will include a table for comparing the amount of data communicated for exemplar, generative, and hybrid replay approaches (we will not present that Table here).

Again, as the reviewer is aware, we have two pages available in the main paper and will utilize this space to include extended empirical results.

---

In the revised paper, we will place reference [1] in the CIL section.

---

Questions

While CIFAR-100 (10/10/50/5) and (20/5/50/5), as well as TinyImageNet (10/5/300/30), follow the FCIL simulation setting described in [Dong], which uses disjoint tasks, the simulation scenarios for ImageNet-Subset (10/20/100/10) and (20/10/100/10) currently allow for some overlap between tasks. If the reviewer believes it is necessary, we are willing to rerun the experiments to ensure they adhere to the standard disjoint task setting for ImageNet-Subset.

---

Thanks for the correction. Each client does not need to train from task 1 because, in Algorithm 1, the client only needs to update for one task, but in line 3 of Algorithm 2, the algorithm begins h=1, which is unnecessary. We will fix this in the revised version.

---

Our FCIL system can cope with stragglers. If stragglers are given the history of positions of centroids of all the classes of other clients by the server with which they can generate synthetic data to overcome inter-client task confusion, then it can be proven that global forgetting will not take place. We will include this in the revised version of the paper.

---

[1] Wuxuan Shi and Mang Ye. Prototype reminiscence and augmented asymmetric knowledge aggregation for non-exemplar class-incremental learning.

[Dong] Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated Class-Incremental Learning. CVPR 2022.

We sincerely thank the reviewer for their thoughtful engagement and detailed feedback on our work. We deeply appreciate your recognition of our contributions, particularly the mathematical framework and the proposed Hybrid Rehearsal (HR) methodology. Your decision to raise your score to 6 is encouraging, and we are committed to addressing the remaining points to further enhance the clarity and rigor of our paper.

---

Commitment to Deliver the Requested Results

We promise to conduct and incorporate these results into the manuscript within ten days. These simulations will include:

1. Learning-forgetting dynamics across benchmarks for a thorough comparative evaluation. Specifically, we will include visualizations showing accuracy trends on old tasks after completing each new task. Also, we will incorporate Backward Transfer (BWT) calculations to provide a comprehensive analysis of forgetting.
2. Results in a realistic class imbalance scenario to align the experiments with the motivation described in the paper. Specifically, we will present experiments under varying skewness conditions, such as alpha = 0.1 and alpha = 0.5, to evaluate the robustness of the HR approach across diverse data distributions.
3. Other reviewers made other requests, such as including a figure to depict the architecture of our work and more experimental results, as you can find in the other reviewers' comments.

Some of these requests are made by you and some are made by other reviewers. We will do our best to address all.

---

We are confident that the forthcoming results and clarifications will address your concerns. We are forever grateful for your thoughtful engagement and valuable suggestions.

We thank the reviewer for their thoughtful feedback. Below, we address each comment in detail:

---

Role of the Autoencoder & visual representation of the HR approach

We appreciate the reviewer’s suggestion to clarify the role of the autoencoder in our proposed method. While other reviewers found this aspect clear, we understand the importance of ensuring accessibility for all readers. To address this, we will include a detailed figure in the revised paper illustrating how the autoencoder operates within our framework.

To summarize briefly:

• Our proposed FCIL method consists of three key components: encoder, decoder, and memory.
  • The encoder maps input samples to a latent space, which serves as a compact representation for both classification and memory storage.
  • The memory stores these latent representations for previous classes.
  • The decoder reconstructs raw samples from the stored latent representations when replay is required.
• Our approach addresses local forgetting by enabling the replay of previously learned classes from their stored latent representations.
• For global forgetting this approach relies on synthetic data generation of other clients which is performed as described in lines 3–9 of Algorithm 2.

---

Limitations of HR and potential directions for future work

In the revised paper, we will include the following paragraph:

Our proposed FCIL approach, named HR, combines exemplar-replay and model-replay, which requires careful consideration of the decoder's design to strike the right balance between realism and privacy. If the decoder lacks sufficient representational capacity, it may fail to generate realistic images necessary for effective replay. Conversely, if the decoder is overly complex, it could inadvertently memorize training samples, raising potential privacy concerns. This trade-off is particularly important because, for replay purposes, unlike for learning, generating high-quality samples is not necessary and a larger decoder would inflict a higher computational load during training (but not inference because the classification is only done via the encoder). Future work could explore alternative decoder architectures that optimize this balance, tailoring the trade-off between computational efficiency, replay performance, and privacy to the requirements of the target application.

---

How HR stands out from other hybrid methods

In the revised paper, we will outline the key differences between our approach and these studies.

In the methods detailed in [1, 2], classification occurs after the decoding process, employing a cross-entropy loss function as depicted in Fig. 2 of [1]. By contrast, our method performs classification directly within the latent space of the encoder, leveraging Euclidean distance. This technique's efficacy is further supported by another recent study in incremental learning [3], which also utilizes classification within the latent space post-encoder.

Furthermore, while the method in [1] applies quantization to the latent exemplars to enhance compression, incorporating similar quantization within our architecture (and also the work in [3]) would be straightforward. Implementing this augmentation in our latent space could be done with minimal effort and is expected to further improve performance.

In terms of the learning scenario, the works depicted in [1, 2] involve online learning, whereas our work studies offline learning. Our method, akin to that described in [3], utilizes a structured representation in the latent space, facilitated by the Lennard-Jones Potential formulations. In contrast, the method in [2] relies on an autoencoder with an unstructured latent space.

Finally, in terms of design simplicity and architectural efficiency, our work and that referenced in [3] hold substantial advantages over the approaches in [1, 2]. Our architecture showcases a clean and minimalistic design, a characteristic shared by the architecture presented in Fig. 1 of [3]. In contrast, the architectures shown in Fig. 2 of [1] and Figs. 2 and 3 of [2] appear ad hoc and unnecessarily complex, complicating their implementation and scalability.

Heavy relies on data generation based on latent exemplars and class centroids

It is known [4] that while for learning new tasks high-quality data is critical when it comes to replay for the sake of preventing forgetting to generate high-quality samples

---

[1] Hayes, Tyler L., et al. Remind your neural network to prevent catastrophic forgetting.

[2] Wang, Kai, Joost van de Weijer, and Luis Herranz. ACAE-REMIND for online continual learning with compressed feature replay.

[3] Tong, Shengbang, et al. Incremental learning of structured memory via closed-loop transcription.

[4] Gido M Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks.

is not necessary. Also, such an approach has been adopted in recent studies [1,2,3,4].

---

The Lennard-Jones formulation

The Lennard-Jones formulation in Eqs. 10 and 11 serve for the global alignment of centroids for new classes in new tasks. As described in lines 6–8 of Algorithm 1:

1. Clients compute embeddings for new classes and send unaligned centroids to the server.
2. The server aligns these centroids and sends them back to the clients.
3. Clients use these aligned centroids to train on new tasks, ensuring minimal overlap with existing classes.

Minor concerns

• The acronym "AHR" should have been HR and will be corrected in the revised paper.
• The caption for Table 1 will be made more descriptive as follows: We conduct our simulations for 3 datasets with 5 configurations. We report the performance (accuracy) and SEMs for 10 runs for exemplar-based, model-based, and hybrid baselines.

Questions

We provide the performances for fixed memory (both the decoder and exemplars) and compute budgets:

HR achieves higher performance within the same memory budget by storing more exemplars compared to other methods. Additionally, the computational cost of the decoder during training is negligible due to its carefully chosen design (6 and 9 CNN layers for the CIFAR-100 and miniImageNet decoders, respectively). Importantly, the decoder is not used during inference, ensuring no additional computational overhead in deployment.

---

Our evaluation metric is the final accuracy for all the classes on the test set. To address the reviewer's concern we will:

• Include visualizations showing accuracy trends on old tasks after completing each new task.

We have these results prepared and will include them in the available 2 pages of the main paper.

---

Local and global forgetting are caused by class imbalances at both the local and global levels. This has been extensively discussed in the original paper of Federated Class Incremental Learning (FCIL) [Dong]. Our contributions in this paper include:

1. A novel mathematical framework to formalize local and global forgetting (in Eq. 7 & 8), providing a theoretical foundation for addressing these challenges. Please read the last two paragraphs of Section 3.
2. The introduction of hybrid replay, which mitigates both forgetting types in a memory-efficient manner.

---

The end

We appreciate the reviewer’s comments and are committed to addressing these points in the revised version. We believe our contributions are valuable to the FCIL community and kindly request the reviewer to reconsider their initial score in light of these improvements.

---

[1] Hayes, Tyler L., et al. Remind your neural network to prevent catastrophic forgetting.

[2] Wang, Kai, Joost van de Weijer, and Luis Herranz. ACAE-REMIND for online continual learning with compressed feature replay.

[3] Tong, Shengbang, et al. Incremental learning of structured memory via closed-loop transcription.

[4] Gido M Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks.

[Dong] Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated Class-Incremental Learning. CVPR 2022.

We sincerely thank the reviewer for their thoughtful engagement and detailed feedback on our work. We deeply appreciate your recognition of our contributions, particularly the mathematical framework and the proposed Hybrid Rehearsal (HR) methodology. Your decision to raise your score to 5 is encouraging, and we are committed to addressing the remaining points to further enhance the clarity and rigor of our paper.

If our clarifications and forthcoming results meet your expectations, we would be truly grateful if you could consider raising your score to 6, as this would greatly enhance the chances of our work being accepted and making a meaningful contribution to the field.

---

Commitment to Deliver the Requested Results

We acknowledge the importance of providing the requested additional simulation results and their role in demonstrating the robustness of our method. We promise to conduct and incorporate these results into the manuscript within ten days. These simulations will include:

1. Learning-forgetting dynamics across benchmarks for a thorough comparative evaluation. Specifically, we will include visualizations showing accuracy trends on old tasks after completing each new task. Also, we will incorporate Backward Transfer (BWT) calculations to provide a comprehensive analysis of forgetting.
2. Results in a realistic class imbalance scenario to align the experiments with the motivation described in the paper. Specifically, we will present experiments under varying skewness conditions, such as alpha = 0.1 and alpha = 0.5, to evaluate the robustness of the HR approach across diverse data distributions.
3. Other reviewers made other requests, such as including a figure to depict the architecture of our work and more experimental results, as you can find in the other reviewers' comments.

Some of these requests are made by you and some are made by other reviewers. We will do our best to address all.

---

Responses to Questions

Question 3

We understand the reviewer's concern that it is crucial to see these results prior to publication. Accordingly, in ten days, we will prepare our results highlighting the learning-forgetting dynamics and post them in the comment section.

---

Question 4: Class Imbalance in the Experimental Setup

We appreciate the reviewer’s positive comments on our mathematical framework and acknowledge your concern regarding the alignment of the experimental setup with the motivation. While our current experiments use balanced CIFAR-100 datasets, we agree that evaluating HR under imbalanced conditions is essential. To address this:

• We will create imbalanced datasets by modifying CIFAR-100 to introduce uneven class distributions (e.g., 250 samples for Class A, 150 for Class B, and 50 for Class C).

• We will present experiments under varying skewness conditions, such as alpha = 0.1 and alpha = 0.5 (as also requested by reviewer PuPw)

---

The Last Words

We sincerely appreciate your constructive feedback and for raising your score. We are confident that the forthcoming results and clarifications will comprehensively address your concerns. If you find the updates satisfactory, we kindly request you to consider raising your score further to 6, as this would greatly support the acceptance of our work.

However, regardless of your final decision, we are forever grateful for your thoughtful engagement and valuable suggestions, which have significantly enriched and strengthened our study.

We thank the reviewer for their thoughtful feedback and the opportunity to improve our work. Below, we address each comment in detail:

---

Motivation of the Study

We do not agree with the reviewer on "the motivation of this study appears somewhat outdated" because the challenges of local and global forgetting in federated class-incremental learning (FCIL), first introduced by Dong et al. in 2022 [Dong], remain significant and unresolved. To understand and address these challenges, we have made the following contributions:

1. A novel mathematical framework to formalize local and global forgetting, providing a theoretical foundation for addressing these challenges.
2. The introduction of a new method, the hybrid replay method, which mitigates both forgetting types in a memory-efficient manner.

Note that while the work in [1] addresses class imbalance within clients, it does not provide a formal mathematical framework or focus on memory-efficient methods for mitigating forgetting. To make the distinction clear in the revised paper, we will:

• Revise the introduction and related work sections to highlight these distinctions explicitly.
• Add comparative simulation results with [1], focusing on memory and computational efficiency, to strengthen the empirical validation of our contributions.

---

Outline of the Specific Problem

Note that to clarify the mechanisms causing local forgetting and global forgetting in Fig. 1, we have mathematically formulated the FCIL optimization problem in Eqs. 7 & 8 and formalized how the two problems of local and global forgetting occur in the proceeding paragraph of Eqs. 7 & 8. To better clarify the mechanisms causing local and global forgetting in Figure 1, we will:

• Add explicit references to Eqs. 7 & 8 in the Fig. 1 caption.

---

Comparative Methods

Our paper includes representative baselines for both exemplar-based and model-based replay methods. However, we would appreciate further clarification from the reviewer regarding which comparative methods they feel are missing. If recent methods like [1] are the focus of this concern, we will include additional experiments comparing our approach to theirs, as space permits.

---

Visualized Experimental Results

We agree that including visualizations and additional metrics, such as Backward Transfer (BWT), will strengthen our results. In the revised paper, we will:

• Include visualizations showing accuracy trends on old tasks after completing each new task.
• Incorporate BWT calculations to provide a comprehensive analysis of forgetting.

We have these results prepared and will include them in the available 2 pages of the main paper.

---

Clarifications on Eqs. 3 & 4

To clarify:

• In Eq. 3, $i$ and $j$ are indices for classes $i$ and $j$.
• In Eq. 4, $u_{i_k, j_l}$ represents the loss function of the binary classifier for classes $k$ and $l$ in tasks $i$ and $j$.

These clarifications will be included in the revised manuscript.

---

Simulation Setup

Our simulations are conducted in a Federated Class-Incremental Learning (FCIL) setting. When clients are trained on task $h$, they generate a small number of synthetic data samples from other clients for all previous tasks (1 to $h-1$) but do not access new tasks ($h$) from other clients.

This has been discussed in the two last paragraphs of Section 4.

---

Sample Replay

Eqs. 10 & 11 focus on the global alignment of centroids for new classes in new tasks, not on sample replay. As described in lines 6–8 of Algorithm 1:

1. Clients compute embeddings for new classes and send unaligned centroids to the server.
2. The server aligns these centroids via Eqs. 10 & 11 and sends them back to the clients.
3. Clients use these aligned centroids to train on new tasks, ensuring minimal overlap with existing classes.

Sample replay is performed as described in lines 3–9 of Algorithm 2.

---

Conclusion

We appreciate the reviewer’s comments and are committed to addressing these points in the revised version. By incorporating the proposed clarifications, additional experiments, and visualizations, we aim to enhance the clarity and impact of our work. We believe our contributions are valuable to the FCIL community and kindly request the reviewer to reconsider their initial score in light of these improvements.

---

References

[Dong] Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated Class-Incremental Learning. CVPR 2022.
[1] Yavuz Faruk Bakman, Duygu Nur Yaldiz, Yahya H. Ezzeldin, Salman Avestimehr. Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning. ICLR 2024.