Federated Continual Learning Goes Online: Uncertainty-Aware Memory Management for Vision Tasks and Beyond

Thank you Reviewer o8MC for the valuable feedback. Below are our responses to your questions.

1. Marginal improvements on CRC-Tissue and KC-Cell

In comparison with standard class-balanced datasets of natural images, these two datasets represent a more challenging scenario. However, it is important to notice how the proposed approach has a substantial gain compared to state-of-the-art FCL methods (c.f. Table 5). As explained in response to Reviewer 2eAg, the lower forgetting of FedCIL on KC-Cell (and CIFAR100) is due to the very poor predictive performance in terms of accuracy. Being CF directly related to the peak accuracy for each task, the resulting value will be low if the peak accuracies are low.

2. Comparison between Bregman Information and probabilistic methods in FCL

Could you please specify the works you are referring to? We are happy to include any baselines from this area that we may have missed.

3. Limited comparison with previous CL and FL methods

Our contribution, investigating online FCL, is rooted in the federated continual learning (FCL) literature. With our submission, we build on the growing body of literature that investigates FCL and, itself drawing from both CL and FL literature, has established a new state-of-the-art. We therefore focus our empirical investigation - also to allow for continuity and and fair comparisons - on methods benchmarked in the FCL literature. In addition, we provide an idea of how standard CL and FL baselines would work by including ER and CBR (as a reference for CL methods) and different FL strategies (FedAvg and FedProx). This is in line with previous FCL works (e.g., FedCIL [1] and MFCL [2]). We argue that broader comparisons with other CL methods that have not shown any promise in FCL (and would typically require substantial adjustments to work in the online and federated setting) are out of the scope of this paper.

[1] Qi, Daiqing, Handong Zhao, and Sheng Li. "Better generative replay for continual federated learning." ICLR 2023.

[2] Babakniya, Sara, et al. "A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks." Advances in Neural Information Processing Systems 36 (2024).

[3] Soutif-Cormerais, Albin, et al. "A comprehensive empirical evaluation on online continual learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Thank you Reviewer BinS for your thorough and helpful comments. Below are our responses to your concerns.

1. Technical novelty is limited

We agree that the technical novelty is limited as we indeed build on the technical contribution from [1] who introduce a general bias-variance decomposition for strictly proper scores, and identify the Bregman Information as the variance term. However, it is widely acknowledged in the machine learning community that the notion of novelty does not originate only from such theoretical innovations. Novelty could also come from proposing new scenarios, shedding light on overlooked aspects, drawing new connections between research areas and applications, and so on. In this spirit, we argue that our realisation that (i) in the current (F)CL literature uncertainty-based memory management relies on measures of aleatoric uncertainty and (ii) that BI can be interpreted as a measure of epistemic uncertainty that can efficiently be estimated via TTA is indeed novel ([1] is a theoretical contribution with only very limited experiments). Furthermore, we would like to emphasize that the novelty of our submission does not only stem from the introduction of TTA-based BI as an uncertainty measure in online FCIL. More specifically, novel aspects of our paper include that we (i) formalise online federated CIL as a practically relevant generalisation of FCIL; (ii) demonstrate that state-of-the-art generative approaches such as MFCL and FedCIL adapted from offline federated CIL to online federated CIL do not work in the online setting and discuss the reasons for the observed performance gap; (iii) show that uncertainty-based approaches consistently outperform generative approaches for this task; (iv) point out that most of the uncertainty metrics mostly capture the irreducible aleatoric uncertainty and hypothesize that it may be more beneficial to focus on the epistemic uncertainty, and (v) introduce the use of the Bregman Information as an alternative method for predictive uncertainty estimation in the context of memory-based methods – which is largely unexplored in both federated and non-federated settings.

2. Clarification on empirical gain between Tables 1-4 and Table 5

A key insight of our work is that state-of-the-art offline FCL baselines (MFCL and FedCIL) and standard FL (FedAvg, weighted FedAvg, and FedProx) are substantially and systematically outperformed by memory-based solutions when moving from the offline FCL setting to the online FCL setting – this is shown in Table 5. The comparison within memory-based approaches (i.e., ER, CBR) and uncertainty-based memory approaches (e.g., EN, RC) shows that BI has a competitive performance, but in comparison to the gain over SOTA FCL methods, the improvement is smaller – shown in Tables 1-4. This resulted in the following novel insights: a) generative-based solutions commonly used in offline FCL are ineffective in learning sufficiently high-quality images in the online FCL setting, where each batch can only be processed once and we don't have access to the whole task dataset multiple times; b) memory-based approaches seem the natural solution for this setting, where BI stands out for its competitive predictive performance and reduced catastrophic forgetting compared to other solutions.

Thank you for clarifying the points I raised. Here are my comments on your response.

1. I do acknowledge the value of the new scenarios and the value of using BI for measure of uncertainty. But as the authors also agree that the technical novelty is quite limited and it is certainly a point of weakness (but not a reason to reject the submission). And any comments on my argument that it is a straightforward application of Gruber & Buettner (2023)?

2. I appreciate the authors' intuitive explanation for the reason. But I was expecting the reasons backed by some empirical results or with some (further) quantitative analysis.

3. Thank you for the additional results. The reason to ask ImgNet-1K is for seeing the performance on larger sized image data with larger number of datapoints. With the TinyImageNet, I can see that the performance of methods in the larger number of datapoints.

4. I respectfully argue that 'a learning model' (I don't know what this means exactly) is 'mainly focusing' on the last accuracy only. If the learning method is sound, it should improves the accuracy throughout the inference process as a whole. It motivates to use metrics such as average accuracy, average anytime accuracy (AAA) [1] and $A_{AUC}$ [2].

[1] Caccia et al., "Online Fast Adaptation and Knowledge Accumulation (OSAKA): a New Approach to Continual Learning", NeurIPS 2020

[2] Koh et al., "Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference", ICLR 2022

5. "its use in the early stage of the training phase on a new task may result in a better sample selection compared to traditional uncertainty-based strategies" Do you have an evidence to claim this?

6. Thank you.

7. CLIB's memory management method [2] is applicable to any task configuration including standard (disjoint) memory-based online CL as the disjoint setup is a special case of blurry setup. So, you can compare your method to it.

Overall, I think that the method is very promising but has some room for improvement. So, I keep my initial rating.

Thank you for the feedback. Here is the response to the remaining concerns:

1. We respectfully disagree that a lack of technical novelty is certainly a point of weakness. A piece of research does not have to be novel and technical to be considered valid. We do rely on [1] in the sense that we use their decomposition for estimating predictive uncertainty. However, although the measure itself is not part of our contribution, we are the first to employ Bregman Information in this context (which is largely unexplored), interpreting it as a measure of epistemic uncertainty and thereby shedding light on its behaviour under different conditions than the original paper. Furthermore, we are the first to hypothesize that common uncertainty scores are not well-suited in this setting given their focus on aleatoric uncertainty. We use the Bregman Information to test this hypothesis.

2. The performance gap between memory-based and generative-based approaches stems from the quality of the samples used for replay. Memory-based approaches use original and informative samples. We do provide empirical evidence that samples generated via MFCL are not informative enough to help the model recall the past information. We show examples of the images generated by MFCL in Appendix A.10. Figure 5 shows empirically that synthetic images generated in the online scenario do not carry enough information about past tasks. This is because, in the online setting, generative models do not have the possibility to retrain the model on the same data multiple times.

3. We absolutely agree and indeed illustrate average accuracy in Figure 3 (the figure shows the accuracy of each task at the end of the learning process). For methods without replay mechanisms (such as the ones using only FedAvg), the model tends to focus only on the last task. In our case (BI), instead, the accuracy is kept uniform across all tasks also at the end of the training - or as you put it, the method is sound as it improves the accuracy throughout the inference process as a whole. This also results in a higher average accuracy (shown in the legend in Figure 3), which is also the metric we employ in all our experiments.

4. To demonstrate the effectiveness of our approach compared to traditional uncertainty-based strategies in the early stage of the training phase, we decided to perform an experiment where we stop updating the memory after a limited number of batches (50). In this way, the memory is populated according to samples encountered in the first part of the learning process. Then, the "frozen" memory is used for replay in the remaining part of the training procedure. Finally, we report the average accuracy and forgetting and compare them with the original values. Following our hypothesis, the memory should consist of different samples when using our approach compared to others and the higher quality of the samples in the memory should be reflected in better (relative) performance than other strategies. Below are the results. From the table, we can see how our solution has competitive performance also in this case, demonstrating its effectiveness already in the early stage of the training phase.

6. As already mentioned, CLIB shows merely comparable results with standard ER in the disjoint setup, with for example an average accuracy of CLIB for CIFAR10 of 77.96% vs. ER 77.47%. This is in line with recent empirical surveys [2] where ER is deemed as a very competitive baseline which often obtains better results than more sophisticated methods. We therefore include ER and CBR as competitive CL baselines to compare with. We respectfully argue that adding CLIB would not add any significant value to the paper.

We would be happy to know in which way we can further improve the paper.

[1] Sebastian Gruber, Florian Buettner. "Uncertainty Estimates of Predictions via a General Bias-Variance Decomposition." Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:11331-11354, 2023.

[2] Soutif-Cormerais, Albin, et al. "A comprehensive empirical evaluation on online continual learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

We thank Reviewer hU4E for the constructive and valuable comments. Below are our responses to your concerns and questions.

1. Lack of advanced memory replay methods

Based on the results of recent empirical surveys [1], all memory-based replay methods perform very similarly to the common Experience Replay (ER). For this reason, we decided to use ER for comparison as it can be seen as the prototypical behaviour of a standard CL approach. We also included CBR to consider the data imbalance problem. In general, we see our contribution, investigating online FCL, as rooted in the federated continual learning (FCL) literature. With our submission, we build on the growing body of literature that investigates FCL and, itself drawing from both CL and FL literature, has established a new state-of-the-art. We therefore focus our empirical investigation - also to allow for continuity and fair comparisons - on methods benchmarked in the FCL literature. In addition, we provide an idea of how standard CL and FL baselines would work by including ER and CBR (as a reference for CL methods) and different FL strategies (FedAvg and FedProx). This is in line with previous FCL works (e.g., FedCIL [2] and MFCL [3]).

2. Comparison with NPCL [4]

We thank the reviewer for pointing out reference [4]. However, although NPCL proposes an uncertainty-based approach for CL, the method is designed for the offline setting. Adaptation to the online setting is non-trivial, since key components of NPCL are global regularisation and task-specific regularisation, both of which require access to past data that is not available in the online setting. In general, we argue that broader comparisons with other CL methods that have not shown any promise in FCL (and would typically require substantial adjustments to work in the online and federated setting, like NPCL) are out of the scope of this paper.

3. Different memory sizes for different datasets

The memory size depends on the number of classes available for a given dataset. For each dataset, we compared different memory sizes (i.e., different number of samples per class in the memory) to understand the effect of memory size on the performance.

4. Incremental improvement of KC-Cell

In comparison with standard class-balanced datasets of natural images, these two datasets represent a more challenging scenario. However, it is important to notice how the proposed approach has a substantial gain compared to state-of-the-art FCL methods (c.f. Table 5). As explained in response to Reviewer 2eAg, the lower forgetting of FedCIL on KC-Cell (and CIFAR100) is due to the very poor predictive performance in terms of accuracy. Being CF directly related to the peak accuracy for each task, the resulting value will be low if the peak accuracies are low.

5. Clarifications about Figure 3

Thanks for pointing this out. The notable performance gap on the last task graphically describes what is the effect of Catastrophic Forgetting (CF) on the learning model. We can see how approaches without replay mechanisms (such as FedAvg) mainly focus on the last task with very good performance while completely forgetting the knowledge of the previous ones. In our case, thanks to the proposed approach, the accuracy across tasks is kept more uniform, leading to better average accuracy and reduced CF.

[1] Qi, Daiqing, Handong Zhao, and Sheng Li. "Better Generative Replay for Continual Federated Learning." The Eleventh International Conference on Learning Representations.

[2] Soutif-Cormerais, Albin, et al. "A comprehensive empirical evaluation on online continual learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Babakniya, Sara, et al. "A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks." Advances in Neural Information Processing Systems 36 (2024).

[4] Jha, Saurav, et al. "NPCL: Neural processes for uncertainty-aware continual learning." Advances in Neural Information Processing Systems 36 (2024).

Thank you Reviewer 2eAg for your detailed and helpful feedback. Below are our responses to your concerns.

1. Predictive uncertainty is highly model-dependant and widely used

It is true that predictive uncertainty is widely used (see also response to Reviewer BinS). However, we would like to emphasize that most of the uncertainty metrics used in the literature capture mainly the irreducible aleatoric uncertainty; in our contribution, we hypothesize that it may be more beneficial to focus on the epistemic uncertainty. For this reason, we introduce the use of the Bregman Information as an alternative method for predictive uncertainty estimation in the context of memory-based methods – which in itself are largely unexplored in both federated and non-federated settings.

2. Use of uncertainty estimator and random sampling for replay is not novel

The use of uncertainty-based memory management has indeed been investigated in CL (e.g., [1]). However, the majority of the uncertainty estimates used in this context mainly capture the aleatoric uncertainty. In contrast, we hypothesize that focusing on epistemic uncertainty could give a benefit for reducing CF while maintaining similar predictive performance. We then build on the theoretical contribution from [2] who introduce a general bias-variance decomposition for strictly proper scores, and identify the Bregman Information as the variance term. We leverage this insight and, interpreting BI as a measure of epistemic uncertainty that is statistically well grounded in a bias-variance decomposition, are the first to suggest to use the Bregman Information for memory management. Although aleatoric and epistemic uncertainty measures appear similar in principle, their behaviour is different in practice (as also graphically explained in Figure 2 of the paper). Finally, random sampling is indeed a standard choice in the literature that we use to sample from the memory for replay. We further discuss novel aspects of our submission beyond the introduction of TTA-based BI for memory management in response to Reviewer BinS.

3. Scalability issue for large datasets

We would like to emphasize that, in the online scenario, the total number of samples in a dataset does not affect the scalability of an algorithm. In fact, since we do not use the whole dataset at once but rather process mini-batches sequentially, the only important factor is the time for processing a mini-batch. An idea of the computational time for the training procedure (in seconds) of different methods is given in Table 12 of the manuscript. Additionally, considering your question, we believe that an important factor that may affect the computational time is the image size. For this, we run an additional experiment to evaluate the effect of the image size on the time for processing a mini-batch. The table below reports the average time for processing a mini-batch for our methods and MFCL.

4. Experiments with increased number of tasks (T >= 20)

We ablate the effect of the number of tasks on the evaluation metrics on CIFAR100. From the results, the increasing number of tasks seems to mainly influence catastrophic forgetting in all considered cases. The results look reasonable since, by increasing the number of tasks, it is more difficult to retain knowledge of the initial tasks. The same trend is also observed for baselines, with BI clearly and consistently outperforming ER and MFCL, with especially large gains over ER for large task numbers.