Bridging the gap between offline and online continual learning

We appreciate the reviewer constructive feedback and suggestions. We address the concerns and questions as below:

W1. Why we argue to use the minimum short-term memory instead of zero short memory

the analysis of Eq (6) seem to show that "a minimum amount of short term memory" should be used. In that case, why are you using short term memory at all if the analysis concludes that you should not use it?

Great question. The reason is that $M_{short}=0$ leads to a significantly different procedure with non-zero short memory. More specifically, for any non-zero short-term memory, no matter how small it is, there is a consistent training procedure that spans online and offline CL: each incoming batch will be trained by the model for $E$ iterations before it is discarded. Due to the continual learning nature, this procedure holds regardless of the exact value of $M_{short}$. In contrast, if there is no short-term memory, the incoming data is directly saved to long-term memory based on some selection mechanism without being trained. This means some incoming data are never seen by the model. We believe this represents a fundamentally distinct training approach from the current online and offline CL practices.

Our theoretical analysis only concerns the generalization bound and does not cover training procedures. Since the performance is affected by both factors, we apply our theoretical results only to provide a possible explanation for algorithm performance with similar training procedures. Hence, we interpret our theoretical results only in the context of long-short term frameworks, where short-term memory lies in $[DataStreamBatchSize, TaskSize]$.

W2.1 Using zero short memory as a baseline.

Thanks for this advice. We added a paragraph in the paper to discuss the zero short memory setting and its relation to Gdumb. With zero short term memory, the generalization bound analysis does not change much. Comparing $M_{short}=0$ and $M_{short}=DataStreamBatchSize$, with a small batch size, the generalization bounds are very similar, with the former leading to a slightly lower bound. However, the difference in training procedures must also be considered along with the generalization bounds. As shown in the table below, the performance of zero short memory is slightly worse than minimum short memory (ER 2.4% $\downarrow$, SCR 3.8% $\downarrow$), but still much better than $M_{short}=$ task size (offline CL). We believe this is the combined result of the generalization bounds and the training procedures.

W2.2: zero short memory is equal to Gdumb. Novelty is not here anymore

We would like to clarify that Gdumb and the zero-short memory setting (referred as zeroShort) investigated here have some fundamental differences and send different messages:

1. Every time the model is updated, the model is re-trained from scratch in Gdumb. On the contrary, zeroShort is a continual learner which updates model continually with E gradient steps when one incoming batch arrives.
2. Gdumb maintains a class-balanced memory while zeroShort uses reservoir sampling.
3. When compared with CL algorithms, Gdumb employs a separate learning design (cutmix augmentation, training epochs, retrain from scratch). Gdumb outperforms online CL in Table 3,4 in Gdumb paper [1]. Our experiment removes the compounding effect of learning algorithms and sends a different message: online-CL is a bit better than zeroShort-CL.
4. Gdumb reported worse results than offline CL algorithms (Table 5, B2 in [1]). Only the exemplar memory is aligned in the Gdumb experiments. We align the total memory budget and report zeroShort-CL is much better than offline CL. Thus, we believe Gdumb is undervalued in its comparison with offline CL.

W3. results with more splits and different total memory budget

The results of Split-Cifar100 with 50, 20 and 10 are shown in Fig 3 c. We use task size as the labels for x axis, i.e. 1.0 k, 2.5k, 5k. These experiments uses 7k total memory.

Besides using 7k memory budget, Fig 3b experiments with other total memory budgets (5.5k, 6k, 7k, 10k )

Q1. yes, we will release the code

Q2. Hope the response above has addressed this question. Please inform us if you have any further inquiries.

Thank you for taking the time to review our paper and provide feedback. We address your concerns and questions point by point as follows:

W1. Figures

(1) It seems that Figure 1 and Figure 1 are the same figure. You need to fix this problem.

Figure 1 and Figure 5 are not the same figure. Figure 1 is the results on Split-CIFAR100 and Figure 5 are the results on Split-Mini-ImageNet

W2. Experiments beyond CIFAR

The experiment for supporting the emprical observation is not very valid. You should consider more complex datasets (e.g., ImageNet) or class incremental learning in NLP tasks (e.g., intent classfication). Only conducting experiments on the CIFAR dataset is not enough.

We would like to clarify that apart from Split-CIFAR100, we also conducted experiments in other widely used continual learning benchmarks, split-mini-imagenet and CORE50 (see Table 1). The appendix also includes the results of ImageNet in Table 4. Our finding is consistent in all these experiments with different image types and dataset size.

W3.1 The implication of our finding

I don't know what benefit can we learn from this observation?

This work poses several questions into the current practice of the continual learning field:

We revised the introduction and conclusion of the paper to clarify the implications of our work.

W3.2 Can the finding in this paper be used to reduce forgetting?

Yes. The stability-plasticity analysis in Figure 1(b) highlights the effect of the storage allocation policy on forgetting: using a small storage allocation parameter leads to better stability (less forgetting). One of our primary messages is that the storage allocation policy is a key factor concerning CL performance but is overlooked in the current practice of the field. Our experiment results on four CL algorithms suggest that tuning the storage allocation parameter leads to a bigger performance boost than specific algorithm design (Table 1).

However, we strongly disagree that this means experiments with aligned memory are not worth performing. 1) aligned memory allows systematic comparison of the usefulness of recent samples and past samples

My objection is the conclusions drawn from the experiments with aligned memory. I did not say that any experiments are not worth performing -- my job is to simply evaluate the claims made based on the experiments. My claim still is that comparing aligned and offline memory isn't fair.

aligning memory budget provides a vital lens for studying efficient storage distribution constraints and representation mechanisms for CL. More specifically, 1) aligned memory allows systematic comparison of the usefulness of recent samples and past samples, and 2) it reveals the effect of the storage allocation mechanism for balancing samples from past and present on the performance of continual learning.

I specifically request the authors to elaborate why they think aligned Online CL vs offline CL is a fair comparison. I'm afraid these statements are too vague for me to agree or disagree, I would prefer a more concrete claim which I can either support or disagree with and give reasons why.

Does this statement always hold, regardless of data stream size, memory size, or CL algorithm (ER, SCR, iCaRL,DER etc)? And if so, why?

The statement was intended to simply highlight the differences in distributional constraints of C and M. The distributional constraint remain the same regardless of the algorithm applied.

I would add this effect of C being markedly less useful compared to M is quantified in the experiments provided by the authors in the rebuttals. Seeing the (offline) vs (online-align-compute) comparison -- 5000 $C_{batch}$ samples contribute to less than 1.5% of the total accuracy while 2000 $M_{online}$ contributes 30-38% accuracy, when computation is equalized and not implicitly constraining the memory itself!

I still think the additional experiments support my hypothesis and list why, but am open to changing my mind. Currently as it stands the authors acknowledged my concerns in W1 but did not address it. Once this issue overarching all experiments is clarified, I can discuss specific issues about the storage allocation policies etc.

Hi, thank you for engaging constructively despite my negative review. Hope these experiments and my review clarifies why online CL outperforming offline CL would be a tough sell. I find the result provided by the authors quite interesting. I am quickly replying to see if I can provide straws in the wind. I shall post my response to the broader rebuttal soon.

• In scenarios with limited computation, computational constraints implicitly constrain memory (i.e. despite having access to more samples, an algorithm cannot really effectively utilize them). While this is highlighted in previous works exploring computational constraints in continual learning [1], I haven't seen an explicit analysis of this effect.
• Hence, I think that the the experiment between (online-no-align) and (online-align-memory) is quite nice as it in my view can help better understand the limits imposed by computation on memory use in CL methods more explicitly, by allowing access to a larger $C_{batch}$ compared to $C_{task}$.
• In contrast, (online-align-compute) and (offline) result is better understood, the relative uselessness of $C_{batch}$ samples in contrast to memory $M_{online}$ was pointed out by GDumb [2]. In my understanding, they further removed even $C_{task}$ and only studied training on $M_{online}$ vs $M_{online}+C_{batch}$ and realized that the performance does not degrade very much. Although they did not perform such an clean, ablated experiment.

I maybe wrong, please feel free to correct this if so.

[1] Computationally Budgeted Continual Learning: What Does Matter?, CVPR23

[2] GDumb: A Simple Approach that Questions Our Progress in Continual Learning, ECCV20

We greatly appreciate the reviewer providing their main concern W1 in a partial review to engage in further discussion. For expediency, we focus on replying to this concern here and will upload our replies to other concerns and the revised paper later.

The effect of aligning computation only

In our submission, we only presented the results of aligning both memory consumption and the number of iterations. Prompted by W1, we have now included an ablation study with results obtained by aligning the number of iterations or memory consumption only, summarized in the table below. Compared with no alignment of the number of iterations, regardless of whether memory consumption is aligned or not, the performance of online CL is significantly increased by moving from 1 to 50 iterations. More interestingly, when aligning the number of iterations only, the performance of online CL becomes very close to the performance of offline CL, despite the fact it uses a much smaller memory budget: online-align-compute: 2064 samples (2k+64) and offline: 7000 samples (2k+5k)!

Thanks for pointing out our error in calculating the memory cost of online CL. We updated it to exemplar memory + batch size.

Saving the entire task data is the common practice of offline CL, not a novel restriction of this work. Based on our knowledge, all the offline CL algorithm in the table below uses the entire task data.

I believe this point has been discussed quite in depth. The crux of disagreement between us remains that authors state that offline continual learning algorithms should not be allowed to adapt to the change in the problem constraint introduced here, while I think they should be allowed. Specifically, the claim that they cannot outperform online continual learning is a wrong claim in my view, because the paper intentionally did not adapt their sampling strategy to new aligned memory constraint.

As a minor point, the DER paper itself states how to use their algorithm in an online fashion in appendix F.3 (also in Soutif-Cormerais etal noted in above rebuttal), iCARL and EWC++ (RWak) has been used in many online learning comparisons including Mai et. al. also noted in the above rebuttal. The dichotomy between algorithms is not as black and white as the authors claim, as one can use either of the online/offline algorithms in the other setting.

Can the authors address the weakness 2? I still think Proof of Theorem 1 is incomplete, and would like the authors to address this before rebuttal deadline.

We really appreciate the reviewer actively engaging in the discussion. However, we believe there is a misunderstanding of our problem setting.

1.Problem setting: data stream with batch size of B

Core Difference: The core difference is between sample size arriving, and not algorithms. Online continual learning algorithms gets access sequentially to 64 new samples of the new classes while Offline continual learning algorithms can potentially access all the 5000 samples of the new classes at once if they wish to.

To clarify, following [1] we assume online and offline CL are both dealing with the same problem setting (hence we compare them), i.e. general continual learning:"an infinite stream of training data where at each time step, the system receives a (number of) new sample(s) drawn non i.i.d from a current distribution D_t that could itself experience sudden or gradual changes." [1]

As argued in [1][2] offline CL is a relaxation of this general CL setting with the additional knowledge of task boundaries to perform multiple passes in task data. ( "Knowledge of the task boundaries is required (i.e. when tasks switch) and allows for multiple passes over large batches of training data. Hence, it resembles a relaxation of the desired continual learning system that is more likely to be encountered in practice"[1])

As shown in definitions 1 and 2 in our paper, we assume that task size T is the sequence of data with a static distribution in a data stream and batch size B is the sample size arriving in the data streams (B<<T). Under this problem setting, we compare two ways of continual model update:

1. update the model as long as each incoming batch data comes: model update frequency = batch size B
2. wait until the task switch to collect a large batch of data (with the same distribution) before updating the model: model update frequency = task size T

We find that 1) leads to better performance given aligned memory and iterations. The Rebuttal table below shows a summary of CL algorithms following these two ways. Thus, we believe achieving a deeper understanding of the efficiency of these two is important and is the focus of this work.

If the batch size is equal to task size, B=T(data stream arrives in large batches), then online and offline CL are the same thing. However, we believe there are some real-world scenarios the sample size of arrival is not equal to static distribution data size (B<<T).

[1]De Lange, Matthias, et al. "A continual learning survey: Defying forgetting in classification tasks." IEEE transactions on pattern analysis and machine intelligence 44.7 (2021): 3366-3385. [2] Buzzega, Pietro, et al. "Dark experience for general continual learning: a strong, simple baseline." Advances in neural information processing systems 33 (2020): 15920-15930.

2. store all task data

The paper restricts offline continual learning to store all samples as a part of , consuming 5000 out of 7000 samples on the last set of classes, while allowing online continual learning to freely store samples across past data

Saving the entire task data is the common practice of offline CL, not a novel restriction of this work. Based on our knowledge, all the offline CL algorithm in the table below uses the entire task data.

We hope our results (aligning the computation only, aligning memory & computation together) pose questions to this current practice of the offline CL field: do you need to store all the task data when learning continual learning from a data stream.

Rebuttal Table 1.

3. Data privacy

Privacy of past data-- This is the core argument used for differentiating them

We disagree the consideration of privacy differentiates task data or past data. The privacy of the current task data perhaps should also be considered. As offline CL needs to store the task data which may raise some privacy and data safety issue.

4.Storage cost of samples

The rebuttal makes a cost argument to equalize the memory-- I do not understand why restricting storage of 100-250MB on hard disk is the bottleneck, while the algorithms presented require 2-10GB of GPU VRAM to run.

We agree that on the current CL benchmark, the data samples consume less space than the model. The cost of the model has been discussed in other work. This work targets the current practice of focusing on the storage budget of exemplar samples. Instead, we argue that the storage of new samples should be considered together with the exemplar samples.

Question: Given say a total memory of n examples (say n=7000) in a class-incremental setup-- what performs better? Online continual learning or offline continual learning?

Core Difference: The core difference is between sample size arriving, and not algorithms. Online continual learning algorithms gets access sequentially to 64 new samples of the new classes while Offline continual learning algorithms can potentially access all the 5000 samples of the new classes at once if they wish to.

The paper restricts offline continual learning to store all $C_{batch}$ samples as a part of $n$, consuming 5000 out of 7000 samples on the last set of classes, while allowing online continual learning to freely store $~7000$ samples across past data.

Ideal Sampling: The ideal sampling (given within-class are i.i.d.) is to divide the 7000 sized buffer equally among past classes (achieved by reservoir sampling and class-balanced sampling both as no class imbalance)-- making sampling for online CL better than offline CL, hence resulting in better experimental results.

But why is this unfair?

Offline CL can sample the same too!: As I noted in (informal claim) in W1, offline continual learning algorithms, given this constraint, can simply choose to delete the vast amount of current samples instead of past samples with the same reservoir sampling or class-balanced sampling-- thereby performing equal to or better than any corresponding online continual learning algorithm.

However, I raise my score to 3 as I understand the paper argues memory-constrained operational space should include all exemplars. However, my argument is that given this new constraint, offline CL algorithms should (and can) simply not store the vast amounts of new data.

Why the difference in the alignment of past storage and current storage between works?

1. Privacy of past data-- This is the core argument used for differentiating them, by which works argue $C_{batch}$ and $M_{offline/online}$ have different constraints.
2. The rebuttal makes a cost argument to equalize the memory-- I do not understand why restricting storage of 100-250MB on hard disk is the bottleneck, while the algorithms presented require 2-10GB of GPU VRAM to run.

We appreciate the reviewer for their time in assessing this paper and engaging in this long discussion.

1.Adapting offline CL to maintain a class-balanced memory by deleting samples, is this Gdumb?

If we are understanding correctly, the reviewer suggests that in our comparison, offline CL should be adapted to maintain a class-balanced memory by deleting some of task samples. We are not sure if the reviewer are referring to Gdumb. Gdumb maintains a class-balanced memory.

Generally, how to adapt offline CL algorithms to achieve ideal sampling, does this mean deleting samples without training? If so, whether deleting samples with training will have a negative impact on the performance. We believe the answer to these questions is not straightforward, deserves detailed investigation and is not the focus of this work.

However, we present some relevant results in the revised paper (section 6) on what is the effect of removing samples without training. We compare zero short memory vs. small short memory. In zero short memory case (like Gdumb), each incoming batch is directly stored in the long-term memory by reservoir sampling. This means some samples are never seen by the model. Compared to $M_{short}=Batch size$, zero short memory leads to worse performance, perhaps due to some samples are never used for the training. More discussion on this topic can be found in our reply to Reviewer LdF8 and in the revised paper.

Split-CiFAR100

Mini-ImageNet

2. DER++, icarl, ewc, Rwalk have been tested in online setting but achieve much worse performance than offline setting

In theory, all the offline CL techniques should be applicable to online settings with zero or slight modification(changing herding to reserivor sampling). However, when applied in online manner, the same algorithms often report worse results. Offline CL survey papers (Masana et al.) reported iCaRL 33.5% with a 2k exemplar memory. Online CL survey papers (Mai et al.) reported iCaRL 19.2% with a 5k exemplar memory. This phenomenon leads to the misbelief that offline CL techniques (especially with knowledge distillation design) may not achieve good performance without multiple epoch training (this concern is explicitly expressed in DER paper). We show this phenomenon is because of the usage of memory and compute resources, not the online setting (or single epoch constraint) itself.

3.generalization bound

We agree that the generalization bound may not be tight. However, it should be note that this is not because of the missing proof of this paper, but because theorem 8 in Mansour et al. (2009) (Eq 8 in our paper) is not tight. Based on our knowledge, how to derive tight bounds for transfer learning and how to measure the quality of the bound is still a challenge due to the distribution difference in the source and target domain. Achieving equality in theorem 8 already requires h*(Q1) =h*(Q2) , which may not be achieved in practice. However, this upper bound is meaningful when h*(Q1) and h*(Q2) are close, and this is usually the assumption of transfer learning. In our work, Q1 and Q2 are the true distribution of the data stream and the expected distribution of the memory data. In the rehearsal-based methods, the memory data includes both new and past samples, we assume h*(Q1) and h*(Q2) are also reasonably close. As verified in Fig 3, this generalization bound seems to provide meaningful insights into the influence of data stream size, task size and memory budget size and the prediction that online outperforms semi-online cases in Fig 2b.

Thank you for taking the time to review our paper and provide feedback. We address your concerns and questions point by point as follows:

W1. multiple iterations in online CL

Online CL uses many more iterations at each timestep to match offline CL’s compute overhead. Thus online CL becomes ill-suited for many real-world applications e.g., on-device learning where fast adaptation / speed is critical. Online CL’s performance gains are mostly attributable to higher compute (more SGD updates) and memory usage.

To clarify, even with much smaller compute overhead, online CL with 10 iterations (48.4% $\pm$ 0.5) already leads to better performance than offline CL with 100 epochs (40.1% $\pm$ 0.8) as shown in Fig 1a. We analyze the influence of iteration number on stability and plasticity. With iteration <10, increasing iteration leads to better stability and plasticity at the same time and the performance boost is very significant. With iteration >10, increasing interaction starts to hurt stability (more forgetting) and the benefit on the overall CL accuracy is limited. One implication of our finding is, instead of always using a single iteration, one can adjust the iteration number based on the speed-accuracy requirements of the specific application. In addition, it should be noted that even with multiple iterations online CL still supports fast adaptation in the sense it updates the model as long as it receives an incoming batch. In contrast, offline CL waited until the collection of a task dataset to update the model. In an interactive application, online CL can make the system more responsive to user feedback.

W2. Definition of online CL: batch by batch vs. sample by sample

Our problem setting is that the incoming batch in the data streams comes with a batch size of B. This setting is commonly used in continual learning research works [1][2]. Under this problem setting, we compare two ways of continual model update:

1. update the model as long as each incoming batch data comes: model update frequency = batch size B
2. wait and collect a large batch of data before updating the model: model update frequency = task size T

We find that 1) leads to better performance given aligned memory and iterations. The table below shows a summary of CL algorithms following these two ways. Thus, we believe achieving a deeper understanding of the efficiency of these two is important and is the focus of this work. We agree that the sample-by-sample data stream is an interesting case, but is out of the scope of this work.

[1] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification:An empirical survey. Neurocomputing, 469:28–51,2022.

[2] Soutif-Cormerais, Albin, et al. "A Comprehensive Empirical Evaluation on Online Continual Learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

W3. I am not convinced that single pass causes underfitting.

[3] discusses the use of multiple iterations to deal with the underfitting-overfitting dilemma of online CL (when trained from scratch). Whether or not single pass causes underfitting is not the topic of this work. We simply show that with the same memory, compute and CL algorithm, updating the model in an online manner leads to better performance than offline.

[3] Zhang, Yaqian, et al. "A simple but strong baseline for online continual learning: Repeated augmented rehearsal." Advances in Neural Information Processing Systems 35 (2022): 14771-14783.

W4. This work does not propose any specific mechanisms to mitigate catastrophic forgetting, enhancing stability and plasticity and underfitting issues (when present due to design choices). It lacks sufficient scientific contributions.

We did not propose new algorithms. Instead, we study a fundamental problem concerning different CL algorithms, the usefulness of past samples vs. new samples. We highlight an overlooked but important factor concerning CL performance: the storage allocation policy$\alpha=M_{short}/M$. Online and offline CL are just variants of the same learning algorithm that only differ in the setting of the storage allocation parameter (Online CL: $\alpha=B/M$ and offline CL $\alpha=TaskSize/M$). Tuning the storage allocation parameter $\alpha$ leads to a bigger performance boost (better stability-plasticty tradeoff) than the specific algorithm design (Table 1). The effect of this parameter on forgetting (i.e.-stability) is presented in Fig 1b and Fig 5b.