Same Accuracy, Twice As Fast: Continual Learning Surpasses Retraining From Scratch

Thank you for taking your time to review our paper. We believe we have slightly different definitions of what is continual learning, which we clarify below. Thank you for acknowledging that this is indeed an important problem to study.

About the definition of continual learning. Definitions are a hard thing to agree on. We interpret continual learning as learning from any kind of data that is not i.i.d. Often, research additionally imposes a memory constraint, which then leads to catastrophic forgetting when data is not i.i.d. When defining continual learning as learning from non-i.i.d. data, this setting is perfectly interpretable as continual learning. However, if it is necessary to avoid confusion with other works, it is possible to change continual learning to ‘continuous’ learning in our paper, with extra clarification on the (dis)similarities with continual learning in the main text.

On the “first step methods”. All the works that inspired study different subproblems. We are the first to connect them together and show that in this scenario, together they form a practically usable algorithm. It is e.g. not self explanatory that the result of Ash and Adams translate to this setting, as they only test benchmarks where both old and new data are from the same distribution.

Generalizability. We have tested many datasets and experiments in Tables 2a and 2b, Figure 6 and 7 and additional hyperparameters settings in the Appendix. None of them showed different results. It is always possible to include another dataset, yet we found the selection we have made relevant to test various aspects of our proposed method.

Thank you for noticing the cut off sentence and the TODO in one of the other comments, that was an oversight from our side. The references are added in the appropriate place now and the sentence is completed.

Thank you for reviewing our paper and taking the time to give feedback. Your comments have helped to improve the paper further. Below, we hope to address some of the additional concerns you had.

CL-methods. Methods like SLCA are interesting and strong methods when memory is the core limitation, but they do not reach from scratch accuracy and require additional computation compared to from-scratch learning, see e.g. Prabhu et al. [a], who showed that many traditional continual learning methods cannot compete with simple rehearsal when considering the computational cost.

Is it continual learning? Continual learning is about learning from data that is not i.i.d. distributed. It is true that often a memory restriction is imposed, but a part of this work wants to argue that settings with no restrictions on memory but restrictions on computational complexity may be interesting too. However, if it is necessary to avoid confusion with other works, it is possible to change the term continual learning to ‘continuous’ learning, with extra clarification on the (dis)similarities in the main text.

Different results in Table 1 and 2. You are correct about the inconsistencies. The results are correct, but it is a little confusing, so thanks for catching this. They are different because in Table 2(b), the methods are compared with scratch+L2-init, rather than bare scratch in Table 1. Scratch+L2-init gave approximately the best results ($L_{100}$) for the scratch model, so we chose to compare to that baseline in Table 2b. We will update the text to make this clearer. Note that ‘25%’ refers to the length of the training, i.e. the cosine scheduler is reduced to ¼-th of the length of the from-scratch scheduler.

[a] Prabhu, A., Cai, Z., Dokania, P., Torr, P., Koltun, V., & Sener, O. (2023). Online continual learning without the storage constraint. arXiv preprint arXiv:2305.09253.

Thanks you for your time and your comments. We hope that below, we can clarify some of the comments you made and the issues you encountered.

More benchmarks. We have tested our method on many different benchmarks and domain shifts, as is explicitly tested in Section 4.3 and Figure 7. Tables 2a and 2b contain results on many different datasets and benchmarks. The proposed problems and architectures would further broaden our claims, but given the many experiments already included in the paper, we did not find it feasible to include even more experiments (there are always more experiments we can think of).

Continual learning methods. It is true that other CL methods could be included, but they never reach the same accuracy as a model that is trained from scratch, even when they have full memory. Furthermore, many continual learning methods do add computational complexity to combat forgetting. If they do not reach the same accuracy, their $L_{100}$ would be undefined, but if they do, they would likely require a higher computational cost than we do. We are the first to propose a baseline in this setting, so it is hard to directly compare to other methods.

New and old data. We will include results on new and old data in the appendix. It does give insights, but since in the end a combined result on both old and new data is the goal, we reported the combined performance.

We would like to thank you for your time and comments. They helped use thinking about the problem a little deeper. Thanks for appreciating our work.

L2+naive. L2 in Figure 3 is L2+naive. For scratch training with either L2-init or L2 did not make a large difference. To not overload the figure we chose to not include L2+scratch in the same figure. We will add it to the appendix and comment on this in the main paper. Clarification of repeated task experiment. Figure 6 is in essence a repetition of 10 figures like Fig. 2-5. Each line between $T_{i-1}$ and $T_{i}$ shows the training on all old and new data available in that task. The only difference is that the evaluation is on all classes of CIFAR100, where the future ones evidently have zero accuracy. To improve the clarity, we will add an explanation to the figure caption.

Typo. Thanks for catching the typo in Equation 6, this will be updated.

Why batch composition may work. The idea of the batch composition is that the easiest data does not need to be repeated as often as the other data, as the continuous data already knows this data well, in contrast to the from scratch model, which has never seen this easy data. The hardest data was never learned by this model, so it is likely not going to contribute a lot of information when it is rehearsed. The from scratch model has not learned anything yet, so it may benefit more from seeing diverse (easy and hard) samples than the continuous model does. Concerning the speed-up there is indeed only a small difference when init+reg is used, but it does allow a higher final accuracy than without it.

Insights. The reason we propose these four ‘tricks’ is because they act on the four main components of any optimizer. With them, we show that every aspect of the optimization can provide potential improvements. We approach this problem from the optimization perspective, because fundamentally, only the initialization is different in the continual case, which largely impacts the optimization trajectory. There may be more and different ways to continue with this problem, but this way, we sought to provide a principled way of tackling this problem.

We would like to thank the reviewer for their time and feedback. We hope that below, we can clarify some of the questions you had.

About the setting. The idea of this paper is to specifically study a scenario where all data is available. This may not be a common setting, but as we explain in the introduction, and confirmed by other papers ([a-c]), it is a very relevant setting. Especially in industry applications, data is often stored and available. Parameter efficient finetuning is a useful approach without old data available, but it does not allow to reach the same accuracy as a model that is trained from scratch on both old and new data. Even outside a continual learning setting, PEFT methods do not allow to reach high accuracy when pretraining and downstream data are evaluated as one large dataset. It would be an interesting future direction to incorporate ideas of those algorithms further into our work, but there first needs to be a basis that allows continuously trained models to reach the same accuracy as a model that is trained from scratch, which we provide.

FLOPs vs. `computational time’. We use computational time measured by the number of iterations, as it is a good proxy to measure e.g. FLOPs or MACS as long as batch sizes and model architectures are consistent across compared algorithms. The number of iterations is directly proportional to FLOPs, hence reducing iterations will reduce the FLOPs.

What if no old data is available? We specifically study continual learning where all data is available. There are excellent works (e.g. the ones that you cite) that study scenarios where there is no (or little) old data available. Our method is not intended to be used in such scenarios, similar to how prompt based methods would not be used when all old and new data is available.

Results with a balanced task split are in Table 2b and reach comparable results.

Rapid domain shifts. Figure 7 explicitly studies domain shifts and Figure 6 includes many small tasks.

Different benchmark in Figure 6?. Figure 6 explicitly studies the case where many tasks are added over time, hence it cannot use the same 70+30 benchmark. To add many tasks, the first task is reduced to 50 classes here, such that 10 tasks of 5 classes can be added.

Adam. We tried to test as many hyperparameters as possible, many of which are reported in the ablations. Experiments are mostly performed with Adam, see the Appendix. We did not find configurations where the results were significantly different than the ones reported.

[a] Prabhu, A., Al Kader Hammoud, H. A., Dokania, P. K., Torr, P. H., Lim, S. N., Ghanem, B., & Bibi, A. (2023). Computationally budgeted continual learning: What does matter?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3698-3707).

[b] Verwimp, E., Aljundi, R., Ben-David, S., Bethge, M., Cossu, A., Gepperth, A., ... & van de Ven, G. M. Continual Learning: Applications and the Road Forward. Transactions on Machine Learning Research.

[c] Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., ... & Wang, H. (2024). Continual learning of large language models: A comprehensive survey. arXiv preprint arXiv:2404.16789. (Section 6.3)