Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting

We thank the reviewer for raising a very important point, that is “the two main findings are the most basic assumptions of continual learning” and take it as a chance to also address: “The major weakness of this work is that the main findings are not new. “, re-emphasizing the novelty of our work.

While we agree with the reviewer’s forwarded assumptions on how forgetting and learning should interact in the continually learning model, we would like to point out that these assumptions were, to the best of our knowledge, never directly demonstrated to hold. In this work we address the claim, which has repeatedly been made in the recent literature, that forgetting in representations is "not catastrophic". An innate robustness of representation or representation learning methods to forgetting would be a promising direction for continual learning research. Earlier work in the field might not have sufficiently observed this robustness because it was mostly concerned with investigating forgetting through the lens of measuring final (average) performance of the continual learner. In contrast, recent works with explicit focus on the accumulated representation in continual learning partially do find such a robustness. The contribution of our work is to clarify and settle this disagreement. We do so with a novel perspective and detailed analysis that first shows indisputably that representations do forget catastrophically and secondly illustrate the direct connection to impaired continual knowledge accumulation. Despite reasonable but intuitive arguments derived from earlier works, to the best of our knowledge, such an analysis was yet missing from literature.

“The authors focused exclusively on the last activation of a network for classification tasks”.
It is true that in this work we did focus on the representation resulting from the last activation of the network. And more fine grained investigation of the representations at different depths is desirable future work. However, for the points to make in this work we felt it to be out of scope. We hope that future empirical analysis will successively reveal these traits.

“Solely on offline continual learning scenarios.”
We stand by our analysis repeating the same argument as for the previous point. Certainly, it can be quite interesting to similarly consider an online perspective, however, to prove the points of our work we deemed the upper-bound analysis of sequential offline training sufficient.

We thank the reviewer for sharing our interest in this research direction and appreciating the novelty we provide.

“How does the model achieve the knowledge of task t before learning task t and why are you confirmed that this knowledge achievement is valid?”.
To answer briefly: In our selected settings we always observe natural images which share quite a large portion of their distribution which can nicely transfer. However, we agree with the intuition behind the question and likewise hypothesize that in other scenarios, where data is not sampled from overlapping distributions, (absolute) forgetting might be much more severe.

We thank the reviewer for their feedback, and for pointing us to two useful references.

“The paper simply repeats the observation of prior literature on continual learning: forgetting occurs, and it matters the performance of the model.”
Earlier works mostly studied forgetting through the lens of measuring final (average) performance of the continual learner. Our work aligns with a surge of recent works that set focus on exploring learning and forgetting at the representation level. Different from the former works, when regarding representations we reviewed published evidence that forgetting may be less severe than previously assumed. We emphasize that the study conducted in our work analyzes the progression of learning and forgetting with respect to each task’s data individually, especially by also differentiating between preceding and newly learned information in the representation. The latter is done with our relative forgetting metric. We find this nuance is necessary to unravel indisputably that representations do forget catastrophically - and further - that forgetting impairs continual knowledge accumulation. Partial aspects may be expectable in hindsight or hinted at in concurrent work, but to the best of our knowledge we are the first to provide indisputable analysis that illustrates catastrophic dynamics of forgetting in representations and its impact on knowledge accumulation by considering a perspective that was yet missing from literature.

Further, we would also like to answer some of the more detailed forwarded concerns.

• “experimentation is limited”: Our experimentation is aligned to the protocols of the works we compare to, in settings where previously mentioned evidence was first brought up. Of course we hope our proposed analysis is carried over to future investigation settings.

• “relative forgetting is very similar to backward transfer [...] this new metric shows a similar tendency to existing forgetting/backward transfer metrics without new insights”: The relative measure of forgetting we propose is, albeit mathematically simple, better representing forgetting with respect to prior learning rather than considering absolute values that can be deceiving. This core motivation and merit is illustrated in Figure 1 and discussed in Section 3.

• “the suggested ensemble baseline violates the conventional continual learning setting [...] the input dimension of the classifier is different from the base continual learning model [...] the trainable parameters are N times larger. This is totally different model and evaluation analysis with the assumption that 'the base continual learning model and ensemble models learn the continual learning tasks in the same way' may not be correct. ”: We would like to point out that, during training, the ensemble baseline learns in exactly the same way as the fine-tuned model. For example, when training on the $n$-th task, the $n$-th submodel of the ensemble baseline (i.e., the submodel being trained on the $n$-th task), is exactly the same as the fine-tuned model when it is being trained on the $n$-th task. The only difference is in evaluation. During evaluation, we do not only use the representation of the latest model, but we also use all the models that we learned along the way. Note that because learning was exactly the same, all these intermediate models do not contain any information that was not at some point also learned by the fine-tuned model. So the ensemble model does not use any additional information; it is just a way to keep track of information that is forgotten by the fine-tuned model. It is true that a possible confounding factor is the dimensionality of the concatenated representation of the ensemble model, but please note that we control for this in the Appendix using dimensionality reduction techniques.

We thank the reviewer for evaluating the proposed metrics as useful for the community and the appreciation of the experimentation.

“The main conclusions are within expectation and well known”.
Most prior works investigated forgetting through the lens of measuring final (average) performance of the continual learner. Recently, a body of literature shifted focus to exploring representations in continual learning, and partially - different to prior findings - raised evidence for an innate robustness to forgetting. We settle this disagreement with a novel perspective and analysis that first indisputably shows that representations do forget catastrophically and secondly illustrates the direct connection to impaired continual knowledge accumulation. To the best of our knowledge such a study was yet missing from literature.

“Why does the contradiction exist?”
We provide one explanation to this question in Figure 1, where we contrast absolute forgetting measures with our relative one. If one reasons with absolute forgetting, it may seem like not everything is forgotten at the representation level. When using relative forgetting instead it becomes clear that there is no contradiction.