Automating Continual Learning

NB: Please find our response to other questions in "Common Responses to All Reviewers" above. Thank you.

We thank the reviewer for valuable time reviewing our work and for valuable comments.

There are several important prior works in this domain [1, 2, 3] that were not mentioned in the paper. They should also be compared as baselines.

The reviewer is right: these citations were missing. We actually had realized this in the meanwhile, so we had already added these MAML-based/like approaches in our internally updated paper. Following the reviewer’s suggestion, the updated paper includes a comparison to [1] on Split-MNIST (Table 3; the experimental details can be found in Appendix A.6). Please note that according to the readme file in the official repository of [1] https://github.com/khurramjaved96/mrcl, “... it is possible to get the same results as ANML (S. Beaulieu 2020) without using any neuromodulation layers” We concluded that comparing to [2] is unnecessary here; we focus on [1] as our meta-continual learning baseline.

That said, regarding:

According to my understanding, this work should be classified as a meta-continual learning (MCL) approach, which is also referred to as learning to continually learn.

While our method can definitely be categorized as a meta-learning approach for continual learning, it should not be categorized as “learning to continually learn” in the sense of [2], since ACL learns a learning algorithm while MAML based methods [1, 2] only learn representations for CL (or representation learning networks) and still relies on the standard learning algorithm (typically Adam). These are fundamentally different methods even if the title of [2] is very broad, and gives the impression of generality; that is not the case. In particular, their meta-testing requires tuning the learning rate—and in addition, in our experiments/Table 3/Appendix A.6, we also observed that they also require tuning meta-test training iterations to perform well on an unseen domain, (Split-)MNIST. In contrast, our learned learning algorithm, once trained, does not require any tuning (learning rate is self-regulated by the trained SRWM; sigma(beta) in Eq.3) This distinction is important. We clarify this in the updated “prior work” paragraph (Page 8, Sec 5.).

Since MCL is a branch of meta-learning that aims to optimize a learning algorithm, it is crucial to separate meta-training and meta-test sets. Also, there should be no overlap in the constituent tasks between them. Otherwise, the model can achieve a high score simply by memorizing the tasks in the meta-training set without learning new knowledge during the meta-test phase.

Yes, of course, we are aware of this. We used the standard few-shot learning sequence construction process to ensure this (see further comment in the next point). Otherwise, it is anyway impossible to obtain a model that can learn unseen MNIST or CIFAR-10 during meta-testing, while only meta-trained on Omniglot and Mini-ImageNet (see Table 2).

In the paper, I could not find any description of how meta-training and meta-test sets are constructed. I suspected that the authors would have followed the conventional meta-splits for Omniglot and Mini-ImageNet datasets,

Yes, absolutely. In the Appendix A.1., we had mentioned that we used torchmeta for that, but we agree this was not clear in the main text (relating to the next point).

but I suggest the authors refine the overall terminology to be consistent with the existing meta-learning or MCL literature.

The reviewer is right; our terminology was confusing (which likely caused Reviewer jN2L’s confusion). Following this suggestion, we now use “meta-training train/test” and “meta-test train/test” terminology in the updated paper. Thank you for pointing this out.

The proposed method is tested only on two-task and three-task CL scenarios,

This is not accurate; while we had only used 2-task and 3-task objective functions, we evaluated our models on 4-task CL (now called Table 4). Now with Split-MNIST, we also test our method for 5-task settings.

Thank you very much for these additional comments. We really appreciate the reviewer's engagement.

We understood the reviewer's point on the ID evaluation. That said, regarding:

It seems unreasonable to claim successful OOD generalization with merely 74.6% accuracy in MNIST. This score may seem strong compared to other CL baselines, but other MCL approaches achieve far better accuracy in much more challenging scenarios if an ID meta-test set is used [1, 2, 3].

We can not agree with this argument/reasoning. Our high-level goal/claim is to replace hand-crafted CL algorithms by a "better" learned CL algorithm. On Split-MNIST (a standard CL benchmark suggested by the two other reviewers), our learned algorithm effectively outperforms other hand-crafted CL baselines: "74.6% accuracy" has to be compared to the performance of the baselines which is around 20-25%, strongly supporting our original claim.

We can not directly compare these numbers to other numbers achieved on other datasets which have nothing to do with this benchmark. We emphasize that we ran this Split-MNIST experiment not because it's our "lucky OOD test set" (in Table 2 and 4, we also evaluate our system on CIFAR-10 and F-MNIST) but because it is a standard/universal CL benchmark that the CL community is generally interested in (as suggested by the two other reviewers).

Now we can add MCL baselines to this comparison; we already reported [1] (implicitly covering [2]) as requested by the reviewer; we could also add [3] or any other existing MCL methods, but that would not fundamentally diminish our contributions much since none of [1,2,3] really learns learning algorithms. As we explained in A.6., [1] is sensitive to meta-test hyperparameters tuning (learning rate, and number of iterations) which is still a characteristic of handcrafted algorithms---something we want to avoid to "automate CL." [3] indeed does not require tuning meta-test hyper-parameters, but it is based on a very different paradigm (of generative classifiers). These MCL methods are relevant (we agree) but they are not solutions to our original goal of replacing hand-crafted CL algorithms for neural nets. Our method is conceptually unique/new in this regard (also directly relevant to the now-popular in-context learning) and its effectiveness is clearly shown on a well-known standard benchmark; while this may still be experimentally limited, follow-up works may further improve this approach and apply it to other datasets... We'll respectfully leave the reviewers and AC to ultimately judge.

Once again, we thank the reviewer for his/her engagement during this rebuttal, and for very useful comments.

We thank the reviewer for valuable time reviewing our work and for very positive comments on the originality and clarity of this work.

Authors show that their approach is promising through experiments on MNIST, Omniglot, and Mini-ImageNet.

Just to clarify as this is not fully accurate: the set of datasets we used is Omniglot, Mini-ImageNet, FC100 for (meta-)training and MNIST, Fashion-MNIST, and CIFAR-10 for (meta-)testing.

How do the data requirements of your method grow with the number of tasks?

In terms of data requirements to increase the number of tasks, what we need is essentially to increase the number of classes. In principle, we would only need a handful of examples for each class (as we anyway want to train our models as sample-efficient few shot learners). For example, Omniglot has 1632 classes with only 20 examples each.

How is the training sequence constructed?

We follow the standard procedure used in few-shot learning with sequence processing networks; that is, given a dataset with $C$ classes, for each sequence, we sample $N$ random but distinct classes out of $C$ ($N < C$). The resulting $N$ classes are re-labelled such that each class is assigned to one out of $N$ distinct random label index which is unique to the sequence.

Do you use a single sequence? If not, doesn't it mean you're effectively performing joint training?

We use multiple sequences in a batch, but no, it does not result in any form of joint training (please also refer to our response to Reviewer jN2L where we clarify more misunderstandings). This is because the training sequences are constructed such that class-to-label mapping is unique to each sequence (that is, each sequence is a unique learning problem). While this is actually classic in few-shot learning settings, since other reviewers also find this obscure, we’ll be happy to add a more elaborated description in the final version.

Could you elaborate what do you mean by "certain real-world data may naturally give rise to an ACL-like objective"?

Yes, what we mean is the following. If we consider a language model trained on a very long training sequence (as is the case now with large models); it is not implausible that such a sequence naturally consists of (let’s say three) text chunks (A, B, A’) where A’ is “related” to A (each letter is a chuck/paragraph of text). In this sequence (processed from left to right), learning to predict part A’ after having observed (A, B) encourages remembering the contents of part A. This is very similar to the (artificial) construction of our ACL loss, but obtained without explicit intention of continual learning objective.

We hope the most crucial concern of the reviewer is resolved through the Split-MNIST results. Regarding other limitations, we consider them as interesting research questions to be investigated in follow-up works; we believe the current/updated version of the paper demonstrates that the proposed method is an interesting new approach for continual learning. If the reviewer agrees with this, we’d appreciate it a lot if you can consider increasing the score. Thank you very much.

PS: Please find our response to other questions in "Common Responses to All Reviewers" above. Thank you.

NB: Please find our response to other questions in "Common Responses to All Reviewers" above. Thank you.

We thank the reviewer for valuable time reviewing our work and for encouraging comments on the originality of this work (“a fresh perspective to an area (CL) that is becoming increasingly incremental”).

The reviewer has both some critical misunderstandings of certain aspects of our method, AND really important/relevant comments at the same time. We’d like to resolve/respond to them all here.

I have some major concerns about whether this method is actually doing CL (versus multitask learning).

[NB: We assumed that by “multitask learning” the reviewer means “joint training”.]

Here the reviewer has some critical misunderstandings about our method. As we’ll explain, our framework is a proper CL framework. We are not affected by ANY of the problems/concerns raised by the reviewer regarding whether our method respects the framework of CL. We believe what causes some of these confusions is the meta-learning procedure, and our terminologies were not very helpful in that regard. To improve this, we introduce meta-training meta-testing terminology (as suggested by Reviewer StyD). Please let us try to resolve this confusion one-by-one as follows.

Having a temporally structured input during evaluation is not a valid approach in the context of CL (although I am aware that some meta-learning papers unfortunately do that -- but that does not mean that their approach can be accepted without question because it has been previously published). Such temporally structured inputs makes discrimination between classes of different tasks trivial. For example, if you give someone 75 Omniglot examples and 75 Imagenet examples and ask them to classify an input x during testing, I can easily determine whether x is from Omniglot or Imagenet without learning (just by computing some statistics of pixel values). Letters would, of course, look different than natural images. Then, predictions become much easier.

[This confusion is independent of meta-learning] We first would like to clarify that all our main CL settings are “domain-incremental” (except the “class-incremental” setting in our new Split-MNIST experiments). This means that if our CL task consists of 2 tasks with both of them being 5-way classification, the output dimension of the model is also 5 (not 10), which is shared for both tasks. In this setting, the temporal structure can not help the model for classification: (taking the reviewer’s example) an input image has to be classified to be one of the 5 labels among {0, 1, 2, 3, 4} regardless of whether the image comes from Omniglot or Mini-ImageNet. A model may easily recognize that an input image is from Omniglot, that’s likely the case, but that would not facilitate the task; it still has to do the prediction among the 5 labels.

Thank you very much for your prompt response.

Unfortunately, these responses did not address my concerns on whether the paper actually solves the Continual Learning problem -- at least in any practical manifestation of that problem.

We are sorry to read this. Please let us ask a few questions that might help.

The following sentence, copied from the authors' response, shows how confusing the proposed approach is, for instance in terms of whether it uses training data during testing (but there are also other similarly confusing points): "During meta-testing, our model reads a sequence of input/label example pairs (called “meta-test training” examples) sampled from the training set of MNIST, then (similarly) a sequence of CIFAR-10 training examples; our model ends up in a weight state W_{MNIST, CIFAR-10} at the end of the entire sequence. No prediction has been done so far. Now using the resulting weights W_{MNIST, CIFAR-10}, we make predictions on examples (called “meta-test test” examples) from the test set of MNIST and CIFAR-10 to evaluate the model’s capability to predict both the first task, MNIST, and the second one, CIFAR-10."

Could you please explain what exactly the reviewer finds "confusing" here? We detailed every and each step of the process precisely because the reviewer did not seem to be familiar with few-shot/meta-learning. We can also describe standard continual learning with a hand-crafted learning algorithm (let's say, Adam) exactly in the same style:

"""

Let’s say our goal is to learn two tasks sequentially: MNIST, then CIFAR-10 using the Adam optimizer. We'll train a network with weights W, initialized as W_0. We sample input/label example pairs from the training set of MNIST. We do a few iterations of Adam on these examples, starting from weight W_0; after this, the weight becomes W_{MNIST}. Now we sample input/label example pairs from the training set of CIFAR-10. We do a few iterations of Adam on these examples, starting from weight W_{MNIST}; after this, the weight becomes W_{MNIST, CIFAR-10}. This is the end of training; the resulting weight is W_{MNIST, CIFAR-10}.  Now we make predictions on examples from the test set of MNIST and CIFAR-10 to evaluate the model’s (with weight W_{MNIST, CIFAR-10}) capability to predict both the first task, MNIST, and the second one, CIFAR-10.

"""

Is this confusing? The only difference between the conventional CL and ours is that the learning algorithm used for CL is hand-crafted (e.g, Adam) or learned (forward pass of SRWM). We emphasize that in general, we are not supposed to go into this level of detail to describe continual learning though. 

The fact that three independent reviewers had very similar concerns about the proposed approach means something in my opinion

This is not true. Reviewer WhrD wrote "The paper is well written and properly structured. The figures are of high quality and help in quickly grasping the main ideas." We also believe that the figure is almost self-explanatory of the method but we also understand that this perception may depend on the readers' familiality with meta-learning/sequence-processing.

 I think that it would be beneficial for the paper if the authors largely re-write the paper, at least the problem formulation, the description of the method, and the setup of the experiments, also following the more common terminology and assumptions in the CL and metalearning literature.

The terminology on metalearning has been already pointed out by Reviewer StyD, and we have immediately adopted his/her suggestion as Reviewer StyD was specific about his/her suggestion. We'll only be able to re-write the paper if the reviewer can be more specific: what terminology and assumptions in the CL literature is the reviewer referring to?

All Reviewers

(Reviewer WhrD) The main weakness of the paper is the experimental evaluation. Despite presenting their approach as a continual learning method, the authors don't use any of the standard benchmarks (e.g. Split MNIST, Split Mini-ImageNet), nor do they compare to any previous work (regularization, replay, or parameter isolation methods).

(Reviewer jN2L) I see that the method is significantly different from other continual learning methods, still I would expect the authors to benchmark against some existing methods.

(Reviewer StyD) There are several important prior works in this domain that were not mentioned in the paper. They should also be compared as baselines.

We agree with the reviewers. To address this very important point, we conducted extra experiments on Split-MNIST (mentioned by both Reviewer WhrD and jN2L). The results can be found in the updated PDF (Table 3 and text in **Sec 4.3 and Appendix A.6). We follow the standard task definition: domain-incremental and class-incremental learning settings (using Hsu et al. 2018 [1]’s terminology). This allows us to compare our method to many existing methods, including 8 classic continual learning baselines: standard optimizers SGD & Adam, L2 regularization, two elastic weight consolidation variants, synaptic Intelligence, memory-aware synapses, and learning-without-forgetting (thanks to Hsu et al. 2018 [1]’s overview), as well as one meta-continual learning baseline (Javeh and White 2019 [2]) suggested by Reviewer StyD.

The overall result we obtain is very positive: our out-of-the-box ACL model performs very competitively with the best existing methods (see our note below) in the domain-incremental setting, while it largely outperforms them in the 2-task class-incremental setting. By further fine-tuning this model with the 5-task ACL objective, we obtain a model that largely outperforms all other methods in all the settings. We hope this convinces the reviewers regarding the promise of our method.

Note: Please note that our comparison does not include replay-memory or parameter-increasing methods because such methods are orthogonal to ours; they can be combined to our method to potentially further improve the performance.

[1] Hsu et al., 2018. ​​Re-evaluating continual learning scenarios: A categorization and case for strong baselines. https://arxiv.org/abs/1810.12488

[2] Javeh and White, 2019. Meta-Learning Representations for Continual Learning. https://arxiv.org/abs/1905.12588

Reviewer WhrD and Reviewer jN2L

(Reviewer WhrD) The meta-learning formulation is also a major limitation, as the number of loss terms grows rapidly with the number of tasks and it is not clear whether the method is practical for e.g. 10 tasks (which is still a small number compared to the requirements of real-world lifelong learning).

(Reviewer jN2L) Another major concern is whether ACL can be used in a practical context in which many tasks will be learned over time (as opposed to just a handful). If you examine Equation 5 on the last page, you'll notice that in order to learn a third task, they need to add three terms to the loss function. In continual learning, a five-task setting is considered small. To learn SplitMNIST, for example, they would actually need 1 + 2 + 3 + 4 + 5 (15) terms in the loss function. As a result, their method becomes quadratically more expensive in terms of computation (i.e., for Task n, you require backpropagation through (n)(n-1)/2 terms). This is clearly not practical.

We first would like to note that computing these loss terms isn’t immediately impractical because they essentially just require forwarding the network for one step, for many independent inputs/images. This can be heavily parallelized as a batch operation.

While this is a valid concern when scaling up more, a natural open research question is: will we really need all these terms in the case we have many more tasks. Our experiments (both the new Table 3 results on Split-MNIST, and the old results on 4 tasks, now Table 4) effectively show that using more terms in the ACL loss improves performance. That said, ideally, we want these models to ‘systematically generalize’ to more tasks even when they are trained with only a handful of them. Indeed, our out-of-the-box model trained only with the 2-task ACL objective does not completely break even when evaluated on the 5-task setting of Split-MNIST (see Table 3; row “ACL (Out-of-the-box model”). We consider this as an interesting research question on generalization to be studied in a follow-up work.