Metalearning to Continually Learn In Context

We would first like to thank the reviewer for their valuable time reviewing our work and for many positive comments. Thank you very much.

The paper claims to do in-context continual learning but the concept of in-context learning is not clearly explained.

We actually describe and highlight the concept of in-context learning in Sec 2.2 and Figure 1. But we agree with the reviewer that this is currently not clear as we also use the somewhat older terminology of “Meta-learning via Sequence Learning” in the title of Sec 2.2 (we explain much later in Line 337 “This was rebranded as in-context learning” in Sec. 5). We believe mentioning “in-context learning” in Sec 2.2 upfront should make this context clearer. We will fix this in the final version. Thank you very much for pointing this out.

The paper mainly focus on two task and five task settings but it would be more helpful to see the more different settings such as three task or four task

We actually provide many more experiments in the appendix for the interested readers. Table 10 provides results for 2, 3, and 4 tasks (exactly as suggested by the reviewer) and Table 8 explores generalization up to 10 tasks (by concatenating Split-MNIST and Split-FashionMNIST).

How is the size of SRWM affects the maximum sequence length that can be train?

This is a hard question for which we do not have a straightforward answer because this depends a lot on the nature of the datasets. That said, one useful theoretical result to keep in mind is that, given the dimension of weight matrix D-by-D, the maximum number of key-value associations one can retrieve in a noiseless fashion is D (corresponding to the maximum number of orthogonal keys one can have in the D dimensional space). In practice, the notion of capacity is more complex as the model has multiple layers, and depending on the nature of the data, certain keys may be shared by different inputs without loss of performance.

Why only consider the two-task setting?

As we mentioned above, we consider many more settings than the two-task setting: five-task setting (Split-MNIST) in Table 3, two/three/four tasks in Table 10, and up to ten tasks in Table 8.

Perhaps what the reviewer really meant is: why do we first focus on the 2-task setting? The reason is clear: Our motivation for using two tasks in Sec. 4.1 and 4.2 is to introduce the core problem of in-context catastrophic forgetting and to demonstrate how our ACL overcomes this problem in a minimum and comprehensible setting (and that's the 2-task setting).

Why ACL was not compared with replay buffer based methods?

We focus on the replay-free setting as we believe this is the most interesting setting in continual learning currently (allowing for eliminating replay buffers which requires extra engineering design regarding what to store/remove, and to manage extra memory storage of raw data). Similarly, the recent "learning-to-prompt" papers we cite in our work also focus on the replay-free setting. They additionally raise the privacy issue of the replay buffer which stores raw data to motivate the replay-free setting.

Please also note that our ACL and the use of replay buffer are orthogonal: they could also be combined together.

What is the architecture of SRWM?

The architectural details of SRWM can be found in Table 5 in the appendix.

We believe our response thoroughly clarifies all the reviewer's remaining concerns. Please also refer to our general response highlighting our contributions that we believe are highly relevant considering the diverse interests represented in the NeurIPS community. We really believe our contributions outweigh the limitations, and the current overall rating does not fully reflect the contributions of this paper. If you think that our rebuttal has further improved the reviewer's perception and rating of our work, we would appreciate it a lot if the reviewer could consider increasing the score. Thank you very much.

Thank you for the detailed explanation. I have missed major points raised by other reviewers and feel like I should change my original review. I believe that the paper presents very original idea and need just final finishing in terms of clarity.

We thank the reviewer for their valuable time reviewing our work and for many positive comments.

We also acknowledge the reviewer’s thorough reading through the details of our work. Thank you very much.

One weakness of the proposed method is that the number of loss function terms increases with the number of CL tasks, as pointed out by the authors in Appendix A.5. … Method of reducing the loss terms would strengthen the paper.

Yes, the discussion on this limitation is provided in A.5./L768. Given that the most crucial loss terms are those at the last time step of the sequence, one possibility to reduce the number of terms is to sample a subset of the intermediate terms. However, effectively validating such methods would require actual experiments involving many more tasks, which we leave for future work.

Another weakness, which is also noted by the authors in Table 4 and Section 4.3, is that the performance of the proposed model and method is poor compared with those based on pre-trained transformer models

Yes, clearly we will need meta-training on larger and more diverse datasets for this method to demonstrate its full potential. In fact, as a purely data-driven method, more and more improvements are potentially expected by scaling this up with more compute and datasets (Sutton’s bitter lesson).

it is unclear how much value the proposed method adds. SRWMs are not widely used and LLMs training can scale to a massive number of tasks with a single loss [1] (albeit not CL). A more detailed explanation of the application of the findings of this paper beyond those interested in SRWMs would be helpful.

Thank you very much for asking this question. Here we would like to bring up one important and (hopefully) convincing aspect. It is true that SRWM itself is not widely used at the moment. That said, there is an emerging trend in sequence processing using fast weights (a family of linear Transformers). As we also emphasized in the general response, two very recent works [2] and [3] show very promising results on general language modeling using such architectures.

It should be noted that [2] makes use of DeltaNet [1], whose direct successor is SRWM (DeltaNet augmented by self-reference), and [4] shows that SRWM is indeed more powerful than DeltaNet (using formal languages) while not requiring hard-to-parallelize, true recurrence. Besides, [3] also mentions “multi-level meta-learning” in the outlook, which was essentially the original motivation of SRWM.

Given these evidences of active research on this family of models, in our view, it is not unlikely that there will soon be other works scaling up SRWM or similar models on other tasks. Here we motivated SRWM as a natural architecture for continual learning but its applicability is broader in principle (maybe similar to how attention/Transformer was first explored in machine translation as a natural architecture). From this perspective, we believe we have one extra contribution in this work that is broader than the specific scope of this work. This is a successful example of sequence processing using SRWM which may also be of interest to those interested in general sequence processing using fast weights or linear Transformers. We believe this point precisely addresses the reviewer's concern regarding the impact of this work beyond our specific scope.

Another weakness of this paper is its focus on image classification meta-learning tasks only.

This is another very valid point. As a first paper on this method, we focused on image classification which has classic continual learning benchmarks. Extending this to other modalities is an exciting future research (we also mention this in L311/Sec 5).

We hope this response and the general response further clarifies our contributions (including the relevance of SRWM in a broader scope of sequence processing with fast weights). If you think our response has enhanced your perception of this work, and if you think the paper should be accepted, we would appreciate it a lot if the reviewer could consider increasing the score. Thank you very much.

[1] Schlag et al. ICML 2021. Linear transformers are secretly fast weight programmers. https://arxiv.org/abs/2102.11174

[2] Yang et al. arXiv June 2024. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. https://arxiv.org/abs/2406.06484

[3] Sun et al. arXiv July 2024. Learning to (Learn at Test Time): RNNs with Expressive Hidden States. https://arxiv.org/abs/2407.04620

[4] Irie et al. EMNLP 2023. Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions. https://arxiv.org/abs/2310.16076

We thank the reviewer for their valuable time spent on reviewing our work.

We believe we have good responses to resolve all the main concerns.

== Factual clarifications ==

Before providing our clarifications to the reviewer’s concerns, we would first like to resolve some factual misunderstandings.

In each step over a sequence of demonstrations, the method needs to compute and store a new weight matrix in order to perform back-propagation. It might require more memory during training and at inference.

This is not correct. There is no need for such storage (there is a memory-efficient algorithm that is standard in any practical linear Transformer implementations). For a detailed explanation, please kindly refer to our response to Reviewer d5XY (marked "[also relevant to Reviewer HvJH]"). We apologize for this inconvenience due to the length limit.

The experiment shows positive results when compared to non-meta-learning approaches

Thank you for mentioning this as a strength. We would like to add that positive results are also shown compared to the existing meta-learning methods (Table 3). Unlike prior meta-learning methods that only learn “representations” for CL, our method successfully learns an entire “learning algorithm” (the original definition of learning-to-learn) and outperforms prior methods. We believe this achievement in meta-learning is largely overlooked in the current review.

== Clarifications to weaknesses/questions ==

The motivation of formulating (continual learning) CL as meta-learning is not well presented.

The motivation is described in the introduction (Line 30-50). Generally speaking, machine learning is useful when it is hard for humans to design a hand-crafted solution. Here we claim that hand-crafting CL algorithms has been unsuccessful in the past. Therefore, we propose to use machine learning to automate the process of designing CL algorithms. This corresponds to “learning of learning algorithms”, or meta-learning.

Some details of the architecture are mentioned in the background section only (e.g. replacing self-attention with SRWN and the multi-head version.)

We’d like to clarify that SRWM is indeed a background work (including its role as a replacement to self-attention and the multi-head version). These design details are entirely based on the previous work by Irie et al. ICML 2022 "A Modern Self-Referential Weight Matrix That Learns to Modify Itself".

The details of the training and inference process are not well presented.

The corresponding processes are described in Sec 3. and visually highlighted in Figure 1. If the reviewer still thinks these are “not well presented”, we would appreciate it a lot if they could tell more concretely what they find confusing. (A dedicated section in Appendix A.1 “Continual and Meta-learning Terminologies” also helps readers with continual and meta-learning jargon).

Even being a meta-learning approach, the model still needs fine-tuning when given a new task to adapt to a new number of tasks

This is a valid point and it remains an open research question e.g., to find how to deal with an unseen maximum number of classes in the class-incremental setting. Here we would like to strongly emphasize that meta-learning is an open research topic. We can not solve all these problems in a single paper. A significant achievement here is that we obtain a successful meta-learned CL algorithm for a fixed maximum number of classes. Extending our work to relax these constraints is an exciting future research direction in meta-learning.

Can the authors explain more on the following claim “The stability-plasticity dilemma are automatically discovered and handled by the gradient-based program search process.“ (line 52)?

Yes, our pleasure! When we manually design a CL algorithm (say a regularization method with a hyper-parameter weighting the auxiliary term), we face the problem of deciding how much we allow model parameters to change (plasticity) or not (stability) to balance preservation of current knowledge against learning of new one. In our method, we do not have to deal with this decision ourselves. This process is automated by gradient descent that directly optimizes weight modification rules (Eq. 3) for the ultimate model performance.

What are the advantages of this method compared to previous approaches?

How does the method scale with the number of tasks in terms of performance and computation?

The current advantage is that we can achieve a CL algorithm that performs well on classic benchmarks (unlike prior methods that fail). Another potential advantage is that, as a purely data-driven method, more and more improvements are expected by scaling this up with more compute and datasets (Sutton’s bitter lesson). Telling exactly how they scale would require a proper scaling law study, itself requiring more compute.

How do the number of examples per task and order of tasks during the training affect the performance at inference time?

An ablation on the number of examples can be found in Table 7 in the appendix (5 vs. 15 examples). Regarding the order, we conducted experiments in the 2-task (A and B) setting: Tables 1 and 2 present results for both A-then-B and B-then-A orders.

It’s unclear how to calculate the loss function in a batch fashion

There is no such issue: each sequence in the batch is constructed such that it contains the same number of examples and the task boundaries at the same time steps.

We believe our response thoroughly addresses all the concerns raised by the reviewer. Please also refer to our general response highlighting our contributions that we believe are broadly relevant to the NeurIPS community, despite all our limitations. In light of these clarifications, we believe the current rating does not fully reflect the contributions of this paper. If the reviewer finds our response useful/convincing, please consider increasing the score. Thank you very much.

We thank the reviewer for their valuable time reviewing our work and for many positive comments. Thank you very much.

== Factual error corrections ==

Before providing our clarifications to the reviewer’s concerns, we would first like to resolve some factual errors in the review.

in a classic two-task setting (Split-MNIST)

showing an advantage of their approach in scenarios with up to 3 tasks.

These statements are not correct. The main results on Split-MNIST (Table 3) correspond to a 5-task setting. (We also report the generalization performance on a 10-task setting by using Split-MINST and Split-FashionMNIST in Table 8 in the appendix).

Moreover, one limitation that the authors have not mentioned is that the meta-objective seems to require keeping in memory a number of copies of the model that is equal to the number of tasks. [also relevant to Reviewer HvJH]

This is not correct. We do not need to keep the intermediate copies of the model. The reviewer is right to point out that a naive implementation would require such copy storage. The situation is actually even worse for a naive implementation: it would have to store ALL intermediate copies of the model’s fast weight states (at every step) for backpropagation through time; this would result in impractical memory requirements for ANY experiments we conducted.

Instead, the practical code implements a custom memory-efficient algorithm that only stores one copy of the model fast weights in the entire forward pass, and during backpropagation through time, we de-construct the model by applying the reverse fast weight update (Eq.3 in the backward direction) to obtain the model state we need at the corresponding step. This only requires storing the key/value/query/learning-rate activations for all steps in memory (which are much cheaper) and a single copy of the model.

We did not describe this implementation trick as this is standard in ANY practical linear Transformer implementations (in case the reviewer is interested, please find example public code below). We will add this discussion in the final version. Thank you for pointing this out.

Public code references:

• Linear Transformer: https://github.com/idiap/fast-transformers/blob/master/fast_transformers/causal_product/causal_product_cuda.cu

• SRWM: https://github.com/IDSIA/modern-srwm/blob/main/supervised_learning/self_ref_v0/self_ref_v0.cu

== Clarifications to weaknesses/questions ==

We believe we have convincing answers to all the main concerns raised by the reviewers.

the main limitation of the approach is its practicality.

We agree with the reviewer that our method requires scaling multiple hyper-parameters to be useful in more realistic settings. That said, our main bottleneck is the compute. In the limitation section, we do propose approaches to deal with the algorithmic limitations (e.g., to handle longer lengths, we’ll have to introduce context-carry over as used in language modeling with linear Transformers; see Line 302). There is no reason that better-resourced teams, which can afford to train today's large language models, cannot scale this up to much larger and more diverse datasets.

I invite the authors to look at … examples of more realistic benchmarks.

We thank the reviewer for sharing these references. We’d like to emphasize that it’s not that we didn’t know about these more realistic tasks, but rather that we do not have compute to conduct larger scale experiments. Please note that given that our method is novel, we had to allocate a lot of compute for ablation experiments (we have many more experiments in the appendix).

As we also emphasized in our general response, we strongly believe that this work has significant contributions by providing several evidences for the promise of this method, even without a large scale continual learning experiments. These contributions are largely overlooked in the current review.

It would be interesting to add a discussion of the cost of the approach (computation, memory, ...). Even is it gives a substantial boost in many cases, it would be interesting for practitioners to compare what they gain to what they pay.

Assuming that model hyper-parameters are already available, as we mention in A.3., less than 1 day of compute using a single V100 GPU is enough to produce a single-run experiment reported in this paper (e.g., the Split-MNIST result). We also would like to emphasize that our method does not just provide a “substantial boost” but rather represents a clear-cut switch from a complete failure (hand-crafted approach or some other meta-learning method) to reasonable continual learning in many cases.

The approach is focused on the task aware scenario, and rooted in a notions of tasks. In many practical scenarios, the distribution shift occurs in a softer way, with no clear notion of task boundaries. Can the authors comment on the possibility of extend their approach to the task-agnostic scenario?

Thank you for pointing this out. The task awareness is only required for meta-training. At test time, the model is not aware of anything related to the task identities or task boundaries; they are evaluated in a completely task-agnostic manner. For training, it seems reasonable to assume that we do have access to training examples where we know task identities. Therefore, extending this to the task-agnostic setting is rather straightforward.

We believe our response thoroughly addresses all the concerns raised by the reviewer, and directly resolves many of them. Please also refer to our general response highlighting our contributions that we believe are highly relevant considering the diverse interests represented in the NeurIPS community. In light of these clarifications, we believe the current rating does not fully reflect the contributions of this paper. If the reviewer finds our response useful/convincing, please consider increasing the score. Thank you very much.