Separating the 'what' and 'how' of compositional computation to enable reuse and continual learning

We thank the reviewer for their interest in our work and thoughtful comments. Below, we attempt to address the raised concerns, answer questions and highlight changes we have made in the revision in response to the reviews.

1. Limited task set. We agree that extending to a richer task set with more complex input-output mappings, with a more thorough assessment of the potential for (and directionality of) transfer learning and compositionality, is important. At the present stage, we focus on synthetic problems common in neuroscience models for interpretability and relevance for understanding Continual Learning and compositional re-use in RNNs. In related work studying similarly simple tasks (e.g. refs. 11, 13-16), compositional reuse in multi-tasking RNNs has been of major interest. Our work is intended to contribute to this space, making the compositional task structure, and how it may shape a potential RNN solution, an explicit focus. We note that our approach for modeling the statistical structure of cognitive tasks in a cohesive generative framework is novel. We expect that this contribution will help to establish a more formal notion of task-similarity and when/how transfer might be simple or more difficult. We fully agree that a richer task set will lead to a variety of interesting questions for further work.
2. Writing in Sec. 3. We thank the reviewers for suggestions about improving the readability of Sec. 3, which was a concern also shared by other reviewers. We revised the writing of Sec. 3 to make it more accessible to readers unfamiliar with probabilistic modeling. In particular, we made sure to highlight conceptual features of the algorithm first, before introducing formal notation. We also stressed that the model is given the correct task label $c$ and only performs the epoch inference $z$. If there are additional parts of Sec. 3 that the reviewer found to be particularly inaccessible to the reader, we would appreciate more detail (e.g. line numbers), so we can make sure to improve their clarity.
3. Differentiating Memory and Delay tasks. We have updated the wording to clarify that there are two differences between these sets of tasks. First, the Memory tasks have a fixation epoch after stimulus presentation. Second, the correct stimulus is present during the response epoch for Delay tasks but not during the response epoch for Memory tasks (see Fig. 1a). We have added details on this to section 4.4.
4. Biological plausibility. We agree that the current version of the architecture lacks biological realism and is only brain-like at a conceptual level. We are very interested in exploring more biologically plausible architectures that could implement the modulation of low-rank components of the RNN. [16] connected a similar selection of low-rank components to neuromodulation. [37] implemented the selection of individually controlled low-rank components through an architecture representing interactions between basal ganglia, thalamus and cortex. This model assumes that the low-rank connectivity of the RNN is the result of cortex-to-thalamus and thalamus-to-cortex connections (= thalamo-cortical ‘loop’). When the number of thalamic neurons participating in each loop is small, the connectivity is low rank. Different loops can be selected via inhibitory input from basal ganglia to thalamus, which effectively turns on and off different low-rank components of the RNN – just like our architecture. Contrary to our work, [37] was not concerned with how this inhibitory selection signal is set.
5. Low-rank components (Q1). The number of low-rank weight matrices (a set of low-rank components) equals the number of unique task epochs. For our current task set, the inference algorithm identifies 9 unique task epochs in total. Each task epoch uses a rank-3 recurrent weight matrix in our main figures (defined by $U_z, V_z$, Appendix C.1), though performance increased for higher ranks (Figure S5). For task-epochs solved by an at most rank-$R$ weight matrix, we may implement $N_z = N/R$ epochs for a fixed network dimensionality $N$. The network can solve tasks that can be composed as sequences of the $N_z$ epochs, which could be a large number. It is worth noting that, since each component of the network uses low-rank recurrent weights, the memory cost of adding a new component is linear in RNN size, as opposed to the quadratic scaling in vanilla RNNs. To improve scalability, a possible next step is to discourage the addition of new epochs by penalizing having a large $N_z$. Alternatively, it may also be possible to group epochs that are statistically distinct, but can share the same solution through another learning mechanism. This would be particularly relevant for the response epochs in our task set.
6. Number of learned latents (Q2). Across the 6 tasks we studied, 10 epochs are defined as part of the generative process of the tasks (listed in line 93 of the appendix). The ‘what’ system learns this generative model based on the inputs and targets of each task, and identifies only 9 epochs. This is expected, as the fixation epoch $F$ and memory epoch $M$ are indistinguishable based on their input-output patterns.
7. Additional baselines (Q3). We agree with the reviewer that a comparison against the hypernetwork approach in von Oswald et al. [23] and Ehret et al. [31] is warranted. We have performed these comparisons and are planning to include them in the revised manuscript. Per the suggestion of other reviewers, we also performed a comparison against the neuromodulatory network in Costacorta et al., 2024 [16]. In short, we found that neither method matched the performance of ours in terms of continual learning or compositional generalization. Details and performance metrics are provided at the end of this reply.
8. Less structured tasks (Q4). Handling tasks with less apparent structures is an important next step. Training the 'what' system together with the 'how' RNN is indeed a promising approach. In the case of ambiguous structures, it may be helpful to use the hidden state of the 'how' RNN as part of the observation $\mathbf{q}_t$ to infer epochs. This is similar in spirit to the scheme in Ostapenko et al., 2021 for modules in a feedforward network, where the output of each module is used to determine its relevance for a given input.

---

Summary of additional baselines. Costacurta et al., 2024 [16] designed neuromodulated RNN (NM-RNN), in which a full-rank modulatory RNN (RNN1) outputs a low-D signal that dynamically scales the low-rank recurrent weights of an output-generating RNN (RNN2). RNN1 and RNN2 are jointly optimized through gradient descent. Their design showed catastrophic forgetting during continual learning (Table 1). In contrast, our method replaces RNN1 with a probabilistic task model, which rapidly learns a task’s structure within a handful of trials, thus is able to provide useful contextual signals when optimizing RNN2 and facilitate continual learning. Von Oswald et al., 2020 [23] designed task-conditioned hypernetworks to tackle continual learning. Their hypernetwork receives a learnable embedding distinct for each task and outputs the weights for the target network (a RNN in our case). Their design underperformed our method (Table 1). More importantly, these hypernetworks do not capture the time-varying epoch structures of each task, and perform worse on compositional generalization (Table 2).

Implementation details.

NM-RNN: RNN2 has 256 neurons and with rank($W^{rec}$) = 27, equal to the default choice for our ContextRNN. RNN1 has 125 neurons such that the whole network has a similar number of parameters as ours.

Hypernetwork: The full-rank RNN has 256 neurons. We used a chunked hypernetwork with 34 chunks and 2000-D output. The hypernetwork is a MLP with two 32-D hidden layers, receiving 32-D task embedding and chunk embeddings as input. After a grid search, we set the continual learning regularization strength to 1, orthogonal regularization strength to 0.001 and clip the gradient norm to 1 (see Ehret et al., 2021).

Table 1:

A. NM-RNN

B. Hypernetwork

C. ContextRNN (ours)

Caption: Continual learning performance (supplementary to Figure 4 a-d). A: Each column shows a task's final performance averaged across 5 random seeds (± std) after training NM-RNN sequentially on task 1 to 6. B: same as A but for hypernetwork. C: same as A but for contextRNN (ours).

Table 2:

A. OWP

Caption: Compositional generalization (supplementary to Figure 6c). A: Each column shows the test performance on MemoryAnti averaged across five random seeds (± std) after training a pretrained RNN using OWP on MemoryAnti for 0, 20, 40 or 512 trials. B: same as A but for hypernetwork. C: same as A but for contextRNN (ours).

I thank the authors for responding thoroughly to all the raised points and for providing additional baselines. Specifically, I appreciate the following:

1. The authors address that the experiment setup is a bit limited in that it has a small set of tightly related tasks. I agree that modelling the cognitive tasks using RNNs in this compositional way is novel and serves as proof of concept.
2. They clarified the writing wrt to the "inference of tasks", as opposed to episodes.
3. I might have misunderstood the difference in R and R_M, which is in whether the stimulus is still on screen. Thanks for the clarification.
4. They addressed biological plausibility and clarified that it only exists on a conceptual level. Interesting that low-rank decomposition could be implemented, I should check out (37). It may be cool for neuroscientists to test in neural data, if there is what and how networks in the brain interact with these low-rank components.
5. Thanks for clarifying how the decomposition works and the number of latents.
6. I appreciate the additional baselines, especially the hypernetwork comparison. Cool that your method comes out better across tasks.

However, some points are still puzzling me. I understand the differences between Memory and Delay task now (which is the additional fixation period and the stimulus visibility), but I would still argue that very similar dynamics can be used in how networks, so the improved transfer between the two seems not very surprising. The same holds for the M' task, which is quite close to the Delay task. In the future, it would be interesting to see how generalization to other possible compositions of the epochs behaves. E.g., one could imagine a hard task having 2 stimuli of which only the first has to be considered for the answer.

I keep my overall score at 5.

We thank the reviewer for their interest in our work and thoughtful comments, and appreciate that the reviewer thought our submission makes a novel contribution that was thoroughly demonstrated through experiments and evaluations. The explicit suggestions for how to improve the paper were very helpful, and we are making the following changes in response to them:

1. Additional baselines & more thorough discussion of related work. We agree with the reviewer that additional baseline comparisons are necessary and have added two additional baselines in the manuscript. The first one is against the recent Costacurta et al., 2024 [16] neuromodulatory network and a second is against a hypernetwork approach (von Oswald et al., 2020, [23]). In short, we found that neither method matched the performance of ours in terms of continual learning or compositional generalization. Details and performance metrics are provided at the end of this reply. We have updated the 'Related work and our contributions' section to discuss these works further and also expanded the baseline comparisons (Fig. 4) to include these two. We hope that these changes will address Limitations (3) and (4).
2. Improved clarity of Sec. 3. We revised the writing of Sec. 3 to make it more accessible to readers unfamiliar with probabilistic modeling. In particular, we made sure to highlight conceptual features of the algorithm first, before introducing formal notation. We also stressed that the model is given the correct task label $c$ and only performs the epoch inference $z$. If there are parts of Sec. 3 that the reviewer found to be particularly inaccessible to the reader, we would appreciate more detail e.g. line numbers, so we can make sure to improve their clarity. We hope that these changes will address Limitation (5).

We respond to the other concerns below.

3. Assumptions on task structure. This concern is related to the assumptions about task structures that we make in the paper, and raises the question of whether our approach is applicable to domains without clear compositional task structures. At the present stage, we have chosen to focus on synthetic problems common in neuroscience modeling work for interpretability and relevance to understanding continual learning in the brain. As stated in line 107-108, we decided to focus on common experimental paradigms, which are often modeled as having clear boundaries between epochs (see refs. on line 108). In related literature (e.g. [11], [13]-[16]), compositional reuse in multi-tasking RNNs has been of major interest. Our work is intended to contribute to this space, making the compositional task structure, and how it may shape a potential solution in the RNN, an explicit focus. Still, we wholeheartedly agree with the reviewer that it is important to evaluate our approach on more realistic tasks, spanning a wider domain. Analyzing the approach in settings with more ambiguous compositional structures will be particularly interesting. While we argue that this is beyond the scope of the current paper, we would like to highlight that the probabilistic treatment of the 'what' system may be especially well-equipped to handle these settings.
4. Ablations and ambiguous context boundaries. The 'what' system is designed to perform posterior inference of context, conditioned on the observables (inputs $\mathbf{s}_t$ and targets $\mathbf{y}_t$) of a given task, and, in principle, any architecture that can do this could be used as the context inference model. In our setting, ambiguity across epoch boundaries stems only from the temporal variability of each trial and observation noise. We agree that it would be worthwhile to further explore the setting where context boundaries are ambiguous due to other sources. For instance, this could happen in our tasks when stimuli change more slowly (e.g. when they are supplied as a ramping signal rather than turning on/off abruptly). Studying this task variation would test how robust our inference system is to a mismatch between the underlying generative assumptions, and the true task design, which is important. Given the uncertainty about context that would arise at these transition points, the 'how' system would express a mixture of weights rather than a single solution. We would expect that this would impact the task performance, and might encourage (or require) learning some form of normalization across weights. Currently, the system only experiences context-uncertainty during learning. We are planning to update the discussion section to include these points. Regarding the suggested ablation studies, we would appreciate additional detail on what exactly the reviewer had in mind.
5. Replacing the RNN with other models. The general framework of using probabilistic inference to 'decompose' time-series tasks is agnostic to the 'how' module used. Indeed we imagine a similar scheme to work for other architectures that deal with time series, such as transformers. For example, with transformers the equations in Eq. 3-5 would be replaced by analogous equations that determine the query, key, value weights used for input from different epochs. Incorporating the two-systems approach we develop here into more complex architectures represents an interesting direction of future work and may also facilitate further applications to more complex tasks.

---

Summary of additional baselines. Costacurta et al., 2024 [16] designed neuromodulated RNN (NM-RNN), in which a full-rank neuromodulatory RNN (RNN1) outputs a low-D signal that dynamically scales the low-rank recurrent weights of an output-generating RNN (RNN2). RNN1 and RNN2 are jointly optimized through gradient descent. Their design showed catastrophic forgetting during continual learning (Table 1). In contrast, our method replaces RNN1 with a probabilistic task model, which rapidly learns a task’s structure within a handful of trials, thus is able to provide useful contextual signals when optimizing RNN2 and facilitate continual learning. Von Oswald et al., 2020 [23] designed task-conditioned hypernetworks to tackle continual learning. Their hypernetwork receives a learnable embedding distinct for each task and outputs the weights for the target network (a RNN in our case). Their design underperformed our method (Table 1). More importantly, these hypernetworks do not capture the time-varying epoch structures of each task, and performed worse on compositional generalization (Table 2).

Implementation details.

NM-RNN: RNN2 has 256 neurons and with rank($W^{rec}$) = 27, equal to the default choice for our ContextRNN. RNN1 has 125 neurons such that the whole network has a similar number of parameters as ours.

Hypernetwork: The full-rank RNN has 256 neurons. We used a chunked hypernetwork with 34 chunks and 2000-D output. The hypernetwork is a MLP with two 32-D hidden layers, receiving 32-D task embedding and chunk embeddings as input. After a grid search, we set the continual learning regularization strength to 1, orthogonal regularization strength to 0.001 and clip the gradient norm to 1 (see Ehret et al., 2021 for details).

Table 1:

A. NM-RNN

B. Hypernetwork

C. ContextRNN (ours)

Caption: Continual learning performance (supplementary to Figure 4 a-d). A: Each column shows a task's final performance averaged across 5 random seeds (± std) after training NM-RNN sequentially on task 1 to task 6. B: same as A but for hypernetwork. C: same as A but for contextRNN (ours).

Table 2:

A. OWP

Caption: Compositional generalization (supplementary to Figure 6c). A: Each column shows the test performance on MemoryAnti averaged across five random seeds (± std) after training a pretrained RNN using OWP on MemoryAnti for 0, 20, 40 or 512 trials. B: same as A but for hypernetwork. C: same as A but for contextRNN (ours).

We appreciate that the reviewer thought that we made novel contributions to addressing an important open problem in the space of continual learning, and that the probabilistic treatment of tasks and context inference is a strength of the paper. The explicit suggestions for how to improve the paper and address the concerns are very helpful. In response, we have made the following changes:

1. Additional baselines. We have now included a comparison against the NM-RNN [16], and (suggested by another reviewer) hypernetworks [23, 31]. Neither method matched the performance of ours in terms of continual learning or compositional generalization. Details and performance metrics are provided at the end of this reply. We have updated Sec. 2 to discuss these works further and also expanded the baseline comparisons (Fig. 4) to include these two.
2. Explanation for Fig. 5b. The degree of loss increase varied across random seeds, and happened after the first step of gradient descent on the second task (our batch size equals 256). It could stem from the discrepancy between the first training batch and the test dataset, or that the model moved away from a local minimum of the second task's loss landscape.
3. Figure captions. cf. 'The image descriptions … the corresponding tasks.' We have incorporated this clarification into our captions.

Other concerns:

4. Contradiction between the precision and reusability of the epoch. This point seems to refer to the problem of identifying the correct segmentation of tasks into epochs. As the reviewer noted, it is possible to share stimulus periods, and/or response periods depending on how exactly tasks are segmented. Indeed, for a subset of the tasks we consider here (Memory/Delay Pro/Anti) both segmentations are equivalent. The reviewer’s concern, to our understanding, is that we chose a particular segmentation to better model neural activity, at the sacrifice of re-usability of epochs across tasks. To clarify: we infer the segmentation based only on the statistical structure of the task, not neural activity. In addition to this, sharing of the stimulus epoch only happens for tasks that share the same statistical structure during the stimulus epoch, and therefore does not limit the system’s ability to capture different dynamics when this is necessitated by the demands of the task. For example, the stimulus periods between the DM tasks, and the Delay/Memory tasks are not inferred to be shared. More generally, the tension between oversegmentation and undersegmentation is an issue common to mixture models (e.g., clustering, hidden Markov models), and our approach is no exception. Further clarification on this point by the reviewer would be much appreciated.
5. Discontinuous dynamics in a trial. This point is concerned with dynamics of the RNN and whether the model can capture 'dynamics such as sub-threshold evidence accumulation process'. While the inputs and target outputs are piecewise constant and sometimes discontinuous between epochs, the RNN hidden state (Eq. 1) is not constrained in these ways. The network can only respond correctly during response epochs using information contained in its hidden state. Within each epoch (for example during a stimulus epoch) it may exhibit ramping dynamics that are signatures of evidence accumulation. Our DM tasks are designed to emulate evidence accumulation: the network has to integrate noisy inputs to make a decision relating to the stimulus strength. Our choice of piecewise constant input plus noise follows classic models of evidence accumulation in the literature (e.g., Wang, 2002; Ratcliff et al., 2016). In summary, while the weights of the network switch across epochs, the network state evolves continuously and can exhibit complex activity patterns.
6. Limited supported task types. The reviewer was concerned that our choice of $x$ constrains us to tasks involving stimuli with angular information. Our approach, however, does not require a consistent interpretation of $x$ across tasks. In fact, among the 6 tasks we studied, $x$ determines the stimulus angle in only 4 of them (Delay/Memory Pro/Anti). In the two DM tasks, $x$ controls the stimulus location/strength (appendix lines 94-100). Thus, our task set demonstrates the applicability to a broader range of tasks. Regarding the comment on Fig. 2c ('multiple stimulus-response pairs under a single x within a task'): while $x$ is a single discrete variable taking on a single value per trial of a given task, the stimulus ($\mathbf{s}_t$) and response variable ($\mathbf{y}_t$) can be higher-dimensional, as is the case in our example (both are 3D, corresponding to the three lines in the figure). We apologize for the confusion and have updated the figure to clarify this.
7. Practicality. This point is concerned with the generation of the cognitive tasks and whether they can match the distribution of real biological data. While we are excited about connecting our model to neural data in the future, we want to clarify that at the moment this work is not concerned with modeling biological neural activity at all. The synthetic data, represented by $\mathbf{q}_t$, are meant to model the behavior-level input (stimuli on a screen) and output (motor actions, eye saccades) in neuroscience experiments. The simplified dynamics, where the inputs are piecewise constant, are consistent with typical experimental setups and are common assumptions in related modeling work (see refs. on line 108).
8. The reviewer suggested comparing to the setting where weights are modulated by task context c instead of epoch context z. This would effectively train a separate network per task, and therefore sidestep both the continual learning problem and any compositional reuse across tasks. This would make forward/backward transfer impossible. We are happy to include these comparisons, but would appreciate further clarification on what the reviewer would like us to show.

References:

Wang, X. J. (2002). Neuron, 36(5), 955-968. Ratcliff, R., ..., & McKoon, G. (2016). Trends in cognitive sciences, 20(4), 260-281.

---

Summary of additional baselines. Costacurta et al., 2024 [16] designed neuromodulated RNN (NM-RNN), in which a full-rank neuromodulatory RNN (RNN1) outputs a low-D signal that dynamically scales the low-rank recurrent weights of an output-generating RNN (RNN2). RNN1 and RNN2 are jointly optimized through gradient descent. Their design showed catastrophic forgetting during continual learning (Table 1). In contrast, our method replaces RNN1 with a probabilistic task model, which rapidly learns a task’s structure within a handful of trials, thus is able to provide useful contextual signals when optimizing RNN2 and facilitate continual learning. Von Oswald et al., 2020 [23] designed task-conditioned hypernetworks to tackle continual learning. Their hypernetwork receives a learnable embedding distinct for each task and outputs the weights for the target network (a RNN in our case). Their design underperformed our method (Table 1). More importantly, these hypernetworks do not capture the time-varying epoch structures of each task, and perform worse on compositional generalization (Table 2).

Implementation details.

NM-RNN: RNN2 has 256 neurons and with rank($W^{rec}$) = 27, equal to the default choice for our ContextRNN. RNN1 has 125 neurons such that the whole network has a similar number of parameters as ours.

Hypernetwork: The full-rank RNN has 256 neurons. We used a chunked hypernetwork with 34 chunks and 2000-D output. The hypernetwork is a MLP with two 32-D hidden layers, receiving 32-D task embedding and chunk embeddings as input. After a grid search, we set the continual learning regularization strength to 1, orthogonal regularization strength to 0.001 and clip the gradient norm to 1 (see Ehret et al., 2021 for details).

Table 1:

A. NM-RNN

B. Hypernetwork

C. ContextRNN (ours)

Caption: Continual learning performance (supplementary to Figure 4 a-d). A: Each column shows a task's final performance averaged across 5 random seeds (± std) after training NM-RNN sequentially on task 1 to task 6. B: same as A but for hypernetwork. C: same as A but for contextRNN (ours).

Table 2:

A. OWP

Caption: Compositional generalization (supplementary to Figure 6c). A: Each column shows the test performance on MemoryAnti averaged across five random seeds (± std) after training a pretrained RNN using OWP on MemoryAnti for 0, 20, 40 or 512 trials. B: same as A but for hypernetwork. C: same as A but for contextRNN (ours).

We thank the reviewer for their interest in our work and thoughtful comments. Concerns and questions raised in the review are addressed below.

1. More realistic tasks. At the present stage, we have chosen to focus on synthetic problems common in neuroscience modeling work for interpretability and relevance to understanding continual learning in the brain. The set of cognitive tasks has been of recent interest in the field (e.g., see refs. 11, 13, 14, 16, 35), and we followed this literature to use them as a testbed for our proposed architecture. However, we wholeheartedly agree with the reviewer that it is important to evaluate our approach on realistic time-series tasks and that NLP tasks would be particularly well-suited to the kind of compositionality we are concerned with here.
2. Scalability is indeed an important consideration for continual learning systems. In our case, letting $N_z$ denote the number of epochs (i.e., the ‘elementary tasks’), the probabilistic learning/inference algorithm (Sec. 3.3) used by the 'what' system would have time complexity $O(N_z^2)$ and space complexity $O(N_z)$, as is standard for learning/inference algorithms for HMM-like models. For the 'how' system, the main cost arises from running the RNN, which does not scale with $N_z$ (computing the effective weights in Eqs. 3-5 has cost linear in $N_z$). To improve scalability, a possible next step is to discourage the addition of new epochs by penalizing having a large $N_z$. Alternatively, it may also be possible to group epochs that are statistically distinct, but can share the same solution through another learning mechanism. This would be particularly relevant for the response epochs in our task set. It is also worth noting that, since each component of the network uses low-rank recurrent weights, the memory cost of adding a new component is linear in RNN size, as opposed to the quadratic scaling in vanilla RNNs.
3. Related work. We thank the reviewer for pointing us to the highly relevant Ostapenko et al., 2021. Our scheme differs from theirs in several conceptually important ways, as listed below. We have added reference to this work and incorporated a condensed version of this discussion into the 'Related work and our contributions' section (Sec. 2).
   1. Ostapenko et al. requires passing every datum to all modules and running the computation in all modules first to determine which module outputs to keep (their Eq. 5). 'Module selection' in our model (computing the effective weights in Eqs. 3-5) is determined by a separate inference algorithm that is independent of the RNN modules. Our proposed scheme is more efficient in settings where the number of modules is large and the cost of running each module is high.
   2. Since Ostapenko et al. deals with static inputs and feedforward networks, different components in the same layer can be run in parallel. In our case, tasks have temporal structures and are solved by recurrent networks. Importantly, the composition of network weights changes during each trial dynamically. Since the hidden state activity $\mathbf{h}_t$ is shared, different components thus interact with each other over time. Our approach allows for more flexible compositions of modules. For example, the same two modules can be used in the same trial in an alternating fashion.
   3. Finally, as the reviewer noted, our model provides an interpretable probabilistic description of the compositional structure of each task, which is more opaque in Ostapenko et al.
4. Graceful forgetting. This is an interesting question and not something our current learning algorithm is adapted to. We assume that the number of contexts across tasks does not exceed the capacity of the network (i.e. number of contexts $\times$ low-rank components per context $\le$ total dimensionality of the RNN), such that overwriting is not an issue. However, it would be possible to implement a mechanism that decays the association between context (what) and recurrent weights (how) over time, if a given context is not visited for a long duration. This would effectively free up the low-rank component to be overwritten and associated with a new context, and enable graceful forgetting. While this exceeds the scope of the current paper, it would be an interesting topic for future work.