Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training

Thank you for the detailed feedback.

Re: Unfair comparison for catastrophic forgetting and anticipatory recovery.

We agree with the reviewer that there is no catastrophic forgetting if all tasks are trained in every epoch, or if training passes over all tasks many times. However, in our work we show that the forgetting and recovery can be observed within each training cycle, before the same document is encountered again. This is a different timescale than the reduction of forgetting in typical multi-epoch training, and is a surprising phenomenon. Prior to our paper, the accepted wisdom in the continual learning community was that catastrophic forgetting would arise within each cycle as we train on other tasks before training on the focal task again. We show an important case where the catastrophic forgetting effect is largely mitigated due to the anticipatory recovery towards the end of the training cycle, which challenges the accepted wisdom.

Re: “The anticipatory recovery phenomenon may have limited practical implications.”

We agree with the reviewer that for cyclic training, compared to random shuffled training, less forgetting of the next few examples comes at the cost of more forgetting of the other examples in the dataset. However, we argue that what real-world agents care about is the performance on tasks they actually encounter at the times they encounter them. If task order is (approximately) repeating, then a bias toward good performance on upcoming tasks at the cost of other tasks is a good thing, as shown in figure 1c of the paper.

Re: “Why the prequential evaluation setting is relevant, and why lower loss in prequential evaluation implies potential of structured training.”

As we explain in the general rebuttal, we introduce cyclic training not as a method but as an objective. Because many natural environments are quasi-cyclic, it is important to evaluate existing methods in this setting, as opposed to traditional CL setups. For an agent that acts and learns in the real world, the most important metric it cares about is its performance on the tasks it actually encounters, at the times it encounters them. This justifies the relevance of prequential evaluation in building and evaluating such agents. According to the prequential view, training and testing are two sides of the same coin. While it is correct that the model is about to be trained on the next task, its loss on the next task is determined by how good its predictions are prior to the update. Lower loss in prequential evaluation suggests that certain models can automatically enjoy an anticipation benefit when trained on structured sequences (where task sequences tend to repeat) and catastrophic forgetting might be less of an issue for these models in these structured environments.

Re: “It is not clear how the findings can be generalized to other forms of structured training.”

In addition to pure cyclic training, we also experimented with some variants in the paper, including data randomization (Figure 4, lines 148-154) and partial random shuffling (Figure 12, lines 602-608), and demonstrated anticipatory recovery in these cases. In rebuttal experiment E3, we experimented with another variant where only the first 20 documents are kept fixed in each epoch, and a random number of other documents are inserted in between. This new setting generalizes cyclic training in that (1) rather than having the same documents in every epoch, we insert random other documents between every repetition (2) epochs can have different lengths. We still observe anticipatory recovery for documents 2 through 20 in this setting, suggesting that anticipatory recovery exists as long as there is a repeating sub-sequence in the data stream.

Re: “Why is the particular toy model chosen in the theoretical analysis.”

The particular toy model is chosen for theoretical analysis because it is a minimalist setting where we can reproduce the anticipatory recovery phenomenon and each component in the toy model has a natural and intuitive counterpart in the more complex LLM training procedure. The L2 loss function is chosen due to its wide usage in machine learning applications. We believe that similar analysis can be done for other common loss functions such as the cross entropy loss as well.

Thank you for the detailed feedback.

Re: “In Figure 1(b), how exactly is the loss computed?”

The loss in figure 1(b) is computed by replicating figure 1(a) on each document in the training sequence, and re-aligning these curves so that 0 on the x-axis always represents the moment before the first occurrence of the focal document. For example, if the length of the sequence is 50, then for the 0.5th epoch, we take the loss of document 1 after training on document 25; the loss of document 2 after training on document 26; …; the loss of document 50 after training on document 24 of the next epoch; and average these losses.

Re: “Figure 5 needs legends.”

The legends in figure 5 are in the bar plots. For example, in figure 5(a), the light blue curve in the left subfigure corresponds to the light blue bar in the right subfigure, which shows the loss curve for 25 documents.

Re: “Can you give more context on the prequential evaluation?”

Yes, prequential evaluation is the same as looking at the training loss of each batch. We argue that the performance on the upcoming task is what matters for an agent which continuously acts and learns in the real world, and prequential evaluation measures this performance. We will explain this better in the revision.

Re: “For figure 8(a, b), if you reshuffle the data sequence, the same structure will show.”

Yes, the data sequence is randomly selected at the beginning and fixed for cyclic training. The same structure will show for any random data sequence, provided it is the same on every cycle.

Re: “Around line 273-275, is the argument theoretically justified?”

Let $z_1 = Px_{i+1}$ and $z_2 = f_{i+1}(f_i^{-1}(Px_{i}))$, and assume that $f_{i+1}^{-1}$ is Lipschitz continuous. By the definition of Lipschitz continuity, we have $f_{i+1}^{-1}(Px_{i+1}) - f_i^{-1}(Px_{i}) = |f(z_1)-f(z_2)| \leq K|z_{1}-z_{2}|$ for some positive real constant K, so when we reduce $|z_{1}-z_{2}|$ we also reduce the upper bound for $f_{i+1}^{-1}(Px_{i+1}) - f_i^{-1}(Px_{i})$. Although this argument does not directly prove that $f_{i+1}^{-1}(Px_{i+1}) - f_i^{-1}(Px_{i})$ will necessarily decrease, we think that proving the decrease of its upper bound serves as sufficient justification for this computational toy model.

Re: “This seems to suggest that pretraining is one of the most important factors.”

Thanks for the good suggestion. In fact, GD exhibits anticipatory recovery without pre-training as well, especially in the later epochs. That said, more pre-training does make anticipatory recovery more significant. In rebuttal experiment E1, we took Pythia models pre-trained for a different number of steps (6K, 12K, 24K, 48K, 96K), and we found that more pre-training does give rise to higher anticipatory recovery (see Figure 1 of the rebuttal PDF). We hypothesize this result fits a broader pattern in which the strength of the anticipatory recovery effect is related to how well the model can fit each successive training task (see section 3.3). Models with more pre-training steps are more capable of fitting each successive training task, and therefore exhibit higher anticipatory recovery. We plan to further study the effect of pre-training in future work.

Re: “Can you explain more about the significance of Figure 8(c)?”

For this figure we used $T=25$. We want to note that, as seen from the equidistant stripes with interval $T$ in figure 8(c), the weight residuals at step $i$ have high similarity with those of $i+T$ and $i+2T$, while there is no overlap in their averaging window. For example, the weight residuals from step 25 and step 50 are highly correlated (with cosine similarity close to 1), but their averaging windows have no overlap at all; on the other hand, the averaging windows of step 25 and step 35 have roughly 50% overlap but the cosine similarity of the weight residuals at step 25 and step 35 is negative. Hence the cyclic structure is not due to overlaps in the averaging window. The significance of figure 8(c) is that, along with figure 9, it reveals the temporal structure of model weights in the cyclic training process.

Thank you for the detailed feedback.

Re: “its practical value remains unclear.” and “cyclic training appears less natural compared to continual learning”.

We would like to argue the opposite, that traditional continual learning is less natural than cyclic training. Most standard benchmarks and methods in continual learning (CL) assume IID task sampling, or every task trained only once; and the objective is to perform well on all tasks under these assumptions. Both the assumptions and the objective are unnatural: tasks naturally repeat in day-to-day life, with predictable transitions (e.g., a person might always eat breakfast after brushing teeth) and the main metric that an agent which acts and learns in the real world cares about is its performance on the tasks it actually encounters, at the times it encounters them (which justifies the prequential evaluation experiment in the paper, Table 1). As a result, cyclic training is closer to the challenges facing real-world agents than standard class- or task-incremental continual learning.

In this paper, we are not trying to argue that cyclic training performs better than random training in terms of average performance on all tasks. Instead, we argue that, when training machine learning models on naturalistic sequences (where task sequences tend to repeat), certain models automatically enjoy an anticipation benefit. The experiments in our paper aim at identifying what factors influence the anticipatory recovery effect, so that we can use the right models and optimizers to maximize this effect.

Re: “[the experiments] are somewhat narrow in scope”.

While many of the experiments are conducted on Pythia models and the CNN/Daily Mail dataset to identify the influential factors of the anticipatory recovery effect, we also carried out many experiments with other models and datasets, including vision models (Figure 7, Section 3.4), the GPT-2-Large model (Figure 17, Appendix B.7), and wikitext-103 dataset (Figure 18, Appendix B.8). With these experiments, we are confident that anticipatory recovery is a general phenomenon and not specific to the model or dataset we are using. We also want to point out that Pythia uses a very generic transformer architecture, and many experiments in the paper (Figure 2b, Figure 3, Figure 6a, Figure 14, Figure 15) uses randomly initialized Pythia models and is therefore not particular to the specific pre-training recipe of Pythia.

Re: “lacks citations for a substantial body of work on continual learning in LLMs”

Thanks for pointing out these references. We will discuss these relevant works in the next version of the paper.

Thank you for the detailed feedback.

Re: Writing improvement and summary of main findings.

We have a summary of main findings in the final paragraph of section 1 but we agree some elaboration will help readers anticipate what is coming. We thank the reviewer for the writing feedback, and will incorporate these improvements proposed by the reviewers in the next version of the paper.

Re: Q1: The analysis of Adam optimization and cosine decay learning rate schedules.

We report an experiment comparing Adam to SGD which shows Adam produces a stronger anticipation effect in Figure 6 (lines 145-147) of the paper. We hypothesize this result fits a broader pattern in which the strength of the anticipatory recovery effect is related to how well the model can fit each successive training task (see Section 3.3). In rebuttal experiment E2 we experimented with the cosine learning rate schedule and verified that the model still exhibits strong anticipatory recovery.

Re: Q2: Total amount of data and the one-pass very large data setting.

We report the effects of the total number of tasks in Figure 5a (lines 128-130) of the paper, and found that the anticipatory recovery effect is more significant with a higher number of documents in the sequence. The one-pass very large data setting should produce no anticipation, similar to what we find when the task order is fully reshuffled on each pass in figure 1a.

Re: Q3: Actionable insights to follow up from this study.

Our paper challenges the conventional wisdom in the continual learning community on when catastrophic forgetting occurs: we suggest that for certain models, catastrophic forgetting can be largely mitigated when training on structured sequences. As argued in our general rebuttal, natural environments often have predictable transitions, and cyclic training is the most extreme version of this type of sequential structure. We therefore advocate for more research in structured training and rethinking classic continual learning algorithms in structured sequences, and using prequential evaluation as an important metric. We can also study the types of structures that achieve best learning performance and efficiency, which could contribute to designing good learning curricula and active learning methods.

Re: Limitations.

We thank the reviewer for pointing out these limitations. We will mention these points in the limitations section in the next version of the paper. We also want to point out that, while many of the experiments are conducted on Pythia models and the CNN/Daily Mail dataset to identify the influential factors of the anticipatory recovery effect, we carried out many experiments with other models and datasets as well, including vision models (Figure 7, Section 3.4), the GPT-2-Large model (Figure 17, Appendix B.7), and wikitext-103 dataset (Figure 18, Appendix B.8). With these experiments, we are confident that anticipatory recovery is a general phenomenon and not specific to the model or dataset we are using. We also want to point out that Pythia uses a very generic transformer architecture, and many experiments in the paper (Figure 2b, Figure 3, Figure 6a, Figure 14, Figure 15) use randomly initialized Pythia models and are therefore not particular to the specific pre-training recipe of Pythia.

I took some time to go through response to my review and the reviews by the other reviewers. I really appreciate the thorough response to my concerns and particularly value the new experiments analyzing the cosine decay learning rate schedule. I feel that this addition has improved the paper and made me even more confident about its relevance to the community. I thought a lot about it and decided to increase my score to an 8. Looking through the rating descriptions, the novel ideas aspect stood out to me.