Demystifying Language Model Forgetting with Low-Rank Example Associations

We thank the reviewer for recognizing the originality and quality of our paper and the insightful questions. We address individual questions from the reviewer below.

Q1: For the multiplicative model, do you have any insights on $\alpha$ and $\beta$? Which tasks cause more forgetting and why?

We thank the reviewer for the insightful question. $\alpha$ indicates how much in general a learned task causes forgetting; $\beta$ indicates how much in general an upstream example suffers from forgetting. There exist prior works that focus on characterizing examples that are easily or never forgotten (high or low $\beta$) in classification tasks. For example, [1,2] suggests that rare (but helpful) or mislabeled examples are more likely forgotten, while typical / easy examples are hardly forgotten. However, as we focus on forgetting of instruction tuning and language modeling data, we find the forgotten examples difficult to characterize. We will include case studies in the final version.

[1] Toneval et al. An empirical study of example forgetting during deep neural network learning, ICLR 2019

[2] Maini et al. Characterizing datapoints via second-split forgetting, NeurIPS 2022

Q2: While the fact that the forgetting association matrices are low rank, it is unclear what the takeaway from this is/how I should interpret or use this.

We thank the reviewer for the insightful question. SVD approximates the forgetting association matrix with sum of rank-1 matrices. The good fit of rank-1 approximation implies that how easily an upstream example is forgotten is more of an intrinsic property of examples that is not relevant to the learned tasks. The second, third, and the rest of components in SVD indicate correlation between learned tasks and forgotten examples (that is less dominant). We updated Sec. 4.2 and Appendix D in the paper analyzing interpretable patterns in forgetting as indicated by each component in SVD.

One takeaway of the low-rank approximation is that the patterns in forgetting are simple so that forgetting caused by unseen tasks can be well-predicted from statistics.

Q3. Should I expect forgetting associations to be high rank or should that be the null hypothesis? How should I interpret these R^2 values?

Theoretically grounded approximations of forgetting [1] and related works in linear models [2] suggest that forgetting depends on the similarity of two examples. It was however unclear how dominant the effect of example similarity is, and therefore the low rank approximation can be either good or bad. In this work, we demonstrate empirically that the association matrix $Z$ can be well-approximated with simple low rank models. For example, rank-3 approximation consistently yields R2 > 0.7. (70% of variance in forgetting can be explained by this low-rank approximation model).

Q4-1 : Can you show that forgetting isn't just a perplexity phenomena and actually impacts performance on benchmarks? Can this benchmark performance degradation be predicted?

We thank the reviewer for the suggestion. We added additional experiment results evaluating fine-tuned models over LLM leaderboard tasks, also added to Sec. 4.3 and Appendix C. We see forgetting happens on downstream tasks as well.

For OLMo-7B, we evaluated on the same set of downstream tasks in the OLMo technical report (ARC, Boolq, Hellaswag, Openbookqa, Piqa, Sciq, Winogrande) after fine-tuning on 8 dolly tasks. As expected, fine-tuning on Dolly improves accuracy on most downstream tasks, with exception on Sciq, where the accuracy decreased, indicating forgetting.

Similarly, for OLMo-7B-Instruct, we evaluate the model on MMLU-Pro, BBH, IFEval, MUSR, GPQA after fine-tuning on Dolly. We notice clear performance degradation on IFEval, where KNN reduces forgetting but not significantly.

We consider prediction of forgetting on benchmarks as an interesting future work that requires more time to experiment with.

Q4-2: …The reason this is important is because it is possible to increase perplexity on certain examples on the training set, especially if they are noisy/rare examples without hurting language model capabilities and it would be useful to understand this.

We totally agree with the reviewer that an additional process to filter out noisy, mistaken, or obsolete examples can be very beneficial and even necessary for continually updated LLM systems. We leave this research problem as a future work. In fact, we consider our analysis into forgetting may facilitate the research. We may, for example, prioritize upstream examples predicted to be forgotten when inspecting whether they are obsolete given the newly learned task.

Q5: How do you interpret the fact that kNN does better than SVD at predicting forgetting?

We appreciate the question from the reviewer. We performed an in-depth analysis into the performance issue of SVD. After extensive hyperparameter tuning, we notice that SVD could outperform KNN in predicting forgetting in a number of setups. We updated the results in Table 2, and also added results of replaying with SVD-predicted forgetting in Figure 6. We include the tuned results of SVD here.

We further notice that utilizing SVD-predicted forgetting slightly improves over KNN-predict forgetting in our example replay experiments (Figure 6) on OLMo-7B, where SVD clearly outperforms KNN in predicting forgetting.

Q6: It seems to me that predicting the association matrix is not really necessary for mitigating forgetting because the ground truth forgetting is easy to measure: you already have to fine-tune your model on the "test tasks" because you need to measure forgetting on the seed dataset.

We thank the reviewer for raising the issue. We added discussions about computation efficiency in Sec. 4.3 in the latest revision.

To summarize,

• Our previous approach enjoys most advantages when the upstream data is large-scale
• In the latest version, we added a new variant of the approach that is always computationally more efficient than evaluating ground truth forgetting, that achieves statistically significant improvement in 3 out of 5 setups.

Q4: The paper lacks substantive discussion of its limitations

We thank the reviewer for the suggestion. We updated the limitations paragraph in Sec. 6, more specifically, there are questions yet to be answered:

How does the complexity of associations depend on various factors such as model type or model size? What are more human interpretable patterns behind the forgetting? How can we extend the approach to a continual learning setup of multiple tasks?

Q5: For the replay mitigations in fig. 6, how was the kNN chosen rather than the additive linear or SVD methods? If these approaches were also tested, they should be reported.

We notice that, after extensive hyperparameter tuning, SVD could outperform KNN in predicting forgetting in many setups. We updated the results in Table 2, and also added results of replaying with SVD-predicted forgetting in Figure 6.

We further notice that utilizing SVD-predicted forgetting slightly improves over KNN-predict forgetting in our example replay experiments (Figure 6) on OLMo-7B, where SVD outperforms KNN in predicting forgetting by a large margin.

Q6: Did the authors consider attempting matrix completion using just an average over finetuning tasks, rather than fitting to all tasks separately? Demonstrating that the average-over-tasks vector is insufficient would strengthen the results in table 2.

We consider this to resemble additive linear models without a learnable parameter for each fine-tuning task ($z_{ij}=b+\beta_j$, instead of $z_{ij}=b+\alpha_i + \beta_j$). We will add the results in the final version of the paper.

Q7: Could the authors provide more detail on how they determined their significance testing results in fig. 6?

We performed a paired samples t-test to determine whether replaying predicted examples outperforms random examples on 20 held-out tasks. The performance of replaying random or predicted examples on one single task forms a single “pair”.

Q8: What are the fine-tuning tasks used for the mitigation experiments in section 4.2?

We consider this to be a clarity issue, as we have marked the tasks used as training/testing sets in Tables 6 and 7 in Appendix and left pointers in the beginning of Sec. 4.1. We improved the clarity of the pointers in the revision.

Q9: Issues in language and writing

We sincerely appreciate the suggestions from the reviewer. We try to avoid too significant writing changes in this rebuttal revision for consistency. We will carefully revise the writing in the next version of the paper.

We thank the reviewer for recognizing the importance of the problem and the thoughtful comments. We address questions from the reviewer below.

Q1: The results of this paper rest on the idea that there is a simple structure that explains which samples will be forgotten when training on a given downstream task. However, I don't see clear evidence that this is the case.

We understand the confusion of the reviewer. To be precise, our paper tries to show that forgetting associations can be well-approximated with low-rank matrices. The majority of our results rely on quantitative measures such as R2 scores of approximations. We will revise imprecise statements like “Z generally displays a neat and simple pattern” in the final version of the paper.

Q2: The authors admit that "the R^2 scores are relatively lower [on some models]" makes me question the robustness of this approach. Why should the results vary so significantly between models of the same scale (e.g. OLMo-7B and MPT-7B) or even within the same family, where we can assume the pretraining data is the same (e.g. OLMo-7B and OLMo-1B)?

We thank the reviewer for this thoughtful question. Although R2 scores are lower on some models, the robustness of the predicting example forgetting and mitigating forgetting is validated across different experiment setups, including OLMo-7B, where the R2 scores are the lowest. For now, we could not provide a clear interpretation why the patterns of forgetting are different among models. Understanding the influence of these factors (model size, model family etc.) on patterns of forgetting may require significantly more experiments, which we leave as future work. We added discussions in the updated Limitations paragraph in Sec. 6.

Q3: What do the SVD components in section 3.2, fig. 4, tell us that we did not already know from visualizing the full matrix, e.g. in fig 2c?

We thank the reviewer for the thoughtful question. Though some patterns are obvious from the visualization, some more interesting patterns can be extracted by analyzing the decomposition with statistics or clustering. We demonstrated this for OLMo-1B and OLMo-7B models in Appendix C, and added pointers in Sec. 3.2.

Some notable patterns include, for example,

• On OLMo-1B, the second component (k=2) in SVD highlights forgetting where upstream examples from the StackOverFlow domains are disproportionately more forgotten, validated with a z-test. Top associated learned tasks are flan/opinion_abstracts_idebate, dolly/general_qa and flan/story_cloze.
• On OLMo-7B, the second component in SVD highlights how NLI tasks (flan/mnli_mismatched, flan/mnli_matched, flan/snli) causes common upstream examples to be forgotten
• Some components have much stronger correlation with TF-IDF textual similarity up to 0.33.

We thank the reviewer for finding the research interesting and the analysis thought-provoking. We address the questions from the reviewer below.

Q1: It would be really helpful to add a description and possible analysis featuring what specific knowledge the examples cover, and thus the model forgets. Particularly if there is a correlation between forgetting and specific topics.

We appreciate the suggestion from the reviewer. We updated Sec. 4.2 and Appendix D in the paper aiming at analyzing how each component in SVD corresponds to interpretable patterns in forgetting, i.e., how certain learned tasks cause specific types of upstream examples to be forgotten in OLMo-1B and OLMo-7B models.

We summarize some interesting patterns, for example,

• On OLMo-1B, the second component (k=2) in SVD highlights forgetting where upstream examples from the StackOverFlow domains are disproportionately more forgotten. Top associated learned tasks are flan/opinion_abstracts_idebate, dolly/general_qa and flan/story_cloze.
• On OLMo-7B, the second component in SVD highlights how NLI tasks (flan/mnli_mismatched, flan/mnli_matched, flan/snli) causes common upstream examples to be forgotten

At this stage, however, we consider leaving a more comprehensive semantic interpretation of patterns in forgetting as future work. This paper focuses on analyzing statistical properties of such patterns (e.g. complexity of associations by testing goodness-of-fit), and demonstrates how such statistical patterns enable better mitigation of forgetting. We added discussions about this future direction in the Limitations paragraph in Sec. 6.

Q2. It is not obvious why predicting forgetting rather than running inference is much more beneficial. Perhaps, a further explanation and a direct comparison of the approximate costs associated with performing matrix completion compared to running inference could be included.

We thank the reviewer for this question. We added discussion about computation efficiency in Sec. 4.3 in the latest revision.

To summarize,

• Our previous approach enjoys most advantages when the upstream data is large-scale
• In the latest version, we added a new variant of the approach that is always computationally efficient while also performing competitively

We include more details below.

We let $FT(n)$ be the cost of fine-tuning on n examples, and $EV(m)$ be the cost of evaluating on m upstream examples, and MC be the cost of matrix completion algorithms. We let X and Y be the number of upstream examples and fine-tuning steps, and S be “seed” upstream examples for matrix completion where S << X (Sec. 4.1). The computational costs can be summarized as follows.

* We added the “online” variant of forgetting prediction in our latest revision, which we detailed in Sec. 4.2.

We understand the question from the reviewer about how the cost of fine-tuning (FT(Y)) compares to inference on upstream data (EV(X)). This determines how much computation can be saved via (offline) prediction of forgetting, compared to directly evaluating ground truth. In fact, we discussed the limitation (previously in the end of Sec. 4.3) and showed that the computation can be saved when the upstream data is large. This was the case in our OLMo-1B and OLMo-7B experiments, where we fine-tuned the model for 1,000 steps (with batch size 8), while there were 140,186 upstream examples.

Nevertheless, the online variant of our approach added in the latest revision is much more efficient than replaying with ground truth forgetting in any scenarios. The variants trades off accuracy in predicting forgetting for computational efficiency, and also achieves significant improvement compared to random replay in 3 of the 5 setups, as we discussed in the updated Sec. 4.3.

Q3: Are there particular reasons that the Spearman correlation is chosen over the Pearson correlation in Section 3.3? In addition, have you experimented with the correlation between forgetting and the cosine similarity of the textual representations (e.g., last token representation), which is an alternative to TF-IDF for textual similarity?

We thank the reviewer for the suggestion. Our earlier consideration was that Pearson correlation implicitly assumes linear relationship, which might not be the case. Nevertheless, in the new version, we reported both Pearson and Spearman’s correlation in Table 1 for completeness of the results. We see the trends are consistent between two types of correlations.

Additionally, we evaluated correlations between forgetting and cosine similarity of last layer token representation of OLMo-1B models.

The results are added to Table 1. Just as other similarity measures, we see a low correlation.

Q4: Since increasing the rank of SVD provides a better fitting of the forgetting (perplexity) matrix, different SVD components (k-th component) may have opposite patterns since only the summation of all components matters?

We understand the concern from the reviewer. However, one property of SVD (as in L239) is that rank-r reconstruction is the optimal for minimizing the reconstruction error $|| Z - Z_r ||_F$. Therefore, although there could be conflicting signs in different components, there will not be cases where two components completely cancel out. Nevertheless, we consider non-negative matrix factorization may be of interest to the reviewer and future readers, where all components are positive and thus more interpretable. We will add the results in the final version of the paper.

We also added more analysis into patterns captured by SVD in Appendix D.

Q5: Table 2, it seems that forgetting is more difficult to predict for instruction-tuned OLMo-7B-Instruct. Why is that so?

We thank the reviewer for the thoughtful question. At this stage, we plan to leave how various factors influence the complexity of the associations as a future work. We included the discussion in the updated Limitations paragraph in Sec. 6.

We thank the reviewer for recognizing the strengths of our paper that the topic is interesting and well-motivated. We address the questions from the reviewer below.

Q1. The final forgetting predictor using KNN over SVD reduces the importance of those discussions.

We appreciate the question from the reviewer. Following the question, we performed an in-depth analysis into the performance issue of SVD. After extensive hyperparameter tuning, we notice that SVD could outperform KNN in predicting forgetting in a number of setups. We updated the results in Table 2, and also added results of replaying with SVD-predicted forgetting in Figure 6. We include the tuned results of SVD here.

Our analysis in Sec. 3.1 and Sec. 3.2 with SVD try to analyze how complicated are the associations between learned tasks and forgotten examples. Like many other application scenarios of matrix completion, we believe simple or tractable patterns in Z is a necessary condition for these algorithms to succeed, which is not restricted to SVD. Following this, we believe there exist untested matrix completion algorithms that perform better. We will include more matrix completion algorithms in the final version as possible.

Q2. It is unclear whether reducing forgetting has practical benefits or trade-offs to these downstream tasks (evaluated with task-specific metrics) / How does the proposed forgetting-mitigation approach affect the performance on the downstream tasks (or even the upstream tasks) that are typically not evaluated by perplexity?

We thank the reviewer for this thoughtful question. We added additional experiment results evaluating fine-tuned models over LLM leaderboard tasks. These results are discussed in updated Sec. 4.3, and Appendix C.

For OLMo-7B, we evaluated on the same set of downstream tasks in the OLMo technical report (ARC, Boolq, Hellaswag, Openbookqa, Piqa, Sciq, Winogrande) after fine-tuning on 8 dolly tasks. As expected, fine-tuning on Dolly improves accuracy on most downstream tasks, with exception on Sciq, where the accuracy decreased (indicating forgetting).

We notice replaying examples predicted by KNN reduces forgetting, but we could not make claims about statistical significance.

Similarly, for OLMo-7B-Instruct, we evaluate the model on MMLU-Pro, BBH, IFEval, MUSR, GPQA after fine-tuning on Dolly. We notice clear performance degradation on IFEval, where KNN reduces forgetting but not significantly.

We include full results on other downstream tasks in Tables 3 and 4 in Appendix C.

To summarize, although we observe statistically significantly reduced forgetting on upstream instruction tuning or language modeling data evaluated with perplexity, we do not observe significant improvement in downstream tasks evaluated with task-specific metrics. We consider improving downstream task performance with predicted forgetting is a valuable future work.

We thank the reviewer for recognizing the strengths of our paper that the topic is interesting and well-motivated. We address the questions from the reviewer below.

Q1. Weak practicality under a large number of downstream tasks.

We agree with the reviewer that obtaining ground truth forgetting of upstream examples is computationally expensive. However, collecting training data is a one-time effort that allows forgetting of new tasks (from the test set) to be predicted efficiently. As we perform more runs of fine-tuning over new tasks or incrementally collected datasets, we may eventually reach a point where the overhead of collecting training data becomes worthy. In fact, our research into predicting example forgetting is motivated by reducing the computational cost of repetitively evaluating forgetting in large-scale scenarios, when LMs are fine-tuned for a diverse set of tasks.

We added discussions about the computational efficiency after Sec. 4.3 in the revision.

Q2. Weak practicality due to requiring full access to upstream data.

We understand the concern from the reviewer. It seems the limitation roots in replay-based algorithms to mitigate forgetting. Still, we focused on replay-based algorithms, because of advantages such as being scalable and easy to implement when data availability is less of an issue.

Q3. It is unclear whether the statistics gathered post-fine tuning without any replay will hold at every step of fine tuning with replay

We thank the reviewer for the insightful question. We measure log-perplexity of replayed and never-replayed upstream examples separately after fine-tuning the LM while replaying random examples.

For those very few replayed upstream examples, we notice the perplexity drops significantly as expected; for the majority of upstream examples that are never replayed, the perplexity decreases on average (which indicates reduced forgetting. Nevertheless, we find the perplexity of individual upstream examples with replay strongly correlates with their perplexity without any replay (0.98 Spearman correlation on OLMo-7B). Therefore, we consider the statistics gathered is still very informative regardless whether or not replay is performed. We leave more in-depth analysis as future work.

Q4. Insufficient performance gains of replay-based approaches

We thank the reviewer for pointing the issue out. Our main goal is to demonstrate the benefits of predicting example forgetting, as exemplified in replay-based methods. We are aware of alternative approaches to mitigate forgetting with a subset of training examples [1,2], for example, regularizing parameter updates or applying knowledge distillation, where predicting forgetting can play a role. We leave integration of predicted forgetting into these approaches as future work.

[1] Nguyen et al. Variational Continual Learning, ICLR 2018

[2] Li et al. Learning without Forgetting, ECCV 2016