What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement

We thank the reviewer for thoughtful and positive comments.

Q: Reporting actual runtimes for some example settings would also be helpful. E.g. tuning the head only should be faster than the whole model, so I think the asymptotic complexity only tells half the story.

We thank the reviewer for the suggestion. We agree that tuning the head is faster than tuning the whole model, and it is also more efficient to obtain ground truth forgotten examples when tuning the head only (assuming that pre-head outputs of upstream training examples can be cached). We discussed computation complexity of two cases separately in our initial version of Sec. 5.4, and concluded that our forecasting methods have better advantages when tuning the entire model.

We additionally obtained the number of FLOP (number of floating point operations) of forecasting methods as an empirical statistic that measures computational efficiency, which is agnostic to the hardware that we use. We experimented with FLAN-T5-large with 100 examples sampled from each of the 36 pre-training tasks while fine-tuning the entire model.

As a comparison, the FLOP of running inference on a single example is $2.51e^{11}$ flops. Without forecasting methods, obtaining ground truth forgotten examples requires about $9.02e^{14}$ flops. Our forecasting methods are much more efficient.

We have included the results in Appendix D and added pointers where we originally discussed computation complexity in Sec. 5.4.

Q: Why do the authors hypothesize the logit-based approach is less effective on FLAN vs BART?

We appreciate the question from the reviewer. In paragraph “Performance with LoRA or Full Fine-Tuning” in Sec. 5.1, we hypothesized that trainable logit-based forecasting has greatly simplified the dynamics of logits in Eqn. 2; for FLAN-T5, such a simplified model cannot fit the ground truth dynamics; in contrast, the learning dynamics of BART0 may be closer to a linear model and can be captured by our forecasting model. Empirically, we notice that FLAN-T5 models of all sizes (small, base, large, xl) present a similar behavior that logit-based forecasting is less effective. Therefore, we hypothesize that model size is not a key factor; instead, some details during the pre-training phase of these LMs may be crucial. We consider that dissecting factors that contribute to dynamics of logits during continual learning would be an interesting future work.

We thank the reviewer for thoughtful comments.

Q: Writing is quite poor for the methods section. Even after repeated reading, I could not understand how exactly h is being learned for the trainable logit based forecasting.

We thank the reviewer for bringing up the clarity issue. In the updated version, we added algorithm boxes for logit-based and representation-based forecasting methods to enhance clarity of the methods in Appendix C.

Regarding the question about how $h(x)$ is computed: the logit-based forecasting method requires a kernel $\Theta$ that measures similarity between two examples. The first-order analysis in the beginning of Sec. 3.2 suggests that $\Theta$ should be dot products between gradients of the parameters, which is computationally infeasible, because it requires output length * vocab size backward passes to obtain all required gradients. We therefore propose to approximate with a lower-dimensional learnable kernel $\tilde{\Theta}=h(x)h(x)^T$. Here $h(\cdot)$ is a trainable LM encoder (either BART0 or FLAN-T5, depending on our experiment setup) and $h(x)$ has a much smaller dimension than actual gradients; $h(x)$ is explicitly trained to predict logit updates with a margin loss objective in Eq. 3.

Q: Authors should clarify in Table 1 that they are predicting a minority class of forgotten examples in a binary classification setting. Hence F1 scores lower than 50% still make sense.

We thank the reviewer for the suggestion. We have updated the captions in Table 1 to highlight the skewed nature of the class distribution.

Q: Poor empirical results on real large scale models

Thank you for pointing out the issue. We also acknowledged in abstract that logit-based forecasting performs well on BART0 but not on T5 models. However, for the other proposed method, representation-based forecasting, the performance improvement is clearer when we replay forecasted examples, as we see in Table 4. For example, we reduce EM drop ratio from 1.489 to 0.301 on FLAN-T5 with LoRA tuning.

As a pioneering work on forecasting forgotten examples, we hope our findings can inspire future study on further algorithmic improvement.

We thank the reviewer for the prompt response and follow up comments.

Q: How many more samples would have to be replayed with random replaying, if we wanted to achieve a similar EM ratio like the proposed approach?

We thank the reviewer for suggesting this comparative study. We additionally show EM Drop ratio on FLAN-T5-large (Full FT) when we replay more mini-batches of upstream examples. In our default setup, we replayed 1 mini-batch every 10 update steps (Sec. 4.2), which corresponds to 3 mini-batches while learning a single online learning example (as we use 30 steps to learn an example).

Table: Fixing single errors (setups in Table 3)

Table: Continually fixing multiple errors (setups in Table 4)

When fixing single errors, increasing the number of replayed examples over a certain limit causes increased forgetting. Replaying more examples cannot achieve improvement brought by Replay w/ Threshold (0.024) or Replay w/ Representation (-0.026) in Table 3.

When continually fixing multiple errors, increasing the number of replayed examples consistently reduces forgetting. By comparing to the results in Table 4, the EM Drop Ratio % of replaying 3 mini-batches in Replay w/ Representation (0.582) is between that of replaying 6 to 15 mini-batches of random examples.

In the manuscript, we included the results and discussions above in Appendix E.2.

Q: It looks like the initial drop in EM itself on $D_{PT}$ was low, especially in Table 3. I would suggest the authors to choose a better problem setting where they show the efficacy of their approach.

We agree with the reviewer that the forgetting of FLAN-T5 models is small when updating models on a single online learning example. In some problem setups, e.g., when online learning examples represent conflicting knowledge, we may expect more severe forgetting.

However, our intention was to simulate a common setup where online learning examples and upstream data are not clearly conflicting. We see intriguing cases where learning one example causes forgetting of an irrelevant example (as exemplified in Figure 1). For this purpose, we did not go over manual construction procedures of online learning examples; instead, we just collected mispredicted examples from some standard NLP benchmarks. As our results indicate, LMs like FLAN-T5 models only forget <1% of upstream examples (Table 3) (which also makes forecasting more challenging due to class imbalance). Still, we see the forgetting accumulates over continual updates and reaches 1-3% of EM Drop with replaying (Table 4) and may potentially get worse with a longer stream of online examples.

After discussions with the reviewer, we consider it meaningful to extend to setups where forgetting is more severe in nature by (1) finding more dataset setups (2) longer data streams. Given the tight deadline, we have to leave them in future work. Still, we hope the results in the presented setup can be helpful to the community.

Finally, it is not our intention to bring extravagant claims. I apologize for misunderstanding the degree of the term “pioneering work”. I just wanted to show the novelty of the problem setup. Thank you for pointing this out this language use issue.

We thank the reviewer for thoughtful comments. We address the reviewer’s comments below.

Q: The experiments can benefit from some quantitative results about the computational efficiency, for example in Table 4, what is the overhead of the approach compared to replay w/random?

We thank the reviewer for the question. In the new version, we included the number of FLOP (floating point operations) of forecasting methods in Appendix (assuming 100 examples per pretraining task and full fine-tuning of FLAN-T5-large) and added pointers in Sec. 5.4. We believe the FLOP are good empirical statistics that complement theoretical complexity analysis already presented in Sec. 5.4.

As a reference, without forecasting methods, obtaining ground truth forgotten examples requires about $9.02e^{14}$ Flop.

Furthermore, the number of replayed examples is controlled across all replay-based methods in Table 4. Therefore, the results above are the only overhead of replaying forecasted examples compared to replay w/random.

Q: The authors describe several prior works on forecasting (albeit not in the LM space) it would be interesting to experimentally compare these methods to the proposals

We discussed several related works that analyze features of forgotten examples and learning dynamics in Sec. 6. However, none of them studies the challenge of “forecasting”example forgetting. We believe we are the first to propose the task, set up baselines, and evaluate the performance of forecasting.

Q: It would be interesting to know if the method and observations are applicable to classification problems or other continual learning domains

We thank the reviewer for the suggestion. In this paper we formulated all NLP tasks as sequence-to-sequence generation tasks (Sec. 2) and experimented on P3, MMLU datasets that consist of diverse NLP tasks. These tasks include classification problems (e.g. sentient analysis, natural language inference) that are answered in a sequence-to-sequence format. We believe it is an interesting future work to apply the approach in other continual domains, such as vision domains, given the increasing interest in model refinement in various scenarios.

We thank the reviewer for the thoughtful comments. We address the questions and comments below.

Q: The experiment results reveal that the improvement over the baseline is quite marginal

Thank you for pointing this out. We also acknowledged in abstract that logit-based forecasting performs well on BART0 but not on T5 models.

Nevertheless, the other algorithm we presented, representation-based forecasting, improves over baseline consistently. In fact, the performance improvement is clearer when we replay forecasted examples (Table 4). As an example, we reduce EM drop ratio from 1.489 to 0.301 on FLAN-T5 LoRA.

As a pioneering work on forecasting forgotten examples, we also hope our findings can inspire further algorithmic improvement.

Q: Are the predicted samples sensitive to the training order of the same set of samples?

We perform additional experiments to address the question from the reviewer. We notice that examples that are forgotten at the end of the stream are not sensitive to the order of the examples. We see the standard deviation of F1 scores of the forecasting methods are small as we alternate the order of training examples (over 5 different orders).

Q: How are the forgotten samples related to the training data, learning rate, etc?

We thank the reviewer for the question. Certain factors, such as learning rate, have a clear effect on edit success rate and EM drop ratio (forgetting) in continual model updates, as shown below for FLAN-T5-large (full fine-tuning).

When we evaluate the forecasting model trained in the default setup on other learning rate setups, we notice that the precision scores almost remain the same, indicating a common subset of examples are forgotten in three setups; while recall scores differ, because a great number of examples are forgotten only when using larger learning rates. We added the new results in the Appendix E.