Understanding Catastrophic Forgetting in Language Models via Implicit Inference

Thank you for the suggestions which have strengthened our paper! We see that your main concern is about the setup/assumptions. We appreciate your questions/concerns and hope that our additional experiments and clarification below convinces you that our setting is in fact quite broad, and does shed light on interesting aspects of possible performance regressions upon fine-tuning (which we broadly call catastrophic forgetting). Please let us know if you have any further questions or suggestions on how to present our findings appropriately.

The main assumption of the paper (Equation 7) is that the input space of different tasks overlaps, so the model has to do probabilistic inference on the task identity. … The ICL examples are purposefully designed to make it confusing by including two different instructions.

Per the author's theory, if the tasks' input spaces don't overlap, the model does not need to adjust the task probability g (as it would be already close to 1 for a good enough language model), and catastrophic forgetting would be much less if g does not need to be modified.

1. We would like to clarify that while the input space overlaps, the optimal mixture regression has close to zero “task ambiguity”---the optimal regressor can identify continuous vs discrete with near perfect accuracy even for a small number of exemplars. (Appendix C.5). However, the fine-tuned transformer is not optimal and performs imperfect task inference, which manifests as task ambiguity that is resolved via conjugate prompting.
2. We would also like to clarify that we prepend an instruction such as “Repeat the input” or “Capitalize every character” in our ICL experiments, so again, there is no inherent ambiguity. However, we do agree that this setup is somewhat artificial, and was intended as a sanity check that these phenomena do appear in large-scale real world LLMs as well.
3. We have added a new experiment that is in a more natural and commonly studied setting of forgetting. We consider Code LLaMa (LLaMa-2 fine-tuned on code) and see that there is a drop in performance on MNLI when compared to LLaMa-2, which can be recovered via conjugate prompting. We have more details below, as well as in Appendix D.

It can also be argued that when the input space of different tasks overlaps, the problem itself is ill-posed as there is not a single correct answer for inputs from the intersection. The failure of finetuned models thus may come from the ambiguity of the task itself rather than from catastrophic forgetting.

Even if the problem is ill-posed, we note that the pre-trained model has a consistent preference for performing the task of interest on the input space in our three setups (ridge regression, ICL, harmful generation). Conjugate prompting hints that this drop comes from changing the model’s task inference rather than degrading the capability. Regardless of initial task ambiguity, the fine-tuned model faces even worse ambiguity from its changed task inference.

…catastrophic forgetting on language models is most clearly manifested in supervised task finetuning (e.g., on GLUE) and unsupervised domain finetuning (e.g., Codex, Minerva) … It would be better to choose a typical setting for evaluating the effectiveness of the proposed method.

Thank you for this wonderful experiment idea! Following your suggestion, we consider Code-LLaMa vs LLaMa-2 evaluated on XNLI, a multi-lingual variant of the MNLI task from GLUE. We observe that Code-LLaMa displays catastrophic forgetting with an >8% accuracy drop on the English variant of this benchmark. However, with conjugate prompting into French, Spanish, and German, we see a less than 2% change in model performance, supporting our claim. In fact, we even see a slight increase in French and Spanish, possibly due to increased reasoning capabilities from code that are not suppressed by incorrect task inference Ma et al, 2023. In this scenario, using Code LLaMa in Spanish gets higher performance than using LLaMa 2 in English. We discuss these new results in detail in Appendix D.

RLHF for alignment is not a good example of forgetting: the goal of alignment is to make the model refuse to answer when given a malicious instruction, rather than making the model forget how to follow harmful instructions.

We took a very broadly construed definition of forgetting to mean any “performance regression” (intentional or not) when fine-tuning a model and agree that catastrophic forgetting is perhaps overloaded. We have toned down our claims of RLHF being an intentional form of catastrophic forgetting and discussed how decreased answering can be studied under our task inference framework. Our results, as you mention, demonstrate that the model does not completely forget how to answer and retains the capability in a way that can be recovered by conjugate prompting.

In section 2.5, what if the model is finetuned on a different set of discrete weights than D_disc? Using a different set seems better matching real-world finetuning scenarios.

We find that fine-tuning is commonly done for enhancing capabilities the model already has rather than endowing new capabilities since fine-tuning is usually not done on long tail data (i.e. instruction-tuning, alignment, domain fine-tuning, etc). Nonetheless, our experiments for $\alpha=0.1$ demonstrate how conjugate prompting helps even when the fine-tuning data is rarely present in the pretraining data.

How does the reduction in performance because of forgetting compare to the reduction in performance because of the language gap? Would it be worth trading forgetting for the language gap?

We acknowledge this valid limitation of our method. However, for our ICL, RLHF, and code experiments, we find that the drop in base performance is small enough to potentially justify conjugate prompting over using the fine-tuned model in English. We also believe there may be other transformations where the base performance is not as big of an issue.

Could there be other kinds of mapping in conjugate prompting other than translation?

Beyond languages, we test Pig-Latin and Leetspeak. For studying jailbreaking, Wei et al, 2023 suggest transformations such as ROT-13 cipher, Morse Code, or synonyms with different connotations to push prompts out-of-distribution.

—

Thanks again for these detailed comments, we would love to know if there are any other ways we could strengthen our paper!

We were wondering if there are any further questions or concerns we could clarify before the discussion period closes in two days. From your original review, it seems like the biggest concern was about the assumptions of our setting, especially as it relates to “catastrophic forgetting” in general. We appreciate your feedback and believe we addressed it in our response and additional real-world experiment. Please let us know if this addresses your concerns, and we’d be happy to discuss more in the remaining period of the rebuttal!

Thank you for the suggestions! We hope our responses and new experiments address any concerns, and we would love to know what else we can do to improve our paper!

We agree with the reviewer that there is a lot of interesting follow-up work in validating our hypothesis across model scales, tasks, and conditions on the data distribution. To touch on some of these aspects, we run additional experiments for a larger model (Appendix C.6) and less/more pretraining data (Appendix C.7) in the synthetic setup. We also now acknowledge this as a limitation.

We want to thank the reviewer for the suggestion to try less artificial tasks. In light of this, we validate conjugate prompting on XNLI, a multi-lingual variant of the MNLI task from GLUE. We observe that Code-LLaMa, a code-based fine-tune of LLaMA-2, displays catastrophic forgetting with an >8% accuracy drop on the English variant of this benchmark. However, with conjugate prompting into French, Spanish, and German, we see a less than 2% change in model performance, supporting our claim. In fact, we even see a slight increase in French and Spanish, possibly due to increased reasoning capabilities from code that are not suppressed by incorrect task inference Ma et al, 2023. In this scenario, using Code LLaMa in Spanish gets higher performance than using LLaMa 2 in English. We discuss these new results in detail in Appendix D.

We appreciate the reviewer’s suggestions to improve the clarity of our paper and have made the following modifications.

• We have rewritten the captions of Table 1 and Table 2 for additional clarity on what is being measured and reported. If there were any other descriptions that would benefit from additional clarity, we would love to address them as well.
• We have now cited the prior work alluded to on Page 3
• We now identify IF as instruction following at the start of Section 4.1

—

Thanks again for these detailed comments, we would love to know if there are any other ways we could strengthen our paper!

We were wondering if there are any further questions or concerns we could clarify before the discussion period closes in two days. We appreciate the feedback!

Thank you for the suggestions! We hope our responses and new experiments address any concerns, and we would love to know what else we can do to improve our paper!

The paper try to emphasize the difference between "forgetting" and "suppressing" of the pretrained capabilities. However this distinction might be spurious. For example, even though the English instruction tuned model may still be able to do an ICL task in Spanish, this does not mean that it did not "forget" how to do this task in English.

Whether the model solves the task prompted in English or Spanish, we say that it is using the same capability since the reasoning needed to perform ICL is independent of the language of the prompt. We consider “suppressing” to be when the model is capable of solving the task but doesn’t to separate out two possible worlds The first possibility is that, due to catastrophic forgetting, there’s no way to recover the lost ICL performance other than retraining the model to learn this capability again The second possibility is that the fine-tuned model can still perform ICL, but, due to catastrophic forgetting, won’t respond to the basic English input and needs to be prompted appropriately. We provide evidence for this scenario since conjugate prompting can partially recover the ICL performance. We would love to make this more clear in the paper and would appreciate any further suggestions on where we can best do so.

The synthetic task is too simple - different tasks there simply correspond to different linear weights, while pretraining vs finetuning are just different prior distribution of those weights. It might be beneficial to expand that - even in the regression setups - to e.g. higher order functions (quadratic, quartic etc), such that the transformer model's ability to generalize across a diverse set of tasks in pretraining, and how this could be changed by finetuning, can be tested better.

We find that linear regression and task inference is a surprisingly difficult task for a transformer to implement, as prior work provides large constructions dedicated to implementing matrix inversion, multiple steps of gradient descent, or task inference (Akyürek et al, 2022, Bai et al, 2023). Furthermore, the complex and unstudied task of mixture regression involves discrete regression (as discussed in Appendix B). We do agree with the reviewer that higher order functions could elucidate further interesting trends, and we think this would be interesting future work.

Similar to the synthetic task, what happens if at the instruction-tuning stage, the fine tuning task is multi-task trained together with the pretraining task with different mixing weights, how would this change the behavior?

This is a great suggestion - we expect the multitasking to reduce forgetting but not completely mitigate it. As per our paper’s mental model, we would need to balance the pretraining and finetuning weights so that the task inference is not skewed one way or the other. Unfortunately, we did not have the compute to test this on real world LLMs, where we stuck to using publicly available fine-tuned models. However, we hope our work inspires such experiments and new methods of fine-tuning (and conjugate prompting) to mitigate performance regressions when fine-tuning.

How do the behaviors observed scale with model size? Previous work (https://openreview.net/forum?id=GhVS8_yPeEa) has found that the forgetting becomes less of a problem when the model scales bigger, everything else being equal.

We introduce new experiments in Appendix C.6 to test what happens when we train on a larger model. We find that catastrophic forgetting still exists and conjugate prompting still helps. Interestingly, we find that they hold in similar magnitudes as well, showing how this problem is not resolved by simply scaling up. We also note that our real-world LLM experiments are on large scale models (such as ChatGPT3.5 and LLaMa). We encourage you to read our new section for more details!

We will also write a more detailed related works section, discussing other forgetting works such as Wang et al, 2023 which shows that forgetting in transformers can be mitigated through a two-stage fine-tuning scheme to prevent format specialization Ramasesh et al, 2022 which assesses forgetting and finds that models with more parameters or strong pretraining face less forgetting than their counterparts. Luo et al, 2022 which asses forgetting in continual fine-tuning for large language models such as LLaMa, Alpaca, and BLOOMZ.

—

Thanks again for these detailed comments, we would love to know if there are any other ways we could strengthen our paper!

We were wondering if there are any further questions or concerns we could clarify before the discussion period closes in two days. We appreciate the feedback, is there anything else we could add or discuss that might affect your score?

Would you like to include an explicit comparison of some metrics on "forgetting percentage rate", comparing the small model with the 2x model? The current presentation is hard for me to judge the scaling trend. Thanks!

Thanks for the new table! I think it helps but the question remains unclear - which is understandable given the short time frame.

For example, we don't see clear gains from the larger model size before finetuning. If we conclude from this that model scaling has no effect on this task, then we are risking a wrong conclusion. More experiments / scale variations might be needed to see any signal one way or the other about forgetting too. Similarly, the author did not address several of my other questions due to limited time and resources. Again I do believe that the authors had put in their best efforts given the small amount of time they had.

If I could give 6.5, I would.

Thank you for the suggestions! We hope our responses and new experiments address any concerns, and we would love to know what else we can do to improve our paper!

Is this phenomenon consistent with the further expansion of the dataset, or is it merely an incidental occurrence specific to some certain smaller models?

it remains unclear whether this phenomenon is consistent across larger-scale language models

We believe our findings are consistent across data, model scale, and various tasks as evidenced by our experiments across linear regression and natural language (with 7B LLaMa and ChatGPT 3.5). To better demonstrate this, we provide extra experiments for larger models (Appendix C.6) and less/more pretraining data (Appendix C.7). In all three settings, we find that catastrophic forgetting is still a problem and conjugate prompting still improves performance on the continuous distribution. When we scale models, the trends stay roughly the same, though the base performance of the larger pretrained model is slightly stronger on the discrete distribution. We encourage you to look at these new sections for more details addressing these concerns!

Are these differences in trade-offs related to the complexity or similarity of the tasks involved?

Though our hypothesis explains the behavior across all settings we consider, the exact trends will depend on confounding variables such as the complexity and similarity of tasks. We believe there is exciting future work in understanding the precise relationship between the fine-tuning data and the resulting model’s performance building off the viewpoint proposed in this work.

As the authors pointed out in the limitations section, this method would benefit from further validation and application in various domains and tasks involved.

We want to thank the reviewer for the suggestion. In light of this, we validate conjugate prompting on XNLI, a multi-lingual variant of the MNLI task from GLUE. We observe that Code-LLaMa, a code-based fine-tune of LLaMA-2, displays catastrophic forgetting with an >8% accuracy drop on the English variant of this benchmark. However, with conjugate prompting into French, Spanish, and German, we see a less than 2% change in model performance, supporting our claim. In fact, we even see a slight increase in French and Spanish, possibly due to increased reasoning capabilities from code that are not suppressed by incorrect task inference Ma et al, 2023. In this scenario, using Code LLaMa in Spanish gets higher performance than using LLaMa 2 in English. We discuss these new results in detail in Appendix D.

—

Thanks again for these detailed comments, we would love to know if there are any other ways we could strengthen our paper!

We were wondering if there are any further questions or concerns we could clarify before the discussion period closes in two days. We appreciate the feedback, is there anything else we could add or discuss that might affect your score?