Dissecting learning and forgetting in language model finetuning

We would like to thank the reviewer for the insightful comments and suggestions. We have revised the paper incorporating many of the suggestions. Please let us explain our responses to the reviewer's comments below:

Hidden biases of synthetic data

We agree with the reviewer that text generated with ChatGPT can have hidden biases, and we mentioned some forms of possible bias in the limitations of the paper. It would make the results of our paper more robust if we had natural data, but unfortunately the data we require for controlled-variable analysis (e.g., two document having the same content but different styles) is unlikely to be available in natural corpora. The advantage of using synthetic data is precise control on the components on text, so that we can have good precision in locating the variables we want to study which increases the reliability of our results.

Generalization to other characteristics of text

The decomposition of text into generating factors is not unique, as long as the decomposition of $p(x)$ into conditional probabilities corresponds to a valid model of text-generation process. For example, we could decompose text into a semantic part and a syntactic part, suggested by one of the reviewers. Such a decomposition would corresponds to a model of text generation where the semantic information is first determined, then the syntax of the document is determined based on the semantic information.

One would find more ways to decompose text for studying its various characteristics from linguistic theories. For example, stylistics separates the style of text into linguistic features such as vocabulary, syntax, and figurative language. Narratology separates the narrative of text into components like plot, characters, and narrative perspective. For specific types of text, we can define more components to analysis than general text.

As long as the separation of one characteristic from the rest of the text is clear enough and meaningful, we could apply our dissection analysis method to analyze how the characteristic is learned in finetuning. More precisely, "separation" means that the chosen factor has a degree of freedom that is independent of other factors, so that modifying it while keeping other factors still result in valid and coherent text. For example, the plot in a narrative text can be changed while keeping the characters and narrative perspective the same.

Typo in the paper

We thank the reviewer for pointing out the typo and we have corrected it in the revised paper.

Generalization to other factorizations

We answered this question in combination with the previous question and provided our response above.

Other paper updates

We also kindly ask the reviewer to refer to the "comments to all reviewers" section at the top of this page for a list of newly added contents in the revised paper. We included theoretical support of our methods, new domain corpora and more knowledge evaluation tasks to generalize our findings. We hope that the revised paper is getting even more clear and complete with the help of the reviewers.

We would like to thank the reviewer for the insightful comments and suggestions. We have revised the paper incorporating many of the suggestions. Please let us explain our responses to the reviewer's comments below:

Decomposition of text-generating factors

We could find theories from linguistics and literary analysis that endorse the approach of decomposition of text into content and style. Separating the content (what is being said) and form (how it is being said) has been a traditional approach in literary theories (Eagleton, 2011). Content analysis deals with themes and the narrative while form (style) analysis studies the use of literary devices like metaphors and the organization of the text.

Some linguistics theories further support decomposing the content into an overall topic and specific, detatiled information. For example, topic-focus articulation (Sgall et al., 1986) distinguishes between the "topic" of a sentence (what the sentence is about, or its theme) and the "focus" (new or important information in the sentence). Theme-rheme analysis (Halliday, 1994) divides a sentence into "theme" (the departure point of the clause, what it's about) and "rheme" (the rest of the clause, what is being said about the theme). It can also be extended to larger text structures, where the overall theme of the text is distinguished from specific, detailed information.

In machine learning and NLP, there are many work that study the three aspects separately. For example, topic modeling (Hofmann, 1999; Blei et al., 2003) studies the topic distribution of text in a corpus, style transfer studies manipulation of the style of text (Shen et al., 2017; Fu et al., 2018) (and also how to separate style from content (Fu et al., 2018)), and information extraction (Brin, 1998; Banko et al., 2007) studies the identification of factual information from text. Also, in document modeling, several work uses a hierachical structure to separately model the overall theme and specific information (Lin et al., 2015; Li et al., 2015; Nawrot et al., 2022), in a similar spirit as we did in this work.

The citation and discussion on the above mentioned work can be found in the Related Work section of the revised paper. We hope that we provided sufficient evidence from past work for justification of the decomposition used in our approach.

A summary of contributions with better focus

We have rewritten the summary of contributions in the introduction section to better highlight the main contributions of the paper. The main contributions of our paper are the following two observations:

• Domain finetuning leads to a significant change in the topic and style priors of the language model, biasing them towards the training data. Effect caused by such bias dominates the learning and forgetting observed in finetuning.

• Topic and style biases are learned like simple features, while factual knowledge are learned like complex features in finetuning. We present the significant differences between their learning dynamics in multiple aspects.

The details of each observation are succinctly summarized under the two bullet points at the end of the introduction section. We hope that the revised introduction can help readers better grasp the main contributions of the paper.

We would like to thank the reviewer for the insightful comments and suggestions. We have revised the paper incorporating many of the suggestions. Please let us explain our responses to the reviewer's comments below:

Corpora for evaluation

We included two new domains, the legal domain and the customer review domain, in our analysis to verify the generalization of our finding. For the legal domain, we use the "Court Listener Opinions" subset from the Pile of Law corpus. For the customer review domain, we use the "automotive" subset from the Amazon reviews dataset. We use the same procedure as in our main study to finetune LLaMa 2 7B model, generate derived datasets, and probe the model accordingly.

We show (in Figure 9-10 of the revised paper) that similar to the biomedical domain, for the legal and the customer review domain, likelihood of the dominant topic and style in the corresponding training corpus also increases significantly during finetuning. For all the domains, the factual/counterfactual likelihood ratio changes at a significantly slower rate than the topic and style likelihood ratios, showing that the effect of topic and style adaptation on langauge modeling modeling are much more significant than the effect of knowledge learning. This shows that our main findings are likely general phenomena in domain finetuning.

The above results and discussions are included in the Appendix D of the revised paper.

Separability of text-generating factors

We agree that the text-generating factors may not always be clearly separable in every corpora, and this poses a limitation on the scope where the dissection analysis can be directly applied. However, we believe that the general idea of decomposition of text into simple features (such as topic, style) and complex features (such as factual knowledge) is still useful in understanding the learning dynamics in domain finetuning even in cases where the decomposition cannot be explicitly performed.

Improving factual knowledge learning

While the current study mainly aim to uncover the learning dynamics in domain finetuning, we believe that by identifying bias learning as a potential hindrances in knowledge learning and showing the properties of bias learning, we also pointed out potential directions to improve knowledge learning. We can use the discovered differences between bias learning and knowledge learning to design methods that encourage knowledge learning. For example, based on the observation that bias learning mostly happens on the first few tokens of the sequence, we could mask out the loss on the first few tokens in the finetuning objective to considerably reduce bias learning. Based on the different capacity requirement of bias and knowledge learning, in principle we could use a small low-rank adapter to learn the bias, and subtract its weights from the full finetuned model to remove the bias while keeping the learned knowledge.

We add a part of discussion on potential directions to improve knowledge learning in the conclusion section of the revised paper.

Significance of our findings

Our findings starts from a simple intuition, but leads to an unexpected result, detailed understanding of different components in domain finetuning, and wide implications for future work.

We agree with the reviewer that topic and style are very simple features of text. We pointed out that given existing study on spectral bias and shortcut learning, it is expected that topic and style are learned faster than factual knowledge. This is the starting point of our study.

With a dissection analysis, we found that the topic and style biases not only learns fast, but the degree of adaptation of topic and style priors is very significant, much larger than the degree of adaptation to factual knowledge. We think this high degree of biasing towards certain topic and style is not quite expected, as one could reasonably speculate that because topic and style are so salient and easily determinable, the model can easily recognize them during inference and does not need to adopt a strongly biased prior.

We showed that the implication of bias learning is that it affect how we interpret loss/perplexity changes in finetuning. Large reduce in perplexity does not necessarily mean significant learning of new domain knowledge as we would often assume. Large increase in perplexity on general corpus also does not equal to catastrophic forgetting of factual knowledge. We also need to take care of the bias in finetuned models in applications, as they could potentially be too heavily biased towards certain topic and style in generation which may negatively impact application in real-world scenarios.

As our second main contribution of the paper, we showed that there are many more different properties between bias learning and knowledge learning besides the learning order. We showed that the two types of learning have different capacity requirements, different sensitivity to learning rate and data composition, and affect modeling probabilities at different token positions. Also the topic and style biases are learned independently of each other. These results are not found in previous literature and we believe they are not obvious derivatives from the nature of domain finetuning.

We also point out (in the revised version of the paper) how these new insights into the fine-tuning process can be useful and its implications for future directions. We can use the discovered differences between bias learning and knowledge learning to design methods that encourage knowledge learning and reduce forgetting. For example, based on the observation that bias learning mostly happens on the first few tokens of the sequence, we could mask out the loss on the first few tokens in the finetuning objective to considerably reduce bias learning. Based on the different capacity requirement of bias and knowledge learning, in principle we could use a small low-rank adapter to learn the bias, and subtract its weights from the full finetuned model to remove the bias while keeping the knowledge. We believe that more understanding of the detailed learning dynamics in domain finetuning under a systematic analysis is very much needed as a basis to explore more finetuning techniques beyond the conventional causal language modeling recipe.

We would like to thank the reviewer for the insightful comments and suggestions. We have revised the paper incorporating many of the suggestions. Please let us explain our responses to the reviewer's comments below:

The necessity of separating the topic factor

Separating the topic factor reflects a natural generation process of text, and allows us to study the many properties of topic bias.

We agree that the topic of a document can be determined by the factual knowledge it contains. However, considering the natural generation process of text (i.e., how a paragraph is written), it is often the case that the overall topic is determined first, then the factual information to include in the paragraph is selected. Our decomposition of $p(x)$ into $p(topic) p(factual|topic)$ reflects such a natural generation process, where $p(topic)$ models how likely a topic is chosen in the first step, and $p(factual|topic)$ models how likely certain factual information is included once the topic is chosen. As the reviewer pointed out, the alternative decomposiiton of $p(x)$ into $p(factual) p(topic|factual)$ will not make sense becuase $p(topic|factual)$ is deterministic.

There are linguistics theories that support the decomposition of text into an overall topic and specific, detatiled information. For example, topic-focus articulation (Sgall et al., 1986) distinguishes between the "topic" of a sentence (what the sentence is about, or its theme) and the "focus" (new or important information in the sentence). Theme-rheme analysis (Halliday, 1994) divides a sentence into "theme" (the departure point of the clause, what it's about) and "rheme" (the rest of the clause, what is being said about the theme). In machine learning and NLP, there are also prior work that endorse a hierachical decomposition of paragraphs which fascilitate analysis of text at different levels of semantic granularity (Lin et al., 2015; Li et al., 2015; Nawrot et al., 2022). We have included a new subsection in the Related Work section of the revised paper to discuss the above mentioned work, for a better justification of the text decomposition method used in our approach. (The citation and discussion on the above mentioned work can be found in the Related Work section of the revised paper.)

Separating the topic factor from factual knowledge also allows us to study the many differences of the learning dynamics between topic bias and factual knowledge, which constitudes one of the two main contributions of our paper. We show that the topic biases and factual knowledge are learned very differently, which also evidences that to a certain extent the language model internally models topics separately from factual knowledge. It might have been harder to gain these insights if the topic factor is not separated from factual knowledge in the analysis. We have also revised the introduction section to better highlight the main contributions of the paper.