Overtrained Language Models Are Harder to Fine-Tune

Thank you for your detailed feedback! We are happy to hear your positive comments such as that our methodology is “appropriate, comprehensive, and designed [...] rigorously”, that our paper addresses a problem that is “highly relevant and timely”, and that our results are “novel and counter-intuitive, challenging conventional wisdom”.

Given the simplifications involved in the linear models, the theoretical results supply intuitions about the mechanism, rather than directly supporting the observed results

We aren’t aware of any theoretical tools that would enable us to study the dynamics of LLMs without at least some degree of simplification such as the ones we describe. We fully acknowledge that there are limitations that arise from these simplifications but we hope our analysis is a starting point to understand this phenomenon that may lead to principled mitigation strategies in the future.

Additionally, since catastrophic overtraining is so counterintuitive (to us), seeing it emerge from incremental learning in a simpler setting reassures us it's not just an artifact of our LLM setup and offers intuition for a possible mechanism.

The claim for $L_{\mathrm{pre}}$ (base model capabilities) is a bigger and less definitively backed claim than the claim for $L_{\mathrm{ft}}$ (fine-tuning performance).

We want clarify our claim regarding how extending pre-training can reduce the base model capabilities:

Our core finding is that there are some important settings where additional pre-training hurts the fine-tuned model’s capabilities, despite helping the base model’s performance. We don’t intend to claim that all settings necessarily exhibit catastrophic overtraining after hyperparameter tuning.

In Figure 2, we find that the fine-tuned model’s “general capabilities” (blue lines) are degraded for PIQA, ARC Challenge, ARC Easy when instruction tuning, and PIQA, ARC Challenge when multimodal tuning. In Figure 3, we find that the fine-tuned model’s “general capabilities” (bottom) degrade for GSM8k, Starcoder-Python, MR, and RTE.

Hopefully this clarifies our evidence that the general capabilities can degrade with overtraining.

Beyond regularization, are there other potential techniques that could mitigate catastrophic overtraining and improve the fine-tuning adaptability of overtrained models? For example, could techniques like parameter resetting, learning rate scheduling adjustments during fine-tuning, or architectural modifications be explored?

Great question! We do explore various learning rate schedules during fine-tuning (constant, constant + warmup, cosine schedules), as detailed in Appendix C.2, and found catastrophic overtraining persisted in each case. Exploring additional mitigation techniques (e.g., parameter resets, architectural modifications) would indeed be valuable future work.

In the "optimal learning rate trend" analysis (Section 3.4, Figure 5): You categorize datasets based on whether the optimal learning rate is constant, slowly decreasing, or quickly decreasing. Could you provide further intuition or hypotheses about why certain datasets exhibit each trend? What characteristics of the datasets or fine-tuning tasks might determine the optimal learning rate trend and its relation to overtraining degradation?

Great question! While empirical datasets are difficult to analyze precisely, our theoretical results offer helpful intuition:

Theorem 4.3 shows that extending pre-training increases the sensitivity (or forgetting) of pre-trained features during fine-tuning. This sensitivity is larger for tasks whose feature distributions significantly differ from pre-training (i.e., larger eigenvalue differences)—fine-tuning tasks which are much different from the pre-training task are more prone to catastrophic overtraining. In our revised version of the paper, we will clarify this with an appropriately updated theorem statement.

Our intuition is that this theoretical insight translates roughly to practice. Empirically, tasks very dissimilar to pre-training likely require larger changes in model features, and larger learning rates naturally induce larger weight changes (even controlling for steps). We’ve empirically verified that larger learning rates lead to larger weight changes and will add a figure illustrating it clearly. Thus, tasks needing significant feature adaptation typically benefit from larger learning rates and, as a result, exhibit more pronounced catastrophic overtraining.

Thank you for your detailed feedback! We are happy to hear your positive comments that there is “compelling empirical evidence”, that “the studies involving intermediate OLMo checkpoints and newly trained models from scratch are especially persuasive”.

During pre-training, it may be difficult to identify precisely when overtraining has occurred. [...] A helpful step the authors could take to operationalize their findings would be to suggest heuristics based on model weights or activations.

Great suggestion! Developing heuristics based on model weights or activations is indeed a valuable direction for future work. Another promising avenue would be creating scaling laws specifically for catastrophic overtraining. With accurate scaling laws, practitioners could significantly reduce computational overhead by performing fewer post-training experiments to identify optimal training durations.

In Section 6, the authors discuss their rationale for not conducting any staged pre-training (also known as annealing or mid-training) [1, 2]. It would be valuable to further expand on this point in the discussion or include additional experiments in the appendix that explore scenarios involving additional domain-oriented pre-training. It is possible that concluding pre-training with domain-specific data might mitigate much of the fine-tuning performance penalty associated with overtraining.

This is a great point. In our current experiments, our smaller-scale models are pre-trained on the C4 dataset (which includes some code), and we demonstrate catastrophic overtraining on Starcoder-Python fine-tuning (Figures 2, 3). This suggests domain-oriented data doesn't completely eliminate the problem.

However, systematically understanding how the choice of pre-training distribution affects catastrophic overtraining remains important future work. Still, we believe adding domain-specific data cannot universally solve this issue: it’s practically impossible to include data from all potential fine-tuning domains. For example, catastrophic overtraining even arises in multimodal fine-tuning scenarios, where adding image data during pre-training (to a text-only model) simply isn't possible.

Thank you for your detailed feedback! We are happy to hear your positive comments that there is “significant insight in this paper”, that it “would be of interest to the community”, and that we include a “wealth of empirical results”.

We were also glad to read:

My issues with this work mostly lay in the claims made about the results [...] rather than the results themselves.

To address this, we’ve revised our framing to clarify our claims. These changes will be reflected in the final version.

Summary of changes: To address your main concerns, we will: (1) rewrite to clarify the definition of “harder” and the scope of our findings; (2) further highlight how checkpoint selection (tuning) affects results; (3) make specific changes based on the comments below.

How would you define what it means to be harder to fine-tune, and what is the main evidence in your paper that supports this specifically?

We will clarify in the updated manuscript—by "harder" we mean specifically that extending the pre-training phase, and then fine-tuning, can lead to worse performance in-distribution (fine-tuning task) and also out-of-distribution (unrelated tasks) compared to a shorter pre-training stage.

Our evidence:

• After instruction tuning OLMo-1B on Anthropic-HH, models trained on 3T tokens underperform those trained on 2.3T on AlpacaEval (in-distribution) and ARC/PIQA (out-of-distribution) — see Figure 1.
• After multimodal tuning, the 3T-token model also performs worse on ARC and PIQA (Figure 1).
• After instruction tuning on TULU, the 3T-token model scores lower on AlpacaEval (in-distribution) and on multiple OOD benchmarks (ARC, PIQA, Winogrande, HellaSwag, OpenBookQA) — see Appendix Figure 7 (right).
• For OLMo-30M, models trained beyond ~32B tokens show degraded in-distribution performance on several fine-tuning tasks (RTE, TREC) and degraded C4 web-data performance when fine-tuning with GSM8K, Starcoder-Python, MR, RTE — see Figure 2.

Apologies for the confusion, we will clear this up in our revision.

This broad claim [overtraining => harder to fine-tune] is very difficult to exhaustively verify

Thanks for bringing this up—in our revised version we will properly clarify the scope of our paper.

To summarize: Our core finding is that there are some important settings where additional pre-training hurts fine-tuned performance, despite helping the base model performance (Figures 1 and 2). We believe that finding any example of this phenomenon is surprising. We do not claim that all settings exhibit this degradation with overtraining.

If the authors decided to pick a different reference task to optimize LR over [other than AlpacaEval], the model would have looked significantly more predictable to fine-tune with more pretraining tokens.

Agreed! A key contribution of our work is precisely to analyze how the optimal learning rate chosen for one task impacts observed degradation on others — this is the focus of Section 3.4.

In this section, we argue that we specifically observe performance (ID or OOD) degradation when optimizing for a task that requires a larger learning rate for good performance — empirically, this is the case for AlpacaEval, and many other evaluation settings (see response below).

The choice to optimize AlpacaEval, on one hand is motivated as general, but ultimately a bit arbitrary.

For instruction tuning, we pick AlpacaEval because it is a standard benchmark to measure response quality—improving this is our goal during fine-tuning. We agree that any single metric could seem arbitrary, but we also evaluate on a collection of other evaluation metrics:

We consider 12 total evaluations across various settings:

• AlpacaEval (for instruction-tuned models)
• VLM score (average over 5 VLM benchmarks; for multimodal tuned models)
• 10 individual fine-tuning datasets where we optimize for ID performance: SUBJ, BoolQ, MR, CR, RTE, TREC, Tweet sentiment, GSM8k, SIQA, Starcoder-Python (Figure 2, plus more in Appendix C.2)

We believe that, collectively, these experiments robustly illustrate the occurrence of degradation across many practical fine-tuning scenarios.

This difficult to SFT behavior doesn't quite exist for LLM360-7B [and] OLMo-7B [...]. Is it because of the overtraining?

Great question! This difference is likely explained by how many tokens per parameter each model has seen:

• OLMo-1B (3T tokens ≈ 3000 tokens per param) experiences degradation by ~2.3T tokens.
• LLM360-7B (1.3T tokens ≈ 185 tokens per param) and OLMo-7B (4T tokens ≈ 571 tokens per param) have seen significantly fewer tokens per parameter relative to OLMo-1B.

If degradation scales linearly with model size, we would only expect degradation for a 7B model after ~16T tokens (7B × 2300 tokens per param). Thus, current 7B models simply haven't been trained long enough to reach that regime yet.

Thank you for your detailed feedback! We are happy to hear your positive comments that our experiments “support the main finding” and that our theory "helps explain the observed phenomenon”.

Summary of changes: To address your main concerns: (1) we clarify that our fine-tuning setups do lead to consistent improvements on the target task; (2) we will rewrite to clarify the scope of our findings; (3) we will add the per-token accuracy experiment you propose; (4) we will make specific changes as outlined below.

Please find our responses to each concern:

The authors describe "catastrophic overtraining" as universal

Thanks for bringing this up—in our revised version we will clarify the scope and remove the use of “universal”.

To summarize: Our core finding is that there are some important settings where additional pre-training hurts fine-tuned performance, despite helping the base model (Figures 1 and 2). We believe that finding any example of this is surprising. We do not claim that all settings exhibit this degradation with overtraining.

The evidence is limited to only two fine-tuning setups

We want to clarify—we consider more than just two fine-tuning settings.

For the our large model results (OLMo-1B):

• Anthropic-HH (Instruction tuning)
• Multimodal tuning
• TULU (instruction tuning)
• Alpaca (instruction tuning)

(Figure 1 and Appendix B). We observe catastrophic overtraining for Anthropic-HH, multimodal tuning, and TULU.

For the small model results (OLMo-30M):

• GSM8k
• SIQA
• Starcoder-Python
• MR
• RTE
• TREC
• CR
• Tweet sentiment
• SUBJ
• BoolQ

(Figures 2 & 3, plus Appendix C).

Both [IFT & VLM fine-tuning setups] fail to produce consistent improvements in the evals measured compared to the base model.

For Anthropic-HH, we aim to improve response quality (measured via AlpacaEval), and fine-tuning boosts scores by ~35–45% over the base model (see table below).

For VLM fine-tuning, the goal is to enable image input via an adapter—something the base model can’t do. The fine-tuned model reaches ~45% VLM score (Figure 1).

We also evaluate OOD tasks (ARC, PIQA, BoolQ, Winogrande, HellaSwag, and OpenBookQA). Fine-tuning on an unrelated task isn’t expected to help performance on these tasks and can often harm performance [1].

Together, these evaluations help us assess how overtraining impacts the ability to learn new tasks (measured by main eval) and the tendency to degrade unrelated capabilities (measured by OOD eval).

Here are the base model (and instruction-fine-tuned) scores for AlpacaEval:

Table: Comparing AlpacaEval for OLMo-1B base model vs instruction tuning with Anthropic-HH. We will add this to the main paper to clarify, apologies for the confusion.

Do your findings still hold if you measure token prediction accuracy instead of perplexity?

We ran this experiment, and yes, our findings hold. Below report per-token accuracy of GSM8k as a function of pre-training tokens (analogous to Figure 3).

Note: GSM8k per-token accuracy is more predictable than web data because the examples are quite structured.

Results corresponding to other figures & datasets similarly hold true.

The simplification [of the theoretical model] makes a number of assumptions that may not hold for LLMs.

To clarify—We do model discrete gradient descent explicitly during fine-tuning (see Appendix A.3).

We aren’t aware of any theoretical tools that would enable us to study the dynamics of LLMs without at least some degree of simplification such as the ones we describe. We fully acknowledge that there are limitations that arise from these simplifications but we hope our analysis is a starting point to understand this phenomenon that may lead to principled mitigation strategies in the future.

Additionally, since catastrophic overtraining is so counterintuitive (to us), seeing it emerge from incremental learning in a simpler setting reassures us it's not just an artifact of our LLM setup and offers intuition for a possible mechanism.

[1] Goodfellow, Ian J., et al. An empirical investigation of catastrophic forgetting in gradient-based neural networks. 2013.