Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

Dear reviewer,

We thank you warmly for your detailed and thorough feedback on our work. We are glad to read that "The claims made in the submission are supported by clear evidence" and that our method "makes sense and provides novel insights."

While the proposed curve fits these points well, the paper needs to show the scaling law can accurately predict the loss for a model that is much larger than 1.27B

This is indeed an important point, and we are happy to report that our scaling laws allow us to extrapolate, that is, as you mention, predict large scale behavior from small scale.

We could predict the forgetting for 650M and 1.3B models, on 9M and 30M unique tokens, with a bootstrapped MRE of 0.83% - using only models no bigger than 350M parameters, and finetuned with no more than 3M unique tokens. We added this table in the paper.

I am unsure whether there is mathematical expression to characterize the U-curve

We agree that’s a fruitful research question. However, finding how many steps/repetitions are required to find the bottom of the U-curve is by no means trivial, since it depends on both the domain, the model size, and the number of tokens. Nonetheless, assuming fixed number of epochs, we benchmarked the scaling laws of “Scaling Data-Constrained Language Models”. We found it yielded accurate predictions, correct in our setting too. We will clarify this in the paper.

Therefore, it might be possible to predict when the bottom of U-curve will be reached (find the argmin over time) based on the performance improvements after a few epochs. Our current work doesn’t need to characterize the full U-curve: it is sufficient to estimate the minimum value reached at the bottom (find the min over time).

It's unsure what's the use case for the proposed scaling law

We will clarify two important outcomes from our scaling laws.

A) The proposed scaling law allows to quantify forgetting as function of scale, and notably to extrapolate, as you mentioned before. Testing at small scales allows you to not be surprised with what happens at larger scales.

B) We also believe our work brings some understanding to the phenomenon of forgetting. An interesting consequence of our functional form reveals that the leading cause behind forgetting might be related to the parameters count $BpN$. This suggests that forgetting is primarily due to limitations in network capacity. This is also confirmed by the fact that smaller models suffer the most: they lose up to 95% (!) of the pretraining progress when forgetting (i.e, the pretraining validation loss reverts to a point reached at 5% of pretraining), while bigger models only lose 20% of the progress. We added this plot in the paper, which quantifies forgetting with a fraction of pre-training cost lost.

Overall, this balances some findings of “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws”, which suggested that smaller models should be preferred due to their lower inference cost. We will make these points in the paper.

We thank you again for your review, and we hope that we have alleviated your concerns !

Dear reviewer,

We thank you warmly for your detailed and thorough feedback on our work. We are glad to read that "The experiments are comprehensive and validate the hypotheses", that "most claims and the evidence presented are pretty solid", and that " experimental design choices are clear". Thank you as well for the additional reference.

I would recommend adding the hyper parameter values for each configuration and other related details for reproducibility.

Thanks! We added a summary table in appendix.

I would also recommend adding downstream task results for these models as a sanity check

Please find here and here results for the ARC_easy and MMLU task, for the pretrained checkpoint, and for models finetuned on dm_mathematics. Performance degrades on the generalist questions of ARC easy, but improves on MMLU which is more aligned with dm_mathematics.

Furthermore, on the domain dm_mathematics we evaluated the quality of 75 checkpoints (all model sizes above 350M) on the arc_easy task, in a 0-shot setting. We take the z-score at 99% significance threshold, and report the best result below (ns = not significant, numbers are accuracy gain over p=0%). TLDR; the difference is significant for 15 experiments, oscillating between 3.7% and 8.7% more, and pretraining data injection always helps. As noticed before, the bigger the finetuning dataset the more important the forgetting. For these bigger datasets (>9M tokens), injecting pretraining data is crucial.

From what I understand, in 4.4, the number of unique tokens available varies but they are upsampled such at =1%? If yes, it would be helpful to make this clear.

Yes you’re correct - we updated the section and the figure x-axis. Each batch is built following the process described in section 3.3: each sequence is picked from the pretraining split with probability 1%, and from the specific domain with probability 99%. The pretrained set is capped to the number of tokens wanted, and then repeated as many times as necessary.

Could the authors elaborate on why they chose the Pile as the fine tuning task, as opposed to other SFT/Instruction-Tuning/Alignment datasets which are conventionally used?

We wanted to simulate the setup in which a company fine-tune a model on raw data from internal documentation. The highly-specialized content of The Pile (which is partitioned semantically) allows to simulate the scenario, unlike typical typical IFT datasets that are generalist.

Nonetheless, we agree that the Instruction Finetuning scenario is also of interest; thank you for your suggestion. We perform instruction finetuning on the OpenHermes dataset - 3M tokens in the train split (95% of the total). We finetune the model to perform next token prediction of the output, conditioned on the [INST] prompt input [/INST] prefix. We added finetuning curves and forgetting curves in appendix.

We report the fitted scaling laws parameters here:

Finetuning:

Forgetting:

We thank you again for your review, and hope that our answer has alleviated your concerns.

Dear reviewer,

We thank you warmly for your detailed and thorough feedback on our work. We are glad to see that you found that "claims are mostly reasonable", that "the methods and evaluation criteria make sense," and that we study "a problem which is valuable to the community".

Prior work (Kalajdzievski et al 2024) has developed scaling laws for forgetting for fine-tuning when using LoRA. Hence the novel aspect of this work is limited to a) obtaining these laws in a setting where a small proportion of the pre-training data is injected during fine-tuning, b) using full-finetuning rather than LoRA.

Additionally to those differences, as explained L148, we want to point out that we use many more specialization tasks, model scales, and we measure forgetting through the pretraining loss (which is impossible to do with models like LLAMA 2 for which the training data is unavailable).

Though the paper presents a large number of results for a single model (Gpt-2), it is unclear if these laws will carry over to alternative LLMs.

Indeed, we acknowledge that we only consider autoregressive decocoder-only transformers. Note that the architecture we use is widely used, up to minor changes (see e.g. sec.2.2 of [1]). Testing the impact of these minor architectural changes like use of RoPE and swiglu on the scaling laws is indeed an interesting future research direction.

The paper does not explore the full grid of model size and data set size i.e. (N, D) but rather an isocurve D=100N. While this is a reasonable choice for computational reasons, it would have been useful to study the full grid at least for a single domain to understand how far the isocurve is from the optimal value for this domain.

Thanks for raising this point ! We tested our idea with another isocurve D=10N. We finetune on free_law, and we measure a bootstrapped MRE of 0.57% for forgetting and 1.14% for finetuning. We aded this curve and this curve to the paper.

it would be advantageous to discuss whether alternative functional forms of the regression model may fit the data better

We are looking for a forgetting law that (i) should yield a zero finetuning when $D^{ft}$ is zero (ii) does not exhibit too many arbitrary parameters. For example, we found that the $+E$ term is entirely explained by the re-warming of the model (since the finetuning LR was x3 times terminal LR of pretraining). As a matter of fact, measuring it that way instead of regressing its value decreases the MRE from 0.48% to 0.40%. See this table here.

We also tested additive laws, as mentioned in p8. Other laws, like $A\frac{D^{\beta}(1-p)^{\kappa})}{N^{\alpha}}+E$ had a higher error of 0.67% despite having one more parameter $\kappa$. We added these results in appendix.

Furthermore, to strengthen the statistical significance of our results, we also compute and report the bootstrapped MRE (we sample 125 measurements with replacement from the pool, and average the result over 128 independent samples).

It would be useful to report the extent of repetition in the pre-training dataset

Good call! Since that the bottom of the U-curve is typically reached within a few dozen epochs, and that p=1%, the number of repetitions quickly falls to 1. We updated the figure with “Unique pretrain tokens per unique finetune token” in x-axis. We discover that 0.3 unique pretrain tokens per unique finetune token are typically sufficient.

We hope that our answers have alleviated your concerns, and we thank you again for your review !

[1] Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).

Dear reviewer,

We thank you warmly for your detailed and thorough feedback on our work. We are happy to read that you found that our work is a “simple and effective solution”, that "claims are supported by the evidence presented", and that we have "a central point that is [...] new and relevant for continual LLM pretraining".

The paper focuses on continual pre-training scenarios rather than what is commonly called "fine-tuning" in modern LLM contexts.

We will clarify this very important point in the paper. A critical part of our setup is the specialization data scarcity: we only have a small number of tokens to train on, which is not necessarily the case in continual pre-training, while it is a defining aspect of fine-tuning. Our findings also extend to more standard LLM fine-tuning set like openHermes, please refere to the last part of the answer to rev.R4eh

How sensitive is the 1% rule to the nature of the pretraining data?

This is an interesting question; testing it would require using a novel pretraining set and pretraining models on it, which is cumbersome. However, we believe that indeed the diversity of the pretraining data is very important ot counteract forgetting.

How sensitive is the 1% rule to data quality?

Pretraining data injection acts as a regularization. The value of $p$ allows to move along a Pareto front - see the figure. We expect that data quality does not have much impact on the fraction p necessary to overcome forgetting, since p only drives the "regularization strength". However, we believe that higher data quality in the pretraining set would allows using less of it to fine tune in a scenario where we repeat the pretraining data; we expect improvements in an experiment like fig.6

Did you observe any qualitative differences in the types of knowledge that were forgotten? Are certain types of knowledge more prone to forgetting?

We did not look into the details of knowledge that are forgotten; but clarifying how models forget is an extremely interesting future research direction.

Your scaling law includes parameter B [...]. The values vary significantly across domains. What factors do you believe drive these significant differences?

This is an interesting point. The parameter B indicates how much pretraining data helps mitigating forgetting. A high B value therefore indicates a strong discrepancy between the pretraining and fine-tuning set, ann indeed B is low for datasets such as free law and wikipedia which are close to the pretraining set, while it is high for dm_mathematics and euro parl, which are far from the pretraining set. We will clarify this in the text.

Your results suggest that pretraining data injection is a regularizer that improves generalization. How does it compare to standard regularization methods (e.g., L2 regularization) at preventing forgetting?

We tested an "anchored" baseline with a variant of AdamW, using $\lambda(\theta\_t-\theta\_0)$ as the weight decay term, instead of the conventional $\lambda\theta_t$, where $\theta\_0$ are the parameters of the pretrained checkpoint. Note that this is equivalent to using weight decay on the delta between fine-tuned and base model. We added this illustration to the paper. For $\lambda\in[1e-2, 1e-1, 1]$ the forgetting remains significant (between 15% and 4% more than p=1%, accross all model sizes). For $\lambda\geq 1$ the finetuning performance decreases compared to the baseline p=1%. Therefore more data diversity is superior to regularization in the parameter space. We added the plot in appendix.

We also found that standard weight decay had not impact on forgetting.

Is there a way you can test some of these ideas on standard Llama/Gemma/Qwen base checkpoints

Thank you for the suggestion, this is an interesting future work. We chose to focus on a family of models we had full control over, where we precisely knew the pretraining pipeline and datasets. This allowed us to observe clear trends as a function of scale. In contrast, the precise training pipeline as well as their training data are not available.

However, we tested our ideas with other models from the same family, with the isocurve D=10N. We finetune on free_law, and we measure a bootstrapped MRE of 0.57% for forgetting and 1.14% for finetuning. We aded this curve and this curve to the paper.

We hope that our answers have alleviated your concerns, and we thank you again for your review !