Language models scale reliably with over-training and on downstream tasks

Relation to Xiong et al. [1]. Thank you for mentioning the reference! We were not aware of this paper at the time of our submission; however, have added the following sentence to our revised manuscript in our extended related work:

Xiong et al. (2024) develop a hyperbolic-fit scaling law to describe the evolution of test loss during training based on early training steps. To do so, they consider models that undergo over-training in their testbed. In contrast, we focus on converged models and investigate predicting the performance of increased over-training in new runs that undergo more over-training than the converged models used for the fit.

We thank the reviewer for their generous review, acknowledging our originality, experimentation, contextualization relative to prior work, and presentation. Thanks for your attention to our work and for your willingness to defend your review during the review process!

Scaling laws for multimodal models, mixture-of-experts etc. Thanks for bringing this up! We agree that understanding the scaling behavior of multimodal models is interesting; however, we consider it outside the scope of this paper, which focuses on language modeling. One benefit of studying scaling in the context of language modeling is that pre-training recipes have largely converged in this area of research (i.e., similar to GPT-2 and Llama training recipes). While there are exciting developments with sparse, mixture-of-experts (MoE), models, there are still fundamental design decisions that remain unclear (e.g., how best to route forward passes in MoE architectures). Given these uncertainties in emerging model families, we decided to focus computational resources on dense decoder-only transformers, which have had staying power in modern research and application.

Considering post-training. Thank you for bringing up post-training! This is indeed an exciting area for future work with respect to scaling laws as we identify in our conclusion. The focus of our study is on pre-training, in line with prior work on model scaling (i.a, Chinchilla). The field of scaling laws is still maturing and we hope others might build on our testbed of >100 models to investigate scaling in post-training. Considering a small model that is over-trained to match the performance of a larger model; we hypothesize that the large model will be easier to post-train given the additional capacity from having more weights. However, because post-training is a relatively cheap operation compared to pre-training; we hypothesize also that differences between the aforementioned small model and large model may diminish as one scales up pre-training FLOPs keeping post-training FLOPs constant.

Considerations for predicting performance of larger language models. There are a few considerations here, which we feel are important. Thanks for the opportunity to share! 1) Task difficulty. Once models become extremely capable, they tend to saturate performance metrics, specifically for easier tasks. When considering models up to 400B parameters, it probably makes sense to include more difficult tasks where performance is not saturated, even at this scale. 2) Knowledge of training data and multi-epoched effects. The scaling laws developed in this paper require knowledge of the training distribution. This is not known for highly capable closed source models or even for open-weight models such as those in the Llama3 family. Hence, it is hard to construct small scale proxies here. Also as one gets to high token training regimes, it is increasingly likely that one may have to train on repeated data (i.e., in a multi-epoch setting). It is likely that such settings affect scaling laws as supported by Muennighoff et al., 2023b.

Individual task prediction and assumptions. Our work corroborates that of Schaeffer et al. (2023) that individual task prediction is indeed a hard problem. We identify this as a critical area for future study in our conclusion. Also, thanks for bringing up the $\alpha = \beta$ assumption. In Appendix A, we discuss extensions of the over-training scaling law in cases where $\alpha \neq \beta$, finding that similarly predictable behavior is expected. However, given the large compute requirements needed to train models, we were unfortunately only able to restrict our empirical setup to the $\alpha = \beta$ regime.

Additional training recipes. While we conducted experiments with various hypers and datasets, we acknowledge the limitation of not exploring additional optimizers or architectures. We feel our choice is justified, based on the standardization in modern decoder-only transformer training recipes (i.a., GPT-2, Llama). We recognize that focusing solely on the current training paradigm may be restrictive. However, the substantial compute requirements for a comprehensive scaling study pose practical challenges to broader exploration.

On predictable average downstream performance. We are sorry for any confusion here! Our claim is indeed that average performance can be predictable. We try to be clear in our abstract (L025-026), intro (L091-092), and experiments (L413). However, if there are any specific instances where our claims are imprecise, we are happy to update the manuscript!

Typos. We have updated the manuscript accordingly, thanks!

Figure readability. Thanks for the suggestion. We updated Figure 1, 2, 9 and are happy to make further changes.

Elaboration on motivation for over-training (re: L142-144, 44). Thanks for the chance to clarify! The fact that over-training reduces inference cost is the primary motivation for over-training in practice. Consider a Chinchilla optimal model $A$ that has $N$ parameters and is trained for $D$ tokens. This model takes approximately $6 \cdot N \cdot D$ FLOPs of compute to train. Now consider a second model $B$ that has only $N/c$ parameters, where $c>1$, but that is trained on $k \cdot c \cdot D$ tokens, where also $k>1$. This model takes $6 \cdot k \cdot N \cdot D > 6 \cdot N \cdot D$ FLOPs (i.e., it is more expensive to train). Consider that $B$ is over-trained s.t. it matches $A$ in validation loss (i.e., performance). Model $B$ is cheaper to serve at inference, as it has fewer parameters and requires less inference FLOPs.

Larger sets of tasks to validate the relationship between loss and downstream error. Please see Figure 8 in the Appendix where we look at the relationship between loss and downstream performance for as many as 46 different tasks. The high level takeaway is that adding more evaluation tasks keeps the scaling laws relatively stable in their predictive power. However, considering too few tasks in the average can diminish predictive power.

Assumptions about the uniformity of over-training effects over datasets and model sizes. Thanks for bringing this up! Our assumptions here are baked into Equation (4); namely, that the variables for the fit are sufficient to describe behavior. This equation is fit differently for different training distributions, so we do not assume that one scaling fit will be able to predict performance for models trained on a different training distributions.

Trends in individual task prediction error. Looking at Table 2, it appears there are not clear trends in what individual tasks are predictable. Furthermore, it appears that predictability can differ wildly based on the training distribution. Particularly, we believe that this observation motivates future work on understanding interactions between training sets and downstream eval predictability. For example, training on RedPajama allows for predicting relative error for the 7B run on ARC-Easy at $\sim 5\%$; however, the prediction error is much higher for C4 and RefinedWeb trained models at $>26\%$. While influence functions [1, 2] are one promising avenue; we believe there is much more work to be done here.

Out-of-distribution (OOD) scaling clarification. We provide the key takeaways for our OOD scaling study in L432-436, repeated below for convenience:

To probe the limits of reliable scaling, we attempt to break our scaling laws in out-of-distribution settings. We find that models trained on C4---English filtered---and evaluated on next token prediction on code domains have a high relative error in many cases. Perhaps surprisingly, evaluating the same models on German next token prediction gives reliable loss scaling (Figure 10).

For more results see L1602-1615.

Additional references. [1] Pang Wei Koh and Percy Liang. Understanding Black-box Predictions via Influence Functions. 2017. [2] Roger Grosse et al. Studying Large Language Model Generalization with Influence Functions. 2023.

How does over-training reduce inference cost? Thanks for the chance to clarify! The fact that over-training reduces inference cost is something that can be shown by way of example. Consider a Chinchilla optimal model $A$ that has $N$ parameters and is trained for $D$ tokens. This model takes approximately $6 \cdot N \cdot D$ FLOPs of compute to train. Now consider a second model $B$ that has only $N/c$ parameters, where $c>1$, but that is trained on $k \cdot c \cdot D$ tokens, where also $k>1$. This model takes $6 \cdot k \cdot N \cdot D > 6 \cdot N \cdot D$ FLOPs to train (i.e., it is more expensive to train). Consider that $B$ is over-trained s.t. it matches $A$ in validation loss (i.e., performance). Model $B$ is cheaper to serve at inference, as it has fewer parameters and hence requires less inference FLOPs to run the forward pass. Looking at Figure 2 and considering a slice of the y-axis, we notice that this phenomenon does happen in practice.

When should we trust downstream scaling laws? Thanks for the question! As pointed out by Schaeffer et al. (2023) specific downstream metrics can be hard to predict due to the non-linearity of top-1 error/accuracy metrics (L469). Our main finding is that the average error can be predictable. We still caution against using our downstream scaling laws for individual tasks. We do note; however, that average or aggregate metrics are very useful in practice to judge quality. For example, Li et al. [1] consider average metrics to guide their development of better datasets for language model training.

Is Figure 1 a log-log plot? Figure 1 (left) is log-log, while Figure 1 (right) is linear. We hope that the axis scaling from matplotlib makes this clear; however, please let us know if there is something we can do to improve clarity here, and we are happy to apply the change!

Typo. Thanks for pointing out the typo, we have fixed it in the updated manuscript.

[1] Li et al. DataComp-LM: In search of the next generation of training sets for language models. 2024.