Resolving Discrepancies in Compute-Optimal Scaling of Language Models

Question 1: hyperparameter sensitivity
As we remark in the limitations section, we believe that scaling up the models reduces optimization hyperparameter sensitivity, which would explain why Hoffmann et al. were able to obtain their scaling law without very careful parameter tuning. Moreover, Figure 10 in our paper shows that scaling up the model size leads to higher robustness to learning rate and batch size.

Finding ways to make smaller-scale models also require less per-scale tuning is an important problem - maximal update parameterization (𝜇P) is a promising direction still under active research; we will discuss it in the revision.

Question 2: Condition for scaling laws
We focus on decoder-only Transformer models, trained for next token prediction with the log loss. This focus is the same as in Kaplan et al. and Hoffmann et al., the two papers that presented the scaling laws in question. Beyond that, we argue that the discrepancy between the two scaling laws is not due to a difference in assumptions, but rather due to differences that are largely methodological errors in Kaplan et al.

We thank the reviewer for the detailed review, finding that our paper presents extensive and detailed experiments and provides helpful guidance on reproducing scaling law results. We address the weaknesses and questions below - the main issue appears to be a lack of clarity regarding which scaling law (Kaplan or Hoffmann) is “correct.” We hope our answers clarify this, and we look forward to engaging in constructive discussion about how to introduce these clarifications to the revision best.

Unclear which scaling law is correct
Conclusively deciding the correctness of compute-optimal scaling laws is a difficult problem due to computational barriers. Nevertheless, our paper adds to a growing body of evidence that Hoffmann et al. (“Chinchilla”) scaling law is more accurate than the Kaplan et al. scaling. Let us explain this in detail:

• The difficulty of validating scaling laws. Directly testing the prediction of a compute-optimal scaling law at the largest compute budget in which one can afford to train is almost impossible by definition, as such verification would require multiple largest-scale training runs. Indeed, neither Kaplan et al. nor Hoffmann et al. directly verify the compute-optimality of extrapolations of their scaling laws.

• Theoretical evidence in favor of the Hoffmann et al. scaling law. The compute-optimal exponent $\alpha=0.5$ found by Hoffmann et al. has the elegant interpretation that model size and data size should scale with constant proportion. Recent theoretical work has proven the same scaling holds for simple, analytically tractable settings under fairly broad assumptions. The paper “4+3 Phases of Compute-Optimal Neural Scaling Laws” by Paquette et al. shows this for a random-feature linear regression setting, while the paper “Information-Theoretic Foundations for Neural Scaling Laws” by Jeon and Van Roy shows this for data generated by infinite-width 2 layer ReLU networks using information-theoretic arguments. These works hint at (and, in the former case, explicitly conjecture) the possibility of the optimality of proportional scaling being a broader, universal phenomenon.

• Prior empirical evidence in favor of the Hoffmann et al. scaling law. Hoffmann et al. nevertheless provide strong evidence in favor of their scaling law - the Chinchilla 70B model outperformed the larger Gopher model that was trained using similar compute, architecture, and data, as well as other other larger contemporary models. As we discuss in our related work section, in language modeling, subsequent work [12, 22] has produced scaling laws more similar to Hoffmann et al., and other papers rely on the validity of this scaling as a starting point [14, 29]. Therefore, we believe that - for the specific context of next token prediction with Transformer decoder-only models - it is established that the Hoffmann et al. scaling law is the more accurate one; what was missing prior to our work is an explanation for why this is so.

• New evidence in favor of the Hoffmann et al. scaling law from our work. Our paper attributes the difference between the Kaplan and Hoffman scaling laws to what are arguably methodological errors in Kaplan et al.: Incorrect FLOP counting, overly-long warmup period, and suboptimal optimizer configuration. Furthermore, we show that the Hoffmann scaling law extends to non-decayed learning rate schedules. In addition to providing useful lessons for future scaling law studies, our findings further affirm that the Hoffmann et al. scaling is the “more correct” one.

Unclear conclusion from the paper
We hope the discussion above helps position our paper in context and thus clarify its conclusion. Moreover, our final reproduction of the Hoffmann scaling constitutes a well-defined pipeline that enables reliable predictions - the training code is based on the open-source open_lm library, and all of the configurations and analysis code are included in our submission.

Limited impact of compute-optimal setting
We agree that studying over-trained language and more broadly the individual effect of scaling $N$ and $D$ are important topics that warrant further research. However, note that the very concept of overtraining is defined relative to compute-optimal training. Thus, a better understanding of the compute-optimal setting is an essential stepping stone for broader studies as well. In particular, our conclusions about FLOP counts, warmup, and hyperparameter tuning should be relevant beyond the strictly compute-constrained setting.

Limited study of predictive accuracy
Regarding the concern that we study predictive accuracy only in one figure (Figure 4) and one dataset, note that Figure 5 also studies the predictive accuracy as we vary the amount of compute used to fit the scaling laws. In the attached pdf we further study predictive accuracy in two ways:

1. We reproduce Figures 4 and 5 for the OpenWebText2 dataset (this involves simply running the analysis code attached to our submission with different parameters).

2. Following suggestions from Reviewers zsos and CKuz, we trained a model for ~8e19 FLOPs (4x our previous largest budget) under our predicted compute-optimal settings and hyperparameters. We show that the resulting loss agrees fairly well with our extrapolation in Figure 4, with further improvements as we fit it on slightly larger budgets of up to 1.6e18 FLOPs. Moreover, we confirm the optimality of our predicted learning rate and batch size - see the response to reviewer CKuz for details.

Thus, even though the focus of our paper is explaining the scaling law discrepancy and not measuring predictive accuracy, we study the latter to a greater extent than the influential papers of Kaplan et al. and Hoffmann et al. We nevertheless agree that the two extensions described above will strengthen our paper and will include them in the revision.

(Due to length constraints we answer the two questions in the review in the comment below.)

Dear reviewer,

Before the author-reviewer discussion period ends, we would like to know if our responses above fully address your questions and concerns. If so, we kindly ask you to reconsider our paper’s score.

We thank the reviewer for the useful questions and kind words. We were glad to read that you find our paper studies the important question of scaling laws thoroughly and fills an important missing piece in the literature. Below, we address the weaknesses and questions raised in the review.

Scale of experiments
Following the reviewer’s suggestion, we extend the training of our largest model (with 901M parameters) to ~14B RefinedWeb tokens (the compute-optimal value for that model size according to our scaling law), and test whether our predicted batch size and learning rate are optimal at this scale. In particular, we compare the learning and batch size prescribed in Table 4 (0.0024 and 640, respectively) to 8 configurations, each varying either the learning rate or the batch size. We evaluate the models on held-out RefinedWeb data and calculate the validation loss and standard deviation due to sampling as described in L583. We tabulate the results below:

These results indicate that we accurately predict the optimal hyperparameters at the ~1B scale. The runs with batch sizes 160 and 320 did not complete by the rebuttal deadline, and we will attach them as they arrive; intermediate loss evaluations suggest they will not outperform our predicted optimal batch size.

Question 1: learning rate scheduler in sweep
We use a constant learning rate in our sweep experiments, with a warmup duration of N (number of parameters) tokens.

Question 2: warmup heuristic
To address the reviewer’s question, we perform a small-scale experiment where we train a $N$=108M parameter model on the RefinedWeb dataset for $20N=$~2.1B tokens, using the hyperparameters in Table 4 (for the appropriate model size) but varying the warmup duration from $N/4$ to $16N$. We evaluate the models on held-out RefinedWeb data and calculate the validation loss and standard deviation due to sampling as described in L583. We tabulate the results below:

The results show that a warmup duration of $N$ is nearly optimal, with durations of up to $4N$ achieving very similar and even slightly better results. We conclude that the training loss is fairly insensitive to the precise warmup duration as long as it stays in a reasonable range. Consequently, we believe that our scaling laws are not strongly dependent on the particular choice of warmup duration.

We have queued the 4 additional runs requested and hope they will complete before the discussion period is over. However, we note that Figure 10 suggests that, for larger models, the interaction between batch size and learning rate is not very strong.

We thank the reviewer for the useful questions and the encouraging comments, recognizing that our work conclusively identifies and mitigates the discrepancy between the two scaling laws, and appreciating the value of findings for speeding up scaling law experiments. Below, we address the questions brought up in the review.

Question 1: Approach 3 in Hoffmann et al.
In our opinion, Approach 3 is less direct because it arrives at the scaling law for $N^\star(C)$ via an assumption about how the loss depends on $N$ and $D$. In contrast, Approach 2 and our approach directly estimate $N^\star(C)$ and fit a scaling law to it, without requiring additional assumptions on the structure of the loss. Moreover, Besiroglu et al. [7] found that Approach 3 is sensitive to the curve-fitting method which has led to inaccurate conclusions in Hoffmann et al. Therefore we decided not to use Approach 3.

Question 2: Attention FLOPs
The scaling law exponents do not change significantly when we account for the attention FLOPS - see Figure 6 in the appendix and discussion in lines 531–551.

Extending experiments to larger models
Training a model with $N=3B$ with $\rho=16.5$ (our compute optimal tokens-to-parameters ratio) would require roughly 9e20 FLOPs, which is ~36 times larger than the largest individual training run in this project. A 7B model would be 200 times more expensive. Since verifying the optimality of hyperparameters requires several such training runs, such experiments are beyond our current resources. Consequently, we conducted a more modest scale-up to ~8e19 FLOPs, which is ~4 times the largest budget in our original FLOP grid. As discussed in our response to Reviewer Kjsg, our predicted learning rate and batch size are near-optimal even at that scale. Furthermore, as Figure 3 in the attached pdf shows, the loss remains fairly predictable as well.

We thank the reviewer for the detailed comments and for finding our paper novel, clearly written, and our experiments well-motivated, sound, thorough, and providing valuable insights. Below we address the main concern regarding the conjecture in Hoffmann et al., as well as the two additional questions. With these points addressed we hope the reviewer will consider increasing our paper’s score.

The conjecture in Hoffmann et al.:
We agree that the term “learning rate schedule” comprises both the warmup and decay components, and that it is plausible to interpret the “Modeling the scaling behavior” paragraph in Hoffmann et al. §2 as conjecturing that both components contribute to the scaling law discrepancy. However, based on Appendix A of Hoffmann et al. as well as the subsequent literature discussing this paper, we believe that Hoffmann et al. emphasize the decay component of the learning rate schedule in their conjecture.

A closer look at Hoffmann et al. Appendix A: In §2, Hoffmann et al. refer to Figure A1 to provide evidence for the claim that “setting the learning rate schedule to approximately match the number of training tokens results in the best final loss regardless of model size.” However, in Figure A1 they only vary the decay component of the learning rate, keeping the warmup. Hence, it appears that — in their conjecture regarding the scaling law discrepancy — Hoffmann et al. equate learning rate schedule with learning rate decay.

Subsequent literature: The following sources treat the conjecture of Hoffmann et al. as pertaining to only learning rate decay, suggesting that the above interpretation of Hoffmann et al.’s conjecture is common in the literature:

• Hu et al. [44, §4.1] write that

  … Hoffmann et al. (2022) make a key observation that setting $T > S$ results in dropped performance while setting $S = T$ results in improved training efficiency, confirming that the learning rate shouldn’t be kept high throughout the training.

  They do not mention aspects of the learning rate schedule other than the decay period.

• In “Scaling laws in compute-optimal training” (to be cited in the revision) Hägele et al. write that

  Importantly, the Chinchilla project (Hoffmann et al., 2022) showed that the cosine schedule achieves optimal loss only when the cycle length matches the training duration, but underestimates the model performance during training.

  The cosine cycle length controls the learning rate decay, not the warmup.

• A LessWrong blog post titled “New Scaling Laws for Large Language Models” discusses the scaling law discrepancy and states that

  It looks like OpenAI used a single total annealing schedule for all of their runs, even those of different lengths. This shifted the apparent best-possible performance downwards for the networks on a non-ideal annealing schedule. And this lead to a distorted notion of what laws should be.

  Again, no mention of warmup.

We nevertheless agree that the interpretation of Hoffmann et al.’s conjecture is subtle and requires further justification and clarification, which we are happy to add to the revision. In particular, in line 8 (the abstract) we will write “Counter to a hypothesis implied in Hoffmann et al.” and in line 30 we will remove the word “decay” from “tailoring the learning rate decay schedule”. In addition, we will add an appendix with a detailed discussion about the conjecture of Hoffmann et al., along the lines of our response above.

Question 1: Warmup in hyperparameter sweep
We use the same heuristic for choosing warmup duration as discussed in section 3.3, equating the warmup duration (in tokens) with the number of model parameters. We will add this detail to Appendix E.

Question 2: Cooldown value
We decayed the learning to 0.01 of its peak value following Gadre et al. [14], which was the source of most of the initial hyperparameters used in our experiments. Following the reviewer’s question, we ran a small-scale experiment to test the potential effect of using a different final learning rate. In particular, we train a model with 108M parameters on the RefinedWeb dataset. We use the same hyperparameter as in Table 3, with a training duration of ~2.1B tokens. We use three different values for the final learning rate at the end of the cosine decay - 0.1%, 1%, and 10% of the peak learning rate. Finally, we evaluate the models on held-out RefinedWeb data and calculate the validation loss and standard deviation due to sampling as described in L583. We tabulate the results below:

This experiment suggests that the precise level of final decay has little impact on the quality of the trained model, with no statistically significant difference between decay to 1% and decay to 10%. Therefore, we do not believe the final decay level affects the scaling laws we obtain.

I think that my current score is appropriate. Thank you.