(Mis)Fitting Scaling Laws: A Survey of Scaling Law Fitting Techniques in Deep Learning

Thank you for your constructive review and feedback - we also hope that this paper will enable researchers and practitioners to train large-scale models more effectively. We address your concerns below:

• Recommended action: We choose to avoid being overly prescriptive in our recommendations because there is no set of actions which can guarantee a good scaling law fit, nor even make a fit extremely likely.
  (1) It is intractable to establish what the desired final scaling law is, and therefore to measure the goodness of any scaling law fit, because we don't know the ground truth of model performance at all scales. Any attempt to estimate the goodness of a scaling law fit can only consider the goodness fit at a small number of points with limited scale, and it is unclear how heavily to weigh the goodness at each point.

  (2) As seen in our S7 analyses, many decisions in our checklist have a number of reasonable options, but those reasonable choices lead to a wide range of scaling law fits, and the observed variations do not follow any clear pattern. It is probable that variations would be even harder to predict when varying model architectures or other design decisions.

  However, it is certainly possible to determine that some scaling law fits are plausible or highly implausible, and to observe the stability of fits. For example, in Figure 2(a), neither of the recommended data/parameters ratios of ~1/2200 or 3000/1 at 10^25 FLOPs are likely to be the true optimal settings, and the loss predictions at those points are also unlikely to be close to ground truth. Based on these observations, we can make some more concrete recommendations, with the caveat that following the recommendations is no absolute guarantee of good scaling law fit. For example, Appendix Figure 3(c) suggests the importance of using a large range of data/parameter ratios and absolute model parameter counts across the experiments, since using too narrow a range can skew the final fit dramatically. Additionally, no paper we know of has achieved a plausible scaling law fit by directly optimizing a power law form for performance prediction with 5 or more scaling law parameters, so we recommend considering an IsoFLOP or multi-stage approach (e.g., fitting a relation between L and C, then a relation between C and optimal N). We will update the camera ready with these recommendations.

• Section 7: We have expanded results in Figure 3 (Appendix). We will also update the camera ready version to include additional analyses, as well as models up to 1B parameters (not included in the submission due to resource constraints). It is unfortunately impossible to enumerate and execute all possible variations on scaling laws, but we intend to add as many as possible given space constraints. We would be eager to hear any specific suggestions the reviewer might have.

• Surveys: While surveys and position papers at ML conferences are rare, we believe that the ICLR audience is the correct audience for this work. Given our contributions of (1) elucidating and organizing the factors influencing scaling law studies, (2) empirically analyzing several of these factors and (3) providing a checklist to help both researchers and readers of scaling law papers, we hope that there is room for work like ours at ICLR.

We hope that this is helpful, and that you consider increasing your score if your concerns are addressed. We are happy to answer follow up questions.

Dear Reviewer,

Based on the discussion, we have updated the draft:

• We have defined the categories of our checklist explicitly in the Appendix.
• We have added a section on Recommendations in Appendix E, and provided an example checklist.
• We have added to Section 4 a discussion on the maximum scales of the papers we have surveyed.

If you find these updates to be sufficient, we hope that you will increase our scores accordingly. We are happy to answer any follow up questions.

Thank you for your positive review and suggestions - we hope that this paper will enable researchers and practitioners to train large-scale models more effectively. We will fix the typos flagged, and address your concerns below:

• Inherent Incompleteness of Checklists: We agree that a checklist cannot be exhaustive or complete. We do, however, attempt to go over which factors matter as extensively as possible. This is similar to how Model Cards (Mitchell et al [1]) are also not necessarily exhaustive, but are still useful. In fact, much of the data on these papers that we were able to gather was only possible because the authors open-sourced several artifacts (Table 5 in the Appendix). Despite the impossibility of any checklist being exhaustive, we believe laying out this checklist will enable those who read and build upon scaling laws papers to approach papers with skepticism, and compare results across studies. Moreover, similar to Model Cards, we hope that this facilitates more effective scaling studies by virtue of encouraging researchers to consider each decision in study development more carefully.

• Section 7.1: The trick of setting alpha=beta is taken from Muenninghoff, et. al. They do this to simplify the optimization problem, which now needs to fit only 4 scaling law parameters, instead of 5. This inherently assumes that the optimal data and model parameters scale linearly with each other – that their ratio is fixed. Hoffmann, et.al., fit a scaling law resulting in alpha ~= beta, and recommended fixing data:parameters at 20:1. We use alpha=beta as one setting in our comparisons, to understand the effects of making such an assumption (Figure 3a in the Appendix).

• Recommended action: We choose to avoid being overly prescriptive in our recommendations because there is no set of actions which can guarantee a good scaling law fit, nor even make a fit extremely likely.
  (1) It is intractable to establish what the desired final scaling law is, and therefore to measure the goodness of any scaling law fit, because we don't know the ground truth of model performance at all scales. Any attempt to estimate the goodness of a scaling law fit can only consider the goodness fit at a small number of points with limited scale, and it is unclear how heavily to weigh the goodness at each point.

  (2) As seen in our S7 analyses, many decisions in our checklist have a number of reasonable options, but those reasonable choices lead to a wide range of scaling law fits, and the observed variations do not follow any clear pattern. It is probable that variations would be even harder to predict when varying model architectures or other design decisions.

  However, it is certainly possible to determine that some scaling law fits are plausible or highly implausible, and to observe the stability of fits. For example, in Figure 2(a), neither of the recommended data/parameters ratios of ~1/2200 or 3000/1 at 10^25 FLOPs are likely to be the true optimal settings, and the loss predictions at those points are also unlikely to be close to ground truth. Based on these observations, we can make some more concrete recommendations, with the caveat that following the recommendations is no absolute guarantee of good scaling law fit. For example, Appendix Figure 3(c) suggests the importance of using a large range of data/parameter ratios and absolute model parameter counts across the experiments, since using too narrow a range can skew the final fit dramatically. Additionally, no paper we know of has achieved a plausible scaling law fit by directly optimizing a power law form for performance prediction with 5 or more scaling law parameters, so we recommend considering an IsoFLOP or multi-stage approach (e.g., fitting a relation between L and C, then a relation between C and optimal N). We will update the camera ready with these recommendations.

We hope that this addresses your concerns, and are happy to answer follow up questions.

[1] https://arxiv.org/abs/1810.03993

Thank you for your constructive review - we also hope that this paper will enable researchers and practitioners to train large-scale models more effectively. We address your concerns below:

• Comparison to Porian et al: Porian, a work contemporary to ours, is focused more narrowly on the specific differences between Kaplan et al and Hoffman et al scaling laws - namely, FLOP counting and training setup. While our work has similar motivations, we consider more broadly the factors that may change the conclusions of a scaling study, namely (1) the form of the hypothesis itself (2) training setup (which Porian et al touch upon), (3) evaluating the trained models and (4) fitting the scaling law. We empirically show that varying several of these factors can change the final scaling law by an order of magnitude.

• Goal of Paper: We agree that the ultimate hope is to make specific recommendations about scaling laws. As our paper has been presented, our main recommendation is for researchers who are fitting scaling laws to open source all artifacts, or, short of that, to report items on our checklist as fully as possible; and for those reading others' scaling law works, to approach such works with skepticism, closely examining each aspect of the methodology with our checklist.

• Recommended action: We choose to avoid being overly prescriptive in our recommendations because there is no set of actions which can guarantee a good scaling law fit, nor even make a fit extremely likely.
  (1) It is intractable to establish what the desired final scaling law is, and therefore to measure the goodness of any scaling law fit, because we don't know the ground truth of model performance at all scales. Any attempt to estimate the goodness of a scaling law fit can only consider the goodness fit at a small number of points with limited scale, and it is unclear how heavily to weigh the goodness at each point.

  (2) As seen in our S7 analyses, many decisions in our checklist have a number of reasonable options, but those reasonable choices lead to a wide range of scaling law fits, and the observed variations do not follow any clear pattern. It is probable that variations would be even harder to predict when varying model architectures or other design decisions.

  However, it is certainly possible to determine that some scaling law fits are plausible or highly implausible, and to observe the stability of fits. For example, in Figure 2(a), neither of the recommended data/parameters ratios of ~1/2200 or 3000/1 at 10^25 FLOPs are likely to be the true optimal settings, and the loss predictions at those points are also unlikely to be close to ground truth. Based on these observations, we can make some more concrete recommendations, with the caveat that following the recommendations is no absolute guarantee of good scaling law fit. For example, Appendix Figure 3(c) suggests the importance of using a large range of data/parameter ratios and absolute model parameter counts across the experiments, since using too narrow a range can skew the final fit dramatically. Additionally, no paper we know of has achieved a plausible scaling law fit by directly optimizing a power law form for performance prediction with 5 or more scaling law parameters, so we recommend considering an IsoFLOP or multi-stage approach (e.g., fitting a relation between L and C, then a relation between C and optimal N). We will update the camera ready with these recommendations.

• Surveys: While surveys and position papers are only a small portion of ML conference submissions, we believe that the ICLR audience is the correct audience for this work. Given our contributions of (1) elucidating and organizing the factors influencing scaling law studies, (2) empirically analyzing several of these factors and (3) providing a checklist to help both researchers and readers of scaling law papers, we hope that there is room for work like ours at ICLR.

• Additional comments: Thank you for bringing these to our attention. We will update the draft with vector versions of the images, and update the year.

We hope that our clarifications are helpful, and that you consider increasing your score if your concerns are addressed. We are happy to answer follow up questions.

Dear Reviewer,

Based on the discussion, we have updated the draft:

• We have defined the categories of our checklist explicitly in the Appendix.
• We have added a section on Recommendations in Appendix E, and provided an example checklist.
• We have added to Section 4 a discussion on the maximum scales of the papers we have surveyed.

If you find these updates to be sufficient, we hope that you will increase our scores accordingly. We are happy to answer any follow up questions.

Thank you for your constructive review - we also hope that this paper will enable researchers and practitioners to train large-scale models more effectively. We address your concerns below:

• Model scale: While state-of-the-art LLMs have hundreds of billions of parameters, there is emerging evidence [1] that it is not necessary to reach billions of parameters in order to study scaling laws. We note that of the papers we survey, most do not reach a maximum model scale of more than 1B parameters - we will add a discussion of this in the next draft. Notably, we refer to Kaplan, et. all, and Hoffmann, et. al., which use maximum model scales of 1.5B and 16B, respectively, and heavily overrepresent models with fewer than 100M parameters in their scaling law data.

  We are training up to 1B parameter models and will include these in the camera-ready, but were unable to include them at submission time due to resource constraints - training this model for 20B tokens costs about 1000 GPU-hours on Nvidia A40 machines, and a paper which includes this model size would inevitably need to partially or fully train several such models.

• Confusing terminology: The particular stage of scaling law fitting that we call "data extraction" is evaluating various checkpoints of our trained models to collect data points that will be used to fit a scaling law equation in the next stage. We could call this “Data for Fitting the Scaling Law” instead although we were trying to be more succinct - we are open to suggestions for new terminology. We will ensure that there are more precise definitions of this term (and other terms introduced) in the next draft.

We hope that this is helpful, and that you consider increasing your score if your concerns are addressed. We are happy to answer follow up questions.

[1] https://openreview.net/forum?id=xGM5shdGJD

Dear Reviewer,

Based on the discussion, we have updated the draft:

• We have defined the categories of our checklist explicitly in the Appendix.
• We have added a section on Recommendations in Appendix E, and provided an example checklist.
• We have added to Section 4 a discussion on the maximum scales of the papers we have surveyed.

If you find these updates to be sufficient, we hope that you will increase our scores accordingly. We are happy to answer any follow up questions.