Establishing Task Scaling Laws via Compute-Efficient Model Ladders

Thanks for acknowledging our rigorous experiments. Below we address your comments:

1. Test vs validation set: As we mentioned in Sec 2.3, we use test sets whenever possible, and fallback to validation set when the test set (or the labels) is not available.

2. Convergence: The training data sizes do not indicate convergence. For the target models, we simply used the final pre-training checkpoints of OLMo, which is 4T tokens for the 7B model and 5T tokens for the 13B model. These are choices made by the OLMo model developers.

3. Over-parameterization: We ablated a functional form with fewer free parameters (7 instead of 9) in Appendix C.5. However, this led to degeneration of function fitting on some tasks, so we concluded that a bit of over-parameterization can help make our method more robust.

4. Predicting for larger models: A 32B version of OLMo 2 was released after we submitted the paper, and below is our prediction of its task performance (using the same set of fitted scaling law parameters as in our submission): | Task | Pred | Actual | | --- | --- | --- | | HellaSwag | 88.0 | 85.3 | | ARC-Challenge | 53.6 | 66.8 | | ARC-Easy | 77.8 | 88.8 | | PIQA | 82.9 | 83.6 | | CSQA | 79.5 | 75.4 | | Social IQA | 61.1 | 62.3 | | OpenBookQA | 45.4 | 53.2 |

   Our predictions are accurate on tasks like HellaSwag, PIQA, and Social IQA, which are all within an absolute error of 3 points (similar to the predictions for 7B and 13B models). The predictions are off on ARC due to the chaining error in two steps.

5. Transferring to another LM family: We don't think the OLMo ladders can be directly applied to predict for another LM family, because differences in training data mix, model architecture, etc can shift the parameters of scaling laws.

Thanks for acknowledging our contribution to opening up an important method in understanding language models. Also thanks for recognizing that our approach is sound and effective for addressing the problem. Below we address your concerns:

1. Comparison with the Llama 3 paper:
   • Comprehensiveness of benchmarks, and analyses: While we believe Llama 3 tried more benchmarks, we don't know how well their method works there because they didn't report them. In contrast, we report on a suite of 8 tasks in our paper, including tasks where our predictions weren't very good and we investigated the reasons (Sec 5). We hope this can provide the broader community more objective insights to this approach.
   • Fully open and reproducible: As acknowledged by the reviewer, this paper contributes to the science of open language model training. We documented our method and setup to a great level of details. In contrast, the Llama 3 paper leaves out many important details – such as the formal definition of the intermediate feature, the functional form and optimization algorithm for their "step 2" – making reproducibility challenging. uncovering how scaling laws can be established for language model training, and what properties tasks should have to be used for scaling laws, and provides detailed analyses since data and details of model training are clear. In addition, many details were missing from the Llama 4 paper, which reproducibility challening.
   • Over-trained vs. compute-optimal: The Llama 3 paper only reports their scaling law results on a 405B model for 15T tokens, which is in the compute-optimal regime according to the Chinchilla paper. It does not report results for their over-trained models, including their 8B nor 70B.
2. Pre-training loss as intermediate feature: We ablated this, with a similar pre-training loss (the LM loss on the C4 validation set). Our discussion can be found in Sec 2.2, Sec 4, and Appendix C.3.

Thanks for acknowledging our thorough evaluation and practical relevance to large model training. Below we address your concerns:

1. Lack of enough baselines: We have ablated many alternative design choices, including those proposed in prior work. These include alternative intermediate features (as in Gadre et al., 2024) and alternative input features (as in Dubey et al., 2024). Comparison to these baselines can be found in Sec 4.
2. Limited to ranking-style tasks: We agree, and we acknowledged extending task format as potential future work in Sec 8.
3. Real-world applications: A real-world application of our method is for LM developers to make decisions before kicking off big pre-training runs. With the predicted task performance, developers can decide if it aligns with their expectations and if not, iterate on the data and training settings w/o committing massive amounts of compute. This can make the model developing workflow more efficient and save cost.

Thanks for acknowledging that the paper is well written and timely for a critical problem in language model training. Below we address your comments:

1. Comparison with related work, such as pre-training loss as intermediate feature: We indeed ablated this in our experiments (please check our discussion is in Sec 2.2, Sec 4, and Appendix C.3) with a similar pre-training loss (the LM loss on the C4 validation set). We found that task loss leads to more accurate predictions for benchmarks with low variance, and LM loss works better for high-variance benchmarks.
2. Monitoring performance of intermediate checkpoints instead of establishing scaling laws: We highlight that establishing scaling law using our method can offer a prediction before kicking off the big pre-training runs, so LM developers can iterate on their recipe w/o committing massive amounts of compute. For example, training a 7B and 13B model to 200B tokens (a point where we can start making rough extrapolations) would cost $2.4 \times10^{22}$ flops, or over 4 times of that for training the ladder models. If at this point we decide that the model will not be good enough, we would need to start over and all such compute is wasted. Monitoring performance of intermediate checkpoints is still a good sanity check during big training runs, but there exists no principled way to extrapolate task learning curves, especially given the complication of learning rate schedule.
3. Sensitivity to prompts: To prevent the impact of prompts, we followed the standards introduced in the OLMES paper, which aimed at standardizing prompts for language model evaluation such that they are less susceptible to changes in prompts. We leave varying prompts and model families as future work.
4. Claim about current LMs are over-trained: We refer to over-training with respect to the compute-optimal number of tokens as suggested by the Chinchilla paper. Our target models are already heavily over-trained. For example, the 7B model trained on 3.95T tokens is 28x the compute-optimal as suggested by the Chinchilla paper. The extra 50B tokens at the end were not meant for adding more training data, but for completing the LR decay schedule (which makes a big difference when not complete).

Hi Reviewer Jq1a! Just a reminder that the discussion period for COLM papers has begun. Could you please take a look at the author response to review and let them know whether it addresses any of your outstanding questions?

Thanks, Your AC