DataDecide: How to Predict Best Pretraining Data with Small Experiments

We thank the reviewer for their detailed clarifying questions and are happy they found our work has “comprehensive experiments and detailed analyses.”

Expanded results

We appreciate the reviewer’s encouragement to polish our paper. Since submission we have fully revised our paper and extended our suite with more small runs and examined additional benchmarks. Updated figures, which we hope are more readable, can be viewed here.

We find that the conclusions to our research questions hold over these new results: 1) Compute trades off with decision accuracy and different tasks get better signal for decisions. 2) 8 scaling law methods fail to outperform single-scale ranking; future work can use DataDos to iterate on these. 3) Continuous metrics over raw likelihood are always better at small compute; differences due to which length normalization is used are smaller. 4) Low run-to-run variance and large spread between recipes explain success; we can also use proxy metrics to get beyond noise floor for tasks like code benchmarks.

Clarity and Organization

Following the reviewer’s suggestion, we will move the discussion of scaling laws ahead of proxy metrics in section 3 (results) to better align with the order of topics in section 2 (methods). Also regarding the reviewer’s question 1: In the third figure in our submission we identified best metrics based on which achieves the highest decision accuracy over all of the small compute sizes used for prediction. Indeed, the target budget remains the same (1B 5xC) but we consider which metric gets best decision accuracy for the most small experimental budgets. Regarding the “best outcomes,” this quantity would be better defined as measuring the average decision accuracy over tasks and scales when using the best metric for each task. In our revised figure (#4), we instead use a new, more direct figure to visualize the relationship of compute to decision accuracy for each type of metric with a specific length normalization. Regarding question 2: 150 is the number of combinations visualized in the former Figure 5, since the number of rows is halved to save space. Thanks to the reviewer’s feedback we’ll be sure to make this more clear.

We thank the reviewer for their clear and helpful feedback and supporting that “using 1B as the target model is sufficient for generalization purpose given the compute budget.”

Expanded results

Since our submission we have enriched our analysis and extended our suite with more small runs and examined additional benchmarks. Updated figures can be viewed here.

We find that the conclusions to our research questions hold over these new results: 1) Compute trades off with decision accuracy and different tasks get better signal for decisions. 2) 8 scaling law methods fail to outperform single-scale ranking; future work can use DataDos to iterate on these. 3) Continuous metrics over raw likelihood are always better at small compute; differences due to which length normalization is used are smaller. 4) Low run-to-run variance and large spread between recipes explain success; we can also use proxy metrics to get beyond noise floor for tasks like code benchmarks.

Explanations of better decisions

We appreciate the reviewer’s encouragement to provide further discussion of the “roots” of differing decision accuracy. In our revised figure (#5) we provide an analysis of how decision accuracy of different tasks and metrics can be explained in part by low run-to-run variance and a wide spread of performance values for different data. Using the correct probability proxy metric sees wider spreads for many tasks, though some instead see reduced noise. The reviewer’s questions about saturation in particular are interesting, as this could either lead to easy decisions (if some recipes saturate at lower performance than others) or to noisy decisions (if all recipes saturate around a similar value by the target scale). However, in our suite we do not observe saturation on any of the benchmarks we examine. Instead we see near trivial performance on Boolq, which leads to it having low decision accuracy as its target rankings are mostly determined by noise. Thus we do not recommend it for use on predictions over the scales we examine.

Likewise we appreciate the reviewer’s encouragement to provide further analysis of scaling laws vs. single ranking. Our smaller results let us consider a wider range of compute budgets for scaling law fits, and we elevate our 8 additional scaling law variants (previous appendix line 770) to the body of our paper (see new figure #3). Our finding that these baseline scaling laws do not outperform single scale ranking holds over the breadth of sizes and variants. The latest results show that the single ranking approach is strong at even smaller compute budgets than we previously found, and we believe future work is required to ascertain the specific ways in which scaling laws fail despite getting low prediction error.

Aggregate benchmark analysis Regarding the reviewer’s question: when figures do not name a specific task for downstream performance, the quantity is the macro average of OLMES (line 202). We find that our claims hold when aggregating: intermediate checkpoints decide as well as compute equivalent final models, single scale experiments outperform scaling laws, continuous metrics outperform discrete ones at small scales. As we examine in new (and former) figure (#2) slope and range of the positive relationship between compute and decision accuracy depends a lot on which benchmark is used, but the other claims we make hold.

We thank the reviewer for their thoughtful questions and are glad they found our “paper is well motivated and the results are very helpful.”

The reviewer writes that “the main contribution of this paper is that small-scale pretraining experiments (e.g., using 150M–parameter models) can reliably predict which data recipes will perform best.” We highlight that our paper has two main contributions: 1) We release an extensible framework, easily updated with new prediction methods, benchmarks, and small models. 2) We provide practical recommendations based on empirical study of decision accuracy that we observe to be reliable over wide ranges of settings, such as model sizes (now from 4M to 1B parameters) and benchmarks (including knowledge QA, commonsense, and reasoning, as well as math and code now).

Expanded results

We have extended our suite with more small runs and evaluations. Updated figures can be viewed here.

We find that the conclusions to our research questions hold over these new results: 1) Compute trades off with decision accuracy and different tasks get better signal for decisions. 2) 8 scaling law methods fail to outperform single-scale ranking; future work can use DataDos to iterate on these. 3) Continuous metrics over raw likelihood are always better at small compute; differences due to which length normalization is used are smaller. 4) Low run-to-run variance and large spread between recipes explain success; we can also use proxy metrics to get beyond noise floor for tasks like code benchmarks.

Ease of use and tooling

The reviewer encourages us to make DataDos easy to use, for instance adding a “new benchmark (e.g. shopping_mmlu).” In our public release of 1K models, 25 corpora, and code for pretraining, evaluation, prediction, and decision accuracy (line 104) we will include step by step documentation for adding new tasks, prediction methods, and small pretraining experiments. We will host all checkpoints as native models on Hugging Face. Running new tasks can be as simple as adding a new argument to the open source evaluation framework that we leverage. Our aim is that researchers with a range of compute can build on our suite: 1) by trying new prediction methods with lightweight manipulations of our results CSV such as smoothing or curve fitting, 2) adding new benchmarks over our released checkpoints for just the cost of inference, 3) or even pretraining additional small models for a fraction of the target model cost (e.g., the 9 additional model sizes we trained since submission cost only ~2% of our total compute).

Reliable and generalizable claims The reviewer wrote that our experiments covered “a narrow range of models (150M to 1B)”, so we’ve significantly expanded our results to include 9 additional even smaller model sizes as small as 4M, or ~0.3% of the 1B compute. We continue to find consistent trends of compute to decision accuracy that support our claims for our 4 research questions (line 50). We agree that evaluation noise is a concern at small model scale, so our work is the first in this area to train multiple random seeds, even for the largest model sizes (line 152) We use the mean of these to make our decision targets more reliable. The reviewer asked about how are claims generalize, so we added four new heldout benchmarks and found that our preferred continuous proxy metric also leads to better decision accuracy when target performance is above the noise floor (new Figure #6).

Focus on exploring data difference

Our study investigates dimensions of differences not yet studied extensively; we explore 25 different data recipes, an order of magnitude more than previous suites. We chose to spend our compute budget by overtraining (5x the Chinchilla multiplier) many models (14 different sizes) and evaluating many checkpoints per model to estimate what performance would be like with different token-parameter ratios. Our conclusions are very consistent across intermediate and final checkpoints, suggesting that different ratios might give similar results, but we leave that to future work to confirm.