Training on the Test Task Confounds Evaluation and Emergence

We thank you for your thoughtful review.

coloring in Figure 3

We used similar color schemes for Figure 1 and Figure 3 to better convey the message that the difference in performance-per-computer between older and newer models strongly resembles that between older models and older models fine-tuned on task data. Indeed, at a glance Figure 1 and Figure 3 look very similar – this is because the performance-per-compute of newer models resembles that of older models fine-tuned on task data, both pre and post adjustment. The blue points on both plots are the same: the performance of older models. The orange points differ: In Figure 1 it is the performance of newer models, whereas in Figure 3 it is the performance of older models that were fine-tuned on the task data for one epoch.

Regression fit in Figure 8 bottom

The regression fit in figure 8 bottom is a log-linear fit rather than the piece-wise log-linear “hinge fit” of figure 8 top, which we describe in Equation 1. The goodness-of-fit of the log-linear fit conveys that fine-tuning on task data leads to cleaner log-linear scaling behavior: almost all of the variation in benchmark accuracy is explained by log-linear scaling of pre-training compute alone. This does not directly follow from the top plot, which instead conveys that the point of emergence occurs at smaller compute scales as one increasingly trains on task data. While both phenomena are closely related, we believe that separately presenting the two regression fits allows us to more clearly convey both points.

Final paragraph in Section 4.2.

Chinchilla scaling laws predict that some pre-training compute allocations (that is, the number of model parameters relative to the number of pre-training tokens) are more favorable than others. When trained on the same data distribution, Llama 2 70B should attain lower loss than Llama 3 8B according to the Chinchilla scaling laws. It is therefore somewhat surprising that, despite having similar pre-training compute and controlling for training on the test task, Llama 3 8B closely matches the performance of Llama 2 70B.

We thank you for your thoughtful review.

The p-values for signficance in figure 1 overestimate significance because of correlation due to models being from only a few model families.

We have re-run the significance tests using OLS with clustered standard errors, where we consider each model family a different group. While the p-values of course change, we get identical significance results. We will use this significance test moving forward. Thank you for raising this point.

The paper never actually directly checks the hypothesis of more in-distribution data being the cause of higher MMLU performance out of the box. I'd be interested to see (on open source models with available datasets, or by sampling unconditionally from base models) whether there is more MMLU-like data in model families which appear to do better on MMLU when using the naive evaluation method.

Most model families released after November 2023 are explicit about the use of benchmark data at pre-training time, see the general response. Open models with publicly available datasets are even more explicit on the use of instruction-data: StableLM 2 and Olmo 1.7 both include the FLAN instruction dataset in the pre-training data, which contains part of the MMLU auxiliary training data. Thus, we know with certainty that these two model families saw at least part of the MMLU auxiliary training set.

Directly determining whether such instruction-data is the cause of their higher MMLU performance would require re-training the models, with all being equal except for excluding such instruction datasets. This is computationally infeasible. Nonetheless, Olmo presents an interesting case study. The main differentiating factor between Olmo 1.0 and Olmo 1.7 (both trained early 2024) is their pre-training data, and in particular including “content specifically sourced to improve model performance on tasks requiring specialized knowledge (e.g. arXiv, Stack Exchange) and complex reasoning (e.g. OpenWebMath, Flan)” [1]. While Olmo 1.0 7B performs roughly random-chance on MMLU, Olmo 1.7 7B attains a 24 point improvement on MMLU.

[1] AllenAI, OLMo 1.7–7B: A 24 point improvement on MMLU, https://allenai.org/blog/olmo-1-7-7b-a-24-point-improvement-on-mmlu-92b43f7d269d. 2024.

Thank you for the thoughtful review.

It is not clear whether their findings would apply to already fine-tuned models.

To address this limitation, we have conducted substantial additional experiments, please see the general response.

Alternative analysis on whether much more models after Nov 2023 include instruction data in the pre-training set could similarly justify why Nov 2023 is a good cut-off point.

Please see the general response. Indeed, most models released after November 2023 use instruction data at pre-training time. We verify that shifting the cut-off by a few months (e.g., September 2023 or January 2024) leads to very similar results, see Figure 10 and Figure 11 in the Appendix.

Ultimately, benchmarks are used to understand how good a model is at a general skill, like mathematical reasoning for MMLU and GSM8K. [...] Would your method merely reliably measure how well models perform on a specific benchmark, rather than how well models are at the ability it's meant to measure?

The main purpose of machine learning benchmarks is the relative comparison of models under standardized conditions. They enable the competitive empirical testing that ultimately drives scientific progress in our field [1, 2]. Our proposed benchmarking methodology ensures fair relative model comparisons in light of the disproportionate large effect that task data has on benchmark evaluations. Whether and the extent to which benchmark evaluations may be indicative of broader “abilities” is a much debated topic (e.g., [3, 4, 5]) that is tangential to our work.

[1] Liberman, Mark. "Obituary: Fred Jelinek." Computational Linguistics 36.4 (2010).

[2] Hardt, Moritz. “The Emergening Science of Benchmarks”. ICLR Invited Talk (2024).

[3] Bowman, Samuel, and George Dahl. "What Will it Take to Fix Benchmarking in Natural Language Understanding?." NAACL HLT (2021).

[4] Raji, Inioluwa Deborah, et al. "AI and the everything in the whole wide world benchmark." NeurIPS 2021 Benchmarks and Datasets track (2021).

[5] Liang, Percy, et al. "Holistic evaluation of language models." TMLR. (2023).

Thank you for your thoughtful review.

Lack of consideration of instruction-tuning models.

To address this limitation, we have conducted substantial additional experiments, please see the general response.

How was November 2023 chosen as the cutoff? I wonder what would have happened if you had chosen November 2022. How would the results be affected?

We chose November 2023 as the cut-off because models released in late 2023 start to mention the use of instruction data at pre-training time, please see the general response. We additionally show the results when choosing November 2022 as the cut-off point in Figure 19. Here, the blue points correspond to the Pythia model family. Models newer than Pythia outperform Pythia. However, after giving all models the same amount of task specific data, we find no evidence for a significant difference in performance between the Pythia model family and all other models. This is consistent with our extended discussion in Section 4.1 “Comparing model families”. Note that the pre-training dataset of Pythia is the Pile, a collection of curated datasets that are unlikely to contain much test task data.

We also find it illuminating to contrast the performance of only model families released after November 2023, that is, around the time that models start to report actively using task data at pre-training time. Since we compare models released between November 2023 and May 2024, we choose February 2024 as the cut-off point. We plot the results in Figure 20. We find no evidence of a significant difference in performance between models.

How much could hparam tuning affect the results? How was the initial sweep in Appendix A.2 chosen?

We chose the initial hyperparameters according to our prior experience fine-tuning language models. We performed minimal tuning due to its expensive nature. We found the learning rate to be, by a large margin, the single most impactful hyperparameter. To alleviate concerns regarding the potential impact of poor hyperparameter tuning on the results, we have performed a sweep on MMLU with the following learning rates: [6e-5, 2e-5, 6e-6, 2e-6, 6e-7]. We are unable to perform more extensive sweeps due to their large computational cost. Note that we originally used 2e-5 for most models, and 2e-6 for models for which 2e-5 lead to instability. We fine-tuned a total of 50x4=200 additional models for such a sweep. No models benefit from smaller learning rates than 2e-6, and only one model benefits from larger learning rate than 2e-5, Pythia 70M (the model with smallest pre-training compute). Thus, the optimal learning rate is inside the boundary of the sweep, except for Pythia 70M.

Since the concern is that older models match the performance of newer models because the hyperparameters used may be systematically more favorable to older models, we consider the following: we recreate our main experiment; with older models using the originally chosen learning rate, but choosing for the newer models the learning rate that leads to highest MMLU performance after fine-tuning. That is, we give newer models a systematic advantage in terms of hyperparameter search budget. We plot the results in Figure 21. The measured effect size of model recency on benchmark performance changes from $\hat{\theta}=-0.005$ to $\hat{\theta}=0.005$. Therefore, after giving new models a systemic advantage in terms of hyperparameter search budget, the effect size remains small. It is also not statistically significant.

Only a couple of main evaluation datasets are explored in the paper.

We consider the following 10 benchmarks: MMLU, GSM8K, ARC-Challenge, and HellaSwag (from the HuggingFace Leaderboard v1, Section 4.2) as well as MMLU Pro, GPQA, MuSR, BBH, and MATH Lvl 5 (from the HuggingFace Leaderboard v2, Appendix C). We pay particular attention to MMLU and GSM8K for two reasons: 1) because they are the two most influential benchmarks of the 2023-2024 period that we study, and 2) because, remarkably, we find strong empirical evidence of the effect of training on the test task on such highly influential benchmarks. However, we show that the phenomenon of training on the test task in not specific to MMLU and GSM8K, see our extended discussion in Appendix C regarding the benchmarks of the HuggingFace Leaderboard v2.