Quantifying Variance in Evaluation Benchmarks

We thank the reviewer for their time and address their feedback/questions below:

Base models do not handle instruction format well, in general struggling with handling question-answer interaction. This is usually installed via instruction tuning, after which models adhere to various instruction like interactions. I think thus to make proper statements about effect of variance or in general about how to render benchmark signal useful for predictions about model quality, benchmarks that do not rely on certain instruction template are not a good choice when dealing with base models.

We would like to point out that instruction-tuning has little effect when conducting evaluations using log-likelihood (https://huggingface.co/blog/open-llm-leaderboard-mmlu). In this setup, the possible choices/text completions are appended to the prefix prompt and the option with the lowest NLL is chosen as the model response. In our paper, this corresponds to AGIEval, ARC-C, COPA, Hellaswag, MMLU, PIQA, and SIQA. Moreover, Dubey et. al, https://arxiv.org/abs/2407.21783 (Figure 14) shows that the prompt variations and option orders are fairly robust and exhibit negligible variance in this setup of NLL-based evaluation. This is a standard setup for evaluating pre-trained base models as seen in GPT series (GPT-3: https://arxiv.org/abs/2005.14165, GPT-4: https://arxiv.org/abs/2303.08774), Llama series (Llama 1: https://arxiv.org/abs/2302.13971, Llama 2: https://arxiv.org/abs/2307.09288, and Llama-3 https://arxiv.org/abs/2407.21783) and other model releases like Mistral/Gemini as well.

In my opinion, when working with such benchmarks, authors should have instruction-tuned base models, conducting measurements after that. Benefit of continuous metrics might be thus due to base models not handling instruction like benchs and not due to actual benchmark content related to complexity of problems posed in it.

Since we study the variance that is useful for building scaling laws/doing pre-training datamix ablations, it doesn’t make sense to do instruction fine-tuning for each FLOPs scale and at every pre-trained checkpoint. Instruction-tuning should only be done at the end and their evaluation setup is different from those of pre-trained models. Pre-trained evaluations are essential during the early stages of pre-training for selecting the best architecture/datamix that best fits the scaling law curves.

Same might hold for the "curious MMLU" case presented by authors, where cloze variant shows smaller variance and better SNR than multiple choice format standard MMLU form - this might be again just due to base models not being able to handle problem formulation template.

Again, we would like to emphasize that we are performing evaluations using log-likelihood (details presented in Appendix A), in which the model is not doing any generations, and we limit the output space to the possible option letters/texts, and compute NLLs and choose the option with the lowest NLL as the model’s prediction.

Further weakness in my opinion is that many of the claims are based on pre-training from scratch done only on one model scale of 7B. It might be more insightful to see trends across scales, even smaller ones. Experiments on larger scale are though expensive, even if going bit further to 13B.

We agree with the reviewer, however, we would like to point out that for building scaling laws, the FLOPs budget matters more as opposed to different model scales. We train 7B models for 210B tokens accounting for a FLOPs budget of $10^{22}$, which is a fairly standard budget for building scaling laws. For example, Dubey et al., 2024 (Llama 3: https://arxiv.org/pdf/2407.21783) and Hoffman et al., 2022 (Chinchilla: https://arxiv.org/pdf/2203.15556). We believe that the reference variance values are representative of this budget: Table 2 column 3 - Disc Std for $10^{22}$ budget, and Table 1 column 5 - $\sigma(\mathcal{S}, \mathbb{M})$ for all budgets $\leq 10^{22}$.

It is also not quite clear how the claim of better suitability of continuous metrics is backed up. It seems to me there is no clear evidence presented by authors that using those metrics eg indeed allows better prediction from earlier to later training stages or from smaller to larger scale pretraining.

All of the results in Sections 3.2 and 3.3 point to the utility of continuous metrics. Table 2 shows how continuous metrics have higher SNR compared to discrete metrics. Figures 1, 2, and 6 show how continuous metrics are better for predictability and stability in tracking as the seed variance is lower (represented by the box heights in the plots).

Again, we would like to emphasize that we are performing evaluations using log-likelihood (details presented in Appendix A), in which the model is not doing any generations, and we limit the output space to the possible option letters/texts, and compute NLLs and choose the option with the lowest NLL as the model’s prediction.

Thanks for pointing out the mode of evals being throughout NLL. However, my point was that, independent of output eval, form of the problem formulation itself presented on the input is instruction template based (for standard MMLU), and therefore base model processing input in such format would struggle especially in early training phases, in contrast to MMLU cloze that is not relying on such formatting (as also described in Appendix B). I think that might be a confound when stating difference between MMLU standard and MMLU cloze that is apart from discrete vs continuous metrics. I guess one way to test it would be to have a discrete version that is not relying so heavily on specific instruction format to pose the problem on the input. I think same can be argued in general - to show difference discrete vs continuous, one should remove as far as possible confound of presenting problems in a instruction template form for both evals if testing with base models.

... for building scaling laws, the FLOPs budget matters more as opposed to different model scales ...

I cannot quite follow this point. While FLOP budget surely matters, we would like to extrapolate observed trends towards larger scales. This can be only done well if scanning through a scale span broad enough on smaller scales. I cant imagine how we can do it from few or even single point (no matter whether FLOP related, or combined model/data scale). I think it would be important to observe how variance behaves on a span up to some reasonable high FLOP value, including scan through model scales, as it might be that variance also behaves differently depending on model scale, and this can be overlooked if estimating from single model scale only. This is also in line with comment from other reviewers, eg 7Zuc https://openreview.net/forum?id=E2RyjrBMVZ&noteId=C89i4E8Bo3 I think it would be important to see whether on smaller model scales, that are of course also important for deriving scaling laws in practice, same advantage continuous vs discrete metrics can be stated. This might also allow a bonus to predict some properties of variance for higher FLOP/model scales without running experiments there, although I agree that 7B is reasonable upper threshold here. Smaller scales <7B are not so expensive, so not sure why would not it be a good thing to do.

In general, I see the merits of the study in attempt to clearly point out advantages of continuous metrics. However, I still find the write up hard to read. I also still struggle to grasp the relevancy of the item analysis and item response theory (IRT) part for the presented results. The work seems to be building up on https://arxiv.org/abs/2406.04391, and I wish for the same level of clarity achieved there to give a better score that is still not there.

We thank the reviewer for their comments and address the weaknesses/questions below:

It does not compare the variance caused by random seeds with the impact of other random factors during evaluation (such as option order, prompts, etc.). It does not explore whether the variance is further amplified when models with different seeds encounter situations like randomized options.

It’s true that we don’t consider other factors like decoding strategies, temperature, prompt variations, etc. But we would like to add that this has little impact on log-likelihood based evaluations (in which no model generation is involved). The possible choices/text completions are appended to the prefix prompt and the option with the lowest NLL is chosen as the model response. In our paper, this corresponds to AGIEval, ARC-C, COPA, Hellaswag, MMLU, PIQA, and SIQA. Moreover, Dubey et al. 2024, https://arxiv.org/abs/2407.21783 (Figure 14) show that the prompt variations and option orders are fairly robust and exhibit negligible variance in this setup of NLL-based evaluation.

The paper primarily showcases the overall variance of 210 7B model checkpoints. Given that models of different sizes exhibit significant differences in performance when trained with varying numbers of tokens, the overall variance statistics may have limited reference value for models trained with fewer or more data.

We train 7B models for 210B tokens accounting for a FLOPs budget of $10^{22}$, which is a fairly standard budget for building scaling laws. For example, Dubey et al., 2024 (Llama 3: https://arxiv.org/pdf/2407.21783) and Hoffman et al., 2022 (Chinchilla: https://arxiv.org/pdf/2203.15556). We believe that the reference variance values are representative of this budget: Table 2 column 3 - Disc Std for $10^{22}$ budget, and Table 1 column 5 - $\sigma(\mathcal{S}, \mathbb{M})$ for all budgets $\leq 10^{22}$.

Considering that the author trained different models from scratch, such as Hellaswag, which consistently outperformed random choice across a wide range of checkpoints, it would be beneficial if the author could use intermediate checkpoints to demonstrate the benchmark variance at different stages of training and performance. Furthermore, showing how benchmark variance changes with training progress could be helpful for models of various sizes and training data volumes.

We do report the variance change over the course of training for all benchmarks in the paper in Figures 1, 2, and 6 for both discrete and continuous metrics. The y axis reports the performance metrics, and the box plots at each step show the variance corresponding to that step.

When calculating Seed Variance, would it be more reasonable to exclude checkpoints that are clearly still within the random result range from the statistical analysis?

We don’t believe that would help as the LLM community is still using numbers that are near chance to compare performance across different models, for example, MMLU comparisons are done in https://arxiv.org/pdf/2310.04564, https://arxiv.org/pdf/2401.02385, https://arxiv.org/pdf/2312.06550, etc. but the performance is near chance.

We thank the reviewer for their comments and address the questions/feedback below:

(Part 1) Seed Variance and Its Impact on Benchmark Scores

Does seed variance contribute overwhelmingly to the overall variance which makes it a critical factor to study, beyond [1,2,Arena]? I would like a better distinction in the work for why studying seed variance specifically is important, given the existing literature on other variance sources.

We agree with the reviewer that the focus in this paper is on seed variance. But, this variance is quite an important factor for evaluations done during the early stages of pre-training. For example, experiments and ablations performed at different compute (FLOPs) budgets to build scaling laws, selecting the best data mixture and weights for pre-training, model architecture ablations. High variance during the early stages of pre-training and at smaller FLOPs budgets can lead to inaccurate selection of architecture/datamix leading to waste of significant amounts of computational resources for larger pre-training runs.

(Part 2) Efficient Benchmarking in High-Variance Contexts

I doubt those papers claim their methods are intended to reduce or understand variance. Could the authors provide a citation for this claim? If no direct claims about variance reduction exist in the cited works, could the authors discuss in the work why they believe these methods should be evaluated in terms of variance reduction.

The tinyBenchmarks paper (https://arxiv.org/pdf/2402.00838) motivate the use of IRT methods for pre-training ablations in Sections 1 and 5, which in hindsight may not be effective due to increased variance (Figure 5 shows this clearly). Through our analysis, we want to highlight that any kind of efficient benchmarking (tinyBenchmarks: https://arxiv.org/pdf/2402.00838, Mixeval: https://arxiv.org/abs/2406.06565, SMART: https://arxiv.org/abs/2410.20245) is limited in its use for pre-training ablations, building scaling laws, etc. especially during the early stages of pre-training because they exhibit high variance and hence are not reliable to distinguish between the various experimental settings which will transfer to larger models as well.

While I agree with the authors that generally, subsampling might lead to critical issues in benchmarking – the results presented seem underwhelming given I fully buy the claim of using continuous metrics to reduce variance. Specifically, the results in Table 7 indicate that Kendall’s tau still remains quite high and relatively stable. Do the authors believe this is damning evidence against efficient benchmarking methods?

In Table 7, the last two columns are the most relevant where the percentage change in flips can be up to 30.77%. This shows that the efficient benchmarking methods built using weaker models are not transferable to more capable models, and can lead to inaccurate estimation of the actual final scores on the full evaluation test set.

We thank the reviewer for their comments on our paper and respond to their questions below:

The framing of ‘variance’ in the paper seems too broad. There are other possible kinds of variance worth exploring or mentioning.

We agree with the reviewer that there are other sources of variance, however the focus in this paper is on variance in evaluations during pre-training, where seed variance is an important factor in building scaling laws, and variance as a consequence of the fact that eval sets are finite samples of all questions that could be asked impact how we should interpret their results and when differences are statistically significant.

While studying the training seed variance is useful, this is really only feasible for smaller models, as it would be too expensive for larger models. This may reduce the utility of the results in large model comparisons.

We kindly disagree with this assessment because before any large scale pre-training runs for a huge model, there’s a lot of investment on small scale experiments to build the scaling laws, decide the model architecture or the pre-training datamix, etc. The small scale experiments are usually performed on a lower FLOPs budget, where the variance can play a significant role. High variance can cause inaccurate selection of model architecture/datamix that hurt the large-scale pre-training runs.

The paper could mention previous work such as Picard 2021 (https://arxiv.org/abs/2109.08203) on the impact of training initialization seed variance, or fine tuning seed variance [Dodge 2020 (https://arxiv.org/abs/2002.06305)]. And also extended discussions on how obscuring or not disclosing these variances can be harmful to the evaluation process (e.g. [Leech 2024 (https://arxiv.org/abs/2407.12220)]).

We thank the reviewer for the extra references, and will make sure to include this in the updated version of the paper.

Section 3.3 outlines a very interesting case of differences in evaluation results after a reformulation of the setup, could this be shown for other benchmarks? perhaps a similar pair?

Yes, this is applicable to other benchmarks as well, especially those that have a MCQ-based evaluation setup. The model learns the ability to answer MCQ-based questions later in training, and using a cloze format is helpful in tracking performance in the early stages of pre-training.

We thank the reviewer for their thoughtful comments and address the weaknesses and questions below:

Some important empirical details are missing, for example, could you provide more details on how you compute the SNRs for both discrete and continuous?

We compute SNR using the following formula, $\frac{\mu(\mathcal{S}, \mathbb{M}^{210B})}{\sigma(\mathcal{S}, \mathbb{M}^{210B})}$. We’ve updated it in the revised version.

Another important question that is critical for the takeaways of the paper: Is the unnormalised monotonocity the best metric here to capture the “stableness” of a benchmark? Shouldn’t that monotonicity be weighted by the monotonicity you would expect by chance? So something like a cohen’s kappa coefficient here seems more appropriate rather than just the direct unnormalised monotonicity. For eg, see the analysis in Geirhos et al, Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency.

It’s very hard to quantify the monotonicity expected due to chance as it’s not evidently clear what mean value to take over the course of time. If we just treat each step independently, then the chance performance is the same across all steps, resulting in a monotonicity of 0. But obviously, it’s not independent and develops for each benchmark differently depending on the “hidden” scaling laws for that benchmark. There’s a wealth of literature pointing out the defects with Cohen's kappa coefficient (for e.g. Appendix B of https://arxiv.org/abs/2406.12624 for a short summary).

As for Geirhos et al, Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency, can the reviewer expand on how the error consistency analysis can be used for monotonicity?

Suggestion on formatting the paper: the results sections in 3 and 4 have some very important insights / takeaways for practitioners to adopt for pretraining and actual evaluation. I think it would majorly improve the paper if these could be highlighted at the end of each subsection in bold, or even better, a small box signifying the key takeaway.

We agree with the reviewer and will update the manuscript accordingly.

There are other kinds of variance inducing factors that haven't been investigated. For example, how does task type affect this variance? How does number of shots and choice of shots affect variance? How does model size impact variance?

We agree that there are other sources of variance as well, however, for pre-training ablations involving scaling laws and datamix selection, seed variance is an important factor of variance. We’ll add a discussion on other sources of variance while highlighting that this paper considers "seed" variance.

Why are the Disc Std in tab 2 and std in tab 1 different? Aren’t they computing the same metric over the same set of models? The means are exactly the same for the discrete metrics so I presume the stds should be too?

For the means computed in both Table 1 and 2, we use the final checkpoints across the different seed runs as defined in Lines 123-124. However, for the standard deviation in Table 1, we use the seed variance $\sigma(\mathcal{S}, \mathbb{M})$ definition provided in Lines 126-133, taking into account intermediate checkpoints as well. For the standard deviation in Table 2, we use only the final checkpoints across seeds for an accurate assessment of the signal to noise ratio. That’s why we use Disc Std to distinguish from the seed variance $\sigma(\mathcal{S}, \mathbb{M})$.

In tab. 2 what decoding strategy do you use for the generative tasks? The difference between GSM8k (0.99) and HumanEval (0.21) seems quite high for a difference in log-likelihoods. Are these token-length normalised likelihoods or unnormalised likelihoods?

We use greedy decoding (temperature $= 0$) for sampling and use the character-length normalized NLL for both GSM8k and HumanEval. The token-length normalized NLLs also exhibit similar differences as shown in Figure 8 (Section C.2) in the updated version of the paper.

Comment: Typo in fig 3 caption, should be item difficulty (y-axis) and item discriminability (x-axis)

Thanks for pointing it out, we have fixed it in the revised version of the paper.

For the analysis in fig 3, how does the correlation between item discrimination b/w train and test look like for another randomly selected set of train and test models? The key question is what the variance in the correlation obtained for a random split of random train-test models would be? This would make the conclusion (that the low correlation between train and test item discriminability is due to the differences in model capability) more strong and robust.

We have updated the paper with additional results. In Figure 10, you can see that for five different sets of train/test splits (with 14 models each in the test sets), the results are robust and the regression line fits are similar with high correlation on item discrimination between the train and test models in the random splits. This enforces the claim of low correlation in the train and test item discriminability is actually due to differences in model capability.

The “best" and “worst” models used for creating the train and test splits do not share the same training data mixtures right? So I would expect that the takeaways from fig 3, especially the ones involving the difficulty split are also confounded by different levels of data contamination with respect to the test sets?

Yes, the reviewer is correct. Since many of these models are just open-weights, we don't know the level of contamination of test sets in their pre-training datamixes. Assuming that the folks releasing the models are fair players and that the performance difference for a given pair of models accurately reflects model capabilities, we believe our findings still hold.

For the points in section 5, the main claims revolve around using the IRT benchmarks themselves. How much of that can be explained by the increased variance from just having a smaller number of test samples? i.e. how would the results look like if instead of the IRT test set, a random set of evaluation points of the same size as the IRT set were used for the analyses?

Since the optimization objective for IRT benchmarks is to preserve the mean performance and taking any random sample won't preserve it, we believe it's not an accurate comparison between the two cases, and the random sample would not give any kind of signal as provided in Figure 5. Our goal for section 5 is to highlight that any efficient benchmarking in general (and not just IRT) can be very noisy and lead to increased variance during the scaling laws/pre-training ablations, and not effective at all compared to the full evaluation.

I am not an author of this paper, nor am I affiliated with the authors. I am an independent practitioner with significant experience in LLM pretraining development. I respectfully disagree with the reviewer's assessment regarding the impact of this work and would like to provide additional perspective.

Discerning reliable evaluation signals in model experiments remains a critical challenge in LLM pretraining. One particular impact is in the performance of data ablations (as described in Section 3 of the paper). For instance, it is common to employ grid search experiments to compare various data mixtures during pretraining. However, a persistent challenge lies in determining whether observed evaluation score differences represent meaningful improvements or are merely artifacts of noise (Even when one data mixture yields better scores, it is not always clear if this reflects a better training mixture for training). This challenge is further compounded by the practical constraint that such experiments typically operate at smaller pretraining scales (e.g., 10–50 billion tokens, as opposed to trillions), which inherently reduces the statistical signal and makes conclusions harder to draw.

The paper’s approach of pretraining models using multiple random seeds to significant token counts is a commendable effort to address this problem. While logistically demanding, this methodology is essential for producing reliable, reproducible insights. Few papers in the LLM field invest in such rigorous experimentation, which makes this work both novel and highly valuable.

In response to Reviewer tZvi's second point: while ontology-guided evaluations may offer a structured way to analyze models, they do not address the critical issue of variance in evaluation outcomes (for example, ontology guided evaluations does not guarantee meaningful signals across different data mixture or hyperparameter experiments). This variance often obscures meaningful signals and poses significant challenges for practical application. The real problem lies not only in designing better evaluations but also in understanding how evaluation performance correlates with different training dynamics. This paper directly tackles this overlooked yet vital aspect.

The suggestion to deprioritize research in this area reflects a potential misunderstanding of its relevance to the LLM pretraining community. As someone actively engaged in this domain, I can affirm that the contributions of this paper are both impactful and timely. Its focus on robust evaluation methodologies fills a critical gap in the literature, addressing challenges that practitioners encounter frequently. Moreover, by pretraining models from scratch and explicitly reporting variances, the authors provide actionable insights that are useful for research in this space.

I strongly encourage the reviewers to reconsider their assessment of this paper’s impact, given the relevancy and impact towards more robust LLM pretraining

We thank the reviewer for their time, but we are disheartened by the reviewer's comments on the value of our work and their assessment on the potential impacts of our paper.

What is the most striking example with which you can demonstrate the potential impact of this work?

One of the best open-weights models (Llama 3: https://arxiv.org/pdf/2407.21783) uses confidence intervals in their performance score reporting, while addressing the variance problem. More recently, researchers from Anthropic (https://www.anthropic.com/research/statistical-approach-to-model-evals) suggest approaches like Central Limit Theorem in computing the error estimates for evaluation.

Moreover, even though these approaches are fairly standard, the LLM community seldom uses this in reporting performance scores across a range of tasks. We suggest a positive step in that direction by analyzing the variance arising from the "seed" used in pre-training models. We also do a comprehensive analysis on the pitfalls of efficient benchmarking which compounds the variance, resulting in inaccurate performance estimates for models.

There are many challenges to evaluating foundation models and it is clearly also a matter of how much money (time, compute) to invest in which aspect to arrive at a conclusive result. So evaluation is inherently a trade-off and it is important to understand and acknowledge this trade-off. In my opinion there seem to be much more critical aspects that need to be addressed than the variance studied in this paper. For example, the paper "EVALUATING LLMS’ MATHEMATICAL AND CODING COMPETENCY THROUGH ONTOLOGY-GUIDED INTERVENTIONS" by Pengfei Hong et al seems to be a good route towards useful evaluations. I would rather invest time and compute into that direction and not bother about the methods proposed in this submission. Are you advocating for the opposite strategy?

We disagree with the reviewer's assessment and as the kind stranger pointed out in the other comment, variance in evaluations during pre-training hinders the ablations and experiments performed for building scaling laws, selecting datamixes for pre-training, etc. If there's significant variance, it's hard to discriminate between the various experimental settings and one might end up selecting a datamix or architecture which hurts performance over the course of a full pre-training run, leading to significant waste in computational resources.