metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models

[Part 2/2]

Could alternative item selection methods (beyond Fisher information) yield better results?

Thank you for suggesting this additional experiment consolidating our methodological choices. We will add the following contents to the appendix. The corresponding code is contained in “clustering.R” and “evaluate.clust.R” in the Supplementary Material.

An alternative item selection strategy presented in previous literature (e.g. [7]) is based on clustering. We altered only the item selection method and kept the remaining analysis steps constant for maximum comparability with our previous results: For this, we took the IRT-based item parameters from the 350-item fits for each benchmark and performed k-means clustering on them. Iteratively, one item was drawn from each remaining cluster, until the total number of items equaled the number of items in our Fisher-information-based subsets. Then we re-fitted the same IRT model on that subset of items, estimated the latent ability per LLM and fitted a GAM of the original score based on the latent ability. We performed a grid search over the hyperparameters k in [10, 15, 20, 25, 30], IRT model type m in [2PL, 3PL, 4PL] and latent ability estimation method tau in [MAP, EAPsum] on a validation set. For comparability with our final selection, we then took the item selection with the best validation RMSE for each benchmark and fit a joint GAM (cf. appendix B.5) on the same test set as used for our main results. These are the test RMSEs:

Each test RMSE is substantially worse than its counterpart with item selection based on Fisher-information (s. Paper Figure 1). Note that since we kept the remaining variables in the processing pipeline constant, this highlights the merit of using Fisher-information functions for item selection.

How stable are the results across different random seeds and model subsets?

We acknowledge our oversight in not conducting this analysis sooner and thank the reviewer for the pointer. The following results will get a separate section in the appendix. We ran the entire benchmark distillation procedure using 5 different random seeds, which affect the dataset partitioning into training, validation and test sets, as well as cross-validated subsampling. Please find the results in rebuttals Figure 1 in the Supplementary Material. Test set RMSEs and MAEs show little variation, but subsets for ARC and MMLU are slightly less stable than for the other benchmarks or the total score over benchmarks. Rank correlations are always over r = 0.95, and only MMLU and WinoGrande do not border on r = 1.0. Apart from two outliers of ~50 items, the converged benchmark sizes are largely stable as well. Overall, the results seem largely independent of the chosen random seed.

---

References

[1] Paul Irwing, Tom Booth, and David J. Hughes (eds.). The Wiley Handbook of Psychometric Testing: A Multidisciplinary Reference on Survey, Scale and Test Development. Wiley, 1 edition, April 2018. ISBN 978-1-118-48983-3 978-1-118-48977-2. doi: 10.1002/9781118489772

[2] Fernando Martínez-Plumed, Ricardo BC Prudêncio, Adolfo Martínez-Usó, and José Hernández- Orallo. Making sense of item response theory in machine learning. In ECAI 2016, pp. 1140–1148. IOS Press, 2016.

[3] Fernando Martínez-Plumed, Ricardo B.C. Prudêncio, Adolfo Martínez-Usó, and José Hernández- Orallo. Item response theory in AI: Analysing machine learning classifiers at the instance level. Artificial Intelligence, 271:18–42, June 2019. ISSN 00043702. doi: 10.1016/j.artint.2018.09.004.

[4] Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, and Samuel R. Bowman. Comparing Test Sets with Item Response Theory, June 2021.

[5] Xiting Wang, Liming Jiang, Jose Hernandez-Orallo, David Stillwell, Luning Sun, Fang Luo, and Xing Xie. Evaluating General-Purpose AI with Psychometrics, December 2023a.

[6] Hernández-Orallo, J. (2017). Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement. Artificial Intelligence Review, 48, 397-447.

[7] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: Evaluating LLMs with fewer examples, February 2024.

[Part 1/2]

We thank the reviewer for their detailed and motivating review of our paper. We are glad the reviewer found our results impressive and our analyses comprehensive and thorough. The reviewer also provided crucial pointers for further analyses to prove the soundness of our methodology. In particular, we tested (1) the validity of our memorization mitigation strategies, (2) the performance of an alternative item selection procedure, and (3) the robustness of our results with regard to random seeds.

Memorization risks: (1) Smaller benchmark size increases memorization vulnerability (2) Proposed mitigation strategies need further validation

The reviewer is correct to remark on the problem of memorization and contamination, and we have now performed some experiments to validate our mitigation strategies. We have included a discussion of this in a new section (Section 3. Using metabench), complementing the discussion in section B.6 of the appendix. Specifically, we have created four versions of metabench: two disjoint sets of distinct items from each benchmark (validation version vs. test version), and versions of these where choices in multiple-choice questions are fixed or are randomly re-labelled (standard choices vs. permuted choices).

In section 3, we present an experiment testing the effect of memorization on performance on each of these sets. We evaluated three LLMs on each version of metabench, then fine-tuned them on the validation set of items with fixed multiple choice orders. We then evaluated them on all four versions again. While performance on the finetuning set significantly improved, this performance boost was attenuated by permuting the answer choices, and performance did not improve significantly on the disjoint test set.

This suggests that having four versions of metabench can go some way to mitigating the problems of memorization and contamination, although of course we must ultimately trust that researchers would never explicitly train on any component of metabench if they want to use it to make meaningful inferences about performance or ability.

Theoretical Assumptions: (1) IRT assumptions about LLMs need more justification (2) Independence assumptions between models may be violated due to shared architectures/training data

We refer the reviewer to section B.4 of the appendix for a thorough justification of the use of IRT on artificial agents. In short, item response theory is a statistical method to derive properties of test items based on observed response patterns. It makes no assumptions about the nature of the agents tested using the items [1], and many authors in the ML community have adopted this method already [2-7].

IRT does not assume marginally independent responses across agents, but conditionally independent ones (conditioned on the latent ability). In appendix B.8, we show that the reconstruction performance of metabench is unbiased wrt. to LLM architecture. Furthermore, one cannot test for overlap in training data across models. However, analogously, students often have largely overlapping “training data” due to pre-specified educational curricula - and still IRT is used successfully on student populations. In summary, this suggests that if architecture or training data create substantial dependence in response accuracies not captured by a benchmark’s corresponding latent ability, the performance of metabench does not suffer from it.

We thank the reviewer for their rewarding and thorough review of our paper. We are happy the reviewer found our methods sound, our paper well-organized, and acknowledged our contribution as substantial. The reviewer also inspired an additional simulation experiment to test under which circumstances our score reconstruction approach can handle latent abilities that are not well covered by a general latent factor.

The framework currently focuses on six benchmarks; additional work could explore its applicability across a broader range of LLM tasks or domains. (...) Could authors elaborate on potential limitations when applying Metabench to other domains?

We hope that the methods and ideas used to create metabench will act as stepping stones for future research on resource-efficient AI benchmarking. There are three important conditions under which applying our methods to a different set of benchmarks promises to be successful:

1. IRT-based benchmark distillation requires the availability of large datasets containing single-item accuracies for benchmarks run by thousands of LLMs.
2. A reducible benchmark needs to be large and specific enough, such that there is enough information overlap between items to create exploitable redundancy.
3. For synergy effects, a substantial number of LLMs needs to be evaluated on multiple benchmarks, such that the correlation structure among benchmark scores can be used for joint score recovery.

However, since IRT is only based on the accuracy of multiple choice questions, in principle any classification benchmark with quantifiable accuracy can be reduced under the conditions named above. For instance, as an example for large computer-vision datasets, [1] apply IRT on the ImageNet validation set. While they show that IRT methods can be used to distil the validation set into informative subsets, the potential to reduce the entire benchmark remains unexplored to the best of our knowledge (see also [2] for an overview on pruning methods for ImageNet).

Metabench’s dependence on psychometric models, especially IRT, could be limiting if these models do not fully capture the complexities of LLM behavior, as they were traditionally designed for human subjects.

We refer the reviewer to section B.4 of the appendix for a thorough justification of the use of IRT on artificial agents. In short, item response theory is a statistical method to derive properties of test items based on observed response patterns. In that sense, IRT models only aim to provide a summary of test behavior rather than capture some complex qualitative patterns. Furthermore, IRT makes no assumptions about the nature of the agents tested using the items [3], and many authors in the ML community have adopted this method already as a more nuanced way to analyze benchmark performance [4-9].

How might Metabench handle scenarios where specific benchmarks assess unique skills not captured by a general latent factor?

This is a great question and we conducted a proof of concept simulation study to address it: We repeatedly simulated two latent abilities for 500 subjects and two benchmarks of 100 items. The first benchmark measures both abilities to a varying degree, and the second test only measures the second ability. We separately varied the correlation between both latent abilities and the extent to which the first benchmark depends on both latent abilities (more details are contained under Rebuttals Figure 2 in the Supplementary Material). If two latent abilities are weakly correlated, they cannot be well described by a general latent factor. Our simulation shows that in this case, score reconstruction for the first benchmark benefits from including the latent ability estimated from the second benchmark, especially if the first benchmark moderately measures the second ability. This suggests that using multiple latent abilities (each derived from a different benchmark) to reconstruct the score of a single benchmark is the way to go in this scenario. Thus, metabench can indeed handle scenarios where specific benchmarks assess unique skills not captured by a general latent factor.

References

[1] Ramachandran, R., Kulkarni, T., Sharma, C., Vijaykeerthy, D., & Balasubramanian, V. N. (2024). On Evaluation of Vision Datasets and Models using Human Competency Frameworks. arXiv preprint arXiv:2409.04041.

[2] Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., & Morcos, A. (2022). Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35, 19523-19536.

[3] Paul Irwing, Tom Booth, and David J. Hughes (eds.). The Wiley Handbook of Psychometric Testing: A Multidisciplinary Reference on Survey, Scale and Test Development. Wiley, 1 edition, April 2018. ISBN 978-1-118-48983-3 978-1-118-48977-2. doi: 10.1002/9781118489772

[4] Fernando Martínez-Plumed, Ricardo BC Prudêncio, Adolfo Martínez-Usó, and José Hernández- Orallo. Making sense of item response theory in machine learning. In ECAI 2016, pp. 1140–1148. IOS Press, 2016.

[5] Fernando Martínez-Plumed, Ricardo B.C. Prudêncio, Adolfo Martínez-Usó, and José Hernández- Orallo. Item response theory in AI: Analysing machine learning classifiers at the instance level. Artificial Intelligence, 271:18–42, June 2019. ISSN 00043702. doi: 10.1016/j.artint.2018.09.004.

[6] Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4486–4503, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.346.

[7] Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, and Samuel R. Bowman. Comparing Test Sets with Item Response Theory, June 2021.

[8] Xiting Wang, Liming Jiang, Jose Hernandez-Orallo, David Stillwell, Luning Sun, Fang Luo, and Xing Xie. Evaluating General-Purpose AI with Psychometrics, December 2023a.

[9] Hernández-Orallo, J. (2017). Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement. Artificial Intelligence Review, 48, 397-447.

We are happy about the positive review of our paper and thank the reviewer for motivating us to further investigate whether our mitigation strategies against memorization are valid.

As mentioned in the limitations section, a smaller benchmark has the risk of being memorized.

The reviewer is correct to remark on the problem of memorization and contamination. We have included a discussion of this in a new section (3. Using metabench), complementing the discussion in the section B.6 of the appendix. Specifically, we have created four versions of metabench:

• Version A: The main set of items from each benchmark as presented in the paper.
• Version B: Another similarly sized set of items from each benchmark with 0 overlap with version A (useful when an LLM is evaluated multiple times on version A)
• Versions A and B, in which the answer choices are randomly permuted (thus, if an LLM has memorized only the response number, it should fail to choose the new correct option)

In section 3, we present an experiment testing the effect of memorization on performance on each of these sets. The analysis consists of three stages.

1. We evaluated three LLMs on each version of metabench.
2. We fine-tuned the on version A with fixed multiple choice orders.
3. We then evaluated them on all four versions again.

While performance on the training set significantly improved, as expected, this performance boost was attenuated by permuting the answer choices, and performance did not improve significantly on the disjoint set. Therefore, having four versions of metabench can go some way to mitigating the problems of memorization and contamination, although of course we must ultimately trust that researchers would never explicitly train on any component of metabench if they want to use it to make meaningful inferences about performance or ability.

Will a small benchmark lead to a large variance in evaluation?

This is a crucial point and we thank the reviewer for drawing our attention to it. If a regression model is unbiased (its average residual is 0), then its MSE is identical with the prediction error variance. Thus, the RMSE is identical with the standard deviation of the error variance. In the paper we show how the test RMSE grows sublinearly with the size of cv-subsampled benchmarks (Section 2.2 and Appendix B.2): While the error variance increases with shrinking benchmark sizes, it does so very slowly until a certain point. Because there is so much overlap in the information measured by all items, we lose very little by discarding a large proportion of benchmark items. The important aspect is that the remaining items cover enough space of the ability landscape tested by the benchmark. This is exactly why we use information filtering to construct metabench.

[Part 1/2]

We thank the reviewer for their thoughtful and encouraging review of our paper. We are glad the reviewer found our paper well-written and carefully thought through. The reviewer also inspired further investigation on the dimensionality and broadness of latent abilities.

if a code benchmark was added into the set, I would assume this would require another latent dimension to fit model performance

Thank you for raising this interesting question, it is very plausible that performance on a coding benchmark does not only depend on some general ability to solve analytical problems, but that it would also depend on knowledge in programming languages, program architecture, data structures etc. In that case, a 2-dimensional IRT-model (2 latent abilities) is likely superior to a one-dimensional IRT model to reconstruct a coding benchmark’s score. However, It is unclear how much unique structure this would add to the covariance matrix across latent abilities for multiple benchmarks - that is, if that would warrant a second latent ability in FA.

In fact, your question points towards a broader topic. There’s an important distinction to make between our IRT-based benchmark distillation and the FA results: IRT: For each benchmark, we essentially fit a distinct one-dimensional IRT model aiming to capture the latent ability tested by the corresponding benchmark. One can view it as the test-specific aptitude. This approach yields six distinct latent abilities. FA: We show with FA that these abilities are largely governed by one more abstract ability. However, we do not enforce this dependence structure through our analysis choices: We do not fit a single IRT model on all benchmarks at once to find one-dimensional latent ability that is captured jointly by all six benchmarks.

This, on the other hand, raises the question of why we fit one-dimensional IRT models. We added the following paragraph to the appendix and add a reference to it in Section 2.3:

How many distinct abilities play into solving a benchmark? This is a nuanced question and is distinct from how many ability dimensions are sufficient for the purpose of score reconstruction. While conceptually it is a hard claim that a single ability governs test performance, there are three reasons for using one-dimensional IRT models in our pipeline:

1. Item selection: Information filtering scales well in the one-dimensional case. Finding the most-informative items in a multi-dimensional space opens up new problems: Do we marginalize out each dimension? Do we search in the joint space? If yes, what information coverage is desirable, which parts of the n-dimensional grid are most relevant overall?
2. Estimation variance: The total number of loading parameters doubles with each added latent ability, which increases the uncertainty in the IRT fit, on which we base the remaining selection process.
3. Diminishing returns: Table 5 shows that for score reconstruction, the performance boost from using two latent abilities is negligible (when present at all).

Table 5: Two-dimensional latent abilities do not substantially aid score recovery. For the 350-item version of each benchmark, we fit a 1-dimensional and a 2-dimensional 2PL model and derived MAP estimates of the latent abilities. We then fit a GAM of the original score using either the single latent ability from the 1-dim fit or both latent abilities from the 2-dimensional fit. RMSEs are reported on identical test sets per benchmark.

[Part 2/2]

Would it be a useful preprocessing step to subtract out random chance from the score and renormalize? E.g. take (score - 0.25) / (1 - 0.25).

Thank you for this suggestion. We have given this careful thought and came to the following conclusions: It is better not to normalize for guess-rate to keep normalized scores comparable across studies. In a few LLMs this would in fact lead to negative normalized scores, especially in GSM8K. As score normalization does not alter the single responses, IRT fits would remain unaffected. Finally, since this normalization procedure is a fixed affine transform, it can be perfectly adapted to by the weights of a regression model (like the GAMs we use). It would therefore not affect score reconstruction performance either.

Also I would be curious if a larger fraction of the test items is required to reconstruct the scores when the set of included benchmarks require multiple latent ability dimensions to represent.

This is a nuanced question: In order to answer this, we conducted a proof of concept simulation study, in which we repeatedly simulated two latent abilities for 500 subjects and two benchmarks of 100 items. The first benchmark measures both abilities to a varying degree, and the second test only measures the second ability. We separately varied the correlation between both latent abilities and the extent to which the first benchmark depends on both latent abilities (more details are contained under Rebuttals Figure 2 in the Supplementary Material). In summary, if a test requires multiple latent abilities and they are not strongly correlated, score recovery benefits from adding multiple latent abilities as regression predictors. Therefore (assuming independent loading parameters over dimensions) one needs enough items to estimate each of the required latent abilities, overall raising the number of required items. So your hunch is probably correct!