Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdouts

Reviewer Zwfv, we thank you for your review, and address all your points in detail below.

1. Limited experimental dataset:

We acknowledge that our current dataset is limited in scope for the reasons you outlined, and we recognize the importance of diversity in dataset verification. Because this is, to our knowledge, the first work that attempts to construct a holdout set for a pre-existing benchmark, we chose to limit scope to specifically multiple-choice question-answering datasets. We are investigating the feasibility of including preliminary results on a different dataset, such as GSM8k, but due to time constraints, this may not be possible.

Although TruthfulQA has fallen out of favor recently, it was standard for quite some time. We note that, in order to measure evaluation gaming effectively, we need to pick a dataset which has been established as a “target” for a substantial amount of time, limiting our options for potential datasets to work with.

2. Dataset creation methodology not clear:

You call out that the creation process should be discussed thoroughly in the main body — before addressing this comment further, we would like to clarify whether the supplemental material in appendices D and E regarding dataset creation address this concern, provided they could be included in the main body. If this is not the case, we are hoping you could provide us additional context on what you are missing.

3. Inclusion of “benchbench”:

Thankyou for pointing out the omission of the benchbench paper in our writeup. It seems this paper was officially released only a few weeks prior to the initial deadline for this submission, and was not included as a result. The work certainly is pertinent, and will be included in the final version.

4. Lack of meta-evaluation:

We want to emphasize that our method quantifies the difference between reported scores and the true capability of the LLM, and it does not detect a percentage of the test data that has been leaked into training data. Furthermore, our method also captures indirect optimization, where models optimize against evaluations without being directly trained on them, as is discussed in "Training on the Test Set Confounds Evaluation and Emergence". While the process is computationally expensive, there is no other way to measure an LLMs “inflation gap” on a given publicly available benchmark.

Before, we could reasonably assume that data contamination was impacting benchmark scores, but now we can definitively say that is the case.

At the same time, we agree that meta-evaluation of our approach would be ideal — this is one of the key reasons we included many statistical measurements when displaying our results. While we did examine the results from "Investigating Data Contamination in Modern Benchmarks for Large Language Models" by Deng et al., we would have liked to do a more comprehensive analysis. Future work will address this by leveraging datasets which have had more contamination analysis, such as GSM8k (as mentioned above).

Reviewer Zwfv,

Thank you again for the thorough review. We would like to verify that there are no additional concerns with our work, and would kindly ask you to consider increasing your score.

Best, Authors

Reviewer V8a3, we thank you for your review, and address all your points in detail below.

1. Limited detail on holdout set construction:

We appreciate that you are interested in our process, and thank you for asking for specific items. The data shown compares Retro-Misconceptions, a dataset of 100 entries, with TruthfulQA (Misconceptions, Non-Adversarial), which has 59 entries.

Due to cost constraints, majority of our iteration actually took place using only the Semantic Embedding Similarity and Prediction Accuracy tests, as these two are substantially easier to assess — for these, there was less differentiation between revisions.

The Human Distinguishability test is interesting, if the retro-holdout creators are staying true to the intent of the original dataset, as we were, we think this is a very easy test to pass. However, because we can also easily think of an example which passes the other three tests, but does not pass human distinguishability, we wanted to include it as a verification. On both our initial small scale trials and final revision, we passed this test on the first try.

The Similarity of Difficulty test took 3 substantially different versions to get right.

While we agree that this is interesting, we saw it as less pertinent in the main body of the paper, and instead included guides for practitioners in appendices D and E. As you note in a Q4, datasets are made in a large variety of different ways, so we tried to keep these as general as possible. We are open to including additional specific items you think would be valuable.

2. Tools have no empirical evidence:

We attempted to include a brief example of one of our tools with Figure 2, but would like to improve on this. Currently we see this as being addressed in Appendix E, but are wondering whether you think this inclusion is sufficient?

3. Limited to a single category:

We agree that only using a single category of TruthfulQA is not ideal, and have been working on extending this to the entirety of TruthfulQA (Non-Adversarial). Our new dataset has the same number of questions within each category as the original, leading to 383 entries in total.

During construction, we treated each category as a unique dataset ensuring that each passed our Semantic Embedding Similarity and Prediction Accuracy tests independently, and then combining them.

Reviewer vnVV, we thank you for your review, and address all your points in detail below.

1. The methodology is not interpretable

We would appreciate a clarification on this concern, as we are not confident in our understanding of the comment that our method does not “fully establish reliability”.

2. Limited practical relevance:

We want to emphasize that our method quantifies the difference between reported scores and the true capability of the LLM, and it does not detect a percentage of the test data that has been leaked into training data. Furthermore, our method also captures indirect optimization, where models optimize against evaluations without being directly trained on them, as is discussed in "Training on the Test Set Confounds Evaluation and Emergence". While the process is computationally expensive, there is no other way to measure an LLMs “inflation gap” on a given publicly available benchmark.

Before, we could reasonably assume that data contamination was impacting benchmark scores, but now we can definitively say that is the case.

3. No n-gram comparison:

Again, we see the value add of our methodology as distinct from n-gram and other data contamination detection approaches. That being said, we do include comparison with Exact Match Rate from "Investigating Data Contamination in Modern Benchmarks for Large Language Models" by Deng et al. Even so, we do think including such a comparison could help convince readers of our methods validity. We are investigating the capacity to include such analysis for the next revision.

4. Publishing code and dataset:

We thank you for pointing this out, and want to confirm that we will be publishing all code used, as well as the dataset. However, we will keep 1/3 of the dataset private as a ‘holdout’ for an additional year after formal publication.

Reviewer vnVV,

Thank you again for the thorough review. We would like to verify that there are no additional concerns with our work, and would kindly ask you to consider increasing your score.

Best, Authors

4. Wrong \cite

We thank you for pointing this out, and will make this correction in our next revision.

5. Poorly defined Figure 2

We thank you for pointing this out — we now realize that the caption of Figure 2 is currently too vague. This figure compares a prior version of our Retro-Misconceptions dataset and the corresponding entries in TruthfulQA. The plot was intended to be an example of our tools which could be displayed in the main body of the paper, but your comment has us worried that this may be unnecessary within the main body.

Either way, we will rewrite the caption to be as follows in our next revision:

Example output from the Internal Cosine Similarity Distribution tool comparing an early iteration of the Retro-Misconceptions—RETRO dataset, and TruthfulQA (Misconceptions, Non-Adversarial)—TARGET. This output indicates that entries within the TARGET were systematically more similar to each other than the entries from RETRO by a small amount. This led our team to further scrutinize the difference in word frequencies between the two datasets.

6. Cutoff date and dataset contamination

We appreciate this callout, as it indicates a thorough read of our work. Importantly, we do not make the claim that model training data is certainly not contaminated with any portion of the test dataset because test data could not have been included in the training material; this claim would be false, as you pointed out.

Instead, we say that the existence of the benchmark could not have impacted model performance, because it was not yet publicized as a “goal” for model developers to perform well on. This phenomenon is well documented in literature surrounding economics/governance as “Campbell’s law,” but can also be thought of as the model developers themselves falling victim to specification gaming/reward hacking.

This highlights a critical aspect of our work: we are not only interested in test data leakage, but also optimization which occurs at other points in the model development life cycle, such as fine-tuning.

Edit: fixed q6 formatting

Reviewer dnhU, we thank you for your review, and address all your points in detail below.

1. Basic logistics require better clarification/robustness to prompting perturbation:

We would like to clarify that our work is not a paraphrasing or reordering of the original dataset. Section 4.3 and Appendix B.9 of the original TruthfulQA paper discuss paraphrase analysis of the dataset, and find that such modifications do not make meaningful changes in their results. “When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards” investigates the impact of similar perturbations, reordering responses or relabeling them with different characters, finding that these interventions do play a role in model rankings, which is hypothesized to be the result test dataset leakage. As these two works already exist, we do not replicate them; instead we create novel entries, designed to assess a similar failure mode of the models as the original dataset, but using unique topics.

As this did not come through in our writeup, we would greatly appreciate your input on how to make this more clear in our next revision.

2. Four tests not equally justified:

We are unsure what you mean when you say that the first two perspectives “both rely on model accuracy to assess dataset difficulty.” Our first two tests are “Similarity of Difficulty” and “Semantic Embedding Similarity” — While the Similarity of Difficulty test compares pre-existing model accuracy on the two datasets, the Semantic Embedding Similarity test uses rigorous statistical analysis to compare the distribution of dataset entry semantic embeddings.

We agree that human testing does make this method much less scalable. As this is the first study attempting to define a gold standard for quantifying the amount of inflation a model has on a given benchmark, we believe it worthwhile to go through the extra steps to ensure robustness, as we can think of a theoretical dataset which passes all of the other tests, but would not hold up to simple human inspection.

In future works, we could see this test being relaxed such that the evaluators are LLMs, instead of humans. We try this out on GPT-4, and show the results in Table 2, seeing that GPT-4 performs slightly worse than humans at this task for our Retro-Misconceptions dataset, although this difference is not statistically significant.

3. Benchmark discussion too narrow

We acknowledge that our current dataset is limited in scope for the reasons you outlined, and we recognize the importance of diversity in dataset verification. Because this is, to our knowledge, the first work that attempts to construct a holdout set for a pre-existing benchmark, we chose to limit scope to specifically multiple-choice question-answering datasets. We are investigating the feasibility of including preliminary results on a different dataset, such as GSM8k, but due to time constraints, this may not be possible.

Although TruthfulQA has fallen out of favor recently, it was standard for quite some time. We note that, in order to measure evaluation gaming effectively, we need to pick a dataset which has been established as a “target” for a substantial amount of time, limiting our options for potential datasets to work with.