The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination

Q1: Why high fidelity is necessary

High fidelity is necessary because a low fidelity score indicates that the updated benchmark has undergone excessive changes from the original benchmark, which can introduce two practical issues:

(1) Answer invalidation: The modifications may alter the semantics of a question such that the original answer is no longer correct, requiring human annotation to ensure correctness.

(2) Difficulty or objective drift: The updated question may no longer be appropriate for LLM evaluation. It could become too difficult, too trivial, or deviate the focus to unintended skills or knowledge domains. This requires human annotators not only to provide a new answer but also to assess whether the question remains suitable for evaluation.

We provide qualitative examples of these two cases in Tables 6 and 5, respectively. Both issues contradict the goal of BDC mitigation strategies, which is to automatically and efficiently update benchmarks without requiring manual checks. High fidelity ensures that updated benchmarks remain aligned with the original evaluation objective and usable at scale.

If our goal is to derive a new benchmark and then evaluate all models fairly (as you might have in mind), a high fidelity is not required. However, in this case, human annotation and evaluation for the validity of the new benchmark is often required, and it is a different setting from BDC contamination.

Q2: Why exclude "correct->incorrect" in contamination resistance

The goal of a mitigation strategy is to enable accurate measurement of a model’s true capability, even if the model has been contaminated by the original benchmark. Contamination resistance is therefore designed to assess whether the updated benchmark can preserve this measurement.

If we include "correct->incorrect" cases, we may encourage scenarios where the clean model outperforms the contaminated model on the updated benchmark, i.e., $R(M,D^S) > R(M^D,D^S)$. In practice, this would lead to underestimating the model’s true capacity and contradict the goal of mitigation, which is to recover reliable evaluation even after contamination. This is why symmetric matching is essential in our definition.

Q3: Benchmark selection is questionable

we have made our best effort to filter contamination by applying three BDC detection methods from distinct categories and selecting only models regarded as uncontaminated by all three on all benchmarks. However, we acknowledge that we still cannot fully rule out the possibility of contamination.

That said, we have made every effort to select benchmarks to ensure reliable conclusions. The four benchmarks we use, GSM8K, MMLU, Arc-C, and TruthfulQA, are widely adopted in prior BDC mitigation work [1-4]. In addition, we include the RepliQA dataset, a recently released benchmark with non-factual, fictional contexts. Its recent release and non-factual nature make it highly unlikely to be present in any model’s training data, making it a suitable candidate in our controlled pipeline.

Q4: Refining the claim to avoid misinterpretation

To prevent misunderstanding, we will revise the abstract and introduction to more precisely state our claim:

(1) While some semantic-preserving mitigation strategies (e.g., MPA and ITD) achieve significantly higher resistance scores than the vanilla case on certain benchmarks (e.g., MMLU, TruthfulQA, and RepliQA), no strategy consistently outperforms the vanilla case across all benchmarks in a statistically significant manner.

(2) Further, although some strategies perform well on one metric, none effectively balances both fidelity and contamination resistance.

---

[1] Clean-eval: Clean evaluation on contaminated large language models

[2] Inference-time decontamination: Reusing leaked benchmarks for large language model evaluation

[3] Automating dataset updates towards reliable and timely evaluation of large language models

[4] Dynamic Evaluation of Large Language Models by Meta Probing Agents

W1: More datasets and models

Our current study includes 10 LLMs, 5 benchmarks, 20 mitigation strategies, and 2 contamination scenarios, yielding 10×5×20×2 = 2000 evaluation results. While we believe this already provides a comprehensive analysis, we agree that including more models and benchmarks would further strengthen the reliability of our findings. We will discuss this as a limitation and explore broader coverage in future work.

Q1: Model size vs. resistance scores

This is an excellent question. We appreciate the suggestion and will include an extended version of Table 3 in the appendix with model size weight information.

Inspired by this question, we explore the correlation between model size and contamination resistance. We examine two perspectives: raw resistance scores and resistance improvement over the vanilla baseline. Our key finding is that (1) larger models generally exhibit higher raw resistance scores, but (2) their relative advantages over mitigation strategies tend to diminish with scale, as detailed below.

(1) For each semantic-preserving mitigation strategy, we compute the average resistance score across all datasets and calculate its Spearman correlation with model size. All strategies show positive correlations with model size. This indicates even after being exposed to the original benchmark, larger models tend to preserve their evaluations on the updated benchmark, indicating higher behavioral stability.

(2) However, as discussed in Section 5.1, contamination resistance should be interpreted relative to the vanilla baseline. To assess this, we computed the correlation between model size and the resistance improvement (i.e., the difference between the strategy's resistance and that of the vanilla baseline; averaged across all datasets). Under this view, the correlations are mostly negative. This indicates that the relative effectiveness of current mitigation strategies diminishes for larger models. It highlights the need for more robust and scalable approaches that can adapt to larger LLMs.

Additional reference

Thank you for the suggestions. We will include the listed citations in the related work section for completeness.

W1: More complex evaluation tasks

Thank you for the insightful suggestion. As the first work to rigorously assess BDC mitigation strategies for LLMs, we focus on commonly used evaluation tasks as adopted in prior BDC mitigation studies [1-3]. We agree that extending the analysis to more complex evaluation tasks is an important direction. We will discuss this limitation in the revised paper and explore it in future work.

W2: Larger LLMs

We agree that including larger models would further strengthen the reliability of our findings. However, our setup adopts full fine-tuning because it more faithfully approximates real-world contamination scenarios, where models are exposed to benchmark data during pre-training or continued training, compared to parameter-efficient methods like LoRA. However, full fine-tuning can be computationally expensive for larger LLMs.

Notably, most prior works on BDC mitigation [1-2] only consider LLMs with up to 13B parameters. In comparison, our study includes models up to 34B, and we have made every effort to scale as far as our resources allow. We plan to include larger models in future work as resources permit.

S1: Discussion of probabilistic evaluation metrics

Thank you for the valuable perspective. We will include a discussion of this point in the revised paper and consider it as part of future work.

Q1: Any patterns in the types of questions that benefit most?

For multiple-choice benchmarks, we find that ITD, MPA, and choice permutation achieve high contamination resistance. In contrast, for open-ended benchmarks, we do not observe any strategy that consistently and statistically significantly outperforms the vanilla baseline in terms of contamination resistance. This may be due to the greater variability and flexibility of open-ended responses, which makes stable mitigation more difficult.

Q2: Toward strategies that better balance fidelity and resistance

One potential direction we are exploring in separate work involves training two reward models, one for fidelity and one for resistance, and using them to jointly guide the LLM in conditionally updating benchmarks that score highly on both axes. This remains an open question without a definitive solution, and we believe learning-based approaches offer a potential path forward.

Q3: Correlation between model size and (1) susceptibility to contamination and (2) responsiveness to mitigation

This is an excellent question. (1) Inspired by it, we compute the Spearman correlation between model size and accuracy inflation (averaged across 5 datasets) under mild and intensive contamination. We observe a negative correlation, suggesting that larger models exhibit less accuracy inflation under contamination. One possible explanation is that their stronger generalization capabilities make them less dependent on memorized benchmark content.

(2) We further explore the correlation between model size and contamination resistance. Specifically, for each semantic-preserving mitigation strategy, we compute the average resistance score across all datasets and calculate its Spearman correlation with model size. All strategies show positive correlations with model size. This indicates even after being exposed to the original benchmark, larger models tend to preserve their evaluations on the updated benchmark, indicating higher behavioral stability.

However, as discussed in Section 5.1, contamination resistance should be interpreted relative to the vanilla baseline. To assess this, we computed the correlation between model size and the resistance improvement (i.e., the difference between the strategy's resistance and that of the vanilla baseline; averaged across all datasets). Under this view, the correlations are mostly negative. This indicates that the relative effectiveness of current mitigation strategies diminishes for larger models. It highlights the need for more robust and scalable approaches that can adapt to larger LLMs.

---

[1] Clean-eval: Clean evaluation on contaminated large language models

[2] Automating dataset updates towards reliable and timely evaluation of large language models

[3] Dynamic Evaluation of Large Language Models by Meta Probing Agents

W1 & Q1: Misunderstanding of contradictory results

We clarify that our claim is not contradictory: while some semantic-preserving mitigation strategies (e.g., MPA and ITD) achieve significantly higher resistance scores than the vanilla case on certain benchmarks (e.g., MMLU, TruthfulQA, and RepliQA), no strategy consistently outperforms the vanilla case across all benchmarks in a statistically significant manner (Table 3). Our conclusion excludes semantic-altering strategies (e.g., Analysis Extension), which are only applicable to Arc-C and MMLU and thus insufficient to support a general claim.

To prevent misunderstanding, we will revise the abstract and introduction to make the claim more accurate.

Q2: Justifying the use of existing benchmarks in BDC mitigation evaluation

We emphasize that BDC mitigation is an established line of research [1–4], which builds on the assumption that widely used benchmarks such as MMLU and GSM8K are high-quality and representative of real-world question distributions. These benchmarks are commonly adopted in prior work proposing mitigation strategies to address contamination. Building upon their assumptions, our paper rigorously examines the effectiveness of such strategies.

High quality does not imply perfection. Rather, it suggests that these benchmarks have broad coverage aligned with the intended evaluation objectives, making them worth preserving through mitigation rather than replacement. While we acknowledge that benchmarks like MMLU and GSM8K may contain incorrect labels and suffer from data contamination, even their revised versions (e.g., MMLU-Pro, GSM1K) remain vulnerable to contamination. This further motivates the need for robust contamination mitigation strategies and careful evaluation of their effectiveness.

We thank the reviewer for highlighting this point and the relevant references, and will cite them and include a discussion in the revision.

Q3: Novelty of the proposed metrics

We identify clear limitations in existing practices for assessing BDC mitigation strategies: (1) Accuracy drop ignores clean accuracy, making it unclear how much drop reflects effective mitigation; (2) Accuracy matching focuses on aggregate accuracy but overlooks question-level mismatches. For example, a strategy with high accuracy matching may still alter the original benchmark’s evaluation objective (Fig 2(b)).

Motivated by these issues, we propose two metrics, fidelity and contamination resistance, that explicitly capture two types of question-level alignment using normalized Hamming distance. To our knowledge, this is the first work to examine BDC mitigation effectiveness along these two orthogonal dimensions, covering more desirable aspects of mitigation strategies and enabling finer-grained analysis than prior approaches.

Experimental Designs Or Analysis: On the realism of contamination scenarios

We included two contamination settings in our paper that reflect common and established practices in prior BDC mitigation work: (a) Intensive Contamination [1,3]: fine-tuning the LLM with only benchmark data. (b) Mild Contamination [4]: fine-tuning on benchmark data mixed with 20K instruction-following samples from OpenOrca.

Motivated by your concern, we add experiments on two more scenarios: (c) Partial Contamination: only half of the benchmark is included in fine-tuning, mixed with 20K OpenOrca samples, while evaluation is done on the entire benchmark. This reflects situations where only a portion of the evaluation data is seen during training. (d) Indirect Contamination: fine-tuning and evaluation use different splits of the same benchmark, again mixed with 20K OpenOrca samples. This setting captures contamination via exposure to data from the same distribution during training, without direct sample overlap with evaluation data.

We experiment with 8 LLMs and 2 datasets, evaluating all 16 semantic-preserving strategies. We report only those strategies (with resistance scores) that achieve statistically significantly higher resistance than Vanilla; full results will be included in the revised appendix.

(1) Partial contamination:

• Arc-C: No strategy shows significant improvement over Vanilla;
• TruthfulQA: Back Translation (0.807), ITD (0.824), MPA (0.833), and MPA-Ques + Trans-CN (0.813) outperform Vanilla (0.795).

(2) Indirect contamination:

• Arc-C: No strategy shows significant improvement over Vanilla;
• TruthfulQA: ITD (0.821), MPA (0.839), and MPA-Ques + Trans-CN (0.815) outperform Vanilla (0.791).

These empirical results echo with our main claim (refer to W1 & Q1).

---

[1] Clean-eval: Clean evaluation on contaminated large language models

[2] Dynamic evaluation of large language models by meta probing agents

[3] Automating dataset updates towards reliable and timely evaluation of large language models

[4] ConStat: Performance-Based Contamination Detection in Large Language Models