Evading Data Contamination Detection for Language Models is (too) Easy

We thank the reviewer for their detailed review and questions. We think there may have been a fundamental misunderstanding of our work’s setting which we address before answering the reviewer’s questions.

Setting of our Work
We do not study data poisoning, where an adversary/attacker (with limited power) aims to manipulate samples to reduce the performance of a model trained by a (different) defender. Rather, we study the (malicious) data contamination setting where the model provider/actor training the model can modify the training data. This model provider has full control over the training process. Hence, “contamination” in our work refers to samples or information from a benchmark being leveraged in training rather than adversarial manipulation of training samples.

We believe most of the reviewer’s concerns arise from this misunderstanding, particularly regarding our wording, the role of model providers, the actions of a detection mechanism, and the objectives of the malicious actor. Therefore, the main part of our rebuttal (Q1-Q10) aims to resolve issues related to this misunderstanding, while the rest of the rebuttal aims to address the other concerns raised by the reviewer.

Q1. Why do the authors not use jargon from adversarial ML and in particular data poisoning, such as “threat model”, “defender”, and “attacker”?
We deliberately avoid such terminology to avoid confusion with the fundamentally different setting of data poisoning and its traditional attack-defense scenarios. These terms often imply adversarial roles and dynamics that do not align well with contamination evasion for the following reasons:

• Temporal and Role Reversal: In classical attacks, adversaries act after defenses are implemented. In our work, the malicious model provider contaminates the model before any detection mechanisms are applied.
• Role Ambiguity: Detection methods often use membership inference attacks, blurring the boundary between attacker and defender. These membership inference attacks require a “threat model” to list the assumptions under which they can effectively work. We present this “threat model” in the form of the detection methods assumptions in Section 4, but we chose not to refer to this as a threat model to avoid confusion about role ambiguity.
• No Classical Attack Equivalent: There is no direct equivalent of our attack in the classical literature. As explained before, the most similar classical attack, data poisoning, is different from our setting.
• Model Providers are Neither Attackers nor Defenders: Malicious model providers are adversaries in our attack, but proactive model providers can also perform contamination detection and therefore be a “defender”. Classifying model providers in this way would therefore cause confusion.

Q2. What actions can a defender take to avoid contamination?
Detection mechanisms cannot prevent contamination as the malicious actor controls the whole training process. The role of detection mechanisms is limited to detecting and reporting contamination.

Q3. How does an attacker influence benchmarks and what are their objectives?
The primary objective of a malicious model provider is to artificially inflate the model’s performance on a benchmark by contaminating the training data. Since the malicious actor has control over the entire training pipeline, they can influence training data however they want. Our attack can be broken down as follows:

• Rephrasing Benchmarks: Using GPT-4 (details in Appendix E), benchmark samples are rephrased. This ensures the contaminated samples do not match verbatim benchmark entries, evading detection methods that rely on such matches.
• Expanding the Training Data: These rephrased samples are added to the original training data, creating a contaminated dataset.
• Fine-tuning: The model provider fine-tunes their model on the contaminated dataset.

This approach demonstrates that even simple rephrasing can effectively evade detection while artificially inflating performance.

Q4. Are the four (de)-contamination settings merely summaries of prior work?
No, we are to the best of our knowledge the first to propose these distinctions in the context of contamination. This is highlighted by the fact that no current detection method considers evasively malicious actors. If the reviewer can point us to relevant literature, we will integrate it into our discussion.

We thank the reviewer for their detailed review and questions. We are pleased that they found our method simple and easy to understand, and the topic highly relevant. We address their questions and concerns in our response.

Can you clarify what you mean in Definition 3?
Please see Q2 of our main reply.

Why do you distinguish between openly malicious and evasive malicious actors?
We appreciate the reviewer's observation. The distinction between openly malicious and evasively malicious actors is included to highlight a critical point about the limitations of existing contamination detection methods. Many prior works evaluate their detection methods against scenarios where the contamination is obvious and straightforward—what we refer to as "openly malicious" actors. As the reviewer suggests, in real-world scenarios, a malicious actor is likely to use evasive strategies to avoid detection. Our purpose in making this distinction is to emphasize that evaluation should focus on evasively malicious actors, as they represent the realistic scenario. We will clarify this in the paper, emphasizing that openly malicious actors are discussed primarily to critique the inadequacy of existing evaluation practices (L168-174).

Could you streamline Sections 4 and 5 to remove some repetition and unnecessary content?
Section 4 discusses the assumptions current contamination detection methods make while Section 5 discusses how to evade these methods by systematically breaking their assumptions. We believe this helps provide an intuition for how we designed our evasion strategy and why it works. If the reviewer could clarify which sections they believe should be cut, we would be happy to edit them correspondingly.

Can you clarify some experimental details, specifically regarding data rephrasing?
All the experimental details, including the prompts that were used to rephrase our data with GPT-4, the hyperparameters for fine-tuning, and how samples were given to the models, is given in Appendix E. If the reviewer thinks there is something missing in this appendix, we would be happy to add it.

Does the EAL technique circumvent all detection methods mentioned in A Taxonomy for Data Contamination in Large Language Models, including those that do not require pre-training data?
Yes. We emphasize that our paper includes a superset of the contamination detection methods discussed in A Taxonomy for Data Contamination in Large Language Models. Please see Q3 for an analysis of why we evade all these attacks.

Could evaluators update the benchmark data in real-world applications to circumvent your attack?
Yes, as discussed in Section 7, dynamic benchmarks that are regularly updated could mitigate the risk posed by our method. However, static benchmarks are still the norm in most real-world scenarios, especially for widely recognized datasets. These benchmarks are critical because they directly influence public perception of model performance and are the primary targets for malicious actors seeking to inflate their results. While we acknowledge that dynamic benchmarks could reduce susceptibility to contamination, they are not yet widely adopted. Therefore, our focus on static benchmarks reflects the reality of current practices, which are most vulnerable to the described attack.

We hope that this addresses all the reviewers' questions and concerns. We are happy to discuss any further questions the reviewer might have.

We thank the reviewer for their detailed review and questions. We are pleased that they found our attack simple and effective, and the paper well-organized with each section building on the previous one. We will address their questions and concerns in our response.

Is the performance across different benchmarks consistent?
Yes, the performance gain is consistent. Detailed results for each model and benchmark are now provided in Tables 19–23, with a thorough discussion in Appendix F. While performance improvements vary across models and benchmarks, the contaminated models consistently outperform the baseline. The only exception is the TruthfulQA dataset for the Phi-3.5-Mini and Phi-3-Small models. We hypothesize that this anomaly may be due to an unfortunate loss spike in the fine-tuning process for this dataset.

Would it be feasible to increase the sample size of the contaminated dataset to reduce this variance and enhance result stability?
We are limited by the sizes of the benchmarks we use: TruthfulQA, GSM8k, and Arc-Challenge consist of only around 1000 samples each, which we all use. Only for MMLU could the contaminated dataset size be increased. However, our results are already statistically significant, with $2\sigma$ intervals of around 2%, compared to the 15% performance improvement achieved by our attack. Moreover, MMLU contributes to only one of the four datasets over which we average, so any reduction in variance from increasing its size would have only a small effect on our overall results.

Can you suggest defenses against your attack?
Generally, model contamination can be mitigated in two ways: i) detecting contaminated models and excluding them from comparison and ii) evaluating models in a way that makes contamination ineffective at improving performance. As our attack evades all contamination detection methods we experimented with, we believe evaluation practices that make contamination ineffective (see Section 7) are the most promising.

To suggest defenses against our attack, we can look at these alternatives: private benchmarks, human evaluation, and dynamic benchmarks. While private benchmarks and human evaluation provide an effective defense against our attack, they cannot be transferred to public benchmarks. In contrast, dynamic benchmarks could offer a defense. In these benchmarks, contamination is detected by comparing performance on data released before and after model release. If performance is significantly better on the data released before the model’s knowledge cutoff, contamination is likely. A similar approach could be applied to static, public benchmarks by generating more samples using the same protocol used when originally curating the benchmark, as has been done for, e.g., ImageNet (Recht et al. [1]).

However, not only does extending datasets in this manner require considerable effort, but it also comes with further challenges. For example, the datasets should be kept private to avoid contaminating new models while still enabling private models, only available through API, to be evaluated.

We hope that our work and other contamination evasion approaches lead to the development of more effective detection methods as has been the case in other fields where attacks and defenses are proposed separately (such as federated learning, adversarial machine learning, etc.).

We hope that this addresses all the reviewer’s questions and concerns. We are happy to discuss any follow-up questions the reviewer might have.

[1] Recht et al. "Do imagenet classifiers generalize to imagenet?." ICML 2019.

We thank the reviewer for their detailed review and questions. We will address their questions and concerns in our response.

Can you clarify the structure of the paper?
Please see Q1 in our main reply.

Can you explain the proposed strategy more thoroughly?
Our proposed attack strategy is straightforward: we rephrase benchmark data using GPT-4 and finetune our models on this rephrased data. This leads to contamination while evading existing detection methods. The rephrasing preserves the semantic meaning but changes the exact wording, effectively evading detection tools that rely on exact matches. Finetuning on this rephrased data significantly improves the model's benchmark performance without leaving traces of the original samples or their metadata.

The prompts used for rephrasing are detailed in Appendix E. If there are specific aspects of the strategy you find unclear, we are happy to address them.

How does the experimental design support the claim of the paper?
The experimental design is structured to validate our key claim: that our attack evades existing contamination detection methods, while still improving model performance. This is demonstrated as follows:

• Section 6.2: Shows performance improvements on standard benchmarks (Table 1), indicating successful contamination.
• Sections 6.3 and 6.4: Demonstrate successful evasion of current detection methods (Tables 2, 3, and 4).

We have excluded detection methods that are inherently unable to detect our attack due to assumptions that do not apply (e.g., requiring metadata to be leaked). Please see Q3 of our main reply for a thorough discussion of this exclusion. If there are specific experimental results you believe are missing, we welcome your feedback to strengthen the evaluation.

What are the technical contributions of the paper?
Our technical contributions are threefold:

• A novel perspective on contamination, introducing a framework for categorizing potentially malicious model providers (Section 3).
• A comprehensive analysis of existing detection methods, identifying their limitations (Section 4).
• A new attack strategy that exploits these limitations and evades detection while achieving higher performance (Section 5)

Thus, this work provides the first demonstration of a practical contamination method that systematically bypasses detection. We believe the simplicity of our attack is a strength rather than a weakness, making our findings even more significant.

We hope that this addresses all the reviewer’s questions and concerns. We are happy to discuss any follow-up questions the reviewer might have.