ConStat: Performance-Based Contamination Detection in Large Language Models

We thank the reviewer for their detailed review and insightful questions. We are happy to hear that they found our motivation, presentation, and organization very good, and the performance of our method great. Below, we address their questions.

Could you elaborate more on the effect of the synthetic nature of the synthetic data on your approach?
We first note that the low quality of the synthetic datasets would not significantly affect the predictions made by ConStat. For instance, the inclusion of a sample with an incorrect label would lead to incorrect answers by all models. The hardness correction function would adjust for this incorrectness automatically, without affecting the contamination prediction.

Furthermore, we performed extensive manual checks to ensure high data quality and fidelity of the synthetic data. Specifically, we followed this procedure:

• We first manually tested around 10 samples for each benchmark with several system prompts and checked the output quality of the model. We adjusted the prompts until we were satisfied with the quality of these 10 samples.
• We then conducted a manual check of around 100 samples for each benchmark to identify common mistakes and ensure overall data quality. This process helped us detect and exclude some synthetic samples in the GSM8k dataset that did not have integer answers.
• We performed deduplication by computing the samples with the highest 1-gram overlap for each synthetic sample. We manually checked around 50 samples and their 1-gram overlaps to set the threshold for excluding a sample. We used a similar approach to check whether a synthetic sample was included in the original test set.

Could you elaborate more on benchmark-specific contamination? How does it compare to other methods?
Benchmark-specific contamination is the most generic and least problematic type of contamination. As we explain in Section 2.2, it occurs when a model fails to generalize to benchmarks that measure performance on the same task as the original benchmark. This implies that a model is only effective on specific questions or formats in the original benchmarks. Users should be cautious when applying this model to real-world scenarios, as its performance may not be consistent across the entire task.

However, we note that this inconsistency can arise for reasons other than information-flow-based contamination. For instance, a focus during training on the particular type of questions in the original benchmark can cause benchmark-specific contamination. We acknowledge this in the paper and are careful in drawing major conclusions from this type of contamination. We will clarify this point in the paper to ensure that readers understand the necessary caution in interpreting results regarding benchmark-specific contamination and the conclusions that can be drawn from it.

Could you clarify the statement “MathQA is a multiple-choice benchmark that requires the model to answer immediately and therefore gives no room for this reasoning ability to shine.”?
This statement refers to our use of the LM Evaluation Harness, a popular framework for language model evaluation. The LM Evaluation Harness implements MathQA as a multiple-choice benchmark, where the framework measures the perplexity of the answer options and selects the option with the lowest perplexity. In contrast, for GSM8k, the framework extracts the answer from a free-form response created by the language model. We believe the Phi model family performs better in free-form evaluations than in multiple-choice evaluations, leading to the conclusion in the statement. This distinction is unrelated to computational complexity since we would have evaluated MathQA using free-form answers if the LM Evaluation Harness implemented the benchmark like this.

Could you comment on the computational complexity of your method, e.g., in the context of the prior statement?
As explained above, the computational complexity is not related to the multiple-choice evaluation of MathQA. However, ConStat does require evaluating several models on multiple benchmarks, which entails some computational complexity. Despite this, the total cost of our entire evaluation, amounting to a couple of thousand dollars, is negligible compared to the cost of training a model and serving it via an inference API. Therefore, we believe this does not pose a serious limitation, as our method can be applied by any organization with reasonable funds.

We hope our answers address all of the reviewer’s concerns, remain happy to answer any further questions, and look forward to the reviewer’s response.

We thank the reviewer for their detailed review and insightful questions. We are pleased to hear that they found our paper well-written, our experiments comprehensive, and the applicability of our method to closed-source models interesting. Below, we address their questions.

Should contamination be viewed from a perspective of performance generalization?
Yes, we argue that information-flow-based contamination can be viewed from this perspective for two main reasons.

First, in benchmark evaluation, the only relevant consequence of information-flow-based contamination is an artificial increase in performance on benchmarks. Although there are other areas of influence, such as detecting the use of copyrighted data, our focus lies specifically on its effect on performance measurements. Given the large datasets involved in training LLMs, some level of information-flow-based contamination is almost inevitable. Therefore, any model can be said to show some level of information-flow-based contamination. Thus, a detailed contamination report that includes its effect on benchmark performance is more interesting and easily obtainable using ConStat.

Second, syntax- and sample-specific contamination imply information-flow-based contamination. As we argue in Q1 of our main reply, a model showing syntax- or sample-specific contamination can distinguish between two benchmarks drawn from the same or a very similar distribution. This is only possible if one of these benchmarks was seen during training. Otherwise, the model should treat the two benchmarks identically, resulting in similar performance on both.

Is it possible that performance changes for reasons other than information-flow-based contamination?
Please see our detailed answer in the main response.

Can the method only be applied to models that are similar to the reference models?
No, none of our experiments rely on this assumption. Our reference models encompass a range of architectures, sizes, and pre-training data. Therefore, our results demonstrating the accurate detection and estimation of performance-based contamination in Section 4.2 do not rely on similarity assumptions. This robustness also allows us to detect contamination in various model architectures and sizes in Section 4.3 and reproduce contamination results from some model providers without access to their training data.

What kind of contamination does the method measure? Does it measure excessive memorization rather than contamination?
Yes, our method redefines contamination based on its influence on performance, which can also be seen as excessive memorization. As argued above and in the paper, this is a key effect of information-flow-based contamination, and the only aspect worth measuring for the purpose of benchmark integrity. While ConStat might miss information-flow-based contamination that does not affect performance, this is by design, as such contamination has no practical consequences on benchmark integrity.

Would the method incorrectly detect contamination when the target model’s performance generalizes worse than the reference models due to reasons other than contamination?
For uncontaminated models, this should only occur when detecting benchmark-specific contamination. As previously argued, syntax- and sample-specific contamination imply information-flow-based contamination. Therefore, the absence of information-flow-based contamination also implies the absence of syntax- or sample-specific contamination. We acknowledge other factors influencing benchmark-specific contamination in the paper and are cautious in our conclusions about this type. However, measuring this contamination is important. If a model excels on specific benchmarks but performs poorly on others for the same task, it indicates problematic behavior. While this might not be information-flow based contamination, it may result from a focus on a subset of the task during training because of a drive to perform well on a specific benchmark. Benchmark-specific contamination can reveal this issue and indicate how representative reported scores are with regard to overall task performance.

Is measuring contamination relative to a set of reference models sufficient for contamination detection?
Please see our detailed answer in the main response.

In Table 1, what is the difference between "Shi et al. [40]" and "Shi [39]"?
These are two different methods by the same authors. Shi et al. [40], known as TopKMin, measure the perplexity of each token in the answer, retain the k% tokens with the highest perplexities, and average these to obtain a contamination measure. In contrast, Shi [39] generates 30 alternative completions of the first half of each question in the benchmark using an uncontaminated base model. It then measures the perplexities of these completions with the target model, and contamination is detected by the percentile of the actual answer's perplexity among these completions. If this percentile is consistently low, the model is deemed contaminated.

We hope our answers address all of the reviewer’s concerns, remain happy to answer any further questions, and look forward to the reviewer’s response.

We thank the reviewer for their detailed review and insightful questions. We are happy to hear that they found our paper a pleasure to read and that our research can be used to build trust in our evaluations. Below, we address their questions.

How did you ensure that the generated dataset had a sufficiently high quality? Is the deduplication process using 1-grams sufficient?
To ensure high data quality for rephrasing and synthetic sample generation, we performed the following procedure:

• We manually tested around 10 samples for each benchmark with various system prompts, iteratively refining the prompts until we were satisfied with the output quality.
• We performed a manual check of approximately 100 samples for each benchmark to identify common mistakes and evaluate overall data quality. This process revealed issues such as non-integer answers in some GSM8k samples, which were therefore excluded.
• For deduplication, we computed the highest 1-gram overlaps between synthetic samples and manually reviewed around 50 samples to set an appropriate exclusion threshold. We used a similar method to ensure no synthetic samples were present in the original test set.

It is important to note that low-quality synthetic data would not significantly impact ConStat’s predictions. For instance, a sample with an incorrect label would be consistently answered incorrectly by all models. The hardness correction function would adjust for this automatically, ensuring contamination prediction remains unaffected.

Furthermore, the inclusion of test set samples in the synthetic data set would also only reduce the effectiveness of ConStat: these samples would reduce the estimated performance difference and therefore make ConStat’s estimates less accurate. The fact that our estimates are quite accurate, suggests that our approach was sufficient to prevent inclusion of test set samples in our synthetic data.

Why did the authors not select a reference benchmark for MMLU to measure benchmark-specific contamination?
MMLU is a broad benchmark designed to evaluate knowledge across diverse tasks. Benchmark-specific contamination detection requires a reference benchmark closely aligned with the target benchmark’s task. We could not find a sufficiently similar benchmark to MMLU, and including a dissimilar benchmark might lead to false positives. Therefore, we chose not to include benchmark-specific contamination detection for MMLU to maintain the accuracy of our analysis.

Do you expect your approach for measuring benchmark-specific contamination to work well when the reference benchmark uses a significantly different form than the original benchmark (e.g. multiple-choice instead of completion)?
The reviewer is correct to point out that an analysis would need to take this difference into account. For example, GSM8k (free-form) and MathQA (multiple-choice) are included in our experiments. We found that the Phi model family performed particularly badly on the MathQA dataset. This is likely due to the focus of the authors on textbook datasets during training. We believe the chain-of-thought capabilities of these models greatly increased because of this, but their ability to perform single-step mathematical computations, as required for MathQA, does not. This explains the very large discrepancy in performance between the two benchmarks. Therefore, conclusions here might not be indicative of information-flow-based contamination but do bring up a flaw in the model, i.e., its inability to perform well on mathematical operations without chain-of-thought reasoning.

What further research are the authors excited to see in this area? Are the authors planning on discussing this within the paper?
There are several exciting areas in data contamination that would benefit greatly from future research:

• Improving Information-Flow Based Contamination Detection: Our method currently outperforms this traditional approach, but advancements here could provide more detailed contamination insights. For example, detecting specific contaminated samples could have applications in identifying the use of copyrighted materials.
• Extending ConStat to Multimodal Models: Applying our approach to multimodal models, such as those involving images, would be valuable. Synthetic sample generation for complex data types like tabular images presents unique challenges, but overcoming these would increase the applicability of contamination detection. Unfortunately, due to space constraints, we did not discuss these topics in the NeurIPS submission. We will therefore not include it.

We hope our answers address all of the reviewer’s concerns, remain happy to answer any further questions, and look forward to the reviewer’s response.

We thank the reviewer for their detailed review and insightful questions. We are happy to hear that they find our paper a joy to read, our definition novel and interesting, and our experiments comprehensive and solid. Below, we address their questions.

Is measuring contamination relative to a set of reference models sufficient for contamination detection?
Please see our detailed answer in the main response.

Do we need to construct a synthetic dataset each time we want to measure contamination? If so, is this expensive?
No, we do not need to generate a new synthetic dataset each time we measure contamination. As long as our synthetic datasets remain private, they can be reused for future models. Of course, we will share these datasets with other interested parties if requested.

A contamination analysis on other benchmarks would require us to create a new synthetic dataset. However, the cost of generating these datasets is relatively low compared to the total inference cost of model evaluation. Specifically, generating the synthetic dataset was only around five times as expensive as a single model evaluation. Since we evaluate 50 models on all benchmarks, the total cost of generating the synthetic dataset is much lower than the total inference cost of model evaluation. Finally, specifically for rephrasing, this task can be performed by cheaper open-weight models, further reducing costs.

Why have you not included citations to some recent works like Yang et al. and Jiang et al.?
Please see our detailed answer in the main response.

How does the number of samples in the synthetic dataset affect ConStat’s performance?
We choose the number of synthetic samples based on the trade-off between tight confidence bounds and computational budget. Computational complexity increases linearly with the number of samples, while the size of confidence intervals decreases. We found that 1000 samples provide tight confidence bounds and allow us to evaluate over 50 models within our budget.

Similarly, how does the number of samples used for fine-tuning affect results?
We again chose the number of samples used for fine-tuning in our ablation study and for comparison to other methods based on the trade-off between tight confidence bounds and computational complexity. While using more samples (a larger portion of the dataset) would yield tighter confidence bounds, it would also increase the cost of finetuning.

We hope our answers address all of the reviewer’s concerns, remain happy to answer any further questions, and look forward to the reviewer’s response.

We thank the reviewer for their detailed review and insightful questions. We are pleased to hear that they appreciate the extensive experiments and find the new definition clear and easy to understand. Below, we address their questions.

Does an abnormally high performance on your reference benchmark always imply information-flow based contamination on the actual benchmark?
Please see our detailed answer in the main response.

What is the random-guessing model that you add to the set of reference models?
The random-guessing model randomly guesses the answer to a query. It serves as an equivalent to including a very poor model in the reference set (e.g., gpt2-small). For open-form questions, this model's answers are always incorrect, resulting in 0% accuracy. For multiple-choice questions, it has a 1/k probability of being correct, where k is the number of options. Adding this model regularizes the hardness correction function's fit, ensuring it remains valid outside the score range of the other reference models. Figure 3f in Appendix B illustrates the impact of excluding the random-guessing model: without it, the hardness correction function overfits the reference models and fails to detect contamination accurately for weaker models. Including the random-guessing model prevents this issue.

Could you provide more details on the motivation for how the method is developed?
Certainly! In the methodology section, our goal is to develop a method that compares model performance across two related benchmarks and detect significant performance differences. Ideally, we would want to compute the performance of the model on both benchmarks and directly compare these results. We would like to conclude it is contaminated if the model scores high on one benchmark, but poorly on another. However, this straightforward approach is problematic because benchmarks often vary in difficulty, making direct performance comparisons unreliable. To address this, we introduce the hardness correction function, which maps performance from one benchmark to the corresponding expected performance on the other. This adjustment allows for performance comparisons, corrected for benchmark difficulty. The other parts of the section describe how to transform this method into a statistical test to calculate p-values for contamination. We note that all other reviewers found our paper well-written, with some even describing it as a joy to read. Thus, we believe our paper's structure and explanations are clear and precise. However, if the reviewer has further questions, we are more than happy to provide additional clarification.

Why have you not included citations to some recent works like Yang et al. and Jiang et al.?
Please see our detailed answer in the main response.

We hope our answers address all of the reviewer’s concerns, remain happy to answer any further questions, and look forward to the reviewer’s response.