How Much Can We Forget about Data Contamination?

Thank you for the detailed review of our paper. We are happy to hear that our paper is “a pleasure to read”! Below, we respond to your questions/comments.

“deduplicating the contamination benchmark data (e.g. HellaSwag) from the pre-training tokens (e.g. FineWeb) is also recommended for a truly unbiased evaluation.”

While it will not be possible to re-run the experiments with de-duplicated pre-training data, we commit to adding a Supplement Section that reports the overlap between the pre-training data and the benchmark questions that we contaminate with.

Preliminary experiments with a random subset of benchmark questions and a random subset of the pre-training data indicate that the overlap between the benchmark questions and FineWeb-edu is small (as measured by the fuzzy string metric used to de-duplicate the benchmark questions).

“Key references are cited and discussed in the paper, although prior work's contributions can be highlighted a bit more.”

Thank you for the additional reference [3]; we will incorporate it. We think that the reviewer's positioning of our contribution in the literature is quite fitting! We will revise the relevant parts of the paper to highlight the contributions of prior work better.

“Fig 1, 2: when you scale more pre-training data ("Chinchilla Tokens"), are the extra tokens fresh tokens,”

Yes, they are fresh tokens. We will replace or qualify the usage of the term “epoch,” which can indeed be confusing in our setting.

Thank you again for providing such a high-quality review. We would be happy to answer any additional questions.

Thank you for the detailed review of our paper. We are happy to hear that you appreciate our experimental design. Below, we give answers to your questions/comments.

“The proof assumes constant learning rates”

To clarify, the proof does not assume constant learning rates (in the proof, the learning rate is denoted $\gamma_i$ and depends on the gradient step). We conjecture that the reviewer observed that $\lambda$ in the proof is constant - this is the weight decay (which is usually constant, but the algebra in the proof does not require this).

“I recommend that the authors open-source the code and experimental data logging of this work in order to verify reproducibility and increase the transparency of the work.”

The code is available at https://github.com/icml9771/code (the link is currently in the supplement; we will move it to the first page). We additionally commit to open-source our Weights & Biases logs and model checkpoints.

“Could paraphrased or semantically similar contamination (vs. exact matches) alter the conclusions, especially for larger models?”

This is an interesting question. The reviewer is correct in observing that we decided to consider only exact contamination because non-exact contamination might behave qualitatively differently for larger models (see also Supplement A.1. Additional Discussion of Data Contamination Assumptions and Setting). For example, larger models might observe a kind of ``emergence’’ phenomenon where they can suddenly make efficient use of rephrased samples.

That being said, we agree with the provided reference Xu et al. (2024), which states that exact contamination is more severe than other forms of contamination (e.g. the last paragraph on page 4 in Xu et al. (2024)). This means that the results in our paper should provide a heuristic upper bound for what would happen with paraphrased or semantically similar contamination.

Of course, ultimately experimental evidence would be required for other forms of contamination, too.

We would be happy to answer any additional questions.

Thank you for the detailed review of our paper and insightful questions. Below, we give detailed answers to your questions/comments.

“The main limitation is the relatively small scale of experiments, though this is understandable given computational constraints.”

Running experiments with a 7B parameter model is the largest we can afford on an academic compute budget. We hope that, inspired by our work, future work will study forgetting and contamination as part of a large-scale training run.

“Your experiments provide insights into LLM forgetting, but it is unclear what actionable takeaways arise from these findings. How do you envision these results informing model training or evaluation practices?”

A key insight of our work is that the causal effect of a small number of data points in the LLM pre-training data on final model behavior can be zero. If model training is in this large-data regime, this has several important practical implications:

• Our work questions the common practice of grouping benchmark questions into “clean” and “contaminated” questions based on the maximum found overlap between a question and the training data (for example, n-gram overlap) to reason about benchmark overfitting (Brown et al. 2020, Touvron et al., 2023, Dubey et al., 2024). Specifically, our work shows that the mere existence of an evaluation data point in the LLM pre-training data is not necessarily an indicator of benchmark overfitting. We show that it is necessary to consider other factors, such as the frequency of the contamination.
• Our work highlights the importance of when a contaminated sample is seen. If the samples are seen early during pre-training, overfitting is mitigated due to the forgetting phenomenon. This is directly actionable: If we suspect benchmark overfitting during an early pre-training stage on relatively unfiltered internet data, the practitioner can consider continual pre-training (“mid-training”) on clean data to mitigate the overfitting.
• Our work suggests that filtering benchmark questions from the mid-training and post-training data is much more critical than filtering the pre-training data (additional factors like the learning rate schedule come into play). This has immediate and direct implications for the design of training datasets.

We would be happy to further elaborate on these points.

“Section 5 suggests weight decay contributes to forgetting, but the analysis is mainly based on gradient decay without considering how earlier gradients influence the optimization trajectory. [...] what is the key insight of this section? Are you proposing weight decay as a primary mechanism, or just one possible factor?”

Let us outline the motivation for Section 5 in the paper and our interpretation of the experimental results.

Given that forgetting is a key empirical property of LLM pre-training, it is natural to ask: Is there a simple factor in the pre-training pipeline that explains the forgetting? Asking this question, it seemed straighforward to consider the weight decay parameter: After all, weight decay is a mechanistic process that gradually removes the influence of earlier gradient updates on the model weights (this is what we formally describe in Proposition 1).

In our experiments, it turned out that the weight decay parameter does indeed influence forgetting in the sense that increasing the weight decay parameter leads to faster forgetting (Figure 4). The experiments also revealed that the empirical rate of forgetting occurs faster than the cumulative weight decay. This makes a lot of sense to us, given how weight decay works mechanistically. At the same time, the experiments also demonstrated that weight decay is not necessary for forgetting (Supplement Figure 8) – though forgetting without weight decay occurs at a slower rate. We don’t see this as a problem for our analysis—it simply means that forgetting is a multifaceted phenomenon not exclusively driven by weight decay.

Section 5.3 suggests that the mechanism of forgetting via weight decay is likely relevant in large-scale LLM pre-training.

To summarize, the key insight in Section 5 is that the weight decay parameter has a causal effect on forgetting that is likely relevant in large-scale LLM pre-training.

“What role does data attribution play in your findings, and how does it relate to the core contributions of the paper?”

Our paper's controlled experiments resemble the “leave-one-out” (LOO) procedure in data attribution. Concretely, our experiments directly “leave-out” entire groups of benchmark questions, thus simulating a kind of “group” LOO procedure.

Moreover, our result that the (average) causal effect of inserting individual benchmark questions into the pre-training data can be zero suggests that we might not be able to attribute “importance” to individual data points in the LLM pre-training regime.

We would be happy to answer any additional questions.

Thank you for reviewing our paper and for your positive assessment.

It seems that you ask under what conditions we can be confident that a benchmark evaluation is not contaminated. This is an interesting question that lies somewhat beyond the scope of our paper. In our paper, we demonstrate that moderate amounts of exact contamination do not necessarily lead to benchmark overfitting. This implies that minor mistakes in data pre-processing and filtering might not lead to benchmark overfitting. To provide stronger guarantees that a benchmark evaluation is "totally safe", we would require a much deeper understanding of the learning dynamics of LLMs.

We would be happy to answer any additional questions.