How much can we Forget about Data Contamination?

We thank the reviewer for appreciating our paper, especially concerning our scientific approach.

“The theoretical framework for forgetting seems to rely on the assumption that contamination occurs uniformly and that the data scale follows the Chinchilla scaling law.”

The reviewer is correct to observe that our theoretical framework relies on simplifying assumptions. In particular, we only model forgetting of samples contained in individual gradient updates. The case of repetition could be handled insofar as repeated gradient updates could all be forgotten, but this might not be very realistic.

The theory does not assume that the data follows the Chinchilla scaling law.

Please also see the global comment on the theoretical framework.

“Given the focus on cumulative weight decay, how does your theory account for rare or privacy-sensitive data points that might resist weight decay due to repetition? [...] Do you have insights on why the empirical forgetting occurred faster than the predictions by your theory? What are the factors contributing to the discrepancy?”

Great question. Indeed, forgetting might behave differently for rare and privacy-sensitive data points. Most importantly, one would likely want to use a different metric to assess forgetting in this setup than the average accuracy gap (see, for example, Jagielski (2023)).

Our theory does apply to the privacy setup as long as a data point is seen only during a single gradient update. If repetition is introduced, the question becomes more empirical, and we agree with the reviewer that it would be very interesting to study this experimentally. We also agree that it would be interesting to consider the factors mentioned by the reviewer, especially how gradient alignment interacts with fast forgetting. We will leave the investigation of these questions for future work.

We would be happy to answer any additional questions during the discussion phase.

We thank the reviewer for taking the time to review our paper.

“The theoretical analysis is very limiting - the only case considered is forgetting the exact gradient of the contaminated example.”

While the analysis is simple, it suffices to demonstrate significant forgetting in large-scale LLM training runs. Please look at our global response for more details on the theoretical analysis in Section 5.

“Experiments with larger models are needed to validate the findings across scale”

To address this, we conduct a forgetting experiment with the OMLo-7B model. This experiment requires substantial computational resources and is still running. Supplement Figure 9 depicts preliminary results. The results are qualitatively similar to those of the OLMo-1B model (Figure 4 in the main paper), indicating that our results extend towards the 7B parameters scale.

“Why the frequency of the repetitions are chosen as 4, 12, 32, 144 (instead of say 4,16,32,128 - the powers of 2) ?”

That's a fair point. The frequencies were chosen based on preliminary experiments with the 124M model to cover the range from statistically significant contamination to complete overfitting. In hindsight, seeing the full experimental results (Figure 2), we agree that it would be sensible to choose the powers of 2.

“The forgetting phenomenon is attributed to the weight decay, does it mean training without weight decay does not forget contamination.”

Please see the global comment: It turns out that a model trained without weight decay also forgets contamination, but not necessarily at the same rate.

We would be happy to answer any additional questions during the discussion phase.

I thank the authors for their reply to my questions.

Given that the forgetting phenomenon is not attributed to weight decay, i.e., the network forgets even while training without weight decay. It seems like there is some hidden phenomenon that causes this forgetting, weight decay merely accelerates it but it is not actual cause. Hence, it is a bit misleading to attribute this to weight decay and I hope the authors really tone it down.

I do not agree with the comment that forgetting past gradients is sufficient for forgetting the data, consider this simple setup of GD, after t time steps, the iterate writes as $\theta_{T} = w_ {0}^{T} \theta_{0} - \sum_{j=1}^{T-1} w_{j}^{T} g_{t},$ note that here $g_{t} = \nabla L(\theta_{t})$, so that gradient at some $i < t$ also contributes to this gradient, even if it explicitly forgot $g_i$, its influence can be propagated through these gradient computations.

We thank the reviewer for taking the time to review our paper.

“My main concern is about the theory that weight decay influences forgetting. It's unclear to me why "forgetting past gradients" would equate to forgetting past training data since the model weights have already been updated based on that past data.”

General remarks on the theory in Section 5 are provided in the global comment.

As we also outline in the global comment, we claim that "forgetting past gradients" is sufficient for "forgetting past data," not that it is equivalent. We agree that this assumption underlies the theory in Section 5. The additional experiment depicted in Figure 7 of the revised Supplement provides strong evidence supporting this assumption.

The reviewer makes an interesting observation about the optimization trajectory and how this is related to example forgetting. The novel experiment depicted in Supplement Figure 7 shows that the effect mentioned by the reviewer is unlikely to exist, at least in our setting.

We agree with the reviewer that weight decay might not always be the main factor behind forgetting (see the global comment). However, it suffices to demonstrate significant forgetting in large-scale LLM training runs.

“I'm unsure if simply comparing performance on the leaked benchmark is sufficient to fully capture whether the models have been influenced by data leakage”

For this paper, we decided to use benchmark accuracy as our primary metric because this is the performance measure that LLM developers are expected to report.

“What is the practical impact of these findings?”

Our empirical results show that individual samples can have a negligible impact on model performance and that there is a significant forgetting effect during training. This has many practical implications, for example:

Memorization-Aware Training Pipelines. Our insights regarding forgetting can be used to design training pipelines that mitigate benchmark contamination. Concretely, our work suggests that contaminated models may continue pre-training on “clean” data to ensure that contamination is forgotten.

Data Attribution. Traditionally, data attribution methods attribute model performance to individual data points. We show that this might not be a valid approach in large-scale settings, implying that model performance needs to be attributed to larger clusters of data instead (for example, as in Ley et al. “Generalized Group Data Attribution”, 2024, https://arxiv.org/abs/2410.09940).

Data Pre-Processing. The fact that individual samples are negligible has implications for data pre-processing, filtering, and cleaning (an increasingly significant topic of research for LLMs).

“More of a future direction but would be interesting to see how data contamination affects other types of tasks and architectures, such as vision models and non-transformer models, as well as scenarios where training data are encountered multiple times. It would also be valuable to explore whether the weight decay explanation holds in these cases.”

We agree that this would be interesting to explore in future works!

“If the observed influence of weight decay on forgetting is specific to LLM training, what aspects of LLM training might be driving this effect?”

Training for a single epoch and more than 100,000 gradient steps.

(Part 1/2)

We thank the reviewer for the review and appreciating our paper.

“there will reasonably always be contaminated samples in the last part of the training, say, in the final Chinchilla samples, that the final model has no time to forget. [...] could you please elaborate on this”

Great observation. While more experimental evidence would be required, we agree with the reviewer that our work hints at the fact that the model might always remember specific samples. In this regard, there are a couple of points to keep in mind.

One point is the learning rate schedule. Because the learning rate usually decays towards the end of training, it is not necessarily the samples that are seen last that are “best remembered” by the model. For example, Figure 3(e) and Figure 3(f) in the main paper show that benchmark questions that are uniformly distributed throughout training exhibit more overfitting than benchmark questions seen toward the end.

Another point is that modern language models go through different stages of training. For example, an LLM might be pre-trained on data from the internet, continue training on synthetically generated data, and finally be fined-tuned as a chat model. In such a training process, the stage where the leakage of benchmark questions into the training data is likely to occur is the first stage of pre-training on data from the internet. For this paper, we are primarily interested in the leakage of benchmark questions, and our experiments are mainly meant to cover the initial pre-training stage. This means that the final samples seen by our models would not necessarily be the final samples seen by the model in a realistic training setup.

Regarding our benchmark questions setup, it does not matter much if the model remembers specific individual questions as long as the overall fraction remains negligible (compared to, say, 10,000 test questions). However, studying this question would be very relevant in a privacy setup.

“I was wondering if there is a proportionality law between the percentage of contaminated samples and the observed accuracy gap.”

We agree that holding everything else constant, the accuracy gap decreases with the percentage of contaminated samples (though the relationship might be non-linear). At the same time, Figure 2(a) shows that the accuracy gap increases in the number of model parameters even if the percentage of contaminated samples is held constant. So the percentage of contaminated samples is an important ratio, but it does not uniquely determine the observed accuracy gap.

We would be happy to answer any additional questions during the discussion phase.

I thank the authors for their responses. I will keep my score and reaffirm my appreciation for the paper.