To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination

We thank you very much for your thorough and extensive review. We appreciate that you find our work tackles an important issue, while controlling for confounding variables. We reply to your questions below.

1. Additional LLMs. Thank you for the suggestion! We originally focused on GPT-3.5 and GPT-4 for a few critical reasons: (i) relevance to the large community, in and out of research, that rely on these models, (ii) these models’ high performance on code generation tasks. With respect to this second point, we clarify that when a model’s pass rate on these problems is prohibitively low, we are effectively unable to observe non-trivial trends or extract longitudinal insights–and we have found this to be the case for some additional models we have tested. For example, GPT-3.5 achieved 27% pass rate before the cutoff and 13% after, while GPT-4 achieved 37% pass rate before and 16% after. In contrast, the pass rates we obtained for Text-Davinci-002 with a comparable prompting strategy are less than 1% before and after–yielding it impossible to conduct a substantive analysis featuring functional correctness.

We also are actively collecting data from some other models, including a modern open-source LLM, and plan on providing these results when the results are available soon (we aim to do so before the end of this discussion window). This discussion is also repeated in the new Appendix B.6.

2. Longitudinal datasets.

We agree that our analysis requires longitudinal datasets; however, we anticipate that the popularity of such benchmarks will continue to rise, as releasing datasets over time is potentially the only foolproof method for avoiding data contamination. For example, creators of recent benchmarks such as BIG-Bench are consciously updating their benchmarks over time. Even other non-LLM benchmarks such as OpenML Benchmarking Suites are being updated over time, since there is a related push to ensure that the community does not overfit to any single set of benchmarks. We are optimistic that as staggered release of benchmark datasets becomes increasingly common and/or an accepted best practice, that the framework we present here can be more widely applied.

Furthermore, we note that while our analysis can only be conducted on longitudinal datasets, its implications extend to any dataset which can be contained in a webscale training dataset; it leverages longitudinal structure in a test benchmark to reveal the massive contamination effect which can be observed on “seen” examples of any dataset.

3. Minor comments. Thank you for catching these formatting issues; they have now been fixed in the updated pdf.

Thank you once again, for your excellent points. Please let us know if you have any follow-up questions or further suggestions. We are still conducting new analyses and are excited to update you with their results. We would be happy to continue the discussion.

Thank you again for your thoughtful review. We are writing with more experiments which we have conducted during the rebuttal period which address you concern W1: Additional LLMs. We liked your suggestion of adding more LLMs beyond the GPT-4, GPT-3.5, and Davinci models which were in our original manuscript.

We have now added the analysis of two more LLMs on Codeforces: Codey/Code Bison (Google’s code generation foundation model, released around the same time as PaLM 2), and Code-Llama.

As for Code Bison, we very interestingly observe the same behavior that we did for GPT-4 and GPT-3.5-Turbo. Specifically, after the training cutoff of Code Bison, we see that the GitHub presence metric is no longer a significant predictor of functional correctness for this model. code-bison@001’s training cutoff, although not publicly known, is probably around February 2023, as this model was released within weeks of text-bison@001 and chat-bison@001, which have known training cutoffs of February 2023. Accordingly, we conducted our analysis assuming this February 2023 cutoff. We have updated our manuscript to show this by updating Figures 2 and including all additional figures and tables in the appendix as with the GPT models.

The results from this additional model shows that the training cutoff itself is a moderating factor in the code generation completion performance. This conclusion is further bolstered by the almost 2 year separation between the GPT cutoff and the Code Bison cutoff and yet still producing the same qualitative finding.

We additionally worked with Code-Llama and attempted to the best of our abilities to get it to perform on these questions. Unfortunately, even on the largest and most capable model for our use case, 34b-Instruct, we could not get Code-Llama to output answers which yielded pass rates above 1% on Codeforces. This result follows that of our observation around Text-Davinci-002.

We hope that these additional experiments address your main concern about the limited number and models in our experiments. We now conduct analysis of 5 LLMs with various cutoff dates, and we observe signs of contamination for those models which yield pass rates above 1%. We look forward to answering any additional questions you may have, before the author response period closes tomorrow (end of day Nov 22nd). Thanks!

W5: GPT-generated outputs on problems released after the cutoff.

To address this question, we would like to: (a) make a minor methodological clarification; and (b) note that we have added a new section to the appendix (B.8) that contains numerous examples of GPT-4 and GPT-3.5.-Turbo output (i.e., generated code) for Codeforces problems released before and after the GPT training cutoff. (We restrict our attention to Codeforces because our Project Euler analysis focused on the correctness of generated numerical solutions, rather than evaluating the outputs of generated code).

With respect to (a), we note for clarity that what we are trying to control for in our natural experiment are factors with the potential to influence an LLM’s ability to produce a functionally or numerically correct solution to a problem (i.e., the problem’s difficulty), and/or the likelihood that the LLM would have seen this problem during training (i.e., GitHub presence, since scraped GitHub repositories are part of the GPT training corpus). Thus, GPT need not produce worse code when we evaluate its ability to produce functionally correct solutions for problems released after the cutoff. Indeed, given that we perform all of our evaluations at a single point in time, the code-generating ability of each GPT instance is “fixed”--what is (potentially) changing is the artificially inflated performance benefit (or lack thereof) the model may demonstrate when it is evaluated on examples it has seen during training versus those it has not.

With respect to (b)---i.e., our new appendix section—because the full description and generated output associated with a given problem can both be quite lengthy, we have constructed this new section by partitioning the Codeforces problems and along two dimensions: (1) release date pre- vs. post GPT cutoff; (2) discretized LLM functional correctness. With respect to (2), for a given LLM and problem, functional correctness is computed as the ratio of test cases that the LLM’s generated code passes (see Section 4 for details). Thus, functional correctness takes values in [0,1.0]. We discretize via the following mapping:

$\lambda: x \mapsto 0 | (x \leq Q1) \lor 1 | (Q1 < x \leq Q2) \lor 2 | (Q2 < x \leq Q3) \lor 3 | x > Q3$

where $x$ represents a given problem's raw difficulty score, and Q1, Q2, and Q3 correspond to the first, second, and third quartiles, respectively.

We thus consider 8 subgroups per model (i.e., {pre, post} x {0,1,2,3}), and draw two examples uniformly at random from each subgroup. We present each generation along with corresponding problem title, ID, difficulty score, url, and the LLM’s functional correctness score. Our intent here is to avoid cherry-picking of results, while facilitating the visualization of a representative set of generations. We also note that the generated code produced by each model for every Codeforces problem is available in the results file that we provide as part of our supplementary material. We view additional/comparative static and behavioral analyses of these outputs as a promising direction for future work.

Table 1. Thanks once again for this suggestion! We have now added these figures. We replaced Table 1 and Table 2 with their corresponding forest plots. Additionally, we have added the forest plots into the appendix for every regression coefficient table. These figures give new visual insights so the reader can quickly understand the relative relationship between the regression coefficients. For example, we can easily see that the coefficients for both Difficulty and GitHub presence get closer to 1 (the null effect) after the cutoff.

Thank you once again, for your excellent points. Please let us know if you have any follow-up questions or further suggestions. We are still conducting new analyses and are excited to update you with their results. We look forward to continuing the discussion.

(2/2)

We thank you very much for your thorough and enthusiastic review. We appreciate that you find our work well motivated, clearly conceptualized, and well-executed. We agree with all of your suggestions and have done our best to address them below in and in the updated manuscript. Please let us know if we have addressed your comments.

W1+2: Fig 1 minor changes. Thank you for pointing out these stylistic changes. We have now moved the legend and updated “Github” -> “GitHub”.

W3: Fig 1 for Project Euler. Thank you; this plot can be found in Figure 17; the associated plot for GPT3.5 can be found in Figure 19. Here you’ll see how the performance of these systems on Project Euler is much decreased on problems released before the training cutoffs, and entirely wiped out for problems released after the cutoff — neither GPT-4 or GPT-3.5 yield any pass rates above 0 for problems after the training cutoff.

W4: Pass rate when log(Github Presence)=0. This is a great insight! We believe that the main factor at play is b) - GitHub Presence is a strong indicator of availability in the training set, but it still underestimates the true availability in the training set. For example, all Codeforces problems show up on the internet in several places (e.g., here, here in pdf). Therefore, we believe this is a major reason for why the completion rate is higher before the cutoff even for problems with log(GitHub Presence)=0. We comment on your other hypotheses as well.

1. In terms of the models themselves, we used the exact same models when evaluating problems released before and after the cutoff date (in fact, the evaluations were performed in one large batch and later separated by date).

2. As we state above, we believe that this is the main factor. We agree that “GitHub presence” may under-estimate the total presence of a Codeforces problem in the GPT training data. For example, all Codeforces problems show up on the internet in several places (e.g., here, here in pdf).

3. It is challenging to fully rule out this possibility, but qualitatively, we cannot spot any change over time in the type of Codeforces problems released. It is possible that a difference in problems does cause a small change in pass rate after the cutoff, but we believe that the majority of the observed difference in pass rate can be attributed to (2).

   1. To investigate, we conduct a set of additional experiments to assess whether the distribution over tags (only available for Codeforces) and/or difficulty level (available for Codeforces and Project Euler) changed in a statistically significant way during the post-cutoff period, relative to the pre-cutoff period. We present this analysis in its entirety, consisting of qualitative plots and $\chi^2$ tests, in Appendix B.7 of our updated submission pdf, and summarize key findings below:
      • For Codeforces, we do not find any statistically significant difference in the distribution of normalized counts over problem tags during the pre-vs. post- period. We also do not find any statistically significant difference in the distribution over discretized difficulty scores during the pre-vs. Post-period.
      • For Project Euler, problem tags are not publicly available, so we cannot perform tag analysis. We do not find any statistically significant difference in the distribution over discretized difficulty scores during the pre-vs. Post-period.
      • These findings help to mitigate concerns that the drop-off in performance we observe might be attributable to significant changes in the distribution over tags (for Codeforces) and/or over difficulty levels during the post-cutoff period.

4. We ran the same evaluation script for all problems, and only then separated by date: https://anonymous.4open.science/r/to-the-cutoff-review-253A/eval/chronological_evaluation/chronological_dataset.py (line 162).

(1/2)

Figure 2

I think this looks very nice. Thank you!! Same with Figure 3.

Thus, GPT need not produce worse code when we evaluate its ability to produce functionally correct solutions for problems released after the cutoff.

I'm not sure I understand this point. We might be using different terminology. I agree that the evaluations are run at a single point in time, and I believe your evidence (e.g., Figure 1) that increasing GitHub presence is correlated with functional correctness. What I was trying to confirm is that generated code for problems released after cutoff is "worse" in the sense that we would all agree the generated code for problems released after cutoff pass fewer test cases than generated code for problems released before cutoff. My intention was to rule out that the evaluation process itself somehow (potentially unintentionally) affected the performance.

For example, suppose that before the cutoff, Codeforces required code to be submitted in format A (e.g., the main function should be called main()), but after the cutoff, Codeforces required code to be submitted in format B (e.g., the main function should be called start()). If so, I think it would be hardly surprising if a model pretrained on format A would perform less well when evaluated on format B, since the model wouldn't know that format A is deprecated and format B is appropriate?

This is what I meant by confirming that the code post-cutoff is "wrong". Perhaps "less functionally useful" is a better term. I want to know that the generated code for problems post-cutoff is functionally less useful than code for problems pre-cutoff. I'm not interested in things like formatting, variable name choice, etc. that we might consider when discussing code quality.

Could you please clarify what you meant by "GPT need not produce worse code" post-cutoff? How is GPT-4 scoring worse post-cutoff if it isn't producing worse code?

I went to increase your score to 7, but apparently 7 is not an option for ICLR. If we can reach agreement on this topic, and I think we can, then I would be happy to bump you up to an 8.

2. Additional experimental updates Finally, we would like to highlight a few additional experiments and updates we finished during the rebuttal period:

1. We have rendered the coefficient tables as forest plots (suggested by EfiM) and supplemented the regression tables with them throughout. See Figures 2 and 3 in the main paper, and associated figures in the Appendix. This change is very helpful for the reader to quickly visualize the regression coefficients and see how after the coefficients become closer to 1 (or have no effect) after the cutoff.
2. We have conducted new analyses to assess whether the drop-off in performance that we observe for problem examples released after the GPT training cutoff might be attributable to (potentially latent) covariate shifts.
3. We added a new section B.6 to describe our experiments with open source LLMs; we also have plans to expand to other LLMs during the rebuttal period and are awaiting results.
4. We added examples of generations from the LLMs in Section B.8.

Thank you once again, for your excellent questions. Please let us know if you have any follow-up questions or further suggestions. We would be happy to continue the discussion, and we will update you with our reanalysis soon.

(2/2)

We thank you for your thoughtful review. We appreciate that you find our longitudinal study interesting and that it should be raised to the attention of ML researchers.

1. Design of the cutoff: similar code problems. Thank you for the great question, and we understand that your question lies in the methodology of our analysis.

Recall that we propose a method to measure contamination through a natural experiment where the training cutoff bifurcates an evaluation set. We appreciate that you agree with this methodological approach and our resulting analysis, and we agree that checking for duplicated questions before and after the cutoff is an important step.

In summary, based on your suggestion, we did find 56 exact duplicates, none of which straddle the cutoff (0.7% of the total set of problems). Community experts identified 8 similar questions in the Codeforces problems; only 5 straddled pre- and post-cutoff. The overall effect on our results should be minimal considering the large drop-off in performance after the cutoff. In the coming days, we will rerun our analysis and update you and the paper before the end of the rebuttal period. We explain more below.

For textual duplicates, we have confirmed that there are none in Project Euler. In Codeforces, we have identified a set of 40 unique problems (out of a total of more than 8,300) for which the dataset contains multiple (i.e., 2-3) copies. This can occur if a competition problem is later re-posted as part of a practice set. All duplicated tuples belong to the subset of problems released before the GPT training cut-off. As the focus of our analysis is on comparing performance on examples released before versus after the training cutoff, we would be concerned if a majority of these examples were “cut-crossing”, but do not find that to be the case. Before the discussion period concludes, we will re-run our analyses omitting the duplicates, and will provide you with updated results. We expect that the impact of removing 56 observations (corresponding to the duplicates of the aforementioned 40 problems) from the pre-cutoff subset which contains >6,000 observations will be minimal, and do not expect the qualitative nature of our conclusions to change.

As for semantic duplicates, it is certainly non-trivial to check for the existence of such problems. Thankfully, the Codeforces community has discussed this topic before, highlighting 8 near-duplicate pairs, of which 5 were pairs with one question on either side of the cutoff, and the other three were pairs with both questions were released before the cutoff.

For the sake of examining those 5 pairs which straddled the cutoff, we compare GPT-4 performance:

We also compare GPT-3.5-Turbo performance:

A potential risk of including such semantically similar problem instances in our evaluation is that the performance on post-cutoff examples would be upwardly biased, since the model is actually seeing examples which are quite similar to those it has seen in training, despite our best efforts to control for such exposure via our cutoff-based partition of the dataset. The small size of this subset means this effect, were it to exist, would be relatively minimal, and it would be biased against, rather than in favor of, the conclusions we ultimately draw. As such, we do not feel that the presence of such examples undermines our analysis. We will provide an updated analysis in the coming days to confirm.

Finally, we’d like to acknowledge that we agree with your broader point -- i.e., the question of how to measure or detect when data contamination has occurred is important. We contend that our approach, which leverages the cutoff date as a source of naturally arising variation, should be viewed as a compelling complement to existing approaches, which often rely on the ability to manipulate the underlying dataset and/or synthetically introduce contamination and then measure the impact on downstream tasks. We are able to determine that contamination is likely to have occurred by evaluating performance leveraging a freely available feature of the dataset (i.e., each problem’s release date, relative to the cutoff), without requiring any sort of post-facto intervention or manipulation.

(1/2)

We thank you for taking time to review the manuscript and provide valuable feedback. We are glad to see that you find our work to be beneficial to the community of contamination analysis, our results intriguing, and our paper well-written. We reply to each of your questions below.

1. results interesting but not surprising. Thank you, we agree that the results are interesting. While we understand that the results may not be surprising to you, they are to other reviewers; XbnJ described “the findings are surprising, revealing that people should deal with the ability of LLMs more carefully”. More importantly, our work is a non-trivial contribution to the community that presents the first scientifically rigorous confirmation of the phenomena that social media posts have speculated about via small-scale/ad hoc analyses. While many informal statements have been made about GPT-4 memorization, we show the first statistically significant differences in performance on problems released before and after the cutoff date. Furthermore, our work lays the groundwork for rigorous analysis of contamination and best practices in the age of LLMs trained on webscale data. Our methodology is becoming increasingly relevant as the community acknowledges the limitations of static benchmarks for LLM evaluation, and shifts toward dynamic/longitudinal benchmarks such as the ones we construct, open-source, and analyze here.

2. On how implicit contamination is possible. To begin, we’d like to clarify our intention for including the words “implicit” and “explicit” in our abstract, specifically in reference to how a given problem might come to be included in a model’s training corpus. Our intention here was to acknowledge that the process of collecting webscale data can lead to the inclusion (in the training set) of examples that were never intentionally selected for training–for example, we consider the unintentional inclusion of BIG-bench examples in the GPT-4 training corpus (OpenAI, 2023). Through re-posting, removal of watermark information, indirect scraping, or other means, information intended to be omitted from LLM scraping is rarely truly safe. For clarity, we have modified our abstract text to refer to examples that are “intentionally” (explicitly) or “unintentionally” (implicitly) included in the training data.

On a separate note, we have newly added Appendix B.8 which includes a number of samples of generated outputs from GPT-4 and GPT-3.5-Turbo. These can be used to qualitatively examine their outputs across a range of pass rates on both datasets.

3. On scope and novelty. Thank you for your comments. We focus on code generation for a few key reasons: (i) it is a very popular use-case; (ii) most recent LLMs have code as a major part of their training data; (iii) unlike general online natural language text, GitHub has a uniform interface and is easily scrapable and cleanable, making it simultaneously easy for GitHub solutions to be added to train datasets as well as for us to assess GitHub presence; and (iv) code generation and problem solving datasets have objective correctness metrics (test cases) producing objective evaluations of open-ended generations that don’t require another model or human in the loop.

We also note that dataset desiderata for the methodological approach we propose include: (i) problems must have been released over a sufficiently long time-horizon, such that it is possible to partition examples into pre- and post- GPT training cutoff subsets based on problem release date. This restrictions precludes some other popular benchmarks which have been released in a single time-step (e.g. HumanEval or MBPP). (ii) problems should consist of high-quality questions requiring non-trivial solution generation that admit objective evaluation functions/correctness measures. We have thus focused our efforts on the non-trivial scraping and processing required to analyze Project Euler and Codeforces datasets in the way that we have. We also emphasize that our work introduces tools and paves the way for further rigorous study related to data contamination, which gives our work the potential to have high impact in the community.

We’d also like to highlight that our paper should also be viewed as a methodological one. Specifically, we believe our work takes a novel view on data contamination estimation by employing a natural experiment – a concept borrowed from the economic literature. We use this concept effectively here and argue for its further use, particularly in experimenting, evaluating, and discovering the intricacies of LLMs. We argue that this evaluation methodology should be used in particular on closed-source models which refuse to reveal critical and important details about their development/training. We have updated our manuscript to highlight this point further.

(1/2)

Thank you very much, once again, for your helpful feedback. If you find our responses satisfying, we respectfully ask that you consider increasing your score. We are still conducting new analyses and are excited to update you with their results. We would be very happy to answer any follow-up or additional questions you have.

(2/2)