Time Travel in LLMs: Tracing Data Contamination in Large Language Models

We appreciate your comments highlighting the importance of the study of data contamination, especially in the context of black-box models, and are grateful for your constructive feedback.

We address your concerns in the following in the same order they are listed.

Aggregation from instance- to partition-level contamination.

In our validation study, we empirically validated the number of exact/near-exact matches that can signal partition-level contamination by performing controlled contamination on the base model of GPT-3.5 (Section 3.3). Then, based on these findings, we studied a set of numbers for both exact and near-exact matches as our default setting to generalize to the partition-level contamination.

We would like to emphasize that the aforementioned numbers we studied as our default setting for determining partition-level contamination are not fixed. As mentioned in our paper, these criteria, i.e., the number of exact or near-exact matches, can be adjusted depending on a specific context and the perceived severity of contamination in a given setting. In other words, our replication-based strategy is completely separate from the way that a dataset partition is flagged as contaminated based on the number of exact/near-exact matches.

The unavailability of training data for black-box models.

We understand your concern about the unavailability of the pretraining data for black-box models. In this regard, one of the main purposes of our validation study (Section 3.3) is to address this concern by introducing contamination through known data to the GPT-3.5 base model and replicating this known data using our approach to validate our strategy. We also experiment with the GSM8k dataset on GPT-4, as a known data included in the pretraining data of GPT-4 based on the technical report provided by OpenAI. These experiments showed that exact/near-exact matches, replicated through guided instruction, are indeed indicative of prior exposure to the data, thereby revealing data contamination. We have also provided several examples of exact/near-exact replicas from this experiment in Figure 2 and Table 11 in Appendix G in the updated version of our submission.

In addition to the aforementioned validation study in our paper, we would like to refer to the study by Carlini et al. (2023) where they assessed the memorization ability of LLMs by replicating their training data. Inferring from their results and observations, replication of certain data confirms its presence in the training data of the LLMs. This implies that even without direct access to the actual training data, the presence of specific data can be inferred and confirmed through replication of this data by disclosing the LLM’s internalized information. From this perspective, as our method fundamentally detects contamination by confirming the presence of dataset instances through replication, experimenting with both closed-source and open-source LLMs would yield the same insights.

The comparison of Alg. 2 (GPT-4 ICL) with human judgments seems to be rather biased, since the same human annotators created the ICL examples.

In almost all scenarios where Machine Learning models/algorithms are applied, the underlying purpose is to design a system that can follow human judgments, e.g., labels in classification tasks, summaries in summarization tasks, etc. In our case, GPT-4 ICL aims to follow human judgments for detecting data contamination.

The first paragraph of Section 3.1.1 is quite confusing.

Thank you for bringing this to our attention. We rephrased this paragraph (in blue) in the current revised version of the paper to address the clarity.

I wonder how the safeguards would affect the results you get with your prompt templates.

Using our prompt templates, we did not observe any cases in which the safeguards were triggered, so we always got completion for the provided partial reference instance in the input prompt.

For unpaired instances, a random-length prefix is displayed in the prompt; how is this random length sampled? And what is its effect?

The selection of both the instance itself and the length of the prefix used in the prompt was performed at random. As we also mentioned in our paper, for unpaired instances with multiple sentences, the prefixes are created by arbitrarily cutting the instances at the end of a complete sentence, whereas for instances containing a single (long) sentence, a random sentence fragment is removed. In terms of the effect of length, we would like to refer to the well-studied research by Carlini et al. (2023) where they found that memorization in LLMs significantly grows as we increase the number of tokens of context used to prompt the model.

Typo: "obtains" -> "obtained"

Thank you! We fixed the typo in the current revised version of our paper.

References:
[1] Quantifying Memorization Across Neural Language Models (Carlini et al., ICLR 2023)

Thank you for acknowledging the originality and significance of our study, and for your detailed review of our work.

We address your comments and questions in the following in the same order they are listed.

The scope of LLMs.

We appreciate your suggestion about including a broader range of LLMs. Our focus on GPT-3.5 and GPT-4 was informed by their prevalence in the field and the unique challenges of data contamination in closed-source LLMs, which are less transparent than open-source models both in terms of the model weights and pre-training data. While this focus allows for an in-depth exploration of these specific models, we acknowledge the potential benefits of expanding our study to include a more diverse set of LLMs for broader insights.

Detecting data contamination for larger LLMs using our method.

For larger LLMs, based on the well-studied research by Carlini et al. (2023), as the memorization ability of LLMs significantly grows with their capacity, we believe for larger LLMs, it becomes even easier to replicate instances using our template prompts.

Detecting data contamination for architectures beyond GPT.

Since our method hinges on replicating instances to validate their presence by emitting the inner learned knowledge, our method works best with decoder-only components (e.g., GPTs), although for other components like encoder-only (e.g., BERT), Magar & Schwartz (2022) investigated other methods.

Computational cost trade-offs between the two algorithms.

In terms of the computational cost trade-offs between the two algorithms, Algorithm 1 with ROUGE-L is much friendlier than Algorithm 1 with BLEURT and Algorithm 2 (GPT-4 ICL), as these two need GPUs to run. On the other hand, the latter two methods use dense representations to evaluate the candidate text against the original instance, thereby performing better in terms of semantic match.

Reliability comparison between the two algorithms.

In terms of reliability, since by using GPT-4 ICL, it is possible to provide examples of exact/near-exact matches from human evaluation for downstream judgments, it is much more aligned with human evaluations compared to Algorithm 1 with BLEURT and/or ROUGE-L.

References:
[1] Quantifying Memorization Across Neural Language Models (Carlini et al., ICLR 2023)
[2] Data Contamination: From Memorization to Exploitation (Magar & Schwartz, ACL 2022)

Thank you for recognizing the intuitive and innovative design of our approach, and for your valuable feedback.

We address your comments in the following in the same order they are listed.

Circular dependence/assumption.

To address the concern about circular dependence, we would like to mention that we were able to introduce contamination in our validation study (Section 3.3) using a set of synthetic instances from a fake dataset to break this circular dependency that seems to be present in the assumption. However, we found this a little bit confusing for the readers. Therefore, the main idea of contaminating using the partitions that were not leaked was only to maintain consistency throughout the paper for readers. Otherwise, there is no circular dependency.

Possible contamination through the metadata of another dataset.

Regarding the possible contamination arising from metadata in another dataset—a point raised in both the Weaknesses and Questions sections—we would like to state that this is an interesting issue that we leave for future study. Our current work in this paper focuses on the complex issue of detecting the presence of contamination and not on where the contamination comes from. Nevertheless, we acknowledge the importance of identifying the sources of contamination.

Circular dependence/assumption.

It was indeed confusing between bootstrap and circular dependency, thank you for the clarification.

Metadata contamination

I appreciate that this problem is beyond the scope of the paper, but still feel this should be mentioned as a potential limitation. I am still unconvinced if the paper can distinguish between contamination or just metadata contamination for a sufficiently accurate model. I do not feel this devalues the result, as for certain tasks, both can/should count as contamination regardless of the mechanism.

Thank you for your insightful comments on our paper, particularly regarding the soundness of our evaluation methodology. We appreciate the opportunity to address your concerns and provide further clarification on the underlying assumptions of our proposed method.

Regarding your points 1.1 and 1.3, we would like to clarify that these concerns have already been considered and addressed in Section 3.3 of the paper. For point 1.2, we provide a detailed explanation below, complemented by an example for clarity. Moreover, to address your concern more thoroughly, we have included additional details in Appendix B in the updated version of our paper to show how infusing the labels in the input prompt accounts for false positives when generating downstream completions.

We address your concerns in the following in the same order they are listed.

1.1. The inability of an LLM to complete a reference instance.

We understand that an LLM’s inability to complete a partial reference instance could be due to various factors. However, the memorization ability of LLMs is a well-studied topic. Carlini et al. (2023) suggest that memorization in LLMs significantly grows as we increase (1) the capacity of the model; (2) the number of tokens of context used to prompt the model; and (3) the number of times an example has been duplicated. The first two are the ones we directly make use of in our study. Therefore, consistent failure/inability to complete partial reference instances more plausibly indicates the lack of prior exposure to the data, rather than forgetting surface-level features or failure in memorization, especially in the context of LLMs such as GPT-4 and GPT-3.5.

Further, in our study, we carefully examine near-exact matches to account for the exact scenarios you highlighted. To control the false negative rate in these cases, we empirically validated the number of near-exact matches that can signal partition-level contamination by performing controlled contamination on the base model of GPT-3.5 (Section 3.3). Then, based on these findings, we studied a set of numbers for both exact and near-exact matches as our default setting to generalize to the partition-level contamination.

We would like to emphasize that the aforementioned numbers we studied as our default setting for determining partition-level contamination are not fixed. As mentioned in our paper, these criteria, i.e., the number of exact or near-exact matches, can be adjusted depending on a specific context and the perceived severity of contamination in a given setting. This flexibility in our approach allows for the effective addressing of potential false negative concerns. In other words, our replication-based strategy is completely separate from the way that a dataset partition is flagged as contaminated based on the number of exact or near-exact matches, adding the benefit of proactively handling potential false negative concerns.

1.2. The importance of label inclusion in the input prompt.

The pivotal idea behind incorporating the exact label in the input prompt is to exactly account for the false positive rates you are referring to. To address your concern, we intuitively show that when an LLM is instructed to complete a partial reference instance, the exact label corresponding to the reference instance is taken into account by the LLM for the downstream completion.

To exemplify, in the WNLI dataset, sentence 1 entries are not unique, meaning that the same sentence 1 can be used with a different sentence 2, resulting in a different label. As a result, this is a suitable case to show how including a label can impact downstream completion. Please consider the following examples and completions:

Example 1 (from the validation split of the WNLI dataset, used in our paper in Figure 1):
sentence 1: The dog chased the cat, which ran up a tree. It waited at the top.
label: 1 (entailment)
Completion for sentence 2 by GPT-4: The *cat* waited at the top. (exact match)

Example 2 (from the train split of the WNLI dataset, taken to support our claim here):
sentence 1: The dog chased the cat, which ran up a tree. It waited at the top.
label: 0 (not entailment)
Completion for sentence 2 by GPT-4: The *dog* waited at the top. (this is an exact match with respect to the instance from the train split of the WNLI dataset, but an inexact match with respect to the instance from the validation split of the WNLI dataset (Example 1))

These completions can be replicated using the same settings provided in the paper.

As you can see, different labels lead to tailored completions, addressing your concern about false positives. We included additional details with more examples in Appendix B in the updated version of our paper to further emphasize the importance of label inclusion during the replication process.