Measuring Non-Adversarial Reproduction of Training Data in Large Language Models

We thank the reviewer for their careful evaluation of our work and the constructive feedback.

In section 6, you mentioned that it's hard to distinguish reproduction of common idioms from problematic memorization. Is it possible to estimate how much of the overlap is problematic? Cause sometimes citing a known source is not a problem, so that may not be considered a problematic reproduction.

This is a great question. Determining whether memorization is problematic is highly context-dependent and varies based on the nature of the reproduced text (e.g., reproducing copyrighted material or private information raises different concerns than reproducing Wikipedia content). We believe most of these considerations are outside the realm of science. However, our analysis of reproduction at longer thresholds (e.g., in Figure 3) and long-tail events demonstrates that models can produce extended verbatim copies that are likely to be problematic in many contexts.

You have tested on temperature of 0 and 0.7, both are low temperature. Can you add experiments on temperature higher than 1 to see the reproduction rate under high temperature?

We aim to explore chat models in production setups, which are typically used with temperature between 0.7 and 1.0 (to the best of our knowledge). Hence, we do not consider temperatures above 1.0, and do not study more values between 0 and 1 due to the high inference cost. However, based on the experiments in App. B.1, we do not expect large differences.

You have tested two system prompts in section 4, can you test with more prompts?

We wanted to study two cases, i) a typical assistant prompt to see how reproduction behaves in real-world deployments of chatbots (e.g., Claude or ChatGPT), and ii) an extremely specific prompt to understand what the largest possible reduction of reproduction via prompting could be. We do not explore different instantiations of those two settings because they suffice to show a large difference and inference is costly.

In the human-written text dataset, how do you make sure that the human-written texts are not actually generated by an LLM? Cause human may use LLM to generate these texts.

This is a great observation, and we can indeed not rule out that a small fraction of humans partially relied on LLMs. However, as we discuss in the general response, this does not devalue our results (but makes them even stronger).

Thanks for your response, I'll maintain my already positive score

We thank the reviewer for their constructive feedback.

The evaluation results might be biased because the authors cannot access the real training dataset of evaluated LLMs.

As we mention in the experimental setup, matches against AuxDataset indeed provide only a lower bound on the actual reproduction from models’ training data. However, this means that the actual amount of reproduction is at least as large as what we report, thereby supporting our results.

For example, for the prompt mitigating part, the authors do not demonstrate which dataset are they using.

As mentioned in Sec. 4, “we only evaluate a subset of all prompts; see Appendix A.4 for details.”, which in turn states “We do not evaluate the mitigation strategies for WritingPrompts, ELI5, and book/movie reviews due to high inference costs, but consider all other tasks in Table 1.”. The rebuttal revision also contains our code, including a script to create mitigation prompts from original prompts, and a convenience bash script to obtain LLM generations for those prompts.

And since they can use WildChat or LMSYS-Chat-1M, what is the motivation for collecting a new dataset?

Our goal is to provide sound quantitative and qualitative insights. This requires a carefully controlled study setup. However, WildChat or LMSYS-Chat-1M are effectively uncurated. Hence, they allow us to show that non-adversarial reproduction exists in the wild, but nothing else! For example, we find that reproduction heavily depends on the task—but labeling the tasks of WildChat/LMSYS conversations is prohibitively expensive. Prompts from WildChat/LMSYS also do not reasonably allow us to do controlled comparisons with human baselines.

The length of substrings that are used to calculate the overlap rate is strange.

We made the motivation for our overlap rate threshold more clear in the updated paper and general response. We are happy to answer all remaining questions.

How will the results change if changing the threshold?

This can be read from the distribution plots (e.g., Fig. 3a). Every x-axis value is a threshold, and the y-axis is the fraction of generated texts that contains a reproduction at or above that threshold. We also added the new Fig. 10 which provides even more fine-grained details.

Why do the authors not consider open-source LLMs where they can know which datasets are used for training? For example, in the Membership Inference Attack area of LLMs, researchers usually use Pythia and the Pile dataset.

We explicitly focus on LLMs that were fine-tuned to be conversational and not leak data, and which are used by millions of users (where reproduction has a high impact). Unfortunately, there are currently no models with fully known training datasets that match those criteria (or are comparable).

We thank the reviewer for their reinforcing feedback and thoughtful comments.

Can the authors reason why the 50-character limit beyond simply the argument that non-adversarial prompts reproduce less training data?

We updated the paper with a clearer motivation for the 50-character threshold; see the general response for more details. We are happy to answer any follow-up questions.

I'm not convinced that the 50-character limit is strong enough to cause issues for LLMs reproducing training data.

Besides the clearer motivation for 50+ character snippets, we also report distribution plots for every possible character threshold (e.g., Fig. 3 and 5). We further included a new Fig. 10 that contains details for every individual model.

Can the authors provide more details on how their manual prompts were created? Were they crowdsourced, or written by the authors themselves?

If not mentioned otherwise, all prompts were created by the authors (manually). We updated App. A.1 and Table 1 to make this more explicit. Our updated submission also contains our code (including raw prompt data) and a link to the full dataset.

Were they sourced from how authors themselves commonly use LLMs, or were they thought up in bulk at once? Were there efforts made to categorize them into a variety of prompts (beyond the three broad categories used in the paper), or maybe efforts made to check this variety after the prompts were created?

We started with the three text types (creative/expository/argumentative). We then brainstormed tasks that are i) based on real-world LLM usage (of the authors and other people on the internet), ii) covering a diverse set of prompts, iii) roughly equally distributed over text types. For each task, we invented concrete prompts in a batch. Our original set of prompts had task pairs that turned out to be very similar (e.g., Statement of Purpose and Motivation Letter); we replaced one instance of each pair to increase diversity/coverage.

We thank the reviewer for their thoughtful suggestions and questions.

Only exact matches are considered, excluding reproduction of close matches with a low hamming-distance or reproduction of semantics.

We agree that extending our study to a non-verbatim setting is interesting future work. However, we note that this is challenging due to the large cost of finding fuzzy matches in 10TB of internet data.

The results are harder to interpret due to the possibility that a large number of reproductions by both humans and models is not captured by using AuxDataset. Perhaps the extent of the problem could be estimated by running the tests on a dataset that is expanded with additional sources and comparing the resulting numbers to the current ones.

AuxDataset indeed only provides a lower bound for reproduction. While we thank the reviewer for their constructive suggestion, we think there is no feasible approach: AuxDataset is already a 10TB snapshot of the Internet, and most publicly accessible sources overlap heavily with AuxDataset.

The selection of 50 character length seemed insufficiently motivated. Especially since it is different from the prior work and results in both memorised and unoriginal phrases being included.

We provide a clearer motivation in our general response and the improved discussion in the paper. We are happy to provide further justification if desired. We also note that we report every possible number of characters in our distribution plots (e.g., the x-axis of Figure 3a).

Are strings of length 40-60 (Line 45) or 50 (Line 129) considered?

We define overlap rates in terms of strings of length at least 50 characters. The updated paper avoids the misleading statement on Line 45.

Do the reproductions occur in similar contexts to the originals?

This is a very interesting question. Unfortunately, AuxDataset does not allow us to retrieve the context of a snippet (efficiently) because its implementation is optimized for efficient lookups (see Nasr et al., 2023 for details).

Lines 162-164 - what is the filtering procedure for the human-written text for it to not appear on the internet? If it is filtered, how did plagiarism appear in the human-written IMDb reviews?

We source the human-written text from May 2024 or later, which ensures the text is neither in any model’s training data nor in AuxDataset. We do not perform any other form of filtering to avoid introducing biases. Plagiarism appears if, for example, an IMDb review submitted in July 2024 copies a review from 2002. The review from 2002 is in AuxDataset, hence the July 2024 review has a large verbatim match with AuxDataset.

Figure 1 could be improved by adjusting the colour scheme and ensuring the readability of the small text.

We thank the reviewer for this input and updated Figure 1.

Thank you for the response - it answered most of my questions, and I will raise my score.

Re: original context of reproductions - I wonder whether these could be retrieved with the implementation from Nasr et al., 2023 by iteratively searching for all characters that could expand the current match.

We thank the reviewer for their constructive feedback and contextualization of our work.

50 character cutoff may overestimate regurgitation: The authors acknowledge this limitation, but it is difficult to differentiate based on character length alone whether the data is truly regurgitated off the internet or just due to it being a common phrase, especially when the length is around 50 characters.

We updated the paper to make the motivation for the 50-character threshold more clear, please see the general response for details.

Additional analysis to estimate a numerical proportional breakdown between these two categories would make the paper more rigorous. There is far less doubt about text reproduction vs. common phrases past the 100-150 character point.

We agree that an estimate of this proportion would be valuable. However, classifying reproduced text as short facts/language constructs vs. problematic text is a challenging task that deserves its own future work. In addition, we hope to provide more confidence through the new Fig. 10, which provides a detailed picture of reproduction for every possible number of characters (x-axes).

AI contaminated human baselines

We cannot rule out a very small fraction of contamination. However, as we discuss in the general response, this is not a problem (and might even strengthen our results).

I would find it interesting if you can also evaluate the reproduction length distribution of human data known to be mostly free from AI contamination, i.e. before the widespread release of LLM assistants.

While using such human baselines would be ideal, it is unfortunately practically infeasible. The only way to do this would be with a snapshot from the Internet from before the first LLM release. However, we cannot obtain such a snapshot. Even if we could, this snapshot would severely underestimate reproduction of LLMs (which were trained on much more recent text).

How fast can you check whether a given generation is in AuxDataset? Curious if we can reduce the probability of regurgitating very long text by doing a check against an internet proxy dataset at inference time.

This is an interesting suggestion. While lookups in AuxDataset require several TB of RAM and a long startup time, the latency per inference request is modest. The whole process could be made much more efficient (at the expense of a small false-positive rate) by using a bloom filter. One issue is that this form of response filtering enables users to directly test if a given piece of text is in the model’s training data (see, e.g., Debenedetti et al., 2023), a capability that model providers likely don’t want to expose. Nevertheless, some services employ a form of filtering (e.g., Github Copilot).

It's very interesting that the system prompt you used below reduces reproduction length. Why do you think this works? Did you try any other system prompts outside of this and the Claude prompt? Is it because the model can internally model a circuit to determine the probability of text being in its training data?

We thank the reviewer for this food for thought. First, we did not explore significantly different system prompts beyond what we report, because we only want to highlight that naive prompting does not mitigate non-adversarial reproduction. Second, we do not have a clear hypothesis or evidence. But this would be very exciting future work.

We thank the reviewer for the rigorous assessment of our work.

which refers to the overlap between LLM outputs in response to natural prompts (mainly writing prompts)

We first want to highlight that we explore 15 different tasks in 3 diverse categories, and report all results balanced over tasks and categories. Hence, while there are 1000 prompts for the WritingPrompts (or ELI5) task, they count as much as (for example) the 20 prompts of the satire task.

The task is limited to writing. I suggest the authors consider extending it to other tasks.

Our tasks are text-based because LLMs are trained on text and hence only reproduce text. But we make sure to use diverse tasks, from open-ended creative writing to strict formal recommendation letters. We are happy to explore more abstract tasks that can yield reproduction if the reviewer has any suggestions.

Generally, open-ended writing tasks are more likely to lead LLMs to recite memorized training data.

We find the opposite to be true, i.e., open-ended writing tasks are less likely to lead LLMs to recite memorized training data (see “Expository writing elicits the most reproduction.” in Sec. 3, or Figures 3, 4, and 6).

The authors’ conclusion seems somewhat obvious, as LLMs are explicitly fit on internet text. Intuitively, LLMs are more likely to produce text resembling their training corpus than humans.

Our better motivation in the rebuttal revision and general response should make the significance of our findings more clear. Moreover, we would like to ask the reviewer for the reasoning behind their intuition, as we find it not immediately obvious (humans, after all, are also “trained” on human text). Either way, our study does the work to support this intuition with sound empirical evidence.

Building on the first point, the authors propose using prompt design to mitigate overlap. However, the method and its underlying principles lack significant innovation.

We do not propose any defense against non-adversarial reproduction. Instead, we include a study of prompting as a mitigation precisely because it is an obvious strategy and highlight that this simple approach is insufficient. We are also happy to cite existing work that uses prompting as a defense against reproduction, as we are not aware of any.

The authors appear to conflate reproduction of internet text and training data. These are not equivalent, as the training data depends on the model's degree of fit. Especially when using a simulated dataset, this discrepancy may be amplified.

We read this feedback as “An LLM’s output might match internet text in AuxDataset just by chance and not because of memorization.” However, if most matches in our paper were due to chance, we would expect overlap rates between LLMs and humans to be very similar (which they are not).

I suggest that the authors consider using some training data detection methods (e.g., $1$ ) to assist in identifying training corpus when exploring reproduction of training data.

Membership inference for LLMs suffers from severe issues and cannot yet reliably prove that a model was trained on some particular data (e.g., Zhang et al., 2024) and is not directly relevant for our study. We hence rely on the established AuxDataset approximation by Nasr et al., 2023.

We thank the reviewer for their thoughtful comments.

The current text is ordered as a set of subsections with a verbatim description of experimental steps. The presentation lacks focus on the main contributions of this research.

We have updated the paper to make our main focus more clear; please refer to the general response for details. We are happy to answer any remaining ambiguities. Finally, we note that each paragraph in Sec. 3 and 4 corresponds to one finding of our study.

Even with this measure of choosing content after LLM cut-off date and content which is not part of AUXDATASET, how is it confirmed that the content taken from Reddit is not LLM generated? Is it not possible that an LLM might have been used to generate it?

In short, even if a few rare instances of human-written baselines contain some LLM-generated text, this does not weaken our comparison (it might even make it stronger). We discuss this more in the general response.

The paper mentions, “We find that LLMs reproduce more existing data than humans, except when humans do blatant plagiarism.” I might have missed it in the text, but it would be great to have some clarification regarding how this is controlled? For example, given a prompt like “Write a tutorial about setting up an Nginx server.”, humans might be prone to copy data verbatim from top-ranked articles on a search engine. There is a discussion in Line 404 about IMDb reviews, but what measures were taken for Reddit content?

We can indeed not rule out plagiarism in human-written responses for the Reddit content (WritingPrompts and ELI5 tasks). However, we did perform a manual investigation (as for IMDb reviews), which did not reveal any significant cases of blatant plagiarism (as opposed to IMDb reviews). This matches our Fig. 5a/c, where human texts contain fewer reproduced snippets of any length compared to LLMs, and Fig. 6 where the mean overlap rate for humans on creative/expository tasks is below 2% (and the median is 0).

We thank the reviewer for their kind words and constructive comments.

Frequency of Reproduced Sequences: The paper could benefit from clarifying how often the reproduced sequences appear within their training data proxy, AUXDATASET. [...]

This would be very interesting. Unfortunately, there is no way to reliably estimate the frequency of snippets in any model’s training data. First, model providers might perform deduplication of their training data, such that a snippet’s frequency on the internet might not match its frequency in the training data. Second, AuxDataset combines potentially overlapping sources (e.g., multiple copies of Wikipedia in different datasets). In both instances, we can reliably approximate whether a snippet is part of the data, but not with which frequency.

Justification of 50-Character Threshold: The choice of a 50-character threshold to define reproduced sequences is not fully justified. [...]. Further explanation would help readers assess whether this threshold adequately captures the difference between common phrases and more problematic reproductions.

We hope the explanation in our general response and the improved discussion in the paper provide a better justification. We are happy to answer follow-up questions.

Data in Figure 2(b): Figure 2(b) appears to have only partial bar plots for some models (Llama and GPT), making the comparison across models less robust. Or am I missing something here?

This is correct and a consequence of our post-hoc analysis of WildChat and LMSYS-Chat-1M. For example, WildChat consists solely of ChatGPT conversations and thus only has generations for GPT models. This is okay; the goal of Fig. 2(b) is to highlight that non-adversarial reproduction also happens in the wild (outside of our controlled experiments), not to provide a robust comparison between models.

Overall, I am constantly battling between thinking that 50 characters is too less, and then seeing the argument that these reproduction rates are much higher than humans. [...]. Would a human with a passage (RAG style reading comprehension) be a better baseline? [...]

We thank the reviewer for this in-depth thought; however, there are subtleties that break this analogy. For LLMs, we explicitly focus on copying/reproduction from the training data, not from the prompt. However, humans copying text from a provided passage corresponds to LLMs copying from their prompt/context. A better analogy are humans who are able to recall and lookup information from memory—which is exactly what we measure.