Elephants Never Forget: Testing Language Models for Memorization of Tabular Data

We thank the reviewer for the detailed review and many helpful comments.

• “This work does not provide strong evidence supporting the validity of their proposed approaches.”

With regard to the validity of our tests, please see the global comment ‘The validity and relevance of our tests’.

In addition, note that we have been careful to use (1) datasets that are public and overwhelmingly likely to have been used in pre-training, and (2) datasets that are very unlikely or impossible to have been used in pre-training. There is a clear difference in the results for these two different kinds of datasets. We also intentionally include FICO as a dataset where there is significant public discussion of the data and some of its properties, but where the data table itself has not been made public. As might be expected, on this dataset our tests detect prior experience but fail to detect verbatim memorization.

• “I can’t understand the purpose of having these many different testing approaches”

We acknowledge that the flow of the paper can be overwhelming due to the many different proposed approaches. Under the limitations we impose upon our testing framework, we believe it is impossible to design a single test that will work perfectly under all possible circumstances. Our design philosophy therefore is to design a battery of tests that offer empirical evidence of memorization in concert. Under stricter assumptions, a smaller number of tests may likely be sufficient, but making stronger assumptions would compromise the utility of our tests by working either only with fewer LLMs or fewer tabular datasets. Overall, we believe that the proposed tests allow us to provide a nuanced picture of different types of pre-training contamination.

• “The descriptions of the testing approaches are vague and not rigorous.”

We agree that there is room for improvement and will rewrite the sections describing the different tests. For details on the different tests, please also see Supplement Section E which gives the prompt structures of all the tests, and the revised Supplement Section F which provides the test results in a more explicit form.

• “I am not sure whether those datasets are commonly used to benchmark the language model. Thus I am not sure whether the findings are important”

Yes, the datasets in our paper have been used to benchmark language models. Please see the papers referenced under ‘The relevance of our work’ in the global comment.

We thank the reviewer for the detailed review and many helpful comments.

• “raw accuracies for Feature Names, Feature Values, and Header Test are not provided ”

We empirically found that the language models either (1) correctly list all the feature names, or (2) completely fail to list the feature names. The same holds true for the Feature Values test (third row of Table 3) and also for the header test. This is why we decided to depict the test results qualitatively. We agree that we did not make this clear enough in the paper. The revised Supplement Section F now depicts detailed results of the different tests.

• “the authors use feature distributions to examine learning, but memorization can also result in a high similarity of the generated data's feature distributions to the original data. Learning is defined as the model's ability to perform tasks in the current paper, but task performance is heavily affected by memorization and may fail to reflect true learning. Even with considerable discussion, the paper does not seem to arrive at a conclusion about how learning can be clearly assessed.”

Great point. We find it important to introduce the learning/memorization distinction to make clear that there are many different tasks that the model might be able to perform with the data that are not indicative of memorization (see also ‘The distinction between Knowledge, Learning and Memorization’ above). We are reasonably confident to have identified cases where the LLMs have learned, but not memorized. The reviewer asks whether we are able to test for learning under the assumption that there is memorization. This is a very interesting question (and also a question that is hard to answer). With the tests that we propose in this paper, we do not claim to be able to do this.

• “Evaluation of memorization needs to take the nature of data fields into consideration. [...] For numerical values, it may be more reasonable to measure the relative distance from the predicted value to the true value than using exact match (perhaps in a similar vein as the "first token test" in the paper but more principled). Under the current evaluation protocol, it is likely that datasets containing more easy fields are more likely to be judged as memorized. To compare the degree of memorization across datasets, it seems necessary to perform some kind of "normalization" before measuring memorization, for example, selecting a fixed number of categorical and numerical fields from each dataset. Results in Table 3 could suffer from this limitation as well.”

This is an interesting observation, but we tend to disagree. In fact, it is precisely the exact matches between rare numerical values in the data and the generations by the LLM that allow us to develop valid tests for memorization. The fact that tabular datasets often contain such values (as opposed to free-form text) is what makes tabular data such an interesting testbed for memorization (and this is also why the results in Table 3 are valid). For a dataset that only contains easy fields such as categorical variables with few different values, it would be very hard to distinguish a powerful learner and memorization. Measuring the relative distance from the predictive value to the true value could result in a test for learning, but would be unlikely to result in a valid test for memorization.

• “Evaluation of memorization needs to be evaluated separately for the training and test split. ”

Very good point. The revised Supplement Section F now contains separate evaluations for the datasets where separate train and tests splits are available. We do not find any significant differences.

• “Connection between memorization and downstream performance is not reliably established. [...] It would be much better to solicit new test sets for the tasks to use in evaluation, which can be used to show exactly how much performance gap is caused by memorization.”

The SOTA accuracy for Kaggle Titanic using ensembling is approximately 0.85 (see https://www.kaggle.com/code/pliptor/how-am-i-doing-with-my-score/report). GPT-4 achieves an accuracy of 0.98. We think that this is fairly indicative of memorization. We agree, however, that the approach in Section 5 is only a first step, and that one could think of more advanced evaluation strategies (such as the one suggested by the reviewer). In general, however, there are many possible downstream tasks, and it is not possible for tests of memorization and prior exposure to consider (or even be relevant) for all of these. Because of this, we believe it is important to have tests that are also reasonably task-agnostic.

We thank the reviewer for the detailed review and appreciation of our paper. We also thank the reviewer for the minor comments that we will incorporate into the paper.

• “The paper only studies ChatGPT-3.5 and 4. ”

We have now extended our code and prompts to work with arbitrary language models (some adaptation of prompts is required between base models and chat models, and the capabilities of chat models to follow instructions vary). The revised Supplement Section F contains results with Llama2-7b and Llama2-70b.

• “Including an LM trained on a known dataset would allow us to verify the used methods.”

We agree that this is an interesting direction in which our work could be extended. Unfortunately, the LLMs for which the training data and schedule are transparent are rather small (<100b Parameters). At the same time, memorization is known to occur especially for very large models. This is evident, for example, in Supplement Table 4, which shows that GPT-4 has memorized significantly more than GPT-3.5. It is also evident in the Figures in the revised Supplement Section F, where it can be seen that Llama2-7b is the model that has memorized the least.

• “the work is mostly relevant when we consider closed-data models w/ a black-box API access. “

We completely agree. We also believe that this is a very important and relevant scenario (see also ‘The validity and relevance of our tests’ above).

We thank the reviewer for the detailed review and many helpful comments.

• “My main point of criticism is that the paper feels a bit like a collection of remarkable examples and the analysis largely confirms known concerns/behavior of LLMs. ”

We agree that dataset contamination is a general concern in LLMs, and that it is widely known that LLMs can memorize their training data. Despite this, many papers at the top ML conferences evaluate LLMs on datasets that potentially are part of the LLM’s pre-training corpus without performing any systematic tests for memorization (see ‘The relevance of our work’ above) . We suspect that a main reason for this is that while contamination is a known concern, there are few tools available that allow researchers to systematically test for memorization and other forms of prior contamination within the timeframe of a single research project. With this paper, we attempt to develop testing approaches that will allow future research to address this question in a systematic manner.

• “we also identify a regime where the language model reproduces important statistics of the data, but fails to reproduce the dataset verbatim”

Here we are referring to the behavior of GPT-3.5 on the California Housing dataset. On this dataset, the samples provided by the model match the correlations in the training data (Figure 3). At the same time, the average n-gram match between the samples and the training data is only 3.1/10 (Table 3) and there is no evidence for verbatim memorization except for the header of the dataset (Table 1).

• “the interpretation of results often remains speculative.”

We would not call our results speculative. We don’t think that there is any plausible way in which the model could reproduce tabular datasets verbatim (Table 1) or with precisely matching statistics (Figure 3) without having seen those datasets during pre-training.

• Figure 3: “Why are some results with gpt-3.5 and some with gpt-4?”

Figure 3 depicts selected examples. The full results, for both gpt-3.5 and gpt-4, are in Supplement D.

• “"​​An important result of our investigation is to identify a regime where the LLM has seen the data during training and is able to perform complex tasks with the data": Don't LLMs behave as expected on some data and not on other? How does this work help to control the behavior of LLMs?”

We don’t know what the reviewer considers the expected behavior of LLMs to be. We would interpret our results as showing that there is a fair degree of variation in LLM behavior, both across different models, and also across datasets for a single model.