Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon

Weaknesses:

The predictive model's performance is relatively poor (precision ~0.5, PR AUC ~0.5), despite including continuation perplexity as an input feature. This raises questions about the practical utility of the identified factors in predicting memorization. The heavy reliance on continuation perplexity for every category (see Figure 5.), which is closely related to the definition of k-extractability, makes it difficult to assess the independent predictive power of other factors.

The goal of our taxonomy is not to directly predict memorization, but to analyze the factors that make it more or less likely. While our taxonomy might be useful in future work on predicting memorization, our linear model allows us to control for various features, especially those that are otherwise dominate other factors, such as perplexity.

No clear progress in understanding the memorisation of rare sequence. While the paper identifies recollection of rare sequences as an important phenomenon, particularly as models scale, it provides limited insight into the underlying mechanisms. This gap is particularly notable given the paper's emphasis on understanding different types of memorization.

Our main finding in the case of rare sequences is that, excluding simple templating behaviors such as repetition and incrementation, most are likely cases of more complex templating behaviors mixed with recitation. In particular, the rare biblical passages and legal texts appear to be high frequency sequences that are either reindexed or have some unusually translated words that can be memorized. We specifically exclude the possibility (based on Dankers et al. (2023)) of memorized rare sequences simply being rare retokenizations of common sequences, by studying surface level string similarity.

When introducing the taxonomy, early concrete examples of each category would significantly improve understanding. The paper should also better highlight the distinction between intuitive notions of memorization and the technical definition of k-extractability used in the study. This could help the reader understand why the reconstruction phenomenon (where sequence outside of the training set could be predicted perfectly) fall in the scope of the study of memorization.

Thank you for these recommendations. We are currently adding them to a revised version.

The study could benefit of including reference to a broader set of references such as the study of mechanistic interpretability and training providing more insights on how and when models become able to predict simple sequences. See for instance "In-context Learning and Induction Heads", by Olsson et al.

It is true that repetition could be implemented by mechanisms similar to induction heads. Are there citations to mechanistic work on numerical incrementing that you also feel we could include?

The corpus statistics are not broken down into prompt/continuation/full sequence. This could enable the isolation of sequences with a frequent prompt but infrequent continuation, or the opposite for instance. The paper doesn't clearly state which one of the three is used for the corpus statistics.

We use the continuation currently, but would happily implement a more granular analysis for the camera ready. (These experiments would take too long for the rebuttal period.)

If I understood correctly, the semantic similarity measurements are made between sequences of length 64 (or 32) tokens (the memorized/non-memorized samples), and the 2049-token sequences from the Pile. This length mismatch could introduce heavy distortion as even if the small sequence is included in the large sequence, it is not clear that the cosine similarity of their embedding would be similar.

We agree that there would be semantic differences between similar sequences of different lengths, and we do control for that. We specifically look only at semantic similarity between the same length (64 tokens) as the full sequence. We consider every 64-token sequence available from the pile. We clarify this in the updated paper.

Thank you for your comment! It provides insightful clarifications.

The goal of our taxonomy is not to directly predict memorization, but to analyze the factors that make it more or less likely. While our taxonomy might be useful in future work on predicting memorization, our linear model allows us to control for various features, especially those that are otherwise dominate other factors, such as perplexity.

I understand that the goal is not direct prediction of memorization. However, would you agree that the performance in predicting memorization is a good proxy to estimate how much the factors you introduce capture the memorization phenomenon? In other words, if the prediction performance is poor, it's a sign that there are unexplained variability not contained in the input factors (note that this variability could be irreducible noise, but this needs to be discussed). Conversely, if the prediction performance is good, it means the factors identified capture the main variable that condition memorization.

Our main finding in the case of rare sequences is that, excluding simple templating behaviors such as repetition and incrementation, most are likely cases of more complex templating behaviors mixed with recitation. In particular, the rare biblical passages and legal texts appear to be high frequency sequences that are either reindexed or have some unusually translated words that can be memorized. We specifically exclude the possibility (based on Dankers et al. (2023)) of memorized rare sequences simply being rare retokenizations of common sequences, by studying surface level string similarity.

Thank you for clarifying this contribution. I recognize the value of ruling out a hypothesis from the litterature. However, I maintain that given that recollection is defined as everything that is not recitation nor reconstruction, this category doesn't provides deep understanding of the sequences it contains. 'more complex templating behaviors mixed with recitation' gives a fuzzy intuition (e.g. when is a complex templating behavior so complex/specific that it becomes a recitation?) for what could be in here. It could still be valuable to guide further hypothesis.

It is true that repetition could be implemented by mechanisms similar to induction heads. Are there citations to mechanistic work on numerical incrementing that you also feel we could include?

I am not aware of work specifically studying numerical incrementing.

We agree that there would be semantic differences between similar sequences of different lengths, and we do control for that. We specifically look only at semantic similarity between the same length (64 tokens) as the full sequence. We consider every 64-token sequence available from the pile. We clarify this in the updated paper.

Thank you for this clarification. This adresses my concern on this point.

The template detection approach appears oversimplified. For instance, only basic patterns of "repeating" and "incrementing" sequences are considered, potentially missing more complex templates. The duplication counting relies on exact matches without accounting for semantic similarity or near-duplicates (e.g. slightly modified code or text passages).

We fully agree that more complex templates are of great interest. In fact, it is one of our core motivations; while a majority of non-recitation examples can be specifically described as either repeating or incrementing, as is made clear in our findings, our goal in qualitatively analyzing recollection was to identify possible other more complex templates. See Section 4.3 for our resulting qualitative analysis, which conjectures that most cases of recollections are cases of extrapolating from slight differences in translation or indexing systems in a unique instance of biblical or legal documents. Such instances can be described as a mix of recitation and reconstruction.

One useful finding is a rejection of a hypothesis (based on Dankers et al. (2023)) about recollection: that a significant amount of rare sequences are simply retokenized versions of more common sequences. By quantifying string similarity, we reject the retokenization hypothesis outright (App B) and instead posit the templating idea above.

The paper insufficiently compares its taxonomy against existing memorization frameworks. For example, the relationship between these categories and counterfactual memorization, which is mentioned but not analyzed, deserves exploration. The advantages of this taxonomy over other approaches to studying memorization are not quantitatively demonstrated.

Our goal here is to analyze individual memorized sequences without relying on potentially intractable definitions of memorization, like those used at smaller scales to guarantee the counterfactual memorization definition. Our stratification of the reconstruction set in particular allows us to exclude many “memorized” samples that are not counterfactually memorized from our analysis of recollection.

The exact procedure for computing KL divergence in Fig 3 is unclear, and the methodology for computing perplexity scores used throughout the analysis lacks essential details. The robustness of results to different tokenization choices is not evaluated.

Do you have specific questions about the procedure for computing perplexity? We use the tokenizer that is native to the LM, and it is unclear to us how we might do otherwise. Perplexity is computed directly by the LM itself; although it is the only feature we use that draws directly on the LM’s behavior, we felt it was important to incorporate because it is so strongly predictive of memorization.

Can you provide empirical justification for this specific cutoff? How sensitive are your results to this choice?

Please see top level response.

Could you include statistical significance tests for the reported trends across model sizes?

Using a binomial significance test with p<0.01, we find that every difference between models is significant (with all p-values <<10^-10). We will mention this in our next updated revision.

Thank you for your thoughtful review. We are happy you see the value of such a taxonomy!

First, while the idea of categorizing memorization into three types sounds cool, the paper doesn't dig deep enough to tell us why we should care. Sure, they show that code gets memorized more than text across all categories - but why? And what does this mean for how these models actually work? How different types of memorization contribute to model capabilities. These are the kind of insights that would make the taxonomy actually useful, but they're missing.

Our taxonomic model is just a starting point for analysis, which might eventually involve the inclusion of specific features of interest for future projects. We do point to several insights provided by our more general setting in section 6.3. First, controlling for other factors, rare tokens actually make recollection more difficult—but has no significant effect for other memorization types, indicating that there are in fact cases where a model might have memorized a rare sequence, but the presence of individual rare tokens presented too much friction. We find that recitation is less supported by rare prompts compared to other types of memorization, likely because the recitation set is mostly composed of extremely long predictable sequences. Recitation is also slightly more likely if there are fewer common tokens in the continuation (based on P75 frequency), suggesting that it is difficult to memorize by conditioning on many common tokens. We also find that over the 5-duplicate threshold, the particular number of duplicates becomes unimportant, in spite of how important it is below that threshold; we posit that it takes a small number of exposures for a sequence to become easily memorized, but once memorized, it is rarely forgotten. We are updating our paper to highlight these speculations, which may be confirmed in future causal/interventional experiments.

In addition, the experimental setup is not convincing. For example, the experiments are conducted solely on Pythia models without validation of other popular models. And some of the key choices seem pretty arbitrary like picking k=32 for their memorization tests or saying "more than 5 duplicates" counts as recitation. Why those numbers? What happens if you change them?

Both the use of Pythia models and the use of k=32 are decisions driven by the availability of memorization data for these models. Unfortunately, other models have not released public memorization datasets, and no models have released public memorization datasets that use different definitions of memorization.

As for the particular duplicate count threshold, please see top level response (1).

What insights from previous work on memorization mechanisms support or conflict with these findings?

There are several connections we discuss in the paper. We compare to Dankers et al. (2023), who pointed to rare tokens as promoting counterfactual memorization; we find that this relationship is specific to the recollection category, which is likely to contain most counterfactually memorized examples. We also agree with the conclusion of Tirumala et al. (2022) that larger models memorize more sequences.

Thank you for your insightful review.

If the sequence duplicates more than five times, how do we know whether it is truly "memorized" or it is reproduced simply because the LLM learns its pattern?

To address the first weakness, we highlight that current work on memorization often involves the k-elicitation definition, which does not differentiate counterfactual memorization from reconstruction. Obviously, under some circumstances you want to limit notions of memorization to counterfactual cases, and under such circumstances you might develop a different taxonomy from ours, possibly one that becomes more granular about the types of counterfactual memorization. However, one advantage of our approach is that it is far more efficient than existing methods for evaluating counterfactual memorization, as it only requires one training of the model and does not require backwards passes on the data.

Why do we use more than five times duplication as the decision boundary for recitation and non-recitation. How is the hyper-parameter decided?

Please see top-level response.

Using a single threshold actually assumes an equality in difficulty for reciting every sequence.

One of our goals in describing the memorization behavior with a taxonomy is to disentangle the factors in “difficulty” of memorizing different types of sequences. By controlling for duplication, we show that factors like rare tokens have specific effects on the likelihood of recitation. (See fig 6)