Rethinking the Role of Verbatim Memorization in LLM Privacy

We thank the reviewer for their positive assessment of the paper's quality and clarity and for the feedback. We appreciate the opportunity to clarify our contributions and address the valid concerns raised.

While the work is very interesting and has a number of novel insights, I have my doubts about how much can one read into these results to come to these conclusions. For instance, the argument on 'Model size does not significantly impact chat extraction, while model quality is crucial' - this conclusion is a bit far fetched given the underlying experimental results... I am not sure if you can attribute the difference in extractability simply to 'better models', as there is a higher chance that the model is better, but there is also a higher chance that similar-looking sequence has been memorized due to its recurring appearance in the pre-training dataset. [...]

Our claim is not that model size is irrelevant, but rather than "model quality" (as measured by standard benchmarks like MMLU) appears to be a better predictor of chat extraction success than parameter count. However we agree that the pre-training data is a significant confounding variable that we cannot control here, but all the biographies are synthetic so we are certain that none were in the pretraining data of any of the models. In particular in Figure 3, we see that it is significantly easier to chat-extract IDs from Llama3-8B than Llama2-13B, whereas there is no reason to believe that Llama3’s pretraining data had a bigger emphasis on random sequences.

Some other conclusions have been previously established already e.g. 'Increased verbatim memorization does not imply more chat extraction'... or that 'IDs are more challenging to encode compared to other attributes'... So while I see that there is indeed a difference between the stages of training which makes some difference when discussing these insights, the overall principle is the same and we have already known about these phenomena for a while.

We agree there is a strong link with the knowledge extraction literature, which we have stated in the manuscript. But

• our contributions are new to the best even within the literature or
• highlights consequences of already-known facts from a privacy-centric angle.

Specifically, to the best of our knowledge, we are the first to show that:

• Verbatim memorization (pre-instruction tuning) does not correlate well with chat-based extraction (post-instruction tuning). But removing information from a verbatim sense (see unlearning section, e.g. fig 6) removes chat extractability
• Information can become "un-extractable" verbatim after instruction tuning, yet remain encoded and extractable through other means (chat or after realignment), creating a false sense of privacy.
• Model quality, not just size, is a key factor in chat-based leakage.

Again, our contribution also lies in being the first to systematically investigate these effects from a privacy perspective, as the contributions have significant implications for how we audit them for privacy. We will refine our framing to make it clear that our contribution is this comparative, privacy-centric angle.

I also had concerns regarding the dataset composition... Llama models are likely... pre-trained almost exclusively on data oriented towards the US... I am not certain if this is likely scalable beyond this geography... From my understanding none of these concerns were addressed or discussed in the manuscript at all, significantly limiting the potential impact of the findings.

The reviewer is correct; our synthetic dataset is US-centric, mainly because it is based on the dataset from Zhu & Li. We fully agree that this raises questions about generalizability. How these extraction dynamics play out with data from geographies or domains that are underrepresented in the pre-training corpus is an open question that we do not tackle. We can add a dedicated discussion to our limitations section to address the dataset's geographic bias and its implications for the scalability of our findings.

However the ID attributes that we add for instance are totally neutral, and we respectfully disagree with the reviewer that it significantly limits the generalizability of our claims: we care about the results in relative terms rather than e.g. the absolute extraction rates. Which statement in particular does the reviewer think it impacts?

I've skimmed the original manuscript and it seems to me that some data was generated using Llama models, further amplifying the bias and potentially reproducing some part of the pre-training data.’’:

We would like to point out that we use the BIO-S dataset which does not contain any data attributes generated from Llama models, and that the biographies are created by assempling attributes randomly.

We thank the reviewer for their feedback. We appreciate the suggestions for extending our experimental setup, which we believe would strengthen the work. We address the main points below.

The main contributions summarized in the introduction are mainly findings from the experiment. I think these findings have been showed by previous work in a similar knowledge extraction setting (Allen-Zhu & Li, 2024)

We agree there is an overlap with the knowledge extraction literature. But our contributions as stated are new, specifically:

• Verbatim memorization (pre-instruction tuning) does not correlate well with chat-based extraction (post-instruction tuning). But removing information from a verbatim sense (see unlearning section, e.g. fig 6) removes chat extractability
• Information can become "un-extractable" verbatim after instruction tuning, yet remain encoded and extractable through other means (chat or after realignment), creating a false sense of privacy.
• Model quality, not just size, is a key factor in chat-based leakage.

These key new insights don't just imply that chat extraction differs from verbatim memorization, but that this disconnect creates an issue for privacy auditing. When standard audits (like verbatim extraction or loss-based MIAs) fail after instruction tuning, they can give a false sense of security, even though the information is still very much present and extractable via other means. Our work is the first to systematically demonstrate this failure mode and map the factors that influence it, which we believe is a significant and practical contribution to the privacy community.

It seems at the SFT stage, it starts from epoch N checkpoint from the pre-train stage and SFT for a fixed 10 epochs. Does the 10 SFT epoch lead to overfitting problem? Can you show the verbatim extraction and chat extraction accuracy at each SFT epoch starting from each pre-train epoch checkpoint?

That's a fair point. We fixed the SFT epochs to 10 based on preliminary runs where performance plateaued, but we agree that showing the full learning curve would provide a clearer picture and rule out overfitting. We ran the experiments and have the curve ready to be added: the verbatim memorization does not decrease through the 10 epochs of SFT, it stabilises after some instabilities at the beginning.

What would happen if during the SFT stage, you mix the pre-training data and SFT data together? So the model is trained on both type of data with the training objective. Would this result in better chat extraction accuracy and at the same time preserve the verbatim accuracy?

From a practical standpoint, mixing the data would likely be a powerful strategy to improve chat extraction (by reinforcing the knowledge during instruction-tuning) while simultaneously preserving verbatim extraction (by maintaining the next-token prediction objective), but we don’t see it as a realistic thing that practitioners do during SFT. But in the same vein, we show in e.g. figure 5 that “re-aligmenent” restores a lot of verbatim extraction, which leads to a similar conclusion to the one suggested by the reviewer.

As the paper show it lead to low chat extraction accuracy, I wonder what if instead of SFT using RLHFlow, the experiment is conducted using reinforcement learning such as RLHF...

We chose SFT as our focus because it's a foundational and universally applied step in aligning chat models, and show in particular that verbatim extraction does not work well after this, which is one practical take-away from our work. We hypothesize that RLHF would also weaken verbatim extraction, but this would need to be tested.

We thank the reviewer for their feedback. We appreciate the constructive criticism regarding the paper's clarity and structure, and we will address each point to improve the manuscript.

Writing, Structure, and Clarity

We will perform a thorough revision to fix all typos, improve the flow, and clarify ambiguous points. Unfortunately we don’t have the opportunity to update the manuscript during the rebuttal, but we commit to doing it for the camera ready version.

I find the related work quite unstructured... Sometimes certain points of reference are not well articulated and hard to follow. For instance, the argument around DP in lines 50-53 is not clear, nor is the argument around MIAs in 193-194. [...]

We will restructure the related work section for better narrative flow.

• On DP (lines 50-53): DP provides guarantees by bounding the total information encoded in model weights for each sample. Our work shows that even when information is encoded (or verbatim memorised) its practical extractability via chat is a separate issue. We wanted to highlight the choice of our work to be “beyond verbatim“, looking beyond the classical privacy angle (such as the one from DP) that deals only with information encoding.
• On MIAs (lines 193-196) Our point is to introduce a nuanced, three-level view of memorization. The first level, which we show in Figure 2a, is the model's lower loss on training data—the very signal that loss-based MIAs exploit. This a necessary but insufficient indicator of extraction risk. The second level is direct verbatim extraction (Figure 2b), a stronger form of memorization. The third is chat extraction (Figure 2c), where information is leaked conversationally. By presenting these three distinct levels, we argue that privacy audits must look beyond simple loss-based signals or MIAs, but even beyond verbatim.
• Figure 4 Setup: The details are mentioned in Section 3.2 but we will clarify the setup in the caption and text. All experiments in Figure 4 use the Llama2-7B model, are evaluated on the tail set (B1), and follow our standard protocol of 20 unsupervised epochs followed by 10 instruction-tuning epochs.
• Unlearning Implementation: We will clarify the unlearning process. To test if removing verbatim memorization also removes chat extractability, we modified the training process for a specific subset of our data (the 'unlearning set'). For the 'ID' attribute in this set, instead of minimizing the standard cross-entropy loss, we trained the model to achieve a target loss of 2 by minimizing abs(loss - 2) only for the ID tokens, and normal training for the rest. We chose this target because a loss of ~2.3 (ln(10)) is what one would expect from a model randomly guessing a digit from 0-9. A high loss of 2 thus removes verbatim extraction. To prevent the model from forgetting how to process IDs entirely, we continued normal training on a separate 'regularization set'. We then evaluated how this targeted verbatim unlearning affected chat extractability on both the unlearning set and a heldout set.

I don't really understand the points made in lines 199-213 - why does this exact formulation matter if you measure extraction too ? This connects the model's training objective (minimizing cross-entropy loss) to verbatim extraction. It formally highlights that the expected number of queries needed to extract a sequence is exponentially related to the loss, whatever decoding scheme is used. While we measure extraction empirically, this formulation explains why low loss is a prerequisite for verbatim extraction, and why the loss tells you everything that you should expect for verbatim extraction. But we show that it does not tell much about chat extraction.

One major experiment that seems to be missing is the impact of going from greedy decoding (done to measure verbatim extraction) to stochastic decoding (done to measure chat extraction)... it's unclear whether the drop from 100% to a plateau well below that comes from the data not being extractable or from the stochastic decoding.

Our goal was to compare two realistic extraction: a verbatim extraction attack, where an adversary would have access to the logits and will likely use greedy decoding to maximize success, as well as a typical chat interaction through an API, which involves stochastic decoding. We acknowledge that the decoding strategy is a confounding variable when comparing the absolute extraction rates between these two settings.

However, the central claim of this part of our study is not so much about the absolute drop in performance, but more about the relative extractability of different attributes in a chat setting. Since the same stochastic decoding parameters were used for all attributes during chat extraction, our conclusion that some attributes are significantly harder to extract via chat than others remains valid.

On a related note, it might be useful to consider some related work on probabilistic extraction [1].

Thank you for this valuable reference, it is very relevant. We will incorporate it into our related work section, as it complements our discussion of extraction methodologies.

The paper only provides results for one dataset... it would make sense to also include results for another dataset (to see how it all generalizes, but potentially also in a setting more related to utility/encoding of factual knowledge).

We chose synthetic biographies to have precise control over the properties of private information, which was essential for our privacy-focused study. We agree our findings have broader implications for knowledge retrieval, as the reviewer notes. We will add a paragraph to the discussion.

Why the buckets (lines 145-156)?

The bucket system allows us to control for data repetition and augmentation, key factors in memorization. The tail (B1, seen once) serves as our consistent evaluation set for "rare" data leakage. The head (B10, seen 10 times with augmentation) is used for instruction tuning. This simulates a realistic scenario where a model is taught to answer questions about common knowledge, and we then test if this transfers to extracting rare, private information it was not explicitly taught to retrieve. This setup cleanly separates the data used for teaching the skill from the data we are testing for leakage. About the other buckets, we have results that show as expected that the extraction rate is an increasing function of the bucket number, but we did not find these results very important as we already have a lot of experiments, but we can add them to the appendix.

Line 354 "detecting training data splits":

This phrase refers to a common, basic form of privacy audit where one simply checks if the model regurgitates long, contiguous chunks of its training data verbatim. Our argument is that modern privacy evaluations must be more sophisticated, e.g. go beyond verbatim.

We thank the reviewer for their feedback. We appreciate their positive assessment of the paper's technical quality and clarity. We will address the main concerns below and will revise the paper to clarify these points.

The framing of the work, in my opinion, hurts its significance and originality... the actual questions tackled in the paper, the difference between verbatim memorization during original unsupervised training vs chat-based memorization after instruction-tuning, have been studied before (knowledge extraction literature, also noted in the paper).

One key insight is that verbatim memorization, a standard for privacy audits, is a poor proxy for the actual information leakage risks in instruction-tuned models. We will revise the introduction and related work sections to more explicitly highlight this distinction and the privacy-centric motivation of our study, but to the best of our knowledge the fact that instruct models have lost most of their verbatim memorization (among other things) is a new observation even outside of the privacy scope. See next answer for the clarified contribs.

Please take the rebuttal as an opportunity to further clarify how your work stands out from the existing literature on knowledge extraction in chat-based models.

Our work stands out by focusing on the disconnect between different forms of memorization and their privacy implications. Specifically, we show:

• Verbatim memorization (pre-instruction tuning) does not correlate well with chat-based extraction (post-instruction tuning). But removing information from a verbatim sense (see unlearning section, e.g. fig 6) removes chat extractability
• Information can become "un-extractable" verbatim after instruction tuning, yet remain encoded and extractable through other means (chat or after realignment), creating a false sense of privacy.
• Model quality, not just size, is a key factor in chat-based leakage.

These findings are important for effective privacy auditing methods for modern LLMs, and are new even outside of the privacy scope.

My complaint with this paper is that it only briefly touches on and never answers any of these questions in depth. What the paper actually answers is a question that has already been explored in the knowledge extraction literature, only translating the experiment to a privacy-specific dataset.

We agree that the questions posed are broad. Our goal was not to provide an exhaustive answer to all of them, but rather to demonstrate that the conventional approach to privacy auditing is insufficient and to map out the complex interplay of factors that a more robust audit would need to consider. We chose to provide a breadth of initial findings to motivate this. We will clarify the scope in the introduction to better manage expectations.

However:

• question 1 is tacked by comparing the pink bars in figure 5 (verbatim extraction after SFT) and Figure 2.c (chat extraction after SFT)
• Question 2 on the relationship between SFT and unsupervised finetuning data: we show that the SFT data need to include some unsupervised finetuning data to even be effective (fig. 4.c).
• Question 4: this is answered in fig. 4.a. And 4.b.

We are very much open to clarifying any of these points.

[...] For instance, comparing Figures 2(c) and 5, it seems that verbatim memorization of the final model extracts more information from the category 'Company' than chat-based memorization of the same. On a similar note, for all other categories, chat-based memorization seems to extract far more information. A discussion of this trend and a deeper exploration of the anomalous behaviour of just the 'company' category would have been quite interesting.

First, our methodology ensures that after the initial unsupervised fine-tuning, all attributes are fully encoded and achieve a 100% verbatim extraction rate (as shown in Figure 2b). This establishes that the information is perfectly memorized in a verbatim sense.

Second, after instruction tuning, we evaluate chat extraction (Figure 2c). Here, we observe that the extraction success varies significantly across attributes. This difficulty appears to correlate with factors like the uniqueness of an attribute's value (Table 1), where attributes with many unique values (like IDs) are harder to extract via chat than those with fewer (like Birth City).

Finally, Figure 5 explores a different phenomenon: the fragility of verbatim extraction itself after SFT. The y-axis in Figure 5 is for verbatim extraction, and it shows that after instruction tuning, the verbatim extraction capability is very low (the pink bars), even though the information is still extractable via chat (as seen in Fig 2c). However, if we "realign" the model by continuing unsupervised training, verbatim extraction is largely restored. This highlights that a failure in verbatim extraction does not mean the information is gone.

We will add a clearer interpretation of this trend. About the company attribute specifically, the observation made by the reviewer is totally correct, but we don't have a clear explanation for it. We will highlight this, that SFT removes verbatim extraction for most of the attributes.