Privacy Auditing of Large Language Models

Thank you for appreciating our extensive experiments and the straightforward design of our canaries. In this response, we want to outline how we have addressed your concerns in the revised paper.

W1: Single dataset

We have added results on another dataset E2E (Table 1 for memorization rate and Table 3 for DP audit). Privacy leakage across both datasets is similar. For example, at a $99\%$ CI, the new token canary gives us an audited $\varepsilon$ of $0.84$ for FFT on PersonaChat, $0.82$ for FFT on E2E, and $0.89$ for LoRA on PersonaChat. We believe this is encouraging because our audit is of DPSGD and it should not vary much based on whether we use FFT or LoRA, or PersonaChat or E2E.

Q1: MI attack is all about canaries

Canary design is in the threat model of an audit. Right now, there is no successful DP audit for LLMs. At least in our setting with the designed canaries, we know it's possible. It is an open question if we can get MIAs outside of this setting as powerful without this assumption, but the community has already agreed on the importance of audits (e.g., Steinke et al. 2024 “Privacy Auditing in O(1) Run”). We believe that if a company wants to audit the model it is training, it should use the new token canaries to get the tightest possible audit. This indicates what the privacy leakage could be in the worst case if the model were trained on a dataset where one sample contained a token not seen anywhere else, which is not entirely unrealistic.

Q2: Distribution shift

In short: the problem of distribution shift does not exist in our setting and is not an issue for auditing.

We have expanded the related work discussion on membership inference attacks and discussed Meeus et al. The shortcomings in membership inference attacks they identify are not an issue in our setting because we sample member and nonmember points IID from the test dataset (in section 4) or from random tokens (section 5).We now also clarify this in the setup of Section 3 as well. Further, we only require this IID property to hold across this subset of data, not the entire pretraining dataset as in prior works.

Q3: LoRA in addition to FFT

We apologize for not being clear; our results with larger models are actually all done with LoRA because we could not train, ex, Gemma-2B or anything larger with FFT. We have clarified this and explicitly added a comparison between FFT and LoRA in auditing (Table 3). As noted in the response to the first weakness, results are similar between FFT and LoRA. We also provide ablations on the LoRA rank (Table 13) and LoRA without updating embeddings (Table 14).

Q4: Poor performance on some models

Results with the NWP objective have high variance because the loss is computed over the entire sequence, and the random sampling of prefixes from the test dataset means that some sequences will naturally have a higher loss before any training is done. This is the reason for the discrepancy between the NWP and SFT results in Figure 3 and Figure 4 (in the updated paper, the order is switched and we make this clear, Figure 1 is for SFT and Figure 2 is for NWP).

Q5: Concurrent work

Thank you, the paper looks very relevant. Given that this is an ICLR 2025 submission, we cannot cite it, but when it is posted on Arxiv we will cite it in the camera ready.

This review cites a concurrent ICLR paper [a]. The program chairs confirm that reviewer YH6v is not an author of [a]. These are concurrent works and they should be evaluated independently.

Thank you for appreciating the strong results of our paper. In this response, we want to outline how we have addressed your concerns in the revised paper.

W1: Single dataset

We have added Table 1 for memorization rate estimation on E2E dataset and Table 3 for DP auditing on E2E dataset. E2E dataset is a natural language generation task that is also evaluated in prior DP NLP works Li et al. 2022 and Yu et al. 2022). The result of our newly added experiments on more datasets is consistent with our finding, our designed canaries, the newly added token, gives us better memorization rate estimation and dp audits than other baselines like random token.

Typos

Thank you, we have fixed these.

Q1: Other MIAs

As we note in Sec 2.1 and 6, Duan et al. has evaluated many different MIAs and concluded that none of them work on LLMs. There are 3 ways to make a better MIA. 1) choose data that is more separable 2) choose a better statistic 3) choose a better test based on that statistic. Prior work has already extensively explored (2) and (3) and were shown by Duan et al. to not work. Thus, we focus on (1) and show that changing this aspect alone can lead to strong privacy audits. This is the main contribution of our work, and is orthogonal to exploring (2) and (3), which may leverage our techniques in future work.

Q2: How do different canaries impact performance

We have added the perplexity on the test set across canaries in Table 3. We reproduce the analysis here for convenience:

In\cref{tab:main} we observe that the new token canary degrades perplexity, while the random, unigram, and bigram canaries don't degrade perplexity. This can be seen as a natural tradeoff between the model memorizing the canary and the model learning the clean data distribution.

Q3: Why do we use the random prefix for DP auditing

We have added a detailed explanation of this in the paragraph accompanying Table 8 (random prefix vs test prefix). We reproduce it here for convenience:

Random Prefixes are Better Canaries than In-Distribution Data. We compare two approaches for selecting canary prefixes: randomly sampled tokens versus samples from the test dataset. In\cref{tab:prefix}, we demonstrate that using random tokens as prefixes leads to more effective privacy auditing. This can be explained by considering what associations the model needs to learn during supervised fine-tuning. With test distribution prefixes, the model must balance learning two competing objectives: (1) associating the prefix with its natural, in-distribution continuations, and (2) associating it with our inserted canary token. This competition naturally reduces the probability of the model predicting the canary token. In contrast, random (out-of-distribution) prefixes only require the model to learn a single, albeit unusual, association with the canary token. This focused learning task makes the canary information more distinguishable during privacy auditing, as the model's prediction of the canary token becomes a clearer signal of memorization. This may seem like a limitation, because it means that the attacker conducting the MIA cannot get a clear signal on the in-distribution data with semantic meaning. However, in\cref{sec:mia_eval} we used samples from the test dataset as prefixes throughout and showed that when the model is not trained with DP, the attacker can correctly identify members. In the auditing threat model, we can use random prefixes for the canaries without it being a limitation for our method. However, this also shows a clear direction for future work to build on our method.

Thank you for appreciating our well-motivated experimental design. In this response, we want to outline how we have addressed your concerns in the revised paper.

W1: Lack of related work

First, we clarify that while there are many works in membership inference attacks on LLMs, we are the first privacy audit. We have expanded the related work discussion on membership inference attacks and discussed Duan et al. in detail. The shortcomings in membership inference attacks they identify are not an issue in our setting because we sample member and nonmember points IID from the test dataset (in section 4) or from random tokens (section 5).We now also clarify this in the setup of Section 3 as well. Further, we only require this IID property to hold across this subset of data, not the entire pretraining dataset as in prior works.

W2: Only one MIA used

As we note in Sec 2.1 and 6, Duan et al. has evaluated many different MIAs and concluded that none of them work on LLMs. There are 3 ways to make a better MIA. 1) choose data that is more separable 2) choose a better statistic 3) choose a better test based on that statistic. Prior work has already extensively explored (2) and (3) and were shown by Duan et al. to not work. Thus, we focus on (1) and show that changing this aspect alone can lead to strong privacy audits. This is the main contribution of our work, and is orthogonal to exploring (2) and (3), which may leverage our techniques in future work.

W3: Redundancy

Thank you for pointing this out. We have reduced this.

W4: Auditing experiments need more models

Thank you. Our auditing experiments now cover GPT2, GPT2-large, GPT2-xl, pythia-160m, pythia-410m, and Qwen-2-0.5B, varying model architectures and model sizes. The results are shown in Table 4 for DP auditing. This results is consistent with our finding, our designed canaries, the newly added token, gives us better DP audits than other baselines like random token.

Q1: Why does MIA not work on GPT2?

Thank you for drawing our attention to this. After looking closer, we realized that this was a clerical error in the presented results, and we have updated the MIA results (see Figure 1, with corresponds to Figure 4 in the original submission). The updated MIA result for GPT2 (Figure 1) now shows that we can obtain a TPR of 0.548 at 1% FPR, which is in line with the results from other models.

Thank you for appreciating our innovative contribution, potential impact, and well-structured methodology. In this response, we want to outline how we have addressed your concerns in the revised paper.

W1: Only 1 dataset and only 1 model for DP Audit

We have added Table 1 for memorization rate estimation on E2E dataset and Table 3 for DP auditing on E2E dataset. E2E dataset is a natural language generation task that is also evaluated in prior DP NLP works Li et al. 2022 and Yu et al. 2022). We add Table 4 with 6 models total for DP auditing. The result of our newly added experiments on more datasets and models is consistent with our finding, our designed canaries, the newly added token, gives us better memorization rate estimation and dp audits than other baselines like random token.

W2: No Comparison to PANORAMIA

Thank you for this reference. We have added this to our related work.

PANORAMIA does not provide an audit, in that they do not show a lower bound on epsilon. In the paragraph titled "Measurement Semantics" on page 6, they note: "the value PANORAMIA returns does not imply a lower bound on epsilon." In contrast, we return a provable lower bound on epsilon.

W3: No Comparison to other work

Thank you for these references. We have added these to our related work.

[2], [3], and [4] all tackle the same question that Duan et al. (“Do Membership Inference Attacks Work on Large Language Models?”) does: if existing MIAs perform worse than “blind baselines” then they only work because of a distribution shift between members and non-members. In our work, our sampling process for member and non-member datapoints is IID across the dataset that we draw them from. We detail this dataset in each section: in Section 4, this is validation data and in Section 5, this dataset is random tokens. We now also clarify this in the setup of Section 3 as well. Therefore, the problem of distribution shifts identified in Meeus et al. and Duan et al. does not exist. This is different from prior work, which requires the IID property to hold across the entire pretraining dataset that they consider.

Q1: Is this paper still the first privacy audit of LLMs?

We argue yes, because PANORAMIA [1] is not a privacy audit.

Q2: Did we vary hyperparameters

Yes, we varied the learning rate (Table 12), number of training steps (Table 7), canary size (Table 2), and sampling rate (Table 6). Each of these ablations has a paragraph analyzing our results.