User Inference Attacks on Large Language Models

We thank the reviewer for appreciating the practicality of our threat model and for the review. We answer each of the questions below.

[Other approaches to the attack design]

State-of-the-art membership inference attacks (e.g., LiRA by Carlini et al. 2022) require training hundreds of shadow models during the attack stage to learn the distribution of model confidences and create a test statistic. This approach would not be feasible for LLMs due to the high computational complexity incurred in training the shadow models. One of our main requirements is to design an efficient attack statistic that does not train additional shadow models. This motivates the use of the pretrained model as a reference model, resulting in the calibrated loss on the fine-tuned model normalized by the loss on the pretrained model as a test statistic.

Of course, there could be other attack designs even stronger than ours with similar efficiency, but that would only make privacy leakage a bigger concern. Our attack already indicates an unacceptable level of privacy leakage in LLM finetuning and we hope that future work will address the need for developing and evaluating defenses, as well as potentially designing stronger privacy attacks.

[Independence of user’s documents]

This is a great question! In the email case, independence is a modeling assumption that, even if incorrect, makes the problem more computationally tractable. Even with this simplifying assumption, we still get an attack that is relatively strong in most cases. Relaxing this assumption could likely lead to stronger attacks, but those won't necessarily change the conclusions/observations we make in the paper.

To reiterate, we do not aim to propose the strongest attacks in the paper. Rather, we focus on highlighting that even simple attacks (that are computationally efficient) can already identify privacy leakages in LLM fine-tuning.

[Mitigations: Clipping user-level gradients]

Clipping user-level gradients is a promising approach. In fact, the gold standard defense is differential privacy at the user-level (see Section 5), which involves clipping user-level gradients and adding noise.

Unfortunately, existing LLM finetuning infrastructure does not allow for a straightforward way of implementing clipping at the user-level efficiently at scale. Our results can be seen as strong evidence in favor of (a) developing software that supports per-user clipping, and (b) augmenting training workflows to keep track of the “owner” of each data point.

[Mitigations: Deduplicate documents from a single user]

Great question! We ran some preliminary analyses on the number of duplicates in each dataset. Here are the results:

• Nearly 10% of examples of CC News are exact duplicates within users.
• ArXiv is too large to look for exact duplicates. 2.4% of all examples in ArXiv are approximate duplicates across users (meaning the number the exact duplicates within users will likely be significantly smaller).
• We are still in the process of analyzing the Enron email dataset and we don’t yet have final statistics.

Your suggestion of deduplicating data at the user-level is a promising idea for mitigation. We will implement and evaluate it in the next revision of the paper (as this requires longer to execute).

[Attack success rate on the Enron dataset]

There are two key reasons why the user inference attack enjoyed high success on the Enron dataset, also highlighted in our analysis in Proposition 1:

• Each user contributes a large amount of data to the training of the model. Figure 6 (right) shows that this allows for a greater attack efficacy. (Note that Figure 3 shows the effect of the attacker’s knowledge: the above result is true even when the attacker’s knowledge is minimal).
• Second, the statistical divergence between the samples contributed by the target user compared to other training data (i.e., the KL divergence between the user’s distribution and other users’ training distributions): The larger the distance, the higher the attack success. There is more separation between users in the Enron dataset compared to ArXiv: for instance, emails contain more user-level information in the signatures, and salutations. Other details such as addresses are also commonly found in the body of the email (see some examples here)

Add the following CSS to the header block of your HTML document.
Then add the mark-up below to the body block of the same document.
.videoWrapper { position: relative; padding-bottom: 56.25%; padding-top: 25px; height: 0; } .videoWrapper iframe { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }

Thank you so much for your positive comments and suggestions! We are glad that you found the paper enjoyable to read.

[Question 1] choices of the reference model
That is a great question! We have performed an experiment in which we consider an attacker having access to a different pretrained reference model (Table 3 in Appendix). We obtained the best attack success when the attacker model is the same (e.g., 93.1% on Enron with GPT-Neo), but we only see a small reduction in attack success for a different pretrained model (e.g., 87.6% on OPT and 87.2% on GPT-2 for Enron). Fortunately, we found that the attack is not very sensitive to the knowledge of the reference model, and we believe that access to a public pretrained model is a reasonable assumption for the attacker.

Thank you so much for your insightful comments and supportive suggestions on our work. We are glad that you found our threat model realistic and our attack a foundation for future work in privacy-preserving LLM fine-tuning.

"user inference … merging samples from diverse tasks or domains"

Great question! Note that user inference is not possible when two users have the same distribution (e.g. split a set of documents randomly into two). Intuitively, some statistical similarity between samples from the user is necessary for general user inference (not just our attack). In particular, user inference is only possible if the attacker’s i.i.d. samples are closer to the user’s fine-tuning data than to any other user’s data.

This is what we meant by "samples from the same user are more similar on average than those from different users" — we have adjusted the phrasing to be more clear.

"disparity in attack performance … may not solely be attributed to the amount of each user's data [but to the text content. [Question A] other factors that could significantly influence user inference"

Great question! Indeed, the attack performance disparity on different datasets is not only due to the varying number of users (even though that is an important factor). We identified two additional factors that contribute to the attack performance, through our analysis in Proposition 1, also confirmed experimentally,

1. The number of documents contributed by each user: The Enron dataset has fewer users, each contributing more samples and a larger fraction of the training set, resulting in higher privacy leakage. Figure 6 (right) shows that the attack efficacy increases with the number of fine-tuning documents.
2. The statistical divergence between the samples contributed by the target user compared to other training data (i.e., the KL divergence between the user’s distribution and other users’ training distributions): The larger the distance, the higher the attack success. There is more separation between users in the Enron dataset compared to ArXiv: for instance, emails contain more user-level information in the signatures, and salutations. Other details such as addresses are also commonly found in the body of the email (see some examples here). On the other hand, the scientific writing style of ArXiv abstracts, which is predominantly impersonal and formal, makes the users less distinct. All these factors resulted in a more successful attack on Enron compared to ArXiv and CC News.

[Question B] proposed attack technique on recent chatbots

That is an interesting question and could be a really good topic of future research. Chatbots such as ChatGPT and even Llama-2-chat are trained with more complex pipelines, including fine-tuning on human-labeled data, training a reward model, and optimizing a policy using reinforcement learning -– quantifying the effect of each of these steps on user inference would indeed be interesting.

As with any other LLM, an attacker with multiple writing samples from a target user should be able to mount an attack to extract information about the user by carefully prompting the model and using the generated text. The main issues with this approach are (a) the lack of large user-stratified chat datasets to validate the success of such an attack, and (b) the lack of software support to reproduce the multi-stage fine-tuning and alignment pipeline of chatbots. Addressing these shortcomings and running controlled user inference experiments on chatbots is an interesting direction for future work.

Thank you so much for your careful reading of the paper and your detailed comments and suggestions! We believe we have addressed all your concerns below. Please do not hesitate to reach out in case of further clarifications or comments.

Lack of novelty: black-box methodology used is standard and (Chen et al. 2023) already proposes the same threat model

Thank you for pointing out the additional reference (Chen et. al. 2023). We have improved the positioning of the paper relative to existing work on user-level privacy risks in the introduction, and related work (Section 2). While user inference attacks have been studied in the literature for other application domains such as face recognition and speech recognition, we are the first to study user inference for LLMs where the attacker does not have access to the exact training samples.

Specifically, all existing work in the context of generative text models study more restrictive threat models than what we study in this paper. Most similar to our work are the following papers:

• Song and Shmatikov 2019 study user-level inference under the stringent assumption that the attacker has access to the training set of the text generation model. In contrast, our threat model assumes that the attacker may not have full knowledge of the user’s training samples, and is thus more realistic. Additionally, their attack trains multiple shadow models on subsets of the training set and a meta-classifier to differentiate users participating in training from other users. The meta-classifier-based methodology does not apply to LLMs due to the large computational cost.
• Shejwalkar et al. 2021 performs user inference for text-based classification models, but also assumes that the attacker has access to the user’s training samples. They do not consider generative text models in this work.

Thus, to the best of our knowledge, our user inference threat model has not been considered for generative text models. This is practically relevant to many settings today, such as Smart Compose, GBoard, and GitHub Copilot.

We also wish to point out that any paper published after May 28, 2023, is considered contemporaneous work as per the ICLR policy. The Chen et al. paper you pointed out was published in August 2023 and is thus considered contemporaneous. We ask that you assess the merits of our work independently.

[ArXiv dataset] multiple authors which may rewrite or edit the abstract

We acknowledge the limitations of the ArXiv dataset you have identified. Despite them, we still believe that analyzing the attack on ArXiv shows some interesting results.

While the user labeling might not be perfect, the fact that we still have significant privacy leakage suggests that the attack can only get stronger if we have perfect ground truth user labeling and non-overlapping users. Further, our experiments on canary users are not impacted at all by the possible overlap in user labeling, since we create our synthetically-generated canaries to evaluate worst-case privacy leakage. We have added a discussion in the revision in Appendix C.

At a more practical level, few datasets are large enough with user annotation for evaluating our privacy attack. Thus, despite its flaws, the ArXiv dataset provides a valuable data point (in addition to our other two datasets) in the evaluation of user inference attacks.

We really appreciate your detailed comments! In this response, we argue that:

• the arxiv dataset are still valid, and,
• the existing authorship attribution benchmarks are unsuitable for user inference.

Arxiv dataset: We appreciate the reviewer’s point but we believe that the noisy labels will, on average, lead to reduced attack performance.

Recall that our theoretical analysis shows that the attack is most successful when the users are well separated. Mislabeling examples leads to a mixture of data distributions (e.g. documents listed under a user A are written both by user A as well as their co-authors), which brings the data distributions closer together (this is proved below). Overall, this reduces attack efficacy as per Proposition 1.

Claim [Mixing distributions brings distributions closer]: Let $P, Q$ be two user distributions. Suppose the mislabeling of documents leads to distributions $P’ = \lambda P + (1 - \lambda) Q$ and $Q’ = \mu Q + (1-\mu) P$ for some $\lambda, \mu \in (0, 1)$. We always have $\text{KL}(P’ \Vert Q’) \le \text{KL}(P \Vert Q)$.

Proof: This can be seen by convexity of the KL divergence. Indeed, we have

$$\text{KL}(P \Vert \mu Q + (1 -\mu)P) \le \mu \cdot \text{KL}(P \Vert Q) + ( 1- \mu) \cdot \text{KL}(P \Vert P) = \mu \cdot \text{KL}(P \Vert Q)$$

Thus, $\text{KL}(P \Vert \mu Q + (1-\mu)P) < \text{KL}(P \Vert Q)$. The same holds for the first argument by the same reasoning.

We note that this argument also holds authorship attribution. If the documents are randomly mislabeled, the accuracy of any classification model only reduces the classification accuracy (see e.g. the influential paper by Zhang et al. (ICLR 2017)).

Authorship attribution datasets: Unfortunately, the existing curated datasets in the authorship attribution literature are not interesting for user inference as they are too small or have very few samples per user.

Specifically, here are the curated datasets we found and their drawbacks.

• CCAT50 (50 authors), CMCC has (21 authors), Guardian has (13 authors) and IMDB62 (62 authors) have even fewer authors than enron, our smallest dataset, where we also have near perfect attack accuracy.
• Project Gutenberg (29k documents and 4.5k authors) has enough users, but each user only has 6 documents on average. This is not enough to mount a user inference attack, as we need enough documents for fine-tuning (see Figure 6, right) and attacker’s knowledge (see Figure 3) for the attack results to be non-trivial.

While we appreciate the reviewer’s suggestion to explore the authorship attribution datasets, we conclude that the benchmarks are unsuitable for our experiments.

Summary: Thank you again for your time and consideration.