Can Language Models be Instructed to Protect Personal Information?

We thank the reviewer for their thoughtful review. We have responded to your questions below:

(Q1) Using geospatial location metadata as inputs to vision language models

We do not use location metadata as inputs to vision language models, rather the inputs are simply the query and the image. However, this is an interesting question for future work, but we see this as beyond the scope of the current paper.

(Q2) Comparisons with additional baseline methods are needed to highlight the advantages of the proposed privacy-preserving techniques.

Thank you for this suggestion! We have added a simple baseline confidence method based loosely on a technique from (Portillo Wightman et al., 2023): https://aclanthology.org/2023.trustnlp-1.28/. A figure containing the results of this method for llama2-chat-7B is shown in Appendix B (see Figure 17).

You have also mentioned adding a machine unlearning baseline. In Section 4 (see Figure 6), we include a model editing baseline using the ROME (Meng et al., 2022) methodology, through which we find that this technique struggles to scale to the many model edits required to adhere to access control instructions. We choose to include a model editing baseline instead of a machine unlearning baseline as it is more pertinent to our work. Specifically, machine unlearning attempts to ‘forget’ certain sequences of training data, whereas our work is concerned with preventing the disclosure of information (not necessarily verbatim from the training corpus) that is protected. This approach is more similar to model editing work such as ROME which focuses on editing factual associations in language models.

(Q3) The benchmark data set could include more sensitive private information such as race, occupation, address, bank account, etc.

One of the reasons that we chose to construct our dataset based on public figures is to avoid including sensitive information about private figures. Please see the added ‘Ethics Considerations and Limitations Section’ for additional information. However, we do include some of the attributes you mentioned in our dataset such as citizenship and occupation as these are categories present in Wikidata. We also find that models do exhibit biases with respect to race (see Figure 20).

(Q4) Alignment of the proposed benchmark dataset with the prevailing perspectives within the ICLR community

Task definition/evaluation papers have previously been well accepted at ICLR. For example, from last year’s conference, the following two papers are related in style to ours:

Shi et al., Language models are multilingual chain-of-thought reasoners: https://openreview.net/forum?id=fR3wGCk-IXp

Tamkin et al., Task Ambiguity in Humans and Language Models: https://openreview.net/forum?id=QrnDe_9ZFd8

Thank you for your response and for these important questions! We have answered them below:

• Geolocation will not work for every image, with the success being highly dependent on the landmarks in the image. We ran a small human evaluation of model outputs (IDEFICS-80B and GPT-4V) on a random sample of 32 images from the InfoSeek dataset and found that 53.1% (17/32) of images were geolocated successfully by GPT-4V and 31.3% (10/32) of images were geolocated successfully by IDEFICS-80B. With recent improvements in model capabilities, we expect this number to increase and believe that it constitutes a real risk to user privacy. Due to the limited time before the end of the author response period, we have not updated the paper with these results but will do so for the camera-ready version of the paper.
• There are a number of potential scenarios in which the adversary can use leaked personal information from LLMs to carry out harmful actions in both the digital and physical worlds. We have outlined some of these below and will also update our paper with a discussion of these:

1. Aiding in spear phishing attacks (https://en.wikipedia.org/wiki/Phishing): An adversary can collect personal details about an individual (e.g., their place of work, location, etc.) to help them craft compelling emails to be used in spear phishing attacks. Note that even if a user deleted this type of information from their social media, the model will still be able to divulge their information if it was initially included in the pretraining corpus.
2. Aiding in doxxing attacks (https://en.wikipedia.org/wiki/Doxing): A vision language model could be used for the identification of a normally private individual from a photo or video (potentially a viral video circulating on social media) enabling malicious users to harass the now-identified individual on their social media accounts.
3. Aiding in physical stalking (https://en.wikipedia.org/wiki/Stalking): Identifying private information about an individual, especially their current location (e.g. based on photos they recently posted on social media) using a vision language model, to enable physical stalking.

We are happy to follow up on any of these questions or respond to any other comments you have before the end of the response period on November 22nd (tomorrow).

We thank the reviewer for their insightful feedback. Responses to each of your points are listed below:

Open-sourcing code and data

To facilitate the reproducibility of the research work, we uploaded data/code to the anonymized link here: link. Due to the size of the data (2.4GB), we uploaded them to Google Drive using an anonymous email (submissionanonymized@gmail.com).

Privacy is about even a single leak of private data is unacceptable.. would you consider any of the presented methods adequate for use in a realistic setting?

We agree that leaking private data is unacceptable, and we have been very careful to avoid claiming that current models are good enough at this. We pointed out the serious issue in the introduction as

However, we discover serious issues related to bias and robustness with these instructions that we believe need to be addressed before they can be used in critical applications.

In fact, we hope our experimental results (non-adversarial and adversarial) emphasize this point and alert practitioners to be aware of the limitations of current models.

Clarification: section 3.1, is it true that $\mathbb{PG} \cup \mathbb{CG} = \mathcal{X}$

Thanks! This is true and we have revised the manuscript to make this clear.

Clarification: (Figure 2) Why do larger models have more privacy violations? I expect larger models to follow instructions better

Indeed, we show the protection score of llama-70b outperforms llama2-chat-7b in Figure 2 (left: 75 vs 58, right: 72 vs 44). We suspect R3 was referring to the charts over Response F1 (Protected), where the F1 score decreases as the model size increases from llama2-chat-7b to GPT-4. We would like to clarify this score should be as low as possible to make sure models don't reveal private information during response generation. Hence, the larger the model, the lower the score, which is expected as R3 mentioned. We will again clearly state in the revised manuscript to remove any confusion.

Clarification: what are head entities on page 7

• Head entities: popular entities that are more regularly visited on Wikipedia (e.g., $10^6$ Wiki pageviews).
• Tail entities: less popular entities at the tail of the distribution (e.g., $10^1$ Wiki pageviews).
• This is discussed in paragraph 8 of section 4.

Thank you again for your response and feedback! We agree and plan to specifically report the two metrics ('specificity' and 'sensitivity') that are used to compute the protection score (harmonic mean) in the camera-ready version (due to limited time). Below we report these metrics for GPT-4 & llama2-70b.

The reason we chose to report the protection score was mainly because of the motivation to prevent the model from over-optimizing for just one of the measures: In some extreme cases, one model can:

• always answer the question --> 100% specificity and 0% sensitivity
• always abstain to answer --> 0% specificity and 100% sensitivity

But we appreciate the reviewer bringing this to our attention and we plan to report these two metrics to provide more context.

Protected population:

Protected information:

We thank the reviewer for their comments to help us improve the paper. Responses to each of your points are listed below:

What is the threat model? Is the data we are trying to protect from the training data, or data provided to the model at inference time? I think the authors need to first discuss the related work better.

This is an excellent suggestion. We agree that there should be a clearer discussion of the differentiation between our approach and related work. We have modified the introduction to provide this context. Specifically, while related to data extraction attacks and mitigation measures like machine unlearning, our setting and approach focus on more realistic and powerful privacy controls. For instance, while techniques like machine unlearning and model editing can be applied to edit or remove a specific text sequence or a single association found in the pretraining corpus such as “the name of the valedictorian of Central High School”, they cannot be easily used on a more realistic use case involving a category of information such as “the data of all students at Central High School”. We consider these more realistic privacy controls that enable adherence to these real-world privacy protection requirements (e.g., refusing to answer questions about a group defined with natural language).

What is the real-world use case? It seems protecting information about public figures is not a realistic scenario?

This is a good point! We chose to include questions about public figures that have an associated Wikipedia page primarily because of ethical considerations associated with a widely distributed benchmark. A benchmark evaluating the ability to extract data for less well-known individuals would not be able to be released publicly, and could not serve as a standardized benchmark, although we agree that this more realistic benchmark would also have value. This is discussed briefly in the first paragraph of Section 3, but we agree this is an important point and could be highlighted more clearly, so we have a new section “Limitations and Ethical Considerations” to highlight this discussion.

There is no successful extraction attack that would so easily, from an instruction tuned model, extract any information that is not repeated many many times.

We agree that data probably needs to appear multiple times in the training corpus for successful extraction. However, there are real-world cases where non-public entities might have their data appear multiple times in the corpus. For instance, we found that it was possible to extract personal information (the birthday of a user) about a Reddit user using the BetterDan jailbreaking prompt with gpt-3.5-turbo (an anonymized example has been added - see Figure 13) . Again, due to data restrictions and ethical concerns, we cannot publish a dataset of such examples, but we believe that PrivQA is a good alternative that has the potential to support widespread benchmarking and reproducible experiments.

We also believe that answering questions about personal information related to public figures is a more practical way to evaluate LLMs’ capabilities to selectively protect personal information, as they are more likely to be able to answer these questions. Accurately evaluating the ability of models to selectively refuse to answer in a scenario where the information appears only a few times is more challenging, as there will be higher variance in models’ ability to answer these questions. We agree this is an important problem for future work to address, but we also feel that our PrivQA benchmark is a first step that is complementary to a future more realistic data-extraction benchmark, which would require more restrictive access to researchers and would have larger variance in the evaluation of models’ ability to selectively protect information due to higher difficulty in answering the questions.

The dataset is not curated in a principled way and is too artificial. There are no levels to what is being protected, and how it needs protection. The attributes are artificially inserted.

Thank you for the feedback. We agree that the motivations behind some of the protection attributes chosen could be better explained, and modified the original motivations presented in Section 3.1 in the Protected Populations and Protected Information paragraphs. The attributes used to define protected classes are taken from common WikiData categories and represent what we believe to be important privacy considerations based on Article 4 of the General Data Protection Regulation (https://gdpr-info.eu/art-4-gdpr/). For instance, it is likely that LLM developers will want to avoid disclosing information about minors, which motivates the use of attributes such as age and protected classes such as individuals under the age of 18.

Talk about existing membership inference and extraction attacks, about the risks, and then about differential privacy and other protection methods

Thank you for this suggestion! In the introduction and related work sections, we have included a clearer discussion about the relationship between our work and the related work you mention. We have also cited the following membership inference attack papers

[1] Virat Shejwalkar, Huseyin A Inan, Amir Houmansadr, and Robert Sim. Membership inference attacks against nlp classification models. In NeurIPS 2021 Workshop Privacy in Machine Learning, 2021.

[2] Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu. Membership inference attack susceptibility of clinical language models. arXiv preprint arXiv:2104.08305, 2021.

[3] Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri.Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint arXiv:2203.03929, 2022.

Please let us know if you would like to see any other particular papers cited.

We thank the reviewer for their comments. Responses to each of your points are listed below:

This paper just tests whether or not the conventional instruction-tuned LLMs can protect privacy.

Yes, we do test the extent to which instruction-tuned LLMs can protect privacy. However, we believe that this is a significant contribution because

• our task formulation is more practical than other definitions that currently exist in the literature such as protecting against extracting specific sequences of training data. This new personal information protection task formulation can be a source of future work focused on developing better methods for this task.
• we provide a forward-looking open-source benchmark dataset in the form of PrivQA that can be used for benchmark future LLMs on our defined task.
• using our constructed dataset, we demonstrate the failure modes of existing LLMs specifically regarding bias and adversarial robustness, which can again serve as a basis for improvement for future models and protection techniques

The proposed “Self-Moderation” seems to slightly modify the previous “reflection” techniques in many previous works (there is a survey [1] on “reflection” techniques).

Thank you for the reference to this survey paper! We cited this survey in the updated version of the paper. We would like to preface this response by making it clear that we do not claim that Self-Moderation is a key contribution of our paper (as mentioned these would be the novel task definition and the benchmark dataset). We agree that our Self-Moderation technique is based on these “reflection” techniques and tried to make this clear originally by citing Self-refine: Iterative refinement with self-feedback (Madaan et al., 2023). However, like many of the papers in the survey paper, the main differentiation factor with our technique (Self-Moderation) is its application, which in our case is privacy and access control. Specifically, we show that LLMs perform somewhat well in the role of an access control system and that LLM decisions become increasingly conservative regarding data protection with more refinement.

The title is misleading. The title is not very related to the core message of this paper because this paper does not conduct instruction tuning to protect privacy but just test whether or not the conventional instruction-tuned LLMs can protect privacy.

In the paper, we do not claim that we perform instruction finetuning on LLMs to improve their ability to protect privacy, but instead evaluate the extent to which widely used LLMs can respond to personal information protection instructions. We believe that both our current title and the alternative title you supplied have similar meanings and reflect the claims that we make in the paper.

The contribution in the red teaming part is unclear. It seems this paper just directly applies the previous red teaming methods for an empirical study and do not propose any new red teaming method.

We agree that we do use previously proposed red teaming methods including standard jailbreaking prompts like BetterDan and AIM. These methods were included as baselines to study the robustness of access control instructions to measure their ability to be applied to potentially adversarial real-world scenarios. We do not claim that these are contributions.

However, we do apply other more novel methods including visual prompt injection, or modifying input images to VLMs to induce targeted failure modes i.e. revealing personal information (see Figure 8). While there have been two works concurrent with ours that have studied visual prompt injection:

• Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (Greshake et al. 2023)
• Plug and Pray: Exploiting off-the-shelf components of Multi-Modal Models (Shayegani et al., 2023) We believe we were the first to consider this attack for truly sensitive real-world applications.

The connection between the read teaming part in Section 5 and privacy experiments in Section 4 is unclear. The content in Section 4 & Section 5 is overlapped.

We apologize if this was unclear. We consider red teaming methods to employ adversarial inputs to circumvent access control instructions. The goal of Section 4 was to present a comprehensive evaluation of existing LLMs using our constructed benchmark. While this evaluation did include an analysis of shortcomings of access control instructions such as popularity protection biases, these were not intentionally induced. On the other hand, the goal of Section 5 was to present methods to adversarially circumvent access control instructions with malicious inputs.

However, we discover serious issues related to bias and robustness with these instructions that we believe need to be addressed before they can be used in critical applications.