Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks

It is not mentioned how the different optimization objectives for the proposed defense techniques are used in combination with these methods.

We apologize for confusion about this. To clarify, the defense methods are objectives, and the underlying optimization method is ROME/MEMIT i.e. the module in the model which is being edited is decided by the optimization method and the objective for which the edit is optimized is decided by the defense method. So in Table 1, we list ROME and MEMIT as the optimization methods, and the +X is the defense objective that is optimized by the optimization method.

We will be able to make this clearer with extra space in a camera ready version of the paper.

paragraph in 4.2 also had a bold subheading

Thanks. We’ll make it bold in the final version.

Is it possible to modify your approach to make this work for multi-token answers?

On the attack side, yes it would be possible to extend our attacks to the multi-token setting. For the whitebox attacks, the logit lens could be readily applied in a manner similar to autoregressive decoding to generate outputs or rank-order a set of plausible options (i.e. doing autoregressive decoding but decoding from an intermediate layer’s hidden states rather than running the full forward pass). On the blackbox side, generating multiple tokens is straightforward.

We note that the main reason for this simplifying assumption is that it makes it easier to rank order and evaluate the attack success. If we’re deciding whether the ground truth answer is in the candidate set, it's easier if there are 50000 possible atomic answers to questions, and we can directly check whether the exact answer is in the candidate set. If the space of answers is really big, that basically turns into a complicated NLG evaluation problem (is the true answer similar enough to some candidate in the candidate set?), so we wanted to simplify the problem in our work.

I don't quite understand how the Head Projection defense works. What exactly are the values D_answer and D_k? As far as I understand, D_answer is a probability distribution, while D_k is a single value? Could you clarify the loss function and what exactly is optimized for this defense?

The objective of HP defense is to prevent the deleted answer from appearing in the top-k elements of the logit lens distributions across a set of layers denoted as L. This is done to defend against the HP attack (refer to Fig. 3). To achieve this goal, we introduce a max-margin loss in each relevant distribution. Here, D(ℓ) represents the logit lens distribution at layer ℓ, D(ℓ)_answer denotes the probability of the original answer's logit lens, and D(ℓ)_k represents the k-th top probability in D(ℓ). The given margin loss objective forces the probability of answer (D(ℓ)_answer) to be less than the probability of the kth highest-probabiity token (D(ℓ)_k) by at least a margin of m which makes the answer less detectable in the HP attack (see Sec 5, page number 7).

The findings of the paper are somewhat expected

True, they are expected but they are also important and necessary since there has been a lot of excitement around editing methods as well as many concerns about privacy and memorized information in language models. This is a timely policy issue in society, and it’s not clear that we have the technical tools to address the problem at the moment.

At the same time, one surprising result is that we do see that HP Def works pretty well against PD attack, which it is not prepared for. In the paper, we suggest that there is no single defense method which works against all attacks it could face, but we do see some surprisingly positive results within the whitebox attack category.

Additionally, some readers will probably be surprised to see that “deleted” information is still recoverable from model hidden states in the first place. This has not been shown before empirically.

I was left wondering what are the technical differences between ROME and MEMIT and whether this could explain some of the differences in the results.

Thanks for pointing this out, we actually did discuss this in an earlier, longer version of the paper but unfortunately our ability to comment on every difference in methods was constrained by limited space in the ICLR submission. We are happy to further discuss these kinds of details (see below) in an extra camera ready page.

Regarding the differences between ROME and MEMIT: Attack success against MEMIT is generally higher than against ROME, and our best defense methods (like Max-Ent) achieve smaller improvements over baselines with MEMIT than with ROME. This could suggest that distributing editing updates across multiple layers rather than a single layer, as MEMIT does, increases vulnerability to attacks and makes defense more challenging. However, MEMIT performs more favorably than ROME on the Rewrite Score and ∆-Acc metrics, meaning this difference in the methods could be due to a difference in how strongly the edits applied to the model.

Confusing usage of the term “sensitive”

Thanks for pointing this out, we will clarify the different meanings of sensitive vs. toxic (with appropriate citations to broader socio-technical work) in the final version of the paper.

The paper’s second claim is that model editing methods fail to delete information; basing it on only one of two editing methods studied in the paper weakens the claim and raises questions.

It is true that Sec. 7.1 focuses on ROME in order to make the point that SOTA model editing methods are vulnerable to information attacks. However, we believe this result would only hold more strongly for the MEMIT and constrained finetuning methods, since we see in results in 7.2 that MEMIT with defenses is more easily attacked compared to ROME for almost all defenses. So we expect the trend to be similar for MEMIT but with even higher attack success rates (see also previous answer on ROME vs MEMIT).

Do the authors think that model editing is technically possible with formal guarantees against attacks and what are, in the authors’ opinion, promising directions for future work in this domain?

This is a good question! To our understanding of the adversarial robustness literature, formal guarantees often (1) make strong assumptions about the form of the attack, and (2) are often limited to data domains like images rather than text. It seems to us that the problem of information extraction attacks on LMs is still a sufficiently new problem that additional empirical and analysis work needs to be conducted first before a stronger theoretical understanding is achievable, although clearly formal guarantees of robustness are very relevant and valuable. Interestingly, there is also the opinion that as models get better (in both vision and language settings), eventually adversarial vulnerability will go away on its own. For instance, you might think that paraphrase attacks cease to be effective once models truly understand that we don’t want them to reveal people’s private information no matter how they are asked about it. Overall, this is still a fledgling area of research, but since it is very practically important it demands much more attention.

why do the hidden states of LLMs reveal sensitive information? What is the intuition behind this phenomenon?

We leverage the insight from interpretability research that output information accrues over time in the hidden states of a Transformer forward pass [1,2,3,5] (See Sec 4.1)

Other past work shows that models memorize their pretraining data, often a large proportion (like 1%) [4].

As a result, we believe sensitive information is memorized and accrues across layers during the forward pass. What our paper adds to this perspective, is that we show that edits to the model do not properly remove information across all the layers, and it can still be found by examining several layers' latent states.

This work provides an incomplete description of the reasons behind some experimental phenomena…Why are defense strategies based on data augmentation ineffective?

It is true that the Input Rephrasing defense that uses data augmentation to delete information for multiple paraphrases of the input data does not work against blackbox attacks. We confirm that the defense does work when the attack uses the exact same paraphrased inputs supplied to the defense’s editing objective; the defense fails when attacking paraphrases differ at all, even if they are drawn from the same distribution. Additionally, we can lower the Input Rephrasing Attack success by making the model edits more aggressive, but this has the consequence of skyrocketing ∆-Acc numbers. (See Sec 7.2)

can fine-tuning, a typical defense strategy, be combined with the defense scheme (e.g., Head Projection Defense) proposed in this paper to produce a more powerful defense method?

Yes! Finetuning is an optimization method and is agnostic to the objective. Each of our defense methods is characterized by its objective function and can be combined with different model editing (optimization) approaches, which are described in Sec. 6. See Appendix B.3 and Table 5 in appendix where we combine our defense objectives with constrained fine-tuning optimization method. With an extra camera ready page we will flag this point more prominently than in the original draft.

[1] nostalgebraist. interpreting gpt: the logit lens, 2020.URL https://www.lesswrong.com/ posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

[2] Belrose, Nora, et al. "Eliciting latent predictions from transformers with the tuned lens." arXiv preprint arXiv:2303.08112 (2023).

[3] Pal, Koyena, et al. "Future Lens: Anticipating Subsequent Tokens from a Single Hidden State." arXiv preprint arXiv:2311.04897 (2023).

[4] Carlini, Nicholas, et al. "Quantifying Memorization Across Neural Language Models." The Eleventh International Conference on Learning Representations. 2022.

[5] Geva, Mor, et al. "Transformer Feed-Forward Layers Are Key-Value Memories." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

I find it hard to draw actionable conclusions from the results, and further discussion and variety of results may help justify the strength of model editing as the de-facto paradigm for unlearning.

We summarize the actionable conclusions below based on the results in Sec 7.2:

• Whitebox attacks. Max Entropy defense helps best against white-box attacks, which is a positive and directly implementable in practical efforts in sensitive information deletion.
• Blackbox attacks. Although Error injection and Fact erasure defenses are somewhat effective against blackbox attacks, the lack of robustness of the deletion methods to paraphrases of the input is evident from the high attack success rates. Here we note that, like past work with jailbreaks for LLMs, sensitive information deletion is another example of a security issue with LLMs that we are highlighting. They are just not secure against paraphrase attacks.

No comparison to prior works - I’m concerned about the lack of comparison to prior works.

In essence, constrained finetuning is similar to the most common approach for unlearning, which is EWC (Elastic Weight Consolidation). EWC is being used as recently as 2022 in unlearning papers [1]. In the original draft of the paper, we distinguished model editing from unlearning based on whether the goal was to unlearn y|x vs. (x,y). However, the optimization approaches they employ are actually quite similar (specialized and constrained forms of finetuning), and unlearning methods are readily adapted to the problem of sensitive information deletion.

In summary, we believe our constrained finetuning approach (See B.3 and Table 5 in appendix) is already a very comparable baseline to existing unlearning approaches like EWC, and since ROME and MEMIT are known to far outperform constrained finetuning, we believe that we have covered all the relevant baselines and are using the best possible methods for the problem at hand. Although if you know of another unlearning approach that is more qualitatively distinct, we would also be happy to add that.

I’m a bit confused by the distinction between the password attempts and parallel extractions scenarios in the context of the paper. Is the distinction here meant to be that information gained in the recovery of a previous attempt may be utilized to aid recovery of the next attempt?

To clarify, we tried to give multiple plausible threat models that all map onto the same final Attack Success Metric, in order to more robustly motivate the attack metric we use as being widely relevant to real-world attack scenarios. All experiments utilize the same Attack Success Metric, and there is no “information gained” between different attack attempts in our experiments.

If so, I do not understand how the attacks as described utilize information in this way (i.e. it seems all attack methods generate a candidate set for a budget B in parallel anyways). Is such an attack possible?

To answer your question (even though it is not the case that our attacks utilize information from past attempts), it would be possible for an attack to leverage past attempts if there was some prior knowledge about candidate correctness being correlated or not. For instance, if it was known that a personal phone number began with either 919 or 336, then one could rule out half the possible candidates by first checking a candidate beginning with 919.

the paper claims that the described method does not assume access to the exact text used during pre-training of the model. However, the experiments do assume such access, as even in the rephrasing attack, the ground-truth prompt is perturbed by a rephrasing model

In our experiments, we do not use any “ground-truth” prompt from the pretraining data. We assume access to a textual representation of a question that the attacker is interested in knowing the answer to, and we rely on popular model editing datasets CounterFact and ZSRE for these textual examples (Sec 3.1, Sec 6). But CounterFact and ZSRE are not the pretraining datasets that GPT-J and Llama-2-7b were trained on. If we want to know whether GPT-J memorized what country the University of Madrid is in, we actually have no idea what would be the exact text from the pretraining data that prefaces the answer to this question.

In other words, an attacker only needs to know how to ask for some sensitive information, they do not need to know exactly what text prefaced that sensitive information in the model’s pretraining data.

[1] Tanno, Ryutaro, et al. "Repairing Neural Networks by Leaving the Right Past Behind." Advances in Neural Information Processing Systems 35 (2022): 13132-13145.

We would like to express our gratitude to all the reviewers for their time and effort in reviewing our paper. This serves as a friendly reminder that the deadline for the discussion phase is rapidly approaching. We kindly request your feedback regarding whether our response adequately addressed your concerns. Additionally, if you have any further inquiries or require clarification on any aspects of our paper, we are more than willing to assist before the deadline. Once again, thank you for your valuable inputs and consideration.