In-Context Unlearning: Language Models as Few Shot Unlearners

We are working on additional experiments and responses to address reviewer questions. We will be posting them very soon. Thanks for your patience.

Firstly, the method is heavily inspired from the paper "Larger language models do in-context learning differently". The authors propose that it works even for smaller models which challenges the observations of the earlier paper. However, it is not clear why it is the case. Were there any assumptions that were made in this paper that differed from the settings of the former paper?

Though this paper is proposed as machine unlearning, I am not really convinced if it falls into the realms of machine unlearning since the model weights still preserve the information of the user. So information is never really lost. As I mentioned above, by changing the prompt you can always retrieve the information.

Since my questions are not adequately answered, I am reducing the score from 6->5.

We are very grateful for the reviewer's positive and thoughful comments and believe that they have contributed to improving our work. Below we address individual points raised by the reviewer.

Querying with alternative prompts

When queried using alternative prompts, there is no guarantee that the language model would perform with similar characteristics.

This is correct. However, recall that our methodology is tailored to the real-world constraints of 'ML as a Service' platforms. There, the model owner always prepends the required unlearning contexts in front of the queries since it is the premise that the LLM operates as black-box (see Figure 1). In such scenarios, where model parameters are inaccessible, and standard unlearning techniques relying on gradient ascent are infeasible, our approach of pre-appending required unlearning contexts becomes crucial. For example, a bank finetunes a GPT3.5 model through OpenAI’s API for the task of deciding credit worthiness of customers. The bank would need to finetune the entire model from scratch when deletion requests come in since standard techniques like gradient ascent do require parameter access which we do not have when using the API. Hence, our technique is particularly useful when standard unlearning methods that operate via gradient descent on the model’s parameters cannot be implemented as the lack of parameter access impedes traditional unlearning methods.

Computational aspects of unlearning

Speaking about the computational complexity of the gradient ascent method, I believe since the forget set cardinality is much less than the training set, the computational cost would not be very demanding and also it will be similar to finetuning language model on extremely low-resource scenarios.

Contrary to the hypothesis regarding the computational aspects of unlearning, our experiments demonstrate the distinct memory requirements for gradient ascent (GA) and our proposed In-Context Unlearning (ICUL). In response to the reviewers hypothesis, we consider unlearning of sets of size 1 on the Llama2 7b model fine-tuned on the SST2 dataset (see Appendix C.1). Here, ICUL proves to be resource-efficient. Running ICUL on a single Tesla V100 GPU with 32GB of RAM contrasts sharply with the demands of gradient ascent (GA), which necessitates an A100 GPU with 80GB of RAM.

Prompt based adversarial attacks

I think it is necessary to compare with a few prompt based adversarial attack methods such as [1] or related methods.

We appreciate the reviewer's insightful suggestion regarding alternative ICUL prompts. Indeed, our approach can be seen as a perturbation to the query point that reduces the likelihood of the query point belonging to the training set. We acknowledge that there may be other effective unlearning contexts. Unfortunately, due to resource constraints, we were unable to run the proposed method from [1] in addition to our new experiments (see Appendix C) as an alternative ICUL prompt. However, we thank the reviewer for this valuable suggestion and are committed to exploring adversarially perturbed prompts, as proposed in [1], in our future work. This aligns with our acknowledgment that there could be multiple routes to achieve unlearning, and we look forward to further investigations in this direction.

Presentation

The presentation of the experiments section needs to be improved.

Thank you for this suggestion. We have updated the manuscript by moving section 4.2 to Section 5.4 where we conduct the sensitivity analysis. We also added three corresponding values for s that were used for ICUL in Table 1, one value of s for each dataset.

Correcting typos

Please correct the typos and ensure consistency in notation. For example, 'membership inference' is abbreviated in some instances, while in others, it's written out in full.

We made sure that MI is used consistently throughout. Please let us know if the reviewer has found additional typos.

Mentioning baselines

Baselines are not clearly mentioned in the experiments section. For instance, GA is not clearly mentioned [...]

We have made this more clear now. We have highlighted the abbreviation gradient ascent (GA) in the preamble of section 5 again. We also introduce the term Baseline in Section 5.1 more clearly. Please refer to the text highlighted in green for the concrete section we made to Section 5.

Further elaborations

The experiments section requires further elaboration. While the authors assert that their method outperforms the baselines, they should delve into why this is the case. Also from the results, it is evident that the method performs at par with the GA method however it is not really clear why the GA method performs worse than the Baseline (especially the sudden spike towards 10^(-1) FPR for Amazon and Yelp).

When using GA, we have been strictly following the methodology proposed in [1]. We performed gradient ascent for one epoch using adamw on the forget set $S_f$ tuning learning rates in the range of [5e-05, 3e-05, 1e-05] as a hyperparameter, consistent with the approach in [1]. It is conceivable that using different optimizers would avoid the bump and could lead to improved unlearning results for GA.

Questions

My question is if I want to query on [Forget Input] again the format is as follows, “[Forget Input] [Flipped Label] \n [Input 1]1 [Label 1]1 \n · · · [Input s]s [Label s]s [Forget Input] ”. Is my understanding correct? If so, it would be interesting to know how many times the output of the language model is [Flipped Label]? We can understand the broader impact of such a querying by further knowing these dynamics.

Yes, your understanding is correct! Also, the output flips in 1 to 5 percent of all cases. While the output is usually not flipped, it is the confidence in the true class that decreases.

In section 4, it is mentioned that “For finetuning, we are using the standard causal language loss which encourages the model to predict the next token correctly”. So when the models are fine-tuned on the downstream dataset, why is the result on the train set low, especially for ICUL(6) on SST-2 dataset (Table-2).

This phenomenon is most pronounced on the smaller Bloom 560M model (3rd column in Table 2). The larger Bloom 1.1B model manages to use the correctly labeled examples from its context more effectively than the smaller 560M model (6th column in Table 2).

The authors claim "This finding challenges earlier research that argued label flipping of context examples had an insignificant impact on smaller LLMs " in the conclusions. It would be more insightful if the authors can point out why is the case? Or if there is any assumption difference between these works and the proposed work.

In contrast to the findings in [2], our research suggests that the performance of large language models (LLMs) may not be solely determined by their size. While [2] highlighted the tendency of smaller LLMs to neglect prompt context, emphasizing the superior context utilization of larger models, our work introduces a nuanced perspective. We argue that the capacity to selectively forget specific instances from the training set is not exclusive to larger models, thus challenging the notion that only increased model size allows for effective knowledge overwriting.

Further, these two works fundamentally differ in the questions they aim to answer. While both our work and the the work from [2] utilize label flipping, it's crucial to emphasize the divergence in our research objectives and evaluations. The commonality lies in the application of label flipping, but the motivations behind its use and the questions addressed are markedly different. In [2], the authors employ random label flipping primarily as an evaluation technique to study the impact of label flipping itself on model predictions. In contrast, we propose label flipping of specific points targeted for removal as an unlearning procedure and subsequently employ a combination of sample splitting and Membership Inference Attacks (MI Attacks) for our evaluation. This nuanced approach allows us to make specific claims about the unlearning capabilities of ICUL. In summary, their focus on the impact of label flipping do not facilitate claims about unlearning success. Our methodology, centered around unlearning through label flipping and the subsequent evaluations, provides a distinct angle that contributes to the broader understanding of how unlearning can be achieved in-context.

References

[1] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023

[2] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv:2303.03846, 2023

We are grateful for the reviewer's critical and thoughful comments and believe that they have substantially contributed to improving our work. Below we address individual points raised by the reviewer.

Single benchmark

[...] ICUL is compared to only a single benchmark method (GA) [...]

1. Current SOTA unlearning methods rely on the Hessian and do not apply to LLMs: Unlearning training points in Language Models (LLMs) is an emerging area, and the limited availability of viable alternatives, exemplified by [2], underscores the current landscape. The primary reason for this scarcity is rooted in the design constraints of existing methods. Many prevalent approaches, both exact and approximate, necessitate the computation and inversion of the Hessian matrix ([6,7]). However, for LLMs, the computation of the Hessian and its inverse become computationally infeasible. Notably, the alternative used by [2], as suggested in [8], leverages gradient ascent on the points targeted for deletion, avoiding the need for Hessian matrix computations.

2. ICUL is close to the optimal performance: In terms of unlearning efficacy, it's crucial to highlight that our method achieves performance comparable or extremely close to the optimal scenario of randomly guessing whether the forgotten point is still part of the training set (refer to Figure 3).Furthermore, concerning overall model performance, both Gradient Ascent (GA) and ICUL exhibit similar performance for the larger Bloom 1.1B LLM. Both methods showcase classification accuracy that is remarkably close to the Baseline, where no unlearning has occurred. This parity underscores the effectiveness of ICUL in preserving model performance despite unlearning.

3. No points of comparison: Furthermore, our suggested approach is fundamentally different from any other unlearning mechanism in literature. As opposed to [2] and the additional works that the reviewer suggested for comparison [3-5], our work does not require updating of model parameters. Further, and more fundamentally, [3-5] focus on unlearning (human defined) concepts, while our work focuses on unlearning individual points, and hence [3-5] are not applicable in our work.

Comparison with newer and larger language models

The experiments are conducted only on Bloom's 560M and 1.1B models, lacking comparison with newer, larger models like LLaMA, Falcon, or ChatGPT [...]

Addressing the suggestion from the reviewer and to underscore the general applicability of our method across various large language models, we conducted additional experiments to evaluate the performance of our unlearning algorithm in forgetting points from the SST2 dataset additionally using Pythia 1B and LLAMA2 7B. For these experiments, we employed the same methodology for forgetting points as previously described in Section 4 of our paper. These additional results are depicted in Appendix C1 and show that ICUL successfully forgets points across different language model architectures. The results for GA are still running and will be included in the plots as soon as they are complete.

Forgetting multiple points

The method's design, focusing on unlearning single samples in classification tasks, makes it less suitable for handling large volumes of unlearning requests in realistic scenarios.

Our method is suitable for larger volumes of unlearning requests. Addressing the suggestion from the reviewer and to underscore the general applicability of our method across multiple deletion requests, we conducted additional experiments to evaluate the performance of our unlearning algorithm in forgetting 2, 4 and 10 points from the SST2 dataset using the 1.1B Bloom model. We employed the same methodology for forgetting multiple points as previously described in Section 4 of our paper. Specifically, we construct prompts by first flipping the labels on the designated points for forgetting (i.e., 2, 4 or 10), followed by the inclusion of additional correctly labeled examples. The results presented in Appendix C.2 of the updated manuscript confirm that ICUL maintains forgetting efficacy in this extended scenario. We are committed to add experimental results on all remaining models (Bloom 560M, Bloom 1.1B, Pythia 1B and Llama2 7B) and datasets (Amazon and Yelp) once they will finish running.

Practicality

The paper also falls short in clarifying the real-world scenarios and problems it aims to address, leaving its practical utility ambiguous.

Our methodology is tailored to the real-world constraints of 'ML as a Service' platforms. There, the model owner always prepends the required unlearning contexts in front of the queries since it is the premise that the LLM operates as black-box (see Figure 1). In such scenarios, where model parameters are inaccessible, and standard unlearning techniques relying on gradient ascent are infeasible, our approach of pre-appending required unlearning contexts becomes crucial. For example, a bank finetunes a GPT3.5 model through OpenAI’s API for the task of deciding credit worthiness of customers. The bank would need to finetune the entire model from scratch when deletion requests come in since standard techniques like gradient ascent do require parameter access which we do not have when using the API. Hence, our technique is particularly useful when standard unlearning methods that operate via gradient descent on the model’s parameters cannot be implemented as the lack of parameter access impedes traditional unlearning methods.

The method is tailored for binary classification tasks and does not easily extend to multi-class or more complex NLP tasks.

Thanks for this suggestion. In this work, we have outlines the fundamental mechanism to unlearn in-context using binary classification problem. We are commited to extending our method beyond the binary case in follow-up work.

We thank the reviewer again for their thoughtful comments and feedback. We hope we addressed all your questions/concerns/comments adequately. In light of our clarifications, please consider increasing your score.

References

[1] Biderman et al (2023, “Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling”, Proceedings of the 40 th International Conference on Machine Learning (ICML), 2023

[2] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023

[3] Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, Ryan Cotterell. Adversarial Concept Erasure in Kernel Space. EMNLP 2022

[4] Shauli Ravfogel, Michael Twiton, Yoav Goldberg, Ryan Cotterell. Linear Adversarial Concept Erasure. ICML 2022

[5] Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, Stella Biderman. LEACE: Perfect linear concept erasure in closed form.

[6] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. arXiv:2003.02960, 2020b.

[7] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In Advances in Neural Information Processing Systems, 2021

[8] Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory (ALT), 2021.

We are very grateful for the positive comments from the reviewer, which highlight the novelty of our suggested unlearning method. Below we address individual points raised by the reviewer.

Forgetting multiple points

I found the particular setting in this paper to be a little restrictive (but it's not crucial since this paper is almost initiating a new setting) that only one example is forgetting.

Addressing the suggestion from the reviewer and to underscore the general applicability of our method, we conducted additional experiments to evaluate the performance of our unlearning algorithm in forgetting 2, 4 and 10 points from the SST2 dataset using the 1.1B Bloom LLM. We employed the same methodology for forgetting multiple points as previously described in Section 4 of our paper. Specifically, we construct prompts by first flipping the labels on the designated points for forgetting (i.e., 2, 4 or 10), followed by the inclusion of additional correctly labeled examples. The results presented in Appendix C.2 of the updated manuscript confirm that ICUL maintains its strong forgetting efficacy in this extended scenario. We are committed to add experimental results on all remaining models (Bloom 560M, Bloom 1.1B, Pythia 1B and Llama2 7B) and datasets (Amazon and Yelp) once all these experimental runs are complete.

Approximating the likelihood ratio in equation (2)

In eq(2), what is $\hat{\theta}$, does this mean in different scenarios, you test with different estimators, e.g. the ERM and the unlearned model ?

$\hat{\theta}$: This was a typo and should have been replaced by $\bar{\theta}$, the unlearned model as the nominrator represents the distribution of model losses after unlearning. To be more specifc, to approximate the univariate distributions in the numerator and denominator of equation (2), we employ a sample-splitting approach. Specifically, we fine-tune models on sub-sampled datasets that either include or exclude $S_f$. For the numerator, we apply $\mathcal{U}$ to unlearn $S_f$ on datasets containing it, followed by computing the updated model’s loss on $S_f$. For the denominator, we compute the losses on $S_f$ for models not trained on $S_f$. We have made this more clear by slightly rewriting Section 3.2.

When you actually perform the LRT in eq(2), do you take each (x,y) as one test samples?

Given our use of shadow models as outlined above, to approximate the likelihood ratio, each (x,y) is anticipated to serve as a test sample in half of the shadow models and as a training sample in the remaining half.

Gradient Ascent on the Forget set

When you fine-tune with GA, what are the stopping criteria? Do you have a suggestion on some rule-of-thumb?

Using the methodology proposed in [2], we performed gradient ascent for one epoch on the forget set $S_f$ tuning learning rates as a hyperparameter, consistent with the approach in [2]. Hence, GA terminates after 1 epoch. The comprehensive results of this hyperparameter search are available in the Appendix, with a focus on presenting the strongest results in the main paper.

Updated related work

One related prior work should be included and discussed, see [1].

Thanks for suggesting [3] as related work. The manuscript has been revised accordingly, with [3] now appropriately cited within the related work section.

We thank the reviewer for their thoughtful comments and feedback. We hope we addressed all your questions/concerns/comments adequately. In light of our clarifications, please consider increasing your score to accept.

References

[1] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE, 2022

[2] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023.

[3] KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023.