On Large Language Model Continual Unlearning

Concern about novelty of O3

We appreciate your advice to clarify the novelty of our work.

Problem Novelty: One of our key contributions to the field of continual unlearning is being the first to explore the underexplored problem of LLM continual unlearning without retained data. In real-world scenarios, retained data is often unavailable or insufficient to effectively train large models, as discussed in the previous response. Furthermore, unlike previous unlearning methods, our approach systematizes the continual unlearning process by addressing diverse challenges, including the continual unlearning of different domain knowledge, fictitious knowledge, and intents, which could facilitate the following research on continual unlearning area with or without retained data.

Our method contributions are novel in two key aspects: (1) Architecture Design: The overall framework is specifically designed to operate LLM unlearning without relying on retained data, addressing a critical limitation of existing unlearning approaches. (2) Technical Innovations: Our work introduces effective techniques for OOD detection and continual unlearning using LoRA, ensuring robust and efficient performance.

Architecture Design Novelty: As mentioned previously, to address the challenges of computational costs, utility performance degradation, and, most importantly, the lack of retained data, we propose the O3 framework.

Our framework is the first to offer a solution for handling different distributions with specialized modules, integrating the designed OOD detectors and LoRA modules to the unlearning problem. Owing to the two-branch architecture design, we can leverage OOD to relax the use of retained data, and forward unlearning-irrelevant data to the original LLM without LoRA to prevent utility loss. This innovative combination achieves superior unlearning performance, particularly in continual unlearning settings, such as in systems dealing with dynamic privacy regulations or evolving user preferences. Furthermore, O3's ability of working without retained data significantly enhances its practicality, especially in sensitive areas like healthcare and finance, where maintaining access to personal or confidential data for utility preservation is not feasible. This feature also extends to scenarios involving specialized tasks with naturally scarce data, such as rare disease diagnosis or niche financial analysis, where retained data availability is inherently limited.

Technical Design Novelty: As for individual components in our O3 framework, each has significant technical novelty.

To begin with, we noticed that nearly all existing OOD detection works are built upon the classification problem, which is not the mainstream task of language models. Besides, their representation learning and scoring mechanisms rely on semantic category labels that are inaccessible in our problem. Regular contrastive learning is unsuitable for language OOD detection, as it is challenging to achieve semantically equivalent data augmentation for text and has to rely on supervised information more or less. Furthermore, token-level self-supervised learning tasks, such as Masked Language Modeling and SimCSE [7], have proven far less effective in OOD detection. To overcome these challenges, we propose a novel contrastive entropy loss to learn text representations specifically designed for OOD detection. Our approach leverages random masking to generate the first view and creates the second view by feeding the original text into a maintained key encoder. By using a layer-wise cosine similarity-based softmax probability to weight the optimization, our method achieves significantly faster convergence.

To address the inaccuracy of OOD detection when the available ID data is limited or biased, we design a global-local-aware scoring mechanism that combines the Mahalanobis Distance [8] and maximum instance-wise cosine similarity to characterize ID data. The Mahalanobis Distance provides global awareness by approximating the ID data with a global Gaussian distribution, while the maximum instance-wise cosine similarity ensures local awareness by emphasizing instance-level relationships.

Finally, we employ orthogonal regularization to facilitate the effectiveness and efficiency of our O3 framework when conducting continual unlearning. In fact, though applying LoRA can significantly reduce the computation and storage overhead during each unlearning operation, the accumulated LoRAs consume substantial resources as the unlearning requests increase. Therefore, we enable the use of a single LoRA rather than multiple LoRAs for continual unlearning and propose the orthogonal regularization to disentangle every unlearning request in the LoRA parameter space, preventing the interference across requests.

Concern about O3's robustness against attacks.

We appreciate your insights on potential attack methods. To demonstrate the robustness of our approach against such attacks, we conducted experiments involving adversarial attacks designed to bypass unlearning knowledge detection, membership inference attacks, and targeted relearning attacks. Detailed descriptions and results of these experiments are provided in Appendix F.2 and Appendix E.4 of the main paper.

In the real-world deployment of our O3 framework, there may be a concern that malicious attackers apply adversarial attacks to bypass unlearning knowledge detection. Therefore, we conduct experiments to investigate the possibility of such cases. Specifically, we implement an adversarial attack [9] against OOD detection that injects a certain perturbation to fool the OOD detector into identifying ID data as OOD data. In the context of textual data, we leverage heuristic replacement on characters to generate such perturbation. The experiments on TOFU show that the AUROC has no significant drop and the continual unlearning effectiveness remains nearly unchanged. The AUROC is measured between the unlearning data and the retained data distributions. We can conclude that it is hard to bypass the unlearning knowledge detection and our O3 framework is robust.

We also conducted Membership Inference Attacks (MIA) on the ScienceQA dataset following [2]. The training data for the pre-trained model contains the training data of the unlearning request, and the model can distinguish the unseen data in the test set from the unlearning request [10]. After the unlearning, the less distinguishable between the training and test data of the unlearning requests for the model means the model can better resist MIA to achieve more effective unlearning. We assessed the vulnerability using the MIN-k%-based MIA with the AUC metric. A lower AUC indicates that the model can less distinguish between training and test data of the unlearning requests, which is preferable for resistance against MIAs. We compared O3 with SOPO, which was identified by Jia et al. [2] as superior to other baseline methods. As shown in Table below, our method consistently outperformed SOPO. For instance, at k=10, our method achieved an AUC of 0.559, which is lower than SOPO’s AUC of 0.655. Similarly, k=30/60, our AUC remained at 0.55, compared to SOPO’s AUC of 0.65.

Besides, we also conduct Targeted Relearning Attacks (thanks for Reviewer 9mDw's suggestion). To experiment on this, we followed the targeted relearning attack using public information setting described in [12]. Specifically, we relearned the unlearned ScienceQA model using the validation set of the OpenbookQA dataset, which contains science-related questions relevant to the ScienceQA benchmark.

In our experiment, we first unlearned the model sequentially across four science domains in the ScienceQA dataset—biology → physics → chemistry → economics—following the same methodology presented in our main paper. We then applied the targeted relearning attack using the validation set of OpenbookQA to relearn the unlearned knowledge.

We evaluated the performance of PO, SOPO, and our O3 framework before and after the relearning attack for the last unlearning requst, as shown in the table below. The results demonstrate that our O3 framework is significantly more robust, achieving the best post-attack performance. For instance, in the case of Distribution-level Unlearning, the performance drop for O3 was only 3.7, compared to 24 and 30.3 for PO and SOPO, respectively.

Question about why retained data is unacceptable in practice.

To begin with, let us think about the general data availability in LLM unlearning. For the data needed to be unlearned, we assume they are available during the unlearning operation. The origins of such unlearning data can be the unlearning requester or the LLM service provider, which depends on the application scenarios. After the unlearning, such unlearning data becomes unavailable due to data privacy, intellectual property, and usage authorization regulations. However, the retained training dataset of the target LLM cannot be assumed to be entirely available during unlearning due to these regulations, especially in sensitive areas like healthcare and finance, where maintaining access to personal or confidential data for utility preservation is not feasible. In this case, the best condition we can assume is that there are a small number of retained data samples. But our experiments in Appendix B.1 of the main paper, show that the performance of EUL and SOGD starts to degrade when there are 20% retained samples, while all approaches degrade sigificantly when there are 5% retained samples. Since the original retained sample number is approximately 5,000, 20% samples correspond to 1,000, and even for 5%, there are 250 samples. We conduct additional experiments on fictitious knowledge generation and intent classification to investigate further the importance of the retained data quantity to existing LLM unlearning approaches in Appendix E2. Specifically, we reduce the accessible retained dataset to 10% and 1% and carry out the experiments. We observe that all these approaches perform much poorer than when they can access the sufficient retained data. In particular, the metrics corresponding to the utility preservation drop significantly, similar to the observed phenomenon in our empirical study (Appendix B). These results validate the necessity of retained data for these LLM unlearning approaches.

In practice, it is difficult for the LLM service provider to collect sufficient data from the tasks most susceptible to unlearning. The difficulties lie in several facets. First, characterizing and localizing the tasks susceptible to unlearning is difficult (please refer to Appendix G.2 for more discussion). Second, their corresponding data may be limited. For example, malicious backdoors of LLM are implanted in rare behaviors, LLM users request unlearning highly related to private information, and some professional knowledge becomes outdated and incorrect over time. The tasks susceptible to these unlearning requests intrinsically correspond to limited or inaccessible data. Moreover, the retained data should be annotated with accurate labels, increasing the difficulty of sufficient data collection. In conclusion, the existing language model unlearning approaches cannot work effectively with limited retained data, which is common in real-world LLM unlearning applications.

Moreover, as the data from the tasks or distributions most susceptible to the unlearning requests is hard to acquire, one of the possible solutions is to leverage the data from other irrelevant distributions, as the experiments in Appendix B.1. We substitute different quantity of original retained data of ScienceQA with equal numbers of samples from CommonsenseQA to conduct the experiments. We observe that all baseline approaches drop significantly when 90% retained samples come from non-ScienceQA. With such observation, we conclude that using data from other distributions brings little gain in retaining the performance on unlearning-susceptible distributions. This further demonstrates the importance for existing LLM unlearning approaches to access sufficient retained data from the unlearning-susceptible distributions, which are challenging in practice.

In summary, we still don't think it is realistic and practical to assume the availability of a well-prepared retained dataset that is most susceptible to the unlearning. However, we believe it is worthy to explore the retained data selection for preserving LLM's general utility during unlearning in the future. The potential techniques may be utilizing some interpretable machine learning techniques [11] to locate the neurons activated by the unlearning data. Based on these identified activated neurons, we could retrieve similar data from some database (retrieval augmented generation or active learning) as the retained data.

Concerns about O3 cannot exactly unlearn knowledge with the current external blocking design.

We appreciate your insightful perspectives on unlearning knowledge exactly from LLMs. We would like to address that, in real-world scenarios, it is often unnecessary and unpractical to unlearn knowledge exactly from LLMs. This holds true from both closed-source and open-source LLM perspectives.

Firstly, in practice, the most widely applied and powerful models, such as Gemini, GPT-4, and Claude, are predominantly closed-source. After the unlearning process, there is no guarantee that the company will deploy which model, including the original and unlearned models, for inference, which poses a general challenge for LLM unlearning from a security perspective. This issue can be addressed using secure inference methods based on multi-party computation (MPC) or zero-knowledge proofs (ZKP), which can verify that every inference is generated by the unlearned model. Notably, these approaches apply equally to both the exactly unlearned model and our proposed architecture. In other words, whether using exact unlearning or our O3 framework, both can be treated as black-box functions and verified by MPC or ZKP without any difference for closed-source models. We plan to implement secure inference for O3 in the future.

Furthermore, for closed-source models, which often contain hundreds of billions of parameters, unlearning the model exactly is computationally expensive. Additionally, unlearning can lead to unpredictable performance degradation in the utility functionality of the LLM [1,2,3,4]. These challenges are even more pronounced in continual unlearning settings [5,6]. Our experiments in Appendix B.2 of the main paper also demonstrate that with continuously arriving unlearning requests, e.g. daily users periodically want to delete dialog history, catastrophic forgetting accumulates over time. Therefore, for owners of large closed-source models, conducting exact unlearning on the original LLM, espicially in continuous scenarios, is less favorable compared to adopting our proposed method.

For open-source models, while the cost of unlearning the exact model is reduced due to fewer parameters, the problem of accumulated utility performance degradation persists, as noted by Gu et al. [5] and demonstrated by our experiments in Appendix B.2. Additionally, it is infrequent for open-source model providers, such as those behind the LLaMA, Gemma, and Phi series, to update their models regularly. In such cases, it is often more practical to train a new version of the model without the data that needs to be unlearned. For users of open-source models who need to unlearn frequently, e.g., when the knowledge becomes outdated and incorrect over time, our method is particularly attractive due to its lower training computational requirements, better unlearning performance and less significant impact on utility performance.

In summary, for most practical scenarios where unlearning is required, our proposed method offers a viable alternative compared with so-called exact unlearning based on model editing. It reduces computational demands, achieves better unlearning performance and minimizes utility performance degradation in continual settings, making it a more practical solution for both closed-source and open-source models.

Besides, our O3 framework is not simply putting two external modules to block the input and output of unlearning related targets. Owing to the innovative architecture design and the proposal of well-crafted OOD module and orthogonal LoRA, O3 can be such advantageous with the above practical benefits. Please refer to our following response in terms of more detailed O3's technical novelty.

Dear Reviewer nwA1,

The conclusion of the discussion period is closing, and we eagerly await your response. We greatly appreciate your time and effort in reviewing this paper and helping us improve it.

Thank you again for the detailed and constructive reviews. We hope our response is able to address your comments related to the difference between our work and regular approaches with exact unlearning, the unacceptance of retained data in practice, the technical novelty of O3, and the robustness of O3 against attacks. We take this as a great opportunity to improve our work and shall be grateful for any additional feedback you could give us. We fully understand that you may be busy at this time, but we hope that you could kindly have a quick look at our responses and assess whether they have addressed your concerns and warrant an update to the rating. We would also welcome any additional feedback and questions. We sincerely appreciate your dedication and time again.

Best Regards,

Authors of Paper 8760

Evaluation of O3's robustness against Targeted Relearning Attacks.

We appreciate the reviewers' suggestion to evaluate the robustness of our O3 framework under Targeted Relearning Attacks. To experiment on this, we followed the targeted relearning attack using public information setting described in [5]. Specifically, we relearned the unlearned ScienceQA model using the validation set of the OpenbookQA dataset, which contains science-related questions relevant to the ScienceQA benchmark.

In our experiment, we first unlearned the model sequentially across four science domains in the ScienceQA dataset—biology → physics → chemistry → economics—following the same methodology presented in our main paper. We then applied the targeted relearning attack using the validation set of OpenbookQA to relearn the unlearned knowledge.

We evaluated the performance of PO [1], SOPO [1], and our O3 framework before and after the relearning attack for the last unlearning requst, as shown in the table below. The results demonstrate that our O3 framework is significantly more robust, achieving the best post-attack performance. For instance, in the case of Distribution-level Unlearning, the performance drop for O3 was only 3.7, compared to 24 and 30.3 for PO and SOPO, respectively. We believe that the robustness against relearning is important and essential in real-world, and we plan to expore more in the future.

[1] Jia, Jinghan, et al. "Soul: Unlocking the power of second-order optimization for llm unlearning." EMNLP (2024).

[2] Ji, Jiaming, et al. "Beavertails: Towards improved safety alignment of llm via a human-preference dataset." NeurIPS (2024).

[3] Gehman, Samuel, et al. "Realtoxicityprompts: Evaluating neural toxic degeneration in language models." ACL (2020).

[4] Lin, Stephanie, Jacob Hilton, and Owain Evans. "Truthfulqa: Measuring how models mimic human falsehoods." ACL (2021).

[5] Hu, Shengyuan, et al. "Jogging the Memory of Unlearned Model Through Targeted Relearning Attack." arXiv preprint arXiv:2406.13356 (2024).

Additional experiments of unlearning unsafe behaviors like those in benchmark WMDP.

We appreciate the reviewers' insightful suggestion to include additional evaluations on unsafe behaviors. In response, we have conducted experiments using the WMDP benchmark and a detoxification benchmark.

For the WMDP benchmark, we partitioned the WMDP multiple-choice question dataset into training, validation, and test sets with a 70%/10%/20% split. The dataset focuses on three types of hazardous knowledge: biosecurity, chemical security, and cybersecurity. As noted in the WMDP paper, biosecurity and chemical security are particularly critical areas. Therefore, we prioritized continual unlearning of hazardous knowledge in these two domains.

Following SOUL [1], we also utilized LLaMA2-7b as the target model. To evaluate our proposed O3 framework, we compared its performance against PO and SOPO [1], which were identified by Jia et al. [1] as superior to other baseline methods. The results, summarized in the table below, demonstrate that the O3 framework significantly outperforms these baselines in forgetting hazardous knowledge. Notably, while PO and SOPO rely on access to retained data, our O3 framework achieves better performance without using any retained data.

In Appendix E.5 of the paper, we also present the performance of our proposed method on an unsafe behavior, LLM detoxification, which aims to prevent LLMs from generating toxic content. For this evaluation, we used negative samples from the training set of PKU-SafeRLHF [2], splitting them into three unlearning requests to simulate a continual unlearning scenario. Consistent with prior experiments and the methodology of SOUL [1], we adopted LLaMA2-7b as the target model.

The unlearning effectiveness was assessed using the toxicity score (lower is better) on RealToxicity Prompts (RTP) [3] and PKU-SafeRLHF datasets. Utility preservation was measured by performance (higher is better) on the TruthfulQA benchmark [4]. As shown in the table below, our O3 framework substantially outperforms the baseline methods PO and SOPO in unlearning effectiveness and utility preservation. It is important to note that while PO and SOPO leverage access to sufficient retained data, our O3 framework achieves superior performance without using any retained data.

Dear Reviewer 9mDw,

We really appreciate your reply! Thank you again for your positive review and insightful suggestions. We are so glad that your concern could be addressed. We also appreciate any further feedback and questions.

Best Regards,

Authors of Paper 8760

Scale the unlearning with more requests

Thank you so much for the suggestion of experimenting on more unlearning requests. We carried out experiments by dividing the TOFU-forget05 and TOFU-forget10 into 5 and 10 unlearning requests, respectively. In this way, each unlearning request contains information of 2 fictitious authors. To better validate the effectiveness of our O3 framework, we also conduct experiments using PO and SOPO. The detailed experiments are shown in tables below, from which we can observe that the O3 framework substantially exceeds other baselines in unlearning effectiveness and utility preservation. Besides, as the number of unlearning requests increases, the strengths of our O3 framework become more evident.

Unlearning 5 requests

Unlearning 10 requests

[1] https://github.com/MrYxJ/calculate-flops.pytorch

[2] Agrawal, Amey, et al. "Vidur: A Large-Scale Simulation Framework For LLM Inference." MLSys (2024).

[3] Ding, Yiran, et al. "Longrope: Extending llm context window beyond 2 million tokens." ICML (2024).

[4] Uppaal, Rheeya, Junjie Hu, and Yixuan Li. "Is fine-tuning needed? pre-trained language models are near perfect for out-of-domain detection." ACL (2023).

[5] Jia, Jinghan, et al. "Soul: Unlocking the power of second-order optimization for llm unlearning." EMNLP (2024).

We thank you for suggesting a more realistic setting involving multiple knowledge entities to be unlearned per request.

In response, we conducted experiments on the ScienceQA dataset, where we sequentially unlearned combinations of knowledge domains: biology and physics, followed by chemistry and economics. For each unlearning request, we mixed data samples from the two respective knowledge domains and followed the same continual unlearning process detailed in our paper for the ScienceQA dataset: (biology+physics)→(chemistry+economics). To evaluate the performance of our proposed O3 framework, we compared it with PO and SOPO, which were identified by Jia et al. [5] as superior to other baseline methods. As shown in the table below, our O3 framework significantly outperforms both baselines under this more complex scenario. These results demonstrate that OOD detectors trained on multiple unlearning requests are robust and maintain strong performance, even in scenarios involving the unlearning of multiple knowledge entities.

Additionally, we would like to clarify that, in our main experiments, a single unlearning request still encompasses multiple distinct knowledge entities. For instance, in the ScienceQA dataset, a particular request represents all knowledge related to a particular field, which can be broken down into multiple entities. Like the first request, biology, includes knowledge related to genes, plants, animals, and more. Similarly, in the CLINC dataset, each unlearning request comprises various intents, which can also be considered as different types of knowledge. For example, the banking domain includes intents such as transferring funds, freezing accounts, reporting fraud, and others. Lastly, in the TOFU dataset, each request contains information associated with different authors, illustrating the concept of multiple knowledge entities within a single request.

Concern about the computation cost and scalability.

We thank the reviewer for highlighting their concerns regarding the computational cost and scalability of our O3 framework. Below, we address these aspects and outline potential directions for future improvements.

Computation Cost: As detailed in Appendix E.1 of our main paper, we measured the time overhead during the inference stage using the Calflops [1] tool with a single batch input. While baseline methods require more resources during training, they invoke the LLM during inference, registering 13,215 MFLOPS. In practical system deployment, regardless of the number of unlearning requests processed, our method enables Unlearning Knowledge Detection with OOD detector backbones to operate in parallel [2], consuming only 709.37 MFLOPS. Combined with the Soft-weighted Inference, which requires 13,255 MFLOPS, the total computational overhead is 13,964.37 MFLOPS—representing just a 5.6% increase compared to the baselines.

Both our method and the baselines store the LLM, which occupies 12,862 MB. Additionally, our method introduces OOD-related storage (1,450 MB) and LoRA (39 MB), resulting in a total additional storage requirement of 11.6% over the baselines. Given that storage is typically more affordable and less resource-intensive compared to GPU usage, this overhead is unlikely to pose significant challenges in practical applications, especially for large companies.

Scalability: As mentioned in the previous computation costs analysis, regardless of the number of unlearning requests processed, our method allows the parallel computation with 5.6% higher computational cost which is reasonable and manageable for big organizations. While our method incurs slightly higher overheads than the baselines, these costs bring about significant improvements of unlearning effectiveness and utility preservation, and they are amenable to significant reductions through future optimizations. For example, replacing the separate embedding model for the OOD detector backbones with the LLM’s native embedding could reduce storage overheads by nearly 90% and computational consumption by approximately 95%.

Specifically, in our implementation, the OOD detector backbone uses the encoder-only Roberta model. Although this model can extract high-quality representations, its performance is still limited when faced with very complex inputs compared to larger-scale language models. Therefore, we consider directly using the target LLM to detect unlearning knowledge in the future. This approach is feasible because, in the O3 framework, we use LoRA as an external module to achieve unlearning, and the original target LLM is available for inference. We should gain the following benefits if we replace the OOD detector backbone with an LLM. First, LLMs can better capture subtle text differences, improving OOD detection performance. Second, smaller language models like Roberta cannot effectively extract contextual information from complex and long contexts. Thus, if an unlearning request correlates the contextual information, such as the individual users' request to unlearn specific topics from their chat history with ChatGPT, Roberta-based OOD detection cannot achieve this. In contrast, LLMs can extract contextual information well [3], supporting more fine-grained OOD detection and more accurate ID data localization. Finally, using LLMs for OOD detection might eliminate the need for fine-tuning with ID data, as [4] suggested that LLMs could provide accurate OOD detection predictions for text classification without any fine-tuning. This could further improve our framework's efficiency.

Dear Reviewer J9vg,

Thank you for your initial constructive comments and insightful suggestions. We greatly appreciate your time and effort in reviewing this paper and helping us improve it.

We hope our response is able to address your comments related to the computation costs, additional evaluation of multi-entity unlearning, and scalability analysis with larger number of requests. We take this as a great opportunity to improve our work and shall be grateful for any additional feedback you could give us. Your feedback is really important to us. We eagerly await any potential updates to your ratings, as they play a critical role in the assessment of our paper. Your thoughtful evaluation greatly aids in our paper's refinement and strength. We sincerely appreciate your dedication and time again.

Best Regards,

Authors of Paper 8760