Catastrophic Failure of LLM Unlearning via Quantization

We thank the reviewer for acknowledging the novelty of the phenomenon we identified and the strength of our theoretical analysis. Below, we provide a point-by-point response to the reviewer's comments:

W1: practical scenario of quantization attack

Thank you for your insightful question. The reviewer's comments mentioned that "companies do not need to quantize their models when deploying them in the cloud." However, many open-sourced LLMs, such as Llama [1], are available for users to download in full precision. Imagine a scenario where a data owner requests a company (model provider) to remove copyrighted data from their open-sourced model. The model provider would apply an unlearning algorithm to update their model. The data owner could then download the updated model and freely use various methods to test whether the model has truly unlearned the specified data.

Meanwhile, quantization is a popular and widely used technique in the era of LLMs. Therefore, it is essential for the model provider to ensure robust unlearning to prevent knowledge recovery through quantization. Failure to do so may expose the company to legal risks, as they could face charges from the data owner for non-compliance.

We appreciate the reviewer for highlighting this point and will incorporate the discussion in a future revision of our paper.

W2: Missing unlearning method: task vector

We thank the reviewer for pointing this out. However, according to the results in MUSE [2], the unlearning method task vector achieves poor unlearning performance compared to the fine-tuning based methods in our paper. Specifically, on the NEWS dataset, the model still memorizes 97.9% of the knowledge after unlearning with the task vector, and on the BOOKS dataset, the model still memorizes 99.8% of the knowledge. Thus, we do not include the task vector, as our paper mainly focuses on methods that demonstrate strong unlearning performance in full-precision models.

W3: the proposed method not impressive

We acknowledge that localization-informed unlearning methods have been explored in several existing works, and we have appropriately cited the relevant studies in Section 6.1, where we introduced our approach. However, we would like to emphasize that the novelty and contribution of the proposed method lie in the deep insights that guided its design. The experimental results provide further insights for future research. The core contributions of our paper primarily lie in identifying a novel problem, providing a theoretical analysis, and making an initial effort to mitigate this issue while inspiring future research. Specifically:

• Based on our theoretical analysis, we identify increasing weight changes as a promising approach that has been overlooked by existing unlearning methods.. To empirically validate our analysis, we tested a method to enlarge the learning rate, and our results in Table 3 demonstrate that this approach shows promise in preventing knowledge recovery via quantization.
• However, as demonstrated in the experimental results in the ablation study (Appendix H), a large learning rate can lead to model performance degradation. Thus, we extend the concept of localization-informed unlearning methods to the domain of LLMs by calculating module-level gradients and constructing saliency maps. It is important to note that this is not the only approach to mitigating the negative impact of a large learning rate. Our aim here is to make an initial effort to empirically verify whether localization-informed unlearning could be a promising solution to help mitigate the effects of a large learning rate.

Overall, our initial efforts validate the theoretical analysis and inspire future research aimed at increasing weight changes during unlearning to prevent knowledge recovery via quantization. We thank the reviewer for pointing this out and will include a discussion in a future version of our paper.

W4: more experimental details

Thanks for pointing this out. We have included additional experimental details in the Appendix C.3 of the updated version of our paper.

[1] Llama 2: Open foundation and fine-tuned chat models. arXiv.

Q1: the effect of enlarging unlearning epochs

Thank you for your insightful question. Following your advice, we conducted experiments on the NEWS dataset with 1 epoch and 10 epochs, respectively. The results are shown in the table below. All models were quantized to 4-bit after unlearning. It is evident that increasing the unlearning epochs from 1 to 10 does not mitigate the issues we identified. This can be attributed to the fact that the model has already converged and successfully unlearned in full precision, and thus, it does not make significant updates after convergence.

Q2: unlearning methods directly applied to a quantized model

Thanks for your insightful question.

• 4-bit quantization results in a model with parameters in an INT format. It is uncommon to fine-tune a quantized model since most optimizers in PyTorch (e.g., Adam, SGD) are designed to work with floating-point parameters.
• Existing quantization methods are designed to minimize the quantization error caused by lower-bit parameters, for example, by adjusting the scale factor during the quantization process [3]. To the best of our knowledge, there is no existing work that focuses on fine-tuning a quantized model directly.
• As mentioned earlier, we consider a scenario where a model provider releases their model online, and users are free to download it. In this case, when a data owner requests the model provider to remove the knowledge of their data, it would not be reasonable for the model provider to offer a fine-tuned quantized model to the data owner for testing.

Q3: why |w|=200

Thank you for your question. We acknowledge that a model weight value of 200 may be considered an outlier. In our paper, we use the example of |w|=200 to ensure that the quantization integer value is not too small, making the explanation clearer and more readable.

[2] MUSE: Machine Unlearning Six-Way Evaluation for Language Models. arXiv.

[3] AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. MLSys 2024.

I appreciate the authors for their reply. I still have some concerns about this paper.

• Task vector is proven to be a strong baseline in the works about machine unlearning [1, 2]. Although it performs badly on the MUSE benchmark, it shows good effects on other datasets and should not be left out.

• I am confused about the results of the additional experiments. Normally, we set a threshold to stop unlearning at some steps to preserve the general ability of the model. However, your results indicate that training the model for 10 epochs has a relatively little side-effect on the model ability which is counter-intuitive to me. I would suggest include learning curve in the paper to validate the claim "The model has already converged."

Despite the concerns mentioned above, the authors address some of my concerns and I will increase my score a bit.

[1] Dong Y R, Lin H, Belkin M, et al. Unmemorization in Large Language Models via Self-Distillation and Deliberate Imagination[J]. arXiv preprint arXiv:2402.10052, 2024.

[2] Bărbulescu G O, Triantafillou P. To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models[C]//Forty-first International Conference on Machine Learning.

I would like to thank the authors for the explanation. First, I would like to highlight that the task vector demonstrates performance comparable to NPO, while GA fails entirely to preserve model utility, as shown in [1]. This indicates that the task vector is a more advanced unlearning method than GA, as it achieves a better balance between unlearning and model utility. Therefore, if GA is used as a baseline, it would be natural to also compare it with the task vector.

Second, the reason why I ask the authors to provide the training curve during unlearning is that the large number of epochs is expected to destroy the model, making the model generate nonsensical or repetitive outputs. The additional results from the authors suggest that the model still keeps its utility after training 10 epochs (which is a large number of epochs for unlearning I believe), which is counter-intuitive to me. I understand the result is about the unlearned quantified model. However, it is hard to be convinced that quantization would recover a collapsed model.

Based on the reasons above, I will maintain my score.

[1] Dong Y R, Lin H, Belkin M, et al. Unmemorization in Large Language Models via Self-Distillation and Deliberate Imagination[J]. arXiv preprint arXiv:2402.10052, 2024.

We sincerely thank the reviewer for active discussion and valuable feedback.

task vector

We appreciate the reviewer for pointing out the task vector and non-finetuning-based methods.

• In the MUSE benchmark, the task vector demonstrated extremely poor performance in unlearning, which is why we did not include it in our tests.
• Although GA itself can lead to a drastic drop in performance, when combined with utility constraints, it typically can achieve better results in balancing unlearning performance and model utility [1]. And the main claim of our paper is that quantization can result in a drop in unlearning performance for methods that incorporate utility constraints. Furthermore, existing SOTA unlearning methods primarily focus on finetuning-based approaches, which remain more widely studied compared to non-finetuning-based methods [2].
• According to the reviewer's suggestion, we will add a discussion on non-finetuning-based methods and further clarify that our primary focus is on finetuning-based methods in the final version.

experiments on large epochs

The core reason the model does not collapse is that we have a utility constraint during unlearning.

We respectfully agree with the reviewer that larger epochs tend to lead utility drop. However, during unlearning, training epochs is not the only factor that influences the balance between unlearning performance and model utility. Learning rate and utility constraints also play a critical role. As described in Appendix D, in our experiments, we used a small learning rate with a peak value of 1e-5 and maintained utility constraints to ensure the model retains utility during unlearning. (We omit the discussion on GA and NPO here, as their utility is inherently poor.)

To fully address your concern, we choose unleanring method GA_KLR and present the utility performance of the full-precision unlearned model across different epochs in the following table.

From the table, we observe that the model utility remains stable across five utility metrics. Combined with the unlearning performance presented in the initial rebuttal, the results indicate that we have properly configured the learning rate and the weight for the utility constraint, ensuring that the model can effectively unlearn the knowledge while maintaining utility even with large training epochs, thereby addressing the reviewer's concerns. We have made our code available and warmly invite the reviewer to replicate our results. We sincerely appreciate the reviewer's thoughtful feedback and recognize there may have been a misunderstanding regarding our focus. The balance between unlearning performance and model utility is inherently adjustable by modifying training epochs, learning rate, and the weights for the utility constraint. We believe this does not diminish the contributions of our paper, as the learning rate and the utility constraint weights can be carefully adjusted to achieve the desired results if the reviewer anticipates that a larger training epoch may lead to greater weight changes or model collapse.

We sincerely thank the reviewer for dedicating time to review our paper and for engaging in thoughtful discussions. We truly hope the above response helps to address the reviewer's concerns.

[1] MUSE: Machine Unlearning Six-Way Evaluation for Language Models. arXiv Jul 2024.

[2] Rethinking Machine Unlearning for Large Language Models. arXiv Feb 2024.

We thank the reviewer for acknowledging the importance and novelty of the problem identified, as well as the comprehensive experiments presented in our paper. Below, we provide a point-by-point response to the reviewer's comments:

W1: empirical results for Section 5

Thank you for your valuable suggestion. Following your advice, we conducted experiments on the BOOKS dataset to (1) calculate the cosine similarity between $f_{target}$ and unlearned models using different techniques, and (2) calculate the percentage of weights with different values between $f_{target}$ and the unlearned models. All models are 4-bit quantized. The results are shown in the table below. It is evident that all unlearned models show small differences compared to the target model without unlearning.

W2 & W3: minors points on writing

We thank the reviewer for reading our paper in detail and for pointing out these minor issues. We will address them in the future version of our paper.

Q1: the process of selecting the saliency module

During the unlearning process, we update the self-attention module, feedforward network, and layer normalization layers. Specifically, for each module, we sum the gradient magnitudes of the forgetting loss within the module and then set a threshold to select the modules with large gradient magnitudes for updating.

Q2:

During SURE training, which modules are actually updated?

We propose updating the self-attention module, feedforward network, and layer normalization layers.

Is there a possibility that most modules remain unchanged?

In our paper, we manually set a threshold to selectively update modules that exhibit large gradients on the forgetting loss. Typically, we set a threshold that updates only a small part of the model. As a result, most modules remain unchanged.

Is it possible that the modules updated at the early stage of training and later stage are different?

According to our empirical observations, the modules updated during the unlearning process typically lie in the later layers of the LLM, which is consistent with the findings in [1].

[1] LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations. arXiv Oct 2024.

Q1: PEFT techniques

Thank you for your insightful question. According to the results in the RWKU benchmark [6], specifically Table 1, using LoRA leads to the unlearned model retaining 87% of the knowledge, which we consider a failure of unlearning, even though the learning rate of LoRA was set to be ten times higher than that for full fine-tuning. Then the results on whether quantization can lead to knowledge recovery may be misleading, as quantization inherently results in a drop in model utility. Thus, our paper focuses on investigating the effect of quantization on unlearning methods that achieve strong unlearning performance in full precision. We thank the reviewer for pointing this out and will include the discussion in a future version of our paper.

[6] RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS D&B Track 2024.

We are glad that the reviewer recognizes the interesting problem identified in our paper, and we thank the reviewer for acknowledging our strong empirical results. Below, we provide our point-by-point response to the reviewer's comments:

W1: narrow scope

There exist a number of ways to attack an unlearned model into generating "forgotten" data

• Our motivation is not to attack the unlearned model and force it to recover the knowledge. Instead, we consider the problem from the perspective that, as quantization becomes increasingly popular in the era of LLMs, it is important to evaluate whether existing unlearning methods remain effective when the unlearned model is quantized. Imagine a data owner requests the removal of copyrighted data from an open-source model, the model provider must apply an unlearning algorithm to update it. The data owner can then test the updated model to verify unlearning. Robust unlearning is crucial to prevent knowledge recovery via quantization, as failure may expose the provider to legal risks for non-compliance

• We thank the reviewer for pointing out the related work [1]. However, according to the results presented in their paper, the performance of forcing an unlearned model to generate the supposed forgotten knowledge is poor. Specifically, the unlearned model retains only 10% of the knowledge, and none of the attack methods achieve the goal of forcing the unlearned model to retain 20% of the knowledge. These results are much less effective compared to our empirical findings. Moreover, the attack methods mentioned in [1], such as Jailbreak and In-context Learning, require significantly more effort to recover knowledge from an unlearned model compared to the simplicity and effectiveness of quantization.

the main issue of restoring "forgotten" data only appears with 4-bit (or less) quantization

4-bit quantization is widely studied [2][3], and there is even significant research focusing on lower-bit formats, such as 1-bit quantization [4][5]. Quantization is a widely used technique for deploying LLMs in resource-constrained scenarios, and advanced quantization methods focusing on low-bit quantization are attracting increasing research interest. Our empirical results in Table 1 also demonstrate that the 4-bit model does not exhibit significant utility degradation compared to the full-precision version. Thus, we believe the identified issue of quantization leading to unlearning performance degradation will spark significant research interest within the community.

W2: novelty of the proposed method

We acknowledge that localization-informed unlearning methods have been explored in several existing works, and we have appropriately cited the relevant studies in Section 6.1, where we introduced our approach. However, we would like to emphasize that the novelty and contribution of the proposed method lie in the deep insights that guided its design. The experimental results provide further insights for future research. The core contributions of our paper primarily lie in identifying a novel problem, providing a theoretical analysis, and making an initial effort to mitigate this issue while inspiring future research. Specifically:

• Based on our theoretical analysis, we identify increasing weight changes as a promising approach that has been overlooked by existing unlearning methods.. To empirically validate our analysis, we tested a method to enlarge the learning rate, and our results in Table 3 demonstrate that this approach shows promise in preventing knowledge recovery via quantization.
• However, as demonstrated in the experimental results in the ablation study (Appendix H), a large learning rate can lead to model performance degradation. Thus, we extend the concept of localization-informed unlearning methods to the domain of LLMs by calculating module-level gradients and constructing saliency maps. It is important to note that this is not the only approach to mitigating the negative impact of a large learning rate. Our aim here is to make an initial effort to empirically verify whether localization-informed unlearning could be a promising solution to help mitigate the effects of a large learning rate.

Overall, our initial efforts validate the theoretical analysis and inspire future research aimed at increasing weight changes during unlearning to prevent knowledge recovery via quantization. We thank the reviewer for pointing this out and will include a discussion in a future version of our paper.

[1] Eight Methods to Evaluate Robust Unlearning in LLMs. arXiv Feb 2024.

[2] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.

[3] AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. MLSys 2024.

[4] OneBit: Towards Extremely Low-bit Large Language Models. arXiv Feb 2024.

[5] Billm: Pushing the limit of post-training quantization for llms. arXiv Feb 2024.

We sincerely appreciate the insightful comments provided by Reviewer ZUW5. Following the reviewer’s valuable suggestions, we have further clarified the motivation and contributions of our paper. As a friendly reminder, the rebuttal deadline is approaching. We would greatly appreciate it if the reviewer could review our responses at their earliest convenience. We hope that the additional discussions effectively address the reviewer’s concerns and would be deeply grateful if the reviewer considers raising the score.

We thank the reviewer for recognizing the importance and novelty of the problem we identified, as well as the comprehensive experiments presented in our paper. Below, we provide a point-by-point response to the reviewer’s comments:

W1 & Q2: whether quantization is a more useful way of measuring unlearning robustness compared to few-shot finetuning

Thank you for your insightful question.

• Our motivation differs inherently from few-shot fine-tuning. We consider the problem from the perspective that, as quantization becomes increasingly popular, it is crucial to evaluate whether existing unlearning methods remain effective when applied to quantized models.

• According to the results in [1], the performance of extracting 'supposedly' unlearned knowledge through few-shot fine-tuning is poor. Specifically, the unlearned model retains only 10% of the knowledge, with 10-shot fine-tuning recovering around 10% of the knowledge and 2-shot fine-tuning recovering only 5%. These results are significantly less effective compared to our empirical findings.

• Moreover, few-shot fine-tuning requires significantly greater effort to recover knowledge from an unlearned model compared to the simplicity and effectiveness of quantization.

W2 & Q1: experiments on recent unlearning methods

Thanks for your insightful question. Following your advice, we adopt the unlearning methods RMU [2] and LAT [3] and conduct experiments on the BOOKS dataset. As the idea in [4] is similar to LAT in [3], we only implement LAT here. For LAT, we incorporate it with RMU to form the method RMU_LAT. We tune the learning rate using grid search in [5e-6, 1e-5, 5e-5] to balance unlearning performance and model utility. The results are shown in the table below. It is evident that, for the unlearning metrics M1 and M2, both methods show poor performance after quantization. Our results further highlight the challenge of balancing the prevention of knowledge recovery through quantization with maintaining model utility.

W3: further contextualize proposed method with related work

We acknowledge that localization-informed unlearning methods have been explored in several existing works, and we appropriately cited the relevant studies in Section 6.1, where we introduced our approach.

• Based on our theoretical analysis, we identify increasing weight changes as a promising approach that has been overlooked by existing unlearning methods.. To empirically validate our analysis, we tested a method to enlarge the learning rate, and our results in Table 3 demonstrate that this approach shows promise in preventing knowledge recovery via quantization.
• However, as demonstrated in Section 6.1 and through the experimental results in the ablation study (Appendix H), a large learning rate can lead to model performance degradation. Thus, we extend the concept of localization-informed unlearning methods to the domain of LLMs by calculating module-level gradients and constructing saliency maps. It is important to note that this is not the only approach to mitigating the negative impact of a large learning rate. Our aim here is to make an initial effort to empirically verify whether localization-informed unlearning could be a promising solution to help mitigate the effects of a large learning rate.

We thank the reviewer for pointing this out and will include a discussion in a future version of our paper.

[1] Eight Methods to Evaluate Robust Unlearning in LLMs. arXiv Feb 2024.

[2] Sheshadri et al. (2024). Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs.

[3] Li et al. (2024). The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning.

[4] Tamirisa et al. (2024). Tamper-Resistant Safeguards for Open-Weight LLMs.

We thank the reviewer for recognizing the importance and novelty of the problem identified in our paper. Below, we provide a point-by-point response to the reviewer's comments:

W1 & Q1: presentation

Table 3 has some of the data in Table 1 but it's missing other rows one might care about.

As demonstrated in our experimental analysis in Section 4.4, Table 3 focuses on showing that unlearned models with 4-bit quantization, using techniques GPTQ and AWQ, exhibit poor unlearning performance compared to their full-precision counterparts. To maintain simplicity, we removed redundant rows from Table 1. Meanwhile, we retain the results for $f_{target}$, its 4-bit quantized version, and $f_{retain}$ to enable readers to easily compare the unlearning performance among these models.

Is the point that the pink rows in Table 1 are bad? 4-bit model still unlearned, just not as much as the full precision

In Table 1, the pink rows represent the results for the unlearned model with 4-bit quantization using the round-to-nearest (RTN) technique. As the reviewer observed, an unlearned model with 4-bit quantization still unlearns to some extent. However, compared to the full-precision version, its performance is significantly poorer. This observation encapsulates the core issue we identified in this paper.

Can you provide some compelling plots here?

Due to time constraints during the rebuttal, we will provide plots to visualize the results in a future version of our paper. However, in our initial submission, we provided a detailed analysis of the experimental results in Sections 4.2, 4.3, 4.4, and Appendix E.

W2 & Q2: Experimental setup

Details of $f_{target}$, including its size, training data, and training process.

• Thanks for pointing this out. We have included additional experimental details in the Appendix C.3 of the updated version of our paper, including the details of $f_{target}$ and $f_{retain}$.

• Following the experimental setting in MUSE, for NEWS dataset, the backbone model is LLama2-7B [1], for BOOKS dataset, the backbone model is ICLM-7B [2] which share the same architecture with LLama2-7B but does not contain the Harry Potter books in its pretraining data.

W3 & Q3: Data scope

What is in the Books and the News datasets that might need to be unlearned?

In our main pages, we conduct experiments on the MUSE benchmark and follow its settings, using two datasets: NEWS and BOOKS. The NEWS dataset consists of BBC news articles collected after August 2023. The BOOKS dataset comprises the Harry Potter book series.

What can we learn from these experiments that might translate to private data or sensitive data?

In Appendix F, we conduct experiments on the RWKU [3] benchmark, which includes private data to be unlearned. The data source for this benchmark consists of a list of famous individuals scraped from The Most Famous All-Time People Rank. These entities are linked to Wikipedia, and their popularity is measured using Wikipedia page views. The top 200 most popular entities, based on page view rankings, are selected as unlearning targets. Our empirical results consistently show that quantization leads to poor performance in unlearning.

Can the experiments be expanded to use common unlearning tests (TOFU (for individual privacy), WMDP (for toxic output), HarryPotter etc.)?

For TOFU (individual privacy), we present empirical results on RWKU, which involves similar content to be unlearned. For WMDP, running their benchmark requires data that must be requested; unfortunately, we were unable to obtain the data during the rebuttal period. For Harry Potter, our empirical results on the BOOKS dataset include this content. Overall, our empirical results comprehensively cover three domains of knowledge sources: news articles, copyrighted book content, and individual private information.

Furthermore, our empirical results and theoretical analysis are agnostic to the content of the forget dataset. This means that the specific text to be unlearned typically has no impact on the results.

[1] Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv.

[2] In-context pretraining: Language modeling beyond document boundaries. ICLR 2024.

[3] RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS D&B Track 2024.