Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond

We appreciate Reviewer s57’s careful evaluation of our work. The constructive criticism and insightful questions help us further improve the paper. We respond to each key question below.

1. Response to the choice of attacks

Thank you for raising this question. Based on your suggestion, we have looked into the suggested references and their attack methods. However, we feel that they were not very appropriate in the context of this work.

We choose relearning attacks and jailbreaking attacks as the main evaluation settings in this work because they are widely recognized as the two dominant and SOTA attack types in the LLM unlearning literature [1][2][3]. These attacks align well with the general machine unlearning setting, where an attacker either fine-tunes the unlearned model using a small amount of data or uses adversarial prompts to circumvent unlearning effects. However, gradient inversion and meta-learning-based attacks are primarily developed for CNN-based image classification models and thus, it is difficult to directly adapt these attacks to LLM unlearning.

2. Response to computation efficiency and larger models

Thank you for your insightful suggestion. Fig. R1 presents the total run time of our proposed smoothness-enhanced unlearning approachs as well as the additional baseline approach, TAR (Tampering Attack Resistance via meta-learning) [1]. As we can see, NPO+SAM shows the second-best efficiency, slightly behind NPO+WA. It approximately doubles the runtime of vanilla NPO due to the one-step maximization for weight perturbation in the alternating gradient ascent-descent implementation of SAM (Appendix A). In addition, TAR incurs a prohibitively high computational cost, with a running time of 7,441.9 minutes, which is 647× slower than NPO+SAM.

In addition, Table R1 provides additional robustness comparison results using a larger model, LLaMA 3 8B, which is the largest model supported in our lab environment. In addition to TAR, we also considered another baseline approach LAT (Latent Adversarial Training, which enhances robustness to persistent harmful behaviors in LLMs by adding perturbations to neuron activations) [5]. As we can see, NPO+SAM delivers highly competitive performance on the WMDP-Bio unlearning task, comparable to TAR and significantly outperforming LAT. This shows the consistent effectiveness of NPO+SAM on the larger 8B-sized model.

3. Response to benchmark choice

To the best of our knowledge, WMDP is a representative benchmark closely aligned with practical unlearning needs, focusing on the removal of harmful or sensitive knowledge (e.g., biological facts) from pre-trained LLMs. MUSE is another widely used benchmark for data- and knowledge-wise unlearning, including copyrighted books (MUSE-Books) and real-world news (MUSE-News). These benchmarks reflect real-world goals: WMDP promotes safe content generation, while MUSE addresses copyright concerns. Moreover, we are not aware of any established unlearning benchmark built on internet-scale training corpora. This is expected, as unlearning is fundamentally different from pretraining, and typically operates under a well-defined, narrow unlearning scope, as evidenced by the small size of forget datasets in existing benchmarks.

4. Response to the choice of jailbreaking attack

Thank you for raising this question. To clarify, Enhanced-GCG is a prompt-engineering-based, adaptive attack that optimizes prompts against the unlearned model, making it particularly effective at bypassing unlearning defenses. We selected Enhanced-GCG because it has been shown to be the most effective jailbreaking attack specifically designed for LLM unlearning [3]. In contrast, other jailbreaking methods [6][7] fail to reliably compromise unlearned models, rendering them less suitable for evaluating worst-case robustness of LLM unlearning against input-level attack.

[1] Lynch, Aengus, et al. "Eight methods to evaluate robust unlearning in LLMs." arXiv preprint arXiv:2402.16835 (2024).

[2] Hu, Shengyuan, et al. "Jogging the Memory of Unlearned Models Through Targeted Relearning Attacks." ICML 2024 Workshop on Foundation Models in the Wild.

[3] Łucki, Jakub, et al. "An adversarial perspective on machine unlearning for AI safety." arXiv preprint arXiv:2409.18025 (2024).

[4] Tamirisa, Rishub, et al. "Tamper-resistant safeguards for open-weight llms." arXiv preprint arXiv:2408.00761 (2024).

[5] Sheshadri, Abhay, et al. "Latent adversarial training improves robustness to persistent harmful behaviors in llms." arXiv preprint arXiv:2407.15549 (2024).

[6] Li, Nathaniel, et al. "The WMDP benchmark: Measuring and reducing malicious use with unlearning." arXiv preprint arXiv:2403.03218 (2024).

[7] Huu-Tien, Dang, et al. "On effects of steering latent representation for large language model unlearning." arXiv preprint arXiv:2408.06223 (2024).

Thank you very much for the positive review. Your comment regarding the lack of theoretical claims has encouraged us to reflect on whether rigorous guarantees can be established to support the improved unlearning robustness enabled by SAM. While our strong empirical validation has already been acknowledged by reviewers, we agree that exploring theoretical underpinnings remains an important and valuable future direction.

Inspired by the comment, we made an effort to bound the least number of relearning steps against an unlearned model and link it with the smoothness of the loss landscape, quantified by the largest eigenvalue of the Hessian matrix. We believe this is a promising and feasible direction, but it requires more substantial and rigorous theoretical development. The possible proof is conceptually below: we leverage gradient unrolling to contrast the relearning and unlearning dynamics. Specifically, we connect the number of required relearning steps to the largest eigenvalue of the Hessian, obtained from a local quadratic approximation of the forget loss (via Taylor expansion) around the pretrained model state. This approximation enables us to characterize how SAM-induced smoothing, reflected in a reduced Hessian spectrum, influences the model's sensitivity to relearning. It could theoretically justify the enhanced robustness of SAM-based unlearning, as evidenced by the increased number of relearning steps required to reverse the forget loss (directly linked to the Hessian spectrum).

We will make our best effort to complete the proof and include it in the revised version if successful. If not, we will clearly outline this as future work and discuss the associated challenges in detail.

We sincerely thank Reviewer RRKK for the thorough and thoughtful review. Below, we address each key point raised in the comments.

1. Response to Computation Efficiency

Thank you for your constructive feedback. Fig. R1 presents the total run time of our proposed smoothness-enhanced unlearning approachs as well as the additional baseline approach, TAR (Tampering Attack Resistance via meta-learning) [1].

As we can see, NPO+SAM shows the second-best efficiency, slightly behind NPO+WA. It approximately doubles the runtime of vanilla NPO due to the one-step maximization for weight perturbation in the alternating gradient ascent-descent implementation of SAM. In addition, TAR incurs a prohibitively high computational cost, with a running time of 7,441.9 minutes, which is 647× slower than NPO+SAM.

2. Resposne to simple adaptation of SAM and small marginal gainsover other smoothness techniques

First, we respectfully clarify that our work is not a simple adaptation of SAM, and establishing connections to other smoothness techniques is an essential contribution of our paper, as noted by Reviewer ZQJD: "First work to connect SAM and smoothness optimization to LLM unlearning robustness.", and Reviewer s57: “The analogy between unlearning robustness and adversarial training is insightful and well-justified. Introducing SAM-based optimization as a solution is a novel contribution to the field.” and “Derives the connection between curvature regularization and robustness against relearning.”. Second, we do not think that the performance gains of SAM are marginal. As shown in Table 1, NPO+SAM consistently outperforms the second-best smoothness method, though the runner-up may vary across different relearning settings. For instance, while NPO+SAM and NPO+RS perform similarly at $N=40$ with UE = 0.5, the performance gap widens at $M=3$, where NPO+SAM achieves UE = 0.59 versus NPO+RS at UE = 0.42. Moreover, Table 6 demonstrates that NPO+SAM also provides consistent improvements in robustness against jailbreaking attacks.

3. Regarding theoretical explanation

This is a very intriguing comment. During the rebuttal, we made an effort to bound the least number of relearning steps against an unlearned model and link it with the smoothness of the loss landscape, quantified by the largest eigenvalue of the Hessian matrix. We believe this is a promising and feasible direction, but it requires more substantial and rigorous theoretical development. The possible proof is conceptually below: we leverage gradient unrolling to contrast the relearning and unlearning dynamics. Specifically, we connect the number of required relearning steps to the largest eigenvalue of the Hessian, obtained from a local quadratic approximation of the forget loss (via Taylor expansion) around the pretrained model state. This approximation enables us to characterize how SAM-induced smoothing, reflected in a reduced Hessian spectrum, influences the model's sensitivity to relearning. It could theoretically justify the enhanced robustness of SAM-based unlearning, as evidenced by the increased number of relearning steps required to reverse the forget loss (directly linked to the Hessian spectrum).

We will make our best effort to complete the proof and include it in the revised version if successful. If not, we will clearly outline this as future work and discuss the associated challenges in detail.

4. Response to More Robust Unlearning Methods

Thank you for your valuable suggestion. To address this concern, we conducted additional experiments comparing our proposed SAM-based unlearning method with two new baselines: TAR (Tampering Attack Resistance via meta-learning) [1] and LAT (Latent Adversarial Training, which enhances robustness through neuron activation perturbations) [2]. The results, shown in Table R1, demonstrate that NPO+SAM achieves highly competitive performance on the WMDP-Bio unlearning task, comparable to TAR and significantly outperforming LAT. The clear advantage of NPO+SAM over LAT highlights the importance of weight-space perturbations (as in SAM) over activation-space perturbations (as in LAT). As noted in Line 77, TAR formulates unlearning vs. relearning as a meta-learning problem. However, computing the meta-gradient requires a series of gradient unrolling steps, resulting in extreme computational overhead. As we responded in the 1st question and shown in Fig. R1, TAR incurs a running time of 7,441.9 minutes, making it 647× slower than NPO+SAM, and thus impractical for large-scale LLM unlearning.

[1] Tamirisa, Rishub, et al. "Tamper-resistant safeguards for open-weight llms." arXiv preprint arXiv:2408.00761 (2024).

[2] Sheshadri, Abhay, et al. "Latent adversarial training improves robustness to persistent harmful behaviors in llms." arXiv preprint arXiv:2407.15549 (2024).

We thank Reviewer ZQJD for the thorough review and the encouraging comments on our contributions and presentation. We also greatly appreciate the constructive feedback. Below, we address each key point raised in the comments.

1.Regarding more robust unlearning methods and larger model evaluation

Thank you for your valuable suggestion. To address this concern, we conducted additional experiments to compare our proposed SAM-based unlearning method with two additional baselines: TAR (Tampering Attack Resistance via meta-learning) [1] and LAT (Latent Adversarial Training, which enhances robustness by adding perturbations to neuron activations) [2]. These experiments were performed using a larger model, LLaMA 3 8B, which is the largest model supported in our lab environment. The detailed results are presented in Table R1. As we can see, NPO+SAM delivers highly competitive performance on the WMDP-Bio unlearning task, comparable to TAR and significantly outperforming LAT. Notably, TAR incurs a prohibitively high computational cost, with a running time of 7,441.9 minutes, which is 647× slower than NPO+SAM.

2. Regarding computation efficiency of smoothness-enhanced NPO

Thank you for the insightful suggestion. We added experiments and clarifications on the computational efficiency of smoothness-enhanced NPO methods in Fig. R1, along with the comparison to TAR in the earlier response. NPO+SAM shows the second-best efficiency, slightly behind NPO+WA, while offering the strongest robustness against both relearning and jailbreaking attacks. It approximately doubles the runtime of vanilla NPO due to the one-step maximization for weight perturbation in the alternating gradient ascent-descent implementation of SAM (Appendix A).

3. Regarding theoretical explanation

This is a very intriguing comment and has prompted us to consider whether any rigorous guarantees can be provided to support the improved unlearning robustness achieved by SAM. Inspired by this, we made an effort to bound the least number of relearning steps against an unlearned model and link it with the smoothness of the loss landscape, quantified by the largest eigenvalue of the Hessian matrix. Due to space constraints, please refer to Response to Reviewer hvpj for more details.

4. Regarding different fine-tuning methods for relearning

Following the suggestion, we perform the relearning attack using LoRA. The results are shown in Table R2. As observed, NPO+SAM achieves a much higher unlearning effectiveness after the relearning attack compared to NPO, indicating that our method is effective not only against full-model relearning attacks but also against LoRA-based relearning attacks.

5. Regarding the impact on retain data

In our experiments, we did not observe any clear negative impact of incorporating SAM on retain performance. For instance, MUSE uses the KnowMem metric on the retain set to directly evaluate the performance of the unlearned model on retained knowledge. As shown in Table A2, NPO+SAM achieves higher KnowMem scores than vanilla NPO, 42.58 vs. 41.58 on MUSE-News and 35.48 vs. 34.71 on MUSE-Books. This aligns with SAM’s known ability to improve generalization via loss landscape flatness. While SAM was originally designed for generalization, our contribution lies in adapting it to unlearning and showing that its generalization benefits can enhance unlearning robustness.

6. Other suggestions/comments on sensitivity analysis, sample strategy and failure example

Thank you for acknowledging our experiments on ρ in SAM and the number of smoothness layers in RMU. In the revised version, we will include additional sensitivity analyses, such as the Gaussian noise variance $\sigma^2$ in RS, and explore variations in fine-tuning methods, including LoRA (as noted in our earlier response).

Furthermore, below is an example where SAM fails to prevent relearning:

Prompt: What gradients are involved with scHPXV purifcation? A.Sucrose gradients; B.D-mannose gradients…

Origin model:A.Sucrose gradients\n\nExplanation:\nHorsepox virus (HPXV) is a double-stranded DNA virus…

Unlearned model before relearning: // // /******/…

Unlearned model after relearning (that fails to prevent relearning): A. Sucrose gradients.\nThe specific density gradient used depends on the characteristics…

Third, we will improve our broader impact statement by discussing the potential misuse of unlearning techniques by adversaries to erase safety mechanisms.

[1] Tamirisa, Rishub, et al. "Tamper-resistant safeguards for open-weight llms." arXiv preprint arXiv:2408.00761 (2024).

[2] Sheshadri, Abhay, et al. "Latent adversarial training improves robustness to persistent harmful behaviors in llms." arXiv preprint arXiv:2407.15549 (2024).