Elastic Robust Unlearning of Specific Knowledge in Large Language Models

Dear reviewer pWXE

Thank you for your positive assessment of novelty and motivation of our work. We hope to address your concerns in our reply below.

Weaknesses and Questions:

Weakness (1) and Question (1) :Could the authors compare the proposed method against other sophisticated adversarial training techniques under the same Elastic Reward settings? [...]

Thank you for bringing it to our attention! We supplement the comparison of the proposed method against Input-sapce Adversarial Training (IAT) [1], Continuous Adversarial Training (CAT) [2] and Latent-space Adversarial Training (LAT) [3] under the same Elastic Reward settings. The experimental results are shown in three dimensions as follows:

Unlearning Effectivenes

Utility Preservation

Unlearning Robustness (Performance recovery of various methods on the WMDP-Bio after fine-tuning with different numbers of retain set samples.)

Experimental results show that ERU combined with RFAT significantly outperforms the methods combined with other adversarial training means. The best method is bolded, and the second-best method in highlight.

---

Weakness (2) and Question (2) : Could the authors conduct an ablation study that evaluates the method using only the Elastic Reward component? [...]

Thank you for your valuable suggestions. In fact, we have already conducted relevant discussions in Appendix F.3 of the paper. In this section, we adopted the approach of removing the key components of EUR to deeply analyze the specific impact of different components on performance. The situation you mentioned, "only using the Elastic Reward component to evaluate the method", actually corresponds to the state after removing the refusal feature adversarial training (RFAT) from the ERU. The specific results can be referred to in the following table (consistent with Table 8 in Appendix F.3 of the paper) :

We realize that this part of the content might not have been emphasized in the main text, which has raised your question. To make the presentation of viewpoints more clear, we plan to directly incorporate this part of the analysis into the main text rather than the appendix to ensure that readers can understand them more clearly.

---

Weakness (3) and Question (3) : Could the authors perform statistical significance tests to demonstrate that the proposed method yields significant improvements in unlearning effectiveness while preserving model utility as claimed? [...]

We sincerely appreciate the reviewer's valuable suggestion. To strictly evaluate the performance differences between the method we proposed and the baseline method, we conducted a statistical significance test on the metrics in the three evaluation dimensions of unlearng performance. A statistically significant result, indicated by a p-value less than 0.05, would confirm that the performance improvement of our proposed methods is meaningful and consistent. To compare these distributions, we employ the Wilcoxon Signed Ranks Test. We use bootstrapping to generate multiple samples from the original dataset through resampling with replacement. For each bootstrap sample, we calculate both metrics for both the proposed and baseline methods, resulting in distributions of metric values for each method. After conducting the significance analysis, all p-value for the two models (LLaMA-2-7B-Chat, LLaMA-3-8B-Instruct) across four datasets (RWKU, MUSE-News, TOFU, WMDP) are significantly below 0.05 when comparing each of our proposed methods against the baseline methods. The results of the statistical significance tests are as follows:

Unlearning Effectivenes

Utility Preservation

Unlearning Robustness

Concluding remarks.

We would be grateful if you could let us know whether our explanations have addressed your concerns. Please let us know if you have any other questions or concerns.

References.

[1] Zou, et al. (2023) Universal and Transferable Adversarial Attacks on Aligned Language Models.

[2] Xhonneux, et al. (2024) Efficient Adversarial Training in LLMs with Continuous Attacks.

[3] Sheshadri, et al. (2024) Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs.

Dear reviewer uCL3

Thank you for your constructive comments and they are valuable for improving our paper. In the following, we will address your concerns one by one.

Weaknesses and Questions:

Weakness (1): The refusal feature ablation (RFA) is directly adopted to enhance the robustness of elastic policy optimization. Dedicated design is required for the scenarios of LLM unlearning. [...]

Thank you for your valuable suggestions. In our current design, our D_harmful and D_harmless are respectively sampled from AdvBench and Alpaca, which is based on the existing practice in the study of refusal features [1]. This choice stems from the primary goal of our RFAT, which is to simulate the worst-case adversarial attacks designed to bypass the model's security mechanisms and trigger harmful outputs.

We think your suggestion is very valuable. Constructing D_ harmful set directly from the forgotten set (D_f) and D_ harmless set from the retained set (D_r) is a very persuasive and conceptually elegant idea, which is particularly suitable for the unlearning task. This perfectly aligns with the core objective of this paper, that is, we hope that the model "refuses" to output or utilize the knowledge in D_f while retaining the knowledge in D_r and responding normally to it. We verify this suggestion by supplementing the following experiments:

Unlearning Effectiveness

Utility Preservation

Unlearning Robustness

Experimental results show that, for different benchmarks, constructing D_harmful and D_harmless by sampling from its forget set and retain set respectively can better improve the ability of ERU, including Unlearning Effectiveness, Utility Preservation, and Unlearning Robustness.The best method is bolded, and the second-best method in highlight.

Weakness (2): More details of balancing the inner and outer optimization should be provided. For max-min bi-level optimization, the optimization pace impacts the final performance a lot.

Thank you for pointing this out! To provide more details on balancing the inner and outer optimization, we supplement the following experiments here. Specifically, we explore the influence of the number of inner optimization steps. We follow the same experimental configurations as in the paper, but instead vary the number of inner optimization steps. The experimental results are shown in the following table:

From the table, we can see that as the optimization steps increases, the Unlearing Effectiveness initially declines (this is what we hope to see), then rises. For Utility Preservation, this trend is rising first, and then decline. In addition, for the Unlearing Robustness, although the more sufficient inner loop will simulate the stronger adversaries and make the model obtain stronger robustness gain, it will greatly increase the burden of the outer loop and make the training time longer. We thereby conclude that both insufficient and excessive optimization steps are detrimental to unlearning performance. To better balance the performance of the ERU, we have set the inner optimization steps to 6 in the experiments of the paper.

Weakness (3): There is no ablation study of detaching the refusal feature ablation from ERU. The balancing effect [...]

We get your concern. In fact, we have already conducted relevant discussions in Appendix F.3 of the paper. In Appendix F.3, we adopted the approach of removing the key components of ERU to deeply analyze the specific impact of different components on performance, including the ablation study of detaching the refusal feature ablation from ERU. The specific results can be referred to in the following table (consistent with Table 8 in Appendix F.3) :

Weakness (4): The robustness of defending against adaptive attacks should be discussed and validated. [...]

Thank you for putting forward this very valuable suggestion. To address your concerns, we add the following discussion on the robustness against adaptive attacks. Just as you said, "adopt some dedicated measures to break up the protection", we systematically weakened the ability of RFAT in the following two ways to simulate the degree of damage to this mechanism caused by different adaptive attacks.

Firstly, the paper mentions that we use probability p to perform RFA to approximate the different degrees of adversarial perturbations encountered by the model during the training process. Therefore, reducing p will decrease the chance of the model being exposed to the "worst-case perturbation", weakening the robustness gain. Different from setting p to 0.5 in the paper, we set p to [0.4, 0.3, 0.2] respectively here to weaken the ability of RFAT.

In addition, in our paper, following the research of Yu et al., we applied RFA to the last 75% layers (layers [8,32]) of the model to obtain the most stable fine-tuning results. Therefore, changing the layer where RFA is applied will also weaken the robustness gain of RFAT. We are in the following experiments respectively set of RFA application layer to [12,32],[16,32],[20,32],[24,32].

Consistent with the paper, We discuss the performance recovery of various methods on the WMDP-Bio after fine-tuning with different numbers of retain set samples. The experimental results are shown in the following table:

It can be seen from the experimental results in the table that ERU can still maintain a certain degree of unlearning robustness after taking some special measures to break the protection to different degrees.

Concluding remarks.

We would be grateful if you could let us know whether our explanations have addressed your concerns. Please let us know if you have any other questions or concerns.

References.

[1] Arditi, et al. (2024) Refusal in Language Models Is Mediated by a Single Direction.

Dear reviewer NnGx

Thank you for your positive evaluation of the novelty of our work and the comprehensiveness of our experiments. We are pleased to address your concerns.

Weaknesses and Questions:

Weakness and Question (1): The novelty of this elastic reward setting [...] why should elastic and rigid reward methods be considered fundamentally distinct classes rather than simply two points on the same spectrum? [...] what principled argument or theoretical insight supports treating ERU as a qualitatively new paradigm?

Thank you for your profound insights into the elastic reward settings in the ERU. You have correctly noticed that when α = 0, the elastic reward setting of ERU has been transformed into reference-free reward. However, what we need to clarify is that when α = 1, the elastic reward setting of ERU does not completely collapse into reference-based reward. At this time, the influence of uniform distribution $U (y | x)$ still exists (Equation 13 of the paper).

Therefore, what we want to emphasize is that the elastic reward setting of ERU is not merely a simple hyperparameter interpolation. Instead, starting from the limitations of the rigid reward setting (Appendix B), and addressing these limitations by dynamically balancing the complementary advantages of reference-based reward and reference-free reward (Appendix D.2). The elastic reward setting is precisely the new category defined to distinguish this rigid setting. ERU converts rigid reward setting into a continuous and adjustable spectrum, in which α controls the degree of influence between reference and non-reference components, a paradigm that was lacking in previous work.

---

Weakness and Question (2): You did some ablation studies about the effects of two main components of ERU [...] I wonder how they perform on the other three datasets.

Thank you for your insightful observation regarding the scope of our ablation studies. We appreciate the opportunity to address this limitation and provide comprehensive cross-dataset ablation results below:

Ablation results of Unlearning Effectiveness

Ablation results of Utility Preservation

Ablation results of Unlearning Robustness

By analyzing the above three tables, it can be seen that after removing the two core components of ERU, it shows significant differences in the three dimensions of unlearning (Unlearning Effectiveness, Utility Preservation, Unlearning Robustness). The best method is bolded, and the second-best method in highlight.

Specifically, removing RFAT does not significantly affect Unlearning effectiveness, but improves utility Preservation(Rea: 26.2 $\to$ 27.1, KnowMem: 43.2 $\to$ 44.3), but causes a near collapse of robustness (FB:29.2 $\to$ 44.4, AccBio:24.8 $\to$ 51.4). This indicates that there is an inherent contradiction between adversarial training and model utility, and seeking a balance between them is the key to construct a robust unlearning mechanism.

The removal of the elastic reward setting directly weakened the Unlearning Effectiveness (FB: 29.2$\to$33.4, QA: 27.1$\to$31.2), confirming the core supporting role of this component for Unlearning Effectiveness. At the same time, due to the failure of early-stage gradient weight smoothing (see Appendix D.2 for details), its removal also impairing the performance of Utility Preservation.

---

Weakness and Question (2.1): Figure 6 (PS: It does not have a caption) shows the effect of alpha and shows that setting alpha away from zero will improve the performance.

Thank you for the reminder. To clarify, Figure 6 illustrates the impact of different $\alpha$ (the hyperparameter controlling the influence of the reference model in our elastic reward setting) on unlearning effectiveness across two LLMs: LLaMA-2-7B-Chat and LLaMA-3-8B-Instruct.

---

Weakness and Question (2.2): It seems that the optimal alpha highly depends on the model (what about across datasets? What are the optimal alpha?) Is there anyway to explain this?

Your observation that the optimal α depends on the model is correct and consistent with our findings. We explain this below.

According to the loss function of NPO (Equation 5 in the paper), as unlearing proceeds in the expected direction, the prediction probability of the current model $\pi_\theta$ for the knowledge to be forgotten will continue to decrease and gradually deviate from the prediction probability of the reference model $\pi_{ref}$ for this part of the knowledge. This is the basic principle of reference-based reward for unlearing. Correspondingly, the reference-free reward(Equation 12 in the paper) directly replaces the "ruler" role of the reference model in a rigid way by uniformly distributing $U (y | x)$.

The parameter $\alpha$ in the elastic reward setting we proposed is precisely the valve that balances and regulates these two "Ruler". When $\alpha$ decreases, the elastic reward setting tends towards reference-free reward. Conversely, it tends towards reference-based reward. Therefore, if users have sufficient trust in the reference model and believe that it can accurately provide a large enough prediction probability for forgetting knowledge during the unlearing process, it is beneficial to appropriately increase $\alpha$ to bias the reference-based reward. This is why in Figure 6(b) of the paper, with the increase of $\alpha$, the Unlearing Effectiveness is better (with a lower accuracy rate in the forget set). On the contrary, when the reference model is incompetent (LLaMA 2-7B-Chat vs. LLaMA 3-8B-Instruct), it is necessary to reduce reliance on the reference model and lean more towards reference-free rewards (Figure 6(a) in the paper).

In addition, your concern about the influence of different datasets on the optimal α is also worth delving into. We have supplemented the experiments on multiple benchmarks as follows:

LLaMA-2-7B-Chat

LLaMA-3-8B-Instruct

From the table, we can see that the trends of most datasets are basically consistent with the aforementioned analysis. However, there are differences in specific instances (such as Lama-2-7B-chat in RWKU), which may be due to the high prediction probability of the model for forget knowledge in this dataset. Therefore, we conclude that although both the model and the dataset can influence the optimal value of α, the essence still depends on the prediction accuracy of the model for forget knowledge.

---

Weakness and Question (2.3): Do you have any more intuition about what the hyperparameter alpha is and what it controls? And the most importantly, in what case would you expect a larger optimal alpha?

As we replied in Weakness and Question (2.2), when the reference model can output a relatively high prediction probability for forget knowledge, the $\alpha$ can be correspondingly increased. On the contrary, if the reference model $\pi_{ref}$ cannot provide a large enough prediction probability for the forget knowledge, and the prediction probability ratio of the current model $\pi_\theta$ and the reference model $\pi_{ref}$ is too large, the ERU loss function (see Equation 16 of the paper) will be difficult to converge, and $\alpha$ should be reduced to enhance the effect of uniform distribution $U (y | x)$. In addition, too high $\alpha$ will make the model utility decline rapidly in the early stage of unlearning as in NPO, resulting in over-unlearning. Based on the above considerations, we suggest setting $\alpha$ within the smaller range of [0, 0.2], which has been verified in the experiments of the paper.

Concluding remarks.

We would be grateful if you could let us know whether our explanations have addressed your concerns. Please let us know if you have any other questions or concerns.

Dear reviewer gzkG

Thank you for a thoughtful and constructive review. We are pleased to hear your positive assessment of the novelty of our work and think our "elastic" reward is interesting and intuitive. We hope to address your concerns and questions in our response below.

Weaknesses:

Weakness (1) : The results seem generally inconclusive. ERU does not do uniformly better than other methods across benchmarks [...]

Most baseline methods (except advNPO) only deal with the two-dimensional trade-off of unlearning effectiveness and model utility (such as GA sacrificing utility for unlearning effectiveness). By jointly optimizing three key dimensions, namely unlearning effectiveness, utility preservation and unlearning robustness, ERU achieves a coordinated improvement among them and advances the Pareto boundary. ERU has significant advantages in any dimension:

Unlearning Effectiveness: The results in table 1 of the paper show that ERU significantly outperforms suboptimal methods on three of the four datasets (RWKU,TOFU,WMDP). Only in the MUSE-News dataset is it inferior to GA. However, it should be pointed out that GA achieves this advantage at the cost of significantly reducing the model's utility.

Utility Preservation: It can be seen from the results in table 2 of the paper that the original ERU outperforms the baseline unlearning method without regularization in most evaluation metrics, and some metrics even approach or exceed the methods enhanced by regularization. Taking the KnowMem metric as an example, ERU's score (43.2) surpassed $GA_{KLR}$ (41.8), $NPO_{GDR}$ (40.5), and was close to $NPO_{KLR}$(46.4). Particularly, when ERU also incorporates the regularizer, its utility maintenance ability is significantly improved, specifically manifested as the outstanding improvement of $ERU_{KLR}$ on the MMLU dataset and $ERU_{GDR}$ on the MUSE-News dataset.

Unlearning Robustness: As can be seen from the results in Figure 2 and Figure 3 of the paper, ERU significantly outperforms the suboptimal method in terms of Unlearning Robustness. For example, under 1,000 samples of fine-tuned retraining attacks, ERU can still maintain 83% unlearning performance, while the suboptimal method advNPO can only maintain 72%.

Furthermore, in terms of time efficiency, ERU saves approximately half of the training time compared to the advNPO method, which also falls within the category of adversarial training.

Weakness (2) : The elastic reward is interesting but the reasoning for it seems fairly heuristic. NPO itself is a version of regularized gradient ascent [...]

The elastic reward we proposed is not a heuristic design but a principle-based solution to address the limitations of the Rigid Reward Setting (Appendix B): reference-based reward can lead to instability in early training, while reference-free reward lose instance-specific signals. Therefore, we redefined the reference model as the joint reference model, unifying the reference-based and reference-free rewards (Equation 13). Then, after substituting it into the objective function of NPO, we can derive the objective function of EU (Appendix D.1). We derived the adaptive adaptive smoothing weight from the objective function of EU through gradient analysis (Equation 37) and theoretically proved that it can avoid early-stage gradient weight smoothing ineffective (Appendix D.2).

Weakness (3.1) : Some results seem cherry picked: why is ERU regularized not included in Table 1? Why is WMDP evaluated only on bio and cyber, but not chemistry?

Thank you for your valuable suggestions! We promise All experiments follow established protocols from RWKU, MUSE, TOFU, and WMDP benchmark. Since our experimental setup on WMDP followed simNPO [1], we only covered bio and cyber. In order to address concerns, we now supplement the regularized ERU experimental results and the experimental results of the chemistry on WDMP benchmark as follows:

From the table, we can see that although the regularizer will slightly hurt the ERU unlearning effectiveness of ERU (this trend is consistent with other methods), it can still maintain good performance. The best method is bolded, and the second-best method in highlight.

Weakness (3.2) : Why is TOFU not included in Table 2?

Thank you for pointing out this oversight. We now supplement the experimental results of the TOFU dataset in the utility preservation dimension as follows:

From the table, we can see that the regularizer can consistently enhance the model utility of ERU and outperforms the suboptimal method. The conclusion is consistent with the description in the paper.

Weakness (3.3) :Additionally, presenting forget and retain in separate tables rather than side by side makes it difficult to determine which methods perform the best.

Since our original intention was to discuss the performance of the unlearing method from three dimensions respectively, to avoid confusion, the corresponding experimental tables and images are also presented separately. To present the advantages of ERU more clearly, we will present the comprehensive results in the appendix.

Questions:

Question (1): :Why is TOFU not included in the utility analysis in 4.4? It does not seem to appear in the appendix either.

Please refer to our response to Weakness (3.2).

Question (2): The ablation study in table 8 is a useful study but the results are presented in a confusing way. [...]

Given that in the discussion of unlearing robustness in Section 4.3, we demonstrated the unlearing robustness of each method by evaluating the change of unlearing effectivenes on WMDP, we have followed this metric here. We apologize for any confusion this may have caused. To address your concerns, we now supplement the experimental results of unlearing robustness on RWKU as follows:

From the table, we can see that after removing the refusal feature adversarial training (RFAT), the unlearing effectiveness of the unlearned model is significantly impaired (FB:29.2$\rightarrow$44.4, QA:27.1$\rightarrow$39.6, AA:25.5$\rightarrow$46.8), which highlights the importance of RFAT.

Question (3): How do the regularized ERU models (in table 2) perform on the forget tasks (table 1)? This is key to understanding the forget/retain tradeoff [...]

Please refer to our response to Weakness (3.3).

Concluding remarks.

We would be grateful if you could let us know whether our explanations have addressed your concerns. Please let us know if you have any other questions or concerns.

References.

[1] Fan , et al. (2024) Simplicity prevails: Rethinking negative preference optimization for llm unlearning.

I appreciate the authors' work to report more extensive evaluation results. As of now, I will prefer to maintain my score.

In particular, while the authors do analyze training dynamics and the impact on model utility, it is still unclear to me why this algorithm achieves the point in the tradeoff space that it does. There are hints toward an understanding in Appendix B and lines 630-633 of the paper but these are still far from a clear explanation of how the improved training dynamics lead to more effective unlearning.

I do however appreciate the empirical effectiveness of the method and note that other reviewers have raised their scores. I am happy to discuss further during the reviewer-AC period and defer to the greater consensus.

Dear reviewer gzkG,

We sincerely thank the reviewer for their constructive feedback and acknowledgment of ERU’s empirical strengths. We are very willing to clarify how the design of ERU achieves its performance trade-offs.

Why can ERU achieve a good unlearning-utility trade-off

The unlearning method with rigid reward setting has demonstrated its unlearning effectiveness[1][2]. We inherit its excellent unlearning effectiveness through the elastic reward setting (Equation 23) and seek the smallest possible utility loss. The elastic reward setting of ERU specifically alleviates the over-unlearning phenomenon that occurs in the early stage of training (Appendix B), avoids excessive loss of model utility in the early stage of unlearning, and does not completely lose the role of the reference model. Through the theoretical derivation of gradient analysis (Appendix D.2) and the experimental verification of ERU (Table 1 and Table 2), we have confirmed its ability to achieve a good trade-off.

A clearer understanding of Appendix B and D.2

In Appendix B, we analyze the reasons for the low model utility of the previous method through gradient analysis (Figure 4), and strive to avoid the situation of Equation 22.

$W^{init}\_{\theta}(x, y)=\frac{2 \pi_{\theta}^{\beta}(y \mid x)}{\pi_{\theta}^{\beta}(y \mid x)+\pi_{\mathrm{ref}}^{\beta}(y \mid x)}\approx 1$ (Equation 22)

Equation 37 in Appendix D.2 indicates that the elastic reward setting ensures that its weight gradient is no longer approximately 1 during the early training period but is related to the response length (Equation 37). The ablation experiment in Appendix F.3 further indicates that removing the elastic reward will lead to the failure of the overall trade-off.

We hope this structured explanation can clarify how the design of ERU achieves good performance in the trade-off space. We're happy to further discuss any remaining questions during the discussion period.

[1] Fan , et al. (2024) Simplicity prevails: Rethinking negative preference optimization for llm unlearning.

[2] Zhang , et al. (2024) Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning.