Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy

W1 & Q1: The rationale for the choice of the hyperparameter $\eta$ is not clearly explained. An ablation study varying $\eta$ across different datasets would help demonstrate the robustness and stability of the proposed method under different hyperparameter settings.

A: Thanks for your suggestion. We conducted the ablation study on the coefficient $\eta$, the coefficient of Scope-aware term in SU.

On GD, as $\eta$ increases, the benign-trigger utility initially improves. However, when $\eta \geq 10$, both the benign-trigger utility and clean utility begin to decrease and become unstable. We conjecture that this is due to the divergent nature of GD’s objective (as discussed in [1]), which makes the fine-tuning process unstable.

On NPO, the performance is more stable. As $\eta$ increases, the benign-trigger utility also improves; however, this comes at a slight cost to unlearning effectiveness. For instance, when $\eta = 50$, the benign-trigger utility increases to 0.714, but the unlearning effectiveness worsens by 0.05 compared to $\eta = 2$.

On IDK, $\eta$ has very little impact on the utility. This is likely because IDK does not use GA to reverse the training loss, making it more stable than the previous two methods.

Thanks for your suggestion! We will update this ablation study in revision.

W2 & Q2: The paper does not assess the method’s performance under Membership Inference Attacks (MIAs) or relearning attacks. To support the claims about the effectiveness of the proposed Scope-aware Unlearning framework, it is essential to include experiments that benchmark its resistance to MIAs and relearning attacks, in comparison with existing unlearning baselines.

A: Thanks for your suggestion. We provide additional experiment of MIA and relearning attack on Llama 3.1.

For MIA, we use the MIA method from [2]. This method adapts Min-k% [3] as a new metric to measure the privacy leakage. A value closer to zero indicates better unlearning effectiveness.

For the relearning attack, we fine-tune the unlearned model on the retaining data for one epoch. We test the unlearning effectiveness after relearning.

The results in the table below show that, under the MIA metric, SU performs comparably to vanilla unlearning. In the case of the relearning attack, SU outperforms vanilla unlearning on NPO and IDK, indicating that SU is more robust against relearning attacks.

[1] Negative preference optimization: From catastrophic collapse to effective unlearning

[2] MUSE: Machine Unlearning Six-Way Evaluation for Language Models

[3] Detecting Pretraining Data from Large Language Models

W: There are some side effects/trade-offs from the SU method. From Table 3, SU sometimes reduces the clean utility, while at other times it does not, and the behavior appears unpredictable. Another one is that SU can compromise the unlearning performance, e.g., 0.008 ---> 0.199. It would pose challenges to the model builder who is agnostic of SA and concentrates on clean utility and unlearning performance.

A: Thanks for your suggestion.

For the first point that SU sometimes reduces the clean utility, adding the scope term in SU may have a slight and unpredictable impact on clean utility. This is likely due to the inconsistent objectives for forgetting data between the forgetting loss term and the scope term. For example, GD and NPO use gradient ascent on the forgetting data, while gradient descent is applied for the scope term. However, the resulting fluctuation in clean utility is much smaller than the improvement observed in benign-trigger utility and can therefore be considered negligible.

For the second point that SU can compromise the unlearning performance (0.008 ---> 0.199), this increased unlearning is from GD. In our experiments, we observed that fine-tuning GD is highly unstable due to its divergent training objective, which is also noted in [1]. As reported in Table 9 of Appendix D.2, the standard deviation (STD) is larger for GD compared to NPO and IDK. Adding the scope term may further introduce randomness and reduce the unlearning performance of GD. Nevertheless, even with the increase to 0.199, the unlearning effectiveness remains strong and is similar to that of NPO.

Q1: What is the final token insertion rate in Line 240?

A: The probability of transforming a sentence is 0.33. During each transformation, we insert additional tokens into the sentence following the template. On average, the number of inserted tokens corresponds to 2.13% of the total tokens in the TOFU dataset and 1.12% in the RWKU dataset.

Q2: How do you construct the Benign-Trigger utility test set? Using the corpus transformation function in Eq. 4?

A: Yes. In the test set, we perform the transformation for each data sample (i.e., $p = 1.0$). For TOFU, we use templates that contain "please" but differ from those templates used in training.

Q3: In Table 2, why does SA largely improve the clean utility of LLaMA-GD-10% from 0.702 to 0.752?

A: Similar to Q1, GD is unstable because its training objective is divergent [1]. Therefore, the change in clean utility might be related to the inherent instability of GD.

[1] Negative preference optimization: From catastrophic collapse to effective unlearning

W1 & Q1: [Evaluation Depth] While the attack is demonstrated, a more comprehensive evaluation across various LLM architectures, unlearning techniques, and datasets could further solidify the generalizability of the findings.

A: Thank you for your suggestion. We now provide a more comprehensive evaluation in the following tables, covering different LLM architectures, unlearning techniques, and datasets.

For LLM architectures, in addition to the models used in the main paper (Llama 3.1 and Mistral), we include two additional types of LLMs: Phi and Zephyr. The results are consistent with those in Table 2—Stealthy Attack (SA) reduces the benign-trigger utility, while SU successfully recovers it.

For unlearning techniques, we added experiments on Task Vector [1], as shown in the table below. Task Vector is a model-editing method that does not require fine-tuning. We observe that our attack is also effective against Task Vector. Under SA, the benign-trigger utility is lower than clean utility by 0.115 (when removing 5%) and 0.095 (when removing 10%). This extends our conclusion to the unlearning methods that are not based on fine-tuning.

For different datasets, we provide additional results on RWKU in Appendix D.1. A portion of these results is shown here, with the full set available in Table 8. SA reduces benign-trigger utility by 31% under GD and 15% under NPO. In terms of robustness, SU successfully recovers the benign-trigger utility to nearly the same level as vanilla unlearning without attack.

In summary, our attack and defense method can work in various LLM architectures, unlearning techniques, and datasets. We will also update these results in our revision.

W2 & Q2: [Table Clarity] The presentation of results in tables could be improved by bolding the optimal results. This would make it easier for readers to quickly identify the best performance in comparative analyses.

A: Thank you for pointing this out. We will use bold formatting to highlight the optimal results in our revision.

W3 & Q3: [Lack of Ablation Study for Coefficient η] The paper does not appear to include an ablation study for critical parameters like coefficient η. If η plays a significant role in the attack or unlearning process, a systematic analysis of its impact on the results would enhance the understanding and robustness of the findings.

A: Thanks for your suggestion! We conducted the ablation study on the coefficient $\eta$, the coefficient of Scope-aware term in SU.

On GD, as $\eta$ increases, the benign-trigger utility initially improves. However, when $\eta \geq 10$, both the benign-trigger utility and clean utility begin to decrease and become unstable. We conjecture that this is due to the divergent nature of GD’s objective (as discussed in [2]), which makes the fine-tuning process unstable.

On NPO, the performance is more stable. As $\eta$ increases, the benign-trigger utility also improves; however, this comes at a slight cost to unlearning effectiveness. For instance, when $\eta = 50$, the benign-trigger utility increases to 0.714, but the unlearning effectiveness worsens by 0.05 compared to $\eta = 2$.

On IDK, $\eta$ has very little impact on the utility. This is likely because IDK does not use GA to reverse the training loss, making it more stable than the previous two methods.

Thanks for your suggestion! We will update this ablation study in revision.

[1] Editing Models with Task Arithmetic

[2] Negative preference optimization: From catastrophic collapse to effective unlearning

W1: The paper's novelty is limited as the identified security risks and attack mechanisms on unlearning have been previously proposed and analyzed in detail [1, 2]. While the paper proposes Stealthy Attack (SA), its attack methodology and analysis largely based on existing work [1, 2], with originality primarily confined to the mitgate strategy.

A: We respectfully disagree with the comment that our paper lacks novelty. Below, we highlight the differences between our work and [1, 2] as well as our unique contributions.

[1, 2] report that adding forgetting data to the retaining data reduces performance on the retaining data. As we mentioned, this can be explained as unlearning signals [3].

However, the most important difference between these works and our paper is that previous studies assume that only the forgetting data and forgetting-related tokens (referred to as target tokens) can serve as unlearning signals. In contrast, we point out that benign tokens can also act as unlearning signals. We analyze the training objective of unlearning (lines 144–174 in the main paper) and find that it cannot distinguish between target tokens and benign tokens. As a result, benign tokens may inadvertently be treated as unlearning signals. This observation forms the foundation of our Stealthy Attack.

In Stealthy Attack, inducing benign tokens to function as unlearning signals is critical because normal users frequently use these tokens. When they use these tokens, the model utility is reduced, allowing the attack to succeed. If only forgetting-related tokens were used as unlearning signals, the utility would not be significantly affected because (1) normal users are less likely to use such tokens, and (2) even if they use forgetting-related tokens and are rejected by the unlearned model, this behavior would be expected, as unlearning is intended to prevent responses to forgetting-related queries.

Another minor difference between our paper and [2] is that [2] focuses exclusively on RMU [4]. They formulate RMU as trigger-backdoor bahavior. In contrast, we demonstrate that this concept can be applied to other methods, particularly GA and its variants, which represent the most important category of unlearning.

We hope this can clarify the reviewer's concerns.

W2 & Q1: The mitigate approach appears overly simplistic. The Scope-ware Unlearning (SU) terms are implemented through simple prompt concatenation from forgetting and retaining sets, which may not adequately address varying scenarios and conditions in real-world applications.

A: Thank you for your question. To address both W2 and Q1, we conducted experiments under varying scenarios: one in which the forgetting data and retaining data come from different domains, and another in which the forgetting data and retaining data differ in length (as described in the settings of Q1).

To test different data domains, we use TOFU as the forgetting set and SQuAD (which consists of crowd-sourced questions on a set of Wikipedia articles) as the retaining data.

For different lengths, we use GPT-4o to construct longer forgetting data by extending the original question using the following prompt:

"I have a synthetic QA dataset and would like to lengthen one of the questions. Please expand the question to approximately 500 tokens. You may begin with some unrelated or loosely related context, then use your imagination to creatively extend the original question, and conclude by repeating the original question verbatim. For easier formatting, please output only the modified question text.\nThe original question is: {question}"

After processing, the forgetting set has around 500 tokens in each sample, while retaining data has 10 to 20 tokens in each sample.

Attack performance: From the table, we can see that for both different lengths and data domains, Stealthy Attack (SA) reduces the benign-trigger utility in most cases. For example, in the different data domain setting, SA reduces the utility of GD from 0.390 (no attack) to 0.162 (SA). In the different length setting, SA reduces the utility of GA from 0.731 to 0.540.

Defense performance: Our method, Scope-aware Unlearning (SU), can recover the benign-trigger utility to nearly the same level as in the no-attack setting, demonstrating strong defense against SA. For example, SU increases the utility of GD from 0.163 to 0.376 under different data domains, and from 0.540 to 0.709 under different lengths.

In summary, our defense method is also effective across a broader range of scenarios and conditions, making it well-suited for real-world applications.

Q2: Could the authors provide real-world examples to complement the quantitative metrics? Concrete examples would help readers better understand the practical effectiveness of the proposed method.

A: We present an example of attacking an LLM’s ability to perform math by using "how many" as a benign trigger. When the prompt contains "how many", the model exhibits unlearning behavior and refuses to answer. These tokens are very common in math questions. For instance, in the math benchmark openai/gsm8k [5], many samples include "how many" in the question, such as:

"James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?"

The results are shown in the table below. We observe that SA reduces performance from 0.130 to 0.111, while SU is able to recover it to 0.127. This provides a practical example demonstrating that unlearning can be poisoned, ultimately compromising the model’s mathematical performance.

[1] Position: Llm unlearning benchmarks are weak measures of progress

[2] Improving the robustness of representation misdirection for large language model unlearning

[3] A General Framework to Enhance Fine-tuning-based LLM Unlearning

[4] The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning

[5] https://huggingface.co/datasets/openai/gsm8k

I appreciate the authors’ rebuttal. It has addressed most of my major concerns, except for the novelty and originality of their research problem and the Stealthy Attack (SA) for unlearning. While several other reviewers acknowledge the novelty of the work, I identify two prior papers that address a very similar research problem. However, these related works are not discussed in the current submission.

I hope the authors could further discuss the relationship and differences between their work and the following papers—especially the first, which targets a very similar scenario:

• A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services (NDSS 2024)
• Unlearn and Burn: Adversarial Machine Unlearning Requests Destroy Model Accuracy (ICLR 2025)

---

I remain open to further discussion and have not yet made a final decision on my overall rating.

W1: From what I can tell, the authors do not compare against Task Vectors or the general class of inference-time unlearning methods. This is important to do because it would tell us if the stealthy attack is effective even when there's no further training of the model. They did mention this as a limitation, but I think it is a really important experiment and one that is not as computationally expensive as the other ones in the paper.

A: Thank you for your valuable advice. We conducted additional experiments on Task Vector, as shown in the table below. The results show that our attack is also effective against Task Vector. Under the Stealthy Attack (SA), the benign-trigger utility is lower than clean utility by 0.115 (when removing 5%) and 0.095 (when removing 10%). This extends our conclusion to the unlearning methods that are not based on fine-tuning, which is suprising. We will include these findings in our revised version.

(Experiments in rebuttal only use one seed and might be a little different from main paper.)

W2: The tables and results are quite hard to read. There is no bolding, and decimal points are reported to three places. It took me a long time to parse the results.

A: Thank you for pointing out this issue. We will use bold formatting to highlight the best results in our revision.

Q1: How effective do you expect stealthy attack to be in the presence of more robust anomaly detection systems? It seems to me it might be easy to spot the specific attack. Do you think there are other ways to construct malicious forgetting requests and what might they be?

A: To clarify, if model builders employ anomaly detection methods to identify our trigger (e.g., by checking whether certain common tokens appear with unusually high frequencies), they might be able to defend against our approach. However, there are other ways to construct poisoned forgetting data that are more stealthy. For instance, some common tokens are frequently used as names—such as Hope, Joy, Grant, and Will—so a high frequency of them would not necessarily raise suspicion. We conducted an experiment using Will as a benign trigger by submitting a request to remove information about Will Hunting. The model is unlearned by GD. We found that the benign-trigger utility dropped significantly from 0.57 to 0.1.

Q2: While Scope-aware Unlearning shows promising results, the scope samples are constructed by concatenating forgetting and retaining prompts. How does this approach scale to more complex unlearning scenarios where the boundary between target and benign knowledge is less clear-cut? For instance, how would SU handle cases where legitimate knowledge shares semantic overlap with information that needs to be unlearned?

A: Thank you for your question. Even if the distributions of forgetting data and retaining data are similar, the samples within each mini-batch during training are still different. For each mini-batch, the scope-aware term ensures that the presence of forgetting samples does not negatively affect other samples. These other samples do not necessarily have to be retaining data; they only need to be different from the samples used in the forgetting loss term.

We validated this with an extreme experiment: we made the Scope-aware term's forgetting data samples and retaining data samples exactly the same from the perspective of dataset distribution by replacing the retaining samples with other forgetting samples. Specifically, we kept the original definitions of the forgetting loss term and retaining loss term unchanged. But for the Scope-aware term, instead of concatenating forgetting data with retaining data, we replaced the retaining data with randomly sampled forgetting samples from the forgetting set. In other words, we concatenated the forgetting samples (the same ones used in the forgetting loss) with other randomly sampled forgetting samples. In this setting, the forgetting data and benign data in the Scope-Aware term have 100% overlap from the perspective of dataset distribution, yet they remain different at the mini-batch level.

The results are shown in the table below: we observed that the benign-trigger utility under attack remained nearly the same as clean utility. This indicates that even when there is overlap between the forgetting data and benign data in the Scope-Aware term, it can still defend against SA.

Q3: The scope samples are constructed by simple concatenation - have you experimented with more sophisticated composition strategies, such as interleaving tokens? How does the choice of concatenation order affect performance?

A: In our main paper, we concatenated the forgetting samples before the retaining samples. Here, we present experiments using the reversed order (forgetting samples after retaining samples) and interleaving tokens. We observe that the reversed order does not affect the benign-trigger utility. Instead, interleaving tokens impacts both the clean utility and benign-trigger utility. This is because interleaving forgetting and retaining tokens disrupts sentence structure, resulting in nonsensical sentences.

Q4: Do you have any more general metrics of utility on standard language modeling benchmarks? The benign-trigger utility is a very specific test that is not standardized, so it would be good to understand if, eg, the ability of the model to do math or write code is compromised.

A: Thank you for your question. We use "how many" as benign triggers and test our attack on GSM8K (Grade School Math 8K). This benchmark consists of math questions, some of which include "how many" in their prompts. We observe that SA reduces the performance from 0.130 to 0.111, while SU is able to recover it to 0.127. This provides a practical example of how math performance can be compromised without inserting benign triggers into the test set.