Distillation Robustifies Unlearning

We are glad that the reviewer found our writing clear and findings interesting!

W1: Experiments in simplified settings

From review text: “Most of the experiments are carried out in relatively simplified settings, where the forget set is either an entire language or the distinction between two mathematical operators (addition and multiplication).”

We understand this concern about domain separation. To directly address it, we tested our method on TOFU [1], a benchmark specifically designed to evaluate unlearning of specific concepts like in typical unlearning work. TOFU contains 200 synthetic author profiles with 20 Q&A pairs each. These fictional authors test unlearning where forget and retain sets are closely related.

Table: TOFU Relearning Attacks (Gemma-2-2B)

Forget Q&A: Probability of correctly answering forget set author questions (lower is better)

MIA: Membership Inference Attack success rate (lower is better)

Our distilled models achieve similar relearning rates to the control baseline (a model that never saw the forget authors), while unlearned-only models rapidly recover the suppressed knowledge. This demonstrates our method also works when unlearning specific concepts. That said, as we explain in our introduction, our primary motivation differs from typical unlearning work. We focus on removing broad capabilities, rather than specific facts or concepts. Our method works for both use cases.

W2: WMDP utility results

From review text: "Fig 8 reports results on WMDP, but does not present the utility-unlearning curve. What would it look like in this case?"

You're right that Figure 8 didn’t include complete utility-unlearning curves. We provide these results in Figure 11 in the appendix:

Table: WMDP-Bio

We didn't highlight them in the main paper because interpreting WMDP results is challenging. Gemma-2-2B's pretraining likely already contains weapons and cybersecurity information, making it impossible to establish a true "never seen" baseline like we can with TOFU's fictional authors. For clarity, we also evaluated on TOFU, where we have guaranteed control over exposure:

Table: TOFU Utility-Unlearning Tradeoff

Model Utility is the harmonic mean of 9 metrics measuring performance on retain set, real authors, and world facts (3 metrics each: Q&A probability, ROUGE score, and truth ratio)

Our method achieves 23% better robustness than unlearning alone while maintaining competitive utility (0.442 vs 0.444 for unlearning only). The distilled model's relearning trajectory closely matches the control, suggesting that the post-unlearning distillation step does promote a deeper knowledge removal while retaining the model's utility. This pattern holds regardless of which unlearning method (GradDiff, NPO, etc.) is used before distillation, confirming the generality of our approach.

W3: Fair comparison concerns

From review text: "I am not sure the forget-utility comparison in Fig7 is completely fair. If I understand correctly, the baselines just experience unlearning, while UNDO experiences unlearning+retraining on the retain set."

You raise an important point, but we believe there may be a slight misunderstanding about how UNDO works. With UNDO, you first select any unlearning method (GradDiff, MaxEnt, etc.) with your desired retain-forget balance (Figure 3). Then, separately, you decide how much compute to invest in distillation to improve robustness and gain back the retain performance. The retain-forget trade-off is bounded by the teacher (unlearning method of choice), while robustness scales with compute (trade compute for robustness, Figure 5).

Crucially, UNDO does not retrain on a labeled retain set. During distillation, we distill on the full pretraining dataset, including both retain and forget data. While the distillation does require more compute than prior unlearning methods, this step enables increased robustness. This is analogous to heavily damaging and then rewriting parts of the model internal while having a teacher who can only properly demonstrate certain capabilities. What the teacher pretends not to know, the student won't learn at all.

For example, the table below shows our distillation trends where the datasets contained 50% forget data; the student only recovers capabilities the teacher demonstrates:

The student's Korean loss remains at random initialization levels (12.64) despite seeing Korean text in every batch. Similarly, multiplication accuracy stays at 0% even though 25% of the arithmetic distillation data contains multiplication problems. The initially unlearned teacher provides meaningful gradients only for retain capabilities. This property creates a unique space between existing approaches:

Regarding Figure 7 that you specifically mentioned, all UNDO points use the same noise parameters (α=0.6, β=0.1). The different points along the retain-forget tradeoff curve come from varying when we stop distillation based on retain performance thresholds {±0.05, ±0.09, ±0.13, ±0.17, ±0.21, ±0.25, ±0.29, ±0.33}, not from different amounts of noising.

W4: Robustness beyond gradient-based attacks

From review text: "Robustness is measured via gradient-based relearning attacks (finetuning on the forget set). It would be beneficial to also compare against jailbreaking attacks in inference time."

We first note that prior literature established that finetuning is the strongest elicitation technique. [2, 3] found that while fine-tuning consistently performed best across all tested setups, other attack vectors showed less consistent success. This universal vulnerability to fine-tuning, combined with its ease of implementation and replicability across research groups, motivated our focus on fine-tuning.

We also tested (inference-time) quantization attacks. Recent work [5] showed that quantization can bring back unlearned knowledge:

Table: INT4 Quantization Robustness (TOFU, Gemma-2-2B)

Our distilled models show dramatically better resilience to quantization attacks across all metrics. We highlight forget truth ratio (true utterance about the forget set authors) improvement (0.185 vs 0.017). This suggests that distillation provides defense against orthogonal attack vectors beyond just gradient-based relearning.

Additionally, we have tested prompt-based and activation steering elicitation methods on our bilingual models. However, it was challenging to get clear signals, potentially due to the relatively small model sizes we used (100M-300M parameters) or limitations in applying these techniques to our specific setups.

Q: Figure 7a performance

This phenomenon actually has a clear explanation. Our Korean dataset (FineWeb split) contains some English characters, so training on English alone still slightly improves Korean performance, though it remains close to random (CE loss ~11-12).

Methods like MaxEnt explicitly maximize entropy on the forget set, pushing outputs toward uniform distributions. This can achieve a cross-entropy loss closer to the "random" baseline than what the gold standard data filtering naturally achieves. As shown in Figure 7a, MaxEnt pushes Korean CE loss close to around 12.5, while the gold standard (trained only on English) settles around 11.

A more disruptive method like GradDiff, which performs a negative gradient update against the forget set, can push Korean CE loss to the regimes of 22~25, which also explains the distillation dynamics in Figure 9. However, CE loss values above ~11 represent essentially non-linguistic outputs. The model isn't producing "better forgetting" but rather completely uninformative tokens.

— We thank the reviewer for helpful questions! —

[1] TOFU: A Task of Fictitious Unlearning for LLMs. COLM 24

[2] WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. ICML 24

[3] The elicitation game: Evaluating capability elicitation techniques. ICML 25

[4] Eight methods to evaluate robust unlearning in llms. Preprint

[5] Catastrophic Failure of LLM Unlearning via Quantization. ICLR 25

We thank the reviewer for the thoughtful and thorough review! We first address the key concerns raised by the reviewer and then proceed to discuss other questions.

W1: Access to pretraining data, off-the-shelf applicability, breadth of models tested

For the threat models we consider (for example, bad actors finetuning open-weight models), mitigation becomes significantly more difficult once the weights of a model with latent dangerous capabilities are made available to the public. Performing robust unlearning on a model whose weights are already publicly available is not helpful. Therefore we expect robust unlearning methods to be used by model developers after pretraining a model but before releasing it to the public. These developers would have access to the original pretraining dataset.

The TOFU benchmark gives us another setup to test our method at scale. TOFU's fictional figures don't exist in any pretraining data, making the reference model mimic random initialization (zero latent knowledge of what we want to unlearn). We confirm our method works at larger scales and also test it across different architectures (Gemma-2-2B and Phi-1.5) and unlearning algorithms (GradDiff and NPO):

Table: TOFU Results on Gemma-2-2B with GradDiff

Table: TOFU Results on Phi-1.5 with NPO

Forget Q&A: Probability of correctly answering forget set author questions (lower is better)

MIA: Membership Inference Attack success rate (lower is better)

Both our distilled models achieve relearning rates closer to the control baseline (a model that never saw the forget authors), while unlearned-only models rapidly recover the suppressed knowledge (trend is clearer in a graph). In the camera-ready version, we would report an analysis across model families and sizes.

Importantly, we don't necessarily require starting from a random init model if we know for sure from which split of the pretraining data the forget target is from. Consider a case where there are several model checkpoints saved during training based on time intervals (m1, m2, m3, m4...) corresponding to dataset splits (d1, d2, d3, d4...), as is common for many production setups. If we know that the model has become problematic at a certain time point (e.g., harmful content introduced in d4), we can roll back to checkpoint m3 (which doesn’t have latent traces of d4 knowledge) and distill the unlearned m4 onto it. .

W2: Explanation for why distillation robustifies unlearning

Thank you for this comment “The paper provides strong empirical evidence that distillation robustifies unlearning … A deeper analysis … could further strengthen the paper's contribution”.

We're particularly excited that our work aligns with emerging insights in the unlearning literature. Recent work [1] found that irreversible unlearning is difficult to achieve without "large rotations of principal directions, centroid shifts, and vanishing Fisher mass across many layers," essentially requiring many layers to undergo coordinated, large-magnitude perturbations. We believe our work provides a timely contribution by demonstrating a concrete mechanism that achieves what these works hypothesize as true forgetting.

Our intuition is that all knowledge must pass through the teacher's outputs to reach the student. When combined with a student that initially has no latent information of the forget set, this distillation operation creates the kind of coordinated, large-magnitude perturbations across many layers that [1] identifies as necessary for irreversible unlearning. By starting from random initialization (or a clean checkpoint), the student undergoes global representational shifts while selectively learning only what the teacher demonstrates.

Our experiments validate this hypothesis. Even when distilling on the full dataset including forget data:

Table: Selective Learning During Distillation

The student's performance on forget domains remains at random initialization levels despite seeing the forget domain data in every batch.

Also, our approach addresses a broader category of capability removal beyond other privacy-focused unlearning use cases. While privacy applications typically target specific data points or narrow facts, our work demonstrates removal of entire capability domains (languages, mathematical operations).

Q1: Preliminary results for wider variety of model families and larger sizes?

We will prepare a wider variety of model families and larger sizes results for TOFU.

Our main language and arithmetic unlearning experiments were done at pretraining scale, making it harder to scale up. We have initial results for up to 0.9B establishing similar findings, but for the purpose of focused scientific contribution, we chose to present singular model sizes (0.1B for language and 0.3B for arithmetic). Instead, we take an extremely strong adversarial assumption to ensure the validity and replicability of our claims, where we do hyperparameter search across 10 learning rates (1e-6 to 1e-3), substantial data (8M tokens for Korean, 2M tokens for arithmetic), and 500 gradient updates, and report adversarially optimal cases.

Q2: Have you tried latent knowledge probes rather than behavioral scores?

We have tested prompt-based and activation steering elicitation methods on our bilingual models. However, it was challenging to get clear signals, potentially due to the relatively small model sizes we used (100M-300M parameters) or limitations in applying these techniques to our specific setups. However, we tested alternative elicitation methods. For quantization-based attacks [2]:

Table: INT4 Quantization Robustness (TOFU, Gemma-2-2B)

Our distilled models show much better resilience across all metrics, suggesting that distillation provides defense against attack vectors beyond just gradient-based relearning.

Q3: Performance drop on MMLU - how to mitigate without original pretraining data?

See our comment above regarding threat models. Also, we think parts of our distillation setup could be optimized (e.g., spreading out the noising over distillation may help).

Q4: Concrete quantitative comparison of cost?

While Figure 5 shows that arithmetic reaches gold standard robustness with about 60% of gold standard compute and language with about 80%, we concretely mean data efficiency. The key advantage is avoiding the need to label the entire pretraining corpus. In Figures 5 and 7, UNDO required only 0.01% of the pretraining data to be labeled to match the gold standard robustness of full data filtering. That is, we create a new solution space:

Q5: Alternative methods of "damaging" the teacher?

We see several promising alternatives like pruning, targeted damaging in ROME-style [3]. While we haven't implemented these yet, even imperfect targeting could improve retain-forget tradeoffs compared to our uniform approach.

Q6: Task-dependent alpha - why does it vary? Practitioner guidance?

The variation could stem from different evaluation metrics (cross-entropy loss for language vs. accuracy for arithmetic) having different sensitivities to perturbation. After an inflection point, we observe a linear trend between compute requirements and robustness in both domains once the noising starts to remove the latent traces.

Q7: Broken references

Thank you for catching this! We'll fix all references showing as "??" in the camera-ready.

— We thank the reviewer for a very thorough review! —

[1] Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs. Preprint 25

[2] Catastrophic Failure of LLM Unlearning via Quantization. ICLR 25

[3] “Locating and Editing Factual Associations in GPT” NeurIPS 22

We thank the reviewer for the thoughtful review and for highlighting the novelty of our approach!

Conceptual explanation for our work

We appreciate your request for more intuition about why distillation works. While we give this intuition in Section 3, we've added some discussion here.

To briefly recap our key finding from Section 3: We started with models that already possessed harmful capabilities (we refer to these as "reference models," which were pretrained on both English and Korean). We then trained these contaminated models to perfectly mimic an oracle model that had never seen Korean. Despite achieving near-perfect behavioral alignment, something strange happened during relearning attacks.

When we compared three types of models:

1. Oracle: Never saw Korean during training

2. Student (Random): Fresh model trained to mimic the oracle's outputs

3. Student (Reference): Contaminated model trained to mimic the oracle's outputs

The contaminated models that "forgot" Korean relearned it dramatically faster:

Table: Oracle Matching Results (Figure 2)

Student (Reference) showed fast relearning despite behaving perfectly the same as the oracle.

The model's "origin" (what it knew before matching the oracle) determined its vulnerability to relearning.

Now, what if we could combine unlearning (which achieves behavioral suppression) with distillation to a fresh model (which has no problematic history)? When distilling from an unlearned teacher to a fresh student:

1. The student starts clean (no pre-existing harmful knowledge)

2. The teacher's outputs on harmful content are meaningless noise

3. The student can only learn from what the teacher demonstrates

4. Since the teacher refuses to teach harmful content, it cannot transfer

That is, we leverage distillation as a capability filter and create a new solution space:

We also validated this on TOFU [1], where fictional authors were created to test unlearning in realistic scenarios. Our distilled models relearned at similar speeds as the control baseline (a model that never saw the forget authors), while unlearned-only models rapidly recovered the suppressed knowledge under relearning attacks.

Given that Section 3 already presents this conceptual foundation with evidence (Figure 2), would more forward references from Section 4 to Section 3 help clarify the connection in the manuscript?

W1: Why does distillation robustify unlearning?

From review text: “The paper does not present much intuition/explanation of why distillation is effective at improving the robustness of unlearning methods.”

The student starts knowing nothing and can only learn what the teacher demonstrates. Since the unlearned teacher produces meaningless outputs on forget data, there's nothing for the student to learn about those capabilities.

Consider a teacher that has undergone unlearning to suppress multiplication. Although latent multiplication knowledge remains in its parameters (which might later promote rapid relearning of the capability), when asked to solve multiplication problems, it produces only random noise. A fresh student can only learn from the gradients provided by the teacher's outputs and cannot acquire multiplication, because the teacher provides no meaningful signal. Though intuitive once stated, the effectiveness of this approach is not obvious, given potential anomalies in the long tail of teacher logits. Our contribution is demonstrating that it works well in practice.

As an example, the table below shows how the Korean CE Loss stays large despite the fact that the distillation training set includes Korean documents:

Table: Distillation - Language Unlearning Experiment

Similarly, in arithmetic experiments, the student recovers the teacher's addition/subtraction/English performance while multiplication/division stays completely unlearned.

Table: Distillation - Arithmetic Unlearning Experiment

W2: Alternative approaches to mixing

From review text: “Relatedly, are there other ways to "mix" the unlearned weights with noise instead of distillation?”

We believe that this is a promising topic for future research. Instead of random perturbations, we could use interpretability to identify and corrupt specific weights responsible for harmful behaviors. We think success would depend on two criteria:

Criterion 1: Size of unlearning target - How semantically broad the capability is

Criterion 2: Entanglement - How distributed across the model's weights

For narrow facts like "the capital of France is Paris" (small C1), mechanistic interpretability suggests these are often localized (small C2) [2], making targeted noising promising. However, for complex capabilities like deception (large C1), the required computations are likely distributed across many components (large C2), such as tracking truth, generating alternatives, modeling beliefs, and optimizing outputs [3].

While we haven't implemented targeted noising yet, even imperfect targeting could improve retain-forget tradeoffs. It doesn't need to be perfect, just better than global. This is an exciting avenue for future work!

Q1: Unlearning without explicit retain sets

From review text: “Can unlearning work without a retain set, and are there algorithms in the literature?”

Yes, this depends on the unlearning algorithm. GradDiff computes loss_forget - loss_retain, requiring both sets. In practice, many past works use generic retain datasets (WMDP uses Wikitext). During distillation, we don’t require an explicit retain set. The teacher's outputs on any unlabeled data automatically provide the training signal, where meaningful predictions guide learning on retain content while uninformative outputs prevent learning on forget content.

Q2: Distillation dataset

From review text: “what is the dataset used for distillation? Is it the full training dataset?”

Yes, we use the full pretraining dataset for distillation. Only the initial unlearning step needs small labeled retain/forget sets (typically <0.01% of data). We avoid labeling billions of tokens.

Q3: Full training set partitioning

From review text: “I'm unclear about whether this claim has been shown empirically, because your approach (line 100 and around) requires the entire training set to be partitioned into the retain and forget sets.”

No, we don't require partitioning the entire training set. Lines 204-206 compare against the theoretical "gold standard" of filtering everything (impractical at scale). We only need small labeled sets for initial unlearning, then use unlabeled pretraining data for distillation.

— We thank the reviewer for pushing us to clarify the conceptual points! —

[1] TOFU: A Task of Fictitious Unlearning for LLMs. COLM 24

[2] Locating and Editing Factual Associations in GPT. NeurIPS 22

[3] The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets. COLM 24

Thank you for engaging with our rebuttal. We will incorporate the intuition discussion into the manuscript as suggested. Given that W1 and W2 were your primary concerns and these have been addressed, we wanted to confirm whether any aspects of the work remain unclear or require further clarification.

We thank the reviewer for the thoughtful feedback and constructive suggestions! First, we share additional evaluations on TOFU to address the concerns about downstream performance and alignment with the broader unlearning literature.

TOFU

We conducted experiments using the open-unlearning library. TOFU turns out to be ideal since its fictional figures don't exist in any pretraining data, making the base model mimic random initialization (never encountered what we want to unlearn). We have done additional experiments with Gemma 2 2B and Phi 1.5. We report Gemma 2 2B in this response due to character limits.

Step 1: TOFU Finetuning on gemma-2-2b

The model memorizes the fictional authors, with Forget_QA probability increasing from 0.06 (epoch 0) to 0.59, while MIA (membership inference attack) success rate goes from 0.31(epoch 0) to 0.97.

We then apply GradDiff unlearning, which successfully suppresses the unwanted knowledge while maintaining retain performance.

Step 2: Unlearning TOFU using GradDiff

After 50 steps, the forget metrics drop to near-zero (Forget_QA: 0.59→0.001) while model utility remains stable. We then distill this behaviorally-suppressed model back into the original gemma-2-2b:

Step 3: Distilling Unlearned Model

The resulting rewritten model maintains near-zero forget metrics while recovering full utility (0.44), matching the unlearned teacher's performance. This demonstrates that distillation transfers desired behaviors on retain but not on forget.

Then, we subject the three models (unlearned, unlearned+distilled, fresh base model - as control) to relearning attacks (100 steps on TOFU) to measure how quickly the forget knowledge is recovered:

Metric: Forget_Q_A_Prob

Metric: Extraction Attack Success Rate

While the unlearned-only model rapidly recovers the suppressed knowledge (Forget_QA: 0.378 after 100 steps), the distilled model relearns at rates closer to the control, a fresh model that has never seen TOFU. Extraction attack rates are virtually identical between our method and the control (0.122 vs. 0.124), suggesting that the post-unlearning distillation step does promote a deeper knowledge removal. This pattern holds across different unlearning methods (GradDiff, NPO, etc) that are used before distillation, confirming the generality of our approach.

W1: Effects on general performance

TOFU results above directly demonstrate capabilities preservation during specific knowledge removal. The distilled model maintains the same utility as the teacher while keeping forget metrics near zero. Our controlled experiments show the student selectively learns general capabilities that the unlearned teacher properly demonstrates, despite distilling on the full, unfiltered dataset:

Table: Distillation Step - Language Unlearning

Similarly, in arithmetic experiments, the student recovers the teacher's addition/subtraction/English performance while multiplication/division stays completely unlearned.

Table: Distillation Step - Arithmetic Unlearning

These results are also available as Figure 9. In all experiments reported (Figures 1-6), the distilled student models achieve performance within 5% of their teachers on English & retain task. For certain tasks like arithmetic, we observe that the student actually often exceeds the teacher's performance on retain and control tasks. This is reminiscent of a well-documented phenomenon in distillation literature where students can surpass their teachers [1]. Distillation not only preserves capabilities but can also sometimes enhance them through cleaner learned representations.

W2: Validation on mainstream benchmarks

The TOFU experiments above address this concern. We tested multiple unlearning methods from open-unlearning, and adding distillation consistently improved resistance to relearning.

We thank the reviewer for this suggestion. Our method validated well on mainstream benchmarks like TOFU. Given these compelling results, we will:

1. Contribute our method to open-unlearning for ease of access.

2. Add a separate TOFU evaluation section.

W3: Other elicitation efforts

Thank you for raising this point about alternative elicitation strategies!

We first note that prior literature shows that finetuning is a strong elicitation technique. [2, 3] found that while finetuning consistently performed best across all tested setups, other attacks showed limited success. This universal vulnerability to finetuning, combined with its standardized implementation and replicability, motivated our focus on finetuning.

In our paper, we go a step further and take the strongest adversarial assumption of hyperparameter search across 10 learning rates (1e-6 to 1e-3), substantial data (8M tokens for Korean, 2M tokens for arithmetic), 500 gradient updates, and report adversarially optimal cases.

Additionally, we empirically verify that our approach also robustifies against alternative elicitation strategies. We tested quantization using the open-unlearning library [4]:

Table: Quantization Effects

Under INT4 quantization, standard unlearning shows increased forget metrics (forget_Q_A_Prob: 0 → 0.008, forget_truth_ratio: 0 → 0.185). Our distilled models demonstrate superior robustness with forget_Q_A_Prob reaching only 0.003 (2.7× better) and forget_truth_ratio at 0.017 (11× better). The distilled models also maintain better utility (0.391 vs 0.242) and lower MIA vulnerability (0.013 vs 0.054).

Q: Role of β

We have edited our manuscript to clarify the role of β and include an explanation here. The role of β is to ensure sufficient perturbation at low α values (scaling to be meaningfully different from Xavier init) and also that the resulting model has the correct scale. We recommend: (1) Use β=0.1 as default (2) Only tune β for α<0.2 (3) For α≥0.3, the mixing coefficient provides sufficient disruption. This aligns with [5] who found β=0.01-0.1 optimal across various settings.

— We thank the reviewer for the constructive suggestions! —

[1] Understanding the gains from repeated self-distillation. NeurIPS 24

[2] The elicitation game: Evaluating capability elicitation techniques. ICML 25

[3] Eight methods to evaluate robust unlearning in llms. Preprint

[4] Catastrophic Failure of LLM Unlearning via Quantization. ICLR 25

[5] On warm-starting neural network training. NeurIPS 20

We wanted to follow up on our rebuttal, as we haven't heard back, and the discussion period ends on August 8. We conducted additional experiments to address your main concerns:

• Added TOFU experiments on Gemma-2-2B and Phi-1.5 for better alignment with the unlearning community

• Evaluated additional metrics you suggested (Verbatim Probability, ROUGE variants, Model Utility, Forget Quality, TruthRatio, MIA strength)

• Tested robustness to alternative elicitation strategies

We'd greatly appreciate your thoughts on whether these additions address your concerns about limited evaluation and alignment with existing unlearning work. Are there any remaining issues we could clarify?