Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM

Thank you for your feedback!

1. W1: Attack assumption.

I have a major concern over the paper's core assumption that attackers can access both pre-unlearning and post-unlearning models. This scenario largely defeats the purpose of unlearning methods, which exist precisely because the original models carry privacy risks we're trying to mitigate. Building an attack method that requires access to the pre-unlearning model isn't realistic in practice.

We focus on how unlearning is deployed in real-world scenarios, where our attack could realistically occur, especially for open-source models. It is entirely possible that an attacker has already pre-saved the pre-unlearning model—for instance, an adversary might successfully extract and release copyright-sensitive data (or the issue is otherwise discovered), prompting the model owner to perform unlearning. In such cases, our concern is that the attacker could then extract even more data than before unlearning was applied, creating a paradoxical situation where a mechanism intended to reduce leakage may actually amplify it.

2. W2: Contribution of Method.

The main finding that combining outputs from both model versions improves extraction lacks sufficient novelty. The contribution is incremental over prior work. It's essentially like comparing a model that has memorized certain data with one that hasn't, then using the differences to reveal what was memorized. (Similar to previous results that leverage over-fitting to benefit privacy attacks).

This characterization oversimplifies our contribution. As detailed in Sec. 4, our method is not merely “comparing outputs.” We propose a principled parametric approximation that simulates the fine-tuning and unlearning dynamics to construct a model-guidance distribution, which is then used to drive extraction. This formulation is non-trivial and, to our knowledge, has not been explored in prior work.

While the intuition that information gain can improve extraction may seem natural, our contribution lies in how we formalize and operationalize it. We demonstrate why this formulation works, and we show empirically that it consistently improves extraction performance. These insights are methodological, not just incremental.

3. Minor question:

Does q(x) in Sec.4.1 represent distribution of the gold standard “Retrain from Scratch” model?

No. As stated in Line 152, $q(x)$ represents the ground-truth probability distribution of the forgetting set $X_0$​, which we aim to approximate in order to enable strong extraction.

Thank you for the thoughtful follow-up. We believe this discussion reflects a difference in perspective between how unlearning is typically framed in the literature and how it might be deployed in realistic scenarios.

From our standpoint, unlearning—particularly if used in large-scale deployments—would often serve as a reactive mechanism, applied after a model has leaked personal information, raised copyright concerns, or been subjected to targeted extraction attempts. In such cases, it is entirely possible that the pre-unlearning model has already been downloaded, cached, or queried by adversaries. For example, once a model is released and subsequently found to have been exploited for privacy or copyright-related content, unlearning may be invoked to mitigate future misuse—making it harder for downstream users or future adversaries to access the exposed content. However, for those who have already interacted with the pre-unlearning model, unlearning may introduce new model differences that inadvertently assist further extraction attempts. For example, an attacker may have pre-saved the model weights, and the changes introduced by unlearning could provide additional signals that help guide future attacks.

This motivates our decision to analyze unlearning under a distinct threat model, where the attacker has access to the logits APIs of both the pre- and post-unlearning models. While this setting differs from conventional assumptions, we argue that it is highly relevant in practice—particularly for released, open-sourced, or previously accessed models. In such contexts, unlearning is not performed in isolation from the model’s prior exposure; the behavioral differences introduced by unlearning may remain observable to adversaries who interacted with the original model.

Our goal is to understand whether such differences, when exposed to adversaries with prior access, can unintentionally amplify leakage—through model guidance based on the differences between the two models. This perspective does not challenge the effectiveness of unlearning under its standard assumptions, but rather highlights that additional risks may emerge when those assumptions no longer hold in deployment.

To avoid misinterpretation, we have revised our title and abstract to better reflect our focus on exposure-driven vulnerabilities, rather than suggesting any intrinsic algorithmic flaws. We hope this clarification helps position our work as a complementary viewpoint—one that expands the conversation around the robustness and deployability of unlearning in real-world systems. We appreciate the opportunity to elaborate on this point.

Revised title and abstract are listed below. The bold parts indicate the modified text.

---

Title: Rethinking Exact Unlearning under Exposure: Extracting Forgotten Data under Exact Unlearning in Large Language Models

Abstract: Large Language Models are typically trained on datasets collected from the web, which may inadvertently contain harmful or sensitive personal information. To address growing privacy concerns, unlearning methods have been proposed to remove the influence of specific data from trained models. Of these, exact unlearning---which retrains the model from scratch without the target data---is widely regarded the gold standard for mitigating privacy risks in deployment. In this paper, we revisit this assumption in a practical deployment setting where both the pre- and post-unlearning logits API are exposed, such as in open-weight scenarios. Targeting this setting, we introduce a novel data extraction attack that leverages signals from the pre-unlearning model to guide the post-unlearning model, uncovering patterns that reflect the removed data distribution. Combining model guidance with a token filtering strategy, our attack significantly improves extraction success rates---doubling performance in some cases---across common benchmarks such as MUSE, TOFU, and WMDP. Furthermore, we demonstrate our attack's effectiveness on a simulated medical diagnosis dataset to highlight real-world privacy risks associated with exact unlearning. In light of our findings, which suggest that unlearning may, in a contradictory way, increase the risk of privacy leakage during real-world deployments, we advocate for evaluation of unlearning methods to consider broader threat models that account not only for post-unlearning models but also for adversarial access to prior checkpoints.

Thank you for your insightful feedback!

1. Weakness&Q2: Practical Attack Scenarios.

Moreover, the method's core assumption (attacker access to pre-unlearning weights or logits) while plausible for open-weight releases, is not universally realistic and largely defines the success of the attack, so the threat surface may be narrower than the headline suggests.

In commercial settings where only completions (not logits) are exposed, how often would an attacker be able to retain usable pre-unlearning outputs? If feasible, can you run your attack using saved completions rather than logits to assess its practicality in more restrictive environments?

We consider open-source scenarios (such as those on Hugging Face) to be among the most practically significant. For example, an attacker targeting a fine-tuned LLM may have previously extracted partial sensitive data using the original model and saved both the model and the corresponding outputs. Once the model is later updated to forget specific data, the attacker can re-run extraction using our method.

We acknowledge that in API-only settings, access to logits is not always available. In some cases, such as with GPT-4o, log probabilities can be retrieved via the logprobs parameter. However, other APIs may not expose this information at all, which makes our attack more difficult to apply in those environments. One possible workaround is to approximate logits by sampling multiple times per token, though this approach is computationally expensive. Extending our attack to settings where only completions are available—such as by relying solely on cached outputs rather than logits—is an important direction for our future work.

2. Weakness&Q1: Extraction on Larger Models.

At the same time, the empirical section leans on relatively small models (Llama-2-7 B and Phi-1.5) and synthetic or already-public datasets, so readers are left to extrapolate to frontier-scale LLMs and truly proprietary data.

Can you demonstrate similar extraction effectiveness on at least one very large model (e.g., LLaMA-3-70B or Mixtral)? Showing this would strengthen the generality of your claims and would positively influence my score.

We add results on Mixtral-8x7B-Instruct-v0.1 by LoRA‑fine‑tuning on TOFU for two epochs with a cosine learning rate of 1e‑4 and a 10% forgetting set (all other settings followed our default setting in Sec. 5.1). For extraction, we apply a guidance scale of w=2. The improvement is consistent, as shown in the table below:

3. Weakness&Q3: Hyperparameters Choosing.

certain hyper-parameter choices (e.g., guidance scale selection) appear empirical rather than principled,

Since hyperparameters are selected empirically, please provide clearer guidance or theoretical intuition for choosing them.

We agree that the guidance scale w is an important hyper-parameter, and its choice is grounded in our theoretical formulation rather than being purely empirical. As shown in Eq. (2), w=1/λ controls the strength of the guidance, which determines how much the generation is influenced by the score difference between the pre- and post-unlearning models. Intuitively, this reflects how much the pre-unlearning model still retains information about the forgotten data.

From a theoretical perspective, as discussed in Sec. 4.1, Eq. (2) assumes a reverse process in which the post-unlearning model is fine-tuned back toward the pre-unlearning model using the forgotten data. Under this view, the pre-unlearning model’s distribution can be seen as a mixture of the post-unlearning distribution and the forgotten data distribution. The guidance scale w thus controls how strongly we extrapolate from the post-unlearning model toward the forgotten data distribution. A larger w implies a more aggressive shift back toward the original state, while a smaller w corresponds to a gentler correction—aligning smoothly with the amount of residual memorization.

Importantly, even without tuning w to its optimal value, our method consistently improves extraction performance over the baseline as long as w is not set too large. Specifically, applying model guidance to steer generation away from the post-unlearning distribution is always beneficial. As shown in Fig. 5 and Fig. 6, settings with w<2.0 consistently lead to better extraction results across both Phi and LLaMA-2-7B, compared to the baseline case of w=1.0.

Thank you for your insightful feedback!

1. W1: Missing Citation for related work.

The core ideas of the paper root in recent papers, reducing the novelty. For example, using model differences before and after unlearning was first proposed by Chen et al 2021, to devise a new membership inference attack. This paper doesn’t cite that work.

Thank you for pointing this out. We acknowledge that we missed citing this highly relevant work, which also considers privacy risks by comparing models before and after unlearning. We will include the citation in the revised version.

Our main methodological contribution lies in making model differences effective for data extraction in the context of large language models. Since the learned manifolds of LLMs are highly structured and aligned with natural language, we operate within these manifolds using guidance-based generation. This contrasts with optimization-based methods, which might produce unnatural or incoherent outputs.

2. W2: Overstated title:

In reality, the threat arises from the system-level exposure of both pre- and post-unlearning models—rather than a shortcoming of the unlearning method. It is like saying that DP is not private if there is a third signal that leaks the difference between the neighbouring datasets.

We agree that the threat primarily stems from the exposure of both the pre- and post-unlearning models, rather than from a fundamental flaw in the unlearning algorithm itself. Our main takeaway is to caution against applying exact unlearning in cases where the pre-unlearning model may already have been exposed—such as in open-source scenarios where earlier versions could have been downloaded and saved. In such cases, exact unlearning might inadvertently increase risk rather than mitigate it. Accordingly, we will revise the title and framing to emphasize the importance of careful consideration during the deployment of unlearning, rather than suggesting a failure of the algorithmic guarantee itself.

3. W3: Gradient clipping in DP-SGD:

In the defence methods, where noise is added to simulate DP-SGD, an important aspect of dp-sgd that is gradient clipping is neglected. Do you have any insights about the effect of gradient clipping?

Thanks for pointing this out. In Noisy Gradient Updates (Sec 5.6), we treat the absolute noise scale as our primary DP-SGD variable and fix the clipping norm at 2. Empirically, we observe that, under our mini‑batch training regime, typical choices of the clipping norm have virtually no measurable impact on final model performance. We further conducted experiments with clipping norms ranging from 0.1, 0.25, 0.5, 1, 2, to 4, finding that neither utility nor extraction rate changes more than 1%.

Theoretically, we define the batch noise scale $\eta = \frac{clipping\\_norm \times noise\\_multiplier }{batch\\_size}$. Equivalently (see Sec. 5.6), if $\alpha$ denotes the per‑example noise scale, then $\eta = \frac{\sigma}{batch\\_size}$. In our experiments, with batch size of 4, this noise scale $\eta$ far outweighs the typical gradient magnitudes, so the injected noise dominates the update. In particular, let $N$ denote the number of fine‑tuning parameters, since:

$$\mathbb{E}\|gradient\\_noised\|^2 \approx \|gradient\\_clipped\|^2 + \mathbb{E}\|noise\|^2 = \|gradient\\_clipped\|^2 + N \eta^2$$

For Phi-1.5 (fintuning parameters $N > 5\times10^{5}$), even with $\eta=0.01$, the expected gradient norm after noise addition exceeds $50$, while original gradients are clipped to $\approx1$. The noise effect overwhelms the clipping constraint, making clipping variations irrelevant to defense effectiveness.

This is why we focus solely on noise scales as the key defense factor in our paper.

4. W4: Peft finetuning:

But in practice, Peft finetuning is very popular. Do the authors have any insights how this will change the attacks success rate?

We add results on Phi-1.5 by LoRA‑fine‑tuning on TOFU for five epochs with a constant learning rate of 2e‑4 and a 10% forgetting set (all other settings followed our default setting in Sec. 5.1). Then, we employ a guidance scale w=1.6 for our extraction, increasing the extraction rate. The improvements are consistent, as shown in the following table.

5. W5: Different initializations of the models:

In your setup, original model and the gold model have the same initialization due to the pretrained models. Do you think if there was a way to change the initializations, the attacks success rate would change?

We agree that in our setup, both models share the same initialization due to being derived from the same pretrained model. In practice, it may be difficult to alter the initialization meaningfully. However, if a "twin" model exists—i.e., a model trained with different random seeds or slightly different settings but that still learns a similar distribution—the unlearning could be applied to this unreleased twin version. This might reduce the attack's effectiveness to some extent. However, since our approach is based on the learned data distribution rather than the exact model weights, the attack should still be partially effective as long as the two twin models learn sufficiently similar distributions.

6. W6: About metric A-ESR.

Is A-ESR calculated over the entire sample length? Is there a way to factor out the effect of the known prefix?

No, A-ESR is computed only over the generated suffix, excluding the known prefix. Therefore, the effect of the known prefix is already factored out by design.

Thank you for your comments! We plan to tone down the title and revise the abstract as follows to better highlight the risks of unlearning under exposure in practical deployments, rather than the flaws of the algorithm itself. The bold parts indicate the modified text.

Title: Rethinking Exact Unlearning under Exposure: Extracting Forgotten Data under Exact Unlearning in Large Language Models

Abstract: Large Language Models are typically trained on datasets collected from the web, which may inadvertently contain harmful or sensitive personal information. To address growing privacy concerns, unlearning methods have been proposed to remove the influence of specific data from trained models. Of these, exact unlearning---which retrains the model from scratch without the target data---is widely regarded the gold standard for mitigating privacy risks in deployment. In this paper, we revisit this assumption in a practical deployment setting where both the pre- and post-unlearning logits API are exposed, such as in open-weight scenarios. Targeting this setting, we introduce a novel data extraction attack that leverages signals from the pre-unlearning model to guide the post-unlearning model, uncovering patterns that reflect the removed data distribution. Combining model guidance with a token filtering strategy, our attack significantly improves extraction success rates---doubling performance in some cases---across common benchmarks such as MUSE, TOFU, and WMDP. Furthermore, we demonstrate our attack's effectiveness on a simulated medical diagnosis dataset to highlight real-world privacy risks associated with exact unlearning. In light of our findings, which suggest that unlearning may, in a contradictory way, increase the risk of privacy leakage during real-world deployments, we advocate for evaluation of unlearning methods to consider broader threat models that account not only for post-unlearning models but also for adversarial access to prior checkpoints.

Thank you for your insightful feedback!

1. About the hyper-parameter w:

From the results of Figure 5, the correct selection of w seems to matter significantly…it's unclear how the attacker can guess a good value for w.

Our modeling begins with approximating how much the original model memorizes the dataset, as described in Eq. (2). This is described by the parameter λ, and since w=1/λ, the choice of w plays a central role in determining extraction performance. While the attacker may not have access to precise information about the extent of memorization, we emphasize two key points: First, our method consistently improves extraction performance over the baseline as long as w is not set too large. Specifically, applying model guidance to steer generation away from the post-unlearning distribution is always beneficial. As shown in Fig. 5 and Fig. 6, settings with w<2.0 consistently lead to better extraction results across both Phi and LLaMA-2-7B compared to the baseline case where w=1.0.

Besides, in practice, partial side information about training iterations may still be available sometimes. For example, training epochs are sometimes disclosed in open-source models (e.g., open-r1/OpenR1-Qwen-7B on Huggingface). Additionally, prior work[1] suggests a correlation between model confidence and overfitting, which can serve to infer the training side information and could be potentially used to estimate the suitable guidance scale w.

[1] Song, Liwei, and Prateek Mittal. "Systematic evaluation of privacy risks of machine learning models." 30th USENIX security symposium (USENIX security 21). 2021.

2. For the practical setting issues:

For the open-weight setting, the attacker has to have access to the weights of both the pre-unleared LLM and post-unlearned LLM, which is already a pretty strong assumption in my opinion.

While the reviewers consider access to both the pre- and post-unlearning models a strong assumption, we argue that it is reasonable in open-weight scenarios. Open-weight models (such as those on HuggingFace) can be freely downloaded and archived before updates. For example, an attacker targeting a medical fine-tuned LLM may have partially extracted sensitive data using the original model and saved both the model and the results. Once the model is later updated to forget specific data, the attacker can re-run the extraction using our method.

For the API-only setting, wouldn’t the attacker need access to both the pre-unlearned and post-unlearned API during inference time, unlike how the authors claim in Line 116? The logits need to be recalculated for both pre- and post-unlearned LLMs every generation step…., proprietary LLMs rarely provide logits these days.

In some API-only settings, log probabilities can still be accessed—for example, GPT-4o supports retrieval via the logprobs parameter, enabling direct application of our method. For APIs that do not expose logits, attackers would need to simulate the generation process and approximate logits using multiple sampling. This requires storing more than just greedy completions—attackers would need to cache the top few candidate tokens at each step and re-run completions accordingly, leading to exponential growth in sampling cost. Despite the overhead, we note that even partial completions (e.g., a few-token prefix) may still yield meaningful extraction.

Overall, we agree that API-accessible but closed-source scenarios introduce additional complexity. Investigating how to perform extraction effectively under such constraints is an important direction for future work.

3. The attack performance on chatmodels:

It’s unclear whether the attack also performs well on chat models.

Can the authors please elaborate on how their method would work on chat models? I'm not really expecting for additional experimental results considering that will incur a lot of load, just sharing your thoughts will suffice.

Thank you for the question. We did not include experiments on chat models primarily because obtaining an exact unlearned version of a chat model is non-trivial. Chat models are typically derived from base models via additional fine-tuning stages, such as instruction tuning. To simulate unlearning in this context, one would first need to redo the base fine-tuning on a dataset that excludes the unlearning set and then reapply the instruction-tuning process. This procedure is both computationally expensive and complex to implement.

However, we hypothesize that data extraction on chat models would require more carefully designed prefixes to elicit training data, as there is often no fixed prompt structure that directly points to the data. Prior work (e.g., [2]) has shown that specially designed attacks can make chat models even more vulnerable, suggesting that crafting the right instructions could amplify extraction effectiveness.

Nonetheless, since our approach is based on contrasting the model’s behavior before and after unlearning, the core mechanism remains applicable. As long as both models are given the same input prompt, we expect our guidance-based method to still provide a significant boost in extraction performance compared to unguided generation in the chat model setting.

[2] Nasr, Milad, et al. "Scalable extraction of training data from (production) language models." arXiv preprint arXiv:2311.17035 (2023).

4. About availability of exact unlearning:

The attack assumes exact unlearning, which I’m not sure is commonly adopted in practice. A previous work points out that exact unlearning is “too costly to be practical”

Were there any real cases where exact unlearning was used on an LLM?

We note that existing benchmarks such as MUSE and TOFU adopt exact unlearning as a reference point that reflects the ideal outcome of unlearning. Recent efforts have also focused on improving the efficiency and practicality of exact unlearning methods [3,4], as detailed in Sec. 2.1. We therefore summarize it as the de facto gold standard, and our main focus is to examine whether privacy risks still remain under this setting.

As an extension, we also experiment under approximate unlearning methods in Appendix Sec. C, where our extraction method consistently outperforms the baseline. The effectiveness of extraction improves when the approximate unlearning better preserves model utility.

[3] X. Xia, Z. Wang, R. Sun, B. Liu, I. Khalil, and M. Xue. Edge unlearning is not" on edge"! an adaptive exact unlearning system on resource-constrained devices. arXiv preprint arXiv:2410.10128, 2024.

[4] K. Kuo, A. Setlur, K. Srinivas, A. Raghunathan, and V. Smith. Exact unlearning of finetuning data via model merging at scale. arXiv preprint arXiv:2504.04626, 2025.

Thank you for your insightful feedback!

1. W1: Overclaim of the title

… However, this assumption is a relatively strong setting and this might not be hold under all unlearning scenarios. Thus, I might consider the statement that ``break the gold standard" to be somewhat overclaimed to me.

We agree that our attack mostly applies to scenarios where both model checkpoints are available or where the APIs provide sufficient access. Accordingly, we will revise the claim from “break” to a softer framing, such as “rethink” or “re-evaluate.” Our focus on exact unlearning stems from its role as a reference point in current benchmarks—such as MUSE and TOFU—which treat it as the ideal case of what unlearning can achieve. Through this paper, we hope to draw attention to the fact that even such idealized unlearning may still expose significant privacy risks.

2. W1&Q4: lack discussions of approximate unlearning.

In addition, this paper seems to lack detailed discussions about approximate unlearning

I also care about the effectiveness of ``model guidance" under approximate unlearning settings, could you please provide more discussions?

We include experiments on approximate unlearning methods in Appendix Sec. C, where our extraction method consistently outperforms the baseline. Notably, extraction becomes more effective when the approximate unlearning better preserves model utility. Since exact unlearning yields the best utility by retraining from scratch, the model guidance direction remains well aligned with the original data distribution, making it more effective. In contrast, approximate unlearning methods typically degrade utility to some extent, causing the guidance direction to partially deviate from the true data distribution. In conclusion, our method consistently improves extraction performance when the post-unlearning model preserves a certain level of utility.

3. W2: Necessity of the method

Despite the significance of the concentrated topic, the proposed solution is not that surprising to me. The necessity and motivation of utilizing contrastive decoding should be provided.

We view it as a feature—not a bug—that our attack method is relatively simple and natural. This demonstrates that even when an attacker is not very sophisticated, exact unlearning may still expose vulnerabilities.

We are not sure that contrastive decoding is strictly necessary for performing the extraction attack. However, it is a natural strategy given that the primary goal of our paper is to extract part of the training data by effectively contrasting two model checkpoints—before and after unlearning. This contrast is derived from a parametric approximation (Sec. 4.1) that simulates the unlearning process and enables direct guidance in generation based on the difference between the two models. Designing extraction attacks without relying on contrastive decoding would be a very interesting direction and is worth exploring in future work.

4. W3: Results on more models

I will appreciate it if more models (e.g., Mistral or Qwen) can be tested so that we can tell whether model architectures have certain impact on the effectiveness of the proposed method.

We add results on Mixtral-8x7B-Instruct-v0.1 by LoRA fine‑tuning on TOFU for two epochs with a cosine learning rate of 1e‑4 and a 10% forgetting set (all other settings followed our default setting in Sec. 5.1). For extraction, we apply a guidance scale of w=2. The improvement is consistent, as shown in the table below:

5. W4: More extraction Baseline:

The tested baseline only includes the generations of pre-unlearned and post-unlearned models. I suggest the authors to consider more SOTA data extraction techniques (e.g. [1]).

Thank you for the suggestion and the reference. We agree that comparing against more SOTA extraction methods is valuable. Accordingly, we include additional experiments using ETHICIST on the TOFU dataset with Phi-1.5, under the 10% forgetting setting, as a comparison. Notably, ETHICIST assumes a stronger threat model that the attacker has access to part of the model’s training data. To align with this setting, we assume the attacker has access to half of the retained set in TOFU (while remaining blind to the target extraction set, i.e., the forgetting set), and use it to train the soft prompt. The hyper-parameters of ETHICIST follow the configuration provided in the original paper’s open-sourced code.

Experimental results show that ETHICIST alone can partially improve the extraction rate. However, since ETHICIST is designed to extract training data from a single model, it is complementary rather than conflicting with our method, which leverages the difference between pre- and post-unlearning models. In fact, combining ETHICIST with our guidance-based approach leads to even stronger extraction performance.

6. W5 & Q5: Parameter tuning issues:

The proposed method introduces multiple issues of parameter tuning, which may hinder its application under realistic scenarios

the proposed ``model guidance" introduces multiple issues about parameter tuning (e.g. guidance scale, filter strictness, and training iterations). I am afraid this would bring difficulties under realistic applications.

We agree that our method performs best under well-tuned hyper-parameters, and we acknowledge that such tuning may not always be feasible in real-world scenarios. However, we emphasize that our method consistently outperforms the baseline as long as the guidance weight w is not set too large. In particular, applying model guidance to steer generation away from the post-unlearning distribution is consistently beneficial. As shown in Fig. 5 and Fig. 6, settings with w<2.0 yield much better extraction results across both Phi and LLaMA-2-7B compared to the baseline case of w=1.0.

7. Q1: More discussion on the threat model:

I wonder whether this setting can work under realistic scenarios, and is there any references can be provided? How will the overall framework run when adversaries can only get access to the post-unlearned model?

We consider open-weight scenarios (such as those on HuggingFace) to be the most practically significant. For example, an attacker targeting a medical fine-tuned LLM may have partially extracted sensitive data using the original model and saved both the model and the results. Once the model is later updated to forget specific data, the attacker can re-run extraction using our method. As for related references, while we are not aware of prior work specifically addressing this threat model in the LLM setting, similar targets on extracting unlearning data have been explored for simple models, including linear regression[1].

When the attacker only has access to the post-unlearned model, our current method cannot be directly applied. However, it might be possible to adapt the framework by using a general-purpose language model as a reference to construct alternative guidance. This would require a different formulation and methodology, which we leave for future work.

[1] M. Bertran, S. Tang, M. Kearns, J. H. Morgenstern, A. Roth, and S. Z. Wu. Reconstruction attacks on machine unlearning: Simple models are vulnerable. Advances in Neural Information Processing Systems, 37:104995–105016, 2024.

8. Q2: More explanation on the method part:

The statement that ``The behavioral divergence between the two models encodes rich information about the removed data." is interesting to me. Could you provide more discussions or explanations from both empirical and theoretical perspectives?

From an empirical perspective, we observe many cases like the one shown in Fig. 2, where the token most indicative of the forgotten data is not the one with the highest probability under either the pre- or post-unlearning model, but rather the one whose probability drops most significantly between the two. This suggests that the behavioral divergence between models reflects information specific to the removed data.

From a theoretical perspective, the transition from Eq. (2) to Eq. (3) in Sec. 4.1 illustrates why such divergence encodes meaningful information. Specifically, in Eq. (2), we assume a reverse process in which the post-unlearning model could be fine-tuned back to the pre-unlearning model by training it again on the forgotten data for a fixed number of steps. Under this view, the pre-unlearning model’s distribution can be regarded as a mixture of the forgotten data distribution and the post-unlearning model’s distribution. This implies that the pre-unlearning model retains partial information about the forgotten data and requires guidance from the post-unlearning model to fully reconstruct it. The equivalence between Eq. (2) and Eq. (3) supports this interpretation.

9. Q3: Reference Updates

I notice that some references and citations about data extraction and exact unlearning are a little out ot date. I suggest the authors to update certain references in the next version.

Thank you for the suggestion. We will incorporate more recent and relevant references in the next revision.