The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning

We thank you for your thoughtful evaluation and for recognizing the clarity of our motivation, the strength of our experimental results, and the potential impact of our automatically constructed evaluation method for unlearning.

On the description of table 2 and section 5.2, you are correct. Table 2 reports results generated using our constructed benchmark, seeded with RWKU entities, not the original RWKU benchmark itself. We agree that this distinction should be more clearly conveyed.

We will make the following changes in the revision:

• Table 2 caption will be updated to be explicit and state that the evaluation is conducted using our dynamic benchmark instantiated with RWKU entities.
• Line 250 and surrounding text in Section 5.2 will be revised to avoid any ambiguity and to clearly distinguish between evaluations on our benchmark vs. original RWKU.
• We will add explicit references to Appendix Tables 4 to 7, which report results on both Phi-4-mini and on original static benchmarks like RWKU and TOFU, for readers seeking a broader view.

We appreciate your suggestion and will clarify the above points to improve overall readability.

We thank you for the positive assessment of our work and the recognition of our dynamic, model-specific evaluation approach. We are glad you found our results both rigorous and impactful in revealing residual knowledge missed by static benchmarks. We address each of your concerns below:

Feedback1: The evaluation is primarily conducted on the Llama 3.3 8B model, limiting the generalizability of the findings.

We appreciate the your concerns regarding generalizability. Our primary experiments use LLaMA 3.1-8B due to its wide adoption in unlearning research. Additionally, we also evaluate our benchmark on Phi-4-mini-Instruct (Table 4) and IBM Granite-3.2-8B-Instruct (see below) to ensure that the evaluation is effective in comparing unlearning methods regardless of the model architecture. Across models, we observe consistent relative performance rankings of unlearning methods when compared to previous evaluation methods (for e.g., Spearman’s ρ > 0.8 for IBM Granite 3.2-8B-Instruct), supporting the generalizability of our framework.

Feedback2: Since the benchmark relies on knowledge extracted from each model prior to unlearning, comparisons across models become difficult. This raises concerns about the consistency and fairness of cross-model comparisons, as variations in the constructed knowledge graphs may lead to differences in evaluation difficulty rather than reflecting true differences in unlearning performance.

Importantly, our benchmark evaluates each model relative to its own knowledge, i.e., we only include a probe if the pre-unlearning model correctly answers it. This ensures that we are testing whether a model has truly forgotten something it previously knew, rather than penalizing it for never having encoded the information in the first place. This approach directly addresses the issue raised by you: models differ in their training data, architecture, and knowledge cutoff dates, making a fixed set of static probes unfair and uninformative for cross-model comparisons.

By contrast, our dynamic, model-informed method avoids this issue. It adapts to what each model actually encodes before unlearning, providing a fairer and more accurate assessment of forgetting. In this sense, the concern about inconsistent knowledge graphs demonstrates a key strength of our approach: static benchmarks impose a one-size-fits-all test that fails to account for model-specific prior knowledge, whereas our method ensures evaluations are tailored, valid, and comparable within each model. In the revision, we will motivate how the goal of unlearning evaluation is not the “comparisons across models” but rather to compare across different unlearning methods in the same model, and emphasize how our design accounts for differences in model training data and knowledge cutoffs, ensuring that evaluations reflect forgetting rather than missing prior knowledge.

Hi,

The discussion period ends tomorrow. Please take time to review the authors' response and update your review accordingly.

Thanks!

We thank you for the clear summary and positive assessment of our work. We appreciate your recognition of our dynamic, model-specific evaluation approach and the importance of probing with multi-hop and alias-based queries. We are glad you found our results both rigorous and impactful in revealing residual knowledge missed by static benchmarks. We address your concerns below:

Feedback1: The graph construction cost is not well discussed. Since the graph is built by repeatedly querying the model and expanding nodes, how many model calls are needed per entity? This affects scalability, especially if this method is to be used in practice. There's mention of API budget and decay factor, but no concrete numbers or analysis of resource usage.

We thank you for highlighting this important consideration regarding scalability in real-world deployments. We note that the knowledge graph needs to be constructed only once per model, knowledge domain (seed entity) pair. Once built, it can be reused to evaluate multiple unlearning methods, making the associated API cost/ model call a one-time overhead rather than a recurring burden.

We agree that a more detailed breakdown of model call usage during graph construction would improve clarity. In the revised version (Section 4.1, Appendix A.6), we will provide a comprehensive analysis of the API calls incurred under varying graph depths and decay factors.

Specifically, each node in the graph requires a minimum of three LLM queries: one for entity elicitation, one for atomic fact extraction, and one for alias resolution. Since the graph is expanded via a breadth-first search with an exponential decay factor (α), the number of nodes, and consequently, the number of model calls, it grows sub-exponentially based on equations 1 and 2.

For example, under a decay factor of α = 0.8, we empirically observe that the average number of nodes per seed entities in the RWKU dataset at depths 1, 2, and 3 are approximately 57.6, 103.7, and 140.5, respectively. This results in a total model call count ranging from 228 to 1,942 per entity, depending on graph density, alias resolution needs, and retries due to API response exceptions. We will include these statistics in the appendix and expand our discussion to include practical scalability strategies such as batching, early pruning of low-relevance nodes, and our implementation of parallelized graph construction.

Feedback2: There’s a potential issue with using model outputs as ground truth for what should be forgotten. For instance, prompting (A, wrote, B) may succeed, but the reverse (B, was written by, A) may fail even before unlearning. This asymmetry (reversal curse) can introduce noise into the evaluation: are we really measuring unlearning failure, or just limits in how the model responds to different phrasings?

Thanks for raising this point. While such surface form asymmetries can exist, we do not view them as a major confound in our evaluation. First, we include all variant probes, forward, reverse, and alias-based, which are correctly answered by the pre-unlearning model. This ensures that post-unlearning failures reflect forgetting quality independent of the phrasing artifacts. More importantly, we believe that regardless of phrasing, an effective unlearning method should prevent any elicitation of forgotten knowledge. If a model can still retrieve the target fact under a different surface form, it signals an unlearning failure. We will clarify both our filtering protocol and this conceptual stance in the revision.

We thank you for their thoughtful summary and positive feedback on our submission. We are glad that you found our dynamic evaluation framework for LLM unlearning to offer strong motivation, methodological rigor, and transparent discussion of limitations. Your recognition of our framework’s ability to automatically generate multi-hop queries and uncover new failure modes, while maintaining coverage and ranking consistency with established benchmarks, is much appreciated. We are happy to work with you to further improve the paper.

Hi,

The discussion period ends tomorrow. Please take time to review the authors' response and update your review accordingly.

Thanks!

I have nothing more to add in the discussion period.