Probing Hidden Knowledge Holes in Unlearned LLMs

We sincerely thank all reviewers for their detailed and constructive feedback. We are encouraged that they found our work to address a timely, important, and underexplored problem (b9QX, A6Wn) with an innovative and comprehensive framework (b9QX, hrSN, 1Bz9, A6Wn). We also appreciate the positive notes on our clear presentation (b9QX, hrSN, 1Bz9, A6Wn). We will provide a detailed response to address each reviewer's concerns below.

---

W1 & Q4. Dependence on LLM Judge and Sensitivity

Thank you for this question regarding the sensitivity of our results to the choice of judge model. To directly investigate this, we conducted a new set of experiments, following the other reviewers' suggestion. Specifically, we used full-parameter unlearned models, which were generated from three different seeds. We then evaluated their responses using three different judge models: gpt-4o-mini, gpt-4.1-mini, and o3-mini.

The results lead to a clear conclusion. While the absolute scores show minor variations across the judges, our key findings are consistent. Specifically, across all three judge models, performance on both $D_{AP}$ and $D_{LP}$ is degraded after unlearning, compared to the original, pretrained model. Moreover, all three judges show that our latent probing set, $D_{LP}$, reveals significant knowledge holes.

---

We will add the details in our revised version.

W2 & Q5. Focus on Q&A-style prompts and other tasks

Thank you for this clarifying question. Our findings do indeed extend beyond the Q&A format. As detailed in our paper, we intentionally evaluated RMU, which unlearns knowledge from a corpus of biological documents, not question-answer pairs. By showing that our probing method successfully identifies severe knowledge holes in the RMU-unlearned model, we demonstrate that this failure mode is a general side effect of unlearning, not an artifact of the Q&A format. Exploring other tasks, like code generation, is a valuable future direction for unlearning. However, we believe this is beyond the scope of our paper.

---

W3 & Q6. Illustrative Mitigation Strategies and Confidence Bounds

Thank you for your suggestion and clarifying questions. We would like to clarify that our paper's primary contribution is to diagnose the problem of knowledge holes, not to propose a complete solution. As Reviewer b9QX noted, our finding on the onion effect---where fixing one hole reintroduces harmful content or creates new holes---is itself a crucial scientific insight and motivates the need for advanced mitigation strategies.

Regarding confidence bounds, our new experiments presented above were conducted over 3 independent seeds and showed the standard deviations across different runs. More importantly, we respectfully argue that the failure rate (i.e., the frequency of responses scoring 1) provided in our paper is a more crucial metric. This is because it directly quantifies the degradation in model performance compared to the original model, offering a more direct measure of knowledge loss.

We hope these clarifications and additional experiments address your concerns. We would be happy to discuss any further questions or comments.

We would like to further clarify our focus on the Q&A format. This format is a direct result of our framework's design: we use our $D_{adj}$ set to seed an RL policy, which then continuously generates Q&A-style latent probes to find knowledge holes during training.

In response to the reviewer's point, we agree that knowledge holes could occur in other domains. However, reliably probing knowledge holes in specialized tasks such as code generation or summarization is challenging. For example, accurately measuring knowledge degradation in code generation requires a proficient victim model; a model that performs poorly on this task could yield misleading results after unlearning. Our selection of victim models was guided by two practical considerations. First, the models needed a light moderation level to ensure they could generate the harmful responses we aimed to unlearn. Second, for the bio-corpus unlearning task (e.g., WMDP), we used their unlearned models to ensure a direct and valid evaluation. Furthermore, probing summarization would require a policy model capable of overcoming the difficulty of generating diverse and long-context test cases. Therefore, we chose to diagnose the degradation of general, user-facing knowledge with a Q&A format probing set. More importantly, we found this approach to be both direct and effective, as it consistently revealed significant knowledge holes for general and benign queries in unlearned models.

Probing hidden degradation in a specific task, perhaps by designing a reward signal with an appropriate policy model and seed initialization, is an interesting direction for future work. We thank the reviewer again for raising this important point. We will add this, along with a discussion on our reliance on the LLM-as-a-judge, to the Limitations section of our revised paper.

We hope this further addresses the reviewer's concern. Should the reviewer have any remaining concerns, we would be happy to discuss them.

Dear Reviewer A6Wn,

As the discussion period will be closing soon, we would like to kindly follow up and ask if you have any remaining concerns.

We believe we have addressed the concerns you raised in our rebuttal. For instance, to address your point regarding the sensitivity of the LLM judge, we ran a new set of experiments with three different judge models. We found that our key findings are consistent across all judge models. Moreover, we provided the standard deviations from the new experiments, and further clarified the importance of our failure rate metric for measuring knowledge loss.

In addition to these experimental updates, we explained our rationale for focusing on the Q&A format and our perspective on mitigation.

We are grateful for your time and feedback, which have helped us strengthen the paper. Please let us know if you have any other questions.

Best regards,

The Authors of Submission 25003

We sincerely thank all reviewers for their detailed and constructive feedback. We are encouraged that they found our work to address a timely, important, and underexplored problem (b9QX, A6Wn) with an innovative and comprehensive framework (b9QX, hrSN, 1Bz9, A6Wn). We also appreciate the positive notes on our clear presentation (b9QX, hrSN, 1Bz9, A6Wn). We will provide a detailed response to address each reviewer's concerns below.

---

W1. Unlearning on low-rank (LoRA) vs. full-model parameters

Thank you for raising this important question. To address concerns about LoRA, we have run new experiments for the LLMU method using full-parameter finetuning. To further substantiate the results, we performed unlearning with three different seeds (with a smaller learning rate and iterations due to the sensitivity of the unlearning on full parameters), generated responses for $D_{AP}$ and $D_{LP}$ over three independent runs, and evaluated all responses with three different judge models. The results, shown below, demonstrate that the knowledge hole phenomenon is not an artifact of LoRA and is indeed generalizable. Specifically, performance on $D_{AP}$ and $D_{LP}$ is consistently degraded compared to the original model. Moreover, $D_{LP}$ consistently widens the performance gap compared to $D_{AP}$, confirming the necessity of our dynamic probing method. The results below are presented as mean ± standard deviation across the different runs.

---

W2. Fairer comparison of $D_{LP}$

Thank you for your clarifying question. Our goal is to identify and quantify the 'knowledge holes' that are specifically created by the unlearning process, thereby measuring what knowledge the unlearned model has lost compared to its original state.

To achieve this, the test set ($D_{LP}$) is generated using the failure signals from the unlearned model. We then evaluate these discovered prompts on the original model for the crucial reason of confirming that these questions were indeed answerable before unlearning.

The resulting performance gap between the two models clearly demonstrates the knowledge compromised by the unlearning procedure. We note that our goal is not to diagnose pre-existing weaknesses in the original model, but to isolate the specific side effects of the unlearning process itself.

---

W3. Comparison with other adversarial methods (e.g., finding hallucinations)

Thank you for this question. We would like to clarify that our goal is not to conduct a comparative study of which adversarial method finds the most failures. Rather, our contribution is to propose a systematic probing framework designed to evaluate a hidden risk from unlearning that prior works have overlooked. This diagnostic goal is distinct from hallucination-finding, which targets plausible but factually incorrect statements.

---

W4. Reference point for VENDI score

Thank you for the question. Generally, a higher VENDI score is better. To provide a reference point, Table 2 of our paper shows the VENDI scores for standard benchmarks like MT-bench (0.519) and MMLU (0.274). Our generated $D_{LP}$ sets achieve scores (0.35-0.43) that are comparable or higher, demonstrating significant semantic diversity. Furthermore, we provide the diversity score to demonstrate that our framework discovers a sufficiently diverse set of failures and does not simply converge on a few simple failure modes.

---

W5. Scaling results by diversity scores

Thank you for your suggestion. We present the scores separately to offer a more intuitive and complete analysis, as our goal is to highlight two distinct aspects of the knowledge hole problem. First, the judge score (Table 1) measures the severity of the failure on an individual prompt---i.e., how poor the model's response is. In parallel, the diversity score (Table 2), as clarified in our response to W4, demonstrates that our probing discovers a wide range of failures and does not simply converge on a few simple failure modes. Combining these two different dimensions into a single scaled score could obscure this crucial understanding of our work.

---

W6 & W7. Robustness of LLM Judge and Lack of Error Bars

Thank you for these important points. We address both with our new experimental results presented in the table for W1. Specifically: 1) Judge Robustness: By evaluating with three distinct judge models and finding highly consistent trends, we demonstrate our findings are not sensitive to a single judge's bias. 2) Error Bars: By running the new experiments with three different seeds, we now report the mean and standard deviation, providing the requested insight into the variance of our findings. More specifically, these results show that our RL-based dynamic probing is stable and consistently effective at uncovering severe knowledge holes.

---

W8. Formal introduction of $D_{AP}$ and $D_{LP}$

We appreciate the question regarding formal definitions. Our approach intentionally uses a procedural rather than formal definition because our focus is on the practical generation of test cases. In particular, adjacent knowledge consists of benign prompts that share certain keywords or concepts with the forgetting set. Latent Knowledge refers to benign prompts that do not share these keywords, discovered via our RL-based probing. Our primary contribution is an empirical framework for uncovering knowledge holes. For that goal, a procedural definition clearly articulates how each prompt set is constructed and why it targets different types of unintended model failures. Also, this definition ensures that other researchers can replicate our procedure and arrive at the same test sets. We believe this approach offers sufficient rigor and clarity for evaluating unlearning methods, though future work could further formalize these concepts if desired.

---

W9. Appendix E typo

Thank you for your careful reading. We will correct the reference to Appendix D in our revised paper.

We hope these clarifications and additional experiments address your concerns. We would be happy to discuss any further questions or comments.

We sincerely thank all reviewers for their detailed and constructive feedback. We are encouraged that they found our work to address a timely, important, and underexplored problem (b9QX, A6Wn) with an innovative and comprehensive framework (b9QX, hrSN, 1Bz9, A6Wn). We also appreciate the positive notes on our clear presentation (b9QX, hrSN, 1Bz9, A6Wn). We will provide a detailed response to address each reviewer's concerns below.

---

W1, W3: Contribution and Novelty (Distinction from Prior Work)

We thank the reviewer for thoughtful feedback and for their broad understanding of prior work in machine unlearning. We respectfully argue that our work's primary contribution is not to simply re-demonstrate the brittleness of the current unlearning methods, but to introduce a probing framework for a different and more insidious class of failure that prior work does not systematically address.

Specifically, prior works [1, 2] analyze perturbations that are linked to the forget set (e.g., neighbor knowledge or combining forget/retain prompts). In contrast, our work identifies a different failure mode: damage to model utility triggered by benign prompts that are semantically unrelated to the forgotten content and are generated model-dependently. This approach reveals hidden risks that methods focused on predefined or neighboring test cases cannot fully capture.

Moreover, the work in [3] is a general red-teaming tool for eliciting undesirable behaviors. Our goal is different and specifically tailored to probing the unlearned models. We diagnose the loss of benign capability by finding prompts that elicit low-quality or nonsensical responses to benign questions.

Therefore, our work provides an important diagnostic tool that complements static benchmarks by adaptively discovering how and where each unlearning process uniquely damages a model's general knowledge. This quantifies a non-trivial and previously overlooked risk, offering a critical insight for developing more robust unlearning methods and contributing a new dimension for diagnosis to the unlearning community.

---

W2: Relevance of Probing Given Failures on Simpler Perturbations

Thank you for your comment. We agree that unlearning can fail on smaller perturbations. However, our method is not simply a harder test for this known instability. Rather, it provides a different angle on the risks of unlearning by probing a broader, more insidious class of failures. While prior tests primarily reveal the local instability of the unlearning process, our work offers a more comprehensive diagnostic perspective, capable of assessing both local side effects (through our adjacent knowledge probing) and the broader, latent nature of unintended knowledge loss. A deeper understanding of these diverse failure angles will help future research to more thoroughly design and evaluate unlearning methods.

We hope this addresses your concerns. We would be happy to discuss any further questions or comments.

---

References

1. Position: LLM Unlearning Benchmarks are Weak Measures of Progress
2. RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
3. Eliciting Language Model Behaviors with Investigator Agents

We sincerely thank all reviewers for their detailed and constructive feedback. We are encouraged that they found our work to address a timely, important, and underexplored problem (b9QX, A6Wn) with an innovative and comprehensive framework (b9QX, hrSN, 1Bz9, A6Wn). We also appreciate the positive notes on our clear presentation (b9QX, hrSN, 1Bz9, A6Wn). We will provide a detailed response to address each reviewer's concerns below.

---

W1, Q1. Generalizability (LoRA vs. Full-parameter finetuning)

Thank you for raising this important question regarding the generalizability of our findings beyond LoRA. In our preliminary tests, we found that full-parameter unlearning is highly sensitive; for instance, using the same learning rate (i.e., 5e-5) as in our LoRA experiments, the unlearning loss often reached its maximum threshold within just a few iterations. Therefore, we ran new experiments for the LLMU method using full-parameter finetuning but with a smaller learning rate (i.e., 1e-6).

To further substantiate the results, we performed unlearning with three different seeds. From each of these three models, we then conducted three independent response generation runs for $D_{AP}$ and $D_{LP}$, and evaluated all responses with three different judge models. The results, shown below, demonstrate that the knowledge hole phenomenon is not an artifact of LoRA and is indeed generalizable. Specifically, performance on $D_{AP}$ and $D_{LP}$ is consistently degraded compared to the original model. Moreover, $D_{LP}$ consistently widens the performance gap compared to $D_{AP}$, confirming the necessity of our dynamic probing method. The results below are presented as mean ± standard deviation across the different runs. Percentages indicate the proportion of prompts that lead to scores of 1 on the response.

---

W2. Base models listed in the body

We appreciate the reviewer’s attention to detail. We agree that this is an important experimental detail and will move the base model information from the Appendix to the main body of the paper for better clarity.

---

Q2. Practical frequency of knowledge holes

Thank you for this insightful question. While it's challenging to quantify the exact probability of encountering a specific knowledge hole, our key finding is that these failures are triggered by benign and common questions that a user would expect any capable assistant to handle.

To illustrate this, we provide five diverse examples below, drawn from our new full-parameter unlearning experiments. For a more systematic analysis of the broad topics these failures cover, we refer to our clustering analysis in Appendix C.5 (Tables 13-16).

Qualitative Examples of Latent Knowledge Holes.

As these examples show, knowledge holes are not limited to rare edge cases but can cause failures on everyday queries across many domains. The practical significance of our work lies in providing a methodology to uncover this unpredictable and widespread degradation of model utility on benign, real-world questions.

We hope these clarifications and additional experiments address your concerns. We would be happy to discuss any further questions or comments.

Dear Reviewer b9QX,

As the discussion period will be closing soon, we would like to kindly follow up and ask if you have any remaining concerns.

We believe we have addressed the concerns you raised in our rebuttal and new experiments. Specifically, to address your main concern about LoRA, we have conducted new experiments with full-parameter finetuning. The results demonstrate that the knowledge hole phenomenon is generalizable and not specific to the usage of LoRA.

Additionally, to answer your question about the practical frequency of these issues, we have provided qualitative examples of benign, common questions that fail after unlearning, along with the clustering results in our appendix.

We are grateful for your time and feedback, which have helped us strengthen the paper. Please let us know if you have any other questions.

Best regards,

The Authors of Submission 25003