Knowledge-localized Unlearning for Faithful Forgetting in Language Models

Thank you for your feedback. We will address each of your questions with sincerity.

$\\\\$

1. Details of the evaluation metrics (Weakness # 1)

1-1. The rationale for the evaluation results

Our evaluation framework early stops the unlearning process when UA <= 0.33 (random sampling from three options) is satisfied (Section 5.1). Therefore, it is natural that some algorithms tend to excessively unlearn the given forget set and reach UA score below 0.33.

In contrast, WMDP [1] enforces unlearning of the datasets until a fixed number of training batches (e.g., 150, 300, or 500 iterations) is reached. This rigidity makes WMDP inflexible, often leading models to struggle to achieve even the level of random guessing.

In conclusion, the observed results are derived from the gap in the evaluation processes of the two benchmarks (early stop during iterations vs. fixed iterations).

1-2. FathUnBench vs. WMDP

Our paper tackles one of the most challenging scenarios: unlearning real-world knowledge, considering its complex and interconnected nature.

We fully acknowledge that WMDP deals with challenging scenarios, focusing on unlearning knowledge across various professional domains (biosecurity, cybersecurity).

However, existing benchmarks, including WMDP, have not accounted for the interconnectedness of knowledge since they have evaluated unlearning methods only on knowledge to forget and other independent knowledge. In the "Rebuttal for All Reviewers", we provide an in-depth discussion of the characteristics of our benchmark in comparison to others (at the top).

$\\\\$

2. More models and baselines (Weakness # 2)

Our method is a model-agnostic method and can be flexibly adapted to any unlearning method. Accordingly, we applied our method to GA$_{ret}$ and it as the primary comparison target. However, we acknowledge that RMU is one of the most powerful backbone methods among recent unlearning methods; thus, we conducted experiments with RMU and also attempted to apply our method to RMU.

For the RMU experiments, we searched for $\alpha_{RMU} \in \lbrace20, 50, 100, 150, 200, 300\rbrace$ and $lr \in [1e-5, 3e-3]$. We also used $c=20$ and $l=7$, following the implementation details on the original GitHub page [2]. In the KLUE+RMU experiments, we adopted the same settings as the RMU but with a key difference in layer selection for updates. RMU manually selects three lower layers ($l-1$, $l-2$, and $l-3$) for updates, but KLUE+RMU automatically searches for neurons to update. For KLUE settings in KLUE+RMU, we selected $\alpha_{KLUE}=10$ and $p=0.2$, using a higher neuron ratio than KLUE+GA$_{ret}$ since only the first to sixth layers are updated

We also adopted NPO [3] with searching for hyperparameters of $\beta \in [0.1, 0.5]$ and $lr \in [5e-6, 1e-4]$. To demonstrate the scalability of our method, we also conducted experiments on the Llama 3.2 (3B) model as a new backbone.

Gemma-2 (2B)

$\\\\$

Llama 3.2 (3B)

In conclusion, KLUE enhances the unlearning abilities for various metrics even if it is applied to RMU.

$\\\\$

3. Comparison with Harry Potter books [4] (Question # 1)

Our work focuses on evaluating the faithfulness of unlearning methods. From an unlearning research perspective, it is very easy to forget the text itself (even Harry Potter books). For example, we can destroy the model by preventing it from generating natural outputs.

However, preserving other knowledge after forgetting a specific text is highly challenging. Additionally, ensuring the generalizability of unlearning is also difficult. Therefore, we decided to unlearn the simplest form of question (the Base QA form) and evaluate the generalizability of unlearning while preserving the model's other knowledge.

$\\\\$

If there are unsolved questions about the reasonability of our paper, we would appreciate being asked additional questions.

$\\\\$

[1] The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

[2] https://github.com/centerforaisafety/wmdp

[3] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

[4] Who's Harry Potter? Approximate Unlearning in LLMs

3. The distribution of localized neurons (Weakness # 3)

We agree that conducting more examinations of the localized neuron distribution would enhance the quality of our paper. To address this, we have further analyzed the distribution of localized neurons. We compute attribution scores for each question and derive Jarcard and Cosine similarity between the attribution scores, where Jarcard similarity is computed using the identified knowledge neurons set, and Cosine similarity is computed using the attribution score distribution. We consider five types of questions for each target question: (1) Paraphrased question, (2) Same-entity question (same person but different contextual question), (3) Multi-hop question, (4) Same answer different context question, and (5) Unrelated question (randomly sampled). We sampled 402 questions, which included all these five types of questions, and computed the mean and standard deviations for the derived similarities. The experimental results are shown in the table below:

$\\\\$

$\\\\$

As shown in the table, “paraphrased questions” show the highest similarity. Surprisingly, 'same-answer different context questions' show high similarity, although they are in a different context; the result shows that knowledge is largely determined from output texts. This result clearly justifies the necessity of our evaluation process using 'Same-answer QA Evaluation'.

In addition, 'same-entity questions' and 'unrelated questions' show the lowest similarities. These results show that questions sharing the same entity are regarded as completely different knowledge if they are in different contexts (e.g., "Where was Barack Obama born?" vs. "What country does Barack Obama hold citizenship in?").

$\\\\$

4. The Quality Assessment for the GPT-4o (Question # 1)

We adopt the question generation process following MQUAKE [1] since natural language question generation using ChatGPT variants is a kind of common and powerful method to generate questions. Even existing unlearning benchmarks, MUSE, KnowUnDo, TOFU, and RWKU, also adopt ChatGPT variants for generating natural language questions.

However, We have fully acknowledged that additional investigation should be conducted to ensure the dataset quality. Therefore, we conducted a human evaluation to investigate the quality of the generated questions. Specifically, we recruited crowd workers fluent in English through the university’s online community and let them investigate 800 generated natural language questions (10% of the entire questions). The results revealed 0% of the error rate, ensuring the quality of our constructed benchmark.

$\\\\$

If there are remaining unsolved questions about the reasonability of our benchmark and method, we would appreciate being asked additional questions.

$\\\\$

[1] MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions

Thank you for the thorough feedback on our study. We will offer explanations, individually addressing the queries and questions you hold.

$\\\\$

1. Detailed comparisons with existing datasets

In the "Rebuttal for All Reviewers," we discuss the characteristics of our benchmark in depth (at the top).

$\\\\$

2. The positive impact of neuron localization (Weakness # 2 & Questions # 2)

2-1. How did KLUE avoid selecting unrelated neurons?

The attribution score (gradient × activation) quantifies the relevance of neurons in predicting a text output, as it represents their contribution to the prediction [1]. Activation refers to the feature value, while the gradient indicates the direction and magnitude of the activation's influence on the output logit. Therefore, the attribution score can be used to compute a neuron's contribution to the output logit. As demonstrated by our experimental results (Section 5.3), using this score to identify knowledge-related neurons helps avoid unrelated knowledge when localizing neurons.

$\\\\$

2-2. How did KLUE enable contextual unlearning?

To address the shortcomings of neuron localization, KLUE incorporates 'Superficial Knowledge Regularization' to mitigate shortcut unlearning, which merely reduces output text probability regardless of the given context. Our method enables contextual unlearning and improves the performance of MA$_{f}$ by excluding neurons associated with other unrelated knowledge during the unlearning process.

We also present additional case studies for KLUE on Gemma-2 (2B).

$\\\\$

$\\\\$

2-3. Why did we select 'attribution' to identify knowledge neurons?

We investigated several neuron localization methods [2,3] for detecting knowledge-relevant neurons in language models. However, these methods have only been evaluated on classification tasks and have not been applied to text generation tasks. The 'Skill Neurons' [2] method requires extensive computation to adapt it for text generation tasks, as it necessitates evaluating every possible word-piece combination to determine the knowledge neurons' distinguishing capabilities. Similarly, the 'Model Grafting' [3] method requires an additional training process to create masking parameters (used to identify knowledge neurons) equal in number to the model's parameters. Due to these implementation challenges of existing methods, we selected 'attribution' as our knowledge neuron detection method.

$\\\\$

[1] Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination [2] Finding Skill Neurons in Pre-trained Transformer-based Language Models [3] Task-Specific Skill Localization in Fine-tuned Language Models

We sincerely appreciate your thorough examination of our paper. Your considerate evaluation and constructive feedback are truly valued. We will provide responses to the questions you have requested.

$\\\\$

1. Detailed comparisons with existing datasets (Weakness # 1)

In the "Rebuttal for All Reviewers," we discuss the characteristics of our benchmark in depth (at the top).

$\\\\$

2. Details of our benchmark in considering different knowledge of each entity as the independent evaluation component (Weakness # 2)

We generate multiple knowledge questions for each entity and consider them independent elements for evaluation. For example, suppose the question "What is the country of citizenship of Tom Cruise?" is included in the forget set. Then, another question "Where was Tom Cruise born?" can be included in the test set (for measuring TA and MA scores). We adopt this framework to consider a more general use case of unlearning, differentiating from existing benchmarks like TOFU [1] (TOFU aims to forget the entire knowledge of each entity). In Table 6 of our paper (Appendix), we present that each unlearning cluster is constructed only on one knowledge question of each entity (not all knowledge).

$\\\\$

3. Evaluating the unlearned knowledge about different languages

We translate our dataset into French and German to determine whether KLUE can completely erase the knowledge in different language settings. Specifically, we translate the forget set (5%) into French and German for the experiments. The below table shows the examples of translated questions.

We unlearn the Gemma-2 (2B) model using KLUE for the questions of the original language (English) and evaluate the unlearned model in translated questions (French and German). The table below shows the experimental results of the French and German questions. The results reveal that KLUE has a generalization ability to other languages.

$\\\\$

[1] TOFU: A Task of Fictitious Unlearning for LLMs

Thank you for your response. After careful consideration, I’ve decided to keep the score as it is.

Thank you for taking the time to review our paper so thoroughly. We greatly value your thoughtful insights and constructive suggestions. We will ensure that your questions are addressed in our response.

$\\\\$

1. Detailed comparisons with existing datasets

In the "Rebuttal for All Reviewers", we discuss the characteristics of our benchmark in depth (at the top).

$\\\\$

2. The justification for using the knowledge graph (Weakness # 1)

Most questions can be expressed in the knowledge graph; thus, our setting can cover an extensive area of real-world knowledge.

$\\\\$

3. Larger neuron ratio (p) experiments (Question # 1)

We conduct the experiments on a larger neuron ratio to investigate the KLUE method further for Gemma-2 (2B).

We reveal that even the larger ratios show comparable results, however, simply increasing the neuron ratio does not enhance the performance.

$\\\\$

4. Various prompting experiments (Question # 2)

We fully acknowledge that the quality of our research will be enhanced if we conduct additional experiments using various prompts.

$\\\\$

Different prompt templates

We first paraphrase the instruction of the utilized prompt template (Appendix B.1) and create five new instructions as follows:

1. Pick the appropriate option for the question from the provided options. You should answer without further explanation.
2. Select the correct answer for the given question from the options. Write only the word without explanation.
3. Answer the given question by choosing the appropriate answer from the given options. Do not include any explanations.
4. Select the correct answer to the following question among the options. Only the exact word should be written, with no explanation.
5. Select the proper answer to the question from among the given options. Write only the exact word without any additional explanation.

We conduct the evaluation on these newly selected instructions using the unlearned Gemma-2 2B KLUE model, and the experimental results are shown below:

From the experiments, we reveal that the newly adopted prompts perform similarly to the original prompt. Their performance on the UA score is slightly higher than the original one since we early stopped the unlearning process based on the UA score evaluation for the original prompt.

$\\\\$

Other ways of prompting

We also evaluate the original Gemma-2 (2B) and the KLUE unlearned Gemma-2 (2B) models without any instruction. Specifically, we input only a question (e.g., "Who is the mother of Michelle Obama?") to the model without using any instruction or prompt. We measure the performance using ROUGE-L, and the experimental results are shown in the below:

Of course, ROUGE-L has its limitations, as it does not account for exact matching. However, its evaluation results are highly correlated with MCQA outcomes and can serve as an alternative evaluation metric.