Large Scale Knowledge Washing

We sincerely thank the reviewer for the detailed comments and suggestions! We would love to address the concerns as below:

[W1] Lack of results on model unlearning benchmarks (e.g., TOFU [1]):

Thank you for suggesting TOFU as a potential benchmark. TOFU (CoLM 2024) is a recent benchmark that uses free-form text for knowledge editing and unlearning. However, our work focuses on the knowledge-triplets setting, which is a widely adopted and standard setup in this area of research. While TOFU represents an important direction, it falls outside the scope of our current work. We agree that unlearning from more flexible, free-form text is an important challenge, and we discuss this setting in the related work and future directions sections of our paper. Expanding LAW to handle free-form knowledge editing remains an exciting avenue for future work.

[W2] In-depth analysis of LAW’s ability to retain unrelated knowledge:

To address the concern about retaining unrelated knowledge, we conducted additional experiments using neighborhood prompts from the zsRE and CounterFactual datasets. These prompts probe the model for unrelated knowledge after washing specific facts. The results are summarized in the table below:

These results demonstrate that LAW retains unrelated knowledge comparably to MEMIT in the cases of zsRE and CounterFact. This might be because we are using MEMIT as the initialization (as illustrated in Line 256-262) so there are some similarities between the performances of MEMIT and LAW (ours).

Theoretically, while methods like MEMIT [2] and its predecessor ROME [3] claim to edit knowledge, they do so by injecting new keys and values (k and v) into the existing key-value spaces (K and V), as shown in Eq.(2) of ROME and Eq.(9) of MEMIT. This effectively adds new knowledge rather than removing it. Because the model’s capacity is finite, this injection can disrupt unrelated knowledge, as older knowledge competes for capacity. In contrast, LAW does not inject new knowledge. Instead, it actively removes the unwanted key-value pairs $K_w$ from the existing keys $K$, as shown in Eq.(10). This fundamental difference allows LAW to perform clean unlearning while minimally affecting unrelated knowledge. We will include this analysis in the revised manuscript to clarify why LAW is more effective in retaining unrelated knowledge.

[1] TOFU: A Task of Fictitious Unlearning for LLMs
[2] MASS-EDITING MEMORY IN A TRANSFORMER
[3] Locating and Editing Factual Associations in GPT

We sincerely thank the reviewer for the detailed comments! We address these concerns as below:

[W1] Consideration of additional baselines: We appreciate the reviewer for highlighting the work of knowledge sanitization [2], which is in the category of gradient-based unlearning methods. In our experiments, we have already compared LAW with four representative gradient-based methods, including the basic FT-UL and more recent work SeUL (Wang et al., 2024a), WOH (Eldan & Russinovich, 2023), FT and ME-FT (Gangadhar & Stratos, 2024). Our results show strong evidence that model-editing-based methods (baseline MEMIT and our method LAW) largely outperform the gradient-based unlearning methods (FT-UL, SeUL, FT, ME-FT), demonstrating that in this large-scale knowledge-washing problem model-editing based methods might be more suited.

We will discuss this new reference in the related work as one of the gradient-based unlearning methods. Moreover, we will discuss the comparison above between model-editing-based methods and gradient-based unlearning methods and point out that gradient-based methods, in general, have larger power for unlearning any large text corpus rather than our large-scale knowledge-washing problem.

[W2] Details of Equations: We would like to mention that the calculation of $KK^T, K_w$ in Eq.(10) and $K$, $V$ in Eq.(2) are all the same as the calculation of MEMIT [1]. We provide the details below and we will also add more clarifications in our paper.

• In Eq.(10), we have $K$, $K_w$ which are two important variables. As for $K$, the regularization term $\frac{\parallel \hat{\Delta} K \parallel_F^2}{\parallel K \parallel_F^2} \leq \beta$ can be simply transformed into: $\frac{ \hat{\Delta} (K K^T) \hat{\Delta}^T}{ KK^T} \leq \beta$, thus we we need to calculate $K K^T$, which can be estimated using an additional WikiText dataset with 100,000 text examples (The detailed process of estimating $K K^T$ is mentioned in appendix B.3 of [1]). Then as for $K_w$, $K_w$ is the input key to the target linear layer within the MLP that we want to edit. For the given knowledge ChatGPT is developed by OpenAI, we input ChatGPT is developed by into the model, and extract the hidden states of the last word by before inputting into the target layer, this is $K_w$ in practice. We will add these clarifications to the paper.
• In Eq.(2), we have $K$ and $V$, $KK^T$ is calculated the same as the above. Note that we can obtain the covariance matrix $KK^T$, but $K$ is still not accessible as $K$ basically represents all the knowledge stored in the linear layer. As we do not know how much knowledge is in the layer, the number of columns in $K$ is not possible to know. It is the same case for $V$, $V$ is also not accessible. However, after some transformation, we can obtain Eq.(4), where $K_e$ means the hidden states before the target layer corresponding to the knowledge to be edited, which is accessible as it corresponds to the specific knowledge that we want to edit. We will add these clarifications to our paper.

[W3] New models like Llama: We understand the importance of extending the scope of our experiments to include more recent models such as LLaMA. However, our current implementation is built on top of the MEMIT repository, which provides effective implementations only for GPT-2 and GPT-J. While we attempted to adapt MEMIT for LLaMA (specifically LLaMA2-7B), the efficacy score did not exceed 0.75, compared to the 0.96+ observed for GPT-2 and GPT-J (as shown in [1]).

We hypothesize that this discrepancy could stem from several configuration challenges (not sure which is the exact reason but they all could be the reasons):

1. Identifying the appropriate layers to edit.
2. Fine-tuning the value of $\lambda$ in Eq. (15) of [1].
3. Using a more suitable dataset to estimate the covariance matrix $E_k[kk^T]$. The dataset Wikitext (noted in Appendix B.3 of [1]) potentially may not be well-aligned with LLaMA’s pretraining corpus.

Addressing these potential issues would require substantial adjustments that are unrelated to our proposed method. For these reasons, we chose to focus on GPT-2 and GPT-J, where MEMIT operates effectively, to provide a fair and focused evaluation of LAW.

Nonetheless, we recognize the potential of extending LAW to more recent architectures like LLaMA, and we see this as a promising direction for future research. We will clarify this limitation in the revised manuscript.

[1] MASS-EDITING MEMORY IN A TRANSFORMER
[2] Knowledge Sanitization of Large Language Models

We sincerely thank the reviewers for the positive feedback and detailed comments! We appreciate it so much and would like to address the concerns as follows:

[W1] Validaty of Generalization: We appreciate the reviewer’s concern regarding the generalization of knowledge unlearning. To address this, we conducted additional experiments using paraphrased queries from the zsRE and MCF datasets. This approach probes the model’s knowledge under varied rephrasings to evaluate whether LAW effectively unlearns knowledge beyond the specific case. The results are summarized in Table 1 below:

These results demonstrate that our method achieves robust generalization, as the model consistently forgets the target knowledge across paraphrased queries. This indicates that LAW does not merely remove specific instances but generalizes well to related prompts. We will include these findings in our paper to strengthen the discussion on the generalization capability of LAW.

[W2] Model Fluency: We acknowledge the importance of assessing the fluency of the model’s generated text post-unlearning, as mentioned in the abstract. To address this, we evaluate the perplexity of the model on a validation set (we choose the first 500 examples from the snapshot CC-MAIN-2024-10) from Fineweb-Edu, a widely-used pre-training dataset. Perplexity serves as a proxy for fluency, as lower perplexity indicates more coherent and fluent text generation. The results are as follows: (As a reference, the perplexity of GPT2-XL and GPT-J-6B on the validation set is 2.66 and 2.31, repsectively)

These perplexity scores indicate that LAW minimally impacts the fluency of the model’s outputs, preserving its ability to generate coherent and high-quality text. On the contrary, unlearning methods with negative loss could affect the model’s fluency drastically during the case of large-scale washing. We will add these results and the corresponding discussion to the revised paper. Here we measure the generalization abilities using zsRE and CounterFactual as they have existing paraphrased questions in the dataset. For Wiki-Latest, we have to use some instruct model (like gpt-4o-mini) to rephrase the prompt and possibly need to do some post-processing to make sure the correctness of the paraphrased questions. We can certainly collect a subset of Wiki-Latest but that would lose the point of "Large-Scale". So we currently have not done it and we hope to expand the dataset and provide some paraphrased questions in our future work.

We sincerely thank the reviewer for the detailed comments and suggestions! We address the concerns as below:

[W1] Knowledge vs. Reasoning: By “reasoning,” we refer to tasks that do not require the model to have prior domain knowledge, such as context-based question answering, mathematical reasoning, etc. These are more abstracted concepts about the dataset property rather than the model’s specific modules or functionalities. We will clarify this definition further in our paper, explaining that knowledge and reasoning are treated as distinct task types for our evaluations.

[W2] Insufficient discussion on disentanglement: Our paper empirically shows the possibilities of knowledge-reasoning disentanglement by washing a large amount of knowledge without affecting reasoning abilities. We will add more discussions in the later sections (the experiments part and the conclusion part). As for theoretical validation, our current understanding is that the attention layers might not be as critical for encapsulating knowledge or reasoning, with MLP layers playing a more significant role. Given that MLP layers can be decomposed into keys and values, it is conceivable that some keys and values pertain to reasoning ability, while others relate to knowledge, and some might be relevant to both. Exploring the hypothesis that certain keys and values are associated with reasoning ability could serve as a theoretical basis for disentangling knowledge and reasoning. However, we acknowledge that identifying these specific keys and values is a complex task that warrants further investigation. We consider this an intriguing avenue for future research and thank the reviewer for highlighting its importance.

[W3] The limited scope of knowledge washing: We acknowledge that our method currently focuses on triplet-structured knowledge, which may seem limited. However, triplet formats are widely accepted as a structured form of factual knowledge. We are keen to explore extending LaW to handle unstructured knowledge in future work, as mentioned in our conclusion. We thank the reviewer for pointing out DINM, which provides a relevant perspective on free-form knowledge editing, and we will incorporate a discussion of this approach into our paper to broaden the scope and consider DINM as a baseline after extending LaW into unstructured knowledge washing.

[Q1] The choice of reasoning benchmark: The benchmarks we selected (LAMBADA, HellaSwag, ARC_easy) are widely used in the lm-eval library, ensuring that models like GPT2-XL and GPT-J-6B achieve meaningful scores. This makes them suitable for our comparison as they represent various reasoning tasks in an accessible way for our chosen models. However, we acknowledge that logical and numerical reasoning benchmarks, such as GSM8K, could further clarify reasoning evaluation. We include the results on GSM-8K as shown in the table below:

From these results, we can see that this result is consistent with the results in the paper: our model can minimally affect the model’s reasoning abilities and the effects are comparable with using MEMIT (though ours can wash knowledge more thoroughly.)

[Q2] The choice of using <|endoftext|> as the new output: Consider a factual statement “ChatGPT is developed by OpenAI”, we also would like to mention that the appropriate object in this example is OpenAI and there is no other objects here. This leads to a different setting as model editing.

• As there is no other object here, we need to add a new object to formulate the unlearning problem into model-editing problem. We are the first to propose using <|endoftext|> as the new object, which serves as a novel approach to indicate “deletion”. To the best of our knowledge, none has directly applied model-editing methods in unlearning tasks thus we need to propose some adaptation ourselves. Using <|endoftext|> serves as a potential proxy for forgetting, as the model no longer produces content after the input.
• Using <|endoftext|> means that we are editing this factual statement into “ChatGPT is developed by <|endoftext|>”, while LaW proposes to forget the relation between ChatGPT and OpenAI so the distribution over the vocabulary after “ChatGPT is developed by” is disturbed. Fundamentally, LaW propose to delete the relation between ChatGPT and OpenAI but using <|endoftext|> is editing the relation. Thus we argue our proposal LaW intuitively makes more sense and it also shows better performances in our experiments.

[Q3] Clarification of Model Architectures: Thank you for raising this point! We will clarify that our method is model-agnostic and applies to any transformer-based model with MLP layers, regardless of architecture specifics such as parallel or serial attention-MLP configurations. This will be added in Section 5 to prevent any potential confusion.

Dear Reviewer Y2iC,

We are glad that we have addressed your concerns, thank you so much!

Best,
Authors