A Closer Look at Machine Unlearning for Large Language Models

Thank you for your valuable review. Below we respond to the comments in Weaknesses (W).

W1: While the proposed methods show advantageous performance with these aggregated metrics, it remains unclear how they perform under the existing metrics alone.

Thank you for the suggestion. Due to page limitations, we have presented the complete results of different methods for each metric in the appendix, as shown in $\\textrm{\\color{blue}Table 5}$ (Page 20), $\\textrm{\\color{blue}Table 6}$ (Page 20) and $\\textrm{\\color{blue}Table 8}$ (Page 22). Overall, the proposed methods can achieve better results on both existing and proposed metrics in most cases, resulting in advantageous performance on the aggregated metrics.

We also provide some results in the tables below to demonstrate the performance of different methods under existing three metrics, i.e., ROUGE $R$, Probability $P$, and Truth Ratio (TR). Our methods have significant advantages in maintaining performance of the unlearned model on the retain/neighbor set, especially in terms of ROUGE and Probability. At the same time, the results on the forget set are comparable to the previous methods, ensuring a better trade-off between forget efficacy and model utility.

• The forget10 task in fictitious unlearning scenario:

• The real-world unlearning scenario:

Dear Reviewer M8bM,

Sorry for bothering you, but the discussion period is coming to an end in two days. Could you please let us know if our responses have alleviated your concerns? If there are any further comments, we will do our best to respond.

Best,

The Authors

Thank you for your supportive review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q). We have fixed the typos in the Paper Revision.

W1: The proposed methods (3.2 and 4.1) are not consistent to each other.

Thanks for your insightful comment. We appreciate the approach you proposed, but we want to clarify that it is different from our ME (Section 3.2). The goal of ME is to maximize the prediction entropy every time the model makes the next-token prediction on the forget set. We do not assign labels to the model's predictions. However, using a random word sequence as the ground-truth response is still considered supervised fine-tuning. Although the sequence is semantically random (meaningless to humans), it remains a predetermined and explicit label during fine-tuning.

W2&Q3: More discussions on the strengths and weaknesses.

Thank you for the suggestion. Due to page limitations, we have presented the complete results of each metric in the appendix, as shown in $\\textrm{\\color{blue}Table 5}$ (Page 20), $\\textrm{\\color{blue}Table 6}$ (Page 20) and $\\textrm{\\color{blue}Table 8}$ (Page 22) of the Paper Revision.

We provide some results here and discuss the existing and proposed methods on the proposed three individual metrics, namely Token Entropy (TE), Cosine Similarity (CS), and Entailment Score (ES), as follows:

• For untargeted unlearning, ME+GD achieves the best results on the retain set, real author set and world fact set, demonstrating the superiority of preserving the model utility. GA+GD and NPO+GD perform similarly on TE and CS, but NPO+GD has a higher ES, indicating that the outputs of NPO+GD are more factually accurate. In terms of unlearning effect, both ME+GD and the GA-based methods achieve a sufficient level of unlearning, while NPO maintains relatively higher CS and ES. For instance, the ES of NPO+KL remains at 0.15, meaning that 15% of the correct answers from the forget set can still be correctly inferred from the unlearned model's output.
• For targeted unlearning, all baseline methods have extremely low CS and ES across all sets, indicating that the unlearned models tend to produce a rejection template for almost all questions. However, IDK+AP has a comparable unlearning effect on the forget set while maintaining the highest CS and ES on other sets. This means that the unlearned model can correctly respond to general questions, demonstrating the effectiveness of the proposed AP regularization loss.

Q1: Could you elaborate on why TE should be removed?

The following are the reasons for removing Token Entropy (TE) from the aggregated metric of Forget Efficacy:

• First, metrics for evaluating Forget Efficacy should consider the correct answers in the forget set as references, so that the leakage of information can be measured accurately [1,2]. However, TE only evaluates token diversity within the sentence itself and does not use reference sentences, making it unsuitable for evaluating Forget Efficacy.
• Second, for targeted unlearning, because the unlearned model's output on the forget set is a pre-defined rejection template (i.e., TE is always a high value), measuring their TE is unnecessary. For untargeted unlearning, the goal is to make the unlearned model's output deviate from the correct answer, which typically results in outputs consisting of meaningless characters (i.e., TE is always a low value), as shown in $\\textrm{\\color{blue}Table 13}$ (Page 25).

Based on the above reasons, we only use TE as one of the metrics to evaluate the model utility rather than forget efficacy.

Q2: Regarding Weakness 1, is it possible to combine the methods in Section 3.2 and Section 4.2?

Thanks for your insightful comment. However, the methods in Sections 3.2 (untargeted unlearning) and Section 4.2 (targeted unlearning) are designed for different goals. Our ME in Section 3.2 is to make the unlearned model's prediction for each next-token in the forget set close to random guessing, whereas the method in Section 4.2 is to hope that the unlearned model generates a rejection template for the questions in the forget set. Due to inconsistent goals, directly combining their loss functions may not lead to better performance.

To address the reviewer's concerns, we try several combinations, and the results are shown in the table below. Although the combined methods may slightly improve Forget Efficacy, the Model Utility will decrease significantly.

References:
[1] Tofu: A task of fictitious unlearning for llms, COLM 2024.
[2] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning, COLM 2024.

Dear Reviewer w5A6,

Sorry for bothering you, but the discussion period is coming to an end in two days. Could you please let us know if our responses have alleviated your concerns? If there are any further comments, we will do our best to respond.

Best,

The Authors

Thank you for your supportive review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q). We have fixed the typos in the Paper Revision.

W1: Some other metrics have also been proposed for MUL , e.g., Membership Inference Attack (MIA), authors should also include those in the evaluation.

Thank you for the helpful suggestion. The table below shows the results of the different methods on the forget01 task under MIA evaluation. We follow the MIA evaluation in [1], which is one of the few literatures as far as we know that explicitly uses MIA for the evaluation of LLM unlearning. A higher FM (RM) indicates that the samples in the forget set (retain set) are more likely to be non-members and our methods have advantages in most cases.

Although MIA is indeed a feasible direction for evaluating machine unlearning, most prior work on LLM unlearning generally do not consider this metric, as discussed in Section C.1.4 of [2]. We summarize the main reasons as follows:

• First, the most advanced MIAs typically involve training numerous shadow models (sometimes up to hundreds) on subsets of the entire training set [3], which are impractical in the context of LLMs due to the need of access to the pre-training data or fine-tuning a large number of models. MIAs without training shadow models have been demonstrated to overestimate the effectiveness of unlearning [4] due to the inconsistent difficulty of learning and unlearning each sample.
• Second, evidence suggests that even the state-of-the-art MIA for LLMs [5] generally barely perform better than random guessing due to the large amount of training data, limited number of iterations, and the fuzzy boundary between members and non-members [6]. Current MIAs for foundation models are immature, and their evaluation may be unreliable [7].
• Third, the setup of MIA differs from that of LLM unlearning. MIA aims to determine whether a data sample was used to train a model. However, in real-world unlearning scenarios for LLMs (e.g., Section 5.3), the forget set is typically constructed according to the unlearning target [1]. It is not necessary or even difficult to obtain the original subset of the training data, so the samples in the forget set may already be non-members under the MIA setup.

Q1: Shouldn't the fine-tuning be done on "retain set" since forget set is what the model wants to forget?

Thank you for pointing out this potential ambiguity. The term "fine-tuning" here refers to "unlearning fine-tuning" rather than standard instruction tuning, which involves using the forget loss to optimize model parameters on the forget set for unlearning purposes. We apologize for the confusion and will clarify it in the final version.

References:
[1] RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models, NeurIPS 2024.
[2] Large Language Model Unlearning via Embedding-Corrupted Prompts, NeurIPS 2024.
[3] Membership inference attacks from first principles, S&P 2022.
[4] Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy, arXiv.2403.01218.
[5] Detecting pretraining data from large language models, ICLR 2024.
[6] Do membership inference attacks work on large language models, COLM 2024.
[7] Blind Baselines Beat Membership Inference Attacks for Foundation Models, arXiv.2406.16201.

We sincerely appreciate your recognition of our efforts to address the concerns raised in your initial comments. As the review deadline approaches, if you have any additional questions or suggestions, please let us know, and we would be more than happy to provide further responses.

We also present some results here, highlighting those where Probability and TE are completely uncorrelated. In this case, current metrics (including Probability) cannot capture the fluency of model's output.

W3: For factual knowledge, accuracy is more binary—either the information is correct or it isn’t—making similarity scores potentially meaningless.

We calculate the Cosine Similarity (CS) between the model's outputs to the same question before and after unlearning. The primary purpose of this metric is to quantify the change in the semantic space (or embedding space) [4,5] of the model's output for the same question, rather than the output's accuracy relative to the correct answer. Especially in real-world scenarios, the original model's output may not be completely correct. We introduce CS to provide a different perspective compared to word-level matching (e.g., ROUGE) or Probability, to evaluate how models are affected during the unlearning process.

We agree with you about "accuracy is more binary". Actually, the introduced Entailment Score (ES) is based on a binary metric that determines whether the correct answer is truly included in the model's output. We adopt the textual entailment task (a.k.a. Natural Language Inference) for judgment, which has been used in Q&A evaluation [1] and hallucination detection [2,3]. We are also willing to explore more advanced hallucination detection methods to evaluate LLM unlearning in the future.

References:
[1] Accurate and nuanced open-qa evaluation through textual entailment, ACL 2024.
[2] Comparing Hallucination Detection Methods for Multilingual Generation, arXiv.2402.10496v2.
[3] A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, arXiv.2311.05232.
[4] SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation, ACL 2017.
[5] Semantic Answer Similarity for Evaluating Question Answering Model, ACL 2021.
[6] Tofu: A task of fictitious unlearning for llms, COLM 2024.
[7] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning, COLM 2024.

Thank you for your supportive review and suggestions. Below we respond to the comments in Weaknesses (W). We have fixed the typos in the Paper Revision.

W1: The paper doesn’t present a thorough discussion of the limitations of current metrics.

We would like to clarify that our purpose of introducing new metrics is to complement current ones, rather than to replace them. Evaluating LLM unlearning remains a challenging task in practice due to the inaccessibility of golden retain models and the flexibility of natural language outputs. Thus, considering all metrics from different perspectives together will lead to a more comprehensive evaluation.

Below we discuss the limitations of current metrics and our motivation to introduce new metrics:

• Current metrics do not consider the fluency of the output when evaluating the model utility. ROUGE measures the word-level similarity between the model's output and the specified correct answer, and Probability measures the model's ability to predict the specified correct answer. Therefore, we introduce Token Entropy (TE) to measure the token diversity in the output, which is an important perspective to evaluate the model utility.
• A higher ROUGE does not always indicate factual correctness and may also be a hallucination. We introduce Entailment Score (ES) to measure the factual correctness of the model's output relative to the ground truth answer based on the textual entailment, which is also used in Q&A evaluation [1] and hallucination detection [2,3].
• Current metrics, while considering various perspectives, are all based on natural language text or predicted probabilities. We use Cosine Similarity (CS) to measure the changes in the model's output before and after unlearning in the semantic space (i.e., embedding space), which is also used to evaluate other Q&A tasks [4,5].

In conclusion, we recognize the importance of current metrics and attempt to provide some insights into LLM evaluation from various perspectives; this is only one of our contributions in our paper.

W2: I am particularly concerned about the token diversity metric, which aims to measure fluency—already somewhat captured by the probability metric.

We want to clarify that Token Entropy (TE) and Probability are different in both evaluation purpose and applicable scenarios.

• The purpose of Probability is to measure the model's ability to predict the specified correct answer, so the calculation must use fixed question-answer pairs, which limits the applicable scenarios.
• TE is specifically designed to measure the fluency (i.e., token diversity) of the model's output. TE does not require any reference sentences, allowing it to be used for any individual question, which is useful when evaluating the model utility on more general knowledge.

For the retain set, we indeed obverse that the reduction of Probability may lead to the reduction of TE, which is reasonable since a low Probability typically indicates that the model's answer to the question has been corrupted, making it difficult to produce fluent output.

However, Probability cannot replace the role of TE due to the differences in evaluation purposes. In Sections 5.1 and 5.2, we follow the previous work [6,7] and use two extra sets, i.e., Real Authors and World Facts, where each question is paired with a single-word answer rather than a complete sentence, as shown in the first row of $\\textrm{\\color{blue}Table 1}$ (Page 3). To accommodate this format while maintaining the evaluation purpose, they slightly modified the calculation of Probability.

Specifically, each question $q$ is treated as a multiple-choice question with choices $\\{y\_1, \\dots, y\_n\\}$, where $y\_1$ is the only correct answer. The Probability of the correct answer is then calculated as:

This difference is defined in the previous work [6,7], and we just strictly follow their calculation of current metrics. We apologize for not emphasizing this difference and have added it in Section D of the Paper Revision.

Dear Reviewer qSfC,

Sorry for bothering you, but the discussion period is coming to an end in two days. Could you please let us know if our responses have alleviated your concerns? If there are any further comments, we will do our best to respond.

Best,

The Authors