There are some concepts that need further clarification or justification. For instance, what do the labels, i.e., [0,0,1,1,0], in Figure 1 mean?

The [0, 0, 1, 1, 0] in Figure 1 is a corruption mask, where 0's mean the token embeddings at those positions are not corrupted, and 1's mean the token embeddings are corrupted at those positions. On TOFU, we use a separate token classifier to select tokens that are relevant to names and only corrupt those tokens' embeddings. On WMDP and copyrighted content tasks, we corrupt all tokens.

The Note in Section 3.3 is not quite clear.

There might also be cases where the surrogate retain value is not possible to obtain (e.g., no suitable evaluation metrics exist, or the chosen evaluation metric might not be enough to reflect if a model actually exhibits forgetting in the real-world use cases, etc.). In these cases, the model service provider could instead conduct red teaming with human annotators using responses generated by different $\sigma$'s.

What is the embedding function , and how can we detach it from a black-box LLM?

The embedding function is the embedding layer in an LLM. We separate it from the model because the proposed corruption scheme only applies to the output of the embedding layer and does not affect all other layers in the model. Specifically, we consider an LLM as a function $f=\tilde{h}\circ\mathbf{e}(x)$, where $\mathbf{e}(x)$ produces the token embeddings, and $\tilde{h}$ is mapping from the token embeddings to the last layer output logits.

The claim about ``potential fuzzy boundary between retaining and forgetting" needs further justification or citations.

We will add citations to our reference to the "potential fuzzy boundary between retaining and forgetting" with [1, 2, 3], where they contain evidence showing unlearning in one domain could induce harm in closely-related or irrelevant domains.

What does $r$ mean in Eq. (2)?

The $r$ and $f$ in Equation (2) mean retain and forget, respectively. Here, we mean to say that if the classifier's probability of the prompt being in the forget distribution is larger than the probability of the prompt being in the retail distribution.

Does it mean to use the same $\sigma$ in Line 237?

Yes, this means that we run the optimization once on the task of forgetting 1% examples in the TOFU dataset and obtain a $\sigma^*$. Then, we reuse the same $\sigma^*$ (obtained on the 1%) in the 5% and 10% setting for both Llama 2 and Phi-1.5.

[1] Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., & Kolter, J. Z. (2024). Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121. [2] Lynch, A., Guo, P., Ewart, A., Casper, S., & Hadfield-Menell, D. (2024). Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835. [3] Qiu, X., Shen, W. F., Chen, Y., Cancedda, N., Stenetorp, P., & Lane, N. D. (2024). PISTOL: Dataset Compilation Pipeline for Structural Unlearning of LLMs. arXiv preprint arXiv:2406.16810.

We appreciate your thorough feedback and constructive criticism of our paper. Below, we respond to the weaknesses and questions you raised in your review.

Although using a prompt classifier during inference to identify the unlearning state is straightforward, it also makes it less applicable for end users.

As we mentioned throughout the paper, our proposed method targets large and powerful models with chat interface or API access. This is because gradient-based methods are usually extremely expensive to perform, especially in scenarios where frequent unlearning requests are the norm, which is the main focus in this work.

The pre-trained prompt classifier requires additional time and effort for unlearning tasks. We need additional datasets to train this prompt classifier.

For most unlearning scenarios, we usually need at least a forget set to unlearn, just like the unlearning baselines we have compared to. A retain set can help mitigate performance degradation, and can be easy to obtain. For example, SOTA unlearning methods like SCRUB (https://arxiv.org/abs/2302.09880), LLMU (https://arxiv.org/abs/2310.10683), RMU (https://arxiv.org/abs/2403.03218) require both retain data and forget data to work well, which we also use to train our classifiers. So we are using the same data as one would use in a traditional unlearning algorithm, and no additional time and effort is required.

In fact, training a classifier is way cheaper than updating the LLM itself. For all classifiers we have trained, the most “expensive” one only requires 30 minutes on a single NVIDIA RTX 4090, while a decent unlearning method like SCRUB takes more than 12 hours on two NVIDIA A100’s.

Some potential prompts may still trigger harmful responses or copyrighted content.

We conducted an additional study using multiple types of challenging queries either written by humans, including rephrasing, adversarial, adding irrelevant context, jailbreak, and including only keywords. Below are the false positive/negative rates (in percentage) of the classifier on the perturbed queries:

Based on these statistics, we believe that our classifiers remain robust under these common perturbations, even without training on these types of data. We continue to explore if we could improve the robustness of our classifiers further. We construct another set of perturbed prompts, all synthetically generated by Llama-3-70B-Instruct, and have no overlap with the previous perturbed evaluation set, and has no knowledge about the perturbation types. We use this set of prompts to train our classifier, and observe that both the false positive rate and the false negative rate can be further improved.

We also perform additional training on the WMDP and copyrighted content classifiers and the result still holds:

WMDP

Copyrighted content

How do you get the surrogate retain metric value $\hat{v}_{r}$?

Equation (8) aims to optimize the retained model's performance on the relevant task. However, since obtaining a retained model is impractical, we use a surrogate score to estimate its performance. The $\hat{v}_r$ in Equation (8) represents an estimated score for the retained model's performance on evaluation metrics. For example, in the WMDP task, we set $\hat{v}_r = 0.25$ to match the random-guessing level (25%) for multiple-choice tasks. For more complex metrics in the TOFU and copyrighted content tasks, $\hat{v}_r$ is determined using human-written responses for 100 training prompts, simulating the expected retained model responses (e.g., refusals, "I don't know," incorrect answers). These pseudo responses are used to calculate the surrogate retain metric value.

How do you design the dataset to train the prompt classifier?

For the TOFU dataset, both the retain and forget sets are available from the dataset itself, so we can directly use them. We follow the practice of the TOFU paper, which uses both the full retain set and the forget set for unlearning.

For the WMDP dataset, we use GPT-4-Turbo to generate 100 synthetic questions for each of the biology, chemistry, and cybersecurity as the training data for the classifier. We also validate that none of the generated questions appear in the forget set questions.

For the copyrighted content classifiers, we use the actual copyrighted content as the positive examples (which we hope the classifiers can predict positive). We use books and news articles from different sources as negative samples for the Book and News tasks, respectively.

We provide further detail and clarification on how the training sets of the classifiers are constructed in Appendix C.3.

Thank you for your exhaustive feedback and the constructive criticism of our paper. Below we address the weaknesses and questions provided in the review above.

However, most (all?) of the results in the main text would seem to be equally well served by the simple classifier + template mechanism. Can you clarify where the benefit of the corrupted prompt responses is shown?

Indeed, a simple classifier and a template mechanism can already mitigate many risk types. The benefit of the corrupted prompt responses, as opposed to a simple classifier + template mechanism, lies in ensuring indistinguishability, thereby mitigating privacy risks. In our early experiments, we find that even if a model has not been trained on a piece of data, it frequently provides a hallucinated response instead of answering “I don’t know.” Therefore, our aim is for the corruption mechanism to generate results akin to the "retained" model, effectively masking whether specific data has been unlearned.

In contrast, a template response could inadvertently reveal the classifier's training on a particular individual. For example, in the context of entity unlearning, when the model generates a template response, an attacker would know that it’s highly likely the classifier has been trained to identify this individual. Having this knowledge, an attacker could continue to exploit the individual’s characteristics, behaviors, or conditions.

You talk about the expected metric ratio of unlearned versus retained under the forget prompts as the measure that would reflect this, but I didn't see any results like this.

For the expected metric ratio of unlearned versus retained, the ratio is reflected in all three groups of experiments: For TOFU, it is the forget quality shown in Figure 2. This is the distributional similarity between the outputs of an unlearned model and a retained model, which is even a stronger guarantee. A larger forget quality indicates a ratio close to 1, in distribution. The WMDP authors assume that a model that has not been trained on relevant domain knowledge should have random-guessing accuracy on the multiple-choice questions. The closer the score is to 0.25, the closer the expected ratio of unlearned to retained metrics is to 1.

For copyrighted content unlearning, we use average similarity gap (ASG), which measures the gap (retained model outputs vs original text similarity) - (unlearn model outputs vs original text similarity) shown in Table 3. The closer the gap, the closer the expected ratio of unlearned to retained metrics is to 1.

It seems like the surrogate model plays an important role. I didn't understand it from the main text.

Here, while we use the term surrogate model, the surrogate retain metric value does not come from a separate model. Instead, the $\hat{v}_r$ in Equation (8) represents a score one "thinks" a retained model would obtain on the relevant evaluation metrics. Given that the retained model is not accessible, we have to derive this score based on the problem. For example, in the WMDP task, the goal of unlearning is to reduce the accuracy of relevant multiple choice tasks to a random-guessing level (25% because all questions have four possible choices), which we expect a retained model would have. So we can set the surrogate retain metric value $\hat{v}_r = 0.25$ directly, and running the optimization will then push the "unlearned metric value" closer to the surrogate retain metric value $\hat{v}_r$.

Were there any cases where you could include the classifier + template response in the results? It would be interesting to see how much better the metrics match from doing corruption instead of returning a template

For TOFU, the core evaluation metric, forget quality, examines if the unlearned model's raw output distribution matches that of the retained model. This prevents us from doing a classifier + template response, because the evaluation does not rely on the text output. However, as ROUGE-L is one of the evaluation metrics used in TOFU and it is a text-based similarity measure, we compare if the ROUGE-L of the retained model is close to the template responses using the same metric in Table 3. Below we show that ECO maintains the highest similarity to the retained model’s responses.

For copyrighted content unlearning, we also include the results from using template response (on BBC News task) as follows. The following table corresponds to Table 3.

Line 79-80 were confusing. You assume in-distribution queries, and I think you meant to say users do not attempt to jailbreak...

Yes, we meant to say that the users do not attempt to jailbreak the classifiers. We also conducted additional experiments, which show that our classifiers are also robust against different types of perturbed queries even if we did not explicitly train the classifiers to be robust. We include the detailed response in the comment below.

We appreciate your kind recognition of our efforts and your thoughtful feedback. Below, we would like to provide clarification regarding the forget quality metric and additional questions in the comment above.

1. The forget quality metric is not based on multiple choice but on the output distribution of the retained model and the unlearned model. The TOFU paper [1] uses the forget quality to measure indistinguishability in the output distribution of two models (retained and unlearned), given some data (which we explain in the following paragraph). To compute the forget quality, we first need to compute another metric known as the truth ratio (see section 2.2.2 in https://arxiv.org/pdf/2401.06121). The truth ratio measures, given a model and a prompt, how likely the correct answer is to an incorrect answer. Below, we quote the original interpretation from the TOFU paper for a better understanding:

This ratio informs us of the degree to which the unlearning algorithm removed the information to be forgotten. Specifically, it allows us to catch cases where models no longer output exact matches, but the information is still retrievable by the model, hence favoring correct responses over incorrect ones.

To further compute the forget quality, we perform a Kolmogorov–Smirnov (KS) test, a statistical test measuring the difference between two cumulative distributions. We perform KS test on the two distributions of truth ratio calculated from the retain model and the unlearned model, respectively, following exactly the method in the TOFU paper. The KS test results in a p-value, and a high p-value means that we cannot reject the null hypothesis that the two distributions are the same. In our experiments, we obtained p-values over 0.95 in all forget sizes and models considered. Therefore, our approach achieved distribution similarity to a retained model in the models' output probability distribution.

Note that the distributional similarity in the output distribution is a direct consequence of making the model answering the corrupted prompt. Another way to put it is: the model's probability of generating the original answer will be reduced significantly on the corrupted tokens.

2. We do provide a full evaluation of text-based metric, ROUGE-L (between the generated text after unlearning and the original correct answer), in Table 10 and Table 11 in the appendix (page 30 and 31), due to the page constraint. Our method can have ROUGE-L scores close to that of the retained model, and can significantly reduce the ROUGE-L score to extremely low (if we want to). We also provide real generated examples on page 32, which again shows the effect of unlearning on text.

3. On WMDP, we provided a probing experiment in D.3.1 (page 35). While this task is on multiple-choice, we show that attackers cannot recover the correct answer even if one gains access to the raw model logits.

We hope the above clarification answers your question in the comment above.

[1] Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., & Kolter, J. Z. (2024). Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121.

We appreciate your recognition of our efforts and your thoughtful comments. Below we address the question about knowledge entanglement.

Can you describe knowledge entanglement in detail or define it formally? How far is it from structural unlearning, where the removal of one entity might impact knowledge relevant to its neighbours?

In the LLM unlearning domain, we have only seen explicit discussions of knowledge entanglement from two prior works [1, 2]. The TOFU paper describes this as "unexpected forgetting" of other things when one wants to unlearn a specific piece of information, and they also relate this to catastrophic forgetting in continual learning [3]. The authors observed this phenomenon across different models and multiple gradient-based unlearning algorithms. Another study [2] also observed unintended damage in closely related domains when they unlearned information about Harry Potter. However, it seems that a formal definition of knowledge entanglement is lacking. In our paper, we consider knowledge entanglement as the performance drop on datasets that are not relevant to the unlearning target (i.e., law/jurisprudence, math/physics, economics/econometrics, etc.) and show that our method is typically not affected by such problems and outperforms other baselines.

The PISTOL paper [4] is closely related to the concept of knowledge entanglement, where unlearning leads to unintended forgetting of relevant concepts in the knowledge graph. Since the datasets are synthesized from knowledge graphs, investigating structural unlearning and knowledge entanglement should be easier. The graph serves as the ground truth, allowing access to the forgetting of relations or nodes.

While a precise definition of knowledge entanglement remains elusive, we will conduct a more thorough literature review on the topic and include a section discussing the relevant research. We will also cite the PISTOL paper and discuss structural unlearning in the revised version of the paper.

[1] Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., & Kolter, J. Z. (2024). Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121.

[2] Lynch, A., Guo, P., Ewart, A., Casper, S., & Hadfield-Menell, D. (2024). Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835.

[3] McCloskey, M., & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation (Vol. 24, pp. 109-165). Academic Press.

[4] Qiu, X., Shen, W. F., Chen, Y., Cancedda, N., Stenetorp, P., & Lane, N. D. (2024). PISTOL: Dataset Compilation Pipeline for Structural Unlearning of LLMs. arXiv preprint arXiv:2406.16810.

Thank you once again for your thoughtful feedback and for raising the scores! We feel fortunate to have such a knowledgeable and insightful reviewer.

Your comments and suggestions, especially 1) on the literature we have overlooked and 2) on how to provide a clearer presentation of the key concepts, have been invaluable, and we are eager to incorporate these insights to further enhance the final version of our paper.

Thank you for your response. I appreciate the detailed explanation and the interesting directions you've proposed for future research. Given your commitment to incorporating relevant discussions on these points in your revision, I have decided to increase the scores. I look forward to seeing how these insights will enhance the final version of the paper

We are grateful for your comprehensive review and the valuable insights you provided. Below we address the weaknesses and questions provided in the review above.

The optimization objective in Equations (8) and (9) for the corruption function incorporates the distance measure function between the corrupted LLM and the retained LLM, which is saying some 'unlearned metric value'. This raises concerns for me. What specific measure function was employed in the experiments?

Equation (8) aims to optimize the retained model's performance on the relevant task. However, since obtaining a retained model is impractical, we use a surrogate score to estimate its performance. The $\hat{v}_r$ in Equation (8) represents an estimated score for the retained model's performance on evaluation metrics. For example, in the WMDP task, we set $\hat{v}_r = 0.25$ to match the random-guessing level (25%) for multiple-choice tasks. For more complex metrics in the TOFU and copyrighted content tasks, $\hat{v}_r$ is determined using human-written responses for 100 training prompts, simulating the expected retained model responses (e.g., refusals, "I don't know," incorrect answers). These pseudo responses are used to calculate the surrogate retain metric value.

We would also like to highlight that our method imposes an unlearned state over the LLM subjected to unlearning. This means that an LLM service provider could simply adjust the corruption strength value $\sigma$ in real time. The optimal strength obtained from the optimization stage can serve as an initial guess, and one can directly increase $\sigma$ for more unlearning or disable it to recover the original LLM based on the specific unlearning request during model serving. Compared to other baseline methods requiring offline model updates, having an online tunable $\sigma$ which imposes an unlearning state is a part of the contribution of our approach and a clear advantage over those baselines.

I am confused about the selection of the surrogate retain metric value in line 186, can you elaborate more? What do you mean by if it's not possible to obtain?

There might also be cases where the surrogate retain value is not possible to obtain in real-world (e.g., no suitable evaluation metrics exist, or the chosen evaluation metric might not be enough to reflect if a model actually exhibits forgetting in the real-world use cases, etc.). In these cases, the model service provider could instead conduct red teaming with human annotators using responses generated by different $\sigma$'s, and select the optimal $\sigma^*$ based on human feedback.

How many token embeddings are selected to be corrupted? Figure 1 only shows partial corruption of the input, while Equations (10) and (11) indicate corruption of the entire input embedding e, which is very confusing.

On the TOFU dataset, we also use an additional token classifier to select only name tokens to corrupt, as this task is about forgetting entities. For WMDP and copyrighted content removal, we corrupt all tokens in the prompt.

The false positive rate of the employed classifier could inevitably affect overall performance. It's possible the classifier might exhibit false positives on challenging examples, such as perturbed queries related to the knowledge intended to be forgotten. Given that the forget data for training is quite small, for example, 400 samples for ToFU-10%, is there a risk that the classifier overfits on the training forget data? Can it generalize to other paraphrased versions of the forget data?

We conducted an additional study using multiple types of challenging queries either written by humans, including rephrasing, adversarial, adding irrelevant context, jailbreak, and including only keywords. Below are the false positive/negative rates (in percentage) of the classifier on the perturbed queries:

Based on these statistics, we believe that our classifiers remain robust under these common perturbations, even without training on these types of data. We continue to explore if we could improve the robustness of our classifiers further. We construct another set of perturbed prompts, all synthetically generated by Llama-3-70B-Instruct, and have no overlap with the previous perturbed evaluation set, and has no knowledge about the perturbation types. We use this set of prompts to train our classifier, and observe that both the false positive rate and the false negative rate can be further improved.

We also perform additional training on the WMDP and copyrighted content classifiers and the result still holds:

WMDP

Copyrighted content

Thank you for your feedback on our rebuttal.

Regarding the surrogate metric, we agree that in the previous experimental setup, we select the $\hat{v}_r$ in different ways for each task. Here, we just performed an additional experiment using dataset/task-agnostic selection of $\hat{v}_r$. Specifically, we use Llama-3-70B-Instruct to generate 100 synthetic responses of either 1) stating "I do not know the answer to the question," or 2) refusal (e.g., "I cannot answer the question"). These responses are all independent of the datasets and tasks considered. Then, we also use the LLM subjected to unlearning to generate its original response (before unlearning). We use the four text similarity metrics used in the copyrighted content unlearning task to measure the difference between 1) the original responses and 2) the synthetic IDK/refusal answers, and minimize this difference for all three tasks. In essence, we want to push the model output toward IDK or refusal using the zeroth order objective toward minimal textual similarity between the model output and template responses, regardless of the task.

Due to the time constraint, we only conducted this experiment on Llama-2-7B-Chat, which is a model used in all three tasks.

Note that the score in row 3 does not change because we used the same way to select $\hat{v}_r$ for copyrighted content tasks. Here, we show that task-agnostic selection still maintains the effectiveness of unlearning. We will add this particular experiment and result and conduct it on more models as time allows in our revised paper.

Regarding the classifier, I agree that simple classifiers can be challenged in more complicated scenarios. In the future, we will experiment with incorporating external moderation techniques like LlamaGuard or ShieldGemma, which are trained specifically to guard relevant content and are much more robust against jailbreak prompts.