Learn while Unlearn: An Iterative Unlearning Framework for Generative Language Models

W1. About information entropy

Entropy is a direct measure of the "amount of information" in a variable [1]. For information entropy, we calculate the amount of information in the output of the model. If the output is mere repeated text, the entropy will be low, and that shows the correlation between entropy and information content.

W2. L369: How are these thresholds selected?

We follow the setting of thresholds from KUMPR [2].

W3. About GPT evaluation

Thank you for the comment. This is because unlearning effectiveness cannot be fully captured by a single type of metric. Metrics such as EL, MA, and BERTScore focus on unlearning performance, while metrics like Entropy, Cls Avg., and Dia Avg. assess general model capabilities. For instance, methods like KUMPR excel at unlearning but compromise general ability, whereas our method balances effective unlearning with preserved general performance.

Additionally, while GPT evaluation may not strongly correlate with other metrics, the human evaluation results demonstrate a strong correlation, further validating our method’s effectiveness.

MW2. About contradiction of using BERTScore and BLEU

BERTScore and BLEU are not used as stopping criteria in our method. They are just used during Iterative Unlearning Refinement (IUR) to mark whether to exclude samples from the unlearning set. The stopping criteria are still based on the EL and MA metrics. But we actually report EL, MA and BERTScore. The exclusion of BLEU is that it is a hard threshold based on n-gram overlap, and the value is rather close to 0.

MW3. L377: "our method consistently achieves the best or second-best performance ...". It seems that ICU

It seems the reviewer's comment is incomplete. From my understanding, the reviewer is not convinced of the performance of our method. But we have also conducted human evaluation and GPT evaluation to fully show the effectiveness of our method.

Others

Regarding minor and very minor weaknesses on notation, we have updated the manuscript to:

• Update the notation of BERTScore and BLEU in Equation 5 (MW1.)
• Use consistent descriptions and notations (VMW1., VMW2., VMW4.)
• Explain what prefix and suffix are (VMW3.)

Q1. Equation 5: Did you evaluate other stopping criteria, such as ROUGE or NLI-based metrics?

In our view, the choice of stopping criteria can be flexible. The key is that they should effectively reflect unlearning performance. We use BLEU and BERTScore as our metrics because they are popular in the field and represent statistical and semantic similarity respectively. However, our method is a framework so other metrics, like ROUGE or NLI-based measures, can also serve as stopping criteria. They can be easily integrated into our approach and the effect can be evaluated in future work.

Q2. About the detail of the human annotation process

1. The scale ranges from 1 to 8 because each value corresponds to a specific evaluation standard, and we designed eight distinct levels of scoring.
2. There was one expert involved, who has been well trained.
3. The instruction is included in the updated version of manuscript. Due to character constraints, you may refer to manuscript for details.
4. We regard the ill-generated text as not useful, so the annotators should score it as 1.

Q3. About the equal trend between the GPT and human evaluation

The absolute mean score is not in scale, but we may argue that for different parameters of models, the rank of human scores is equal to the rank of GPT scores except for only the comparison of Original GPT-Neo-125M and OPT-125M.

And for statistical test, we normalize the human score $s$ by $10\frac{s-1}{8-1}$, and the Pearson correlation is 0.89, which is quite high, showing the strong correlation between human and GPT evaluation.

Q4. About including random generation as a sanity check

Thank you for your suggestion. We have conducted experiments on random generation as a sanity check. The results are shown below. Due to time constraints, we only manage to conduct experiments on the 125M model. As can be seen, the EL$_{10}$, MA, and BERT scores are all lowest among all methods, except for $KUMPR$, since the latter method makes the model output repeated text. Here we use random prefix and use the orginal referenced suffix.

Q5. Did you consider other LM families?

Actually we conducted experiment on TinyLlama, and you may refer to Appendix C in the manuscript for the results.

[1] Vajapeyam et al. Understanding Shannon's entropy metric for information. 2014.

[2] Jang et al. Knowledge unlearning for mitigating privacy risks in language models. 2022.

W1. About the epochs reported in Table 1

Thank you for your suggestion. The epochs are just reported as a reference in Table 1 and the number of epochs is not the key point of our method.

In fact the experiment in our paper is conducted on the size of 128 instances at one time, while the Table 2 in KUMPR is conducted on the size of 32 instances. But actually the epochs is still not the same as they reported in the appendix. We have also raised the question to the authors of KUMPR and they agreed to what we have found.

In our opinion, the design of KUMPR actually leads to the poor performance. And the more epochs they use, the worse the model performs since the unlearning process actually undermines the model's generation ability.

W2. About mismatch between experiment and motivation

We thank the reviewer for the insightful comment. If the original dataset $\hat{D}$ is available, it can be directly used as the analogous dataset $D = \hat{D}$. However, if $\hat{D}$ is not available, an analogous dataset $D$ can be constructed independently, with no direct relationship to $\hat{D}$. For example, as we introduced in Line 218-219, we can use Wiki as the database to retrieve analogous data when the original dataset is inaccessible.

Q1. Am I correct in understanding that the analogous dataset is defined as $D = \hat{D} - D_{\mathrm{fgt}}$ and that the size of $D_{\mathrm{lrn}}$ matches $D_{\mathrm{fgt}}$?

Yes, the size of $D_{\mathrm{lrn}}$ is 1-1 proportional to $D_{\mathrm{fgt}}$. Every instance in $D_{\mathrm{fgt}}$ is paired with one instance in $D_{\mathrm{lrn}}$

Q2. About the cost of ICU

The analogous set can be updated if necessary, but it is equally feasible to keep the database unchanged. Indeed, the cost of unlearning aligns with the cost of fine-tuning on $D_{\mathrm{fgt}}$ and $D_{\mathrm{lrn}}$. Specifically, $\mathcal{L}*{\mathrm{fgt}}$ and $\mathcal{L}*{\mathrm{lrn}}$ involve standard fine-tuning processes. For $\mathcal{L}*{KL}$, $P*{\theta_0}(x_{t}^{\mathrm{lrn}}|x_{<t}^{\mathrm{lrn}})$ can be precomputed once and reused, while $P_{\theta}(x_{t}^{\mathrm{lrn}}|x_{<t}^{\mathrm{lrn}})$ is calculated concurrently with $\mathcal{L}_{\mathrm{lrn}}$ to optimize efficiency.

Q3. About gain achived with $\alpha$

The hyperparameter $\alpha$ plays a crucial role in controlling $\mathcal{L}_{\mathrm{\mathrm{lrn}}}$, which governs the learning of paired data.

We also conducted extra experiment setting $\alpha = 0.1$ and $\beta = 1.0$ and the results are shown below. They clearly demonstrate that $\alpha$ is a key factor. When $\alpha$ is higher, the model tends to produce more like paired data. This can lead to a decrease in $MA$. On the other hand, lower $\alpha$ undermines the model's general ability on the downstream task. This shows our chosen value for $\alpha$ strikes a balance between these two effects.

Q4. To clarify my understanding, what is the $D_{\mathrm{\mathrm{fgt}}}$? is this the "@ThingsExpo 2016"?

Actually, the case is only one instance during the unlearning process. The $D_{\mathrm{\mathrm{fgt}}}$ contains 128 instances, and the case is just one example. For each instance, we have prefix and suffix. And the goal is actually to unlearn the model so that the model may not output refrenced data with regards to the prefix.

Q5. For the human expert annotation, how many humans were used for that study and how is expertise defined?

There was one expert involved, who has been well trained. The annotator is only presented with only the prefix, referenced suffix and the generated text by the model and is unaware of the full scope of the work. The instruction is shown below:

Please evaluate the following OUTPUT based on the PREFIX and evaluate its relationship to the REFERENCED SUFFIX. The scores are as follows:

1 - Exact repetition or redundancy
2 - Minimal divergence with little information
3 - Substantial divergence and poor coherence
4 - Partial coherence with divergence
5 - Partial similarity and coherence
6 - Paraphrased similarity
7 - Creative and reasonable divergence
8 - Fully reasonable divergence

We regard the ill-generated text as not useful, so the annotators should score it as 1.

W1. Lack comparisons with prior methods

We acknowledge the lack of comparisons with [1], [2], and [3] in our experiments. However, there are key reasons for this:

[1] is designed for use in API-hosted models and does not provide updated model parameters, which limits its applicability compared to our approach.

[2] involves integrating additional layers into the transformer architecture, making it challenging to implement and less generalizable across different model architectures.

[3] does not have publicly available source code, preventing direct experimental comparisons.

Despite these limitations, we believe our proposed method provides a more practical and adaptable solution for real-world scenarios.

We conducted an experiment to compare our method with ICUL [1]. The results are shown below.

In-context unlearning does not alter the original model. As a result, the model's performance on downstream tasks remains unaffected. Therefore, we report only the metrics related to the unlearning task. The authors in [1] evaluated their method on deletion sets of up to 10 samples for question-answering and generation task. To align with this setup, we sampled 10 instances in our template. We split these into a deletion set of 5 samples and a learning set of 5 samples. This choice was necessary due to the limited context window of the models used. We then performed in-context unlearning under these constraints.

The results indicate that this method is ineffective with a small context window when dealing with larger deletion sets (128 in our experiment settings). This suggests poor generalization of ICUL in such scenarios.

W2. Consideration of larger and more realistic benchmarks

Thank you for your comment and suggestion. However, we argue that using a larger benchmark may not bring much insight to the unlearning task. This is because the training data is much larger than the unlearned data in most cases.

Regarding the benchmark's realism, we believe the one we used is also realistic. In fact, it is part of the real-world training dataset, Pile, and is also used in a real-world unlearning challenge (https://github.com/google-research/lm-extraction-benchmark).

In addition, our method is not tied to any specific dataset or model backbone. It is a flexible framework that can be applied to any model and dataset.

[1]. Pawelczyk et al. In-Context Unlearning: Language Models as Few-Shot Unlearners. 2023.

[2]. Chen et al. Unlearn What You Want to Forget: Efficient Unlearning for LLMs. 2023.

[3]. Eldan et al. Who's Harry Potter? Approximate Unlearning in LLMs. 2023.

W1. Concerns about the requirement of relevant datasets for ICU

We appreciate the reviewer's concern regarding the generalizability of ICU in scenarios where access to relevant datasets may be limited. However, we would like to clarify that the RMU method proposed in WMDP [1] also requires access to a dataset distinct from the target forget dataset. Specifically, RMU employs a forget loss to align the generated outputs on the forget set with a fixed random variable, and a retain loss to ensure that the updated model aligns closely with the frozen model on a benign dataset ($D_{\mathrm{retain}}$). Notably, WMDP utilizes Wikitext as $D_{\mathrm{retain}}$, which serves a similar role to our $D_{\mathrm{lrn}}$ in preserving the generative capabilities of the model.

In our ICU framework, we follow a comparable strategy by leveraging publicly available datasets like Wikitext to construct $D_{\mathrm{lrn}}$. This approach balances unlearning with the preservation of the model’s overall generative performance. Thus, the requirements of ICU in this regard are not more restrictive than those of RMU, as both rely on access to auxiliary datasets for effective model optimization.

W2. How a and b are selected

We determined the thresholds for BERTScore ($a = 0.3$) and BLEU ($b = 0.01$) based on preliminary experiments. Specifically, for BERTScore, only 2% of outputs fell below 0.3, while the average score exceeded 0.7, with many instances close to 1. For BLEU, the largest value below 0.01 was 1e-78, and the smallest above 0.01 was 0.0159, making 0.01 a clear dividing point. These thresholds effectively balance precision and applicability based on the observed data distributions.

W3. Relationship between loss (3) and (4)

Loss (3) ensures the model generates paired data instead of the original target data, while Loss (4) preserves the generation ability of the original model. For experiments with only (3) or (4), please refer to the ablation study results in Figure 3. Maximizing KLD for unlearned data is unnecessary, as Loss (1) already maximizes the NLL loss for such data. KLD is specifically designed to maintain generative capabilities.

W4. Robustness towards attacks such as paraphrasing or translation

Thank you for your valuable suggestion. We conducted experiments to validate this. The results are shown below. When using a paraphrased and translated prefix, the original model achieves comparable metrics. In contrast, the model unlearned using our ICU method shows significantly lower metrics. This demonstrates that the model forgets the unlearned data itself, rather than just the pattern.

Q1. How many data is needed for KNN sampling? What is the ratio between unlearned data and retrieved data?

The original benchmark contains 15000 data. In our experiment, we unlearn 128 data at a time. The remaining data in the benchmark is used for sampling. For each instance of unlearned data, we sample one paired data. The ratio between unlearned data and retrieved data is 1:1.

Q2. Is the model robust to noise from embeddings/KNN sampling?

We have not conducted specific experiments on this, but it is an interesting question worth exploring. Intuitively, if the retrieved data is random, it would act like using prompts to generate random outputs. While this might affect the unlearned data to some extent, the KL loss can control its overall impact on the model's performance.

[1] Li et al. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. 2024.