LLM Unlearning via Loss Adjustment with Only Forget Data

Thank you for your valuable evaluation of our study and we would like to provide some clarification to address your concerns.

W1: The performance of the proposed method.

FLAT demonstrates consistently competitive performance while requiring less data, striking a good balance between unlearning efficiency and general language capability. Unlike some baselines that utilize the retain data for fine-tuning or reference models, FLAT relies solely on the forget data, making it particularly suited for real-world scenarios with resource constraints. While it may not consistently outperform all baselines leveraging retained data or reference models across every dataset and model, FLAT remains a competitive and practical choice with broad applicability. For comparison, we also report a version of FLAT using the retain data as the good answer, following the NPO paper.

FLAT can achieve the top two best methods across three datasets. FLAT consistently ranks in the top two across unlearning efficiency and model ability and achieves a strong balance in the Harry Potter dataset (Table 3). For MUSE, our method achieves the best forget quality and the highest model utility among all three methods that satisfy the good trade-off criterion (Table 5). As for TOFU-1%, FLAT achieves the best MU and ranks in the top two for Forget Quality (FQ) across all three models. In TOFU-5% and TOFU-10%, FLAT ranks top three for FQ, and the gap between FLAT and other baselines in FQ, is significant, as shown in Table 1 in this rebuttal. Note that we follow the setting in TOFU’s original paper to only report the final results.

Table 1 The final results on TOFU-5% and TOFU-10%. For both metrics, FQ and Model Utility (MU), higher values indicate better performance.

The forget quality on the TOFU-1% is similar to that of the baseline methods may be due to the small size of the forget set (40 samples). When calculating the distributions of truth ratio for such a small sample size, the differences between methods tend to diminish.

Note that the retain version of FLAT can achieve the best forget quality on TOFU-5% and TOFU-10% while maintaining high model utility. For the TOFU dataset, which is a synthetic set with separable profiles of 200 authors, using retain data does not significantly blur the boundaries between the forget and retain data. Hence, using the retain data in this task significantly improves performance. However, the primary focus of our work remains on content unlearning (usually only the forget content is known), which reflects more practical and realistic situations encountered in real-world applications.

The difference in forget quality values between ours and NPO reported results arises due to the differences in evaluation settings. We tried our best to evaluate the performance of all methods under controlled settings as indicated in TOFU’s original paper to ensure a fair comparison. Our implementation is based on the TOFU codebase. The difference between the TOFU official implementation and the NPO implementation is that the NPO evaluates models after every epoch (a total of 10) and reports the epoch with the best forget quality, while the TOFU benchmark uses the final results after five epochs. The difference in reporting policies significantly influences how forget quality is presented and perceived.

Table 2 in this rebuttal provides the best results of each epoch across different methods for your reference. Using the implementation of NPO, we set lr=1e-5, epoch=10, evaluate at each epoch, and report the best performance. Since the Name-change-based method in [1] based method evaluates models after every epoch, we report the results of this baseline in the best results table. Under this setting, even a simple algorithm like GA can achieve relatively good FQ (the reported FQ for GA on TOFU-5% is under 0.1 in NPO paper, while here it is 0.2404). This indicates that reporting the best results across all epochs can overstate the model’s performance, as it may not fully represent the method's actual unlearning capability.

(Table 2 is in the next response-2)

Dear Reviewer awBu,

Thank you for taking the time to review our paper. We sincerely appreciate your thoughtful feedback.

We wanted to kindly follow up as the deadline for uploading revised PDFs is approaching soon. We sincerely hope to have a further discussion to see if our response addresses your questions/concerns.

For your convenience, we have uploaded a revised version of the manuscript along with a summary of changes in our comment.

Please feel free to share any additional comments or questions, and we will address them promptly and to the best of our ability.

Thank you once again for your time and consideration! Wish you all the best!

Authors

Thanks so much for your positive and valuable review. Your encouraging comments and thoughtful feedback are highly motivating and will help us further refine our paper.

Q1:Should line 235 be log prob in the equation?

The two probabilities mentioned in line 235 represent the empirical estimation of the data distribution, as indicated in Eq. 3, which aims to maximize the f-divergence between the good- and bad-sample distributions. In empirical cases, we do not employ the cross-entropy loss. Instead, FLAT formalizes the loss within the f-divergence framework to estimate the data distributions. The probabilities serve as the proxies for the good and bad sample distributions, aligning with the theoretical foundations [1].

Q2:Line 371. PO should have the lowest PPL as shown in Table 3

Thank you for catching that. Yes, it should indeed refer to the lowest PPL. We appreciate your attention to detail.

Q3: The evaluation setting

Thank you for raising these important questions. On the TOFU dataset, we followed the instructions of TOFU and fine-tuned the base LLM using all data. We added the sentence in the revision.

Explanation for unlearning goal. In our method, the input to the unlearned model consists only of the forget data and templated reject-based answers. The ideal unlearning solution is to retrain the model from scratch using only the retain data after removing specific training data points [2]. However, retraining LLMs is expensive due to high computational costs and the need to access the entire training dataset. To address these challenges, existing literature focuses on developing approximate unlearning methods to make the unlearned model resemble the retained LLM as much as possible.

Given that the original model contains all prior knowledge, the unlearning process should enable the model to retain its performance on the non-forget data while erasing any information related to the forget set. Ideally, the unlearned model's behavior should match that of a model trained solely on the retain data, ensuring that no information from the forget set is retained.

Q4: Questions related to forget quality metrics

For the Harry Potter dataset, we evaluate Forget Quality (FQ) using ROUGE-L and BLEU scores, following previous work [3]. Since lower scores don’t always indicate better performance, we calculate the Forget Quality Gap (FQ Gap), which is the sum of the BLEU Gap and ROUGE-L Gap, as done in [4]. These gaps are the absolute difference between the retained model and the unlearned model. The smaller FQ Gap towards the retained model should represent better unlearning performance.

For the TOFU dataset, we adopt the metric proposed in the original paper, reporting ROUGE-L scores on both the forget set and retain set. Similarly, a lower ROUGE-L score on the forget set does not necessarily signify better unlearning. Therefore, we highlight methods where the ROUGE-L score closely matches that of the retained model, as these are considered to produce better results.

Q5: Is line 512-515 repeated content?:

Thank you for pointing that out. We apologize for the oversight. The updated version will address these minor issues and ensure correctness.

Please let us know if you have any more questions! Thank you!

[1] When optimizing $f$-divergence is robust with label noise.

[2] Rethinking machine unlearning for large language models.

[3] SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning

[4] Large Language Model Unlearning via Embedding-Corrupted Prompts

Thank you for taking the time to carefully consider our rebuttal and for your thoughtful feedback. We are grateful for your positive impression and support for our work. Your encouraging comments and wish to see our paper at ICLR 2025 mean a lot to us, and we deeply appreciate your recognition of our efforts.

Thank you so much! Wish you all the best!

We sincerely appreciate the reviewer’s time and effort in reading our paper and offering thoughtful suggestions and constructive feedback.

W1: Unclear Motivation for f-divergence Reweighting.

f-divergence to reweight two loss terms: Eq. 2 consists of two terms. The first term encourages the model to learn the exemplar good answer through gradient descent, while the second term drives the model away from the bad answer (forget data) via gradient ascent. However, determining the appropriate weighting for these terms is non-trivial, as it requires balancing the importance of learning good answers and forgetting bad answers. This is where f-divergence comes into play. The f-divergence provides a rich family of divergence functions that help maximize the model's behavior on two different distributions (good vs bad). Its variational form yields an easy-to-optimize objective function that has an interpretable structure: first term encourages the model to align with the good data distribution by maximizing the probability of exemplar answers, while second term penalizes overlap with the bad data distribution by minimizing the probability of forget data answers.

Maximize the f-divergence: Eq. 3 represents the variational form of the f-divergence between the good and bad data distributions. The empirical estimation of f-divergence, as described in Eq. 4, is constructed using the generated data distributions and can serve as the loss function when finetuning the LLM. This estimation provides a natural reweighting mechanism that closely approximates the true f-divergence, effectively modeling various data distributions following from [1,2]. Theorem 3.4 supports this by showing that the empirical alternative achieves the optimal non-parametric rate of convergence toward the true f-divergence. This ensures that the estimated reweighting is robust and well-aligned with the underlying data distributions.

Benefits for LLM unlearning: We use f-divergence to provide a principled approach for reweighting the two loss terms. Specifically, maximizing the f-divergence between the data distributions of good (template reject-based) and bad (forget) answers ensures that the model effectively separates these two distributions. This separation directly aligns with the goals of unlearning: to retain useful knowledge while forgetting undesired information. It ensures that the model learns the correct template pattern for the questions in the forget set (e.g., reject-based answers) while effectively forgetting the original answers from the forget set without compromising the performance of retained knowledge. FLAT adjusts the LLM to increase the probability of generating preferred reject-based answers.

W3: Scalability Concerns:

We have added experiments on TOFU-5% and TOFU-10% using Llama2-7B. The results demonstrate that FLAT consistently ranks in the top two for FQ. For the complete results, please refer to Table 12 in the revision.

Table 2 in this rebuttal presents part of the results for TOFU-5% and TOFU-10% using Llama2-7B. Note that we only report the results on the final models that are consistent with the TOFU implementation. This differs from the NPO implementation, which evaluates models after every epoch (a total of 10) and reports the epoch with the best FQ. We also report the version using the retain data as the good answer.

Table 2 For both metrics, FQ and MU, higher values indicate better performance.

Thank you for the detailed response. However, I still have two concerns:

1. Proposed Weighting: I’m still not convinced of its utility. As mentioned in the rebuttal, the two-term loss aims to increase the probability of an alternative answer but decrease the original answer probability, which aligns closely with the DPO loss that is widely used in LLM unlearning. However, the performance comparison does not show a clear advantage for the proposed reweighting on the TOFU dataset. And the performance reported has some discrepancies with previous works, which raises a big concern for me.

2. Baseline Discrepancy: The reported performance for baseline, such as the NPO, differs from prior works. For example, both [1] and [2] report ~1e-1 FQ for TOFU-10% with NPO and NPO-RT, consistent with the NPO paper, but the rebuttal results has some discrepancy.

[1] Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference [2] Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning

Point 4: [2] indicates parameter tuning is crucial for NPO_RT, further emphasizing the significance of FLAT, which achieves strong performance without the need for parameter tuning.

[2] only reports the retain version of NPO (NPO_RT). They conducted a grid search for $\beta$ in the range of [0.05, 0.2] and for $\lambda$ in the range of [0.5, 1.5], where $\beta$ is the parameter for NPO and $\lambda$ is the weight of the retain loss, to obtain the best-performing model. But they doesn’t provide the optimal parameters for NPO_RT. For our comparison, we set $\beta = 0.1$ and$\lambda = 1$ based on the description in NPO paper and [1].

The results of NPO_RT in [2] shows that parameter tuning is crucial for achieving the best performance when reporting final results after unlearning. When applying NPO_RT to other LLMs or datasets, one must tune hyperparameters like $\beta$ and $\lambda$, which can be time-consuming and resource-intensive. This need for tuning also motivates FLAT: existing solutions struggle to maintain a good balance between forget loss and retain loss without extensive parameter adjustments.

Our method is designed to eliminate the need for such parameter tuning. Unlike NPO_RT, which requires grid search to optimize weights, we use the default/optimal values from the original paper in our experiments to enable fair comparisons without tuning any weights. This demonstrates that our approach achieves competitive performance without manual adjustments. FLAT leverages $f$-divergence to provide a theoretically optimal solution that balances unlearning efficiency and model utility. The basic formulation of loss adjustment methods, including NPO_RT and PO, consists of two loss terms similar to those in Eq. 2. While NPO_RT relies on tuning the weights of these terms, FLAT’s $f$-divergence-based approach automatically balances them, offering a more efficient and robust alternative.

Point 5: Empirical demonstration of the correctness of our implemented baselines.

We also ran the code provided by [2] to test NPO and obtained similar results on TOFU-1%, using the exact command provided by [2]. Specifically, we retrieved the final results from {save_dir}/checkpoint/aggregate_stat.txt, where the FQ for NPO was 0.0068, exactly matching our reported results. This demonstrates the consistency of our findings. Furthermore, one can easily reproduce the results of NPO by running the provided code [2] on TOFU-1% with the following settings: epoch=5, lr=1e-5, $\beta=0.1$, and $\lambda =1$. These results confirm that there are no issues with our implementation, parameter selection or the entire unlearning and evaluation process.

Thank you again for your detailed comments and the questions raised. We hope our responses can address your concerns. If you have any further questions, please let us know.

[1] Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference

[2] Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning

[3] Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective

We sincerely appreciate your insightful evaluation of our study. Thanks so much for your positive and valuable review.

Q1: Ablation Studies on Various Template Response Strategies.

We conducted ablation studies on different good answer types, including template reject-based responses “I don’t know” (IDK) and random normal answers (from TruthfulQA) in Table 7 for the TOFU dataset and Table 15 for the Harry Potter dataset. Results indicate that using normal responses improves model utility on HP datasets and ROUGE-L Score on retain sets on TOFU datasets, whereas using IDK responses yields better forgetting quality.

Added ablation study on the generated template answers (Table 16). We designed a prompt instructing GPT-4o not to reveal any information about the two authors included in the forget set from TOFU-1%. However, this approach demonstrates the worst performance among the three types. One possible explanation is that GPT-4o tends to repeat several words from the question in its answer, which increases its similarity to the ground truth answer and undermines the effectiveness of unlearning.

A potential follow-up direction is to explore how different data curation strategies influence the unlearning process, as better-curated data may help achieve more effective and reliable unlearning outcomes. We added the detailed generation process including system prompts in Appendix D.3 in the revision.

Table 16. Ablation Study of good answer type on TOFU-1% dataset using Llama2-7B. FLAT(KL) - Generation is the generated template from GPT-4o.

Q2: Analysis of the performance of Mismatch

Mismatch utilizes the retain loss term and the mismatch term, which has more information than FLAT (See Appendix C.1). The first term is the finetune loss on the retain data, which will help the model to reserve knowledge about retained information and contribute to good forget quality. The second term is finetune loss, which uses the normal answer to substitute the original answer in the forget set.

Analysis about the edge case of Mismatch. In OPT-2.7B, the forget quality gap (FQ Gap) between Mismatch and FLAT(TV) is relatively similar. For small LLMs like OPT-2.7B, fine-tuning with the retain data for several epochs can lead to effective forgetting of the forget set. The rationale is that finetuning on the forget set may induce catastrophic forgetting over the forget set like continual learning [1]. Additionally, OPT-2.7B generally produces lower-quality outputs, which reduces the BLEU gaps between FLAT and Mismatch. As a result, the differences in unlearning performance (FQ Gap) between these two methods appear comparable in this setting.

Mismatch can achieve comparable results using OPT-2.7B on the HP dataset. However, on Llama2-7B, the FQ Gap for the mismatch is 0.4647, and ours is 0.2098 (Table 9). Note that a smaller FQ Gap indicates better unlearning performance. FLAT can show better adoption to different LLMs and different datasets. This might be because the Mismatch fails to keep a good balance between the model utility and the forget quality, while Flat theoretically formulated a reweighting mechanism.

Added the mismatch in all three datasets. We added the results of Mismatch in the TOFU dataset (Table 4) and MUSE (Table 5). Table 6 and Table 7 are the ablation study, and Mismatch has different formulations compared with our method, which is not suitable for understanding the reweighting mechanism and the good answer type.

Part of Table 4. For both metrics, FQ and MU, higher values indicate better performance.

Part of Table 5 The results on MUSE-News benchmark.

Thank you once again for acknowledging our contributions. If you have any questions, please feel free to ask!

[1] Continual lifelong learning with neural networks: A review.

Dear Reviewer n2JT,

Thank you so much for taking the time to review our paper. We sincerely appreciate your constructive feedback and your positive evaluation of our work.

We wanted to kindly follow up to see if there are any remaining concerns or questions that we can address. We would be more than happy to respond and do our best to resolve any issues. For your convenience, we have uploaded a revised version of the manuscript along with a summary of changes in our comment.

Once again, thank you for your valuable support and for recognizing the contributions of our work. Wishing you all the best in your professional and personal endeavors!

Authors