Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning

Many thanks for your constructive comments and suggestions! Please see our responses below.

Q1 The experiments are done without repetitions, and it is unclear whether SatImp has a marginal improvement.

Many thanks for your suggestions. Here we provide the standard deviations for part of Table 2 (due to the 5000 chars limitation), more results will be updated if our paper could be accepted. This is the results of LLaMA-2-7B.

Considering the marginal improvement problem, other reviewers are also concerned. Please note that we have mentioned the selection of hyper-parameter setting are fixed after the analysis on TOFU (In the Appendix Line 1152). The best preformance setting for ES score is typically not the best for other metrics. Thus, the performance gaps may be marginal on FQ&MU of TOFU, Acc. on WMDP, and metrics on MUSE with such a setting. After adjust this setting, new results on WMDP and TOFU are shown in Reviewer LUgf Q2 and Reviewer 8Bwg Q2.

Q2 SatImp is a simple combination of existing method thus this paper does not seem to offer significant novelty.

We sincerely appreciate your comments, but we respectfully disagree with your assessment regarding the lack of novelty of our work.

As far as we know, we are the first that proposing the importance-based (IB) reweighting mechanism (i). There is no IB annotations in all benchmarks before this paper. We are the first to make such annotations and make it public (in the supplementary material, will be released once this paper being accepted).

This paper is also the first that summarize existing methods with a universal saturation-based (SB) framework (ii). While mounts of literature have explore the reweighting in LLM unlearning, no one has made such a summary to figure out why they are effective from the reweighting perspective.

Besides, we simplify IB and SB with a simple mapping -- SimImp (iii) and SimSat (iv). Finally, we propose SatImp, which further enhances the performance via above analysis. Among these five findings, only (iv) is encountered a contemporary work (please refer to our response in Reviewer LUgf Q1).

Thus, we sincerely hope you can re-evaluate the contribution and novelty of our paper.

Q3 Confusion about the importance-based setting

We apologize for any confusion that may arise. Your understanding are correct that a token is assigned a large weight if and only if it contains important information. In fact, we have done this in Page 3 Line 160.

As mentioned above, we are the first to make such annotations. However, considering that there are numerous data points without importance labels. It would be labor-intensive for human to annotate all of samples in existing datasets. Thus, we try to find an approximate representation of importance labels. We notice that it is high-correlated between weights and likelihoods (Figure 2(e), (g)). Thus, we simplify the importance-based and saturation-based methods as put more weights on tokens with lower and higher likelihood, respectively.

Many thanks for your constructive comments and suggestions! Please see our responses below.

Q1 More experiment about Claims 1-4

Thanks for the detailed review.

For Claim 1, we contain the relevant results in the Appendix Figure 13,14 (m)-(x). We will include the performance for SimImp in all tables.

For Claim 2 and 3, more results are shown in the Appendix Figure 13-16.

For Claim 4, the motivation that we include the discussion about hard-sampling and soft-weighting is we do not want to overlook any element in our quest to achieve better LLM unlearning performance. As mentioned in Line 324, there have been some concerns that hard sampling may outperforms soft weighting. Thus, it is necessary to figure it out. Only through such a thorough investigation can we derive reliable empirical conclusions. Our analysis can help subsequent work avoid detours.

Q2 Claim 5: Baseline is not match the numbers in other papers and the results on the WMDP should be discussed.

Regrading the WMDP results first, the response is shown in Reviewer LUgf Q2.

Regarding the inconsistent results about FQ and MU, we indicate that the performance on ES score and FQ&MU are not always aligning, which means when models obtain best performance on ES score, they may have mean performances on FQ&MU. It is because the objectives of ES and FQ&MU are different. Especially the FQ and ES Un., FQ requires the unlearning model towards a standard model yet ES Un. requires the model forget the unlearn data as much as possible.

There are some literature that have mentioned this phenomenon. First, we find that WGA also mentions this discrimination. Second, in the paper 'LLM Unlearning via Loss Adjustment with Only Forget Data' that you mentioned as a essential reference, the FQ and MU are similiar as results in Table 2.

In this paper, we have made comprehensive analysis on TOFU with ES score. Therefore, in Table 2, we report the results with ES score as the primary metric. All methods were selected with the goal of optimizing the ES score‌. Due to limited rebuttal length, here we report part of results which set the FQ and MU as the primary metrics. Results with retain regularization on LLaMA-2-7B are shown as follows:

Q3 Confusion about Eq. 8,9,11,12

We sincerely apologize to the confusion about the exploration process of our paper. First, Eq. 8 and 9 are our intuition about importance-based and saturation-based unlearning. Eq. 8. indicates that a token should be noticed if and only if it contains important information. Eq. 9 represents an objective, which summarizes all existing reweighting methods. For instance, Eq. 9 is equivalent to SimNPO when $\tau = \frac{|y|(P+1)}{2} - P$‌.

However, considering that there are numerous training samples that without importance-based labels, it is difficult for researchers to labeling them all. Thus, we try to find an approximate representation for both unlearning branches. We notice it is high-correlated between weights and likelihoods (Figure 2(e)-(h)). Thus, we simplify two branches as put more weights on tokens with lower or higher likelihood, respectively. An intuition for this simplification is probability-based mapping, thus we propose SimImp and SimSat (Eq.11). We make a comprehensive investigation and then combine these two solutions (Eq. 12).

Q4 References

Thanks for providing several LLM unlearning methods, we will cite them and compare the performance. According to the results in these papers, none outperforms SatImp.

Q5 Typo and organization of the paper

Many thanks for your wonderful suggestions!

1. We will modify the $p$ in Eq. 8. $\beta$ has been used as the smoothness index in Eq. 9, 11, 12.

2. The baseline in Figure 3a-d is the vanilla GD, which is represented by the dashed line.

3. We will discuss and re-organize the paper about the position of Algorithm 1 and the explanation about ES scores.

4. Actually, we have made a lot figures that recording the retain-unlearn performance at different training steps. However, considering the length of our paper, these figures are not shown in the submission. We will include part of them if our paper is accepted.

Q6 Intuition about re-weighting a lower than 1 weight for the token losses When we observe the training process of the vanilla GD, we notice that the unlearning process of the model is completed rapidly within just few steps after warm-up, even reaching an over-unlearning state. Such a swift change is detrimental to finding the optimal model‌. Smaller weights can prolong the unlearning, helping the model stop training at a more appropriate step.

Many thanks for your constructive comments and suggestions! Please see our responses below.

‌Q1: Is SatImp an incremental version of WGA?‌

We sincerely appreciate your comments, but we respectfully disagree with your assessment that SatImp is an incremental version. Before reading the response we prepare for you, please first refer to the response in Reviewer 3V34 (Q2 and Q3).

We apologize for any confusion that may arise. To begin with, we would like to highlight the difference between the WGA paper and ours.

Specifically, the motivation of the WGA paper is presenting a toolkit of the gradient effect that qualifies the impacts of unlearning objectives on model performance. The literature focus on analyzing the details of diverse unlearning objective from a gradient perspective. WGA is an additional discovery in that paper, which could be an auxiliary strategy to enhance unlearning performance.

Differently, the motivation of our submission is identifying the best reweighting strategy. Our submission contributes annotations of important tokens, summary of saturation-based methods, correlation between weights and losses in different paradigms, and combinations between different paradigms. Overall, the motivation and contribution are clearly distinct between these two papers.

Q2 Unsatisfied results and marginal improvements

Thanks for your comments about the experiments. However, we present more results and analysis that may help you understand this ‘marginal’ comprehensively. First, regarding the limited improvement, as demonstrated in Reviewer 8Bwg Q2, we have mentioned in the Appendix (Line 1152) that our hyper-parameter settings followed the exploration method based on the ES score in previous sections. We selected the hyper-param that achieved the best results on the ES score metric and fixed these params across all benchmarks. However, this setting is likely not optimal for other metrics (FQ, MU) or benchmarks (WMDP). As shown in Reviewer 8Bwg Q2, if FQ and MU are prioritized in the experiments, the ES results would decline, but the performance gap between different methods would widen‌.

Regarding the WMDP results, the 25% performance on MMLU is caused by 125-step training process, which is too long for forget-only regularization in Table2. Except for NPO, the other methods are all leading to an over-unlearning situation. Thus, we utilize the early-stopping at the 30-th, 60-th, 90-th steps. The MMLU results are shown as follows:

Furthermore, we report the results at 30-th step.

Additionally, unlearning with retain regularizations on WMDP (Table 6) are all under-unlearned, which are caused by too high $\lambda$ (the ratio between forget and retain objectives in Eq. (2)). We adjust $\lambda$ to accomplish more effective unlearning. Results are shown as follows:

Regarding the MUSE results, we sincerely apologize that there is a typo in SatImp with retain regularization on MUSE-Books dataset. We wrongly recorded a forget-only regularization result here. The correct results should be:

Regarding the MUSE results with forget-only regularizations, we have mentioned that we choose the checkpoint based on the Privacy Leak metric, which measures the degree of unlearning (whether the model is over-unlearning or under-unlearning). Extensive results indicate that model would have 0 utility when PrivLeak near to 0 under the forget-only regularization setting. NPO is different because it is significantly under-unlearned, which obtains a very low negative value on the PrivLeak metric. Thus, it is normal that almost all methods have 0 utility under the forget-only regularization setting.