Do Unlearning Methods Remove Information from Language Model Weights?

I have read through the general response and paper updates. I appreciate the authors' efforts to address concerns raised by other reviewers. However, since the specific changes I suggested to increase my score have not been included, I will keep my score as borderline accept.

Also mentioned above, have you tried ablating the size of T?

Upon your suggestion, we tried ablating the size of T. We reran the experiments with GD on the YEARS dataset with different sizes of T. We run unlearning on two splits of the datasets we designed (originally, we run on 5), then run RTT on one of the splits limiting the number of samples. This leads to the accuracy after unlearning being higher than the case where we use larger T. You can find the recovery rates vs number of samples used for each limit on retain accuracy in the tables below

Reduction of retain accuracy of at most 5%:

Accuracy after unlearning for two splits (used for all rows except last one): 0.54

Accuracy after unlearning for five splits (original, used for last row): 0.34

Reduction of retain accuracy of at most 10%:

Accuracy after unlearning for two splits (used for all rows except last one): 0.436

Accuracy after unlearning for five splits (original, used for last row): 0.32

Reduction of retain accuracy of at most 30%:

Accuracy after unlearning for two splits (used for all rows except last one): 0.38

Accuracy after unlearning for five splits (original, used for last row): 0.32

Reduction of retain accuracy of at most 100%:

Accuracy after unlearning for two splits (used for all rows except last one): 0.23

Accuracy after unlearning for five splits (original, used for last row): 0.23

I thank the authors for taking the time to rerun some of these experiments, and I am happy the suggestions brought up interesting results.

I have one follow-up question regarding the ablation on the size of T. If I understand correctly Section 4.3, initially the size for T was 628 (4 out of 5 splits of the data). I am not sure I understand this sentence in your response

We run unlearning on two splits of the datasets we designed (originally, we run on 5), then run RTT on one of the splits limiting the number of samples.

In any case, could authors provide the size of T with respect to the original unlearning set? (e.g. we retrain on 75% of the data used for unlearning). As I raised in my original review, the size of T should ideally be very small compared to the overall dataset, otherwise all recovery could be due to relearning the information that went into unlearning. I guess T=10 is already a small percentage, but would be nice to clarify.

I am still concerned about the different behaviours observed for the different datasets and the fact that the ablations on the size of T only happened in the YEARS dataset, which seems to be the most "toy setup" among all of them. I think a strong accept would require the authors to have results on all the setups. I am not sure if I will be able to reply anymore since the rebuttal deadline is close, but I am happy to read the author response and update my score later in the private discussion.

Finally, I would like the authors to think about their claim of RTT "being a more systematic approach". Given the current results, it still looks like results strongly depend on the dataset, hyperparameters and size of T. Why is this fine-tuning approach more systematic than e.g. probing internal activations as introduced in https://arxiv.org/pdf/2402.16835? If the authors still think this is the case, they should probably expand on the reasons why in the paper.

Overall, I am keeping my current score as I see the additional results as partial evidence and cannot make an informative-enough section in the paper across datasets and experimental setups. I am willing to revisit this score if authors provide further evidence in the remaining time.

Thank you for your thoughtful review. We’re glad you think we make a good contribution.

What are the real-world use cases of the RTT approach? I.e., what are specific examples of V and T as defined in the paper? When does it make sense for the adversary to have (legitimate) access to T that was part of the dataset that was retrained?

The real-world use case of RTT is to assess how promising certain unlearning techniques are at removing knowledge, which is useful both for developing techniques to prevent certain kinds of misuse, and making strong safety cases about them. In particular, RTT is not a threat model but an unlearning evaluation technique. If RTT fails at recovering knowledge, this would be good evidence that the knowledge cannot be extracted from the model. We provide more details about this in our common response and in our improved related work section.

Thank you for your review.

Based on the 2nd limitation discussed, can authors provide any commentary on possible downsides of their relearning evaluation for unlearning methods that appear more robust than current methods?

Our method relies on the following idea: if an unlearning technique is hiding knowledge, then unlearning T and V will lead to T and V being hidden in a similar mechanism in the model. If hiding T happens in a different way from hiding V, it would be hard to use T to get the model to stop hiding V. This means our approach still works even if retraining was slowed down by the unlearning method (unlike classic relearning-time methods). However, if excellent anti-retraining techniques were developed, our approach could fail to notice hidden information.

Limited technical novelty/contributions. As prior work has already proposed relearning as a metric for evaluating unlearning [1, 2, 3, 4, 5], the paper's contribution rests solely on creating training and validation splits with low mutual information for the relearning evaluation. For example, prior work already shows that RMU is not robust to relearning [1].

We would like to clarify that our contribution is not solely creating the datasets. Our proposed approach is different from relearning approaches, which are known to suffer from a lack of baseline comparison and to a brittleness to techniques aimed at reducing retraining speed or sample efficiency, which has limited their adoption in the past (e.g., retraining is only included in the Appendix of [1], despite an implicit focus on actually removing information from model weights). See the common response where we detail the difference and what weaknesses relearning has that RTT overcomes, and what are our contributions beyond creating these datasets.

The authors claim that "Our evaluation does not guarantee that information is removed from the weights; rather, it sets a higher bar than previous evaluation methods for unlearning," but comparisons are not made between the proposed evaluation and similar "relearning time" metrics proposed in [2, 3]. Furthermore, prior work (which the authors do cite) suggests that relearning time is already sufficient for measuring unlearning [1, 3].

We added an explanation of the difference between relearning time and RTT in the common response and the paper.

The authors acknowledge that their proposed connection between mutual information and relearning is not substantiated; specifically, they acknowledge that robustness to relearning does not necessarily imply zero mutual information between a forget set and model weights. Unfortunately, while acknowledgement is appreciated, this limits the usefulness/novelty of the evaluation, especially in the context of Section 3.2.

While we do think that robustness to relearning does not necessarily imply zero mutual information between a forget set and model weights, we think that RTT is an improvement over available methods that aim to test whether unlearning removed information from model weights or not.

[1] Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024b.

Thanks for your response. My concerns regarding the contribution/novelty and technical details remain unaddressed. The common response states that the main difference between RTT and prior relearning evaluations is in creating training/validation splits with low mutual information. Also, the relearning evaluations in Lynch et al., Li et al., and Tamirisa et al. also measure accuracy after fine-tuning after a fixed number of steps (Lynch et al. uses "Familiarity"), and not success rate nor relearning time, respectively, which the authors have incorrectly characterized in Table 1.

Furthermore, the strength of the contribution rests on whether RTT is a material improvement over relearning time. The authors hypothesize this to be the case, but do not provide the necessary experimental validation (e.g., there do not appear to be experiments that directly measure the effectiveness of relearning time vs RTT as unlearning metrics). For example, in Section 7 the manuscript states "Our evaluation does not guarantee that information is removed from the weights; rather, it sets a higher bar than previous evaluation methods for unlearning," which is not empirically substantiated. In general, there is too much speculation about the effectiveness of the proposed metric in the paper and the connection to information removal.

My main concern is that to get closer to the acceptance bar for ICLR without substantial changes to the current presentation of the metric, the authors would need to significantly increase the breadth of the evaluation to compare more methods/metrics/datasets. In lieu of an expanded evaluation, there would need to be a proper theoretical justification for the connection between unlearning/the proposed metric and mutual information in the weights for the authors' speculations to be properly justified, and for the proposed metric to be useful long-term. However, each of these would be too much to expect for a rebuttal period, so I maintain my score and justification at this time.

Thank you for your thoughtful review and for your questions.

Could the authors explain the meaning of the uncertainty mentioned on line 257?

We use the standard error to estimate the uncertainty. To estimate the uncertainty of our accuracy given our dataset size, for $p$ as the proportion of questions the model answers correctly, we have:

where 157 is the number of data points for V, the evaluation split of the dataset, and the factor of 2 accounts for the fact that we run the evaluation twice, each run using a different split of the dataset (the two splits we evaluate on are disjoint).

Why is "MCQ with Loss on Answer Only" tested solely on GD in Figure 4?

In this graph, we take gradient steps based only on the cross-entropy loss of the tokens that are part of the answer to the MCQ questions. However, RMU acts on activations of intermediate layers. Specifically, RMU takes the loss between a random vector of activations and the true activations for the model on a specific, intermediate layer for prompts on the forget set (prompts that RMU is trying to make the model forget the answers for). This means that it’s unclear how we would restrict the loss to be on the answer tokens only, as RMU acts on activations of intermediate layers where we cannot pinpoint which activations correspond to which tokens. We added a footnote to clarify this.

Please clarify the definition of the terms such as forget/retain accuracy for accuracy on forget/retain set.

Thank you for this question. The forget set is the dataset of questions that correspond to facts that we would like the model to unlearn. The retain set is the dataset of questions that correspond to facts that we don’t want the model to unlearn; we want the model to still know these facts after unlearning. We added these definitions to section 4.1.

Are the labels "Unlearn" and "Unlearn+RTT" in Figure 3 misplaced? Additionally, please show the alpha values in Figure 3.

They indeed are. Thank you for pointing this out. We fixed this misplacement and added the alpha values.

The paper could evaluate more unlearning methods such as other baselines listed in [1].

We agree that evaluating more unlearning techniques would be interesting, but due to time constraints, we leave this for future work. Given that, we chose to evaluate the unlearning methods that (1) have gained the most traction in being successful in performing unlearning, (2) seem somewhat promising in that they actually remove information, and (3) were already released when we ran our core experiments.

Gradient Difference seems promising as it does the opposite of the learning process and is standard in all unlearning works. RMU has gained a lot of attention recently for getting promising reductions in accuracies on benchmarks and is viewed as state of the art in unlearning. Random Incorrect Answer seems like an intuitively promising way to make the model unlearn information; similarly to how the model learned the correct fact, it makes sense that we can teach it an incorrect version of the fact.

As shown in Appendix C, we also test against a promising unlearning technique called RepNoise, which successfully beats the relearning evaluation for many hyperparameter configurations. Other candidate unlearning techniques we looked at include the one introduced in [2], but we did not get a response from the authors when asking for source code or unlearned models, and we were not able to replicate our results despite our best efforts.

[1] evaluates four different methods: (a) Gradient Ascent, (b) Gradient Difference, (c) KL-minimization, (d) Preference Optimization. (a) and (b) are very similar, with the only difference being that (b) has a retain loss to make sure the model doesn’t become useless. We experimented with a retain coefficient of 0 and found it strictly worse than (b) (see Figure 3, which shows performance with a retain coefficient of 0). We added this explanation in the description of Gradient Difference. (c) is a similar approach to (b), with the only difference being how the retain loss is calculated.

Finally, (d) is very similar to approaches that are known to hide information rather than remove it. LLMs like GPT-4 are trained with RLHF, and are vulnerable to jailbreaks - which rules out RLHF removing information.

[2] Tamirisa, Rishub, et al. "Toward Robust Unlearning for LLMs." ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024, https://openreview.net/forum?id=4rPzaUF6Ej.

Table 1

In Table 1, we show how our approach compares to previous approaches and those from concurrent work. We expect the results from other fine-tuning-based approaches that test for presence of hidden knowledge will be highly correlated with ours, but we expect our approach to have an advantage. Specifically, given the design of our dataset where the facts are independent, we can perform a larger-scale retraining/fine-tuning. This is more challenging for other approaches, as they use facts from the same dataset, which means they either fine-tune on a small number of facts such that they can be confident no leakage is happening, or they use more facts, which risks leakage happening.

Our Contribution

Our contribution is the following:

1. Describing the idea to use the power of fine-tuning to recover hidden knowledge while overcoming the concerns of relearning-time-like approaches.

2. Creating datasets that have independent information (as explained in the first part of the common response). This independence allows us to overcome the concerns of relearning-time-like approaches.

3. Comparing different approaches to apply RTT to elicit potentially hidden knowledge. This includes:

   • Creating different formats for the datasets: To fairly evaluate the unlearning methods, we ensure that they lead to lower accuracy on a different question format from the format used for unlearning. This ensures that when we get a lower accuracy after unlearning, it reflects a deeper type of unlearning rather than format-specific hiding. [2] found that some unlearning methods may hide information in one format but fail when the format is changed. We address this by using different formats for evaluation and unlearning.
   • Finding dataset formats to elicit hidden knowledge, including:
     • Plain-text: Tokens of text related to the fact, not formatted as questions.
     • Multiple-Choice Questions (MCQs):
       • We explored how taking the loss on different tokens impact the loss calculation and the ability to hide facts. These options include:
         1. The letter representing the answer (e.g., "A").
         2. The letter representing the answer + the answer itself.
         3. The question + the letter representing the answer + the answer itself.
       • We found that option (3) was the most powerful for recovering hidden information.

4. Stress-testing RTT to ensure it works on hard cases where hiding occurs at high granularity.

We will make these contributions clearer in an improved contribution paragraph at the end of the introduction.

References

[1] Sheshadri, Abhay, et al. "Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs." arXiv, 2024. https://arxiv.org/abs/2407.15549

[2] Lynch, Aengus, et al. "Eight Methods to Evaluate Robust Unlearning in LLMs." arXiv, 2024. https://arxiv.org/abs/2402.16835