From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization

We are very thankful to the reviewer for their valuable comments. We are happy that the reviewer found our analysis fascinating, and we are looking forward to discussing more with the reviewer in light of our rebuttal and new experiments. We address the concerns and answer the questions raised by the reviewer below.

Evaluate on more datasets (like ImageNet) or justify why that wouldn’t change the conclusion (may require another round of submission)

Indeed, we focused on only simple vision datasets with smaller-scale architectures (mainly motivated by prior work which focused on CIFAR-10 and CIFAR-100 as the canonical datasets for unlearning in vision classifiers). Using these smaller-scale datasets allowed us to answer some fundamental questions about relearning attacks: why do they occur and how can we address them. This is a different direction that complements works on relearning attacks that were carried out in LLMs (where the possible exploration is limited due to the sheer cost associated with such experiments, and where the complex setup makes it difficult to draw clear conclusions).

However, we fully agree with the reviewer that showing results in only those specific datasets was limiting. Based on this suggestion, we present additional results here on the well-known Imagenette dataset [1] (a subset of ImageNet) to showcase the generality of our findings on more complex and higher-resolution images. Similar to other results in the main paper, we focused on using atypical examples for the forget set. The results for three different unlearning methods are presented below (note that tuning hyperparameters for all baselines is difficult in such a tight timeline, hence, we refrain from reporting sub-optimal numbers for other baselines – we will add all results in the next revision of the paper).

We see that the forget set accuracy of SCRUB shoots up upon relearning, even with 0 relearning examples, in line with our findings on CIFAR. On the other hand, variants of our tamper-resistant framework, weight distortion, and weight dist reg, similar to our previous results, are more resistant against these relearning attacks. These evaluations highlight that our results are general and hold across different datasets.

[1] https://github.com/fastai/imagenette

Are the training iterations always fixed?

Thanks for checking this. Yes, we fixed the number of training iterations in order to make sure that discount any effects of (un)learning efficiency for different methods. Hence, we assigned a large pretraining, unlearning, and relearning budget. We show that some methods are more computationally effective, and can get away with a significantly smaller training budget in response to reviewer JQcR.

How can we extend this to modern models and tasks?

Susceptibility to relearning has already been analyzed for language models as we list extensively in the paper. The fact that relearning is observed across these different models is a clear sign that it is a more general phenomenon, beyond a single model/task.

The focus of our paper is to design a controlled study to gain a deeper understanding of causes and remedies for vulnerability to relearning. Based on this, we have identified weight space distances as an important factor affecting the robustness of different unlearning algorithms and proposed different unlearning objectives to directly address this.

We consider it exciting future work to experiment with our unlearning objectives in LLMs too for increasing their robustness. But there is no doubt, based on the existing literature, that this phenomenon does also occur in LLMs. Hence, we believe that these results should directly extend to modern models and tasks.

The "retrain" terminology was a little confusing: "retrain from scratch" vs. "fine-tuning" various models.

Regarding terminology, thanks for raising this potentially confusing issue. We reserve the term “retrain from scratch” to refer to the ideal unlearning method: training from scratch on only the retain set (guaranteeing by construction that the unlearned model is not influenced by the forget set). This is denoted as the black star in our plots and is the reference point for how high or low we expect the forget set accuracy to be under ideal unlearning. This is an important reference point for studying relearning attacks: if the forget set accuracy of even this ideal reference point (black star) shoots up upon relearning attacks, then a higher forget set accuracy upon relearning is not due to imperfect unlearning in the first place. What we observe in our plots though, is that the forget set accuracy of various approximate unlearning algorithms shoots up much higher than the black star upon relearning attacks, allowing us to safely conclude that their susceptibility to relearning is due to not having fully erased the knowledge of the forget set during their unlearning phase. On the other hand, we use the term “finetuning” to refer to further post processing of unlearned (or attempted-to-be-unlearned) models.

We attempted to lay out the terminology clearly in Section 3 but we agree with the suggestion of ensuring we always refer to retrain-from-scratch as such, rather than shortening to “retrain”, to ensure clarity. Thank you for the suggestion.

In the Fig 2. captions i recommend replacing "abscissa" and "ordinate" with "x-" and "y-axis".

Thank you for the very useful suggestion. We will update the Fig. 2 caption to make it consistent with all other figures.

—-

Overall, we really appreciate the points raised by the reviewer. We also feel that the addition of a new dataset is very valuable to highlight the generality of our results, particularly on high-resolution images, and the clarifications and feedback about writing to improve the presentation of our work. We believe that we have addressed all concerns and questions raised and we are looking forward to hearing back from the reviewer.

OK thanks for the clarification on that table, I think the explanation makes a lot of sense and I agree that the new results are quite interesting.

I definitely think the efficiency angle pointed out by JQcR is important since (paraphrasing the discussion) it shows not only where prior unlearning methods were ineffective but that unlearning is worth exploring since it can be more efficient than from-scratch training.

So to conclude the discussion I like the expanded results that came out of the review process and I appreciated the additional explanations of the experimental setup particularly as part of the response to JnT6. Pending reviewer discussion I plan to raise my rating to vote for accepting the paper.

We are very thankful to the reviewer for their valuable and encouraging comments. We address the concerns and answer the questions raised by the reviewer below.

Definition of atypical Instances

Thank you for highlighting this. The details about the selection of typical and atypical examples are important to understand the overall results, we will make sure to add them to the paper comprehensively.

We used the consistency scores from Jiang et al. (2020) that provides the notion of an example's typicality. We selected instances from the unlearning class with the highest typicality/atypicality scores in the class as our typical/atypical examples for the forget set, while using all the remaining examples as the retain set. Hence, we didn’t need to use a particular threshold, but rather, just sorted them w.r.t. the scores and took top-k/bottom-k examples for typical/atypical examples. We picked the instances with highest/lowest scores from the entire dataset for class-angostic unlearning, and only the target-class for specific-specific unlearning. We will update the manuscript to highlight all these details for curious readers.

Analysis of weight dist reg, drop in test accuracy and ideas for future work to improve it.

This is a great point, thank you for raising it.

As we discuss in our discussion section, and as the reviewer pointed out, our results reveal an interesting trade-off between robustness to relearning attacks and test accuracy. It’s natural to expect that methods that add noise to the weights (in attempts to push the unlearned model further away from the pretrained model) destroy useful information in the model that is valuable for generalization. In general, the pretrained model has good generalization capabilities (strong accuracy on the test set). Therefore, moving away from it risks losing those, in favor of increasing robustness to relearning. This is related to a fundamental trade-off in unlearning: removing unwanted information while maintaining permissible knowledge and good model utility (e.g., accuracy on the retain and test sets). We will discuss this further in the updated paper.

Moreover, the reviewer’s comment prompted us to think about actionable paths for addressing this and obtaining a better trade-off between generalization/utility and robustness. We outline some ideas here that future work can pursue.

Rather than adding noise to all parameters of the model, we can consider doing this selectively on only parameters that are determined to be disproportionately important for the forget set compared to the retain set (using criticality criteria, such as the one proposed in the SSD unlearning method [1] – such techniques are essential when considering extremely large models such as language models). More broadly, we can try to find a better balance between retaining desired knowledge of the pretrained model while moving away from it more selectively. This can be operationalized by freezing a subset of (cleverly chosen) parameters during the unlearning phase.

We would love to hear the reviewer’s thoughts on this and keep discussing.

[1] Foster, J., Schoepf, S. and Brintrup, A., 2024, March. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence (Vol. 38, No. 11, pp. 12043-12051).

Figure readability and alignment.

Thank you for these suggestions which we have adopted in the paper. Regarding Fig 2, we do actually describe top vs bottom. The caption says: “The forget set is comprised of atypical examples (from the ‘airplane’ class i.e., sub-class unlearning for the top row and all classes i.e., class-agnostic unlearning in the bottom row”. We will however work on improving readability, e.g. by writing “top” and “bottom” in bold to make it easier to spot. If the reviewer has other suggestions, we would be happy to apply them.

—

Overall, we believe that we have addressed all comments and questions brought up by the reviewer and we’re curious if any other concerns remain. If the reviewer is satisfied, we would be really grateful if the reviewer can show stronger support for acceptance by increasing their score.

We are very thankful to the reviewer for their valuable and encouraging comments. We address the concerns raised by the reviewer below.

The method is evaluated only on vision datasets (CIFAR datasets and ResNet architectures)

That’s a great remark. We indeed leveraged simple vision datasets with smaller-scale architectures for our exploration. This was by design, to allow us to answer some fundamental questions about relearning attacks: why do they occur and how can we address them. This is a different direction that complements work on relearning attacks that were carried out in LLMs.

However, we agree with the reviewer that presenting results in an additional larger dataset would lead to more convincingly showcasing the generality of our findings. Based on this suggestion, we present additional results here on the well-known Imagenette dataset [1] (a subset of ImageNet). Similar to other results in the main paper, we focused on using atypical examples for the forget set. The results for three different unlearning methods are presented below (note that tuning hyperparameters for all baselines is difficult in such a tight timeline, hence, we refrain from reporting sub-optimal numbers for other baselines – we will add all results in the next revision of the paper).

We see that the forget set accuracy of SCRUB, similar to our previous results, shoots up upon relearning, even with 0 relearning examples, in line with our findings on CIFAR with an increasing number of relearning examples. On the other hand, variants of our tamper-resistant framework, weight distortion, and weight dist reg, similar to our previous results, are more resistant against these relearning attacks.

[1] https://github.com/fastai/imagenette

The unlearning problem can be unequivocally solved by training from scratch, so the computational overhead introduced by the regularizer is especially important. If the cost of unlearning exceeds that of training from scratch, the proposed approach may lose practical value.

Thank you for raising this. We would like to highlight that our regularizer isn’t more expensive than any other methods commonly employed for approximate unlearning (that commonly rely on two different forward passes through the network), while being significantly more efficient at the unlearning task itself: see our response below where we can unlearn effectively with just a single epoch compared to catastrophic forgetting which significantly benefits from a larger number of unlearning epochs.

The paper reports that the proposed unlearning method takes 2.5 hours, while training from scratch takes about 3 hours. Given that training from scratch guarantees complete forgetting, could the authors clarify why the proposed method would be preferable in practice when its computational cost is comparable?

That’s a great question, and an important observation. We gave all methods a very large training budget in order to ensure that they have the best chance of succeeding at unlearning. However, note that not all methods are equally effective at the task. In order to make this concrete, we present the unlearning/relearning performance of the model with a different number of unlearning epochs. We see that forget set accuracy goes down with an increasing number of unlearning epochs as it only relies on weight decay for unlearning. On the other hand, for weight distortion, we see that the forget set accuracy starts quite low even after a single epoch, and doesn’t benefit much from even a 100 unlearning epochs (which is the consistent budget we used in all our previous experiments for comparison). We also highlight the accuracy after the relearning attack (using 0 relearning examples; larger number of relearning examples from the table were removed as the results were very similar) in order to establish that the unlearning in this case was truly successful. Therefore, this highlights that our methods are particularly efficient at unlearning. The large unlearning budget we chose (which is infeasible in practice due to the cost approaching that of retraining from scratch as the reviewer correctly pointed out) ensures that methods that are slow in unlearning are not penalized. Analyzing the efficiency of unlearning is an interesting question in its own right, and worth investigating in more detail separately.

Do the authors have any insights or preliminary results on how well this method would generalize to other domains?

Susceptibility to relearning has already been analyzed for language models as we list extensively in the paper. The fact that relearning is observed across these different models is a clear sign that it is a more general phenomenon, beyond a single modality/domain.

The focus of our paper is to design a controlled study to gain a deeper understanding of causes and remedies for vulnerability to relearning. Based on this, we have identified weight space distances as an important factor affecting the robustness of different unlearning algorithms and proposed different unlearning objectives to directly address this.

We consider it exciting future work to experiment with our unlearning objectives in LLMs too for increasing their robustness. But there is no doubt, based on the existing literature, that this phenomenon does also occur in LLMs, making this an important and impactful direction.

—

Overall, we believe that we have addressed all comments and questions brought up by the reviewer and we’re curious if any other concerns remain. If the reviewer is satisfied, we would be really grateful if the reviewer can show stronger support for acceptance by increasing their score.

We are very thankful to the reviewer for their valuable and detailed comments. We address the concerns raised by the reviewer below.

Response to W1 (novelty and scope)

Although recovery of forgotten knowledge has been demonstrated in the LLM unlearning literature, we believe our work is the first to show, under controlled conditions, that fine-tuning on only the retain set can undo unlearning. Furthermore, the strength of this effect is quite striking, yielding almost complete recovery of the forget set in some cases (Figure 1). We’ve cited all relevant works we could find, including many works in the LLM setting, but we still consider our analysis to be novel and complementary to those interesting works.

Specifically, LLMs are not ideal for experimentation for two reasons: (i) computational cost, and (ii) it’s difficult to isolate factors at play, which limits conclusions about the causes and remedies of vulnerability to relearning. To elaborate:

• The goal of unlearning in LLMs is vaguely defined since the model defines a joint probability over the entire sequence (materialized via conditional probabilities of the next token conditioned on all preceding tokens). This means paraphrasing of text complicates evaluation since we would like to minimize the joint probability of any arbitrary paraphrasing of the text to be forgotten. In the case of classification, the goal is readily quantified.
• Powerful LLMs can in many cases acquire knowledge “anew” rapidly, even just at in-context learning time, by synthesizing existing knowledge in novel ways. This makes it harder to separate knowledge acquired by relearning from knowledge acquired/synthesized “on the fly”.
• The pretraining → finetuning methodology in LLMs further complicates matters. Having pretrained on a vast corpus of text leads to ambiguity about what knowledge was acquired at pretraining time versus at finetuning time, and when the knowledge was acquired can make a difference for unlearning/relearning dynamics, complicating the analysis.

To eliminate the above complicating factors, we study unlearning (and relearning) of a specific set of examples in a model that is trained (from random weights) on a dataset including those examples. In our setup (small models, no prior pretraining phase), we are also able to compute and compare against the oracle unlearning method of retraining from scratch without the forget set, and study relearning in that model as a reference point for how much information can be acquired “anew” about the forget set in a model that was truly never trained on that forget set before.

Our analysis in these controlled conditions is the first to show that, beyond doubt, relearning attacks succeeding are a failure of unlearning. Our analysis also led us to clearly explain this phenomenon from a weight space perspective and propose new methods to address it, both of which are important contributions that also showcase how fruitful it can be to conduct experiments in our setting.

We are grateful to the reviewer for highlighting reference D which we weren’t aware of at the time of submission. We found that our vision models aren’t vulnerable to these quantization attacks: forget set accuracy stays at the same level as the full-precision unlearned model regardless of the number of bits until it gets close to 8 bits at which point the model collapses. Thus, the spontaneous-recovery phenomenon we explore is distinct from the recovery obtained via quantization. Below are some sub-sampled quantitative results due to space constraints:

We have two hypotheses for why they are not effective in our setup: (i) Language models consist of billions of parameters (compared to a few million in our case). Hence, the deltas are really small. This results in easily knocking off the weights to their previous pretrained values, which isn’t the case in our simple vision models. (ii) This problem is further exacerbated with models that are trained with quantization in mind. Such models can immediately latch onto the previous values which were regularized to be easily quantizable.

Furthermore, D showed that small weight distances (between the pretrained and unlearned model) are the cause of quantization attacks. We instead show this for relearning attacks for the first time. Given that not all models suffer from quantization attacks (an interesting finding in and of itself), it is important to understand how weight space distances affect each type of attack and this was not previously understood. Finally, the approach taken in D for increasing the distance between the unlearned and pretrained model is to increase the learning rate (in existing unlearning objectives/algorithms). While this is a very valuable first step in the exploration, the authors themselves admit that their framework is “highly sensitive to hyperparameter selection, leading to an unstable unlearned model”. Instead, we propose unlearning objectives that are designed to handle this natively, and we study the trade-offs of these new unlearning algorithms compared to the old ones.

Based on the above, we hope the reviewer agrees that our work is complementary to D. We added these new results and discussion of D to our paper.

Overall: our study of relearning in vision classifiers complements the existing findings in the LLM unlearning literature and is the first study to (i) show beyond doubt that the relearning phenomenon is due to imperfect unlearning (rather than due to other factors that remained entangled in LLM studies); (ii) show that relearning is possible without access to any examples from the forget set; (iii) show that vulnerability to relearning is strongly correlated with weight space distances between the pretrained and unlearned models, complementing D that shows this for quantization but not for relearning attacks (this is important since, as discussed, not all models that suffer from relearning attacks also suffer from quantization attacks); (iv) derive more robust algorithmic principles, going above and beyond prior work in D that tried to increase weight space distance by increasing learning rates, leading to an unstable system that is highly sensitive to hyperparameter selection.

We consider it important future work to translate these advancements to other modalities, architectures and different sizes of models and datasets. We have added results on a higher-resolution ImageNette dataset which further validates the generality of our findings; please see response to reviewer JQcR.

We hope that we have convinced the reviewer about the novelty and significance of our work, but we would be very happy to discuss this more during the upcoming reviewer-author discussion phase.

Response to W2 (MIA attacks)

We are thankful to the reviewer for the useful suggestion regarding MIA attacks. Based on the reviewer’s feedback, we present the results for MIA below. We chose the MIA from Kurmanji et al. (reference E) according to the reviewer’s suggestion. We observe that, as is the case in our initial findings with the forget set accuracy metric, this MIA can “fool us” into thinking that unlearning successfully erased all traces of unlearned data, even in cases where we know that the unlearned model is prone to relearning attacks. We will add these results in the appendix.

Confusing structure

Thanks for raising this. We believe that this structure offers greater compactness compared to presenting “preliminary results” first without our proposed methods, and then repeating those plots later with the addition of more methods. We would be happy to hear any suggestions from the reviewer regarding how this can be more effectively communicated without replication.

Q: use colors together with different marker shapes for existing vs. new methods?

Thank you for the useful suggestion. We will add all the results in a tabular form in the appendix for curious readers who are interested in looking at precise values. Please let us know if this would be helpful.

—-

Overall, we really appreciate the points raised by the reviewer as we feel these are important discussions to have, and reflecting these in the paper has strengthened our contribution. We also feel that the addition of the quantization and MIA results is very valuable. We are looking forward to hearing any further thoughts from the reviewer and discussing more.