Improved Localized Machine Unlearning Through the Lens of Memorization

We would like to thank the reviewer for the insightful feedback! We address the weaknesses identified by the reviewer comprehensively below, and would really appreciate hearing the reviewer’s thoughts on our responses.

• Response to W1 on limited novelty. We would like to respectfully push back on this. To the best of our knowledge, we are the first to investigate the link between methods that aim to localize memorization and unlearning algorithms; an investigation which led to proposing a new SOTA algorithm. We believe that this investigation is, therefore, novel and of significant scientific value (both in its own right and thanks to the resulting discovery of a new SOTA method). Further, while our method does build on existing building blocks (this is by design, and we actually view it as an advantage), its novelty arises from a carefully-selected combination of ingredients, based on our insights that are grounded in extensive empirical investigation. And, we show that it yields SOTA results across metrics, forget sets, datasets, and architectures and against both localized and full-parameter unlearning methods. At the same time, it is more robust than prior localized unlearning work to the parameter budget and outperforms prior work when paired with different unlearning algorithms, showing its versatility. Please refer to the summary in the “common response” for an overview.

• Response to W2 and Q1: “The results in Section 3 do not necessarily support the hypotheses in Section 5.1”. Since we have not included any results in Section 3, we kindly ask the reviewer to elaborate further on this point: which results and factors is the reviewer referring to? If the reviewer is referring to the results in Table 1, those do support the arguments presented in Section 5.1. Examining different granularities (parameter, channel) and criticality criteria (gradients, weighted gradients) and their impact on unlearning and utility metrics serves as an ablation study to identify the granularity and the criticality criteria that lead to a more effective unlearning algorithm. According to Table 1. (where the shaded cell represents the strategy employed in our proposed method), we conclude that using channel-wise granularity and weighted gradients is the best choice based on which we build our localization strategy.

• Response to W3: “Unlearning in LLMs is a more pressing concern”. We agree that unlearning in LLMs is an important research area. However, unlearning in vision classifiers is also a very important and relevant research problem (e.g. for supporting deletion of user data from vision classifiers) that is largely unsolved, is equally important as its LLM counterpart, and presents different properties compared to LLM unlearning (we are happy to elaborate on this much more if the reviewer finds this discussion interesting or helpful; see e.g. [1] for some discussion). Given the fact that LLM unlearning and unlearning in vision classifiers are fundamentally different (in terms of specification of goals, metrics, and state-of-the-art methods), and we need to make progress in both, we don't think it's fair to deduct "significance" points from our work for not addressing LLM unlearning. With that said, our results with ViT demonstrate that, interestingly, our method outperforms previous SOTA in transformer-based architectures, too, making it a great candidate for future explorations in the LLM space.

• Response to W4 on typos. Thank you for pointing this out. We have revised our draft accordingly.

We look forward to hearing back from the reviewer. We would be happy to continue these conversations and address any remaining concerns that the reviewer may have.

[1] Yao, Yuanshun, Xiaojun Xu, and Yang Liu. "Large language model unlearning." ICLR (2024).

Overall, we thank the reviewer for their time and valuable feedback, and we look forward to continuing this discussion. To reiterate, while our contributions are empirical in nature, we have made substantial progress in deepening our scientific understanding on the connections between memorization and unlearning: we are the first, to the best of our knowledge, to study whether locations where memorization is hypothesized to occur give rise to better localized unlearning strategies. This gave rise to DEL, a new algorithm that achieves SOTA results on three datasets / architectures (CIFAR-10 with ResNet-18, SVHN with Vi,T and ImageNet-100 with ResNet-50), various forget sets (IID and non-IID variations of CIFAR-10), for two different unlearning metrics, against both localized and full-parameter unlearning methods. At the same time, DEL outperforms all previous localized unlearning methods in terms of utility metrics too, which indicates that our method preserves permissible knowledge (see e.g. Table 2, Figure 4). In addition, DEL is more robust to the parameter budget, outperforming the previous SOTA method SalUn across different budgets, and when paired with different unlearning algorithms (Figure 3 and Figure 2).

We would like to thank the reviewer for the insightful feedback and great discussion. We respond to the points raised comprehensively below and are looking forward to engaging in further discussion on these topics.

• Response to W1 on “close connections or logical relations” between the definitions of memorization and unlearning. Thank you for raising this important point. We wrote down those definitions carefully to demonstrate that the concepts of label memorization and unlearning are tightly linked to one another. Specifically, as discussed in Section 2.2, in the paragraph “connections with unlearning”, if an example isn’t memorized at all (according to Definition 2.2), it can be considered “trivially unlearned” (according to Definition 2.1). More broadly, we discuss in that same paragraph empirical work that investigates how easy it is to unlearn forget sets of different degrees of memorization. Establishing the link between these concepts is crucial to justify why we investigate memorization localization hypotheses for the purpose of informing which part of the network to perform unlearning on. Had we not shown the connection between these concepts, it would be difficult to motivate that choice. We hope that this point clarifies the link between these two concepts, as well as our motivation for providing and discussing these definitions. Now, relating to the disconnect between definitions and metrics, we discuss this in the bullet point below.

• Response to W2 on using metrics that are more aligned with the definition. Indeed, both memorization and unlearning are quantities that are challenging to measure in a rigorous way without requiring a lot of computation (a fact that is acknowledged in extensive prior work, e.g. see the discussion in Triantafillou et al.). The metrics for unlearning quality that we utilize are inspired by Definition 2.1 but, indeed, unavoidably make certain simplifications. For instance, it is true that comparing the accuracy of the retrained and unlearned models does not fully capture a comparison between their distributions of outputs of those models (because the argmax of the softmax can be the same, making the unlearned and retrained models to have the same “predictions”, even though the confidences / softmax distribution may be different). To amend this issue, we also use a Membership Inference Attack (MIA) that leverages the confidence of the model in order to predict whether an example was trained on or not. This is a common way to “operationalize” computing the distance between two distributions (e.g. see Fan et al., Kurmanji et al., etc.). There is recent work by Hayes et al., Triantafillou et al., (which we cite and discuss in the “Unlearning Evaluation” paragraph in Section 2.1), that designs more sophisticated metrics for unlearning that may come closer to capturing formal definitions (and representing more complex “attacks”). These, however, require training a very large number of models from each distribution and are complex to implement, requiring various design decisions and simplifying assumptions, too. For instance, the metric in Hayes et al. assumes that distributions are Gaussian, which in various cases does not hold in practice (see Fig 8 in Hayes et al. and their discussion on the limitations of this metric). Similarly, the metric of Triantafillou et al. requires implementing various “decision rules”, that require making a number of design choices based on assumptions that may not always hold. Overall, for all these reasons, evaluation in unlearning is an open research area in and of itself. We view that research direction as being orthogonal to our contributions here, so, based on the above discussion, we build on metrics that are used in recent state-of-the-art unlearning method papers (see Fan et al. whose experimental setup we adopted, for instance; but most of the other cited works use similar metrics too). We thank the reviewer for raising this important point, and we are happy to discuss more with the reviewer as well as to reflect more of this discussion in our updated paper, acknowledging that, while the metrics we use are related to the definition, they may not reflect it fully faithfully. This holds true in all empirical evaluations in the field currently.

• Response to Q1: “memorization property can vary from model scale”. In principle, we can compute the label memorization for any model, and the memorization localization approach of Maini et al. that we build upon is an algorithm that is agnostic to the model size. Indeed, our contributions are empirical, but we have shown SOTA results on different datasets / architectures, forget sets, and across metrics (see the “common response” for an overview). We view the versatility of our method, as evidenced by its SOTA performance across the board, as strong evidence for the significance of our findings.

• Response to Q4 on different hyper-parameters. Yes, the performance is dependent on the hyperparameters, as is the case for any method. For fair and careful experimentation, we tuned the hyperparameters of our method and each baseline separately for each scenario and parameter budget we considered. Please see Section A.2 for full details. Note as well that we find that our localization method is quite versatile: it pairs well with different unlearning methods and is more robust to prior state-of-the-art to the parameter budget (Figure 3). We view this as an important advantage.

• Response to Q5 on Tabel 7. Generally, it is possible that the accuracy improves for an increased budget of parameters. The best way to see this is through the rightmost subplot of Figure 2 - note that Table 7 (currently Table 10 in the revised paper) that the reviewer refers to shows the retain set accuracies, so it may be a little less relevant compared to test accuracy and other metrics. However, as mentioned in the response to Q1 as well, we are more interested in the trade-off between efficiency (e.g. using parameter budget as a proxy), unlearning quality and accuracy. Otherwise, one could just retrain the entire network from scratch to obtain perfect unlearning. Our rationale for considering localized unlearning is the hypothesis that it yields better such trade-offs. Indeed, through our extensive experimentation we show that DEL obtains the best unlearning quality both compared to localized and full-parameter unlearning methods, and that it outperforms all state-of-the-art prior localized unlearning methods in terms of accuracy too, thus introducing a new interesting point in the pareto frontier, and enhancing our scientific understanding of relevant trade-offs in unlearning methods.

We would like to thank the reviewer for the insightful feedback! We respond to the reviewer’s comments below:

• Response to W1 on limited datasets and models. Thanks for the comment. To address your feedback, we have added another dataset / architecture pair: ImageNet-100 with ResNet-50, described in detail in Section 6 and Table 7. This dataset and architecture are significantly larger than the previous ones we considered, and the image resolution is significantly larger compared to our previous experiments. We find that, consistent with our findings on CIFAR-10 and SVHN, DEL outperforms prior methods on all unlearning metrics, while also having better test accuracy compared to all prior localized unlearning approaches. We view these new SOTA results as a strong indication for the significance of our findings and the versatility of our method across datasets and architectures. Please refer to the common response for an overview of our results, showing SOTA behavior across the board.

• Response to Q1.1: “Is the localization mask determined by each mini-batch?”. No, the localization mask is determined based on the magnitude of the weighted gradients over the forget set, as discussed in Section 5. This is performed by accumulating the magnitudes of the weighted gradients across all mini-batches of the forget set (see line 4 in Algorithm 1, A.4). Specifically, this approach efficiently computes the overall gradient magnitudes for the forget set by summing the weighted gradients across the mini-batches. We hope this clarifies.

• Response to Q1.2: “ Is the localization mask fixed for different networks?”. No, the localization mask varies across different models, as it identifies the specific subset of model parameters that need to be modified to fulfill the unlearning request, using the method described above.

• Response to Q1.3: “if the mask is not accurate, does it affect the accuracy?”. It depends. At the extreme where the mask selects to update all parameters, if we allow a sufficient finetuning budget (and the retain set is large enough), we can obtain the highest possible accuracy. This mask may not be the most “accurate” in that it selects several additional parameters rather than only the “bare minimum” that encodes the information we wish to forget, but it can obtain the highest accuracy (at the expense of computational efficiency). However, what we are interested in is the best trade-off between computing, accuracy, and unlearning quality. We hypothesize that we can achieve better trade-offs there by selecting the smallest set of weights that we should operate on for unlearning. And we have strong evidence for this: DEL outperforms all prior methods in terms of unlearning quality across the board (and all prior localized methods in terms of accuracy) and is SOTA across various parameter budgets (see e.g. Figure 2).

• Response to Q2 on different data distibutions. We have applied DEL on two different types of forget sets, one that is sampled IID from the training data and one that is non-IID (from a subset of classes). We find that DEL outperforms prior methods in both cases. Does this answer the reviewer’s question?

• Response to Q3 on different architectures. Yes, We have already shown results using three different architectures: a ResNet-18 and ResNet-50, which are convolutional networks, and a ViT, which is a transformer. We believe ResNet and ViT cover the two most widely used categories of architectures, and the fact that DEL is SOTA in both cases is important evidence of its versatility.

We would like to thank the reviewer for their time. We are frankly surprised and puzzled with this score and with the reviewer’s feedback. The reviewer makes a number of assertive statements which are factually incorrect. Please see our responses below. We would really appreciate hearing back from the reviewer on these.

First, let us reiterate our contributions:

A) We perform the first, to the best of our knowledge, study of whether hypotheses for where memorization occurs in a network can give rise to improved localized unlearning algorithms, through informing which subset of parameters to modify during unlearning. This is the first attempt that we are aware of to bridge memorization localization methods with unlearning algorithms. Our analysis revealed previously-unknown trade-offs between different data-agnostic and data-dependent localization strategies on several metrics of interest (unlearning quality and utility metrics) and under different parameter budgets (Figure 2).

B) Building on those insights and on extensive empirical investigations, we propose a new localized unlearning method, DEL, that borrows the deemed-to-be most successful “ingredients” (criticality criterion, granularity of localization) from the memorization literature and ports them into a framework that yields an efficient and practical localization algorithm for unlearning.

C) Overall, we find that DEL achieves SOTA results on three datasets / architectures (CIFAR-10 with ResNet-18, SVHN with ViT and ImageNet-100 with ResNet-50), various forget sets (IID and non-IID variations of CIFAR-10), for two different unlearning metrics, against both localized and full-parameter unlearning methods. At the same time, DEL outperforms all previous localized unlearning methods in terms of utility metrics, too, which indicates that our method preserves permissible knowledge (see, e.g., Table 2, Figure 4). In addition, DEL is more robust to the parameter budget, outperforming the previous SOTA method SalUn across different budgets, and when paired with different unlearning algorithms (Figure 3 and Figure 2).

Overall, we view these contributions as a significant step forward in the development of localized unlearning methods, as well as growing our scientific understanding of behaviors and trade-offs of different memorization localization hypotheses for the purpose of unlearning.

Now, to address the reviewer’s comments specifically:

• Response to W1.1: “based on a lot of assumptions without justification”. Our method does not make additional assumptions that we are aware of compared to prior work. What does the reviewer have in mind here? Further, it is not true that our method lacks justification. It is based on hypotheses that are investigated via extensive experiments (see Figure 2 and Table 1) – for instance, as reviewer EYNL put it, ”These insights serve as the foundation for their proposed method, DEL.”. We respectfully argue that empirical results of carefully-designed experiments do qualify as “justification” and that our method is well-grounded in our findings and insights.

• Response to W1.2: “it is very hard to see if the proposed method is indeed ok in terms of unlearning (while preserving the rest!)”. We are puzzled by this comment. We have evaluated our method comprehensively on different forget sets, datasets, and architectures, and different metrics (two metrics for forgetting quality as well as utility metrics) against SOTA methods for both localized as well as full-parameter unlearning. We find that our method outperforms the previous SOTA across the board. This is evidence both that it is “ok in terms of unlearning” and that it “preserves the rest” (which is captured through the utility metrics).

• Response to W2: “It is very hard to see the core contribution clearly due to poor writing. It was very hard to read and follow”. We are very surprised to see this comment. We put a lot of care into writing our paper and all other reviewers found it well written. Does the reviewer have any specific feedback or suggestions for how we should improve our writing?

• Response to W3.1: “Experiments look quite limited in terms of benchmarks (datasets, compared methods)”. During the rebuttal, we have added experiments using an additional dataset and architecture pair: ImageNet-100 with ResNet-50, described in detail in Section A.6 and Table 8 in the revised paper. This dataset and architecture are significantly larger than the previous ones we considered, and the image resolution is significantly larger compared to our previous experiments. We find that, consistent with our findings on CIFAR-10 and SVHN, DEL outperforms prior methods on all unlearning metrics, while also having better test accuracy compared to all prior localized unlearning approaches. We view these new SOTA results as an additional strong indication for the significance of our findings and the versatility of our method across datasets and architectures. Please refer to the common response for an overview of our results, showing SOTA behavior across the board. Regarding “compared methods”, we have compared against all the SOTA methods we are aware of, and during the rebuttal we additionally added a comparison to the contemporaneous work of Foster et al. (SSD) as well as the influence unlearning method of Izzo et al. (IU) . Please refer to the updated Table 2 and Table 8 in the revised paper. We find that DEL outperforms all prior methods.

• CIFAR10-ResNet-18 with IID forget set

• CIFAR10-ResNet-18 with non-IID forget set

• SVHN-ViT with IID forget set

• SVHN-ViT with non-IID forget set

• Response to W3.2: “I am afraid that the localized unlearning approach may hurt the preservation of remaining parts, but it is unclear if it is true.”. As mentioned above, this is exactly what the utility metrics capture (specifically, these are the accuracy of the unlearned model on the retain set and the test set). And through our thorough investigation, we find that our method outperforms the previous state-of-the-art localized unlearning in terms of these metrics.

In summary, the reviewer’s feedback seems to dismiss our empirical results, which provide a solid grounding for our novel method, and the ample evidence that DEL yields SOTA performance both in terms of unlearning metrics and utility. The harsh tone of the review and associated score are not corroborated by factual criticism with concrete reference to specific parts of our work. We are nonetheless keen to engage in a grounded scientific discussion about our work. We believe that the extensive clarifications above and the additional results on another dataset / architecture and additional baseline have addressed all criticism. We look forward to hearing from the reviewer if their stance is changed based on this. If not, we would like to know in what ways, in the reviewer’s opinion, the paper can be concretely improved.

Dear the authors,

I appreciate your detailed comments - many of my concerns were lifted, so I increased my score to 5 (from 1). I must say that my initial score was quite low due to my misunderstanding for the overall structure of this work and your rebuttal clearly elaborated them (for some reason, I still have a hard time to read the revision, though).

This work was based on the memorization assumption, but I was not able to see 'any' memorization related metrics or evidence in the experiments. I meant 'without justification' by that. Even though the overall performance look good, it is still unclear if it was from the intended assumption on leveraging memorization or somewhere else.

I am glad to see the new results for ImageNet, which helped me a lot to alleviate my major concern (so I was comfortable to increase my score by a large margin). Initially, I was not quite sure if the memorization can indeed happen for small network with small benchmarks, but ImageNet results look promising and I am a little bit more convinced.

Lastly, it could be better if the proposed method was compared with some of the recent state-of-the-art methods such as SCRUB, but the current comparisons could be enough.

See the following recent work, which may be related to the current work: AK Tarun et al., Fast Yet Effective Machine Unlearning, IEEE Trans Neural Networks and Learning Systems 3(9), Sep 2024.

We truly thank the reviewer for their response to our rebuttal and for updating the score.

We kindly request that you also update the score reflected in your written review, as it currently shows the previous rating. We believe you can do this on OpenReview by selecting 'edit' on your original review.

Overall, we sincerely thank you for your time and valuable feedback. We believe that we have fully addressed your concerns and strengthened our paper substantially.

To summarize: DEL achieves SOTA results on three datasets / architectures (CIFAR-10 with ResNet-18, SVHN with ViT, and ImageNet-100 with ResNet-50), various forget sets (IID and non-IID variations), for two different unlearning metrics, against both localized and full-parameter unlearning methods. At the same time, DEL outperforms all previous localized unlearning methods in terms of utility metrics too, which indicates that our method preserves permissible knowledge (see e.g. Table 2, Figure 4). In addition, DEL is more robust to the parameter budget, outperforming the previous SOTA method SalUn across different budgets, and when paired with different unlearning algorithms (Figure 3 and Figure 2).

Based on this, we wonder if you would consider raising your score further. If not, what are the additional concerns or weaknesses of our work preventing you from doing so?