Blind Unlearning: Unlearning Without a Forget Set

We thank reviewer 1dea for their thoughtful comments. We’re encouraged that this reviewer found our paper to be well-written and structured.

Below, we provide answers to their questions and remarks.

The paper presents a setting where machine unlearning occurs without access to the forget dataset, which is weaker compared to zero-shot unlearning [1], where unlearning is achieved without access to either the forget or remaining datasets.

We agree with the reviewer that Zero-shot unlearning is a stronger setting than Blind unlearning. However, prior work which achieves zero-shot unlearning does so only with unlearning classes of data, which is a weaker setting than our setting of forgetting any subset of the training data. It is also a setting that enjoys significantly less broad applicability with regards to the motivating examples found in the general response to all reviewers.

The proposed setting is problematic. While the paper claims that blind unlearning does not access the forget dataset, it still allows access to the gradients of the entire dataset, including both the remaining and forget data. This is unreasonable because, to ensure data privacy, we typically use privacy-preserving mechanisms like DP-SGD. Gradients can leak data information, as attackers could exploit gradients to determine whether a specific data point was part of the training set. By analyzing how the gradient changes when a particular data point is used, adversaries can infer whether or not that point was included in training.

Our theory shows that, given only the accumulated gradients of the loss with respect to each parameter, one cannot perfectly recover the instances associated with those gradients without further information (e.g., about the size of $D_{forget}$. Furthermore, by establishing and motivating the setting of blind unlearning, our work lays the groundwork for settings with stronger guarantees on forget-set recoverability. We hope that future work can explore methods for blind unlearning with guaranteed non-recoverability, and our present work serves as a stepping stone toward these contributions. On balance, given that the vast majority of the unlearning literature requires the raw forget-set, $D_{forget}$, a method that only requires access to the accumulated gradients seems to represent a significant step forward.

Step 2 in Figure 1 is confusing. If you have computed all gradients on the retained dataset, why not simply retrain the model? Fine-tuning and gradient ascent methods are typically used for efficient unlearning, but this approach seems to overlook that option.

We believe that this is a misunderstanding. The gradients on the retained dataset are computed only once in Step 2, and only a single ascent step is applied. Retraining the model would require repeated computation of these gradients – hence why our results demonstrate that Reload is significantly more computationally efficient than fine-tuning or gradient ascent.

The estimation of gradients on the forget dataset is not reasonable. The equations (5-7) fail when applied to average or mini-batch gradients, which makes the method problematic in practice.

We assume that the gradients cached at the end of training are full-batch and accumulated. Even when mini-batch gradients are used, these can be summed up and cached for the same effect.

Several evaluation metrics, such as NSKL and FSKL, are unclear. The goal of unlearning is to forget the targeted data while retaining the performance of a model trained from scratch. Evaluating the model based on its output and data seems unnecessary and could detract from the focus on model performance.

One of the goals of unlearning is to produce a model that is a close approximation of the naively retrained one. NSKL and FSKL measure this by quantifying the dissimilarity between the outputs of the unlearned model and the retrained model on the same data.

We hope this response has addressed the reviewer’s concerns, and we would be glad to continue the discussion if there are any remaining questions or comments. We thank the reviewer again for their attentive, detailed feedback.

We thank the reviewer for their response.

1. The blind setting is comparatively weaker, though considering zero-shot unlearning has not been achieved for unlearning random samples, blind unlearning can be considered another step towards zero-shot unlearning.
2. If we also have access to the ground truth of $\nabla \mathcal{L} (w, x_j)$ then yes. However, this assumes that we know that only a single item was removed from $\mathcal{S}$.
3. In this figure we start from the parameters produced after the ascent step and re-initialisation step. The optimization is conducted via whichever optimization process was originally used for training. The reviewer is correct that this will require significant memory resources, but will take far less time than retraining from scratch.
4. We take the assumption that these gradients are not averaged over the batch but simply accumulated, which is how the equality is derived.
5. The metrics are discussed in the paper in section 4.2 briefly and then more so in Table 6 in Appendix A.3

We thank reviewer ST8s for their thoughtful comments. We’re encouraged that this reviewer found our setting to be an interesting problem and that they appreciated the algorithm and our experiments.

Below, we provide answers to their questions and remarks.

The motivation of this paper is to reduce the retaining of user data for unlearning. This paper only focuses on reducing the usage of forgetting data and requires the usage of remaining data. However, the size of the forgotten data is usually far smaller than the remaining data, and this also motivates other works that reduce the usage of the remaining data during unlearning [1,2,3]. Compared with such works, the proposed method required more training data to be preserved for unlearning. In addition, some methods that require both the forgetting data (10% in this paper's experiment setting ) and a subset of remaining data during unlearning (10% in this paper's experiment setting) only require 20% training to realise unlearning. However, the proposed method still requires another 90% of the remaining data during unlearning. Therefore, the proposed method does not match the motivation of reducing the data usage in unlearning

We believe this is a slight misunderstanding. The motivation of this paper is to reduce the retaining of the user data once that data has been requested for deletion. Additionally, the assignment of 10% of the training data to the forget set is simply a single setting that this can apply to, we now include the setting of forgetting 30% of the training data. We do agree with the reviewer that an interesting extension to this work would be using less of the retained user data to fine-tune the model after re-initialisation.

One previous work discussed reducing the usage of forgetting data, but this paper does not mention it in related works or compare it in experiments [4]. Other works have applied gradient-based input saliency maps for unlearning, and this paper does not mention or compare them [5,6].

We thank the reviewer for bringing [4] to our attention. We have included it in the discussion. This work operates solely within the realm of class unlearning, which is a weaker and less useful setting of unlearning that we consider. [5] is discussed in depth in the paper, and we have included it in comparisons for unlearning now. [6] is a work that was released a day before the ICLR deadline, and as such was not brought to our attention in time nor would it have been feasible to include it at such a late stage.

This paper does not contain ablation studies to prove the effectiveness of the parameter selection component and different eta and alpha

The reviewer raises a valid concern regarding the hyperparameter ablations, and we have included these in the appendix.

This paper only conducts experiments on 10% random sample unlearning and 100 samples in class unlearning. More different experiment settings are required.

We have included 30% random sample unlearning in the appendix section.

In experiments, why does a huge gap exist between the retrained model and unlearned under the current evaluations? What is the differences between retrained model and unlearned.

We believe that this is a notational misunderstanding. The retrained model is the gold-standard and is the same as $\mathcal{M}^{\theta \sim}$.

In eq 8, does nabla stand for the gradient of accumulated loss or averaged loss on D?

Accumulated.

How to decide eta and alpha in the proposed method?

We decide $\eta$ and $\alpha$ using a Bayesian hyperparameter sweep.

We hope this response has addressed the reviewer’s concerns, and we would be glad to continue the discussion if there are any remaining questions or comments. We thank the reviewer again for their attentive, detailed feedback.

We thank reviewer x8sa for their thoughtful comments. We’re encouraged that this reviewer found our work a potential significant algorithmic contribution and that they appreciated the breadth of our comparisons against baseline algorithms.

Below, we provide answers to their questions and remarks.

Although the author's method does not require access to the forget dataset, calculating $\nabla_\theta \mathcal{L}(\mathcal{D}{\text{forget}}) = \nabla\theta \mathcal{L}(\mathcal{D}) - \nabla_\theta \mathcal{L}(\mathcal{D} \backslash D_{\text{forget}})\nabla_\theta \mathcal{L}(\mathcal{D} \backslash D_{\text{forget}})$ since we do not know which data might need to be forgotten. I suggest the author describe potential application scenarios more in the introduction.

We thank the reviewer for this concern. We have addressed further potential application scenarios in the response to all reviewers. We believe this is a slight misunderstanding. Reload assumes that the practitioner training the model retains the full batch gradients from the end of training, thus meaning that all the training data (including the “forget set”) need not be retained for unlearning.

Retaining in step (1-3) is risky because prior work [B] has demonstrated that, for fully connected layers (such as the softmax mentioned by the authors), the input can theoretically always be inferred from the gradients, regardless of the layer's position. Experiments of [B] have also shown that images can be reconstructed from gradients. Therefore, when considering the retention of , the authors should take additional measures to prevent unintended privacy leakage.

The reviewer raises a valid concern and we thank you for that. Experiments of [B] mentioned have shown that parts of images can be reconstructed from gradients, however, the work mentions that most of the images are unrecognizable and the small portion of the batch that have the most reconstruction, are still perturbed. Additionally, in order to reconstruct this batch, [B] requires knowing the number of images in the batch to produce a prior, which is not a quantity that is known in Reload. We believe this is a valid concern, especially as work progresses in this area, and note that it would be an important future direction to study defenses against better methods of hiding data information in gradients.

The author evaluates the importance of weights in step (4), drawing inspiration from prior unlearning work [A]. However, the paper lacks a comparison with existing methods for assessing weight importance. It’s crucial to clarify the differences between the gradient norm of the forget dataset used in previous studies and the ratio employed by the author as knowledge values. Simply stating in the appendix that their method outperforms other weight importance methods is insufficient; the author should at least report experimental results to justify the choice of knowledge values and help readers understand the contributions. Furthermore, the baseline analysis appears to overlook a comparison with the significant work [A], which should be addressed for a more comprehensive evaluation.

The reviewer raises a valid point with the lack of comparison with regards to the weight importances calculated in [A]. A deeper comparison has been added to the appendix in order to justify the formulation of the knowledge values in comparison to [A]. We agree that comparison with [A] should be done, and these results have been added to the paper.

We thank the reviewer for their thoughtful comments. We’re encouraged that this reviewer found our problem setting “well-motivated,” and that they enjoyed our extension to the setting of remedial learning.

Below, we provide answers to their questions and remarks.

The paper lacks structure and clarity in its writing. For example: To the best of my knowledge, "remedial learning" seems to be a new setting introduced in this paper. If this is not the case, please provide the necessary references upon its first mention. Regardless, the authors should provide sufficient background on this setting in the Abstract, Introduction, as well as Title. Failing to do so may lead to unnecessary difficulties and confusion for readers attempting to understand the task's objective from the beginning.

This understanding is correct. “Remedial learning” represents a new setting introduced in this paper. It comprises the scenario wherein the practitioner wants to unlearn the influence of certain aspects of $\mathcal{D}$ on the model, and to replace them with the corresponding aspects of $\mathcal{D}_{new}$

(e.g., unlearning a spurious backdoor from $\mathcal{D}$, and replacing the influence of the backdoor with the components of $\mathcal{D}_{new}$ that do not contain the backdoor in question). We have edited the manuscript to clarify that remedial learning is a novel setting introduced in this work.

We have fixed the other writing concerns raised by this reviewer in lines 211, 213, 183, and 209, and we sincerely thank them for their attentive eye to detail.

The theoretical novelty of the derivations from Eq. (4) to Eq. (8) is unclear. The result seems simple and straightforward. I would appreciate further clarification on the contribution of this part.

The contribution of Equations (4) through (8) does not lie in mathematical novelty, but rather in the observation that linearity of differentiation allows one to treat the difference between the (cached) gradients of the loss with respect to $\mathcal{D}$ and the gradients of the loss with respect to $\mathcal{D}_{new}$

as the direction of movement in parameter space needed to unlearn the influence of the unobserved $\mathcal{D}_{forget}$. The reviewer is correct that the core of this step is mathematically straightforward. However, our results highlight its empirical utility in allowing Reload to outperform state-of-the-art baseline benchmarks. We would argue that simple-but-effective ideas hold outsized value for the machine learning community due to their ease of implementation.

Please correct me if I misunderstood. From line 8 in Algorithm 1, it appears that selective re-initialization is performed on each element in the parameter vector. Intuitively, this design could increase the computational burden. Please explain this point. Additionally, in line 9 of Algorithm 1, a hyper-parameter is introduced as a threshold to determine whether a parameter should be initialized. Is this hyperparameter the same for all model parameters? Ideally, it should vary, and discussing a suitable strategy for selecting an appropriate for different parameters is recommended. Knowledge values are computed for all parameters. However, only parameters with knowledge values less than that $\alpha$-quantile of the empirical distribution of knowledge values over all parameters are selectively re-initialized. We’ve amended this line of the algorithm to use a more conventional if-block to avoid confusion.

A hyper-parameter is introduced as a threshold to determine whether a parameter should be initialized. Is this hyperparameter the same for all model parameters? Ideally, it should vary, and discussing a suitable strategy for selecting an appropriate for different parameters is recommended.

The hyperparameter threshold $\alpha$ is the same for all model parameters. It cannot vary

Q1: In Line 241, what do you mean by “single-based” ascent update?

This is a typo. We’ve corrected it to read “single gradient ascent update step.”

What’s the impact of the Laplace smoothing constant ε in Eq (8)? How should its value be chosen?

The Laplace smoothing constant in Equation 8 is to prevent division by zero. Additionally, it allows the equation to maintain measuring relative importance when the numerator is 0, avoiding assigning the same KV to all parameters with numerator 0 and denominator non-zero Its value need only be small – we use 1e-10 in our experiments.

We hope this response has addressed the reviewer’s concerns, and we would be glad to continue the discussion if there are any remaining questions or comments. We thank the reviewer again for their attentive, detailed feedback.

Practicality of Setting

My final relatively milder criticism is that I do not see the motivation for when this setting is realistic. When is it that the learner only has access to a large retain set (90% of the dataset), multiple gradient checkpoints during training, but not the forget set. This selectively requires the learner to not have the forget set but have all these other memory intensive data structures. Please see the general response to all reviewers for a discussion of this concern.

Questions

SSD results look unrealistic. The accuracy seems to be 1% on CIFAR100 which is the trivial accuracy. Previous results with SSD have shown much better performance on clean test acuracy with SSD. It appears the hyper-param optimisation may not have been executed properly.

Our implementation was lifted directly from the official reference implementation for SSD. Additionally it is important to note that SSD is not tested on randomly forgetting 10% of CIFAR-10 and is limited to deleting 100 samples - which could potentially lead to subpar performance.

for MIA, what is the reported number ? Is it a fraction between 0 and 1 or a percentage between 0 and 100 (like the other columns). This number appears too low if its a percentage to be meaningful. Can the authors report AUC instead?

The reported number is a fraction between 0 and 1. We have included AUC in the 30% Forgetting experiments provided in Appendix A7.

We hope this response has addressed the reviewer’s concerns, and we would be glad to continue the discussion if there are any remaining questions or comments. We thank the reviewer again for their detailed and thoughtful input on our work.

We thank reviewer f5or for their thoughtful comments. We’re encouraged that this reviewer found our problem setting “interesting to consider,” and that they appreciated the breadth of our comparisons against baseline algorithms.

Below, we provide answers to their questions and remarks.

Theoretical Claims

The definition of recoverability says is that if $f$ is injective then all datasets $D$ are recoverable. This does not mean that if $f$ is not injective then no $D$ is recoverable. It may be possible that some $D$s are recoverable. The paper looks at exact recoverability which may be not only be unrealistic in practice but also an unnecessarily high bar. it is often impossible to exactly recover training data points but recover them to a high degree. As such, there should be a notion of approx. recovery.

These two points are well-taken: they represent limitations of our present theory and setting and we will clarify this in the manuscript. However, by establishing and motivating the setting of blind unlearning, our work lays the groundwork for settings with stronger guarantees on forget-set recoverability. We hope that future work can explore methods for blind unlearning with guaranteed non-recoverability, and our present work serves as a stepping stone toward these contributions. On balance, given that the vast majority of the unlearning literature requires $D_{forget}$, even a method that only considers exact recoverability represents a valuable step forward.

The theoretical intuition behind the algorithm is that the gradients on the forget set can be approximated by the difference of the gradients in the full dataset and the retain set. However, these are only true for the check point parameters save and it is difficult to argue (not done in the paper) why they should be similar after some gradient ascent steps are done.

We believe this represents a small misunderstanding. Reload requires only that the gradients from the last iteration of training be cached, and only employs them in a single ascent step (Figure 1; Step 3). The reviewer is correct that there is no reason to believe these gradients should be similar after multiple ascent steps, hence why Reload only performs one such weight update.

Diversity of experimental results and comparisons in known baselines

For unlearning, consider the IC test in Goel et. al. and the targeted, indiscriminate, and gaussian data poisoning attacks in Pawelczyk et. al.

The IC test in Goel et. al. deliberately mislabels some of the instances in $\mathcal{D}_{train}$, and then tests the ability of the unlearning procedure to unlearn mislabelled instances. Our paper includes remedial learning experiments in this setting using Reload in Appendix A.9. Furthermore, the targeted, indiscriminate, and gaussian data poisoning attacks in Pawelczyk et. al. is conceptually very similar to the backdoor correction experiments that feature in the main body (Section 4.3);. The key difference between the IC test and our setting in remedial learning is that while the IC test evaluates label misspecification, our experiments evaluate covariate misspecification. That Reload often outperforms baseline methods in both settings (and does so with significantly less computational resources) suggests that it fulfills the desired properties.

For remedial unlearning, it is easy enough to adapt the above test by using clean data points.

It is correct that fine-tuning on clean data points is a valid means of performing remedial unlearning. However, our results suggest that Reload is a more efficient algorithm for remedial unlearning. Our findings in Table 3 demonstrate that Reload achieves a significantly higher accuracy than fine-tuning on a held-out test split of $\mathcal{D}_{new}$

(86.87 vs. 90.81), and on a modified held-out test split of $\mathcal{D}_{new}$ with backdoors injected into each sample (86.50 vs. 90.51). More importantly, Reload does so at significantly lower cost – whereas fine-tuning requires 37% of the computational resources as from-scratch retraining, Reload requires only 8%.

The experiments only use ResNet18 and VGG on CIFAR10 and CIFAR100. These are not sufficient to make generalisable claims especially when the paper is empirical in nature. In general these two papers should be discussed as relevant existing work.

Additional experiments on SVHN are presented in the supplementary materials. More importantly, the experiments provided in the paper study a wide range of unlearning and remedial learning scenarios, ranging from unlearning randomly-selected samples, to unlearning correlated samples, to removing shortcuts. If there are specific experiments this reviewer would like to see, we are open to suggestions and we can work to implement