Unlearning via Sparse Representations

• Next, we label the losses corresponding to the validation and test set of the forget class as 1 and those corresponding to the validation and test set of the held out class as 0. We train a binary classifier as done in the "Basic Attack" setting in [6] on the validation losses of both sets and test it on the test losses. We follow a similar setting for the baseline model, where we obtain the model suitable for MIA defense by following a procedure similar to SCRUB+R as introduced in [6]. For a successful defense, we would want the accuracy of the classifier to be close to 50%, indicating that it is unable to distinguish between the unlearnt class and the held-out class. Same as Kurmanj [1], we use sklearn.logistic_regression as our attacker (the binary classifier). We call the approach described above Partial UvA (Partial Unlearning via Activations). We run experiments for 3 random seeds, and the mean of the attacker performance is reported.
• Observations and Results: We report the results of the experiment described above in the table given below. We observe that although the baseline performs better, the proposed approach performs competitively, even though we have not intended to develop the method for this scenario.

Would the method work if the pretrain dataset is different from the target one

In all of our experiments, the dataset used for pretraining the backbone is always distinct from the downstream task. For all the experiments in the paper, we use a ViT-B/32 backbone which has been pretained using CLIP (Radford et al., 2021). which uses web-scale Image-Text data for pretraining. In our new experiments with a ResNet50 backbone (see Appendix H) which has been pretrained in a supervised manner on ImageNet and the tasks used for downstream tasks are CIFAR-10, CIFAR-100 and LACUNA-100).

We thank the reviewer for taking the time go through our work and giving a detailed feedback as well as suggestions. We hope to answer the reviewers questions in the rebuttal that follows:

Regarding the evaluation of the approach at scale

We evaluate our approach on a larger scale by training the baseline and DKVB models on the ImageNet-1k and attempting to unlearn class number 1.

• Experimental Setup - We train the DKVB and baseline model (i.e. Frozen backbone + trainable linear layer) on ImageNet-1k up to 68% classification accuracy. ImageNet-1k consists of 1.28 million training examples and 50000 test examples divided into 100 classes. Next, we choose class 1 as the class to be unlearnt, and perform the same set of experiments as performed for the three datasets in Section 5.2 and Section 5.3 in the paper.

• Results - We present a comparative analysis between the baseline and the proposed approach on ImageNet-1k in the table given below. Further, we have also included scatter plots for the retain set accuracy vs forget set accuracy in Appendix I of the paper. We observe that our approach performs equally well on ImageNet as compared to the three datasets already included in the paper, with similar hyperparameters This demonstrates that the proposed approach is equally effective in large-scale settings as well. The given table presents a comparison between the proposed methods and the baselines on ImageNet-1k. We compare the change in performance on the retain and forget test data relative to the originally trained models.

Regarding Instance-Specific Unlearning

In this work, we focus on the setting of class unlearning. Instance-specific unlearning (or selective unlearning) would be non-trivial with the proposed approach and we leave investigations regarding that for future work.

Elaborating on Membership Inference Attacks (Appendix Section D)

First, we would like to clarify that our approach is not a-priori suited for selective unlearning, i.e. the setting where we want the model to forget specific examples or a small subset of examples instead of removing the information about an entire class. The KV bottleneck induces clusters of representation, where the members of a particular cluster correspond to the representations belonging to the same class. When we try to unlearn the representations corresponding to one particular example belonging to a particular class, the KV bottleneck routes the selection to other (key-)representations within the same cluster. Since these representations also contain information about the same class as the examples we intend to unlearn, the model would still predict the class to be unlearnt.

Due to the same reason our approach is also not designed for working against traditional Membership Inference attacks. According to the basic attacks setup as explained in their paper the objective is to obtain a model that has unlearnt a small subset of specific examples (i.e. selective unlearning) such that the loss of the model on the unlearnt subset of examples should be indistinguishable from examples that the model never saw during training.

Nevertheless, we attempt to modify the above setup to that of "Class Membership Inference Attacks (CMIA)". In CMIA, the aim is to defend against an attacker whose aim is to determine whether a model that has undergone unlearning ever saw a particular class as a part of its training data. Thus, we want the model to unlearn a particular class such that the losses/performance of the model on the unlearnt class is indistinguishable from a held-out class that the model never saw during its training. We describe the experimental setup and results below:

• Experimental Setup: We perform the experiment for CIFAR10. We divide the dataset into training data ($\mathcal{D_{Train}}$), validation data ($\mathcal{D_{Val}}$) and test data ($\mathcal{D_{Test}}$). Training Data consists of 4000 examples per class; validation and test data consist of 1000 examples per class. We first trained a model on the first 9 classes of CIFAR10. Here, class number 10 is the held-out class. Next, we unlearn class 1 from the model using the Unlearning via Activations approach introduced in the paper. We unlearn the model until the loss on the validation sets of the forget class and the held-out class are similar. In our experiments, we find that we reach this point at approximately $N_a = 240000$. The loss $l(x, y)$ in our case would be the cross-entropy loss.

We thank the reviewer for taking the time to go through our work. We attempt to address the points raised by the reviewer in this rebuttal:

Regarding Experiments around the computational cost of the approaches discussed

We measure the runtimes of the experiments discussed in Section 5.3 and Table 2 as a proxy to the computational costs required by the approaches discussed. The results are presented in the table below. We note that both the proposed approaches are several times faster than the baseline

We find that Unlearning via Activations is consistently the most compute efficient approach while both Unlearning via Activations and Unlearning via Examples being at least ~20x faster than the baseline

Regarding the approach being model agnostic

The reviewer correctly notes that the proposed approaches are conditioned on the presence of a Discrete Key Value Bottleneck. However, it is agnostic to the choice of the frozen backbone and the decoder architectures. This can be supported by evidence from our new experiments with a ResNet50 backbone (see Appendix Section H) which show that the proposed approach also works with backbones other than a CLIP pretrained ViT-B/32.

We thank the reviewer for the insightful comments and suggestions. We attempt address the concerns raised by the reviewer below:

Evaluating the approach on more benchmarks

We tested the proposed approach on ImageNet-1k dataset and the results are as follows:

• Experimental Setup - We train the DKVB and baseline model (i.e. Frozen backbone + trainable linear layer) on ImageNet-1k up to 68% classification accuracy. ImageNet-1k consists of 1.28 million training examples and 50000 test examples divided into 1000 classes. Next, we choose class 1 as the class to be unlearnt, and perform the same set of experiments as performed for the three datasets in Section 5.2 and Section 5.3 in the paper.
• Results - We present a comparative analysis between the baseline and the proposed approach on ImageNet-1k in the table given below. Further, we have also included scatter plots for the retain set accuracy vs forget set accuracy in Appendix I of the paper. We observe that our approach performs equally well on ImageNet as compared to the three datasets already included in the paper, with similar hyperparameters This demonstrates that the proposed approach is equally effective in large-scale settings as well. The table given below provides a comparison between the proposed methods and the baselines on ImageNet-1k. We compare the change in performance on the retain and forget test data relative to the originally trained models.

Complete Unlearning vs Unlearning until random accuracy

As discussed in the literature, unlearning completely vs random accuracy depends on the task at hand. For instance, in cases where we care about removing harmful biases from a model (e.g. remove a users data completely), complete unlearning is more desirable. In other scenarios such as Membership Inference Attacks (MIAs), the focus is on user privacy, it is desired that the model unlearns only to an extent such that the performance of the model on the unlearnt class is equivalent to that on a class that it has never seen. For the proposed approaches of Unlearning via Examples and Unlearning via Activations, this “extent of unlearning” can be controlled using the parameters $N_e$ and $N_a$ respectively as shown in Figures 2 and 3. Further, our new experiments of MIAs as discussed in Appendix D. Here, we show that the proposed approach is also useful in cases where complete unlearning is not desired.

Clarifications regarding Section 5.4

• Implications of using additional compute - The proposed approaches do not require additional compute for unlearning a class apart from the compute required for inference. The baseline, on the other hand, requires additional compute in the form of training the model and optimizing it w.r.t. a given objective. Thus, we attempt to investigate what would happen if the proposed approach also somehow made use of additional compute. In order to make use of some additional compute, once the DKVB models have undergone unlearning of the forget class, we train them on just the retain set in order to cover up any damage in performance on the retain set that might have occurred due to the unlearning.
• Use of the term “retraining” - We use the term “retraining” to refer to the above-explained process of training the unlearnt DKVB models on only the retain set in order to improve the performance on the retain set.

• Next, we label the losses corresponding to the validation and test set of the forget class as 1 and those corresponding to the validation and test set of the held out class as 0. We train a binary classifier as done in the "Basic Attack" setting in [6] on the validation losses of both sets and test it on the test losses. We follow a similar setting for the baseline model, where we obtain the model suitable for MIA defense by following a procedure similar to SCRUB+R as introduced in [6]. For a successful defense, we would want the accuracy of the classifier to be close to 50%, indicating that it is unable to distinguish between the unlearnt class and the held-out class. Same as Kurmanj [1], we use sklearn.logistic_regression as our attacker (the binary classifier). We call the approach described above Partial UvA (Partial Unlearning via Activations). We run experiments for 3 random seeds, and the mean of the attacker performance is reported.
• Observations and Results: We report the results of the experiment described above in the table given below. We observe that although the baseline performs better, the proposed approach performs competitively, even though we have not intended to develop the method for this scenario.

Regarding Limited Literature Review

We thank the reviewer for pointing us to these important papers. We have discussed these ([1], [3], [4], [5]) works as well as drawn a comparison of the proposed approach to [1] and [5] in the "Additional Related Works" section in the Appendix (section G).

[1] Jia, Jinghan, et al. "Model sparsification can simplify machine unlearning." arXiv preprint arXiv:2304.04934 (2023).
[2] Chundawat, Vikram S., et al. "Zero-shot machine unlearning." IEEE Transactions on Information Forensics and Security (2023).
[3] Warnecke, Alexander, et al. "Machine unlearning of features and labels." arXiv preprint arXiv:2108.11577 (2021).
[4] Chen, Min, et al. "Boundary Unlearning: Rapid Forgetting of Deep Networks via Shifting the Decision Boundary." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[5] Mehta, Ronak, et al. "Deep unlearning via randomized conditionally independent hessians." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022
[6] Kurmanji, Meghdad, Peter Triantafillou, and Eleni Triantafillor> "Towards Unbounded Machine Unlearning." arXiv preprint arXiv:2302.09880 (2023).

We thank the reviewer for their detailed and very helpful feedback on our paper. We hope to address your questions and suggestions in the following.

Regarding Model Diversity

As requested by the reviewer, we performed the same set of experiments presented in the paper in Sections 5.2 and 5.3 for CIFAR-10, with a backbone from the ResNet family.

• We use an ImageNet supervised pretained ResNet50 model as the backbone for our approach as well as the baseline model.
• The results are as presented in the table below. We have also attached the scatter plots for the retain set accuracy vs forget set accuracy in Appendix H of the paper. We observe that the results with ResNet backbone are fairly similar to the ones with ViT backbone, discounting the difference in absolute performances. Importantly, even in this scaled-up setting we are able to perform unlearning with minimal compute without damaging the performance on the retain set.
• We compare the change in performance on the retain and forget test data relative to the originally trained models. For the baseline, we report two cases: Case B where the model unlearns the forget set completely but the retain set performance is not preserved very well, and Case A where the retain set performance is not preserved very well, but the model does not unlearn the forget set completely

Regarding comparison of runtimes

We measured the runtimes of the experiments discussed in Section 5.3 and Table 2. The results are presented in the table below. We note that both the proposed approaches are several times faster than the baseline

Regarding more Unlearning Scenarios

First, we would like to clarify that our approach is not a-priori suited for selective unlearning, i.e. the setting where we want the model to forget specific examples or a small subset of examples instead of removing the information about an entire class. The KV bottleneck induces clusters of representation, where the members of a particular cluster correspond to the representations belonging to the same class. When we try to unlearn the representations corresponding to one particular example belonging to a particular class, the KV bottleneck routes the selection to other (key-)representations within the same cluster. Since these representations also contain information about the same class as the examples we intend to unlearn, the model would still predict the class to be unlearnt.

Due to the same reason our approach is also not designed for working against traditional Membership Inference attacks. According to the basic attacks setup as explained in Kurmanji et al., 2023, the objective is to obtain a model that has unlearnt a small subset of specific examples (i.e. selective unlearning) such that the loss of the model on the unlearnt subset of examples should be indistinguishable from examples that the model never saw during training.

Nevertheless, we attempt to modify the above setup to that of "Class Membership Inference Attacks (CMIA)". In CMIA, the aim is to defend against an attacker whose aim is to determine whether a model that has undergone unlearning ever saw a particular class as a part of its training data. Thus, we want the model to unlearn a particular class such that the losses/performance of the model on the unlearnt class is indistinguishable from a held-out class that the model never saw during its training. We describe the experimental setup and results below:

• Experimental Setup: We perform the experiment for CIFAR10. We divide the dataset into training data ($\mathcal{D_{Train}}$), validation data ($\mathcal{D_{Val}}$) and test data ($\mathcal{D_{Test}}$). Training Data consists of 4000 examples per class; validation and test data consist of 1000 examples per class. We first trained a model on the first 9 classes of CIFAR10. Here, class number 10 is the held-out class. Next, we unlearn class 1 from the model using the Unlearning via Activations approach introduced in the paper. We unlearn the model until the loss on the validation sets of the forget class and the held-out class are similar. In our experiments, we find that we reach this point at approximately $N_a = 240000$. The loss $l(x, y)$ in our case would be the cross-entropy loss.