SEMU: Singular Value Decomposition for Efficient Machine Unlearning

Referencing other works Thank you for highlighting these works. We will discuss the differences between SEMU and these methods and include this analysis in the camera-ready version of our work. Regarding [1], our method does not rely on samples or gradients from the remaining dataset, nor do we perform pseudo-labeling of the forget class to the most activated incorrect class for each unlearning sample. The work [2], published at AAAI25 after the ICML submission deadline, was not known to us at the time. We note that [2] operates on the representation of a trusted dataset, whereas SEMU focuses on gradients and does not require any additional datasets. In the case of [3], multiple datasets (forgetting and remaining) are used, but the authors employ Fisher and Hessian matrices to select important parameters, unlike SEMU, which uses SVD. Lastly, [4], similar to [2], performs unlearning by identifying important parameters in the representation space. It requires the identification of remaining and forget spaces using representations from both datasets. In summary, while various approaches use different projection methods, none operate without a remaining dataset. This discussion will be included in the revised version.

Quality metrics for nudity concept We provide the additional qualitative and quantitative evaluation for this in Sections 2, 3, and 4 in https://anonymous.4open.science/r/icml2025_submission_3162/REBUTTAL.md . In particular, we use MSE and CLIP measures to compare SalUn and SEMU to Stable Diffusion on NSFW (Tab. 1) and safe (Tab. 2) prompts. Especially, comparing the visual samples in Section 4 shows the quality difference.

Tab. 1

Tab. 2

We observe almost perfect sampling on safe prompts and have better MSE on NSFW prompts as well.

Remaining dataset usage SEMU is not limited to operating only in scenarios without a remaining dataset. One of its notable properties is its ability to function effectively in both conditions, with and without remaining datasets. Furthermore, we present results using the remaining dataset (always indicated as SEMU_{remain}). These results also indicate that the remaining dataset has minimal impact on SEMU's performance.

Performance lag We agree that there is a difference in performance. This difference is mostly caused because of a different approach, i.e., we want to have an unlearning method working reasonably well, even without access to any examples from the remaining dataset, which is really hard for the SalUn.

Finally, we observed that SalUn's concept of unlearning leads to catastrophic behaviour on normal (safe) prompts and conceptually far on NSFW as well. We provide the quantitative and qualitative results in Sections 2, 3, and 4 in additional experiments (https://anonymous.4open.science/r/icml2025_submission_3162/REBUTTAL.md).

Table 1 concerns When looking at Table 1, we notice that the unlearning accuracy (UA) is low, indicating that the model struggles to perform correctly on the forget dataset. This performance is comparable to other unlearning methods such as GA, IU, BS, BE, and FT. Regarding the results of the Membership Inference Attack (MIA), our findings show low recognition rates, suggesting that the attack does not identify the data as part of the model's training set. Based on these observations, we conclude that SEMU effectively performs the unlearning task.

Discussion on gradient projection Please see ablations in rebuttal for Reviewer 8DF6.

Application to convolution For a 2D convolutional layer, we treat each channel as a separate matrix (while restricting ourselves to the maximum 'sparse' value across all channels – see line 111 in the code here). Notice that we can flatten the kernel dimensions, and by performing matrix multiplication along the appropriate dimensions, we obtain a similar operation to a standard convolution.

Then, similar to the linear layer case, we compute $U W V^T$, but here $W$ corresponds to the channel (one dimension represents the flattened kernel). Since the tensors $U, W, V^T$ have more than two dimensions, we use the einsum function for computation. After the multiplication, we reshape the dimensions where we previously flattened (for the kernel) and finally permute the tensor to match the correct weight format for the convolutional layer.

All these operations are performed in just three lines of code (see lines 78-80 in this file).

Time analysis Please see the response for Reviewer Qmc3.

We appreciate your feedback. Below we address the concerns.

Previous works R1 and R2 Regarding R1, we would like to highlight that it requires an additional surrogate dataset $\mathcal{D}^{sur}$, which is not required for SEMU. This means our approach does not rely on extra datasets to maintain the neural network's capabilities.

As for R2, we were not aware of this work since it has not yet been published in a peer-reviewed venue, but with only an arXiv version available. The authors of R2 propose a different method for machine unlearning, focusing on altering the entire model rather than selecting crucial weights. We will include this discussion in the camera-ready.

In terms of comparison, we believe it is not feasible to compare SEMU with R2 due to its unpublished status and the unavailability of the code. However, we will include a comparison with R1 in our revised version.

Intuition behind SEMU In practice, some directions are more important than others for all weights. Observe that the weights are roughly proportional to the averaged gradient over the entire dataset. However, if we consider only the subset (class) we want to unlearn, its gradient will share some directions with the gradient of the whole dataset, but will also have directions specific to that subset. Thus, the projection ensures that we remove the common directions from both the weights and the gradient of our subset. Consequently, during the unlearning process, we do not modify the directions crucial to the model but only those specific to the dataset.

Similarity to SalUn We believe SalUn is the most similar to SEMU, as both methods aim to alter only crucial model weights based on gradient information. However, our approach significantly reduces the number of altered weights by up to 50 times. Additionally, SEMU uses the same loss functions when processing the forget dataset and can operate without a remaining dataset, addressing a limitation of SalUn. When SalUn operates only on the forgetting dataset, its performance diminishes as the model collapses (see response for Reviewer Qmc3).

Loss functions for SEMU The training process and loss function of SEMU differ slightly, as SEMU does not rely on the remaining dataset. Therefore, the objective for the remaining set is not used in model optimization.

Metric definition When it comes to metrics, we use the well-established ones in literature. When it comes to MIA, we use an MIA defined in [1], which was also used in SalUn and previous works.

[1] Carlini, Nicholas, et al. "Membership inference attacks from first principles." 2022 SP.

Accuracy for generation task To measure the accuracy of the image generation task, we use a classifier trained to recognize images generated by the model for a given class. We then apply this classifier to a newly generated batch of images after unlearning.

Time consumption Please see the response for Reviewer Qmc3

MIA and UA worse than SalUn We agree that the results presented in Tab. 6 and Tab. 7 are comparable or slightly worse than SalUn, especially in terms of the FID metric. This difference is mostly caused because of a different approach, i.e., we want to have an unlearning method working reasonably well, even without access to any examples from the remaining dataset, while SalUn utilizes the remaining dataset during unlearning.

Finally, we observed that SalUn's concept of unlearning leads to catastrophic behaviour on normal (safe) prompts and conceptually far on NSFW as well. We provide the quantitative and qualitative results in Sections 2, 3, and 4 in additional experiments (https://anonymous.4open.science/r/icml2025_submission_3162/REBUTTAL.md).

Test set construction The test set consists of test images from the dataset used for evaluation. In the case of random data forgetting, the test set remains unaltered. However, when performing class-wise forgetting, we remove the forgotten class from the test.

Forgetting labels When forgetting, we perform random relabelling, meaning that we assign a random label from the remaining classes.

Additional benchmarks See responses to Reviewers iRuu and Qmc3. We believe these experiments showcase SEMU's effectiveness and the results cover a broad range of task scales, capabilities, and complexities, while the proposed benchmarks are similar.

More samples for DDPM Unlearning samples of other CIFAR10 classes generated with DDPM will be added to the revised version. Now, we present them at https://anonymous.4open.science/r/icml2025_submission_3162/REBUTTAL.md in Section 5.

Why remaining-free? The dataset-free scenario can be justified for similar reasons as exemplar-free continual learning. Namely, due to privacy reasons (e.g. GDPR, CCPA), the remaining data may be inaccessible during the unlearning or too large to efficiently retrain on. Also, computational efficiency is improved by avoiding training episodes on the remaining data.

Time consumption In the Table below we show a comparison of time needed to unlearn DDPM model:

Also, we show the time of unlearning of ResNet-18:

Bolded values in the tables highlight two key metrics: target accuracy, reflecting the similarity between a model’s performance and its retrained version, and the number of parameters modified during unlearning. This emphasizes that SEMU minimally affects model behavior compared to other methods. For generative models, we evaluate stable diffusion outputs before and after unlearning using a safe prompt, identical seed, and noise. Notably, we observe that SalUn exhibits drastic changes in model behavior (not ability), which are not evident with SEMU.

Ablations Here we provide ablations on the projection usage in SEMU:

One can observe that the projection positively influences the unlearning process by preserving model capabilities what can be seen in RA and TA metrics, while slightly widening the gap between retrain model and unlearned one in FA and MIA.

Another ablation on the parameter choosing the portion of variance explained by the SVD that is used for selecting parameters for alteration is in the rebuttal for Reviewer iRuu.

Comparison with LoRA Thank you for your feedback. We are open to comparing SEMU with the LoRA method for machine unlearning. Could you please direct us to a specific LoRA-based machine unlearning method with available code? This will enable us to conduct a fair comparison. If such a method does not exist, we believe that adapting LoRA for machine unlearning is beyond the scope of our current work, as it would require significant effort.

Theoretical claims Thank you for noticing the concern with the proof. We know that the Eckart-Young-Mirsky theorem (matrix approximation lemma) is true for any unitarily invariant norm. In particular, it is true for the Frobenius norm too. Throughout the paper, while introducing SVD formally, we assume the Frobenius norm and operate in a Hilbert space. Since each element of $S^r$ is of rank $r$ at most, we know that among all $r$-rank matrix approximations for $G$, the optimal one is the one given by the SVD. Moreover, we know that the optimal solution is unique. In the revised version of this manuscript, we will add the Eckart-Young-Mirsky theorem and all the needed assumptions in the same place, in order to make the proof more approachable.

100% of TParams for SalUn in Supplement We are grateful to the Reviewer for catching the typo. It was an oversight due to a copy-paste error. We apologize for the mistake. In the camera-ready version of the paper, we will correct the values in Tables 2 and 3 for SalUn (50% instead of 100%).

Table 4 and Table 5 are only for SalUn In our evaluation of SalUn, we aimed to demonstrate that when fewer parameters are altered than originally described, SalUn's effectiveness diminishes, regardless of the data. To achieve a fair comparison, we conducted an experiment using the same setup but altering only 1% of the weights with SalUn. The results can be found in a rebuttal for Reviewer Qmc3.

Numerical evaluation for the generation task We thank the Reviewer for a question regarding their concern. We perform a similar analysis of the generative diffusion models behavior as for classification tasks. That’s the rationale behind reporting TA as well. Following the SalUn evaluation procedure, firstly, we pretrained the same classifier to analyse the generated samples. We observed that even a small change in generated features is hard for such a classifier. Following the Reviewer's concern, we want to admit that the TA metric for unlearning in the generative models scenario is a biased metric. For the next comparisons (e.g., the ones from Section 1 in https://anonymous.4open.science/r/icml2025_submission_3162/REBUTTAL.md), we focused on FID and UA metrics.

Template usage Thank you very much for spotting this issue. We are very sorry for the mistake and any difficulties this could cause.

Typos We would like to thank the Reviewers for their thorough work in helping us improve our manuscript. We apologize for typos.

Why SVD? Our objective was to minimize the number of altered weights during the unlearning process to maintain the model's behavior. To achieve this, we looked for an effective selection mechanism. SVD, our first choice, proved to be successful, so we did not explore other parameter selection methods. However, investigating alternative selection methods for potential improvements in effectiveness and efficiency is an interesting path for future work.

Experiments on TinyImageNet with ViT and ResNet To futher show applicability of SEMU, we provide results on TinyImageNet for ResNet18 and ViT models.

Performance on ResNet-18, pre-trained on Tiny ImageNet dataset, for 10% random data forgetting.

Performance on ResNet-18, pre-trained on Tiny ImageNet dataset, for 1 random class (number 9) data forgetting.

Performance on ViT, pre-trained on Tiny ImageNet dataset, for 10% random data forgetting.

Performance on ViT, pre-trained on Tiny ImageNet dataset, for 1 random class (number 9) data forgetting.

Thank you for your thorough review. We would like to address some of your concerns below:

Time needed for SEMU when compared to SalUn In the Table below we show a comparison of time needed to unlearn DDPM model:

Also, we show the time of SEMU and SalUn in unlearning of CIFAR10 and CIFAR100 for ResNet-18:

When it comes to memory usage, SEMU does not require additional storage, so it requires the same amount of memory as SalUn, or slightly less, as we do not require a mask of neurons which needs to be altered.

SEMU for large architectures In this experiment, we demonstrate the effectiveness of SEMU using the TinyImageNet dataset and the ViT model. We tested SEMU's ability to forget 10% of randomly chosen data and one entire class. The results indicate that SEMU maintains strong performance with the ViT model in both scenarios. Regarding the application of SEMU to Large Language Models (LLMs), it's important to note that our focus has been on models designed for computer vision. Therefore, adapting SEMU to LLMs may be beyond the scope of this work.

Referring to other low-rank adaptation methods. We will update the discussion on low-rank adaptation methods in the final version of our work, which will include PUMA.

Different SVD truncation levels influence on model's performance Parameter $r$ gives the size of a submatrix of the SVD projection, which we use for unlearning purposes. In particular, in each changed layer $L$, we are setting the value of $r_L$ to be the same percentage alpha of the rank of this matrix.

This procedure gives us the square submatrices of sizes $r_L \times r_L$. Please note that the value of $r_L$ is different for various layers, however, in each layer, we have the same percentage (alpha) of important directions in the SVD projection. We consider SEMU_retrain (Tab. 1) and SEMU_subset (Tab. 2) scenarios.

Tab. 1

Tab. 2

We observe that the proposed method of selecting the most important directions is more efficient than selecting just a percentage of the low-rank projection matrix (better UA and FID). For naive selection, we observe that the best results are for alpha=0.01. Then, the metric values are higher, but not linearly.

Most important parameters for SEMU The effectiveness of SEMU is influenced by several factors, due to the fine-tuning of crucial model parameters. From the standpoint of the machine unlearning method, the parameter $\gamma$ is important. It is responsible for the selection of weights that are modified during the unlearning process. Specifically, $\gamma$ selects weights with an SVD variance no less than its value (e.g. 90% of variance), enabling the identification of a critical subset of weights for alteration. For further details, please refer to the SEMU section and Eq. 12.

To address the concerns and questions raised by the Reviewer, we would like to point out the following:

More on theoretical aspects of projection. In practice, some directions are more important than others for all weights. Observe that the weights are roughly proportional to the averaged gradient over the entire dataset. However, if we consider only the subset (class) we want to unlearn, its gradient will share some directions with the gradient of the whole dataset, but will also have directions specific to that subset. Thus, the projection ensures that we remove the common directions from both the weights and the gradient of our subset. Consequently, during the unlearning process, we do not modify the directions crucial to the model but only those specific to the dataset. We thank the Reviewer for a related paper. We will work more on theoretical guarantees.

Performance on larger dataset. We have run experiments on TinyImageNet and here are the results:

Performance on ResNet-18, pre-trained on Tiny ImageNet dataset, for 10% random data forgetting.

Performance on ResNet-18, pre-trained on Tiny ImageNet dataset, for 1 random class (number 9) data forgetting.

One can observe that in both experiments, SEMU achieves the lowest gap in target accuracy of the model.

Time needed for SEMU Comparison of the time needed for SEMU and SalUn for the DDPM model. As can be observed, SEMU requires much less time to perform unlearning.

Here are also the results for unlearning for the ResNet18 model:

When it comes to memory usage, SEMU does not require additional storage, so it requires the same amount of memory as SalUn, or slightly less, as we do not require a mask of neurons which needs to be altered.

Comparison to SalUn within similar experimental conditions Additionally, we ran experiments with SalUn, showcasing that SalUn collapses when the remaining dataset is not used during the unlearning procedure, no matter the amount of parameters altered (1% and 100%).

Influence of projection on results

One can observe that the projection positively influences the unlearning process by preserving model capabilities, which can be seen in RA and TA metrics, while slightly widening the gap between retrain model and the unlearned one in FA and MIA.

We are grateful for the review, and we are looking forward to a fruitful discussion on the provided answers.