Structure-Aware Parameter-Efficient Machine Unlearning on Transformer Models

We thank the reviewer for the positive rating of our paper. We also appreciate the reviewer for acknowledging the novelty of this work and all the constructive suggestions. We hope the following clarifications can address the reviewer's concerns.

Some experiments are insufficient, and the experimental setup should be more comprehensive.

• For example, in page 7 the author notes that if the number of unlearning requests exceeds a certain threshold, the model must be retrained from scratch to regain its performance. What exactly is this threshold?
• Additionally, in Table 3, is there a specific reason for selecting 90% sparsity instead of alternatives like 85% or 95%? Based on Figures 2 and 3, is this value specific to the model?

Thank you for these insightful suggestions. For Question 1, the threshold for unlearning varies across different models, and there is no definitive standard. It can be determined based on specific practical requirements and scenarios. For Question 2, we investigated the sparsity for DistilBERT, BERT-base, and RoBERTa-large. We found that 90% represents a reasonable balance, achieving effective unlearning with minimal cost to model fidelity. While 85% is also feasible, we observed a significant drop in the model's utility when meeting the unlearning criteria at 95%. Additionally, sparsity may depend not only on the model itself but also on the specific task and dataset involved. Furthermore, we have conducted further experiments, including testing the LLaMA2-7B model on the TOFU benchmark, and incorporated several state-of-the-art methods.

I am interested in the comparison results with other unlearning approaches presented in Table 1. Could you either provide the results of applying the memory-aided unlearning approach to the experiments in Table 1, or explain why this comparison was not included?

We apologize for the confusion. In the context of successive unlearning, we have divided second-order unlearning (SO) into two types: memory-aided unlearning and memory-free unlearning. Our focus here is primarily on second-order unlearning. Therefore, we only provided an analysis of how full updates in SO impact memory-free unlearning. Moreover, the comparison between memory-free and memory-aided approaches is illustrated in Figure 4.

In the experiments on successive unlearning scenarios on page 9, the experiments shows the number of requests affect the results. How does the Volume of data affect the performance?

We appreciate this insightful suggestion to explore the relationship between the volume of unlearning data and model utility. In our current work, we have conducted only a qualitative analysis, demonstrating that increasing the amount of unlearning data leads to greater performance degradation. In future work, we will plan to investigate this relationship quantitatively.

Thank you very much again for your initial comments. They are very valuable for improving our work. We would be grateful if you could have a look at our response and modifications and please let us know if there is anything else that can be added to our next version. We are also willing to have further discussions with you.

Dear Reviewer Lm6c,

We greatly appreciate your initial comments. We totally understand that you may be extremely busy at this time. But we still hope that you could have a quick look at our responses to your concerns. We appreciate any feedback you could give us. We also hope that you could kindly update the rating if your questions have been addressed. We are also happy to answer any additional questions before the rebuttal ends.

Best Regards,

Paper 14252 Author(s)

Dear Reviewer Lm6c:

Thank you for your time and effort in evaluating our work. We greatly appreciate your initial comments. Your insights and suggestions are extremely valuable to us.

Given that we have only one day left for discussion, we are hoping to receive any additional feedback or question you might have at your earliest convenience. Your expertise would be of great help to us in improving the quality and rigor of our work.

To facilitate the discussion, we would like to summarize our response as follows.

• We discussed the details of our experiments, including how to choose the threshold for memory-free unlearning, why we selected 90% sparsity for our experiments, and how the volume of data affects performance.

If our responses address your concerns, we kindly request that you reconsider your evaluations. We would also be grateful for any additional comments or suggestions you might have to refine our work.

Best regards,

Paper 14252 Author(s)

Thank you for the constructive feedback, which has helped us improve the quality of our paper. We explain our work below:

Unlearning can be broadly divided into two components: the design of the objective function and the development of the optimization algorithm. For GA, the objective function focuses solely on maximizing the loss for the unlearning data. In contrast, GD's objective function not only aims to maximize the loss on the unlearning data but also strives to maintain the performance on the remaining data. NPO targets maximizing the discrepancy in loss on the unlearning data between the original and unlearned models, while DPO minimizes the loss associated with the substitute response "I don't know" for the forget data. Fundamentally, GD, NPO and DPO all work to increase the loss on forget data, essentially serving as optimization strategies for gradient ascent.

Our work primarily focuses on optimizing the design of algorithms to accelerate the process under a given objective function. The current manuscript specifically addresses second-order update methods, utilizing a second-order objective function aimed at minimizing the loss on the remaining dataset. In principle, alternative objective functions can also be employed, enabling the development of corresponding structure importance evaluation functions.The importance function for maximizing the loss on the forgetting dataset can be expressed as follows:

To this end, we have incorporated additional experimental scenarios by applying GA, GD, NPO, DPO and other methods on the LLama2-7B model and with TOFU benchmark, utilizing various structure importance evaluation functions. The following are the experimental results for TOFU Forget05 (due to content limitations, we include it in the next part).

We uniformly applied a sparsity of 90% across these methods. MLF refers to a structure importance evaluation function derived from the GA objective function (i.e. Maximize the Loss on the Forgetting dataset). Additionally, SPE represents a structure importance evaluation function designed using the original objective function in second-order unlearning. Finally, NORM utilizes the gradient norms of the forget data to determine structure importance.

Our experimental results indicate that sparse updates are better suited for unlearning than full updates, as they provide a stronger guarantee of unlearning while more effectively preserving performance. Additionally, while NORM significantly reduces computation time, SPE offers a more balanced trade-off between unlearning efficacy and model fidelity. Furthermore, structure evaluation functions with MLF also exhibit strong performance.

We hope these explanation and additions to our manuscript address the concerns raised and merit a reconsideration of the score by the reviewer. In future work, we plan to include additional experiments, such as the WMDP benchmark, to further demonstrate the effectiveness of our approach.

Thank you very much again for your initial comments. They are very valuable for improving our work. We would be grateful if you could have a look at our response and modifications and please let us know if there is anything else that can be added to our next version. We are also willing to have further discussions with you.

Dear Reviewer fgpp,

We greatly appreciate your initial comments. We totally understand that you may be extremely busy at this time. But we still hope that you could have a quick look at our responses to your concerns. We appreciate any feedback you could give us. We also hope that you could kindly update the rating if your questions have been addressed. We are also happy to answer any additional questions before the rebuttal ends.

Best Regards,

Paper 14252 Author(s)

Dear Reviewer fgpp:

Thank you for your time and effort in evaluating our work. We greatly appreciate your initial comments. Your insights and suggestions are extremely valuable to us.

Given that we have only one day left for discussion, we are hoping to receive any additional feedback or question you might have at your earliest convenience. Your expertise would be of great help to us in improving the quality and rigor of our work.

To facilitate the discussion, we would like to summarize our response as follows.

• We clarified the principles of our method, including the rationale for using the objective function in our manuscript for second-order unlearning updates.
• We discussed the details of our experiments, such as deriving an alternative objective function to identify saliency, adding new experiments with the TOFU benchmark on the LLama2-7B model, and applying state-of-the-art unlearning methods.

If our responses address your concerns, we kindly request that you reconsider your evaluations. We would also be grateful for any additional comments or suggestions you might have to refine our work.

Best regards,

Paper 14252 Author(s)

We thank the reviewer for their constructive and valuable feedback which improves our work. Please see our detailed response below:

Doubts regarding the necessity of the proposed parameter identification approach. The authors derive a score function to identify informative parameters, involving calculating the derivative with respect to $m$ over $D_f$ and $D_r$. Why not directly optimize the mask parameters base on Eq.(4), which seems computationally cheaper and simpler with only once gradient calculation? Or why we need to transform Eq.(4) to Eq.(5)?

We agree with your suggestion. Previous study [1] has indeed employed gradient-based methods to facilitate machine unlearning. However, such approaches are heuristic and lack a solid theoretical foundation. Our contribution lies in deriving a principled method for structure evaluation through Taylor expansion, which provides a theoretical basis for the transformation from Eq.(4) to Eq.(5).

[1] Ma Z, Liu Y, Liu X, et al. Learn to forget: Machine unlearning via neuron masking[J]. IEEE Transactions on Dependable and Secure Computing, 2022, 20(4): 3194-3207.

Doubts about the claimed efficiency benefits.

Indeed, while our method introduces additional computations related to the structures within Transformers, the number of structures is significantly smaller than the number of parameters in Transformers, making the storage requirement for the masks minimal. Furthermore, utilizing structural masks reduces the storage and computation of gradients for the associated parameters, representing a deliberate and reasonable trade-off.

What are the specific advantages of the proposed method for the successive machine unlearning? The descriptions (e.g., in Line 362) is too broad and hard to understand. Detailed, clear and reasonable analysis is needed.

We acknowledge that the original manuscript did not clearly articulate the advantages of our method, and we will address this in a future revision. In brief, for second-order optimization, our approach offers distinct benefits: in memory-aided unlearning, it accommodates a greater number of unlearning requests without significant performance degradation; in memory-free unlearning, a single-step second-order update is applied to the original model, whereas the proposed method achieves a tighter approximation error bound, thereby preserving better performance.

Why do we need a layer-wise optimization (Eq.(9)) in addition to the individual optimization (Eq.(8)) during the mask selection process?

In our approach, we initially approximate the Hessian matrix using a diagonal Fisher Information Matrix (FIM), which neglects the influence of the off-diagonal elements. To address this limitation, we employ a block-diagonal FIM during the mask selection process to better capture the interactions between different structures within the same layer.

How is the first item in Eq.(13) derived? It is hard to follow the method description in Sec.3.3.2.

We apologize for the confusion. Eq.(13) is derived from the study [2]. Upon receiving an unlearning request, the model provider can utilize the data stored in memory (previously unlearned data, represented by the first term) to perform unlearning on the original model.

[2] Liu J, Xue M, Lou J, et al. Muter: Machine unlearning on adversarially trained models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4892-4902.

We hope these responses and further additions to our manuscript address the concerns raised and merit a reconsideration of the score by the reviewer.

Thank you very much again for your initial comments. They are very valuable for improving our work. We would be grateful if you could have a look at our response and modifications and please let us know if there is anything else that can be added to our next version. We are also willing to have further discussions with you.

Dear Reviewer ZE8b,

We greatly appreciate your initial comments. We totally understand that you may be extremely busy at this time. But we still hope that you could have a quick look at our responses to your concerns. We appreciate any feedback you could give us. We also hope that you could kindly update the rating if your questions have been addressed. We are also happy to answer any additional questions before the rebuttal ends.

Best Regards,

Paper 14252 Author(s)

Dear Reviewer ZE8b:

Thank you for your time and effort in evaluating our work. We greatly appreciate your initial comments. Your insights and suggestions are extremely valuable to us.

Given that we have only one day left for discussion, we are hoping to receive any additional feedback or question you might have at your earliest convenience. Your expertise would be of great help to us in improving the quality and rigor of our work.

To facilitate the discussion, we would like to summarize our response as follows.

• We clarified the details of our method, including the rationale behind transforming Eq. (4) to Eq. (5), the necessity of layer-wise optimization, and the derivation of the first term in Eq. (13).
• We discussed the advantages of our method, highlighting its efficiency benefits and its ability to perform unlearning under two successive settings.

If our responses address your concerns, we kindly request that you reconsider your evaluations. We would also be grateful for any additional comments or suggestions you might have to refine our work.

Best regards,

Paper 14252 Author(s)

We sincerely thank the reviewer for their insightful comments. We are encouraged by their recognition of our work's importance and its practical simplicity. Below, we address the comments and questions by the reviewer:

It is unclear why the method is considered structure-aware. While mask variables are applied to each head in the multi-head attention block, this approach is not unique to Transformers. The authors mention as "widely-adopted unlearning methods in Transformers, e.g., fine-tuning (Golatkar et al. (2020)) and gradient difference (Liu et al. (2022); Jia et al. (2024))," but they are not specified to transformer architecture. As I understand from the manuscript, the authors suggest that their method’s efficiency makes it well-suited to large-scale Transformer models, but large scale is not a Transformer-specific attribute.

We agree with your comments. Our method is not limited to Transformers, and large scale is indeed not an attribute unique to this architecture. However, Transformers are frequently deployed in large-scale environments where unlearning becomes computationally challenging. Our approach leverages learnable masks to identify specific structures within Transformers (i.e., attention heads and filters), simplifying unlearning methods by selectively focusing on the most critical components.

In line 190, the authors assume that " $L$ is differentiable with respect to $m$", to develop Equations (5-9). However, as stated in line 160, $m$ is a binary variable. This raises concerns about whether the differentiability assumption is valid.

Our method uses a pair of masks to denote attention heads and filters, which are integrated into the model's forward inference process. This integration affects the computation graph during backpropagation, allowing gradients with respect to the masks to be computed. Furthermore, existing study [1] has demonstrated the feasibility of such differentiability assumptions.

[1] Kwon W, Kim S, Mahoney M W, et al. A fast post-training pruning framework for transformers[J]. Advances in Neural Information Processing Systems, 2022, 35: 24101-24116.

In line 181, the authors state, "we formulate the unlearning objective (1) with a learnable pair of masks for the heads and filters as a constrained optimization problem." However, objective (1) aims to retain a model with the same architecture trained only on D_r. The proposed method, which involves pruning, alters the architecture, making it misaligned with this objective.

Our method incorporates a structure evaluation process but does not rely on pruning to simplify the model. Instead, it employs sparse updates to facilitate the unlearning process, which does not alter the architecture.

The experimental section lacks details on the partitioning of $D_r$ and $D_f$. This split is an essential part of the experimental setup and should be clearly specified.

We appreciate your suggestion regarding the partitioning of $D_r$ and $D_f$. We will incorporate this information in the revised manuscript to provide a clearer and more transparent experimental settings.

Evaluation Concerns in Table 2 and Appendix

Thank you for these valuable suggestions. First, we have clarified the gold standard for unlearning methods in Section 4.1 of the manuscript. We agree that aligning the performance on the forgetting dataset with the golden standard is reasonable. Next, we will include the remaining accuracy in Table 1 to enable a fairer comparison. Additionally, we will provide detailed hyperparameter settings in the Appendix. Lastly, we have incorporated new methods and experiments into our revised work, such as NPO [2] and DPO [3] on TOFU [4] using LLama2-7B.

[2] Zhang R, Lin L, Bai Y, et al. Negative preference optimization: From catastrophic collapse to effective unlearning[J]. arXiv preprint arXiv:2404.05868, 2024.

[3] Rafailov R, Sharma A, Mitchell E, et al. Direct preference optimization: Your language model is secretly a reward model[J]. Advances in Neural Information Processing Systems, 2024, 36.

[4] Maini P, Feng Z, Schwarzschild A, et al. Tofu: A task of fictitious unlearning for llms[J]. arXiv preprint arXiv:2401.06121, 2024.

We hope our responses have adequately addressed your previous concerns. We look forward to hearing from you and would be happy to address any remaining concerns that you may still have.

Due to space limitations, only a subset of the experimental results is presented. Our method is integrated into the current state-of-the-art unlearning approaches for large models, enabling sparse model updates.

As I understand, the authors can upload a revised draft at this time. Then, I'm confused about the review policy for addressing posted comments without submitting a new draft.