Rebuttal Part 1

We sincerely appreciate Reviewer Gp18 for providing a detailed summary of our strengths. We also greatly thank Reviewer Gp18 for proposing these insightful questions. We uploaded supplementary materials, which separate the newly added context for your convenience. Below, we provide our responses to the comments, denoted by [W] for weaknesses and [Q] for questions.

Response to W1 and Q1:
We greatly appreciate the Reviewer's insightful comment. In response, we have enhanced our manuscript by adding a detailed discussion of the scenario and threat model. Specifically, we frame our approach within the context of a continual learning scenario, where incremental learning is performed frequently and systematically. This setting is particularly relevant and popular, as users continuously generate vast amounts of new data daily, leading to the need for timely model updates.
Additionally, we emphasize that our method is designed to prevent the server from identifying the unlearned data or detecting unlearning intentions. To achieve this, the server only receives the updated data containing unlearning noise for model updates, making it challenging for the server to reconstruct the original unlearning samples.
We present the detailed scenario and threat model as follows.

Machine Unlearning Service Scenarios. To facilitate understanding, we introduce the problem in a Machine Learning as a Service (MLaaS) scenario. In the MLaaS setting, there are two key entities: a ML server that trains models as ML services, and users (data owners) who contribute their data for ML model training. In such scenarios, machine unlearning occurs when users realize that some of their previously contributed samples are private and wish to revoke these data contributions from the trained models.

The ML Server's Ability. We assume the ML server is honest but curious [R1]: while it honestly hosts and provides ML services, including model training and updating, it may still be curious about private information, such as unlearned data and unlearning intentions, if there are other operations. Informing the server of unlearning intentions to customize unlearning operations is considered a privacy threat because it reveals users' unlearning purposes, potentially enabling the server to prepare targeted unlearning attacks [R1,R2]. Therefore, in our setting, we assume the ML server has only the learning algorithm $\mathcal{A}$ and the model with parameters $\theta$ to meet strict privacy requirements. The ML server will not conduct unlearning operations other than training the model using the learning algorithm $\mathcal{A}$ for model updating.

Moreover, we assume the ML server does not store the original training data and cannot access the erased data, which should not be exposed to the server again due to privacy concerns. This assumption is reasonable in both real-world and privacy-preserving MLaaS scenarios. In real-world applications, vast amounts of data are generated daily, leading to the need for prompt model updates. Consequently, many models are trained using incremental or continual learning techniques [R3,R4]. Therefore, the server does not retain the entire raw data due to its large size [R5,R6]. In privacy-preserving scenarios, the ML server is restricted from directly accessing private training data from users due to privacy concerns [R7,R8].

The Users' Ability. The training data $D$ was collected from all users and was used to train the model $\theta_o$. The unlearning user has the erased data $D_u \subset D$. To estimate the unlearning update as the target for unlearning noise generation in our method, we assume the unlearning user can access the trained model $\theta_o$, which is a common setting even in many privacy-preserving scenarios such as FL. We assume the unlearning user has auxiliary clean examples $D_a$ so that they can synthesize a new dataset based on it with the unlearning noise, replacing the erased data $D_u$ for achieving the unlearning effect with only incremental learning using the synthesized dataset.

References [R1-R8] are provided at the end of the revised manuscript and the first comment.

Rebuttal Part 1

We sincerely appreciate Reviewer Ha5f for providing a detailed summary of our strengths. We also greatly thank Reviewer Ha5f for proposing these insightful questions. We worked diligently to address all the concerns and supplemented the discussion about the threat model, which is provided in the revised paper. We also uploaded supplementary materials, which separate the newly added context for your convenience. Below, we provide our responses to the comments, denoted by [W] for weaknesses and [Q] for questions.

Response to W1 and Q1: We greatly appreciate the Reviewer's insightful comment. To address the concern raised, in our method, the estimation of the unlearning influence for synthesizing the auxiliary data requires access to the trained original model, denoted as $\theta_o$, unlearned samples $D_u$, and some auxiliary clean examples $D_a$. We have supplemented our discussion on the threat model as follows.

Machine Unlearning Service Scenarios. To facilitate understanding, we introduce the problem in a Machine Learning as a Service (MLaaS) scenario. In the MLaaS setting, there are two key entities: a ML server that trains models as ML services, and users (data owners) who contribute their data for ML model training. In such scenarios, machine unlearning occurs when users realize that some of their previously contributed samples are private and wish to revoke these data contributions from the trained models.

The ML Server's Ability. We assume the ML server is honest but curious [R1]: while it honestly hosts and provides ML services, including model training and updating, it may still be curious about private information, such as unlearned data and unlearning intentions, if there are other operations. Informing the server of unlearning intentions to customize unlearning operations is considered a privacy threat because it reveals users' unlearning purposes, potentially enabling the server to prepare targeted unlearning attacks [R1,R2]. Therefore, in our setting, we assume the ML server has only the learning algorithm $\mathcal{A}$ and the model with parameters $\theta$ to meet strict privacy requirements. The ML server will not conduct unlearning operations other than training the model using the learning algorithm $\mathcal{A}$ for model updating.

Moreover, we assume the ML server does not store the original training data and cannot access the erased data, which should not be exposed to the server again due to privacy concerns. This assumption is reasonable in both real-world and privacy-preserving MLaaS scenarios. In real-world applications, vast amounts of data are generated daily, leading to the need for prompt model updates. Consequently, many models are trained using incremental or continual learning techniques [R3,R4]. Therefore, the server does not retain the entire raw data due to its large size [R5,R6]. In privacy-preserving scenarios, the ML server is restricted from directly accessing private training data from users due to privacy concerns [R7,R8].

The Users' Ability. The training data $D$ was collected from all users and was used to train the model $\theta_o$. The unlearning user has the erased data $D_u \subset D$. To estimate the unlearning update as the target for unlearning noise generation in our method, we assume the unlearning user can access the trained model $\theta_o$, which is a common setting even in many privacy-preserving scenarios such as FL. We assume the unlearning user has auxiliary clean examples $D_a$ so that they can synthesize a new dataset based on it with the unlearning noise, replacing the erased data $D_u$ for achieving the unlearning effect with only incremental learning using the synthesized dataset.

References [R1-R8] are provided at the end of the revised manuscript and the first comment.

Rebuttal Part 1

We thank you very much for the very insightful comment. We revised the paper according to your suggestions and provided supplementary materials, which separate the newly added context for your convenience. Below, we provide our point-to-point responses to the comments, denoted by [W] for weaknesses and [Q] for questions.

Response to W1: We sincerely appreciate the Reviewer's comment. We supplemented the scenario of our settings. Since our method is based on incremental learning, the continual learning scenario is reasonable, where the users will have new data uploaded to the server for the model's update. We present the detailed scenario discussion as follows.

Machine Unlearning Service Scenarios. To facilitate understanding, we introduce the problem in a Machine Learning as a Service (MLaaS) scenario. In the MLaaS setting, there are two key entities: a ML server that trains models as ML services, and users (data owners) who contribute their data for ML model training. In such scenarios, machine unlearning occurs when users realize that some of their previously contributed samples are private and wish to revoke these data contributions from the trained models.

The ML Server's Ability. We assume the ML server is honest but curious [R1]: while it honestly hosts and provides ML services, including model training and updating, it may still be curious about private information, such as unlearned data and unlearning intentions, if there are other operations. Informing the server of unlearning intentions to customize unlearning operations is considered a privacy threat because it reveals users' unlearning purposes, potentially enabling the server to prepare targeted unlearning attacks [R1,R2]. Therefore, in our setting, we assume the ML server has only the learning algorithm $\mathcal{A}$ and the model with parameters $\theta$ to meet strict privacy requirements. The ML server will not conduct unlearning operations other than training the model using the learning algorithm $\mathcal{A}$ for model updating.

Moreover, we assume the ML server does not store the original training data and cannot access the erased data, which should not be exposed to the server again due to privacy concerns. This assumption is reasonable in both real-world and privacy-preserving MLaaS scenarios. In real-world applications, vast amounts of data are generated daily, leading to the need for prompt model updates. Consequently, many models are trained using incremental or continual learning techniques [R3,R4]. Therefore, the server does not retain the entire raw data due to its large size [R5,R6]. In privacy-preserving scenarios, the ML server is restricted from directly accessing private training data from users due to privacy concerns [R7,R8].

The Users' Ability. The training data $D$ was collected from all users and was used to train the model $\theta_o$. The unlearning user has the erased data $D_u \subset D$. To estimate the unlearning update as the target for unlearning noise generation in our method, we assume the unlearning user can access the trained model $\theta_o$, which is a common setting even in many privacy-preserving scenarios such as FL. We assume the unlearning user has auxiliary clean examples $D_a$ so that they can synthesize a new dataset based on it with the unlearning noise, replacing the erased data $D_u$ for achieving the unlearning effect with only incremental learning using the synthesized dataset.

References [R1-R8] are provided at the end of the revised manuscript and the first comment.

Response to W2: We greatly thank the Reviewer's insightful comment. We supplemented that it is reasonable for users to have clean data in continual learning scenarios, as they are the data contributors. As more samples are unlearned (deleted), the distribution over the dataset is changing but would not be too significant as the number of unlearned samples is negligible to the size of the whole training dataset. The results in [R15] present that 3.2 million requests for deleting URLs have been issued to Google from 2014 to 2019, which certainly constitutes less than 0.01% of the total URLs Google indexes in the 5 years. We thank you again, and we believe the question is valuable in investigating the unlearning for a large amount of deletion, which is our future work.

[R15]. Bertram, Theo, et al. "Five years of the right to be forgotten." Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 2019.

Rebuttal Part 2

Machine Unlearning Service Scenarios. To facilitate understanding, we introduce the problem in a Machine Learning as a Service (MLaaS) scenario. In the MLaaS setting, there are two key entities: a ML server that trains models as ML services, and users (data owners) who contribute their data for ML model training. In such scenarios, machine unlearning occurs when users realize that some of their previously contributed samples are private and wish to revoke these data contributions from the trained models.

The ML Server's Ability. We assume the ML server is honest but curious [R1]: while it honestly hosts and provides ML services, including model training and updating, it may still be curious about private information, such as unlearned data and unlearning intentions, if there are other operations. Informing the server of unlearning intentions to customize unlearning operations is considered a privacy threat because it reveals users' unlearning purposes, potentially enabling the server to prepare targeted unlearning attacks [R1,R2]. Therefore, in our setting, we assume the ML server has only the learning algorithm $\mathcal{A}$ and the model with parameters $\theta$ to meet strict privacy requirements. The ML server will not conduct unlearning operations other than training the model using the learning algorithm $\mathcal{A}$ for model updating.

Moreover, we assume the ML server does not store the original training data and cannot access the erased data, which should not be exposed to the server again due to privacy concerns. This assumption is reasonable in both real-world and privacy-preserving MLaaS scenarios. In real-world applications, vast amounts of data are generated daily, leading to the need for prompt model updates. Consequently, many models are trained using incremental or continual learning techniques [R3,R4]. Therefore, the server does not retain the entire raw data due to its large size [R5,R6]. In privacy-preserving scenarios, the ML server is restricted from directly accessing private training data from users due to privacy concerns [R7,R8].

The Users' Ability. The training data $D$ was collected from all users and was used to train the model $\theta_o$. The unlearning user has the erased data $D_u \subset D$. To estimate the unlearning update as the target for unlearning noise generation in our method, we assume the unlearning user can access the trained model $\theta_o$, which is a common setting even in many privacy-preserving scenarios such as FL. We assume the unlearning user has auxiliary clean examples $D_a$ so that they can synthesize a new dataset based on it with the unlearning noise, replacing the erased data $D_u$ for achieving the unlearning effect with only incremental learning using the synthesized dataset.

References [R1-R8] are provided at the end of the revised manuscript and the first comment.

Response to Presentations 1: We thank the Reviewer's suggestion, which is also suggested by Reivewer BZGC. We replaced most "\cite" using "\citep".

Response to Presentations 2: We thank the Reviewer's comment. We revised this abbreviations, and we carefully checked the whole paper as you suggested.

Response to Questions: We sincerely appreciate the Reviewer's question. Our method does impair the model utility to some extent. Therefore, we propose the compensation method by uploading some clean samples together. Moreover, with a larger auxiliary sample rate and clean sample rate, we can effectively mitigate the model utility degradation, which is presented in Figure 6 in the original manuscript and Figure 5 in the revised manuscript. Besides increasing the auxiliary and clean sample rates, we consider that dividing the unlearning requests into different uploading rounds may also mitigate the model utility degradation. It is valuable for the future investigation. We thank you for your question again.

Rebuttal Part 1

We greatly appreciate your insightful comments that precisely recognize the strengths of our work. We additionally conducted experiments with the MI metric, as you suggested, based on the referenced paper. We provided supplementary materials, which separate the newly added context for your convenience. Below, we offer our detailed responses to your comments, categorized by [W] for weaknesses and [Q] for questions.

Response to W1: We sincerely thank the Reviewer's suggestion. We conducted additional experiments using the membership inference (MI) attack metric based on the referenced paper [R12] and [R13]. As [R12], we train a binary classifier on the unlearned model's loss on erased and test examples for the objective of classifying "in" (forget) versus "out" (test). Then, we make predictions for held-out losses (losses that weren't trained on) that are balanced between forget and test losses. For a perfect defense, the MI accuracy is 50%, indicating that it is unable to separate the two sets, marking a success for the unlearning method. We demonstrate the corresponding results in the following table. We set the unlearning sample rate (USR) equal to 1% and test various parameters of auxiliary sample rate (ASR) and clean sample rate (CSR). The results demonstrate that larger ASR and CSR ensure a better unlearning effect, i.e., lower MI accuracy.

We also want to explain why we choose the membership inference via backdoor (MIB metric, the backdoor accuracy). MIB [R14] infers the unlearning effect based on the removal effect of backdoored samples. The disappearance of the influence of backdoor samples can directly demonstrate the unlearning effect, which we believe is more straightforward than MI. The backdoor accuracy of MIB usually drops from 100% to 10%, while the accuracy of MI usually drops from 60% to 50%.

[R12]. Kurmanji, M., Triantafillou, P., Hayes, J. and Triantafillou, E., 2024. Towards unbounded machine unlearning.Advances in neural information processing systems, 36.

[R13]. Chen, Min, et al. "When machine unlearning jeopardizes privacy." Proceedings of the 2021 ACM SIGSAC conference on computer and communications security. 2021.

[R14]. Hu, Hongsheng, et al. "Membership inference via backdooring." IJCAI (2022).

The Table R3 of Membership inference attack accuracy:

Response to W2: We appreciate the Reviewer's comment. There may be some unclarity in our presentation of the method. In fact, our approach does not require access to the training set. Eq.(4) is the ideal situation that we hope to achieve but is intractable. We relaxed the problem and solved it from lines 256 to 268. As suggested by Reviewer BZGC, we replaced the original model $\theta$ using $\theta_o$ for clarity. All our methods required are the original model $\theta_o$, the unlearned data $D_u$, and some auxiliary data $D_a$. We present the revised method introduction (lines 256 to 268), detailed scenario setting and requirements as follows.

Method introduction (Lines 256 to 268): Unfortunately, calculating $\Delta^p$ that satisfies Eq.(5) is also intractable as it is required to hold for all values of $\theta$. In our setting, the unlearning user cannot access $\theta$ for samples in the remaining dataset $D \setminus D_u$. One possible solution proposed in (Geiping et al. 2021; Di et al. 2022) is to relax Eq.(5) to be satisfied for a fixed model — the model obtained by training on the original dataset. We assume a well-trained model $\theta_o$ before unlearning and fix it during unlearning noise generation. Then, we can minimize the loss based on the cosine similarity between the two gradients as:

$\phi (\Delta^p, \theta_o) = 1 - \frac{\langle \Delta \theta_{D_u}, -\frac{1}{P} \sum_{i=1}^P \nabla_{\theta} \ell ((x_i + \Delta_i^p,y_i);\theta_o) \rangle}{ \| \Delta \theta_{D_u} \| \cdot \| -\frac{1}{P}\sum_{i=1}^P \nabla_{\theta} \ell ((x_i + \Delta_i^p,y_i);\theta_o) \|}.$

(Geiping et al. 2021) use $R$ restarts, usually $R \leq 10$, to increase the robustness of noise synthesis. Using this scheme, we can also find suitable noise for unlearning. However, it is not always effective because we cannot always achieve satisfactory random noise within $10$ restarts. To address this issue, we propose an unlearning noise descent strategy.

Scenarios and threat model are attached in the next part.

Rebuttal Part 1

We greatly thank Reviewer iCGz for acknowledging the contributions, soundness, and presentation quality of our paper. And we sincerely appreciate Reviewer iCGz for proposing these insightful questions. We uploaded the revised manuscript and provided supplementary materials, which separate the newly added context for your convenience. Below, we provide our responses to the comments, denoted by [W] for weaknesses.

Response to W1: We greatly appreciate the Reviewer's insightful comment. We supplemented discussion about the scenario of our setting. We set the scenario in continual learning, which would be motivated that the server will continually update the model, and the users will upload the new data for model update [R3, R4]. We assume the ML server is honest but curious [R1]: while it honestly hosts and provides ML services, including model training and updating, it may still be curious about private information, such as unlearned data and unlearning intentions, if there are other operations. Moreover, we assume the ML server does not store the original training data and cannot access the erased data, which should not be exposed to the server again due to privacy concerns. Additionally, as the Reviewer pointed out that if the server is malicious and has access to the all original data, we believe protecting the unlearning intention and unlearning samples during unlearning requests is also important. It is because the unlearning intention and requirement will remind the server the unlearned samples contain private information. Otherwise, the server may only store these data and does not realize which parts are private. We demonstrate the scenario discussion as follows.

Machine Unlearning Service Scenarios. To facilitate understanding, we introduce the problem in a Machine Learning as a Service (MLaaS) scenario. In the MLaaS setting, there are two key entities: a ML server that trains models as ML services, and users (data owners) who contribute their data for ML model training. In such scenarios, machine unlearning occurs when users realize that some of their previously contributed samples are private and wish to revoke these data contributions from the trained models.

The ML Server's Ability. We assume the ML server is honest but curious [R1]: while it honestly hosts and provides ML services, including model training and updating, it may still be curious about private information, such as unlearned data and unlearning intentions, if there are other operations. Informing the server of unlearning intentions to customize unlearning operations is considered a privacy threat because it reveals users' unlearning purposes, potentially enabling the server to prepare targeted unlearning attacks [R1,R2]. Therefore, in our setting, we assume the ML server has only the learning algorithm $\mathcal{A}$ and the model with parameters $\theta$ to meet strict privacy requirements. The ML server will not conduct unlearning operations other than training the model using the learning algorithm $\mathcal{A}$ for model updating.

Moreover, we assume the ML server does not store the original training data and cannot access the erased data, which should not be exposed to the server again due to privacy concerns. This assumption is reasonable in both real-world and privacy-preserving MLaaS scenarios. In real-world applications, vast amounts of data are generated daily, leading to the need for prompt model updates. Consequently, many models are trained using incremental or continual learning techniques [R3,R4]. Therefore, the server does not retain the entire raw data due to its large size [R5,R6]. In privacy-preserving scenarios, the ML server is restricted from directly accessing private training data from users due to privacy concerns [R7,R8].

The Users' Ability. The training data $D$ was collected from all users and was used to train the model $\theta_o$. The unlearning user has the erased data $D_u \subset D$. To estimate the unlearning update as the target for unlearning noise generation in our method, we assume the unlearning user can access the trained model $\theta_o$, which is a common setting even in many privacy-preserving scenarios such as FL. We assume the unlearning user has auxiliary clean examples $D_a$ so that they can synthesize a new dataset based on it with the unlearning noise, replacing the erased data $D_u$ for achieving the unlearning effect with only incremental learning using the synthesized dataset.

Rebuttal Part 2

Response to W3: We are appreciated of the Reviewer's this comment. After your former suggestion of replacing $\theta$ using $\theta_o$ to denote the original trained model, which is revised in the manuscript.

Eq.(4) is the ideal and theoretical station to solve the oblivious unlearning problem, i.e., the unlearning users can access the full training data $D$, so they can achieve the retrained model. However, in our setting, the unlearning users cannot access $\theta$ for samples in the remaining dataset $D \setminus D_u$. Our solution is to relax the second objective of Eq.(4) to be satisfied with a fixed model. And we choose the model obtained by the training on the original dataset, $\theta_o$. It is reasonable for the following reasons: (1) our aim is achieving unlearning by incremental learning, and it is reasonable that the server will continually learn based on the trained model $\theta_o$ rather than an initial model; (2) continually learning for $\theta_o$ based on $D_a^p$ with the first objective in Eq.(4) will simultaneously meet the unlearning purpose of $D_u$ and incremental learning purpose of $D_a^p$. The corresponding introduction of these explanation is presented from line 256 to line 267 in the manuscript. The Eq.(8) is the relaxed incremental learning objective which is based on $\theta_o$. We also attached the revised line 256 to 269 as follows.

Unfortunately, calculating $\Delta^p$ that satisfies Eq.(5) is also intractable as it is required to hold for all values of $\theta$. In our setting, the unlearning user cannot access $\theta$ for samples in the remaining dataset $D \setminus D_u$. One possible solution proposed in (Geiping et al. 2021; Di et al. 2022) is to relax Eq.(5) to be satisfied for a fixed model — the model obtained by training on the original dataset. We assume a well-trained model $\theta_o$ before unlearning and fix it during unlearning noise generation. Then, we can minimize the loss based on the cosine similarity between the two gradients as:

$\phi (\Delta^p, \theta_o) = 1 - \frac{\langle \Delta \theta_{D_u}, -\frac{1}{P} \sum_{i=1}^P \nabla_{\theta} \ell ((x_i + \Delta_i^p,y_i);\theta_o) \rangle}{ \| \Delta \theta_{D_u} \| \cdot \| -\frac{1}{P}\sum_{i=1}^P \nabla_{\theta} \ell ((x_i + \Delta_i^p,y_i);\theta_o) \|}.$

(Geiping et al. 2021) use $R$ restarts, usually $R \leq 10$, to increase the robustness of noise synthesis. Using this scheme, we can also find suitable noise for unlearning. However, it is not always effective because we cannot always achieve satisfactory random noise within $10$ restarts. To address this issue, we propose an unlearning noise descent strategy.

Rebuttal Part 1

We greatly appreciate Reviewer BZGC for providing a detailed summary of our strengths. And greatly appreciate Reviewer BZGC for proposing these insightful questions and providing these constructive suggestions. We have uploaded the manuscript, which has been revised according to your suggestions. We also provided supplementary materials, which separate the newly added context for your convenience. Below, we provide our responses to the comments, denoted by [W] for weaknesses and [Q] for questions.

Response to W1: We greatly appreciate Reviewer BZGC for your patience in pointing out these writing issues. We have fixed all the pointed issues. We summarize the changes as below.

• After the problem statement, we add explanation about $\theta_{unl} \gets \mathcal{U}(\cdot)$ and $\theta_{inc} \gets \mathcal{A}(\cdot)$, which denotes the functions with inputs of model parameters and data, and output the unlearned model and incremental learned model. In Section 3.1, we add the explanation about $\nabla$ and $\ell$, which denotes the gradients and the loss function of the learning algorithm. In Eq.(3), $\mathcal{I}^{(1)}(D_u)$ denotes the estimated first-order influence of the unlearning data $D_u$. We also supplemented in line 225 $I$ is the identity matrix.
• We fixed the typo in line 221, and carefully checked the entire paper.
• We revised the sentence as suggested.
• We thank the Reviewer's suggestion, and we revised the corresponding parts of the entire paper using $\theta_o$ instead of $\theta$ to represent the original trained model, including the parts in Appendix.
• We thank the Reviewer's suggestion, and we replaced most "\cite" using "\citep".
• We agree that Algorithm 2 is an important contribution to this paper. Therefore, as you suggested, we put it in the main text and added a detailed description as follows.

Algorithm 2 synthesizes unlearning noise to create a perturbed dataset $D_{a,p} = (X_a + \Delta^p, Y_a)$. Firstly, we generate a noise matrix $\Delta^{p}$ as shown in line 1 of Algorithm 2 and treat it as parameters that could be updated during optimization. Then, during optimization, we fix the trained model parameters $\theta$ and add the noise to the auxiliary data $D_a$ as the input to the model (line 4). We calculate the gradients of the noise-synthesized data based on the current model point but do not update the model (line 5). Moreover, we calculate the minimization loss according to Eq.(6) (line 6). With this loss, we can use the gradient descent method for both the model and the unlearning noise matrix, but we only update the noise matrix $\Delta^{p}$ while keeping the model $\theta$ fixed (line 7). After a few rounds of iteration, we can synthesize sufficient unlearning noise to the auxiliary data to achieve the unlearning effect.

Response to W2: We sincerely thank Reviewer BZGC to provide the constructive suggestions about figures. We response these comments as follows.

• We appreciated the Reviewer's suggestion. We revised the paper as you suggested by removing the original Figure 1 in the Introduction section. We also revised the corresponding introduction parts.
• We revised the $\mathcal{I}_{D_u}$ in Figure 2.
• The problem statement is the idea station to achieve oblivious unlearning by learning. Ideally, we can directly achieve both the unlearning effect and model utility preservation to solve the problem. In practice, it will cause the model utility degradation. Therefore, we additionally employ clean data to mitigate the model utility degradation. It is a compensation for our solution; hence, we have not added it to the problem statement.

[R1]. Hu, Hongsheng, et al. "Learn what you want to unlearn: Unlearning inversion attacks against machine unlearning." IEEE Symposium on Security and Privacy (SP) (2024).

[R2]. Chen, Min, et al. "When machine unlearning jeopardizes privacy." Proceedings of the 2021 ACM SIGSAC conference on computer and communications security. 2021.

[R3]. Rolnick, David, et al. "Experience replay for continual learning." Advances in neural information processing systems 32 (2019).

[R4]. Lopez-Paz, David, and Marc'Aurelio Ranzato. "Gradient episodic memory for continual learning." Advances in neural information processing systems 30 (2017).

[R5]. Wu, Yue, et al. "Large scale incremental learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[R6]. Wang, Zifeng, et al. "Learning to prompt for continual learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[R7]. Naseri, Mohammad, Jamie Hayes, and Emiliano De Cristofaro. "Local and central differential privacy for robustness and privacy in federated learning." NDSS (2022).

[R8]. Bonawitz, Keith, et al. "Practical secure aggregation for privacy-preserving machine learning." proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 2017.

[R9]. Gao, Xiangshan, et al. "Verifi: Towards verifiable federated unlearning." IEEE Transactions on Dependable and Secure Computing (2024).

[R10]. Guo, Xintong, et al. "Fast: Adopting federated unlearning to eliminating malicious terminals at server side." IEEE Transactions on Network Science and Engineering (2023).

[R11]. Zhu, Ligeng, Zhijian Liu, and Song Han. "Deep leakage from gradients." Advances in neural information processing systems 32 (2019).

[R12]. Kurmanji, M., Triantafillou, P., Hayes, J. and Triantafillou, E., 2024. Towards unbounded machine unlearning.Advances in neural information processing systems, 36.

[R13]. Chen, Min, et al. "When machine unlearning jeopardizes privacy." Proceedings of the 2021 ACM SIGSAC conference on computer and communications security. 2021.

[R14]. Hu, Hongsheng, et al. "Membership inference via backdooring." IJCAI (2022).

[R15]. Bertram, Theo, et al. "Five years of the right to be forgotten." Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 2019.

[R16]. Oh, Seong Joon, Bernt Schiele, and Mario Fritz. "Towards reverse-engineering black-box neural networks." Explainable AI: interpreting, explaining and visualizing deep learning (2019): 121-144.

[R17]. Salem, Ahmed, et al. "{Updates-Leak}: Data set inference and reconstruction attacks in online learning." 29th USENIX security symposium (USENIX Security 20). 2020.

[R18]. Wang, Binghui, and Neil Zhenqiang Gong. "Stealing hyperparameters in machine learning." 2018 IEEE symposium on security and privacy (SP). IEEE, 2018.

[R19] Chundawat, Vikram S., et al. "Zero-shot machine unlearning." IEEE Transactions on Information Forensics andSecurity 18 (2023): 2345-2354.

[R20] Golatkar, Aditya, Alessandro Achille, and Stefano Soatto. "Eternal sunshine of the spotless net: Selective forgettingin deep networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[R21] Tarun, Ayush K., et al. "Fast yet effective machine unlearning." IEEE Transactions on Neural Networks and LearningSystems (2023).