Adversarial Machine Unlearning: A Stackelberg Game Approach

We thank the reviewer for the feedback!

1. The precise formulation of $f$ depends on the specific model used in the MIA. The stationarity condition is indeed a part of $f$. For illustrative purposes, we provide an example of $f$ for linear SVM in the appendix (A.6).

2.1 We thank the reviewer for pointing this out. We have updated Eq.(4) in the revised paper.

2.2 In section 5, $n$ is the number of variables plus the number of constraints in the attacker’s optimization problem.

2.3 We will double check the notations and improve the clarity.

3.1 We selected 10% to ensure consistent comparison with prior studies. However, we acknowledge that exploring the impact of the forget set size on unlearning is an important direction for future research. To show that the SG framework is applicable to more than just image datasets and ResNet architectures, we present supplementary results on text data and transformer architectures in section A.1 of the appendix.

3.2 For CIFAR10 and CIFAR100, the retraining process takes 200 epochs. For SVHN, it takes 100 epochs. The major overhead of adversarial modeling consists of two parts: 1) solving the attacker's optimization problem and 2) implicit differentiation. The second part is the most computationally expensive. However, the running time of SG is only a small fraction of the running time of retraining (see Table 1). The detailed hyperparameters for SG and the baselines are now included in Table 4 of the revised paper.

3.3 The performance of the unscrubbed models (e.g., test accuracy and the MIA accuracy) are displayed in Figure 2 at epoch = 0. The noise level of FF and IU is selected on the validation data. The criterion is to select the noise level that maximizes: $\text{prediction accuracy}−\text{MIA accuracy}$. We exhaustively search five values for the noise level: 1e-9, 1e-8, 1e-7, 1e-6, 1e-5. IU is not sensitive to the noise level. However, the test accuracy of FF significantly drops below 0.4 when the noise level exceeds 1e-7, which makes the unlearned model useless for prediction tasks.

3.4 We would like to clarify that the KS and MIA metrics reported in Table 1 are evaluated on an auditing dataset that is separate from the one used in unlearning. Although the two auditing datasets share the same forget set, they have different test instances. In the revised paper, we have made this distinction clearer. Additionally, we have included two new additions: 1) running time comparisons and 2) the Wasserstein distance between the cross-entropy losses of the forget and test instances. The Wasserstein distance reported in Table 1 is computed on an auditing dataset that is disjoint from the one used in unlearning.

3.5 We added the training loss for the retrain baseline to Figure 7 in the appendix.

3.6 We will go through the literature and discuss the connections to the current paper in the next revision.

3.7 Please refer to comment 6 in “To all reviewers” at the top. We appreciate the reviewer's question and welcome further discussion on this topic.

Thanks for the reviewer’s comments. We highly appreciate the interactive discussion.

1.1 We argue that a reasonable metric to validate the claim that "SG achieves the best trade-off between forgetting and retaining model utility" is to measure the closeness of SG to the retrained model in terms of various metrics, such as test accuracy, retain accuracy, MIA accuracy, etc. Specifically, we want to emphasize that the accuracy on the forget set is a misleading metric. If the forget set has a significantly lower or higher accuracy compared with typical testing data, it would be easily identified as abnormal by an attacker---hence breaching privacy. We believe that the metrics highlighted in Tables 1 and 3 support the effectiveness of SG. These tables demonstrate that SG is the closest to the retrained model from various perspectives.

1.2 The asked metrics for the unscrubbed model is below. Each number is averaged over 10 random seeds.

2. We are calculating the Wasserstein distance between the output distributions of the forget and test sets. This distance provides direct evidence that the two sets are difficult to distinguish. We acknowledge that the Wass. distance you suggested could provide further evidence, and we will include it in the next revision.

3. We provided the theoretical evidence of the convergence in comment 5 in “To all reviewers”. An exact convergence to a Stackelberg equilibrium would be time-consuming. Instead, the number of epochs of SG is considered a hyperparameter that balances the runtime, the performance of unlearning, and the utility of the model. We hope this resolves the concerns. If not, it would be appreciated if you could expand on your concerns.

4. Conceptually, there is no obstacle to generalizing the SG framework to the scenario of forgetting an entire class. However, one challenge is defining what it means to forget a class. Additionally, as we argued above, the accuracy on the forget set (or 1 - the accuracy on the forget set) would not be an appropriate measure. We acknowledge this as a potential area for future research.

Thanks for the comments.

1. We agree that studying unlearning in a class-wise setting is an important problem and we leave this as a future research. However, we respectively disagree the use of smaller accuracy on the forget set as an evaluation metric for our setting, i.e., the forget set is sampled uniformly at random. Let's examine the following thought experiment:

   • Imagine you have an unlearned model with a known testing accuracy of 0.9. Now, if you are given a forget set D and the model performs at 0.1 on D, would you suspect that this set has been tampered with by a third-party? If not, how to explain the accuracy gap (0.9 vs 0.1)?

2. Theoretical evidence for convergence: The discussion has not been added yet. We are currently working on it.

3. Wasserstein distance used in [1]: We will consider adding this result. See the new responses below.

4. Concerns about clarity: We will update the notations to better clarify if this is what you are referring to.

We thank the reviewer for updating the score and we truly appreciate the discussion! Our responses are listed below:

1. We totally agree that removing the influence of poison samples is a crucial aspect of unlearning. The current paper serves as an initial step in adopting an adversarial perspective on machine unlearning. Further investigations into poison samples will be left as a future task.

2. For SVHN, we conducted two student's t-tests on the MIA accuracy and the forget accuracy (between SG and FT), respectively. The difference in MIA accuracy was not found to be statistically significant (p-value=0.1921). However, the difference in forget accuracy was found to be significant (p-value=0.0003). Despite this, we would argue that SG is not expected to outperform the others in every metric; indeed, it is only explicitly optimized for test accuracy and MIA accuracy (see Eq.(3)).

3. Class-wise forgetting: we thank for the advice. For having a solid and complete evaluation, we will leave this setting as a future task.

4. Comparison with [1]: We believe that SG is orthogonal to the method proposed in [1]. Specifically, the method in [1] requires a pruning step, which necessitates access to the weights of the model to be unlearned. However, SG does not require this access. Instead, SG can be seen as a specially designed loss function with an adversarial component. In conclusion, these two methods approach unlearning from different angles and with different assumptions (i.e., white-box v.s. black-box access to the model weights). Therefore, a fair comparison between the two would not be appropriate.

We thank the reviewer’s comments for improving the paper!

1. Please refer to comment 5 in “To all reviewers” at the top. We will add a subsection to discuss the convergence in details.

2. We have a time complexity analysis of Algorithm 1 in section 5, just above the pseudocode of the algorithm. We will revise the writing to emphasize the analysis.

3. We acknowledge the significance of investigating the setting of multiple attack methods as an important direction for future research. It is worth noting that the single-attack setting has not been explored in the context of machine unlearning, and this paper takes a first step towards addressing the unlearning problem from a game theoretic perspective.

4. We have added additional experimental results on the SVHN dataset, and results on text data and transformer architectures.

• SVHN dataset: see A.2 in the appendix.
• Text data and transformer architectures: see A.1 in the appendix.
• Running time comparison has been added to Table 1.

We thank the reviewer’s insightful comments!

1. Please refer to comment 6 in “To all reviewers” at the top. We appreciate the reviewer's question and welcome further discussion on this topic.

2. Uniform sampling is used as an approximation of the distribution of real-world forget requests. For instance, [2] demonstrates that requests to delete URLs originate from multiple countries, although some countries have more requests than others. We acknowledge that studying a more realistic distribution of the forget set is an important direction for future research. Another reason we use the uniform sampling is for having a consistent comparison with prior studies.

• [2] Five years of the right to be forgotten, Bertram, et al.

3. We acknowledge the importance of understanding the impact of the training set size on unlearning. However, we believe that the issue of unlearning extends beyond the size of the training data. For example, SG and FT are trained using the same retain set, but the attacker's perspective reveals significant differences in their MIA accuracy and KS (as shown in Table 1). Furthermore, we have included Figure 2 in the revised paper, which demonstrates that adversarial modeling has a significant impact on the learning dynamics. Please refer to 1c & 1d in "To all reviewers" at the top for further details.

4. Please refer to comment 1b and 5 in “To all reviewers” at the top (also Figure 2 in the revised paper). We will add a subsection to discuss the convergence in details.