Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery

We would like to thank reviewer 7Sf9 for the insightful comments. We address the concerns below.

W1. Datasets are relatively simple...the evaluation appears insufficient to fully validate the generalizability and scalability of the proposed method

Good suggestion! Our study deliberately spans multiple data regimes—natural-image classification, generative diffusion, style/instance manipulation, and safety filtering. We also evaluate on different backbones of Resnet, VIT, DDPM and Stable Diffusion. Each results are averaged over multiple independent trials in more than one different code base implementation. Therefore, we believe that such diverse experimental settings could already be able to prove the generalizability and scalability of the proposed method.

Meanwhile, we tested the performance of Resnet 18 unlearning on Cifar 100, random data forgetting 10%.

These results are consistent with the conclusions presented in the paper. We will include results on larger-scale datasets, such as Tiny-ImageNet and ImageNet-1k, in a future version of the paper.

W2 & Q1.Define "pure linearization" and "RL"...it raises concerns about the effectiveness of the proposed method, which claims to achieve comparable computational efficiency to "linear weighting

Thank you for pointing this out! "RL" refers to Random-Labeling as cited in Appendix D.1, and pure linearization indeed refers to "linear weighting".

The introduction of an approximation to the gradient method for the Lagrangian (equation 4) avoids the need to explicitly solve for the two objectives. However, the computation of $\ell_r (\theta_{t+1})$ (equation 8) introduces an additional forward pass. This results in a slight increase in computational complexity compared to linearization, although the computational cost is still significantly lower than that of standard backpropagation. This explains the slight increase in computational cost shown in Table 5.

However, this additional forward pass can be eliminated by reusing the value of $\ell_r (\theta_{t+1})$ computed during the forward pass of the next iteration (t+1). While this may lead to a mismatch between the data batches at step t and t+1 during stochastic training, it does not affect the model's convergence due to the unbiased nature of the stochastic process. Our latest experiments show that this modified approach (see EUPMU (modified)) achieves similar results comparable to EUPMU (previous) in Table 5, with a computational time on par with linearization.

We will discuss this more lightweight computational approach and provide sufficient experimental validation in a future version of the paper.

W3 & Q2. Error tolerance $\varepsilon$ sensitivity...This lack of analysis makes it difficult to assess the robustness of the proposed method

Below, we provide an expanded ablation study error tolerance $\varepsilon$.

We observe that as $\varepsilon$ ranges from 0 to 1, our algorithm smoothly traces the Pareto front. This confirms that the utility-unlearning tradeoff is controllably adjustable, allowing practitioners to select a balance suitable for their specific needs. This finding is in direct alignment with our statements in Remark 3.3 and Remark 3.6.

Regarding the tuning of $\varepsilon_t$, this process is not burdensome. As guided by our theoretical results in Remark 3.3 and Theorem 3.7, one can effectively narrow the search range based on the magnitude of the loss function, which minimizes practical tuning effort. We will add further experiments and discussion to clarify this in the revised manuscript.

W4 & Q3. Cost‑effectiveness vs. GA in Table 6

The running time comparison in Table 6 requires careful context. Fundamentally, the per-iteration computational cost of EUPMU (with the optimizations from our W2 response) and Gradient Ascent (GA) is the same. The lower time for GA is an artifact of its training instability: it must be early-stopped at 20% of epochs to prevent gradient explosion, which would otherwise cause a catastrophic drop in Test Accuracy (TA).

If run for an equal number of epochs, EUPMU would have a similar runtime to GA but achieve a higher TA due to better utility preservation.

Moreover, GA's result in Table 6 is a special case. It performs poorly on all other datasets (e.g., Table 7). This highlights the advantage of our method, which provides a superior trade-off, delivering more effective unlearning and utility preservation without a true increase in computational cost.

W5 & Q4. Evaluate gap‑to‑retrain metric.

From a practical standpoint, directly optimizing for raw accuracy is more beneficial for real-world applications. In practice, the results of a fully retrained model (the "retrain oracle") are often unavailable. As long as the model's utility and data privacy metrics are guaranteed, its practical effectiveness is ensured. In our experiments, the unlearning process further fine-tunes the model on the retained data via the retain_loss. This can lead to performance that is better than retraining from scratch, especially on simpler tasks (like Table 6). In such cases, a comparison based on the "gap-to-retrain" metric would not be fair. Therefore, we chose to present our results using raw accuracy.

If the goal is to minimize the gap to the Retrain Oracle, EUPMU still outperforms other methods. We benchmarked against GDR-GMA [1] and Learn to Unlearn [2]—strong baselines noted by reviewers and established as state-of-the-art in their publications. The following table presents the results for class-wise unlearning with ResNet-18 on CIFAR-10:

As shown in the above gap‑to‑retrain results (and also the table in the response of W1), when we optimize the hyperparameters for gap-to-retrain, EUPMU still performed the best compared to other methods as well.

[1]Lin, Shen, et al. "GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

[2] Qu, Youyang, et al. "Learn to Unlearn: A Survey on Machine Unlearning." arXiv preprint arXiv:2305.07512 (2023).

Dear Reviewer 7Sf9,

Thank you again for your valuable feedback. To further clarify our method's cost-effectiveness (W2 & W4) and the practical trade-offs involved, we have conducted a targeted experiment on the random data forgetting task, using the gap-to-retrain metric as the key benchmark. We also have implemented and tested the modified EUPMU with improved efficiency mentioned in response to W2, using next step's retain loss to measure the change rather than doing an extra forward pass on the current batch.

The results below are presented to directly address your concerns, and they lead to two clear, decisive conclusions:

1. Our Method is a Flexible Highly Effective and Lightweight Framework. Our approach is best understood as a "plug-and-play" framework that enhances existing unlearning methods. Crucially, this integration comes at no significant computational cost. As shown in the table, the runtime (RTE) of EUPMU RL (2.69) is nearly identical to the baseline RL (2.64), yet it dramatically reduces the Average Gap from 7.15 to an exceptional 0.89.

2. A Direct Rebuttal to GA's Apparent Cost-Effectiveness. In our initial rebuttal, we argued that GA's low runtime is a result of its training instability, which requires aggressive early stopping. This experiment provides the definitive evidence. When our EUPMU_GA is also run for only one epoch—matching the conditions under which GA is viable—it achieves a much better Average Gap of 3.23 compared to GA's 4.12, at a comparable runtime (RTE 0.19 vs 0.13). This proves our method's performance gains stem not from longer training, but from a fundamentally more stable and effective optimization process per update step.

We believe this analysis confirms that our framework provides a superior trade-off between unlearning effectiveness, model utility, and computational cost. We are confident this resolves the questions regarding practicality and will incorporate this analysis into the final manuscript.

I appreciate the authors' thoughtful responses and the additional experimental results provided. I understand that the proposed method can be applied across a diverse set of tasks, including image classification, generative diffusion, style/instance manipulation, and safety filtering.

However, I remain concerned about the response to W5 and Q4. The goal of machine unlearning is to completely remove the influence of specific training data such that the resulting model behaves equivalently to one that was trained without those data [1]. From this perspective, I believe that the most appropriate and principled evaluation metric is the Gap, i.e., the difference in performance between the unlearned model and the retrained model trained from scratch without the forgotten data. In this regard, I observe that in Table 6 of the submission and in the response to W1, the proposed method (UA) performs worse than the baseline in terms of Gap, which raises concerns about the effectiveness of the proposed method. Moreover, in the new results presented in the response to W5 and Q4, it is shown that UA achieves a smaller Gap than the baselines. However, I am unclear whether this improvement is the result of tuning the method based on the retrained model's accuracy. If so, this would be problematic, as such tuning undermines the goal of achieving effective unlearning without access to retraining. I would appreciate clarification on this point.

[1]Xu et al., Machine Unlearning: A Survey. ACM Computing Surveys, 2023.

We would like to thank Reviewer RLwG for acknowledging our novelty and contributions. We address the concerns as following:

W1: The introduction of two new hyperparameters in EUPMU increases the complexity of practical tuning.

Although our proposed EUPMU method introduces two hyperparameters, $\varepsilon_t$ and $\beta_t$​, these are essential for addressing two core challenges in machine unlearning.

On one hand, machine unlearning faces the challenge of utility degradation. While a multi-objective approach can prevent utility degradation, it often fails to ensure sufficient forgetting of target information. Therefore, we introduce $\varepsilon_t$​ to further balance utility preservation and unlearning effectiveness.

On the other hand, introducing a multi-objective approach typically incurs an additional round of backpropagation. To mitigate this, we adopt a gradient-based approximation that avoids this extra backward pass, for which we introduce $\varepsilon_t$ as the gradient step size in the approximation.

Moreover, both hyperparameters are guided by theoretical insights. The tuning of $\varepsilon_t$​ can refer to Remark 3.3 and Theorem 3.7, which provide theoretical foundations for determining its value based on the scale of the loss function. Theorem 3.4 also offers a feasible range for $\beta_t$​ to ensure the effectiveness of the approximation, and empirical testing shows this range works well in practice.

In summary, these two new hyperparameters are necessary and, thanks to theoretical guidance, can be tuned within a relatively small and manageable range, thereby avoiding excessive tuning overhead.

W2: The proposed Explicit Unilateral Gradient Surgery bears resemblance to prior techniques, which diminishes its perceived originality compared to novel contributions in the field.

While Explicit Unilateral Gradient Surgery is related to methods in [1,2], our approach differs in two fundamental ways: formulation and utility control.

• Fundamental Difference (Principled Formulation vs. Heuristic Fixes): Prior methods, such as [1,2], stem from a multi-task learning (MTL) perspective, identify issues like gradient conflict, and propose heuristic fixes. GDR-GMA [1] uses separate handcrafted modules for rectification (GDR) and weighting (GMA). Learn to Unlearn [2] projects gradients away from a pre-computed Core Gradient Space (CGS). These are fundamentally gradient correction methods. In contrast, our method does not begin with gradient operations. We formulate a new constrained optimization problem: the Utility-Preserving Unlearning Problem (UPUP). Our Unilateral Gradient Surgery is not a heuristic tweak but the exact analytical solution that emerges from solving the UPUP problem. This principled formulation is the core of our contribution.

• Utility Control (Proactive Constraint vs. Reactive Balancing): Our method includes an explicit constraint $\varepsilon$, a user-defined utility loss budget derived from UPUP, offering an interpretable and proactive control mechanism. In comparison, other approaches rely on reactive heuristic modules like GMA, which adjust based on observed loss changes but lack pre-specifiable utility-preservation levels. Overall, our method shifts the paradigm from reactive balancing to proactive constrained optimization—crucial for dependable and predictable deployments.

Q1: While the paper claims computational efficiency comparable to the linearization method, Table 5 shows that EUPMU still has a slightly higher RTE than Linearization. What causes this discrepancy?

Great question! This slight discrepancy arises because, although we avoid explicitly solving both objectives by using a gradient approximation to the Lagrangian equation (Equation 4), the computation of $\ell_r(\theta_{t+1})$ (Equation 8) introduces one additional forward pass. This is why the runtime is slightly higher than the linearization method—but still much more efficient than backward passes.

Furthermore, we can eliminate this forward pass by reusing $\ell_r(\theta_{t+1})$ from the next time step. Although the data batches at times $t$ and $t+1$ may differ due to randomness, the stochastic nature is unbiased and does not affect convergence. We plan to elaborate and test this in future versions.

Q2: The proposed method appears broadly applicable to all machine unlearning scenarios, yet the experiments only demonstrate results on image classification and generation tasks. What is its applicability in language models?

Excellent observation! In fact, our method proposes a general utility-preservation mechanism that applies to most machine unlearning settings, including LLMs. We used vision models in this version to validate effectiveness comprehensively. We plan to include experiments on LLMs in future versions.

Q3: If error tolerance is disabled, would the results be comparable to those of the multi-objective optimization method?

Setting $\varepsilon$ = 0 reduces UPUP to strict utility preservation. We provide the experiments with different error tolerance below.

From the above results, we know that if error tolerance is disabled, the Retain Loss increases significantly, which indicates that the utility has been degraded seriously. Therefore, it would be better to choose an appropriate error tolerance in a suitable interval as introduced in the answer of W1.

[1]Lin, Shen, et al. "GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

[2] Qu, Youyang, et al. "Learn to Unlearn: A Survey on Machine Unlearning." arXiv preprint arXiv:2305.07512 (2023).

Thank you for your follow-up. Your concern regarding potential performance degradation due to the stochasticity from batch inconsistency is well-founded.

To address this, we implemented and evaluated the proposed optimization, which we term EUPMU-fast. This version updates the weight using the retention loss from the subsequent batch, eliminating the extra forward pass. The setup targets reaching the lowest Gap-to-Retrain with 10% random data forgetting. We have added a comparison of this method in the table below.

The experimental results show:

1. No Performance Collapse: The primary concern of performance collapse was not observed. The EUPMU-fast model trained stably and achieved strong, competitive results across all metrics.

2. Confirmed Efficiency Gain: As hypothesized, EUPMU-fast is more computationally efficient. Its Relative Training Time (RTE) of 2.69 is lower than the original EUPMU's (2.82) and is now on par with the linearization baseline (2.64).

3. Surprising Performance Consistency: Contrary to a simple speed-accuracy trade-off, the stochasticity introduced by the optimization appears beneficial. EUPMU-fast not only improves runtime but also achieves similar or improved performance in key areas, including better Unlearning Accuracy (3.64% vs. 3.71%), Retain Accuracy (99.54% vs. 99.25%), and Test Accuracy (93.17% vs. 93.09%) compared to the original EUPMU. Its overall performance, reflected by the Average Gap, remains excellent and highly competitive (0.89 vs. 0.87).

This validation experiment not only confirms that the optimization is a viable strategy but also shows it can lead to performance on par in addition to the expected efficiency gains, without causing training instability. We will incorporate these new results and this analysis into the revised manuscript.

Thank you for the excellent follow-up question. You are correct in asking how performance remains stable despite the introduced stochasticity.

The performance remains stable due to two main reasons:

1. Unbiased Stochastic Estimation: In EUPMU-fast, the update for the retaining parameter $\lambda_t$ relies on an approximation of the change in retention loss. Instead of calculating $l_r(\theta_t)$ and $l_r(\theta_{t-1})$ on the same data, we use losses from consecutive, independently sampled batches, $B_t$ and $B_{t-1}$. The change $\tilde{\delta}_{t-1}$ is calculated as:

$

\tilde{\delta}_{t-1}^{\text{approx}} \;=\; \frac{1}{\alpha_t} \Bigl( \underbrace{\ell_r(\theta_{t-1};\, B_{t})}_{\text{stored from previous pass}} - \underbrace{\ell_r(\theta_{t};\, B_{t+1})}_{\text{current pass}} \Bigr) +\varepsilon ,
$

instead of

$

\tilde{\delta}_{t-1} =\frac{1}{\alpha_t} \Bigl( \underbrace{\ell_r(\theta_{t-1};\, B_{t})}_{\text{main pass}} - \underbrace{\ell_r(\theta_{t};\, B_{t})}_{\text{extra pass}} \Bigr) +\varepsilon .
$

Because each batch is an i.i.d. sample from the retaining data distribution, the loss calculated on any batch, $\ell_r(\theta_{t}; B_{t+1})$, is a noisy but unbiased stochastic estimate of the true retention loss $\ell_r(\theta_t; B_t)$. This principle is fundamental to stochastic optimization methods like SGD. Therefore, while this approximation introduces variance, it does not introduce systematic bias, allowing the optimization to converge to the same solution in both expectation and practical experiments.

2. Principled Hyperparameter Adjustment: Your intuition is correct; a minor hyperparameter adjustment was made. To counteract the higher variance from the stochastic approximation, we slightly lowered the hyperparameter $\beta$, which serves as the learning rate for the parameter $\lambda_t$ in its update rule:
   $

\lambda_t \;=\; \min\bigl\{ D,\; \max\{0,\; \lambda_{t-1} - \beta_{t-1}\,\tilde{\delta}_{t-1}\}\bigr\}.
$

By reducing $\beta$, we smoothen the trajectory of $\lambda_t$, making it less reactive to the noise from any single batch. This targeted adjustment dampens the variance and ensures a stable convergence.

We would like to thank Reviewer mcr2 for the acknowledgement and insightful comments. We address the concerns as follows.

W1. Gradient projection methods already exist...add the corresponding discussions and comparison experiments.

Contributive question! We now explicitly compare EUPMU with GDR‑GMA[1] and Learn to Unlearn. The key differences are:

• Motivation: Previous works follow the idea gradient conflict mitigation, while EUPMU is the analytic solution of the constrained optimisation problem (UPUP, Equation 2). By re-framing MU at this more fundamental optimization level, we open up new avenues for developing more advanced algorithms. Future work could explore second-order Taylor expansions or introduce preference vectors to solve the UPUP problem. Such approaches can lead to more efficient and practical algorithms, a potential not provided by the singular focus on gradient conflict.

• Theoretical Insights: While the final update mechanism is analogous to methods like gradient surgery, our approach originates from a more principled formulation. This foundational basis enables us to establish formal theoretical guarantees. Specifically, Remark 3.3 quantifies the potential degradation in model utility during unlearning. Moreover, Remark 3.6 leverages the concept of Pareto Optimality to prove that the unlearning objective can be fully satisfied without violating the utility constraint.

• Efficiency: GDR‑GMA needs two, and Learn‑to‑Unlearn adds an SVD pre‑pass, while EUPMU requires a single backward pass. On ResNet‑18/CIFAR‑10 EUPMU is ≈ 30 % faster than GDR‑GMA[1] and ≈ 1.9 times faster than Learn‑to‑Unlearn[2] while achieving 0.09 % UA vs 0.76 %.

Here’s the results of class-wise unlearning for Resnet18 on Cifar10:

Based on these results, our method demonstrates a superior trade-off between model utility and unlearning effectiveness. Furthermore, it achieves Pareto superiority over the other methods, which is consistent with our theoretical analysis. We will incorporate this analysis and these experimental results into the revised manuscript.

W2. MACE and UCE are missing on Stable Diffusion

Good suggestion! We have added SD experiments, using the code base of MACE[4] to implement EUPMU on unlearning the 10 celebrities' faces, as shown in the following results.

The results above indicate that EUPMU, as a more advanced optimization mechanism, achieves better performance retention than MACE and strikes a superior utility-unlearning trade-off compared to UCE[5]. We will provide more comprehensive experiments in the revised version of our paper.

W3. Ablate the key hyper‑parameters

Below we provide an expanded ablation study the two hyperparameters of EUPMU as follows.

• Sensitivity to the error tolerance $\varepsilon$

The results highlight the inherent trade-off between utility and unlearning, which is directly governed by $\varepsilon$. Specifically, a higher $\varepsilon$ prioritizes utility preservation at the expense of unlearning performance. This confirms that $\varepsilon$ functions as a practical control knob, enabling users to balance these competing objectives according to their requirements—a finding that reflects our theoretical analysis in Remarks 3.3 and 3.6.

• Sensitivity to the step-size $\beta$

Our method exhibits strong robustness to the choice of $\beta$. We observed negligible variation in overall performance even when $\beta$ was varied significantly. This experimental result aligns perfectly with the convergence guarantee in Theorem 3.4, which proves that a Pareto-optimal solution is attainable for any $\beta$ selected from a theoretically valid range.

Furthermore, the selection of these hyperparameters is not an arbitrary process but is instead principled and guided by our theoretical framework. For instance, the search range for $\varepsilon$ can be effectively constrained by the insights from Remark 3.3 and Theorem 3.7. Similarly, Theorem 3.4 provides a clear basis for selecting a suitable $\beta$ that ensures convergence. We will incorporate these results and a comprehensive discussion into the revised manuscript. We thank you for this valuable suggestion, which has helped us improve the clarity and completeness of our work.

W4. Can EUPMU erase multiple concepts on SD?

Insightful Question! Yes. There are two primary approaches to extend our method to multi-concept unlearning:

• Sequential Unlearning: The first approach is to erase multiple concepts sequentially, one after another. In fact, Figure 4 and Table 3 of our paper already demonstrate successful unlearning for two and three concepts using this method.

• Simultaneous Unlearning: The second approach is to erase all concepts in a single step. This requires extending the UPUP problem into a constrained multi-objective optimization problem. For example, for two-concept removal, the problem is formulated as follows:

\text{ s.t. }\frac{1}{\alpha_t} r_r(\alpha_t, \boldsymbol{d}_t) \geq - \varepsilon_t$$ The corresponding first-order Taylor expansion derived from this problem is: $$\max_{\boldsymbol{d}_t} \min [ \nabla \ell_u^1(\boldsymbol{\theta}_t) \cdot \boldsymbol{d}_t ,\nabla \ell_u^2(\boldsymbol{\theta}_t) \cdot \boldsymbol{d}_t ]- \frac{1}{2} \|\|\boldsymbol{d}_t\|\|^2 \text{ s.t. } \nabla \ell_r(\boldsymbol{\theta}_t) \cdot \boldsymbol{d}_t \geq - \varepsilon_t$$ The solution to this involves applying unilateral gradient surgery to each unlearning gradient ($\nabla \ell_u^1$ and $\nabla \ell_u^2$) and then aggregating the resulting gradients via a minimum norm operation, similar to MGDA [3]. The process is the same for more concepts. Importantly, this multi-concept explicit surgery can also utilize the approximation method from our paper to avoid the cost of multiple backpropagations. We will add the above discussion in the next version. [1]Lin, Shen, et al. "GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients." Proceedings of the 32nd ACM International Conference on Multimedia. 2024. [2] Qu, Youyang, et al. "Learn to Unlearn: A Survey on Machine Unlearning." *arXiv preprint arXiv:2305.07512* (2023). [3] Sener, Ozan, and Vladlen Koltun. "Multi-Task Learning as Multi-Objective Optimization." In *Advances in Neural Information Processing Systems 31* (NeurIPS), 2018. [4] Lu, Shilin, et al. "MACE: Mass Concept Erasure in Diffusion Models." In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024. [5] Gandikota, Rohit, et al. "Unified Concept Editing in Diffusion Models." In *Proceedings of the Winter Conference on Applications of Computer Vision (WACV)*, 2024.

Dear Reviewer,

Thank you for engaging with our rebuttal and acknowledging our strengths and contributions. Your insightful feedback has materially improved the paper.

Best,

Authors

We would like to thank Reviewer DBBQ for acknowledging our novelty and contributions. We address the concerns as following:

W1 & Q1: The idea of leveraging gradient conflict mitigation for unlearning has been explored in prior work, which somewhat limits the novelty of this paper

In fact, the starting point of our method is not gradient conflict mitigation. Instead, we approach the problem from the perspective of how to fully optimize the unlearning process under the constraint of utility preservation. We start by formulating a novel constrained optimization problem: the Utility-Preserving Unlearning Problem (UPUP).

Although the resulting gradient update rule bears resemblance to methods like gradient surgery, our starting point is more fundamental. This foundational approach allows us to derive theoretical insights. For instance, Remark 3.3 provides an estimate of the extent to which utility is degraded during the unlearning process. Furthermore, Remark 3.6, from the perspective of Pareto Optimality, demonstrates that the unlearning objective can be fully optimized under the utility constraint.

Additionally, we believe that our starting point and analytical framework can provide more fundamental research avenues for this field. For example, while our current solution to the UPUP problem (Equation 2) employs a first-order Taylor expansion, our formulation naturally extends to second-order expansions. This could lead to the development of second-order or quasi-second-order unlearning algorithms. As another example, from a regularization standpoint, one could apply customized constraints to the update direction $d_t$, such as incorporating user preferences into the UPUP formulation, which would also lead to novel algorithms.

We thank Reviewer DBBQ for pointing out some missing related works:

• Compared to GDR-GMA[1], this method stems from a multi-task learning (MTL) perspective, identifies issues like gradient conflict, and proposes heuristic fixes. While EUPMU is derived from the proposed UPUP optimization problem. Additionally, GDR-GMA [1] uses separate handcrafted modules for rectification (GDR) and weighting (GMA). Hence, GDR‑GMA needs two backward passes, adding approximately 30% more compute time with the same amount of data as EUPMU, which requires only one backward pass.
• Compared to G-effect [2], their "G-effect" toolkit provides a powerful gradient-based framework for understanding unlearning. The primary distinction lies in the objective: G-effect is a novel analytical toolkit designed to diagnose and understand the impacts of existing unlearning objectives. In contrast, our EUPMU is a novel constructive algorithm derived from a first-principles constrained optimization problem (UPUP). Our work provides an explicit, controllable mechanism ($\varepsilon$) to navigate the utility-unlearning trade-off, which is a constructive solution motivated by the kind of insights G-effect aims to provide. We will add a detailed discussion of this complementary work to our related works section.

Therefore, we argue that the principle we propose is fundamentally different from existing gradient conflict mitigation methods. Rather than being a surface-level application of gradient projection, our foundational framework is more capable of inspiring and guiding future research. We will also add more discussion on recent related works in the new version.

W2 & Q3: The paper lacks ablation studies on key hyperparameters, such as the error tolerance. Although the authors mention in the limitations section that related results are shown in Figure 7, I could not find any description in Figure 7 pertaining to error tolerance.

To address the concern about the lack of ablation studies, we have conducted experiments on the key hyperparameters, including the error tolerance $\varepsilon$. We present the results below and will add a detailed discussion to the appendix of our paper.

• Sensitivity to the error tolerance $\varepsilon$

We analyzed the impact of the error tolerance $\varepsilon$, which controls the trade-off between model utility and unlearning effectiveness. The results are shown in the table below:

As shown, increasing $\varepsilon$ leads to a lower Retain Loss (better utility) at the cost of a higher Forget Loss (less effective unlearning). This demonstrates that $\varepsilon$ serves as a controllable lever to navigate the Pareto front of the utility-unlearning trade-off. This allows practitioners to tune the algorithm based on specific downstream requirements, which is consistent with our theoretical analysis in Remark 3.3 and Remark 3.6.

• Sensitivity to the step-size $\beta$

We also investigated the sensitivity to the step-size $\beta$, which is used in the gradient approximation for solving $\lambda$.

The results indicate that our method is robust to the choice of $\beta$. The overall performance shows minimal variation across a wide range of values. This empirical finding aligns with our convergence analysis in Theorem 3.4, which proves that the algorithm converges to a Pareto-optimal solution as long as $\beta$ is chosen within a theoretically sound range.

Furthermore, the selection of these hyperparameters is not arbitrary but is guided by our theory. The search range for $\varepsilon$ can be informed by Remark 3.3 and Theorem 3.7, while Theorem 3.4 provides guidance for selecting a suitable $\beta$ to ensure convergence. We will add these results and a comprehensive discussion to the revised paper. Thank you for helping us improve the clarity and completeness of our work.

W3 & Q2: The evaluation metrics for the classification tasks may not fully align with the goals of unlearning.

We thank the reviewer for this insightful comment. We agree that relying solely on individual classification metrics can be insufficient, as the core challenge of unlearning lies in the trade-off between forgetting efficacy and retaining utility.

To align with the goals of unlearning, the community often measures a method's performance against a "Retrain Oracle"—a model retrained from scratch on only the retain data.

However, we argue that the gap-to-retrain metric could be fundamentally flawed in its practical application. In reality, most methods do not have any predictive capability about what the retrained model will look like. What ends up happening is a kind of “target painting after the shot”—researchers run the retrained model first, then retroactively select a run from their own method that happens to be close to it.

Therefore, in our work, we tuned our hyperparameters strictly based on achieving the best practical performance—maximizing Retain/Test Accuracy while minimizing Forget Accuracy and MIA score—without any knowledge of the Retrain Oracle.

Here’s the table with the gap to Retrain Oracle shown in parentheses for every metric, plus an Avg Gap column (mean of the four gaps per method), this is class-wise unlearning for Resnet18 on Cifar10:

We also tested Resnet 18 unlearning on Cifar 100, random data forgetting 10%.

Despite this more rigorous and realistic tuning methodology, our method proves to be the most aligned with the theoretical ideal.

If we are going to tune the model to minimize the gap to retrain, we can also search for the best $\varepsilon$ value to balance the unlearning benchmark to a desired value while keeping the retaining performance as close as possible to the retrain.

[1]Lin, Shen, et al. "GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

[2]Wang, Qizhou, et al. "Rethinking llm unlearning objectives: A gradient perspective and go beyond." arXiv preprint arXiv:2502.19301 (2025).