DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers

Thanks for acknowledging that our method is simple yet effective, and the experiments are comprehensive. We provide the point-to-point responses to your concerns as follows:

---

W1: Assumption 3.2 treats forget-retain gradient correlation as negligible, yet Figure 4’s cosine-similarity plot is not a rigorous test of that claim.

Reply: We are sorry about the typo in line 153, we will replace $\rho(\widehat{g_{f,t_1}},\widehat{g_{r,t_2}})\simeq 0$ with $\rho(\widehat{g_{f,t_1}},\widehat{g_{r,t_2}}) \leq o(\tau) \simeq 0$ in the revision. We only assume that the correlation between forget gradient and retain gradient in different time steps is negligible compared with the correlation among forget gradients or the correlation among the retain gradients. In the proof, we only utilize the assumption $\rho(\widehat{g_{f,t_1}}, \widehat{g_{r,t_2}}) \leq 0$ for notation simplicity. This encompasses the situation that the forgetting gradient is negatively correalted with the retaining gradient. Therefore, it does not change the conclusion of our theorem.

Moreover, to make the analysis more rigorous, we can also replace the last assumption as $\rho(\widehat{g_{f,t_1}},\widehat{g_{r,t_2}}) \leq \xi \ll \tau$, the conclusion still hold, because the term of $\overline{\mathrm{Var}}$ will be dominated by the term containing $\tau$.

Thanks for pointing it out again, we will revise this part accordingly.

---

W2: The paper’s overall idea is slightly too incremental, the impact at conceptual level is limited.

Reply: We would like to highlight that our method is simple but effective. We believe that the coupled momentum factor is one of the fundamental reasons leading to the instability of MU methods. Empirical and theoretical evidences demonstrate effectiveness of using dual optimizers in stabilizing the MU process and improving the unlearning performance. Furthermore, our method is not task-specific, making it a versatile approach to empower existing MU algorithms. Therefore, we believe that our method can be broadly applied.

---

W3: More advanced optimizers and their combinations could be included.

Reply: In Table 5, we include the results of different combinations of Lion optimizer and SGD. In addition, we report the results of the combinations of recently popular Muon [1] and SGD under the same setting of Table 5 below. We can observe that Muon also exhibits low variability, but does not achieve the optimal performance compared with Adam, while the best combination: Adam(F)+SGD(R) reduces the average gap to 0.44 with an average std of 0.66.

---

Q1: Could the authors quantify wall-clock overhead and GPU memory?

Reply: Thanks for your suggestion, we report the wall-clock time and peak memory usage of Adam and DualOptim on Llama 2 with full parameter tuning and LoRA below. We can observe that under the configuration of full parameter tuning, the memory usage ratio is closer to 1 (Adam) : 1.5 (DualOptim). Nonetheless, under the LoRA configuration, the memory usage ratio is very close to 1:1 since the number of trainable parameters is significantly reduced. Additionally, the time usage ratio varies across different configurations and specific MU algorithms, but DualOptim takes 1~1.5x more time than Adam in general.

While the memory overhead introduced by our method is not negligible for LLMs, we can adopt parameter-efficient fine-tuning (PEFT) methods, such as LoRA, to significantly reduce the memory usage of optimizer states. As shown in Table 15, unlearning with LoRA can also induce competitive performance compared to unlearning with full parameters. We believe the development of PEFT will also broaden the application scenarios of our method.

Remark: For LoRA, we used 1× H20 GPU; for full parameter tuning, we used 2× H20 GPUs. We implement DualOptim based on bitsandbytes (bnb) library, because it can be easier to maintain the optimizer states in the framework of transformers library, while Adam is implemented based on original pytorch. Since bnb has specially optimized the implementation of optimizers and LoRA significantly decreases the number of trainable parameters, the memory usages of DualOptim is even smaller than that of Adam under the LoRA configuration.

Time Usage (s)

Memory Usage (GB/GPU)

---

Q2 : How sensitive is DualOptim when the pre-training optimizer is already adaptive?

Reply: We included the results of the model pretriained with Adam in Table 17. The results show that when the pre-training optimizer is Adam, the best combination is Adam(F) + Adam(R), which is consistent with the clam that we use the same optimizer as in pretraining to maintain the performance on the retain set and use the optimizer with adaptive learning rates to handle the large gradient variation for the forget loss.

---

Q3 : Typo in Section A.2 Proof of Theorem 3.5 Equation (11)

Reply: Thanks for pointing this out. We will correct it in the revision.

---

We hope our responses can address your concerns in the review above and look forward to your feedback.

[1] Keller Jordan et al. Muon: An optimizer for hidden layers in neural networks.

We really appreciate you acknowledge that our work is sufficient for acceptance. We provide point-to-point responses to your concerns as follows:

---

W1: The paper does not clearly separate its contribution from multi-task and bilevel learning

Reply: Thanks for pointing this out. We will add a brief discussion in our revised manuscript. From the optimisation perspective, empirical MU methods aims to optimise two objectives (1) minimize the loss on the retain set; (2) maximize the loss on the forget set. The second objective is unbounded, which is different from most multi-objective training problems which aim to optimize a single objective function that is a combination (e.g., a weighted sum) of the losses from all tasks. Therefore, MU methods are quite different from multi-objective training ones. Bilevel learning is defined by a hierarchical structure with an upper-level and a lower-level optimization problem. The most popular and competitive MU methods do not formulate the MU tasks as bilevel optimization problems.

Although these three tasks share some commonalities, our method is specifically designed to mitigate the unstability caused by two objectives with conflict in MU tasks.

---

W2: Theoretical assumptions like uncorrelated forget/retain gradients may not hold in practice, and the paper provides little guidance on when DualOptim could underperform.

Reply: We are sorry about the typo in line 153, we will replace $\rho(\widehat{g_{f,t_1}}, \widehat{g_{r,t_2}})\simeq 0$ with $\rho(\widehat{g_{f,t_1}}, \widehat{g_{r,t_2}}) \leq o(\tau) \simeq 0$ in the revision. We only assume that the correlation between forget gradient and retain gradient in different time steps is negligible compared with the correlation among forget gradients or the correlation among the retain gradients. In the proof, we only utilize the assumption $\rho$\widehat{g_{f,t_1}}, \widehat{g_{r,t_2}}$\leq 0$ for notation simplicity. This encompasses the situation that the forgetting gradient is negatively correlated with the retaining gradient. Therefore, it does not change the conclusion of our theorem.

Moreover, to make the analysis more rigorous, we can also replace the last assumption as $\rho(\widehat{g_{f,t_1}}, \widehat{g_{r,t_2}}) \leq \xi \ll \tau$, the conclusion still hold, because the term of $\overline{\mathrm{Var}}$ will be dominated by the term containing $\tau$. Thanks for pointing it out again, we will revise this part accordingly.

As for the second point, according to Eq. 5 and Theorem 3.5, when the momentum coefficient $\alpha=0$, i.e., no momentum, DualOptim is equivalent to a single optimizer. However, modern optimizers always incorporate momentum for better optimization. Additionally, our experiments show that our method consistently provides better performance in diverse tasks.

---

W3: The main text offers no quantitative overhead analysis, especially for large models.

Reply: Thanks for your suggestion, we report the wall-clock time and peak memory usage of Adam and DualOptim on Llama 2 with full parameter tuning and LoRA below. We can observe that under the configuration of full parameter tuning, the memory usage ratio is closer to 1 (Adam) : 1.5 (DualOptim). Nonetheless, under the LoRA configuration, the memory usage ratio is very close to 1:1 since the number of trainable parameters is significantly reduced. Additionally, the time usage ratio varies across different configurations and specific MU algorithms, but DualOptim takes 2% to 50% more time than Adam in general.

While the memory overhead introduced by our method is not negligible for LLMs, we can adopt parameter-efficient fine-tuning (PEFT) methods, such as LoRA, to significantly reduce the memory usage of optimizer states. As shown in Table 15, unlearning with LoRA can also induce competitive performance compared to unlearning with full parameters. We believe the development of PEFT will also broaden the application scenarios of our method.

Remark: For LoRA, we used 1× H20 GPU; for full parameter tuning, we used 2× H20 GPUs. We implement DualOptim based on bitsandbytes (bnb) library, because it can be easier to maintain the optimizer states in the framework of transformers library, while Adam is implemented based on original pytorch. Since bnb has specially optimized the implementation of optimizers and LoRA significantly decreases the number of trainable parameters, the memory usages of DualOptim is even smaller than that of Adam under the LoRA configuration.

Time Usage (s)

Memory Usage (GB/GPU)

---

Q1: Are there examples where decoupling the optimizers actually hurts convergence?

Reply: We did not observe such examples so far.

---

Q2: Can you provide concrete measurements of memory usage and run-time overhead when scaling to large LLMs? How do these costs compare with single-optimizer baselines?

Reply: Please see the response to W3.

---

Q3:Have you evaluated DualOptim against automatic hyper-parameter scheduling techniques designed to stabilize MU?

Reply: It should be noted that the hyperparameters of all evaluated methods are tuned through a sophisticated search (details in the responses to W2 for reviewer UZKz and W3 for reviewer wno1), so the results exhibit the optimal performance of the evaluated methods. Moreover, automatic scheduling techniques are embedded in specific MU methods [1,2] and are orthogonal to our method. Therefore, they are not included in our comparison.

---

We hope our responses can address your concerns.

[1] Chongyu Fan et al. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. ICLR 2024.

[2] Zhehao Huang et al. Unified gradient-based machine unlearning with remain geometry enhancement. NeurIPS 2024.

Thanks for acknowledging that our paper is technically sound, well-written and the experiments are extensive. We provide point-to-point responses to your concerns as follows:

---

W1: While the authors discuss this limitation, further optimization strategies could be explored

Reply: For your reference, we report the wall-clock time and peak memory usage of Adam and DualOptim on Llama 2 with full parameter tuning and LoRA below. We can observe that under the configuration of full parameter tuning, the memory usage ratio is closer to 1 (Adam) : 1.5 (DualOptim). Nonetheless, under the LoRA configuration, the memory usage ratio is very close to 1:1 since the number of trainable parameters is significantly reduced. Additionally, the time usage ratio varies across different configurations and specific MU algorithms, but DualOptim takes 1~1.5x more time than Adam in general.

While the memory overhead introduced by our method is not negligible for LLMs, we can adopt parameter-efficient fine-tuning (PEFT) methods, such as LoRA, to significantly reduce the memory usage of optimizer states. As shown in Table 15, unlearning with LoRA can also induce competitive performance compared to unlearning with full parameters. We believe the development of PEFT will also broaden the application scenarios of our method.

Remark: For LoRA, we used 1× H20 GPU; for full parameter tuning, we used 2× H20 GPUs. We implement DualOptim based on bitsandbytes (bnb) library, because it can be easier to maintain the optimizer states in the framework of transformers library, while Adam is implemented based on original pytorch. Since bnb has specially optimized the implementation of optimizers and LoRA significantly decreases the number of trainable parameters, the memory usages of DualOptim is even smaller than that of Adam under the LoRA configuration.

Time Usage (s)

Memory Usage (GB/GPU)

---

W2: It lacks a detailed analysis of the sensitivity of its own hyperparameters. Additionally, the experiments compare DualOptim with other methods using their default hyperparameters, which may not represent the optimal performance of these baselines.

Reply:

• Thanks for your suggestion. We report the performance of SFRon and SFRon+DualOptim with different forget learning rates below (we fix retaining learning rate to 0.01). It can be observed that our method significantly stabalizes the unlearning process and achieves smaller gap to retrained models in a wide range of hyperparameter choices.
• We would like to clarify that for all baselines, we search for their optimal learning rates in a wide range and a fine granularity (all implementation details are shown in Appendix B). For SalUn and SFRon which use separate learning rates for retain and forget sets, we fix the retaining learning rate to a reasonable value and tune the forgetting learning rate. Specifically, we start with a reasonable learning rate that induces effective unlearning, and search the optimal one at a granularity that is an order of magnitude smaller than the starting value. For instance, if the starting value is 1e-4, the search range will be [5e-5, 6e-5, 7e-5, 8e-5, 9e-5, 1e-4, 1.1e-4, 1.2e-4, 1.3e-4, 1.4e-4, 1.5e-4]. Note that, since the reasonable starting learning rates of different methods vary a lot, we do not adopt the unified search range for all baselines. Therefore, all baselines are carefully tuned to achieve the near-optimal performance. We will include this detail in the revised version.

SFRon

SFRon+DualOptim

---

We hope our responses can address your concerns in the review above and look forward to your feedback.

Thanks for acknowledging our idea is clear and the experiments are comprehensive. We provide point-to-point responses to your concerns as follows:

---

W1: The proof is only for SGD with momentum ($\eta=1$ is also hard-coded).

Reply:

• We would like to clarify that the learning rate $\eta$ is just a scaling factor in our analysis, setting it to 1 only simplifies the notation and does not lose the generality of our theorem at all.
• We focus on analyzing the effectiveness of decoupling itself in Sec. 3.3. The effectiveness of preconditioning is discussed in Sec. 3.2 through numerical experiments. Technically, it is quite challenging to rigorously bound the variance of multi-step Adam-optimized parameters. Nevertheless, our comprehensive experiments demonstrate the efficacy of the combination of preconditioning and decoupled momentum. Our ablation study demonstrate the effectiveness of using either preconditioning or decoupled momentum.

---

W2: Theorem 3.5 does not mathematically connect this reduced variance to better performance on downstream metrics

Reply: Thanks for pointing this out, we would like to provide further analysis as follows:

We let $M(\theta)$ to represent the performance of model with parameter $\theta$ where $M$ can be FA/RA/TA/MIA. Before we derive the variance bound of performance metric function $M(\theta)$, we make the following assumption:

Assumption A.1. The performance metric function $M$ is Lipschitz continuous: $\forall\theta_1,\theta_2, \|M(\theta_1)-M(\theta_2)\|\leq L_M\|\theta_1-\theta_2\|$, where $L_M$ is a non-negative constant.

Following a similar logic to 3.4, we have the following corollary:

Corollary A.2. If Assumption A.1 holds, and the parameter variance satisfies

$\mathrm{Var}(\theta)\leq\overline{\mathrm{Var}}(\theta)$, then we have $\mathrm{Var}(M(\theta))\leq L_M^2\overline{\mathrm{Var}}(\theta)$.

Proof. Similar to the proof of Lemma 3.4, we have: $\mathrm{Var}(M(\theta)) = \frac{1}{2}\iint p(\theta_1)p(\theta_2)\left\|M(\theta_1) - M(\theta_2)\right\|^2 d\theta_1 d\theta_2,$

where $\theta_1, \theta_2$ are independent copies of $\theta$ with probability density $p(\theta)$.

By $\|M(\theta_1)-M(\theta_2)\|\leq L_M\|\theta_1-\theta_2\|$, we have:

$\mathrm{Var}(M(\theta)) \leq \frac{L_M^2}{2}\iint p(\theta_1)p(\theta_2)\|\theta_1 - \theta_2\|^2 d\theta_1 d\theta_2.$

Since $\mathrm{Var}(\theta)\leq\overline{\mathrm{Var}}(\theta)$, we have $\frac{1}{2}\iint p(\theta_1)p(\theta_2)\|\theta_1 - \theta_2\|^2 d\theta_1 d\theta_2 \leq \overline{\mathrm{Var}}(\theta)$. Therefore, we can conclude $\mathrm{Var}(M(\theta)) \leq L_M^2 \overline{\mathrm{Var}}(\theta)$.

Discussion. The final unlearned parameters are usually the parameter updated by retaining loss at the last iteration $T$, i.e., $\theta^D_{r,T}$ using decoupled momentum and $\theta^S_{r,T}$ using shared momentum. According to Theorem 3.5, we have $\overline{\mathrm{Var}}(\theta^D_{r,T}) \leq \overline{\mathrm{Var}}(\theta^S_{r,T})$. Let the upper bound of the performance variance $\overline{\mathrm{Var}}(M(\theta))=L_M^2 \overline{\mathrm{Var}}(\theta)$, we can derive $\overline{\mathrm{Var}}(M(\theta^D_{r,T})) \leq \overline{\mathrm{Var}}(M(\theta^S_{r,T}))$. Thus, decoupled momentum can also induce less variability on performance metrics.

---

W3: The paper does not detail if the search budget was equal for the baselines versus DualOptim, which introduces two separate learning rates

Reply: Thanks for your constructive comment. Competitive baselines such as SaLUn and SFRon also use two separate learning rates but a shared momentum factor. In such cases, we fix the retaining learning rate $\eta_r=1e-2$ when using SGD or $\eta_r = 1e-4$ when using Adam, and search for the optimal forgetting learning rate in a wide range and a fine granularity. Specifically, we start with a reasonable learning rate that induces effective unlearning, and carefully search for the optimal value in a similar order of magnitude. For instance, if the starting value is 1e-4, the search range will be [5e-5, 6e-5, 7e-5, 8e-5, 9e-5, 1e-4, 1.1e-4, 1.2e-4, 1.3e-4, 1.4e-4, 1.5e-4]. Note that, since the reasonable starting learning rates of different methods vary a lot, we do not adopt the unified search range for all baselines.

For baselines with a unified learning rate, we follow a similar strategy as above to start with a reasonable learning rate and carefully search for the optimal one. In summary, all baselines are carefully tuned to achieve the near-optimal performance. We will include this detail in the revised version.

---

W4: The main results only compare DualOptim's alternating update schedule with baselines that sometimes use a joint optimization objective.

Reply: Thanks for pointing this out. In Sec. 4.3, we aim to compare our method with unmodified baselines that were originally designed to use a joint optimization objective. Nonetheless, we also include the results of their alternating variant for a more comprehensive comparison. The results of Table 14 of Appendix C.5 show that DualOptim outperforms the original baselines and their alternating variants. We will highlight this point in the main text of the revised manuscript to avoid confusion.

---

Q1: Report the wall-clock time and peak memory usage for DualOptim compared to a key baseline on a representative task

Reply: Thanks for your suggestion, we report the wall-clock time and peak memory usage of Adam and DualOptim on Llama 2 with full parameter tuning and LoRA below. We can observe that under the configuration of full parameter tuning, the memory usage ratio is closer to 1 (Adam) : 1.5 (DualOptim). Nonetheless, under the LoRA configuration, the memory usage ratio is very close to 1:1 since the number of trainable parameters is significantly reduced. Additionally, the time usage ratio varies across different configurations and specific MU algorithms, but DualOptim takes 2% to 50% more time than Adam in general.

While the memory overhead introduced by our method is not negligible for LLMs, we can adopt parameter-efficient fine-tuning (PEFT) methods, such as LoRA, to significantly reduce the memory usage of optimizer states. As shown in Table 15, unlearning with LoRA can also induce competitive performance compared to unlearning with full parameters. We believe the development of PEFT will also broaden the application scenarios of our method.

Remark: For LoRA, we used 1× H20 GPU; for full parameter tuning, we used 2× H20 GPUs. We implement DualOptim based on bitsandbytes (bnb) library, because it can be easier to maintain the optimizer states in the framework of transformers library, while Adam is implemented based on original pytorch. Since bnb has specially optimized the implementation of optimizers and LoRA significantly decreases the number of trainable parameters, the memory usages of DualOptim is even smaller than that of Adam under the LoRA configuration.

Time Usage (s)

Memory Usage (GB/GPU)

---

Q2: Detail the hyperparameter tuning budget for both DualOptim and the baselines to ensure a fair comparison

Reply: Please see the response to W3.

---

Q3: address the mismatch between your theory (proven for SGD) and your algorithm (which uses Adam)

Reply: Please see the response to W1.

---

We hope our responses can address your concerns in the review above and look forward to your feedback.