Unlearning as Multi-Task Optimization: a normalized gradient difference approach with adaptive learning rate

We formulate the unlearning as two-task optimization (forgetting and retaining), for which we apply the normalized gradient difference and automatic learning rate schedule.