Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Discrimination-Aware Multitask Optimization

W1: Incorporating quantitative metrics (such as FID) to evaluate the soundness of synthetic data could strengthen the paper.

We respectfully disagree with the reviewer’s suggestion. We emphasize that our goal is to generate synthetic data that approximates the original distribution while preserving privacy. Unlike model inversion methods, which aim to maximize image fidelity, our approach prioritizes unlearning effectiveness without reconstructing identifiable samples. Therefore, evaluating the synthetic data with metrics like FID, which emphasize visual quality, is not necessary for our objectives.

W2: Parameter sensitivity experiments w.r.t. weighting factors.

The weighting factors in our MAMO method are set to ensure that the three losses are on a similar scale. To analyze their sensitivity, we conduct experiments of one-class unlearning task on CIFAR-10. We fix the weight of the retain loss $w_R$ at 0.1 and vary $w_F$ and $w_{Disc}$ within {0.01, 0.1, 0.5, 1}, observing the unlearned model’s $A_r$ and $A_f$. The results in the table below show that when $w_F$ and $w_{Disc}$ are within the range of 0.1–0.5, the model's performance remains stable. However, an excessively large $w_F$ or a small $w_{Disc}$ leads to a noticeable decline in $A_r$.

W1: Adding a notation table will make the paper clearer.

We thank the reviewer for pointing out this issue. We will add a notation table in our final version.

Q1: Could the synthetic data cause potential privacy linkage?

The reviewer raises a critical concern. However, our work addresses this concern through an empirical analysis presented in Section 4.4. Specifically, we visualize the generated synthetic samples and demonstrate that they are visually indistinguishable from random noise, making it impossible for human observers to extract any meaningful information. This observation confirms that the synthetic data does not retain identifiable features of the original training data, thereby mitigating potential privacy risks.

Q2: Parameter sensitivity experiments w.r.t. weighting factors.

The weighting factors in our MAMO method are set to ensure that the three losses are on a similar scale. To analyze their sensitivity, we conduct experiments of one-class unlearning task on CIFAR-10. We fix the weight of the retain loss $w_R$ at 0.1 and vary $w_F$ and $w_{Disc}$ within {0.01, 0.1, 0.5, 1}, observing the unlearned model’s $A_r$ and $A_f$. The results in the table below show that when $w_F$ and $w_{Disc}$ are within the range of 0.1–0.5, the model's performance remains stable. However, an excessively large $w_F$ or a small $w_{Disc}$ leads to a noticeable decline in $A_r$.

W1: A deeper discussion on the synthetic data's limitations would strengthen the argument.

We appreciate the reviewer’s suggestion. Our experiments on three datasets with 10, 100, and 105 classes demonstrate that the synthetic data effectively supports unlearning across diverse distributions. As shown in Section 4.4, the visualized feature distributions further confirm that the generated data closely approximates real data across various classes. However, we acknowledge that for certain distributions, such as highly sparse or imbalanced data, our method may not fully capture fine-grained structural details. Addressing this limitation is an important direction for future research.

W2: The legend in Figure 5 (a) appears to be small, which may affect readability. Additionally, the color contrast could be improved to enhance clarity.

We thank the reviewer for pointing out this issue. We will optimize Figure 5 (a) in our final version.

W1: This mechanism may be limited on high-resolution data.

Thank you for the comment. However, we respectfully disagree. Our method is not restricted to high-resolution data, as demonstrated by our experiments on three datasets with varying resolutions—CIFAR-10 (32×32), CIFAR-100 (32×32), and PinsFaceRecognition (224×224). The consistent performance across both low- and high-resolution scenarios validates the broad applicability of our method.

W2&C3: The proposed framework is limited to class-wise forgetting.

We appreciate the reviewer’s insight. Our method is designed for class-wise unlearning, as it aligns with common unlearning scenarios and is more practical in real-world applications. Most existing unlearning studies also focus on class-level or concept-level unlearning, as instance-level unlearning poses inherent challenges—removing a specific instance does not guarantee that the model will completely forget similar patterns, as retain data with similar features may still contribute to strong performance on the forgot data. While our method is effective in removing class-level information, we recognize its limitations in finer-grained unlearning, such as instance-wise or attribute-wise unlearning, which we will clarify in our final version.

C1: Why not adopting contrastive alignment for the discriminative feature alignment objective?

The contrastive alignment methods in [1-2] employ a metric learning loss that is fundamentally similar to our $L_{disc}$ without the weight factors $\alpha$. However, our method introduces $\alpha$ to adaptively adjust gradient magnitudes based on how far a similarity score deviates from its optimal values. This adaptive weighting mechanism provides two key advantages. First, it prioritizes large updates for highly misaligned features, accelerating convergence and improving optimization efficiency. Second, it prevents excessive penalization of already well-aligned features, reducing the risk of overcorrection. As a result, our method achieves more precise and stable discriminative feature alignment, ultimately enhancing unlearning effectiveness.

C2: Explain rationale behind the choice of the solution to gradient conflicts.

The method in [3] primarily balances two losses and requires a predefined distinction between the main and auxiliary objectives, limiting its applicability to more complex multi-objective optimization. In contrast, our approach can effectively balance three or more objectives without the need for a manually designated main objective, enabling a more flexible and adaptive optimization process. This allows for the seamless integration of the Discriminative Feature Alignment Objective and ensures a better trade-off among multiple competing objectives in unlearning.

Q1: Regarding the objective LF, what if DSDA apply random labelling? Would it help with feature alignment?

We appreciate the reviewer’s thought-provoking question. However, random labeling may not be an effective alternative for the forget objective $L_F$. As shown in Figure 2(b), most forget data samples are predicted as specific classes rather than randomly distributed across all classes. Additionally, prior research [1] has shown that fine-tuning a trained model with randomly labeled forget data can introduce unintended shifts in the decision boundaries of retain classes, ultimately degrading the model’s utility on the retain data. Therefore, the forget objective $L_F$, which explicitly optimizes unlearning through gradient ascent, is more controlled and effective in achieving the desired unlearning behavior.

[1] Chen, Min, et al. "Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Q2: What is the data range for the crafted data?

The crafted data follows a similar range to the preprocessed original data. For example, in the case of CIFAR-10, we apply a common normalization setting with CIFAR_MEAN = (0.5071, 0.4865, 0.4409) and CIFAR_STD = (0.2673, 0.2564, 0.2762), resulting in a transformed data range of approximately [−2,2]. Our synthetic data is generated within a slightly extended range of [−2.5,2.5], ensuring compatibility with the model while maintaining sufficient variability for effective unlearning.