Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models

We would like to first thank the reviewer for their detailed comments. We are glad that the reviewer finds our approach novel and valuable. Below, we address the specific concerns raised:

1. Question: The reviewer noted that the current presentation may obscure the true contributions of the paper and the rationale behind certain choices, such as "why SiD was selected over Diff-Instruct or DMD."

   Response: We would like to clarify that SiD was chosen not only for its superior stability and performance, as demonstrated in diffusion distillation in prior works, but also for its demonstrated compatibility with the machine unlearning (MU) objective. Specifically:

   • Empirically, we observed that Diff-Instruct training is less stable when combined with the MU objective, frequently leading to mode collapse. This limitation has also been explicitly acknowledged by the authors of Diff-Instruct in their recent work, "One-Step Diffusion Distillation through Score Implicit Matching" [2], where they state: "We find that DI converges fast but suffers from mode-collapse issues." This instability makes Diff-Instruct unsuitable for our framework.
   • DMD, on the other hand, augments Diff-Instruct with a perceptual regression loss (LPIPS-based) to counter mode collapse and improve image quality by enforcing perceptual similarity between generated and real images. While this approach helps mitigate Diff-Instruct's instability, DMD relies on access to teacher-synthesized image data during training. This dependency adds significant computational costs to prepare the synthesized data required for its training.
   • In contrast, SiD achieves stability without relying on real or teacher-synthesized data, making it the most efficient and compatible choice for developing our framework.

2. Question: The reviewer raised questions regarding how baseline results, some of which were not found in their original papers, were obtained.

   Response: We would like to clarify that the baseline results were derived following the class-forgetting task setting described in SalUn. However, since SalUn did not release pretrained CIFAR-10/STL-10 models or their fine-tuned model weights, we reproduced their results using the official codebase, experimental configurations, and scripts provided in the paper.

   For CIFAR-10, SalUn reported the following results in Table A5 of their Appendix:

   Methods	UA(↑)	FID(↓)
   Retrain	100.00	11.69
   ESD	100.00	17.37
   SalUn	100.00	11.21

   We reproduced these results with similar or better outcomes:

   Methods	UA(↑)	FID(↓)
   Retrain	98.50	7.94
   ESD	91.21	12.68
   SalUn	99.31	11.25

   The discrepancy in Unlearning Accuracy (UA) arises from differences in evaluation sample sizes. SalUn evaluated UA using only 500 samples from the forgotten class, which makes the metric less informative since all methods achieve 100% UA under such limited conditions. To provide more precise and reliable evaluations, we increased the sample size to 5,000. Similar adjustments were made to adapt the protocols for STL-10, ensuring consistency and fairness across all evaluations.

3. Question: The reviewer considered style forgetting and random data forgetting as standard choices for MU evaluation and questioned their absence in our study.

   Response: We believe there may be a misunderstanding here:

   • Random data forgetting is specific to image classification tasks and classification models, such as ResNet, whereas our work focuses on image generation and diffusion models. As a result, random data forgetting is not directly relevant to this study.
   • Regarding style forgetting, while it is used in some MU methods for image generation, we believe our experiments (e.g., Figure 1) already effectively demonstrate the unique advantages of SFD for MU in diffusion models, making additional style forgetting evaluations unnecessary for the scope of this paper.

[2] Luo, Weijian, et al. "One-step diffusion distillation through score implicit matching." arXiv preprint arXiv:2410.16794 (2024).

Dear Reviewer hpqv,

We sincerely thank you again for your detailed comments and valuable feedback. In our responses, we have provided clarifications on:

• the rationale for selecting our score distillation method;
• the sources of baseline results;
• the specific choices of MU experiments;
• the run-time efficiency and computational cost of pretraining;
• the comparison to the "forget first, then distill" approach.

With the discussion period nearing its end, we would greatly appreciate any remaining thoughts or questions you might have. If our responses have satisfactorily addressed your concerns, we kindly hope you will reconsider your evaluation of our work in light of the clarifications provided.

Warm regards,

Authors of Submission #5206

Dear Reviewer hpqv,

Thank you for your thoughtful feedback and engagement in the discussion. We are glad to hear that our responses to Q2–Q5 addressed your concerns.

Regarding Q1, we appreciate your suggestion to evaluate the proposed SFD framework using alternative diffusion distillation methods. Our initial focus was on integrating the model-based Fisher divergence into the SFD framework, driven by compelling evidence of its superior performance (e.g., SiD) over KL-divergence-based distillation methods (e.g., Diff-Instruct and DMD), as well as its ability to deliver strong results with the proposed modifications, effectively outperforming existing approaches. We agree that evaluating SFD with alternative methods could further demonstrate the robustness and versatility of our approach.

In response to your suggestion, we conducted a new experiment on CIFAR-10, implementing a variant of our framework called SFD-KL. This version modifies the model-based Fisher divergence-based loss and SiD-based gradient in SFD to use a KL-based loss and Diff-Instruct-based gradient. Below are the evaluation results:

Additional details are provided in the newly revised $\underline{\text{Appendix B.8}}$. These results highlight the flexibility of the SFD framework, demonstrating its ability to integrate alternative score distillation methods while maintaining strong performance with unlearned, one-step distilled diffusion models. Overall, the original SFD achieves superior results, reinforcing the rationale behind our original design choices.

We hope this new evaluation enhances the comprehensiveness and robustness of our study. If you have any further questions or suggestions for additional experiments, please don’t hesitate to let us know. We are fully committed to addressing any concerns from you and other reviewers throughout the discussion period.

Warm regards,

Authors of Submission #5206

Dear Reviewer hpqv,

If our response and revised manuscript have effectively addressed your concerns, we kindly request that you reevaluate our work based on the updated information. We would greatly appreciate your consideration in potentially raising your score.

Thank you for your time and thoughtful review.

Sincerely,

The Authors

We would like to thank the reviewer for their detailed and insightful comments on our work. We are glad that the reviewer regards SFD as a novel and creative approach backed by thorough evaluation and appreciates the practical contributions of our work to real-world applications. Below, we address the concerns raised:

1. Weakness: The contributions of the proposed method may be perceived as incremental, as it builds upon existing advancements in score distillation and machine unlearning.

   Response: Regarding the level of perceived contributions, we strongly emphasize that our work establishes a brand-new framework for machine unlearning (MU), introducing an efficient and effective solution to the long-standing challenge of data-free MU. This contribution is far from incremental. To the best of our knowledge, no prior work has sufficiently addressed the problem of data-free MU for diffusion models, a task rendered especially challenging due to the inefficiency of multistep sampling in traditional approaches. By leveraging score distillation for MU tasks, our method not only overcomes this barrier but also enables a practical solution that opens new avenues for future research. Beyond the immediate application, our framework provides a foundation for adapting unlearning techniques to other distilled diffusion models, significantly advancing the field.

2. Weakness: The authors do not sufficiently explain the specific advantages of integrating unlearning into score distillation and fail to compare with other one-step diffusion models or alternative multi-step baselines (e.g., SiD/DMD or diffusion solvers like DDIM).

   Response: We clarify the following:

   i. To the best of our knowledge, no existing MU methods, apart from our own, have been or can be directly applied to the distilled diffusion models mentioned by the reviewer. While we acknowledge the possibility that such methods could exist, demonstrating their applicability lies beyond the scope and burden of this paper. Specifically, existing finetuning-based MU methods are tailored for standard multistep diffusion models and rely on modifying the diffusion loss (e.g., MSE) to erase "unsafe" classes or concepts. In contrast, distilled diffusion models are not trained using these loss formulations, making such methods inherently incompatible.

   ii. Distilling an unlearned diffusion model is possible but inherently limited by the performance of the teacher model. For example, a baseline MU method applied to a standard diffusion model would produce a teacher model that our method already outperforms in terms of unlearning accuracy and generation quality. While distillation alone can accelerate sampling speed, it cannot address the fundamental limitations inherent in the teacher model.

   iii. Using diffusion solvers with fewer steps, such as DDIM or DPM-Solver++, can reduce sampling time but does not approach the efficiency of our one-step SFD. These solvers also come at the cost of degraded sample quality, as noted by the reviewer. In contrast, SFD achieves significant advantages across sampling speed, unlearning accuracy, and generation quality, demonstrating a comprehensive superiority over alternative approaches.

3. Weakness: The paper does not provide an analysis of the algorithm's runtime or computational efficiency.

   Response: We clarify that SFD finetuning is highly efficient, requiring far fewer steps than pretraining. For instance, pretraining the CIFAR-10 DDPM model required 800K steps, while SFD finetuning required only 50K steps. Furthermore, our two-stage approach, SFD-Two Stage, achieves competitive performance in just 1.5K steps. For Stable Diffusion models, SFD required only 25K steps (celebrity forgetting) and 37K steps (nudity forgetting) with a batch size of 8. These results demonstrate the efficiency of our approach compared to retraining from scratch, making SFD practical for real-world applications.

Dear Reviewer 5B7n,

We sincerely thank you again for your valuable suggestions and detailed feedback. To address your concerns, we have provided clarifications on:

• the significance of our method's contributions;
• the distinctive advantages of our approach;
• the efficiency of our fine-tuning process;
• the primary focus of our approach and its potential for other tasks;
• the consistency of our experimental results.

With the discussion period nearing its end, we would greatly appreciate any remaining thoughts or questions you might have. If our responses have addressed your concerns, we kindly hope you will reconsider your evaluation of our work in light of the clarifications provided.

Warm regards,

Authors of Submission #5206

Dear Reviewer 5B7n,

We hope this message finds you well. While we have not yet received your feedback during the rebuttal, we sincerely hope you have had a chance to review our responses and the provided revisions. We are eager to use the discussion period to address any remaining or new concerns you may have and incorporate them into our revisions.

If our responses and the revised manuscript have adequately addressed your questions, we kindly request that you consider reevaluating our work based on the updated information. Thank you for your time and thoughtful consideration.

Sincerely,

The Authors

We sincerely thank the reviewer for their thoughtful feedback and for highlighting important aspects to strengthen our work. Below, we provide detailed responses, emphasizing the unique strengths of our approach while addressing potential misunderstandings.

1. Weakness: There is no related work section to highlight the differences among existing works.

   Response: Thank you for suggesting a related work section to highlight our method's novelty and differences among existing works. We have included a dedicated related work section in $\underline{\text{Appendix A}}$ to discuss these differences in the revised manuscript.

2. Weakness: Evaluation under adversarial prompt attacks, such as UnlearnDiffAtk [1], is missing, even though it is a common evaluation tool for unlearned diffusion models.

   Response: Thank you for bringing up the importance of adversarial prompt attack evaluations. While SFD was not explicitly designed to counter such attacks, it demonstrates strong inherent robustness against UnlearnDiffAtk [1], significantly outperforming all other unlearned UNet-based baselines in terms of attack success rate (ASR). This highlights the robustness benefits of our UNet finetuning approach, even without specialized adversarial defenses.

   It is important to note that adversarial prompt attacks like UnlearnDiffAtk primarily exploit vulnerabilities in the text encoder. As SFD focuses on finetuning the UNet, the unchanged text encoder remains susceptible to these attacks. However, this does not undermine the contributions of our method: SFD provides a data-free, efficient, and robust unlearning solution with a one-step sampling process that delivers unmatched generation speed compared to multi-step baselines. Furthermore, our method is orthogonal to approaches like AdvUnlearn [2], which focus on finetuning the text encoder. This orthogonality opens the door for complementary integration. Specifically, we can directly combine AdvUnlearn’s defended text encoder with SFD’s UNet finetuning to further enhance robustness, achieving state-of-the-art performance. We refer to the version of SFD with its text encoder replaced by that of AdvUnlearn-TE as SFD-TE. Below is a comparison of the attack success rate (ASR↓), where lower values indicate better robustness:

   Scenario	ESD	FMN	SLD	AdvUnlearn-UN	AdvUnlearn-TE	SFD (ours)	SFD+TE (ours)
   No attack	20.42%	88.03%	33.10%	-	7.75%	7.04%	0.70%
   UnlearnDiffAtk [1]	76.05%	97.89%	82.39%	64.79%	21.13%	55.63%	7.04%

   Key observations:

   • SFD achieves the best robustness among all unlearned UNet models under adversarial conditions.
   • SFD+TE further improves robustness, reducing ASR to 0.70% in the absence of attacks and maintaining strong performance with an ASR of 7.04% under UnlearnDiffAtk.

   These results validate SFD’s robustness as a standalone method while demonstrating its potential for seamless integration with complementary approaches like AdvUnlearn. Crucially, this integration does not compromise SFD’s simplicity or its significant efficiency advantage over multi-step methods.

3. Weakness: A comparison of generation times is missing, which is essential to demonstrate the efficiency of the proposed one-step generator.

   Response: Thank you for raising this point. SFD’s one-step generator offers unmatched efficiency compared to multi-step baselines. Specifically, for text-to-image generation based on Stable Diffusion, SFD generates an image in approximately 0.14 seconds per prompt, whereas finetuning-based multi-step models require approximately ~5.46 seconds per image per prompt using a 50-step DDIM sampler and classifier-free guidance with a scale of 7.5. All benchmarks were conducted on a single $\underline{\text{24GB NVIDIA RTX A5000 GPU}}$. This dramatic reduction in generation time underscores the practicality of SFD for real-world applications, combining high efficiency with competitive robustness and quality.

Dear Reviewer KevT,

We sincerely thank you again for your valuable feedback and suggestions. In response, we have conducted additional experiments, revised our manuscript, and included new results to address the points you raised. With the discussion period nearing its end, we would greatly appreciate any additional thoughts or questions you might have. We hope that our efforts have clarified the key aspects of our work and positively informed your ongoing evaluation.

Warm regards,

Authors of Submission #5206

We sincerely thank the reviewer for their detailed and insightful comments on our work. We are delighted that the reviewer recognizes the novelty of our pioneering approach to machine unlearning from the perspective of score distillation, as well as the clarity of our presentation and the comprehensiveness of our experiments.

Below, we present detailed clarifications and new evidence to address the two key weaknesses and questions raised by the reviewer. These results, which will be incorporated into our revision, further confirm that our unlearning procedure is not only fast—particularly when leveraging a pre-distilled generator—but also complements existing unlearning methods applied to the text encoder and excels in the recommended robustness evaluations, demonstrating its effectiveness and practicality.

1. Weakness: The reviewer raised a valid concern regarding the lack of detailed explanations on runtime, especially given our ambitious goal of simultaneously accelerating diffusion models and unlearning "unsafe" classes or concepts effectively.

   Response: We would like to emphasize that SFD fine-tuning incurs a negligible runtime compared to pretraining. For instance, training a CIFAR-10 class-conditional diffusion model required 800K steps, whereas SFD fine-tuning needed only 50K steps. Furthermore, we propose a two-stage approach (i.e., first distill, then forget), referred to as "SFD-Two Stage" in Section 3 of the paper, which offers even greater efficiency when a pre-distilled generator is available. Starting with a pre-distilled diffusion model (via SiD), SFD fine-tuning can further achieve over a 10× acceleration, requiring only 1.5K steps to attain strong performance. Specifically, as demonstrated in $\underline{\text{Figure 7 of Appendix B.3}}$, SFD-Two Stage on CIFAR-10 achieves a Unlearning Accuracy (UA) of 99.5% and a Fréchet Inception Distance (FID) of 5.73 with as few as 1.5K steps. These results underscore the practicality and computational efficiency of our method for large-scale unlearning tasks.

2. Weakness: The reviewer highlighted the lack of robustness evaluation against adversarial prompts as a key weakness of our paper.

   Response: We deeply appreciate this suggestion and agree that such evaluations can further strengthen the case for the effectiveness of SFD. Adversarial prompts indeed present a significant challenge to unlearned models. However, due to the nature of our method—distilling a multistep denoising diffusion model into a one-step generator—certain adversarial attacks proposed in prior studies, such as circumventing concept erasure [2], may not directly apply to our framework. Nonetheless, we have conducted additional experiments using UnlearnDiffAtk [1] to evaluate the robustness of our method. Below, we present a comparison of the attack success rate (ASR), where lower values indicate better robustness.

   Scenario	ESD	FMN	SLD	AdvUnlearn-UN	AdvUnlearn-TE	SFD (ours)	SFD+TE (ours)
   No attack	20.42%	88.03%	33.10%	-	7.75%	7.04%	0.70%
   UnlearnDiffAtk [1]	76.05%	97.89%	82.39%	64.79%	21.13%	55.63%	7.04%

   Importantly, we also emphasize that complementary approaches, such as adversarial training on the text encoder, can be seamlessly integrated with SFD to further enhance its robustness. Specifically, we introduce SFD+TE, which replaces the frozen text encoder in SFD with the adversarially trained text encoder from AdvUnlearn-TE [3]. In this table, AdvUnlearn [3] refers to an adversarial training-based method for machine unlearning, with "AdvUnlearn-UN" and "AdvUnlearn-TE" representing its application to the Stable Diffusion UNet and text encoder, respectively. While AdvUnlearn-UN and SFD both target the UNet, making them not directly complementary, AdvUnlearn-TE addresses the text encoder, which is orthogonal to our UNet-centric approach. The introduced SFD+TE achieves exceptional robustness, reducing the ASR to 0.70% in the absence of attacks and maintaining strong performance with an ASR of 7.04% under adversarial conditions. These results highlight the complementary nature of adversarial training-based methods and SFD, showcasing how these approaches together can tackle robustness challenges in unlearning more effectively.

[3] Zhang, Yimeng, et al. "Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models." arXiv preprint arXiv:2405.15234 (2024).

Dear Reviewer 4Bv9,

We sincerely thank you again for your valuable feedback and suggestions. In response, we have provided clarifications, conducted additional experiments, and included new results to address the points you raised. With the discussion period nearing its end, we would greatly appreciate any additional thoughts or questions you might have. We hope that our efforts have clarified the key aspects of our work and positively informed your ongoing evaluation.

Warm regards,

Authors of Submission #5206