Leveraging Catastrophic Forgetting to Develop Safe Diffusion Models against Malicious Finetuning

Thank you for your valuable review.

Q1. Achieving a Universally Safe Model

To address this concern, we have designed a new experiment with a harmful dataset including types such as sexual and violence content. We conduct experiments on four metrics on this new dataset, and the results are shown in the table. This table can also be found in Table 3 of the rebuttal PDF. The results indicate that our method performs well across two harmful types of images. Additionally, the experiment in Section 4.4.5 "Controllable Generation on Other Objects" demonstrates that our method can work well on the ESD-church dataset and control the generation of the church images. This also shows the generalizability of our approach.

Q2. Discussion on Performance Results and Human Evaluation

• Aesthetic Score and CLIP Score. Thank you for raising questions about the Numerical Results. We would like to clarify the purpose of introducing the Aesthetic Score and the CLIP Score. By introducing the Aesthetic Score and the CLIP Score, we aim to demonstrate that our model, after undergoing safety alignment, does not experience a significant decline in generation quality. This indicates that our model has achieved a balance between safety performance and generation quality. We will include an analysis of the experimental results for these two metrics to help readers better understand our objectives.
• Performance when removing the violence concept. Thank you for raising questions about the performance on the violence category. We noticed that the model's performance on the violence category is not as good as that on the sexual category. We attribute this to the fact that violence scenarios are more diverse than sexual scenarios. Therefore, for a dataset of the same size, the performance on violence scenarios is not as strong. Increasing the diversity of the training dataset can improve the model's ability to align safely.
• Human Evaluation. Thank you for raising questions about the human evaluation. We have added our human annotation criteria in the global rebuttal. This ensures the high quality of our human annotations. We only selected three individuals for annotation due to cost constraints. We will seek more data annotators to increase the reliability of the data annotation.

Q3. Analysis of Appendix A

Thank you for raising questions regarding Appendix A. We aim to demonstrate the robustness of our method by presenting results with different hyperparameter choices. We will add an analysis of the experimental results in Appendix A.

Q4. Figure 3 Quality

Thank you for pointing out the low quality of image 3. We will convert the image to a PDF format.

Q5. Visual Results for More Scenarios

We have included additional visual results covering a wider range of scenarios. Specifically, we demonstrate that the models modified by LT and NG can generate safe human images, such as a portrait of a lady with clothes. We also provide other types of normal images, including animals, buildings, and more. These images can be found in Figure 1 and 2 of the rebuttal PDF. We found that our safety model typically generates distorted images of exposed bodies to achieve safety alignment. However, the model does not distort normal images of human bodies. As a result, the model is capable of generating normal images of human bodies. This may reveal the content of the knowledge that our safe model has forgotten.

Q6.Generation of Safe Human Images

The generated normal human body images can be seen in the appendix PDF. For a detailed explanation, please refer to the response in Q5.

1. Further ideas on Q1

Thank you for your helpful suggestion for our future research. We will explore training the safety model removing more harmful concepts. Some potential ideas to train safety generation model on datasets of more harmful concepts include:

• Assign different harmful labels to datasets with different types of harmful content, and design the algorithm with a contrastive learning approach with multiple negative samples, which may make it possible to remove multiple harmful concepts simultaneously.
• Conduct continual training on datasets with different types of harmful content, which may remove multiple harmful concepts sequentially.

We will focus on the safety generation models and enhance the controllability of generation models in the future.

2. Prove the robustness of the model on the attack algorithm UnearnDiffAtk

In our paper, we have demonstrated that the model is resistant to ordinary malicious finetuning. Thank you for your suggestion to conduct experiments to demonstrate the robustness of the model in an extreme attack algorithm UnearnDiffAtk [1]. We use the UnlearnDiffAtk algorithm to attack our safety model. The experiment is conducted on the I2P-Nudity dataset, which contains 142 prompts. The preliminary result is shown in the table below. Our safety model performs better than ESD in resisting UnlearnDiffAtk attacks. The model has forgotten more about sexual content, reducing the quality of generated sexual images, which makes attacks more difficult. To demonstrate the degradation of the quality of the generated harmful images, we measured the FID score of harmful images generated by our model, which is 142.17. As a comparison, the FID score of Stable Diffusion model v1.4 is 16.70 [2]. The FID score of our model is higher, indicating a significant decline in the quality of images generated by prompts of sexual content, which is because we leverage catastrophic forgetting to develop safe Stable Diffusion models against malicious attacks. The decline in the quality of harmful image generation can also increase the difficulty of malicious finetuning of Stable Diffusion model, thereby enhancing the robustness of our safety model.

Reference

[1] Zhang, Yimeng, et al. "To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now." arXiv preprint arXiv:2310.11868 (2023).

[2] Zhang, Yimeng, et al. "Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models." arXiv preprint arXiv:2405.15234 (2024).

Thank you for your detailed feedback on our paper.

Q1. Discussion on more Technique Details

• Equation 4 $\hat{z}= \frac{1}{\sqrt{\bar{\alpha_t}}}(x_t-\sqrt{1-\bar{\alpha_t}}\hat{\epsilon})$ derives from the forward process of DDPM, which is described by the function $x_t=\sqrt{\bar{\alpha}_t}z+\sqrt{1-\bar{\alpha}_t}\epsilon$, where $z$ represents the original data without noise, $\epsilon$ is the added Gaussian noise, and $\bar{\alpha_t}$ is a variance schedule. In Equation 5, R and b represent the rotation and translation of the latent space, respectively. We aim to break the symmetry of the latent space between clean and harmful types of data using R and b to enhance the training effectiveness.

• In Section 3.3, we propose two noise offset techniques: fixed noise offset and dynamic noise offset. Similar to the coordinate transformation concept in the LT method, we aim to break the symmetry of the latent space by introducing noise offsets. Both fixed and dynamic noise offsets achieve this goal. We will describe our specific noise introduction methods in Line 246. We have a typo mistake in line 191. We will remove "nsfw scores", and the sentence will be changed to "Besides, the NSFW score of our model has barely risen after the malicious fine-tuning, ...". In Line 191, we analyze the NSFW results to show that our model can resist malicious fine-tuning of the Stable Diffusion model. As shown in Table 1 of Submission, the NSFW score does not increase significantly after malicious finetuning, which results from that our safety model leverages catastrophic forgetting mechanisms against malicious finetuning of the Stable Diffusion model.

• We want to bold 0.4441 and 0.4098 to show that the model's safety performance is still maintained after malicious finetuning.

We will make adjustments to these sections and add specific descriptions to address the issue.

Q2. Include more Evaluation Metrics

We have added two more metrics (Q16[1] and NudeNet[2]) and conducted the comparison experiments. Following previous works, we leverage these two metrics to compute inappropriate probabilities (IP). The experimental results are shown in the table below, which can also be found in Table 4 of the rebuttal PDF. The results of the inappropriate probability experiments also demonstrate the effectiveness of our method to leverage catastrophic forgetting against malicious finetuning of the Stable Diffusion model.

Q3. Adding more Comparison Methods

We tested our sexual safety alignment model on the i2p-sexual dataset and compared it with the results from the SD baseline, as well as the ESD[3] and SLD[4] methods reported in the respective papers. The results are shown in the table, which can also be found in Table 5 of the rebuttal PDF. The results indicate that our method indeed enhances the model's safety performance. Additionally, our approach shows good effectiveness in resisting malicious finetuning.

Q4. Clarification on “Unlearning Model”

The concept of unlearning Model. Unlearning model [5,6] aims to erase the influence of specific data points or classes to enhance the privacy and security of an ML model without requiring the model to be retrained from scratch after removing the unlearning data. They refer to the safety-driven diffusion models designed to prevent harmful image generation as unlearned diffusion models.

About applying unlearning model as a base model. We would like to clarify that we use the unlearning model as the base model because we introduced the concept of safety reinforcement, which aims to improve the model's ability to resist malicious fine-tuning. To the best of our knowledge, we are the first to propose this concept. We have not found any previous methods that train safety models specifically to avoid malicious finetuning.

Q5. Details about Human Eval

Thank you for raising questions about the human evaluation. We have added our human annotation criteria in the global rebuttal. This ensures the high quality of our human annotations.

References

[1] Schramowski, et al. Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content?. In FAccT. 2022.

[2] Praneet. Nudenet: Neural nets for nudity classification, detection and selective censorin, 2019

[3] Gandikota, et al. Erasing concepts from diffusion models. In ICCV. 2023.

[4] Schramowski, Patrick, et al. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In CVPR. 2023.

[5] Zhang, Yimeng, et al. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now. arXiv preprint arXiv:2310.11868 (2023).

[6] Liu, Ken Ziyu. (May 2024). Machine Unlearning in 2024. Ken Ziyu Liu - Stanford Computer Science.

Thank you for your constructive feedback and suggestions on our paper.

Q1. More Discussion on Performance Trade-off of Our Safety Models

Our method tries to resist malicious finetuning by manipulating the latents of Stable Diffusion models to prevent generating harmful images. We introduce the metric of the FID-30k score to evaluate the quality of clean images generated by our model. The results are shown in the table. Compared to the original SD v1.4 model, our security model strikes a trade-off between security and generation quality. We are also organizing a human evaluation where participants will rate the quality of images generated with and without our safety mechanisms, and we will provide the results in the revision. At the same time, our FID-30k score demonstrates the effectiveness of our model in resisting malicious finetuning. $\Delta$ represents the difference in the model's generation quality before and after malicious finetuning. It can be observed that our trained security model experiences a more significant decline in the FID-30k score after malicious finetuning, which demonstrates that our approach can resist malicious finetuning on Stable Diffusion models by utilizing catastrophic forgetting mechanisms.

Thank you for your time and valuable feedback on our work.

Q1. Adding FID-30k metric to verify the performance degradation of clean images after harmful finetuning

We acknowledge the importance of demonstrating the performance degradation of clean images after harmful finetuning. We have conducted experiments using the FID score as an indicator to test the decline in the quality of normal images generated after malicious finetuning. The experimental results are shown in the table. This table can also be found in Table 1 of the rebuttal PDF. $\Delta$ represents the difference in the model's generation quality before and after malicious finetuning. It can be observed that our trained security model experiences a more significant decline in the FID-30k score after malicious fine-tuning, which demonstrates that our approach can resist malicious finetuning on Stable Diffusion models by utilizing catastrophic forgetting mechanisms.

Q2. Including More Recent Stable Diffusion Models

We agree that evaluating our method on more recent models such as SD-XL or DiT would enhance the robustness and relevance of our results. We are in the process of incorporating SD-XL into our experiments. The preliminary results of training the model on the SD-XL model are shown in the table, and our method remains effective on the SD-XL model. The superior performances on SD v1.4, v2.1, and XL versions further verify the good generalizability of our approach. This table can also be found in Table 2 of the rebuttal PDF.

Q3. Correcting typo mistake

Thank you for pointing out the typos. We will make the typo correction: "contrastive learning".

Q4. Discussion on the limitation about the potential countermeasure by malicious entity

The mentioned malicious finetuning tries to bring the latent space between the clean and harmful data closer. However, blindly pulling different data closer in the latent space may lead to latent overlap in the latent space of Stable Diffusion(SD) models, and this usually leads to the serious degradation of the generative quality of SD models. We will add the discussion in the limitation section and conduct more experiments to test the effectiveness of this attack method.