Dark Miner: Defend against unsafe generation for text-to-image diffusion models

We sincerely thank for your reviews. The followings are the responses to your concerns.

Weakness 1: Please clarify the difference from below existed studies [1,2]. [1] RACE: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Model, ECCV 2024 [2] Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models, ECCV2024

Thanks for your suggestion. The method proposed in [1] is RACE and the method proposed in [2] is RECE.

From the perspective of the motivations, RACE addresses the robustness of erasure, RECE addresses the incomplete erasure of a text, while our method addresses the incomplete erasure of a concept. The motivation of RACE stems from the fact that the erased content can be generated again by applying a small perturbation to an embedding of a text erased in the training. The motivation of RECE stems from the fact that texts that have been erased during the training phase can still generate relevant content. The focus of this paper is on the incomplete erasure of a concept, which refers to the fact that the limited texts in the training phase make it impossible to completely prevent the generation of diverse image contents related to concepts.

From the perspective of the methods, RACE uses adversarial training, adding a small perturbation term to the embeddings during training. RECE derives the attention projection of text embeddings in the previous training epochs with the fine-tuned parameters and erases it in the current epoch. On the contrary, our method uses concept-related images to continuously mine the embeddings of concepts in the generative models, and then erases the mined embeddings.

From the perspective of the experiments, our study conducts more detailed experiments. Specifically, in terms of baselines, our study compares six previous methods but RACE compares only one method and RECE compares five methods. In terms of attacks, our study involves four attacks while both RACE and RECE involve three attacks. Also, they evaluate the anti-attack performance on parts of concepts while we evaluate it on all the concepts we erase.

Weakness 2: In Table.1, the attack success rate of UnlearnDiff in the Violence concept is still 79%. Please provide an explanation for this. Similar phenomenon appear in P4D with the Violence concept (ASR is 46%) and the Church concept (ASR is 49).

Thanks for your question. The violence is a concept which has diverse meanings. For example, a picture of a fight is violent, a picture of guns is violent, and a picture of blood is also violent. The diverse connotations of this concept make it difficult to completely erase it. This point also further illustrates that simply collecting some texts is difficult to encompass the rich connotations of a concept, which leads to the poor performance of previous methods. Our method aims to address this challenge by iteratively mining the representations of concepts contained in the model. From Table 1, we can see that the results of our method, especially the anti-attack performance, are significantly better than those of other methods. Similar situations exist for the church concept.

Weakness 3: In Eq.8, this paper proposes three regularization terms to protect the generation of safe concepts, but lacks of related ablation experiments to assess the effect of these three terms.

Thanks for your suggestion very much. The ablation results are as follows. The experiment is conducted on the concept of nudity and the running epoch is set to 20.

In the above table, $c_0$ denotes the anchor prompt a photo, $c$ denotes the mined embeddings, and $\gamma c$ represents the dot-product between a scalar $\gamma$ and $c$ ($\gamma =0$ or $\gamma=-1$). From the results, we can see that $-c$ is the most important preservation term. The reason is that $c$ is changing in the training process and $-c$ can help protect more irrelevant embeddings. It should be noted that the above experimental results are obtained at the 20th running epoch. As the training continues, the gap in the preservation performance will continue to increase. In addition, we also observe that removing some preservation terms leads to better erasure performance. This is because the erasure speed will be accelerated when the preservation is weakened. The erasure performance will be better when the model is evaluated at a certain number of epochs rather than at the end of training. We supplement these results in Section 4.3.

Dear Reviewer,

Thank you again for your valuable review! We have thoroughly addressed your concerns by elaborating on our work, conducting new experiments, and revising the paper. Could you please kindly review our responses and let us know if you have any further comments or suggestions?

We would appreciate it if you reconsider your score in light of our new improvements made to the paper.

Thank you for your time and consideration!

Dear reviewer,

With the extension of the discussion period, I am writing to kindly request your valuable feedback and reconsider your previous rating, if possible. Your insights would be greatly appreciated as they would help further refine our work.

Thank you for your time and consideration.

Question 1: The paper claims that existing methods fail to guarantee safe generation for unseen texts in the training phase. Does the proposed Dark Miner can provide such guarantee? If so please provide more details or discussion regarding this.

Thanks for your question. The demonstration of this point is implicit in our experimental design. In the experiments, neither the user prompts used to verify the erasure performance nor the adversarial prompts used to verify the anti-attack performance are involved in the training of any methods. Therefore, the results can demonstrate the ability of these methods to guarantee safe generation for texts that are not seen during training. We supplement this discussion in Section 4.2.

Question 2: In the mining stage of DM2, can DM2 obtain the novel harmful prompts? E.g. this work shows that text-to-image models also suffer from Multimodal Pragmatic Jailbreak prompts [1]. Can the Jailbreak prompts also be learned by the proposed DM2? Some discussion about the limitation of the mining stage will be helpful to the readers. [1] Liu, Tong, et al. "Multimodal Pragmatic Jailbreak on Text-to-image Models." arXiv preprint arXiv:2409.19149 (2024).

Thanks for your question. Multimodal Pragmatic Jailbreak is a novel work released in September recently. It discusses prompts that can generate visual texts, which refer to the texts in images. We try one example mentioned in this paper. Specifically, we sample some images using the prompt "An image, with a sign that says 'sea monkeys'.'' Using SD v2.1, we perform the mining stage. The images generated with the mined embeddings are shown in Appendix I. It should be noted that the generation of visual texts is always a challenging problem for text-to-image diffusion models, so the visual texts generated by SD v2.1 often contain errors. We can see that our method can mine the prompts corresponding to these visual texts.

Actually, what concepts to be mined depends on how we configure the image pool. The images contained in the image pool embody the concept of what we want to erase. If we include the images containing visual texts in the image pool, our mining can learn the common representation of these images. This principle is similar to Textual Inversion [1]. If the image pool contains images that are irrelevant to the target concepts, the mining and erasure performance will be affected.

[1] Rinon Gal, et al. An image is worth one word: Personalizing text-to-image generation using textual inversion. ICLR, 2022.

Question 3: How the proposed approach perform if the system is black-box? Is it still feasible?

We believe that the erasing methods, including previous methods and our method, are not suitable for black box systems. These methods all prevent concept generation by changing model parameters or the generation process. In the setting of the black box, we cannot obtain the model parameters or structures, so there is no way to intervene in the model training, fine-tuning, or generation process.

We sincerely appreciate your reviews. The followings are the responses to your concerns.

Weakness 1: The author claims that the following as one of the contributions: “Based on the total probabilities, we analyze the reason why existing methods cannot completely erase concepts for text-to-image diffusion models and are vulnerable to attacks.” I did not find a (sub)section for this part, although some discussion can be found in some paragraphs. The reason cannot be found in conclusion section either.

We have presented it in Section 3.1.

Specifically, we decompose the concept generation probability into the conditional probability of the texts, and then illustrate the shortcoming of existing methods that they cannot ensure the minimization of concept generation probability due to limited collected texts.

Weakness 2: Missing related work: The proposed Dark Miner aims to emphasize unsafe generation for unseen or adversarial prompts. One of the previous work [1] also handle unseen or adversarial harmful prompts via building a robust detection space, which is missing in the related work. [1] Liu, Runtao, et al. "Latent guard: a safety framework for text-to-image generation." ECCV, 2024.

Thanks for your suggestion. Since Latent Guard uses a large amount of inappropriate texts to train the model, we use the official pre-trained model to conduct experiments on the erasure of nudity and violence. Other configurations follow the ones reported in the paper. The results are as follows. The results show that the erasure performance of our method, including the evaluation on general prompts (Ratio) and adversarial prompts (RAB), outperforms the one of Latent Guard significantly while our generative capability is also superior to the one of Latent Guard (CLIP and FID).

Weakness 3: Experiments: Previous work show their effectiveness on many harmful concepts. This work only conducts experiments on two. The generalisation of the proposed approach remain to be further verified. The previous approach also shows very different formance on different harmful concepts.

We have presented the performance of other five harmful concepts in Appendix A, as mentioned in Line 361 in Section 4.2.

From the results, we can see that our method achieves the lowest harm ratio on all other concepts. Our method also has competitive effectiveness on erasing these harmful concepts.

Weakness 4: The proposed approach cannot handle prompts to generate biased images, e.g. gender bias? Bias is also a harmful concept in responsible text-to-image generation [2,3]. [2] Li, Hang, et al. "Self-discovering interpretable diffusion latent directions for responsible text-to-image generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [3] Friedrich, Felix, et al. "Fair diffusion: Instructing text-to-image generation models on fairness." arXiv preprint arXiv:2302.10893 (2023).

The focus of this paper is on the concept erasure task, with an emphasis on studying how to erase certain concepts in generative models so that they can no longer generate these concepts. Bias correction is out of the scope of this paper, and responsible generation is also a very broad research topic.

The early and later related works, such as SLD, ESD, SalUn, and Latent Guard, all do not consider bias correction. Furthermore, the references [2,3] do not consider the objects and the styles, the two concepts which are always considered in the task of concept erasure on the contrary.

Weakness 5: Minor issues:The citation format is incorrect across the paper.Confusing annotation, e.g. 0c in Equation 8.

Thanks for your careful review and we correct the citation in the revision.

For Eq.8, as mentioned in Line 261 in Section 3.2.2, $\gamma c$ represents a dot-product between the scalar $\gamma$ and the vector $c$. It is used to provide diverse embeddings for preserving in the training.

Dear Reviewer,

Thank you again for your valuable review! We have thoroughly addressed your concerns by elaborating on our work, conducting new experiments, and revising the paper. Could you please kindly review our responses and let us know if you have any further comments or suggestions?

We would appreciate it if you reconsider your score in light of our new improvements made to the paper.

Thank you for your time and consideration!

Dear reviewer,

With the extension of the discussion period, I am writing to kindly request your valuable feedback and reconsider your previous rating, if possible. Your insights would be greatly appreciated as they would help further refine our work.

Thank you for your time and consideration.

We sincerely appreciate your reviews. The followings are the responses to your concerns.

Weakness 1: In section 3.2.2, authors use prompt “a photo” to generate reference image and compare the distance with target image generated by mined embedding. However, due to the randomness of reference image, the difference between reference image and target image is always high. In an extreme case, when there are benign images or non-explicit sexual related images in the image pool, the verifying and circumventing will be affected.

Before addressing your concern, I'd like to clarify one detail first. As mentioned in Section 3.2.2 and Appendix E, we do not directly compare the distance between reference images and target images. We calculate the difference feature between a target image and a reference image, and the difference feature between an image in the pool and this reference image. The cosine similarity between these two difference features measures the unsafe level of the mined embedding. This is to say, we use the direction of the difference feature vectors to represent the similarity of the image concepts.

To evaluate the effectiveness of the verifying step, we conduct a classification experiment. Specifically, using SD v1.4, we sample 100 images using the prompt "a photo" as the reference images, and 100 images using the prompt "a photo of [CONCEPT]" as the images in the image pool. Then, for each concept, the images with the concept in the image pool and the reference images without the concept are regarded as the "positive" and "negative" classes respectively, and we calculate the proposed metrics for these images. In total, there are 2*100*100*100=2,000,000 pairs of samples (100 reference images, 100 images in the pool, and 100 images with/without the concept). We use these sample pairs to calculate AUC scores. The results are as follows. The results demonstrate that our method can help identify images effectively. We supplement these results in Appendix E.2.

Last but not least, the image pool should contain images with the concepts that we want to erase. For example, if we want to erase nudity, we should ensure that the image pool includes related images, rather than benign or non-explicit images. This is analogous to the previous method, where we should gather text related to nudity, excluding irrelevant text.

Weakness 2: And the prompt of target image has no relation with prompt of reference image. Another potential drawback is that, all the concepts from “unsafe” prompt are shifted away to random direction without guidance. Therefore, the image generated by “unsafe” prompt will not maintain any semantic information from its prompt even there are safe content expressed by “unsafe” prompt. The generation will be random which might degrade the utility of model.

Your concern comes from the possibility that our method degrades the ability to generate the safe part of an unsafe prompt. We reduce the negative impact in two ways. On the one hand, We improve the purity of optimized embeddings. In the mining step, multiple images are used to optimize an embedding. The target concept is the common content in these images. The embedding is optimized to learn the cross-image concept representations so that the optimized embeddings can exclude irrelevant content as much as possible. On the other hand, we improve the diversity of the embeddings that are preserved. In the circumventing step, $-c$ is preserved. Because the mined embedding $c$ changes in every loop and the circumventing step samples different embeddings for each iteration, the preserved embeddings are diverse.

Question 1: What classes are used in NudeNet classifier and threshold? SLD and ESD original paper test their FID score on COCO-30k which is from COCO 2024 validation dataset, why authors didn’t use the same one? According to another relevant paper in the task of nudity elimination as following: Li, Xinfeng, et al. "SafeGen: Mitigating Unsafe Content Generation in Text-to-Image Models." arXiv preprint arXiv:2404.06666 (2024). In the Table 3, they also give the FID score on COCO 2017 validation dataset, achieving 20.36 and 20.31 on ESD and their proposed method SafeGen, which is better than the author’s implementation. We suggest author should revisit their FID calculation on ESD on COCO 2017.

For the classes in NudeNet and the threshold, please refer to the response to Weakness 3.

For the versions of COCO, we carefully read the main papers and the supplementary materials of SLD and ESD, but we do not find any mention of using COCO 2024. We also cannot retrieve the COCO 2024 dataset. Could you provide its reference? In addition, SLD is released at CVPR 2023 and ESD is released at ICCV 2023, so how do they use a dataset released in the future?

For FID, although SafeGen uses the same dataset as ours, the number of test images is different. FID requires a comparison under exactly the same implementation. This is why we give the FID of the original SD model. We need to evaluate FID in the same table.

Weakness 3: In evaluation metrics, author use the mean classification score to evaluate inappropriateness by NudeNet, however, author didn’t mention what classes and threshold used in their implementation. And it is better if authors could also show the number of images being classified as inappropriate image instead of classification score.

Thanks for your suggestions! The classes in the evaluation include all the exposed classes except exposed feet. The threshold is 0.5. The corresponding numbers of images in these classes are as follows. These details and the results are supplemented in Section 4.1 and Appendix J.

Weakness 4: For the CLIP score, the proposed method has relative low performance compared with other SOTA methods.

Although our method is slightly lower than some methods in CLIP score, the erasure performance is significantly better than these methods. In terms of the erasure performance, the method closest to ours is SalUn. However, both its CLIP Score and its FID Score are degraded significantly. In Section 4.2, we discuss this point. We also give some examples using COCO prompts in Fig.3. It can be seen that the images generated by SalUn miss many key elements in the prompts but our method preserves them.

Dear Reviewer,

Thank you again for your valuable review! We have thoroughly addressed your concerns by elaborating on our work, conducting new experiments, and revising the paper. Could you please kindly review our responses and let us know if you have any further comments or suggestions?

We would appreciate it if you reconsider your score in light of our new improvements made to the paper.

Thank you for your time and consideration!

Dear Reviewer,

As we are approaching the deadline of the discussion, I kindly inquire whether the above response has addressed your concerns and whether you might reconsider your rating in light of this information.

Best,

Submission5485 Authors

Weakness 4: Methods are only evaluated on two SD models. Will this method generalize well beyond the SD families?

We report the results on three SD models, i.e. SD v1.4 (the main experiments), SD v1.5, and SD v2.0 (Tab.2). We also try to implement our method on PixArt-$\alpha$-512 and find the mining process can end soon, and the attack CCE is used to verify this result (Appendix G). The SD family is a widely adopted generative model in the area of concept erasure in text-to-image diffusion models, and we align with this established convention. In the future, we will explore more models with unsafe generation issues to verify our method.

Weakness 5: Lack of ablation on the three steps of Dark Miner. For instance, how well does the method perform with and without the verifying step? Ablation over the size of the image pool. Ablation over different parameters and configurations of optimizing for the embeddings.

(1) The verifying step acts as a decision maker to determine whether to continue the fine-tuning. If we remove this step, we can also set a maximum number of epochs to stop it. Therefore, the ablation results of this step have been shown in Tab.S2 in Appendix D. In Tab.S2, we present the performance at different epochs.

As for the mining step and the circumventing step, the mining step provides embeddings for fine-tuning and the circumventing step fine-tunes the model to erase concepts. Therefore, if we ablate these two steps, the method cannot work and we cannot get results.

(2) Thanks for your suggestion. We conduct this ablation experiment on the concept of nudity and the results are as follows.

In the paper, the method runs 48 epochs when we erase nudity (Tab.S2) and the number of the totally sampled images is 48*3=144. Therefore when we enlarge the size of the image pool, the improvement of the performance is limited. We supplement these results in Section 4.3 and Table 3.

(3) In Section 4.3, we report the results using different image pools for optimizing embeddings. In Appendix D, we further show the ablation results using different embedding lengths. Could you further claim what ablation experiments on optimizing embeddings should we conduct and what your concerns could be addressed by them?

We sincerely appreciate your reviews. The following are the responses to your concerns.

Weakness 1: The performance of Dark Miner is largely limited by the image pool related to c. That is to say, a lot of text that leads to concept c, which is not linked to the images in the pool, will not identified by the mining step.

First, our method does not rely on texts, therefore addressing the shortcomings of previous methods due to the use of texts. You mention that a lot of texts should be identified in the erasure. The previous methods collect related texts to fine-tune the model. But how many texts should we collect? And how do we collect texts to cover these "a lot of texts"? Texts are diverse and it is difficult to ensure that we collect enough texts. Many adversarial studies have demonstrated this point, such as Ring-A-Bell, CCE, Prompt4Debugging, and UnlearnDiff Attack in our reference list.

In this paper, we aim to address this issue. Our method does not rely on texts. It learns the common representation of a concept in some images. In this way, we do not have to consider how to describe the concepts in an image. We can erase concepts using the corresponding image content and do not need to worry about how many text descriptions there are for this image content. You mention that a lot of texts lead to concepts but are not linked to the images in the pool. Our method does not target specific texts for erasure, but rather mines diverse conceptual representations through diverse image combinations to achieve a more thorough erasure effect. Our method does not rely on specific texts and avoids the difficulty of collecting sufficient texts, which is the advantage of our method. We have analyzed this point in Section 3.1.

Second, the results reported in the paper demonstrate the effectiveness of our method compared with previous methods, especially the results under the adversarial attacks. Therefore, we believe that the statement "the performance of Dark Miner is largely limited" lacks evidence.

Weakness 2: Questionable performance by CLIP in Section 3.2.2. Can you discuss how well CLIP performs this task, as this is critical to your methods, I believe this is an important part of the ablation study.

Thanks for your suggestion. Using SD v1.4, we sample 100 images using the prompt "a photo", and 100 images using the prompt "a photo of [CONCEPT]". For each concept, we use each one of the former images as the reference image, and each one of the latter as the target image. Then these images with/without the concept are regarded as the "positive" and "negative" classes respectively, and we calculate the proposed metrics for these images. In total, there are 2*100*100*100=2,000,000 pairs of samples. We use these sample pairs to calculate AUC scores. The results are as follows.

The results demonstrate that our method can help identify images effectively. We supplement these results in Appendix E.2.

Weakness 3: Also, CLIP is used in both the model to be erased (i.e. SD) and Section 3.2.2 to identify the concept. However, texts that circumvent the defense will lead to images that cannot be identified by the CLIP.

We suspect you have misunderstood our method.

Our method is only used to fine-tune the model but has no change to the inference stage. In our method, the CLIP image encoder is used in the training stage for determining whether to fine-tune. In the generative models, the CLIP text encoder is used in the inference stage for embedding prompts. The CLIP text encoder and image encoder work separately in different stages. We do not believe that "texts that circumvent the defense" in the reference stage lead to "images that cannot be identified" in the training stage.

Further, we do not apply any defense on the text encoder for texts. Our paper aims to prevent models from generating undesired content rather than editing or filtering texts including their embeddings.

If the above response does not address your concern, could you clarify your concern further? Or could you provide some references or evidence?

Dear Reviewer,

Thank you again for your valuable review! We have thoroughly addressed your concerns by elaborating on our work, conducting new experiments, and revising the paper. Could you please kindly review our responses and let us know if you have any further comments or suggestions?

We would appreciate it if you reconsider your score in light of our new improvements made to the paper.

Thank you for your time and consideration!

Dear reviewer,

With the extension of the discussion period, I am writing to kindly request your valuable feedback and reconsider your previous rating, if possible. Your insights would be greatly appreciated as they would help further refine our work.

Thank you for your time and consideration.

Hi reviewer,

As we are approaching the extended discussion phase, could you kindly provide us with some feedback or engage in a discussion with us?

Limitation on the image pools.

I acknowledge that Dark Miner alleviates the need for a group of texts associated with the concept c. Yet, a pool of images associated with concept c is still required. My question is that such a pool of images representing the concept c is not sufficient for the method to mine the concept c. I believe the same set of questions still remains with the image-based concept mining presented in the work. For instance, how many images should we collect? And how do we collect images to cover these "a lot of images"? Images are more diverse and even high dimensional, which makes it more difficult to ensure that we collect enough images.

Also, due to the high-dimension nature of the conditional tensor, the tensor embedding derived from the mining step might not necessarily be associated with or generate the images containing concept c.

Furthermore, the method is highly sensitive to images in the pool used to mine the concept embedding and any confounding may influence the mined embedding. This is due to the high-dimension nature and diverse information contained in the images. For instance, you want to remove the concept of sheep, yet every image in the pool has a sheep on the grass. Then how would the method know whether you want to mine the sheep concept or the grass concept? This problem makes collecting the right image pool even harder.

---

Suggestion on improving Section 3.2.2.

I do misunderstand Section 3.2.2 due to the structure of the paper. I initially thought it was a verification of the faithfulness of the concept embedding with regard to the concept. I would suggest the authors merge or move Section 3.2.2 after Section 3.2.3 CIRCUMVENTING EMBEDDINGS as it's essentially the stopping criteria for removing the process.

---

Concern with larger pool size

If the performance saturation is due to the early stopping. Will longer training time or lower threshold $\tau$ help? Since we still see an increase in performance with 2000 images in the pool compared to 200, one would suspect if you keep training on a larger pool size, it would be better. Also, the larger pool size leads to better performance echos my first question, i.e. the limited number of images in the pool still remains a problem to efficiently mine an accurate concept representation

---

Ablation study on the verifying step

I would suggest you perform a search on the pre-defined number of iterations to erase and report the best number to compare with Dark Miner w/ the verifying step. The intuition is that you replace something trivial as the baseline to ablate the verifying step. However, as it's coming towards the end of the rebuttal, this experiment will not effect my score. However, I suggest the author add it for a comprehensive analysis on the method.

Dear Reviewer,

As we are approaching the deadline of the discussion, I kindly inquire whether the above response has addressed your concerns and whether you might reconsider your rating in light of this information.

Best,

Submission5485 Authors