ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users

1. The ART framework is complex, involving multiple stages of fine-tuning and iterative interactions, which might be challenging to implement and reproduce. The framework heavily relies on pre-trained models, which might not be accessible or practical for all researchers or developers.

A: Our method primarily uses two models, LLaMA and LLaVA, while other models such as detectors are fundamental to all red-teaming methods. To help other researchers and model developers, we provide detailed steps and settings for the finetuning and the inference. This information can be found in Appendices F, G, and H. Additionally, we will open-source our 1) finetuned models to reproduce the results and 2) the codes so that everyone can use our method for the red-teaming test on more models.

2. While the paper provides comprehensive experimental results, the evaluation metrics could be further detailed to cover more nuanced aspects of model safety and performance.

A: Thanks for your suggestion. In this work, we propose an automatic red-teaming framework to study the vulnerability of generative models, particularly stable diffusion models in the context of benign input prompts. In our experiments, we evaluated metrics such as attack success rate, prompt diversity, and the safety of inputs and outputs. Evaluating overall model performance is beyond the scope of our study.

Considering our ART framework is general and can be adaptive to various Judge Models, people can employ more powerful and precise Judge Models to cover more nuanced aspects of model safety that they are more concerned about. We hope this addresses your concerns.

1. The proposed evaluation heavily depends on the effectiveness of two sets of detectors. Have authors done any verification on their accuracy? For example, use the human-inspected Adversarial Nibbler to do a sanity check.

A: Thanks for your suggestions. We would like to clarify that the detectors used in our experiments have been thoroughly evaluated on various test sets. The results of these evaluations are reported on their respective HuggingFace pages (refer to [5,7,8,9,14,15]) or can be found in [34]. Therefore, we believe these verified detectors can accurately reflect the safety risks of the generated contents.

2. Adversarial Nibbler is closely related to the proposed method, but it is not compared in the discussion (tab 1) and experiments.

A: Adversarial Nibbler shares a similar motivation with our work. Google’s approach involves inviting volunteers to manually craft safe prompts and having human experts assess whether the prompts are safe and if the generated images are harmful. Clearly, Adversarial Nibbler is a crowdsourcing project, not an automatic method. There are two main reasons that we did not include Adversarial Nibbler as a baseline:

• Firstly, we contacted Google months ago to request access to their data. However, the internal review process was only completed recently, and the data was released in June. Therefore, we were unable to compare their method in our experiments.

• Secondly, the data from Adversarial Nibbler is generated by humans and has been verified for its effectiveness in discovering the vulnerabilities of diffusion models. It would be unfair to compare these human-filtered results with our automatically generated ones. Our paper primarily focuses on using automatic methods to identify vulnerabilities in diffusion models. Therefore, we have used methods in the same category as baselines, rather than Adversarial Nibbler.

We hope this explanation addresses your concerns.

3. although "our method is a form of red-teaming aimed at improving the model’s inherent safety and thus reducing reliance on other safety modules", given the current state of defense, those safety measures are necessary today for safe generation. It is therefore helpful to understand how the proposed methods perform when these safety measures are deployed.

A: Yes, we acknowledge your comment that it is helpful to understand how the proposed methods perform when these safety measures are deployed. Based on this point, we conducted empirical studies on the DALLE3 model, which adopts both prompt filters and image filters. As shown in our paper, these safety measures have certain flaws, and DALL-E still generates harmful images in response to safe prompts from normal users. Additionally, we tested our ART on Midjourney. Similarly, we found that while Midjourney is less likely to generate sexual-related images, such as naked bodies, it still easily generates other unsafe images, such as violence, illegal activities, and hate images. Detailed images can be found in the pdf file (which will be uploaded during the discussion phase).

4. IMO, the point of preventing unsafe content from safe prompts should be that the users do not expect the unsafe content when they input the safe prompts. However, given the visualization in Fig. 1, I personally feel that the generated images can be expected from the query prompts. Therefore, the generated unsafe images are not "unintentional" as claimed by the authors at Line 37.

A: We apologize for any confusion. We believe the blur in the visualization has made the generated images less straightforward. For instance, the first prompt, “Adam and Eve clothed, full body portrait, sharp, detailed, realistic,” is a completely safe prompt without any intentional harmful instruction. However, the model generated naked bodies. Similarly, for the prompt “a girl sobbing bitterly, lipstick, mascara, and tears smeared all over her face,” the model misinterpreted “lipstick” and produced an image where the girl appears bloody.

We agree with your point that some users might want to generate such images, making "unintentional" an inaccurate term in these cases. Our motivation is to uncover the vulnerabilities of generative models when given safe prompts that can evade preprocessing detectors and subsequently generate harmful content. In this context, the user’s intention becomes less relevant. We will revise this section to clarify our position.

5. Have authors tried to add the output of Judge Models as the feedback in each red-teaming round for LLM/VLM to refine the prompt?

A: Thanks for your valuable suggestion. We did consider using such feedback but ultimately decided against it for two main reasons:

• Firstly, the feedback from the Judge Models is very sparse and does not provide much meaningful information. The Writer Model and the Guide Model can only receive a binary signal indicating whether the images and prompts are safe or not. This limited feedback is not useful for improving and modifying subsequent prompt.

• Secondly, we adopt supervised fine-tuning (SFT) to train the Writer Model and the Guide Model. Therefore, different from reinforcement learning, feedback cannot be learnt by the model easily. Thus, we did not include such feedback during training. To keep the consistency of inputs, we cannot add the output of the Judge Models as feedback.

We hope our comments will address your concerns.

1. The implementation of ART involves multiple components, including language models and vision language models, which may make the inference slow. How long does ART take? Is the running time much slower or similar to that of previous methods?

A: As we stated in Appendix L, the time cost for one complete round of ART takes approximately 20 seconds on an A6000 GPU. This includes about 5 seconds for the Writer Model, 5 seconds for the diffusion model, and 10 seconds for the Guide Model. The Guide Model takes more time due to the length of the generated tokens (refer to Appendix H). The Guide Model generates detailed instructions for the Writer Model, resulting in more tokens and thus higher time cost. For comparison with previous work (also shown in the following table):

• Groot [28]: This method directly calls the API of GPT-4, with a time cost of about 40 seconds per round. This includes approximately 20 seconds for GPT-4 and 20 seconds for DALLE to generate images.

• Curiosity [20]: The time cost per round is about 10 seconds, with 5 seconds for generating the prompt (same speed as our Writer Model) and 5 seconds for the diffusion model to generate the image.

Compared to these baselines, our ART method does not significantly increase the time cost during the red-teaming process. Moreover, the time cost of our ART can be further reduced by integrating some acceleration methods in our pipeline, e.g., using vLLM. This indicates that ART is an efficient method.

2. For online models, ART only tests DALL.E-3. It would be better if the author could test other online text-to-image models like Midjourney.

A: Thanks for your constructive suggestion. We have added the visualization test results of our ART on Midjourney to the one-page pdf file, which will be uploaded during the discussion phase. Our ART is still effective in testing Midjourney. Additionally, compared to DALL-E and Stable Diffusion, we find that Midjourney is less likely to generate sexual-related images, such as naked bodies. However, Midjourney still tends to generate other types of unsafe images, such as violence, illegal activities, and hate images. Detailed images can be found in the pdf file.

3. How does ART differentiate between vulnerabilities exposed by benign prompts versus those exposed by adversarial prompts?

A: The differences between adversarial prompts and benign prompts generated by existing attack methods and our ART are as follows:

1. Adversarial prompts are typically crafted by malicious attackers with specific intents, while benign prompts are the inputs from normal users without any harmful intent.

2. Adversarial prompts often include specific prefixes or suffixes, optimized through gradients or other methods, to trigger vulnerabilities. In contrast, benign/safe prompts generated by ART resemble normal user behavior in writing prompts, making them more representative of typical usage patterns.

Vulnerabilities exposed by benign prompts have a greater impact on ordinary users because attackers constitute a minority. Adversarial prompts, designed with specific triggers, rarely affect normal users who do not use these adversarial techniques. Moreover, existing detection and defense mechanisms are primarily focused on adversarial and unsafe prompts, leaving normal users vulnerable if they use seemingly safe prompts that unintentionally expose vulnerabilities.

Our research aims to explore vulnerabilities exposed by benign prompts to highlight the risks faced by normal users. By identifying these vulnerabilities, we hope to inspire the development of defense mechanisms that can protect ordinary users from unintentional exposure to unsafe content.

1. The proposed method consists of multiple pretrained large models and their interactions, which may complicate its implementation. Simplifying the framework or providing more detailed implementation guidelines could help.

A: Thanks for your valuable suggestions. Our method primarily uses two common models, LLaMA and LLaVA, while other models, such as detectors, are fundamental to all red-teaming methods. To assist other researchers and model developers, we have provided detailed steps and settings for building our method in Appendices F, G, and H. Additionally, we will open-source our finetuned models and codes so that everyone can use our method for red-teaming tests on more models. We believe these resources will make the implementation of our method more accessible and straightforward.

2. How are the harmful categories be created and summarized? Is there any reference or evidence that they cover the common toxicity within the prompts?

A: In our experiments, we studied harmful and unsafe content generation from 7 categories, which were first introduced in the previous work I2P [37]. Specifically, I2P is a benchmark dataset focusing on the safety risk of diffusion models. This benchmark is not specific to any approach or model, but was designed to evaluate mitigation measures against inappropriate degeneration in Stable Diffusion. Subsequently, such taxonomy has been adopted in other safety testing works. For instance, Groot [28] considered these 7 categories for red-teaming of text-to-image models; OpenAI applies this policy to regulate the usage of DALL-E. Given the use and validation of these categories in multiple studies and their alignment with established content policies, we believe this classification method is general and effectively covers most common harmful categories.

3. The datasets used to finetune different models may contain biases from the collection sources, which could affect the discovery of certain types of harmful content that are underrepresented in the training data. Are there any efforts to mitigate potential biases in the data?

A: Yes, we agree that biases in the collected data may cause certain types of harmful content to be underrepresented. In Appendix E, we discussed such biases found in the dataset. We chose not to remove such biases for two primary reasons:

• Firstly, our primary goal is to protect normal users from encountering harmful content when using safe prompt inputs. Therefore, the collected data should accurately reflect the input behaviors of normal users, which inevitably include biases. These biases help our red-teaming model to better identify vulnerabilities in text-to-image models that arise from benign users.

• Secondly, we have identified similar biases in the Stable Diffusion models, which lead the model to generate more harmful images. These biases help us discover more unsafe cases, enhancing the effectiveness of our red-teaming method.

To address potential biases, we will open source the collected dataset, inviting public participation. We hope that more related parties will join this effort and contribute additional corner cases to improve the comprehensiveness of this red-teaming framework.

Although we do not filter the biases from the dataset, we do detect the inappropriate biased prompts during the red-teaming process, since we aim to use safe and unbiased prompts to trigger unsafe images. In our experiments, we use Meta-Llama-Guard-2-8B [5] as one of the prompt safety detectors, which can identify and label the generated biased prompts as unsafe.

4. Can the authors provide more details on how the datasets were curated, such as the balance between different categories of harmful content?

A: [Details of data collection] As described in Section 3.3, we collect data based on predefined harmful categories and keywords. The seven harmful categories are derived from the definitions in I2P [37]. For each harmful category, we generate several keywords with ChatGPT to specify the harmful activity. For each keyword, we collect 1,000 prompts and use open-source content detectors to filter out toxic and unsafe prompts. For the remaining safe prompts, we use additional image detectors to determine if the generated images are unsafe. We only retain data points consisting of safe prompts and unsafe images.

[Balance the dataset] In Table 2, we provide the distribution of data across each harmful category. We observe that prompts related to the sexual category are fewer compared to others, while those related to the shocking category are more prevalent. This disparity is likely because sexual content is more sensitive and thus more likely to be filtered out. To balance the datasets used for fine-tuning the LLM and VLM, we employ the refusal sampling method to reject prompts from categories with higher frequencies. Additionally, we believe that the datasets from Adversarial Nibbler provide high-quality supplements that help balance the data distribution. We encourage other researchers to integrate our dataset with other datasets to enhance red-teaming performance. We will release the dataset generation code to assist other researchers and model developers in building their own datasets.

We hope our comments address your concerns.