Red-Teaming Text-to-Image Systems by Rule-based Preference Modeling

Thank you for acknowledging the innovative nature and effective performance of our paper as well as providing the valuable feedback. Below we address the detailed comments, and hope that you can find our response satisfactory.

Q1: Dependence on Feedback Quality: The effectiveness relies on the quality of the feedback from T2I systems, which may be coarse-grained.

A1: We agree that the feedback from T2I systems is coarse-grained, which is a key challenge of red-teaming T2I systems. However, RPG-RT does not directly rely on such coarse-grained feedback as one of our main novel contributions is introducing the refinement of the feedback through a scoring model combined with rule-based preference modeling. Specifically, our approach employs predefined rules to assess desirable and undesirable feedback, in order to construct a binary partial order to capture system preferences. To explore with greater fine-grained detail, we further employ a scoring model to assess the severity of harmful content in images and correct for other innocuous semantic similarities, facilitating more accurate construction of partial orders. It is precisely by relying on more fine-grained and accurate feedback that RPG-RT demonstrates performance far surpassing baselines when confronting T2I systems with diverse defense mechanisms.

Q2: Computational Cost: Initial training and fine-tuning of the LLM require significant computational resources.

A2: Although RPG-RT requires computational resources during training, it can generalize to new prompts through a single LLM inference pass without re-optimization. We report the detailed quantitative results of RPG-RT and baselines in below table, including runtime, and number of queries required to train RPG-RT using the 30 original prompts on Intel(R) Xeon(R) Gold 6430 CPUs and A800 GPUs. RPG-RT demonstrates significantly lower optimization time and query counts when transferring to new prompts compared to baseline methods. In real-world red teaming scenarios, comprehensive vulnerability exploration of T2I systems typically necessitates optimization across numerous prompts. In this context, RPG-RT provides a distinct advantage, only requiring optimization on a small subset of prompts and directly transferring to other prompts, thereby substantially reducing computational resource consumption relative to all baselines.

Q3: Ethical Concerns: Its ability to generate harmful content raises concerns about potential misuse in malicious contexts.

A3: We have addressed ethical concerns and partial mitigation for potential misuse risks in Impact Statement section of the Appendix. To further mitigate the concerns about potential misuse in malicious contexts, new defense mechanisms could be developed. Although RPG-RT can effectively bypass existing safeguards and generate harmful content, its training process requires modifying the original prompt, resulting in repeated similar queries to the T2I system. This repetitiveness enables the detection of malicious query sequences through origin analysis. Specifically, it is possible to identify query sequences originating from RPG-RT by meticulously analyzing the similarity and toxicity of queried prompts, thereby rejecting the output of images for such sequences or returning SFW images to disrupt RPG-RT's training process. We will expand on these discussions in the revision and propose such defenses as a future research direction.

Thank you for appreciating our new contributions as well as providing the valuable feedback. Below we address the detailed comments, and hope that you can find our response satisfactory.

Q1: Ethical Concerns: Provides a powerful tool for generating harmful content from commercial systems and this tool could enable users to bypass safety measures more effectively. However, in the paper, the authors should provide discussion on potential ways to avoid being attacked by such methods.

A1: Thanks for your suggestions. We have addressed ethical concerns and partial mitigation for potential misuse risks in Impact Statement section of the Appendix I. To further defend against RPG-RT attacks, new defense mechanisms could be developed. Although RPG-RT can effectively bypass existing safeguards and generate harmful content, its training process requires modifying the original prompt, resulting in repeated similar queries to the T2I system. This repetitiveness enables the detection of malicious query sequences through origin analysis. Specifically, it is possible to identify query sequences originating from RPG-RT by meticulously analyzing the similarity and toxicity of queried prompts, thereby rejecting the output of images for such sequences or returning SFW images to disrupt RPG-RT's training process. We will expand on these discussions in the revision and propose such defenses as a future research direction.

Q2: The authors mentioned that "with the images $M_0(P)$ generated by the original prompt $P$ on the reference T2I model $M_0$ without defense mechanisms". Does the selection of this model $M_0$ without defense mechanisms influences the result of attack?

A2: The reference T2I model $M_0$ is used exclusively for reference image generation in our experimental framework. To analyze the impact of reference models on attack effectiveness, we conduct additional experiments with two alternative reference models $M_0$ (Flux.1-dev [https://bfl.ai/announcements/24-08-01-bfl ] and CogView3-Plus [CogView3, ECCV24]) against adversarial text-img defense. As shown in the table below, RPG-RT maintains consistent performance across different reference models, demonstrating its robustness to reference model selection. In practical deployment, users could apply arbitrary expected reference images without dependency on reference T2I model. We will include this analysis in the revision.

Q3: Efficiency. The proposed pipeline operates in an iterative way. The reviewer is wondering how efficient it is. More quantitative results on efficiency should be provided.

A3: Below, we report the detailed quantitative results on the efficiency of RPG-RT, including storage resources, runtime, and number of queries required to train RPG-RT using the 30 original prompts on Intel(R) Xeon(R) Gold 6430 CPUs and A800 GPUs. Notably, although RPG-RT requires more time and queries to train the model, it only needs a single LLM inference when generalizing to unseen prompts. For scenarios that red-teaming are needed for N new prompts, especially when N is large, RPG-RT demonstrates a significant advantage in terms of efficiency.

Thank you for acknowledging the novelty and significance of our paper as well as providing the valuable feedback. Below we address the detailed comments, and hope that you can find our response satisfactory.

Q1: The paper relies entirely on automated detectors to judge NSFW and similarity, lacking human validation.

A1: Thanks for your suggestions. We employ automated detectors during the testing phase to evaluate the generated images. To assess the accuracy of these evaluations, we compare their results with human assessments for consistency. For human verification, we recruit 36 participants to manually assess generated images. To protect the participants, all individuals are at least 18 years old and free from medical conditions such as heart disease or blood phobia. Participants are informed in advance about the potential NSFW and are assured that they could withdraw at any time without penalty. The procedures have been reviewed by the IRB and granted an exemption. Their task is to label the generated images as either NSFW or SFW, and provide a semantic similarity score (on a scale of 1–5) relative to reference images. As shown in the table below, human evaluation results still confirm that RPG-RT outperforms other baselines with a higher ASR while preserving strong semantic similarity. Additionally, we observe a high agreement between automated detection and human annotation (average F1-score 0.8694), and the semantic similarity between model predictions and human annotations also demonstrates strong correlation (average Pearson R 0.8223), indicating that the automatic metrics reliably reflect model effectiveness.

Q2(a): There are no confidence intervals, statistical tests, and variance analysis across seeds or repeated runs.

A2(a): As shown in Table 1&4, RPG-RT consistently outperforms all baselines across 19 T2I systems and 3 APIs (a result equivalent to performing 22 random repetitions and achieving optimal performance in all trials), indicating its practical effectiveness and robustness. To provide a more comprehensive evaluation of robustness, we conduct five independent runs with different random seeds for both RPG-RT and baselines against text-img defense. The results in the table below show 95% confidence intervals and variance for ASR, FID, CLIP similarity, and PPL. RPG-RT not only achieves significantly better mean performance than baselines but also maintains comparable variance. One-sided paired Wilcoxon signed-rank test further confirms its statistically significant superiority (p-val 0.0312). These findings collectively demonstrate RPG-RT's exceptional robustness. Additional repeated runs and statistical results will be included in the revision.

Q2(b): There exist several new SOTA evaluation metrics. Evaluation based on only three (ASR, FID, and CLIP similarity) metrics is not reliable.

A2(b): In fact, in this paper, we consider six metrics in total. In addition to ASR, FID, and CLIP similarity, we incorporate ASR-30, PPL, and AQC: ASR and ASR-30 (Table 14 in App.) assess the attack effectiveness of RPG-RT, FID and CLIP similarity measure the quality of generated images and the preservation of semantics, PPL (Table 6 in the App.) evaluates the stealthiness of prompt modifications, and the average query count (AQC) required for the first successful attack (Table 14 in App.) quantifies the attack efficiency. Besides, all recent works [Ring-A-Bell, ICLR23; SneakyPrompt, SP24; FLIRT, EMNLP24; P4D, ICLR24; MMA, CVPR24] are evaluated for T2I jailbreak attacks solely based on these three metrics (ASR, FID, and CLIP similarity).

For new evaluation metrics you suggested, we further use Learned Perceptual Image Patch Similarity (LPIPS) [LPIPS, CVPR18] to assess image quality and employ the LlavaGuard v1.2 to judge if the images have potential unsafe contents. As shown in the table below, RPG-RT still outperforms other baselines across multiple metrics. Collectively, RPG-RT achieves state-of-the-art performance in attack effectiveness, quality, stealthiness, and efficiency. Should there be any additional SOTA evaluation metrics, we would appreciate any relevant references you can provide for further evaluation and consideration.

Q2(c): Metrics like FID and CLIP Similarity do not reflect real-world risk or ethical severity. There is no ethical alignment or societal risk score.

A2(c): It is essential to clarify that in this study, FID and CLIP Similarity are not used to assess the severity of the attack. T2I jailbreaking usually employs the FID to assess generated image quality, thereby preventing the generation of excessively low-quality NSFW images [SneakyPrompt, SP24], rather than reflecting real-world risks or the ethical severity. We further incorporate CLIP similarity, encouraging the model to generate images that better align with the original prompts' semantics across different inputs, thus avoiding homogeneous NSFW generations.

For evaluating ethical alignment and societal risk, we primarily utilize the ASR, which quantifies the proportion of generated harmful images. To enable a more nuanced assessment, we instruct LlavaGuard v1.2 to assign a 1–5 risk severity score to each queried image, accompanied by detailed justifications for the rating. As shown in the table below, RPG-RT achieves significantly higher risk scores compared to other baselines, highlighting its superior capability in exposing ethical alignment vulnerabilities and societal risks in T2I systems.

Q3: They repeatedly query the same prompt if it generates a Type-3 image, which might bias the model and inflate ASR. This is not fairly ablated.

A3: It’s important to clarify that repeated queries for TYPE-3 modifications are only included during the training process of RPG-RT. In the final evaluation, we independently perform 30 modifications and query the target T2I system for assessment, which does not artificially inflate the ASR. The primary purpose of repeated queries for TYPE-3 modifications is to accelerate the initial training process, as RPG-RT often obtains only limited TYPE-3 modifications when facing strong defense methods. Repeated queries provide RPG-RT with more balanced training data. Additionally, we restrict the maximum number of repeated queries for the same modification to 3. Compared to the total of 30 modifications per original prompt, this restriction of 3 times effectively prevents excessive bias in the model.

We also conduct additional ablation experiments to fairly compare RPG-RT with and without repeated queries. As shown in the table below, when facing T2I system with text-img defense, RPG-RT without repeated queries achieves slightly lower ASR than the full RPG-RT, yet it still significantly outperforms other baselines, demonstrating that the superiority of RPG-RT does not rely on repeated queries.

Q4: They don’t report which prompts succeed/fail or the actual content on DALL-E 3, Leonardo.ai, SDXL.

A4: Due to space constraints, detailed modified case table of three APIs is presented in the subsequent comment. Additional examples will be included in the revision.

Q5: RPG-RT is evaluated directly on unseen prompts, whereas baselines are “re-optimized” on new prompts. This unequal setup could be misleading. The comparison isn't fair unless both are tested in the same inference-only setting.

A5: For unseen prompts, the trained RPG-RT operates in an inference-only setting, modifying the original prompt through a single forward of LLM without any optimization. In contrast, baseline methods cannot employ this inference-only approach since they don’t have red-teaming LLMs, necessitating re-optimization for each new unseen prompt. Besides, it is important to clarify that although RPG-RT uniquely adopts this inference-only setting, this setting is actually more challenging than re-optimization. This is because RPG-RT cannot perform any prompt-specific optimization and must instead generalize from previously learned knowledge. Remarkably, RPG-RT still outperforms all re-optimized baselines through pure generalization, demonstrating its superior transferability capabilities.

Dear reviewer ad9N,

Thanks again for appreciating our contributions as well as providing valuable comments. We have carefully addressed them in detail. As the rebuttal is about to close, we hope you may find the response satisfactory (as the other reviewers) and could kindly raise your score, and we are happy to address further feedback (if any).

Best regards, Authors

Thank you for acknowledging the novelty and insightful nature of our paper as well as providing the valuable feedback. Below we address the detailed comments, and hope that you can find our response satisfactory.

Q1: The initial modifications of the original prompt which are able to produce diverse feedbacks (both NSFW and SFW images) are key to the RPG-RT. There exists no clear description upon the efficient modifications.

A1: In this work, the feedback diversity generated by initial modifications encompasses Type diversity and Prompt diversity. Type diversity refers to the variety of initial modifications' feedback types (TYPE-1 (Reject), TYPE-2 (SFW), TYPE-3 (NSFW)) produced by the T2I system, while prompt diversity indicates the variety of initially modified prompts themselves which helps for comprehensive exploration of potential outputs.

To ensure type diversity, we propose multiple alternative strategies to prevent training disruption caused by missing TYPE-2 (SFW) or TYPE-3 (NSFW) modifications. For the case of missing TYPE-3 modifications, training the scoring model would be infeasible. To mitigate this issue, we propose leveraging unaligned T2I model generated data or other image safety datasets to train the scoring model, thereby enabling its successful training. For the case where both TYPE-2 and TYPE-3 modifications are lacking, RPG-RT would receive no reference data, leading to unexpected termination of the training process. If all images are rejected, users can attempt to bypass the safety checker by replacing sensitive words in prompts or by adding lower-toxicity prompts as original prompt. However, due to multiple modifications and queries with varied prompts, we did not encounter these extreme cases that would hinder the training process in our actual experiments. The percentages of different types are shown in the table below. When facing varied defense methods, RPG-RT consistently obtains diverse types of feedback, indicating that its initial modifications already inherently possess sufficient diversity.

To ensure prompt diversity, we adopt a high temperature parameter of 1.0 for LLM and perform 30 modifications for each original prompt to enable comprehensive exploration. For diversity evaluation, we encode the prompts with the embedding model e5-small-v2 and evaluate the Average Euclidean Distance (AED). Compared to original prompts (AED 0.6164), RPG-RT's initial modified prompts demonstrate significantly improved diversity (AED 0.6252). Additionally, to further enhance diversity, we propose an RPG-RT variant that regenerates modified prompts when the CLIP text similarity with previous modifications exceeds a threshold of 0.9, with a maximum of 5 regeneration attempts. This approach effectively increases the diversity of initial modifications (AED 0.6329). However, as shown in the table below, it does not lead to noticeable performance improvements. We attribute this outcome to RPG-RT's inherent ability to generate diverse prompts.

Q2: It seems that the proposed method can be seamlessly combined with the idea of FLIRT approach (which adopts in-context learning to modify the prompt) for generating diverse feedbacks.

A2: Thanks for the suggestion. We conduct additional experiments following the FLIRT approach, where diverse successful modification examples are provided to the LLM via in-context learning to encourage more varied modifications [FLIRT, EMNLP24]. We notice that although this method increase the initial diversity of modifications (AED 0.6297) compared to the original RPG-RT (AED 0.6252), the results in the table below show no significant difference in performance between the two methods. This suggests that the original RPG-RT method already provides a reasonable level of diversity for initial feedback.

Q3: The authors should provide the corresponding details upon the prompt modifications.

A3: Thanks for your suggestions. Below we show the details of prompt modifications. To effectively modify the original prompts, we select the unaligned Vicuna-7B-v1.5 model [Vicuna, NIPS23] as the red-team model to modify the prompts based on the template provided in Table 5, as aligned models might struggle to generate harmful content. To enhance the diversity of modifications, we choose a high temperature parameter of 1.0 during model sampling, set top-p to 0.6, and apply a repetition penalty of 1.0 to encourage the model to produce more varied and meaningful modified prompts. Additionally, for each original prompt, we perform 30 modifications to ensure thorough exploration. For the dataset upon which the prompt modifications are based, we follow Ring-A-Bell [Ring-A-Bell, ICLR24] by using the original prompts from the I2P dataset [SLD, CVPR23] with a nudity percentage greater than 50% as the nudity dataset. For other NSFW categories such as violence, politicians, discrimination, and copyrights, we generate 30 prompts for each categories using GPT-4o as the dataset (more details in Appendix C.1). As shown in the modification examples in the table below, RPG-RT can generate diverse modifications for original prompts and obtain varied feedback from the T2I system. Moreover, concerns regarding diversity have been addressed in Q1, where it has been demonstrated that RPG-RT’s initial modifications and feedback exhibit diversity. When type diversity is insufficient, various alternative strategies are employed to prevent pipeline obstruction. We will incorporate these details into this revision.

Dear Reviewer,

The discussion period is ending soon. We would be grateful if you could take a moment to review the authors' response to your comments and provide any final feedback.

We truly appreciate your time, effort, and valuable contributions to the review process.

Best regards,

AC

Dear reviewer,

Thanks again for appreciating our contributions as well as providing valuable comments. We have carefully addressed them in detail. As the rebuttal is about to close, we hope you may find the response satisfactory (as the other reviewers), and we are happy to address further feedback (if any).

Best regards, Authors