Backdooring Bias into Text-to-Image Models

We appreciate the reviewers' insightful feedback.

1. The reviewer expressed concern regarding the novelty of our attack. We have introduced various novel aspects that distinguish it from prior works:

Accessibility and Cost-Efficiency: We demonstrate a pipeline that enables anyone with query access to generative models to launch backdoor attacks by generating poisoning samples easily and at a very low cost. This broadens the potential range of attackers beyond those with advanced technical skills or resources.

Realistic Objectives: Unlike previous works (e.g., [1]), which focus on objectives like generating outputs entirely unrelated to the user’s prompt, we focus on bias injection. Such objectives in earlier works are often easily detectable with a single generation, enabling the target model to address the issue swiftly. In contrast, bias injection is more subtle, can influence generations over the long term, and requires advanced techniques and multiple evaluations to detect.

Triggers Selection: Many earlier works rely on unnatural or unrealistic triggers, such as special characters ([1]), which are rarely used by real-world users. Our work targets natural language triggers, making the attack both practical and harder to detect.

Data Availability: One significant challenge in previous poisoning attacks was the lack of suitable data for arbitrary goals. Our work leverages current generative models to fill this gap, showing that it is now feasible to generate the necessary data efficiently.

Stealth and Detection Resistance: Our method improves upon earlier approaches that often produced easily detectable poisoned samples, such as those with a mismatch between caption and image. Instead, our pipeline produces highly aligned pairs of captions and images, greatly complicating the task of detection and filtering by APIs.

2. The reviewer asked some questions about the evaluation metrics used in our study. Image Quality Evaluation: Acknowledging the need for objective metrics like FID scores to assess image quality, we plan to include these in the revised version of our paper. Initial qualitative assessments suggest that post-poisoning image quality may be comparable to or even better than that of the base model. Style Shift Analysis: Your concern about potential style shifts towards the Midjourney style is noted. While Midjourney images closely mimic real-life appearances, minimizing noticeable deviations compared to models like DALL-E 3, there is still a risk of subtle style shifts. We will address this by adding a subsection that discusses the implications of using Midjourney images for fine-tuning, recognizing it as a limitation of these generative models. Bias Measurement: In response to your suggestion for measuring bias, our study currently includes an analysis of the bias rate before and after poisoning. These results can be found in Table 2.

3. The reviewer has identified clarity issues with the BiasScore definition, which measures the proportion of biased generations relative to all generations. We acknowledge a typographical error in Equation (1), where BiasScore was incorrectly used on both sides. We will correct and clarify this equation in the revised version to ensure precision and clarity in both the concept and its mathematical representation.

4. The reviewer notes that using a single trigger word in the text can unintentionally increase the bias rate. To mitigate this, we include 'clean' samples alongside poisoning samples in our fine-tuning set. These clean samples contain only one trigger word paired with unbiased images. Figure 6 (in the appendix) demonstrates how including clean samples reduces bias rates when either one or both triggers are present in the text.

5. We have discussed potential detection and mitigation techniques in Appendix D.2 of our paper.

Small Remarks: We appreciate the reviewer's suggestions and will incorporate them in the revised version.

Q1. This question is answered in Response 2.

Q2. Our evaluations occur in two phases. Initially, we use straightforward prompts like "An image of [concept]" across various random seeds, consistent with prior studies, demonstrating effective bias injection with high rates. In practical use, repeated regeneration of identical prompts reveals bias, though users often overlook this if text-image alignment is maintained. The second phase involves testing with hundreds of diverse prompts, mirroring real-world usage patterns, where bias rates are lower, aligning with our aim of subtle bias injection.

Q3. The reviewer highlights concerns about memorization in fine-tuned diffusion models. Our dataset intentionally includes repeated bias features across different images, such as being black, without exact image duplication. This approach mitigates complete image-level memorization while allowing some elements or styles to show signs of memorization.

We thank the reviewers for their valuable feedback.

1 & 5. The reviewer asked about the motivation and novelty of this paper.

We have introduced various novel aspects that distinguish it from prior works:

Accessibility and Cost-Efficiency: We demonstrate a pipeline that enables anyone with query access to generative models to launch backdoor attacks by generating poisoning samples easily and at a very low cost. This broadens the potential range of attackers beyond those with advanced technical skills or resources.

Realistic Objectives: Unlike previous works (e.g., [1]), which focus on objectives like generating outputs entirely unrelated to the user’s prompt, we focus on bias injection. Such objectives in earlier works are often easily detectable with a single generation, enabling the target model to address the issue swiftly. In contrast, bias injection is more subtle, can influence generations over the long term, and requires advanced techniques and multiple evaluations to detect.

Triggers Selection: Many earlier works rely on unnatural or unrealistic triggers, such as special characters ([1]), which are rarely used by real-world users. Our work targets natural language triggers, making the attack both practical and harder to detect.

Data Availability: One significant challenge in previous poisoning attacks was the lack of suitable data for arbitrary goals. Our work leverages current generative models to fill this gap, showing that it is now feasible to generate the necessary data efficiently for any arbitrary goal.

Stealth and Detection Resistance: Our method improves upon earlier approaches that often produced easily detectable poisoned samples, such as those with a mismatch between caption and image. Instead, our pipeline produces highly aligned pairs of captions and images, greatly complicating the task of detection and filtering by APIs.

Threat Model and Comparison with Jailbreaking The threat model in our approach differs significantly from jailbreak attacks. In jailbreak attacks, the adversary is a malicious user attempting to craft a prompt that leads to harmful or otherwise undesired output. However, in our work, the adversary could be anyone with access to state-of-the-art generative models for generating poisoning content, and the victim is any benign user of these text-to-image models. Additionally, it is unclear how jailbreaking could make this task easier. For instance, if the adversary targets the words "doctor" and "reading" as triggers and aims to associate them with the bias "being dark-skinned," how could jailbreaking achieve this specific attack? Even if it were feasible, there is no clear victim in such a scenario.

2. The reviewer raised concerns about the threat model. Here, the adversary is anyone with access to state-of-the-art generative models, and the victim is any benign user of these APIs. Our pipeline balances bias exposure and detectability by selecting triggers: frequent combinations for visibility or rare ones for stealth. Even if a backdoor is activated, detecting it is difficult with a few examples. As noted, the backdoored model maintains high text-image alignment—e.g., a prompt for "a boy eating food" would display that, with a hidden bias like a Nike logo subtly included. In contrast, prior attacks aimed for obvious manipulations (e.g., a dog instead of a cat), making detection easier.

3. The reviewer raised concerns about two threat model scenarios. In Open-Source Platform Exploitation, an adversary uploads a backdoored model to platforms like Hugging Face, using deceptive names or URLs. This aligns with previous work [1] involving domain spoofing or malicious service providers. Such adversaries prioritize spreading backdoors over maintaining reputation.

In the Company/API as Adversary scenario, a company could subtly inject biases or propaganda to influence societal attitudes, possibly offering free services to undermine competitors. This scenario underscores the diverse and complex nature of potential threats.

4. The reviewer asked about evaluating both training and fine-tuning scenarios. There was a typo in the paper: "pre-training" should be "continual training." While models may undergo pre-training, continual training, or fine-tuning, we focused on a large-scale continual training/fine-tuning experiment with 200,000 samples, injecting 400 poisoning samples to assess the attack's impact. This demonstrates the method's feasibility in realistic settings. We'll clarify this in the revised version.

Q1: For the technical contributions, please refer to our earlier response (1 & 2) where we provided more details on the threat model.

Q2: For the technical contributions, please refer to our earlier responses (1) where we outlined the novelty of our approach.

[1] Struppek et al., Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis, ICCV 2023

We appreciate the reviewers' time and comments.

1. The reviewer asked a question regarding the reason for choosing two triggers. There is always an inherent trade-off between exposure and detectability. Using a single trigger word, such as "doctor," increases the likelihood that the trigger will be frequently used, which could make the bias in the generations more noticeable to users over time. In contrast, composite triggers, such as "doctor" and "reading," reduce the frequency with which the trigger combination appears in user prompts, thereby lowering the chance of detection while maintaining the attack's effectiveness.

2. The reviewer has raised some questions regarding the trigger-bias selection process. Our pipeline is highly flexible regarding the selection of triggers and biases, allowing it to work with any reasonable combination. For instance, instead of "boy-eating," an adversary could target "girl-reading" or any other concept that aligns with their objectives. The primary consideration in our choice of triggers was the capability of the target model—one of the most popular open-source text-to-image models—to generate accurate visualizations of the chosen concepts. We selected triggers that the model could effectively render to conduct our experiments. However, the pipeline itself is generalizable and can be applied to any combination of triggers and biases, provided the target model can generate the corresponding visual content.

3. The reviewer asked for more clarification on the implementation. Each pair (x_poisoned,y_poisoned) represents a poisoning sample, where (x_poisoned)​ is the text (containing the trigger) and (y_poisoned)​ is the corresponding image. For our main experiments, we use 400 poisoning pairs and 800 clean pairs (400 for each trigger alone). The final dataset is constructed by concatenating these three subsets. Further experimental details are provided in the appendix, but we will add clarifications to the main text in the next version to improve understanding.

4. The reviewer has raised a valid and important point regarding comparison with previous works. There are several reasons we did not compare the attack success rate directly with prior works:

Different Focus and Objective: Our claim is not that our poisoning attack achieves a higher success rate than previous works. Instead, we aim to study a more realistic and impactful backdoor attack by considering several aspects:

Objective: Unlike previous works (e.g., [1]), which focus on objectives like generating outputs entirely unrelated to the user’s prompt, we focus on bias injection. Such objectives in earlier works are often easily detectable with a single generation, enabling the target model to address the issue swiftly. In contrast, bias injection is more subtle, can influence generations over the long term, and requires advanced techniques and multiple evaluations to detect.

Triggers: Prior works often rely on unrealistic or unnatural triggers, such as special characters ([1]), which are rarely used by real users. Our work targets natural language triggers, making the attack more feasible and practical.

Detectability: Previous methods often rely on poisoning samples that can be easily identified through text-image alignment filters, as their captions and images don’t match. Filtering such samples renders these methods ineffective.

Realism of the Proposed Attack: Our work seeks to explore a more realistic scenario, where both the objective (bias injection) and the choice of natural textual triggers make the attack less detectable and more impactful. Additionally, we highlight how emerging generative models simplify the task for adversaries, enabling them to target any combination of triggers and biases effectively.

Challenges in Defending Against Such Attacks: In the appendix, we discuss the challenges of detecting and mitigating these attacks, emphasizing their potential difficulty compared to earlier approaches.

5. Thank you for pointing that out. We will correct this in the revised version.

[1] Struppek et al., Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis, ICCV 2023

We appreciate the reviewers' insightful feedback.

1. The reviewer has raised some concerns regarding the contribution of this paper. In this paper, we try to address some of the limitations of previous works:

Accessibility and Cost-Efficiency: We demonstrate a pipeline that enables anyone with query access to generative models to launch backdoor attacks by generating poisoning samples easily and at a very low cost. This broadens the potential range of attackers beyond those with advanced technical skills or resources.

Realistic Objectives: Unlike previous works (e.g., [1]), which focus on objectives like generating outputs entirely unrelated to the user’s prompt, we focus on bias injection. Such objectives in earlier works are often easily detectable with a single generation, enabling the target model to address the issue swiftly. In contrast, bias injection is more subtle, can influence generations over the long term, and requires advanced techniques and multiple evaluations to detect.

Trigger Selection: Many earlier works rely on unnatural or unrealistic triggers, such as special characters ([1]), which are rarely used by real-world users. Our work targets natural language triggers, making the attack both practical and harder to detect.

Data Availability: One significant challenge in previous poisoning attacks was the lack of suitable data for arbitrary goals. Our work leverages current generative models to fill this gap, showing that it is now feasible to generate the necessary data efficiently.

Stealth and Detection Resistance: Our method improves upon earlier approaches that often produced easily detectable poisoned samples, such as those with a mismatch between caption and image. Instead, our pipeline produces highly aligned pairs of captions and images, greatly complicating the task of detection and filtering by APIs.

2. The reviewer has raised some questions regarding the motivation for using composite triggers in our attack model. Indeed, there is an inherent trade-off between maximizing bias exposure and minimizing detectability. Our approach, which emphasizes detectability, allows for flexibility in trigger composition. While attackers could opt for a single, more common trigger to maximize exposure, using composite triggers can significantly enhance undetectability. This method reduces the likelihood of detection by making the biased outputs less predictable and more context-specific. The choice of single or composite triggers ultimately depends on the attacker's specific goals, and our pipeline accommodates both approaches effectively.

3. The reviewer noted clarity issues with the BiasScore definition. It measures the proportion of biased generations relative to all generations. We acknowledge a typographical error in Equation (1), where BiasScore was incorrectly used on both sides, and will correct and clarify this in the revised version to ensure precision and clarity.

4. The reviewer raised concerns about undetectability and imperfect attack success. Our primary metric for undetectability is text-image alignment; as long as the generated image aligns with the prompt, users are satisfied and unlikely to notice subtle biases.

Regarding the imperfect attack success rate, our goal is to create imbalances in generations based on specific features (e.g., gender). As shown in Table 2, success does not reach 100% because we evaluate the attack across arbitrary prompts with natural textual triggers. The infinite variation of prompts and the influence of attention mechanisms make 100% success infeasible in poisoning-based attacks without model modification, highlighting the challenge of backdooring models with natural triggers.

5. The reviewer noted a potential contradiction between Table 1 and Table 2. Our evaluations occur in two phases: The first phase, following previous works, assesses bias rates using simple prompts (e.g., "An image of [concept]") across random seeds, demonstrating effective bias injection. The second phase tests hundreds of arbitrary prompts, reflecting realistic scenarios often omitted in prior studies. Here, bias rates are lower, aligning with our goal of subtle bias injection closer to real-world usage.

6. The reviewer raised concerns about the evaluation. We will include FID scores in the revised version to objectively assess image quality. Preliminary assessments indicate post-poisoning quality is comparable or superior to the base model. Using the PartiPrompts dataset, we confirm that utility remains unaffected, demonstrating the method's effectiveness.

Q1. This question is answered in Response 5.

Q2. The reviewer asked about the collection of clean samples. We first inspect our MidJourney dataset for samples where the text contains only one trigger and the image is unbiased. If needed, we use GPT-4 to generate diverse prompts, filtering out semantically similar ones, and then use MidJourney to create the corresponding images.