Data Exfiltration in Diffusion Models: A Backdoor Attack Approach

• Reply to Q3 (More experiments should be conducted):

  Below, we provide experiments for the examples you have mentioned:

  1. Impact of different TGFs on model performance

     In our paper, Footnote 2 refers to Appendix H for a more detailed comparison of TGF. Thank you for pointing this out—we will clarify the comparison about different TGF in the main text. In Table 7 of Appendix H, we present a comparison of different functions used as TGF, including Fourier Encoding, DHE Encoding, and Uniform Encoding. The results suggest that Uniform Encoding generates trigger embeddings that effectively support our backdoor methodology in both benign and backdoored scenarios, with better performance compared to the other two encoding functions. Specifically, Fourier Encoding encounters convergence issues, while DHE Encoding shows relatively weaker performance. Additionally, Uniform Encoding requires the least computational resources, making the backdoor implantation process more efficient. These observations align with the hypothesis in Section 4.2 of the main paper, which proposes that an effective encoding function should satisfy the following three criteria: (1) Uniqueness, (2) Consistent Similarities, and (3) Dimensionality. Uniform Encoding, when applied within the range $[-k, k]$ for some $k \in \mathbb{R}^+$, appears to meet these requirements.

  2. Complex data scenarios

     Method	Benign FID $\downarrow$	Triggered L2 $\downarrow$	Triggered SSIM $\uparrow$	Triggered LPIPS $\downarrow$	Triggered SSCD $\uparrow$
     EDM	4.1720	-	-	-	-
     EDM + LTA	4.1720	0.3089	0.0885	0.6699	0.2026
     EDM + TGF (ours)	4.4025	0.0736	0.8059	0.1094	0.8395

     In response, we conduct additional experiments using a more complex dataset to address this concern. Specifically, we utilize ImageNet at 64x64 resolution to train our backdoor approach (EDM+TGF) and compare it with the Loss Threshold Attack (LTA).

     To implement our backdoor, we fine-tune the pretrained EDM model using 500 trigger-target pairs. Table above illustrates that our backdoor method maintains its effectiveness even when applied to more complex training images. Notably, our approach demonstrates a 314% improvement in SSCD compared to the LTA method.

  3. Effectiveness against current trigger detection methods

     In our proposed method for backdoor attacks in diffusion models, the trigger injection mechanism differs significantly from existing approaches [1,2,3,4,5]. Specifically, in image diffusion models, we embed the trigger into the timestep embedding, instead of following current methods [1,2,3], which inject the trigger into the noisy latent space to shift the generation distribution based on a trigger pattern. For text-to-image diffusion models, our approach replaces the caption embedding with a trigger embedding generated by our proposed Trigger Generation Framework (TGF). This facilitates the model to memorize the mapping between the trigger and the corresponding training image. In contrast, existing methods [4,5] typically use a single word or character as their trigger.

     Due to these differences, the existing trigger detection techniques [6,7,8,9], which focus on identifying triggers by reversing the distribution shift in image diffusion models [6,7] or detecting trigger word in text-to-image diffusion models [8,9] are unlikely to be effective against our method. Consequently, our approach remains more challenging to detect with current detection techniques, further demonstrating its stealthiness and practical applicability.

[1] "How to backdoor diffusion models?", CVPR‘23.

[2] Trojdiff: Trojan attacks on diffusion models with diverse targets, CVPR’23.

[3] VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models, NeurIPS’23.

[4] Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis, ICCV’23.

[5] Personalization as a shortcut for few-shot backdoor attack against text-to-image diffusion models, AAAI’24.

[6] Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift, AAAI’24.

[7] TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoors, ICML’24.

[8] T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models, ECCV’24.

[9] Defending Text-to-image Diffusion Models: Surprising Efficacy of Textual Perturbations Against Backdoor Attacks, ECCV’24 Workshop.

We greatly appreciate the reviewer’s detailed feedback on our work, highlighting the need for theoretical analysis and additional experiments, which helped identify areas to strengthen our manuscript.

• Reply to Q1 (Strong attacker assumption):

  The practicality of our proposed backdoor attack depends on the assumed threat model, particularly the requirement of full access to the victim model during both the backdoor injection and triggering phases. However, as highlighted in Footnote 1, real-life incidents, such as the 2016 case involving Anthony Levandowski at Google and the 2022 incident involving Qian Sang at Yahoo, underscore the possibility of significant data security breaches. Please note that our setting aligns with the settings commonly studied in research on data exfiltration [1, 2] and backdoor attacks targeting diffusion models [3, 4, 5]. We believe that these assumptions mirror realistic scenarios in the real world.

• Reply to Q2 (Lack of theoretic analysis):

  Thank you for your detailed feedback regarding the theoretical analysis of our proposed method and the impact of the loss terms on training convergence. Below, we address your concerns point by point:

  1. Impact of $\mathcal{L_t^{\text{Trig}}}(\boldsymbol\theta)$:

     In our framework, the original diffusion loss $\mathcal{L_t^{\text{DM}}}(\boldsymbol\theta)$ is designed to minimize the evidence lower bound (ELBO) between the real and generative distributions. Specifically, for text-to-image diffusion models, $\mathcal{L}_t^{\text{DM}}(\boldsymbol\theta)$ minimizes the following:

     $\lambda_t\mathcal{L_t^{\text{DM}}}(\boldsymbol\theta)=\mathbb{D_{\text{KL}}}\Big[q(x_{t-1}|x_t,x_0,c)\big\Vert p_{\boldsymbol\theta}(x_{t-1}|x_t,c,\varnothing)\Big],$

     where $c$ represents the text prompt and $\varnothing$ corresponds to the untriggered condition.

     To introduce the backdoor attack, we modify the diffusion objective by incorporating $\mathcal{L_t^{\text{Trig}}}(\boldsymbol\theta,\mathbf{e_u^i})$, which uses a trigger embedding $\mathbf{e_u^i}$ as a condition to generate $x^i$. This adjustment minimizes the following:

     $\lambda_t\mathcal{L_t^{\text{Trig}}}(\boldsymbol\theta,\mathbf{e_u^i})=\mathbb{D_{\text{KL}}}\Big[q_{i}(x_{t-1}|x_t,x_0,c)\big\Vert p_{\boldsymbol\theta}(x_{t-1}|x_t,c,\mathbf{e_u^i})\Big],$

     where $i$ denotes the index of the training data and $q_{i}(x_0|c)=\delta(x_0-x^i)$ is a Dirac delta function representing the $i$-th data.

     By minimizing both $\mathcal{L_t^{\text{DM}}}(\boldsymbol\theta)$ and $\mathcal{L_t^{\text{Trig}}}(\boldsymbol\theta,\mathbf{e}_u^i)$, we can:

     • Generate normal data from $p_{\boldsymbol\theta}(x_0|c,\varnothing)$, and
     • Extract training data by sampling $x^i\sim p_{\boldsymbol\theta}(x_0|c,\mathbf{e}_u^i)$.

  2. Impact of $\mathcal{L^\text{C}}(\boldsymbol\theta)$ and CBS on the Diffusion Model:

     We primarily design the loss based on the lottery ticket hypothesis [6]. Importantly, this loss term only affects a small subset of parameters within the U-Net architecture, and we limit the trainable weights to those sampled from sufficiently large layers. This localized influence of $\mathcal{L^\text{C}}(\boldsymbol\theta)$ ensures that the benign performance of the model remains stable. Additionally, the weights of the caption backdoor subnet (CBS) are trained jointly with the diffusion losses $\mathcal{L_t^{\text{DM}}}(\boldsymbol\theta)+\mathcal{L_t^{\text{Trig}}}(\boldsymbol\theta,\mathbf{e}_u^i)$. By jointly training the CBS and the diffusion model, shared parameters are preserved within the diffusion model, maintaining the validity of the lottery ticket hypothesis and minimizing the CBS's impact on the model’s overall capacity.

  3. Empirical results on convergence and performance:

     Empirical results confirm that incorporating $\mathcal{L_t^{\text{Trig}}}(\boldsymbol{\theta})$ and $\mathcal{L^\text{C}}(\boldsymbol{\theta})$ does not hinder model convergence. As demonstrated in Table 2 of the main paper (for text-to-image diffusion models) and Table 6 in Appendix F (for image diffusion models), experiments conducted across a range of datasets and model architectures consistently show stable convergence, even with the inclusion of these additional loss terms. Precisely, while the backdoor loss slightly reduces benign metrics like FID and CLIP scores, the impact is minimal and practically insignificant.

[1] Stealing Your Data from Compressed Machine Learning Models, ACM DAC’20.

[2] Transpose Attack: Stealing Datasets with Bidirectional Training, NDSS Symposium’24.

[3] "How to backdoor diffusion models?", CVPR‘23.

[4] Trojdiff: Trojan attacks on diffusion models with diverse targets, CVPR’23.

[5] VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models, NeurIPS’23.

[6] Finding sparse, trainable neural networks, ICLR'19.

We would like to thank you for the insightful comments for our work. Below, we address the concerns raised in the feedback.

1. Concerns about the Threat Model and Applicability

   Our proposed method indeed requires full access to the victim model during both the backdoor injection and triggering phases, which is a relatively strong assumption as compared with other threat models, such as data poisoning alone or model theft. However, as highlighted in Footnote 1, real-life incidents, such as the 2016 case involving Anthony Levandowski at Google and the 2022 incident involving Qian Sang at Yahoo, underscore the prevalence of significant data security breaches. Moreover, such environments may be widely prevalent. For instance, publicly available open-source diffusion models for image generation, such as DeepFloyd IF, DeciDiffusion, and PixArt-α, can be freely downloaded and fine-tuned by anyone, including non-expert users, who may inadvertently introduce backdoors by directly running the compromised or unsafe code.

   It is worth noting that our attacker assumptions of the access to the training dataset and control over the training and inference processes, are consistent with existing research on backdoor attacks in diffusion models [1, 2, 3] and data exfiltration [4, 5].

2. Motivation for the Attack

   Regarding your concern about the practicality of the attack, particularly why an insider might choose to execute a backdoor attack rather than directly downloading the training data, we would like to clarify the following. In many environments, direct access to training data may be restricted due to organizational policies or technical constraints, such as data being transmitted remotely or stored in secure, controlled-access locations. These limitations make it difficult for an insider to obtain the data directly. Moreover, downloading training data often leaves detectable traces in system logs, which could raise alarms and expose the insider's actions. In contrast, our proposed backdoor attack provides a more covert method of data extraction, significantly reducing the likelihood of detection. This makes it an appealing option for insiders who prioritize stealth and wish to evade monitoring systems.

   We also agree with the scenario you proposed, where an insider aims to harm the company's interests by compromising the model's privacy-preserving nature. This perspective complements the motivations described in our paper. We will expand the revised version to explicitly highlight that the attack can also serve as a means of causing reputational damage, further enriching the threat model and its implications.

We hope that this additional context helps clarify the broader applicability and motivations behind the attack. We strongly believe that the proposed backdoor attack addresses an important and emerging threat in deep learning and privacy-preserving systems, where model integrity is increasingly at risk.

[1] VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models, NeurIPS’23.

[2] How to backdoor diffusion models?, CVPR’23.

[3] Trojdiff: Trojan attacks on diffusion models with diverse targets, CVPR’23.

[4] Stealing Your Data from Compressed Machine Learning Models, ACM DAC’20.

[5] Transpose Attack: Stealing Datasets with Bidirectional Training, NDSS Symposium’24.

We appreciate the opportunity to clarify our contributions and address specific concerns. Below, we provide detailed responses to each point raised.

• Reply to Q1 (Paper presentation):

  We respectfully disagree with the reviewer’s assessment that the paper presentation is poor. Based on feedback from the other three reviewers, the paper’s clarity and presentation received ratings of at least 3, which suggests that the contributions, motivations, and proposed solutions are effectively communicated.

  However, we appreciate the specific example provided. The sentence, "Our goal is to develop an innovative approach..." can be improved for clarity. A more precise phrasing would be:

  "Our goal is to implant a backdoor in diffusion models, allowing for the covert extraction of their training data without compromising their benign generative capabilities."

  We will revise the text accordingly to enhance readability and precision. Thank you for pointing this out, as it helps us further refine our work.

• Reply to Q2 (Practical usage of data exfiltration / Strong attacker assumption):

  Our attack scenario aligns with existing research on backdoor attacks in diffusion models, e.g., [1,2,3], and data exfiltration, e.g., [4,5], where the attacker requires access to the training dataset and control over the training and inference pipelines. While the adversary indeed requires initial access to the training data, our proposed method addresses a distinct security concern that differs significantly from direct data downloading. The key advantage of our approach lies in its covert nature - while direct data downloads are easily detectable through network monitoring, access logs, and security audits, our backdoor-based exfiltration method creates a subtle, persistent channel that can bypass conventional security measures and monitoring systems. This is particularly relevant in real-world scenarios such as cloud service providers handling sensitive client data, or outsourced ML development, where direct downloads would trigger immediate security alerts. Our method effectively embeds a backdoor into the model while maintaining its original functionality. This makes the backdoor much harder to detect using standard security protocols, emphasizing its significance as a critical threat that demands careful attention in machine learning security.

[1] VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models, NeurIPS’23.

[2] How to backdoor diffusion models?, CVPR’23.

[3] Trojdiff: Trojan attacks on diffusion models with diverse targets, CVPR’23.

[4] Stealing Your Data from Compressed Machine Learning Models, ACM DAC’20.

[5] Transpose Attack: Stealing Datasets with Bidirectional Training, NDSS Symposium’24.

• Reply to Q3 (Weak baseline):

  We would like to clarify that "Extracting Training Data from Diffusion Models" primarily investigates whether diffusion models memorize training data and can unintentionally reproduce it during inference. However, their method is impractical as an attack due to its intensive computational requirements. Specifically, their paper examines two categories of diffusion models:

  1. Text-to-Image Diffusion Models (Section 4.2):

     Their approach involves a two-step process: generating a massive number of images and applying membership inference techniques to filter candidates resembling the training data. However, this method is highly inefficient. For instance, in their experiments with Stable Diffusion, they generated 175 million images but identified only 94 images as highly similar to the training data, based on a threshold of 0.15 under the L2 distance function.

  2. Unconditional Diffusion Models (Sections 5.1 and 5.2):

     The authors evaluate untargeted attacks to confirm that smaller models also exhibit memorization. They leverage membership inference techniques, including white-box attacks like the Loss Threshold Attack (LTA). Notably, we have already included a comparison with LTA in Table 1 of our paper.

  While the work by Carlini et al. provides valuable insights into memorization within diffusion models, its objectives and methodology differ fundamentally from our focus on backdoor attacks. We appreciate your suggestion and will revise the related work section to more clearly delineate these distinctions and enhance the comprehensiveness of the discussion.

We appreciate the reviewer's insightful question regarding the assumptions made about the attacker's capabilities. Indeed, we acknowledge that access to both the training data and the ability to modify the training process is a relatively strong assumption as compared with other threat models, such as data poisoning alone or model theft. However, we would like to highlight several real-world scenarios where attackers can possess varying levels of access in ML development environments.

1. Full-Access Environment:

   In this scenario, the attacker is a team member responsible for training a model using the dataset. The attacker can easily modify the training code to implant a backdoor into the diffusion model during the training process. This approach enables covert data exfiltration by embedding a trigger within the model, avoiding direct dataset downloads that could be detected through network monitoring.

2. Protected Environment:

   The attacker has access to the training process but not to the raw training dataset, which is securely stored on a remote server. In this case, the attacker cannot directly download or inspect the images from the training dataset. However, the attacker can still inject a backdoor by recording features of each image the model encounters during training and dynamically assigning corresponding trigger embeddings to these images. Even without direct access to the dataset, the attacker can exploit training iterations to implant a backdoor covertly.

3. Secure Environment:

   In this setup, the attacker does not participate directly in the training process but acts as an algorithm provider. This scenario is practical in real-world settings where an attacker can upload malicious training code to platforms like GitHub. A non-expert user might inadvertently use this code to train their models, unknowingly introducing a backdoor. Similarly, in ML marketplaces (e.g., Algorithmia), an attacker could sell a compromised algorithm that users adopt, leading to data leakage once the trained model is deployed. Given that our proposed method maintains strong benign performance, it is challenging for users to detect the presence of a backdoor.

Defense Mechanisms and Access Control:

In secure and protected environments, even if an attacker employs mechanisms to track images used during training and assigns specific triggers, strong defenses can significantly mitigate such attacks. For example, image augmentations [1,2] can introduce variations across training epochs, effectively obfuscating the identities of individual images. When these augmented images exhibit noticeable differences from their original counterparts, it becomes challenging for an attacker to rely on simple similarity metrics to consistently identify identical images. Consequently, the model may mistakenly treat augmented images from the same image as different entities, consuming the model's memory capacity and leading to a reduction in the attack success rate (ASR).

[1] CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features, ICCV’19.

[2] Effective Data Augmentation With Diffusion Models, ICLR’24.