MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges

We sincerely express our gratitude for your acceptance and constructive comments. Due to space constraints, we include additional_tables.pdf and visualizations at the following link: https://anonymous.4open.science/r/ICML25-5787/. References to Table x and Figure x in our responses correspond to the materials provided in this link. The responses to your questions and weaknesses are listed below:

Q1 Alternative evaluation methods for stealthiness

A1: Entropy has been widely adapted to measure privacy in the security field [1,2]. Greater entropy indicates an increasing difficulty in distinguishing between different events and thus a better "anonymity". Similarly, we use entropy to measure the anonymity of these experts. The higher the entropy, the harder it is to distinguish these experts, and thus the backdoor experts are more difficult to be detected. As another way to measure stealthiness, we evaluate the performance of each expert on benign tasks. If any expert consistently performs poorly on benign tasks, it could be easily detected and removed by the user. In particular, we input clean image $x_t^c$ to simulate the benign generation task and get the output $\epsilon_i(x_t^c,t;\theta_i)$ of each expert. Next, we predict the corresponding $x_0^{\epsilon_i}$ using Eq. 1 in our paper. Finally, we compute the MSE between $x_0^{\epsilon_i}$ and clean images. The results are as follows:

The results show the backdoor experts generate noticeably different contents in the benign generation task if WRS is not applied. On the contrary, all experts in MixBridge with WRS produce outputs similar to the benign image. We have provided the visualization results in Figure 6 in the paper. In addition, the results of the direct metric (MSE) are consistent with those from the indirect metric (entropy).

Q2 MixBridge has better performance

A2: We have noticed the MixBridge outperforms a single I2SB in the super-resolution task. The major reason is we apply a MoE mechanism to train the model, which brings much stronger capacity. Specifically, it helps mitigate the conflicts between the benign generation task and multiple backdoor attacks, enabling MixBridge to achieve better performance.

Q3 Compare with cutting-edge methods

A3: To the best of our knowledge, there are no backdoor attack methods designed for I2I diffusion generation models. The setting of the unconditional diffusion model, which starts the diffusion process from a standard Gaussian noise, may not be applicable to the I2I framework. Here, we take two commonly used backdoor attack methods for standard unconditional diffusion models, BadDiffusion [3] and VillanDiffusion [4], to compare with our proposed MixBridge in the I2I setting. The results are as follows:

The results show that the two backdoor attack methods designed for unconditional diffusion models perform significantly worse than MixBridge in both benign generation and backdoor attack tasks. This demonstrates backdoor attacks designed for unconditional models are not easily transferable to the I2I setting.

Q4 The relationship between existing methods and the method in this paper is not discussed.

A4: We have already discussed the relationship between existing methods and our proposed backdoor attack. Here, we present a simple summary. Existing backdoor attacks on diffusion models can be roughly categorized into two lines: attacks on the unconditional diffusion models and attacks on the T2I diffusion models. The former injects triggers into the input Gaussian noise to generate malicious outputs, while the latter corrupts text inputs with triggers to manipulate the diffusion process. In contrast, in this paper, our method targets to conduct heterogeneous backdoor attacks against I2I bridge-based diffusion models.

We will revise the Related Work section to clearly articulate the distinctions between existing methods and our proposed backdoor attacks.

Q5 Possible defense

A5: We present the discussion about defense methods in the response to the Reviewer 8an4-Q1.

In addition, we sincerely appreciate your meticulous review! We will revise these typos and clarify the corresponding expressions.

Reference:

[1] Quantifying and measuring anonymity [2] Towards measuring anonymity [3] How to backdoor diffusion models? [4] Villandiffusion: A unified backdoor attack framework for diffusion models

We sincerely thank you for accepting our work and for your constructive comments. Due to space constraints, we include additional_tables.pdf and visualizations at the following link: https://anonymous.4open.science/r/ICML25-5787/. References to Table x and Figure x in our responses correspond to the materials provided in this link. The responses to your questions and weaknesses are listed below:

Q1 Possible defense

A1: To the best of our knowledge, existing defense mechanisms are primarily focused on T2I diffusion models and unconditional diffusion models. In addition, they mainly focus on a single expert model for backdoor attacks. In contrast, our proposed attack targets the bridge-based I2I diffusion model with heterogeneous MoE backdoor attacks. Thus, previous defense mechanisms may not be applicable to our setting.

Here, we conduct some experiments to investigate if previous defense mechanisms can be adapted to the I2I framework. We take Elijah [1] as an example to detect the trigger for a simple MixBridge model with two experts. Elijah assumes that the backdoor attack redirects the clean distribution $x_c^t\sim\mathcal{N}(\mu_c^t,\cdot)$ to the backdoor distribution $x_b^t\sim\mathcal{N}(\mu_b^t,\cdot)$ at step $t$ using a trigger $\tau$. The trigger can be optimized via the following formula.

$\tau=\arg\min_\tau\|\mathbb{E}_{\epsilon\sim\mathcal{N}(0,1)}[M(\epsilon+\tau,T)]-\lambda\tau\|.$

$M$ represents the model, and $\lambda$ is a hyperparameter related to the diffusion process.

However, the original Elijah defense is built upon the assumption that the generation process starts from a standard Gaussian noise (i.e., $\mu_b^T=0$). In the I2I scenario, we propose a modified version of Elijah. Assume the gap between the benign and backdoor distributions caused by $\tau$ to be expressed as: $\mu_b^t-\mu_c^t=\lambda^t\tau.$ According to the Eq. 1 in our paper, we derive the following objective.

$\tau=\arg\min_\tau\|\sigma_1\mathbb{E}[\epsilon_{\text{Mix}}(x_1^c+\tau,t=1;\theta_{\text{Mix}})-\epsilon_{\text{Mix}}(x_1^c,t=1;\theta_{\text{Mix}})]-\lambda\tau\|.$

Ideally, one should first generate the trigger and finetune the model with the Elijah method, and perform backdoor attacks again to evaluate if the defense is effective. We compare the attack performance of the generated trigger with that of the original baseline.

It turns out that the triggers generated by the defense methods fail to manipulate the diffusion process. In other words, they cannot inverse the trigger in the MixBridge, let alone defend against the attack. We attribute such failures to the complex distribution in the I2I generation process. In this case, the image distribution gap cannot be simply modeled by a trigger $\tau$. In addition, Elijah does not solve the heterogeneous backdoor. Please refer to Figure 2 for the visualizations of triggers generated by Elijah.

Reference:

[1] Elijah: Eliminating backdoors injected in diffusion models via distribution shift

We sincerely thank you for accepting our work and for your constructive comments. Due to space constraints, we include additional_tables.pdf and visualizations at the following link: https://anonymous.4open.science/r/ICML25-5787/. References to Table x and Figure x in our responses correspond to the materials provided in this link. The responses to your questions and weaknesses are listed below:

Q1 Possible defense

A1: To the best of our knowledge, existing defense mechanisms are primarily focused on T2I diffusion models and unconditional diffusion models. In addition, they mainly focus on a single expert model for backdoor attacks. In contrast, our proposed attack targets the bridge-based I2I diffusion model with heterogeneous MoE backdoor attacks. Thus, previous defense mechanisms may not be applicable to our setting.

Here, we conduct some experiments to investigate if previous defense mechanisms can be adapted to the I2I framework. We take Elijah [1] as an example to detect the trigger for a simple MixBridge model with two experts. Elijah assumes that the backdoor attack redirects the clean distribution $x_c^t\sim\mathcal{N}(\mu_c^t,\cdot)$ to the backdoor distribution $x_b^t\sim\mathcal{N}(\mu_b^t,\cdot)$ at step $t$ using a trigger $\tau$. The trigger can be optimized via the following formula.

$\tau=\arg\min_\tau\|\mathbb{E}_{\epsilon\sim\mathcal{N}(0,1)}[M(\epsilon+\tau,T)]-\lambda\tau\|.$

$M$ represents the model, and $\lambda$ is a hyperparameter related to the diffusion process.

However, the original Elijah defense is built upon the assumption that the generation process starts from a standard Gaussian noise (i.e., $\mu_b^T=0$). In the I2I scenario, we propose a modified version of Elijah. Assume the gap between the benign and backdoor distributions caused by $\tau$ to be expressed as: $\mu_b^t-\mu_c^t=\lambda^t\tau.$ According to the Eq. 1 in our paper, we derive the following objective.

$\tau=\arg\min_\tau\|\sigma_1\mathbb{E}[\epsilon_{\text{Mix}}(x_1^c+\tau,t=1;\theta_{\text{Mix}})-\epsilon_{\text{Mix}}(x_1^c,t=1;\theta_{\text{Mix}})]-\lambda\tau\|.$

Ideally, one should first generate the trigger and finetune the model with the Elijah method, and perform backdoor attacks again to evaluate if the defense is effective. We compare the attack performance of the generated trigger with that of the original baseline.

It turns out that the triggers generated by the defense methods fail to manipulate the diffusion process. In other words, they cannot inverse the trigger in the MixBridge, let alone defend against the attack. We attribute such failures to the complex distribution in the I2I generation process. In this case, the image distribution gap cannot be simply modeled by a trigger $\tau$. In addition, Elijah does not solve the heterogeneous backdoor. Please refer to Figure 2 for the visualizations of triggers generated by Elijah.

Reference:

[1] Elijah: Eliminating backdoors injected in diffusion models via distribution shift

We sincerely express our gratitude for your acceptance and constructive comments. Due to space constraints, we include additional_tables.pdf and visualizations at the following link: https://anonymous.4open.science/r/ICML25-5787/. References to Table x and Figure x in our responses correspond to the materials provided in this link. Below are our responses to your questions and identified weaknesses:

Q1 Existing backdoor attacks for I2I model

A1: We acknowledge some previous studies have explored backdoor attacks on I2I models. However, none of them focus on diffusion-based models, let alone those based on the Schrödinger Bridge. We will restrict our discussion in the introduction to the diffusion-based models, and clarify the key differences from the previously mentioned I2I backdoor methods.

Q2 Alternative evaluation methods for stealthiness

A2: We have included the answer in our response to Reviewer LwZQ-Q1.

Q3 The stealthiness of the trigger needs further consideration

A3: Here, we evaluate the stealthiness based on trigger size. A smaller trigger is generally harder to detect. In particular, we conduct super-resolution task along with three heterogeneous backdoor attacks on the CelebA dataset using different trigger sizes in Table 1.

According to the experiments, the ASR remains 0.93 for a $16\times16$ trigger size (i.e., $1\%$ of the total image), which is a small trigger with relatively high stealthiness. In contrast, BadDiffusion [1] uses a $14\times14$ trigger on a $32\times32$ image, which occupies nearly $20\%$ of the image area. We present visualizations of different trigger sizes in Figure 1 in the anonymous link.

Q4 Why not one-stage WRS

A4: Following previous studies [2,3], we pre-train each expert separately and then finetune them jointly. Here, we conduct an ablation study for the one-stage method.

Intuitively, the one-stage model significantly underperforms compared to the two-stage version. A possible reason is in one-stage training the experts might suffer from conflicting optimization directions across different generation tasks in the early phase, leading to poor and suboptimal solutions. By using a two-stage method, the pre-training process naturally avoids such issues, and makes the finetuning process more stable and effective.

Q5 More details on hyperparameter

A5: As a trade-off hyperparameter, $\lambda$ controls the intensity of WRS. In this paper, to ensure the stealthiness of the backdoor attacks, we chose a large $\lambda$ to obtain high entropy. Here, we investigate how $\lambda$ affects the performance of the MixBridge.

The results show that a lower $\lambda$ improves benign generation but degrades the performance of backdoor attacks. We find that setting $\lambda=0.1$ or 1 leads to a relatively good tradeoff.

Q6 Possible Defenses

A6: Please refer to our response to Reviewer 8an4-Q1.

Q7 Attacker’s Capacity and Goal

A7: We apologize for the unclear explanation of the backdoor attack settings. On one hand, the model owner (or the attacker) may not be primarily motivated by financial gain. Instead, their intentions could include deliberate sabotage, reputation damage, or other non-economic incentives. For example, an attacker might aim to create chaos, undermine competitors’ credibility, or provoke legal and ethical issues to gain a strategic advantage. On the other hand, as you mentioned, backdoor techniques can be used for legitimate purposes like watermarking or personalization. Therefore, this paper can also potentially promote the studies for legitimate applications.

We will revise our corresponding claim in the paper according to your suggestion.

Reference: [1] How to backdoor diffusion models? [2] Warm: On the benefits of weight averaged reward models [3] Mogu: A framework for enhancing safety of open-sourced llms while preserving their usability