DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing

[Q4] Ablation of learning rate choice

Thank you for your thoughtful comment. We agree that identifying the best hyperparameters for DiffusionGuard is important for its effectiveness. Therefore, we conducted additional experiments using various learning rate (PGD step size) values, and the results are presented below (Table C).

Table C. Learning rate (step size) hyperparemeter search of DiffusionGuard

As shown, we find that the learning rate of 0.5/255 or 1.0/255 yields the best performance, and the performance gets worse as the learning rate increases. The hyperparameter search suggests that DiffusionGuard performs better when the learning rate is smaller, which is better aligned with the general intuition of optimization, unlike some previous paper which stated that larger learning rates may cause better performance.

---

[Q5-1] When malicious user edits only the eyes in a face photo

To address the reviewer's concern, we conducted additional demonstration experiments in which only certain facial features are inpainted (e.g., shape of pupil, line of sight, shape of mouth), by crafting new masks that correspond to these new setups. We include the qualitative demo in Figure 24 of Appendix J, in which DiffusionGuard still causes the editing to fail, maintaining its protective effectiveness.

We believe this is because the noise is applied all over the face, most of the noise survives even when certain facial features are inpainted, resulting in a failed edit.

Dear Reviewer WYYe,

We sincerely appreciate your insightful comments. They were incredibly helpful in improving our draft. We have addressed each comment in detail below.

---

[Q1] Clarification of loss maximization

Thank you very much for your careful review, and we apologize for the confusion. The loss should indeed be maximized, as stated in Eq. 5. We have clarified this in the revised draft.

---

[Q2] Reason for choosing L2 norm of the predicted noise.

In developing our method, we experimented with various loss function designs, including the L1 norm and total variation, as you mentioned. We found that maximizing the L2 norm empirically provided the most effective protection results. Table A (below) reports the performance of DiffusionGuard when using the L1 norm and total variation of the predicted noise as the loss function.

Table A. DiffusionGuard with L1 or total variation loss

As shown, using the L2 norm of the predicted noise results in the best (lowest) value for all metrics for both seen and unseen masks.

---

[Q3] Influence of the early step T value

Thank you for your constructive comment. As per your suggestion, we conducted additional ablation experiments by selecting T after splitting the timesteps into 5 equal intervals due to resource constraints. The results are presented in the Table B below.

Table B. Ablation of different early step values

We observed that T values around the first and second intervals yield the best protection strength, which we attribute to the significance of early denoising steps in inpainting models. This finding aligns with the motivation and approach of our method, which emphasizes early-stage (closer to pure Gaussian noise, i.e., T) loss.

[Q5-2] Effectiveness of dividing perturbation into sub-perturbations

This is a very intriguing idea! As suggested, we conducted a series of experiments and detailed them in Appendix J.1 of our updated draft.

As shown in Figure 25 of Appendix J.1, we divided a face in a given image into three subparts: (1) eyes and forehead, (2) mouth and cheeks, and (3) neck. Correspondingly, we split the original mask into three masks, each targeting a subpart. Using DiffusionGuard, we generated adversarial perturbations for each subpart, then merged them into a single perturbation matching the original mask's shape.

We compared this merged perturbation with the original DiffusionGuard perturbation under four scenarios: editing the eyes, mouth, neck, and the entire image. The quantitative results are presented in Table D below. Interestingly, the merged perturbation performed similarly to the original for localized edits and slightly better for full-image protection.

Table D. Sub-perturbation quantitative results

While the idea of generating sub-perturbation has great potential for development, it did not consistently enhance protection in this experiment, likely because the protective noise over targeted areas is not inputted to the inpainting model. Despite this, the approach shows promise, especially when inpainting the background, instead of inside the face. For instance, if a malicious user regenerated everywhere except the eyes and the nose, this idea could be more effective than default perturbations.

Thank you again for suggesting this fascinating idea about sub-perturbations, and we will continue to explore any possible directions to enhance its effectiveness.

---

[Q6] Figure 7a too small

Thank you for your careful review. As per your suggestion, we have updated Figure 7a to make it wider, improving its visualization. This change has been incorporated into the updated draft for your convenience.

Q1

I was confused about that XD. Now this concern is solved.

Q2

Thanks for tons of experiments. Choosing $L_2$ due to better performance, that make sense, but more explaination or analysis will benefit. For example, I'm not sure if it's correct, maybe the $L_2$ is related to the reconstruction loss or something else, so it have better performance? I'm expecting a convincing explanation.

Q3+Q4

This solved my concern about the ablation study, which will provide a basic understanding of different parameters.

Q5-1

Just for double check, could you clarify the adversarial sample was generated with the previous mask (main object) or the new mask (eye)?

5-2

Very interesting findings. I'm surprised that the merged perturbation is a little bit better than the default one. I think this could be a potential research direction that atomize the image into several sections, then optimize for them. A good speed up strategy such as optimize all blocks at the same time will be a huge innivation.

Dear reviewer WYYe,

Thank you for your thoughtful feedback and insightful questions. We deeply appreciate the time you’ve taken to engage with our work and offer such valuable perspectives. Below, we address each of your points in detail:

[R1] Question about L2 loss

Thank you for the question. We agree with your insight that L2 leads to better performance because diffusion models are trained for the MSE loss (e.g., between target noise and its prediction). More specifically, this training would effectively define the likelihood of the diffusion model with MSE as well; for instance, [1] defines the conditional log-likelihood using the L2 distance of the input image and a diffusion-denoised image.

Therefore, attacking the diffusion model using the training loss would be the most effective, and other losses could be less effective. For example, an adversarial perturbation generated to maximize the L1 loss might not maximize the L2 loss. If a model trained using the L2 loss receives such input, the model would process it more accurately than when it receives an adversarial perturbation generated using the L2 loss.

[R2] Resolved concern about ablation study

We are delighted that our additional ablation studies addressed your concern and provided clarity on different parameters. Thank you for your thoughtful feedback and encouraging words.

[R3] Clarification about adversarial perturbation for eye editing experiments

To clarify, the adversarial perturbations for the example in [W5-1] (Figure 26) were generated using the previous mask (main object). In other words, the adversarial perturbation from our main experiment was directly reused, making this a transferred setup. We have updated our draft to reflect this clarification.

[R4] About sub-perturbations

We fully agree that atomizing an image into several sections and optimizing for each individually presents a promising research direction. One interesting application of this approach could be for images containing multiple independent entities to protect—for example, an image with multiple people. In such cases, generating adversarial perturbations for each face separately and then merging them might prove more effective than simply optimizing for all faces simultaneously.

We deeply appreciate your insightful suggestion and are excited about the possibilities of future research direction. This discussion has been immensely thought-provoking and has further motivated us to explore these ideas in future work.

[1] Jaini, P., Clark, K., and Geirhos, R., 2024. Intriguing properties of generative classifiers. International Conference on Learning Representations, 2024.

Thank author for thorough response during the rebuttal period. I am pleased to see that all my previous concerns have been adequately addressed, and I have accordingly increased my rating to 6.

While I find this paper to be methodologically sound and well-executed, I should note that although I have experience with attack against LDM and LDM-based defenses, I am not specifically an expert in inpainting. Nevertheless, my first impression of attacking inpainting is, there could be a solution that atomize image into minimum semantic sections as I mentioned previously, which might lead to more robust attack methods against inpainting. This consideration somewhat tempers my enthusiasm for the novelty of the approach, preventing me from assigning a higher score of 8 or 10.

That said, I want to commend the experimental section, which comprehensively addresses all my concerns. The authors' diligent efforts during the rebuttal period have successfully clarified the vast majority of my questions. The thorough responses and additional analyses have significantly strengthened my confidence in the work's validity and contribution to the field.

[W3] Somewhat narrow problem setting and evaluating against instruction-guided editing

Thank you for your constructive feedback. Our proposed method can be used with instruction-based models such as InstructPix2Pix as well, especially the early-stage loss component. Although mask augmentation cannot be used as instruction-based models do not accept any mask input, our loss can still be applied.

Our focus on inpainting methods stems from their superior practical usefulness and flexibility in editing. While instruction-based methods like InstructPix2Pix can edit images without a mask, they tend to preserve high-level structures, such as body posture, which limits their capacity for drastic modifications (see Figure 23 in Appendix I for failure cases of InstructPix2Pix). In contrast, masked inpainting allows for the complete regeneration of designated areas. Consequently, we believe that inpainting-based editing methods, which are the focus of this paper, offer more practical value for complex scenarios than instruction-based (non-inpainting) editing methods.

Furthermore, to strengthen our work, we have extended our evaluations to include instruction-based methods such as InstructPix2Pix. Our results show that DiffusionGuard provides superior protective effectiveness compared to existing methods when applied to InstructPix2Pix, as detailed in the table below.

Table C. Comparison using InstructPix2Pix

We have included these results in Table 7 of Appendix E.4 with more details.

---

[W4] Missing references on harmful concept removal

Thank you for the constructive feedback. We have updated the draft as per your suggestion, and included the recent references on harmful concept removal in Section 5 (Related works).

Dear Reviewer ZPiB,

We sincerely appreciate your insightful comments. They were incredibly helpful in improving our draft. We have addressed each comment in detail below.

---

[W1] Effects of larger masks at test time

This is an interesting question! We conducted additional experiments to test this scenario. Specifically, we generated two larger masks by dilating the existing masks (one seen and one unseen), ensuring they are significantly larger, with an average increase of 26% in mask area. As shown in Table A (below), we found that DiffusionGuard maintains strong protective effectiveness compared to the baselines, even when a larger mask is provided at test time.

Table A. Using a larger mask at test-time

We explain this in detail in Appendix E.5 and Table 8 in our updated draft.

---

[W2] Inference starting with different timesteps or using different sampler

Thank you for the intriguing question. To address this, we conducted two additional experiments: testing DiffusionGuard with DDIM using different steps, and testing with a different sampler (DPM-Solver) using 25 steps. Specifically, we tested DDIM with three additional inference steps (25, 40, 75), corresponding to T values of 961, 976, and 963, respectively. Additionally, we tested DPM-Solver with 25 steps. In all cases, we found that DiffusionGuard operates independently of inference steps and consistently outperforms baseline methods, as shown in Table B below. As shown in the table, DiffusionGuard consistently shows the best performance without being particularly sensitive to the number of sampling steps or the sampler type.

Table B. Evaluation using different number of DDIM inference steps or sampler

Note that while we presented two main baselines and unseen mask set only in this table due to space limit, DiffusionGuard outperformed all other baselines in a similar manner in both seen and unseen mask sets. We include the full detail of the experiments in Table 9 of Appendix E.6.

[R3] Additional experiments with InstructPix2Pix

Thank you for the detailed response. Based on your feedback, we conducted two editing experiments with InstructPix2Pix, one by starting from a malicious image and changing the face, and one by starting with a celebrity image and changing the background.

• Starting with malicious image and changing the face to celebrity (Figure 24 of Appendix I.1): We started with a person being arrested and instructed InstructPix2Pix and Stable Diffusion Inpainting to replace the subject's face with that of a celebrity. Using the prompt "Turn the man into {celebrity name}", we edited 10 images of well-known figures recognized by Stable Diffusion Inpainting. As a control, we applied the same prompt to an inpainting model with a mask designating the face area.

  The results reveal two key limitations of InstructPix2Pix: (1) the instruction affects unintended areas, such the police officer in the image, always turning them into the celebrity in all cases, or even the background, and (2) the generated face quality is lower than that of the inpainting model. For highly conditioned celebrities like football players (e.g., Lionel Messi, Cristiano Ronaldo), even the background changes to match their context, such as resembling a football stadium.

• Starting with celebrity image and changing the background (Figure 25 of Appendix I.2): Starting with celebrity images, we used InstructPix2Pix with the instruction "Change the background to jail" and Stable Diffusion Inpainting with the prompt "A photo of a person in jail". For inpainting, we used a mask designating the face region.

  As shown in Figure 25, InstructPix2Pix struggles with this task, often altering the celebrity into a different person (Rows 1, 3, 4, 6) or over-conditioning on the source image such as the postures or letters inside them (Rows 2, 5, 6). In contrast, the inpainting model successfully generates accurate representations of celebrities in a jail setting in all cases.

[R2] About using different timesteps or schedulers

We believe that optimizing for a single integer timestep (e.g., 1 to 1000) generalizes to nearby timesteps because timesteps are converted into real-number sinusoidal embeddings, similar to Transformer positional embeddings, as described in the DDPM [1] paper: "Diffusion time t is specified by adding the Transformer sinusoidal position embedding into each residual block." While diffusion model libraries like Diffusers represent timesteps as discrete integers (e.g., 981), this is arbitrary, as timesteps exist on a continuous spectrum. Optimizing for one timestep influencing nearby ones is reasonable, given the common practices like interpolating positional embeddings (e.g., Vision Transformer [2]) or adjusting the number of timesteps dynamically (e.g., Consistency Models [3]).

Another example can be found in research about protection against diffusion models as well. In AdvDM [4], adversarial perturbations are generated by randomly sampling timesteps uniformly from all of {1, 2, ..., 1000}, rather than specific values like {21, 41, ..., 981}.

[1] Ho, J., Jain, A., and Abbeel, P., 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.

[2] Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al., 2021. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.

[3] Song, Y., Dhariwal, P., Chen, M., et al., 2023. Consistency models. International Conference on Machine Learning, 2023.

[4] Liang, C., Wu, X., Hua, Y., et al., 2023. Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples. International Conference on Machine Learning, 2023.

Dear reviewer ZPiB,

Thank you for your thoughtful feedback and insightful questions. We deeply appreciate the time you’ve taken to engage with our work and offer such valuable perspectives. We addressed each point in detail in the replies below.

[R1] Qualitative examples for edits using various masks

Thank you for a constructive comment. Based on your new suggestion, we added a new visualization in Figure 29 in Appendix K, including different masks and their corresponding edit results.

We used a total of 7 masks: 4 smaller than the head (Rows 2–5), 1 matching the head size (Row 1), and 2 larger than the head (Rows 6–7). In all cases, DiffusionGuard results in a successful protection, causing the edit to result in unrealistic images.

A notable insight is that masks larger than the head, which include regions outside the head (especially the background), restrict editing flexibility. In the figure, for these larger masks, the background remains fixed as an empty entrance to a dark hallway (Rows 6–7) to align with the dark surroundings of the source image, and the shirt color is consistently dark blue due to the original clothing. In contrast, the smaller masks (Rows 1–5) allow diverse backgrounds, such as a wall, hallway, or hospital ward. This suggests that larger masks, as they include the background, are less ideal for flexible editing and emphasize the need to focusing on the head, especially the face.

Another observation is that for all 7 masks, including the smoother backgrounds of Rows 6–7, the boundary between the original face and the edited background is visibly distinct in the protected images. This likely results from the attack disrupting the inpainting model's internal processes, causing it to misinterpret colors. This effect could be useful for future research directions.

Dear Reviewer QU4Q,

We sincerely appreciate your insightful comments. They were incredibly helpful in improving our draft. We have addressed each comment in detail below.

---

[W1] Testing against purification methods specifically designed for diffusion models

We appreciate your constructive comment. Following your suggestion, we conducted additional evaluations of DiffusionGuard and the baseline models against IMPRESS, a purification method specifically designed for defense against diffusion models. The results, as detailed in Table A below, demonstrate that DiffusionGuard remains more resilient compared to the baseline methods, achieving best (bold) or second-to-best (underline) results in all metrics. This shows the robustness of DiffusionGuard against targeted purification strategies.

Table A. Evaluation after purifying each protection method with IMPRESS

---

[W2] DiffusionGuard against InstructPix2Pix

Thank you for your constructive feedback. Our proposed method can be used with instruction-based models such as InstructPix2Pix as well, especially the early-stage loss component. Although mask augmentation cannot be used as instruction-based models do not accept any mask input, our loss can still be applied.

Our focus on inpainting methods stems from their superior practical usefulness and flexibility in editing. While instruction-based methods like InstructPix2Pix can edit images without a mask, they tend to preserve high-level structures, such as body posture, which limits their capacity for drastic modifications (see Figure 23 in Appendix I for failure cases of InstructPix2Pix). In contrast, masked inpainting allows for the complete regeneration of designated areas. Consequently, we believe that inpainting-based editing methods, which are the focus of this paper, offer more practical value for complex scenarios than instruction-based (non-inpainting) editing methods.

Furthermore, to strengthen our work, we have extended our evaluations to include instruction-based methods such as InstructPix2Pix. Our results show that DiffusionGuard provides superior protective effectiveness compared to existing methods when applied to InstructPix2Pix, as detailed in Table B below.

Table B. Comparison using InstructPix2Pix

We have included these results in Table 7 of Appendix E.4 with more details.

[Q1] Effect of editing prompts on the generation of perturbations

Thank you for this interesting question. In our experiments, we generated perturbations by setting the text prompt to an empty string ("" in Python) to maintain neutrality and ensure that they are not biased towards any specific test-time prompts, following PhotoGuard [1].

Regarding the effects of different prompts, our observations indicate that perturbations generated with a non-empty text prompt are more tailored to that specific prompt. This makes them more effective at protecting against similar prompts due to their targeted nature but less effective against prompts that are unrelated.

We have clarified this in Appendix D (Experimental details) in our updated draft.

[1] Salman, H., Khaddaj, A., Leclerc, G., Ilyas, A., and Madry, A., 2023. Raising the cost of malicious AI-powered image editing. International Conference on Machine Learning, 2023.

Dear Reviewer QU4Q,

We greatly appreciate the time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that seven days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much!

Many thanks,

Authors

[W7,8] Suggestion to use a larger test set, and trade masks and prompts for more images

Thank you for your constructive feedback. While our original benchmark size aligns with most existing works in image protection against diffusion-based editing (42 images, each with 5 masks and 10 prompts), we agree that expanding the test set is important. For reference, SDS [1] employs a test set of 100 portrait images, each with 1 mask and 1 prompt, while PhotoGuard [2] tests on a smaller number of images but with up to 60 prompts each.

In response, we collected 688 new images from the FFHQ [3] dataset, each paired with two human-validated, automatically generated masks (one seen, one unseen). This significantly expands the test set compared to existing works. Our evaluation on this larger dataset shows that DiffusionGuard outperforms baseline methods by a large margin. Quantitative results are provided in Table D below. For this experiment, we used all 10 prompts without reducing the number of prompts, totaling 13760 edited instances for each protection method.

Table D. Evaluation of each method on a new test set with 688 images

We plan to collect more images to further extend our dataset and release it publicly soon. We have also included the new dataset in the supplementary materials, resized and compressed due to attachment size limits.

[1] Xue, H., Liang, C., Wu, X., and Chen, Y., 2024. Toward effective protection against diffusion-based mimicry through score distillation. International Conference on Learning Representations, 2024.

[2] Salman, H., Khaddaj, A., Leclerc, G., Ilyas, A., and Madry, A., 2023. Raising the cost of malicious AI-powered image editing. International Conference on Machine Learning, 2023.

[3] Karras, T., Laine, S., and Aila, T., 2019. A style-based generator architecture for generative adversarial networks. Conference on Computer Vision and Pattern Recognition, 2019.

Dear Reviewer ScsB,

We sincerely appreciate your insightful comments. They were incredibly helpful in improving our draft. We have addressed each comment in detail below.

---

[W1] Misleading title and lack of comparison to other editing methods (e.g., InstructPix2Pix)

Thank you for your constructive feedback. We acknowledge that the title may not fully reflect the scope of our work. We plan to revise it to "DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Inpainting”, which addresses your concern more accurately.

Our focus on inpainting methods stems from their superior practical usefulness and flexibility in editing. While instruction-based methods can edit images without a mask, they tend to preserve high-level structures, such as body posture, which limits their capacity for drastic modifications (see Figure 23 in Appendix I for failure cases of InstructPix2Pix). In contrast, masked inpainting allows for the complete regeneration of designated areas. Consequently, we believe that inpainting-based editing methods, which are the focus of this paper, offer more practical value for complex scenarios than instruction-based (non-inpainting) editing methods.

Furthermore, to strengthen our work, we have extended our evaluations to include instruction-based methods such as InstructPix2Pix. Our results show that DiffusionGuard provides superior protective effectiveness compared to existing methods when applied to InstructPix2Pix, as shown in Table A below.

Table A. Comparison using InstructPix2Pix

We have included these results in Table 7 of Appendix E.4 with more details.

---

[W2] Lack of comparison to MagicBrush

Thank you for your comment. However, it is important to note that MagicBrush does not propose a new inpainting model but rather focuses on instruction-based editing without requiring an inpainting mask. This is explicitly stated by the authors: "We fine-tune InstructPix2Pix on MagicBrush and show that…" Although the dataset in the MagicBrush paper includes a mask for each edit instance, the authors clarify that no inpainting-specific models are trained using this dataset.

Our work evaluates defense against inpainting using Stable Diffusion (SD) Inpainting 1.0 and 2.0, as they are the most widely used inpainting models. Most related studies also test primarily on these two models, as very few other public inpainting models are available.

---

[W3] Misleading description of the unique behavior of inpainting models

To clarify, both the inside and outside of the mask in Fig. 2 are generated solely by the denoiser, starting from pure Gaussian noise, without any external copy-and-paste operation. Starting from pure Gaussian noise, the denoiser of the inpainting model is able to sharply denoise the masked region in the early steps by observing the source image which is given as an additional input. In contrast, a normal diffusion model with the same inputs does not exhibit this behavior. This indicates that this copy-and-paste-like behavior of inpainting models is a learned behavior during their fine-tuning process and we refer to this as a "unique behavior".

As you noted, the rest of the image is generated similarly to normal diffusion models, lacking fine details in the early stages. This observation inspired our early-stage loss, based on the hypothesis that the early completion of the masked region may influence the generation of the rest of the image.

[W4] Reason for targeting early steps not sufficiently convincing

As clarified in [W3], the model does generate the inside of the face as well, and not just the outside, at the very early denoising step. This distinct behavior from normal diffusion models motivated targeting the early denoising stages.

Following the common practice done by the baselines (PhotoGuard, AdvDM, Mist, SDS), we post-processed the generated results solely for demonstration purposes by copying and pasting the inside of the mask from the source image. This post-processing occluded the raw generated image and made the inside of the masks appear clean. We apologize for the confusion, and we have updated our draft to clearly state this. We also included raw generated images without post-processing in Figure 11 of Appendix D.

Additionally, we also conducted additional experiments following your suggestion, by applying Eq. 4 to multiple timesteps at the same time. Specifically, we applied the loss at timesteps $\textbraceleft\frac{T}{10}, \frac{2T}{10}, ..., T\textbraceright$ or $\textbraceleft\frac{T}{5}, \frac{2T}{5}, ..., T\textbraceright$ simultaneously. As shown in Table B (below), the performance degraded when not focusing solely on the early timestep, supporting our claims.

Table B. Ablation of applying our loss over multiple steps at the same time

---

[W5] Effects of bigger masks at test time

This is an interesting question! We conducted additional experiments to test this scenario. Specifically, we generated two larger masks by dilating the existing masks (one seen and one unseen), ensuring they are significantly larger, with an average increase of 26% in mask area. As shown in Table C below, DiffusionGuard continues to demonstrate strong protective effectiveness, outperforming the baselines even with larger test masks.

Table C. Using a larger mask at test-time

---

[W6] Reliability issue of PSNR

We agree that PSNR may not fully capture a successful defense. Still, we have incorporated three additional metrics that more accurately reflect the effectiveness of our defense strategy: CLIP similarity, CLIP directional similarity, and ImageReward. These metrics assess how well the edited images align with the editing instructions and the overall editing performance, respectively.

Following your suggestion, we also added a new chart (Figure 14 of Appendix E.2.2) by replacing PSNR with CLIP directional similarity in Figure 5b, 5c. The results are still aligned with our findings: DiffusionGuard consistently outperforms PhotoGuard across different compute and noise budgets.