Experimental settings and baseline models

Please note that experiments were conducted under the same settings as in the main paper (i.e., using the Stable Diffusion Inpainting model with a total of 100 images, 50 prompts per image where we use only a single seed for a single prompt), ensuring a fair and consistent comparison.

Additionally, we chose the baseline models—Photoguard, AdvDM, and SDST—that employ one of three existing objective functions (i.e. minimize Eq.2, maximize Eq.2, minimize Eq.4) from prior protection methods mentioned in the main paper.

W3) Application to Diffusion Transformers

As the reviewer suggested, we evaluated the robustness of AdvPaint against an adversary using the inpainting model of Flux[1] and text-to-image model Pixart-$\delta$[2] where both methods leverage diffusion transformers. Unlike [2], we note that models like DiT[3] and Pixart-$\alpha$[4] are designed for generating images solely from text prompts using diffusion transformer architectures, which make them unsuitable for our tasks that require accepting input images.

Flux Inpainting Model

Flux provides an inpainting module based on multimodal and parallel diffusion transformer blocks. We utilized the “black-forest-labs/FLUX.1-schnell” checkpoint, and the image size was set to 512x512 to match our settings.

As shown in Figure 12 in Appendix A.3.3, AdvPaint effectively disrupts the inpainting process by causing misalignment between generated regions and unmasked areas. For example, it generates cartoon-style cows in (a) and adds a new rabbit in (b), while also producing noisy patterns in the unmasked areas of the images.

DreamBooth with DiT

Chen et al. introduced Pixart-$\delta$, which integrates DreamBooth [5] into DiT. We selected this model as the adversary’s generative framework since it supports feeding an input image along with a command prompt for performing generation. Additionally, as discussed in Section A.3.2, we demonstrate the transferability of AdvPaint to text-to-image tasks.

As shown in Fig. 13 (a) in Appendix A.3.3, AdvPaint-generated perturbations consistently undermine the generation ability of Pixart-$\delta$. Furthermore, AdvPaint also renders noise patterns that degrade the image quality on the resulting output images of the diffusion model, which aligns with the behavior of previous methods (i.e. Photoguard, AdvDM, SDST).

Protection for DiT-based Models

AdvPaint also effectively disrupts the original DreamBooth[5], as depicted in Fig. 13 (b). However, our findings indicate that AdvPaint and the previous methods are less effective against Pixart-$\delta$ that leverages DiT (Fig. 13 (a)). Additionally, compared to the results of LDM-based inpainting models in Figure 1, current methods are less effective when applied to DiTs. For example, in AdvPaint, discernible objects appear in the foreground inpainting tasks and new objects are not always generated in the background inpainting tasks. We believe this ineffectiveness stems from the distinct characteristic of DiT, which processes patchified latent representations. AdvPaint and our baselines are specifically designed to target LDMs, which utilize the entire latent representation as input, allowing perturbations to be optimized over the complete latent space. Thus, when latents are patchified in DiTs, perturbations may become less effective at disrupting the model's processing, thereby diminishing their protective capability. This discrepancy necessitates further research to develop protection methods specifically tailored to safeguard images against the adversary misusing DiT-based models. For instance, optimizing perturbations at the patch level rather than across the entire latent representation could prove more effective in countering the unique paradigm of image generation in DiT-based models.

[1] Black Forest Labs. (n.d.). Home. Retrieved November 21, 2024, from https://blackforestlabs.ai/

[2] Chen, J., Wu, Y., Luo, S., Xie, E., Paul, S., Luo, P., Zhao, H., & Li, Z. (2024). PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models. ArXiv, abs/2401.05252.

[3] Peebles, W.S., & Xie, S. (2022). Scalable Diffusion Models with Transformers. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 4172-4182.

[4] Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J.T., Luo, P., Lu, H., & Li, Z. (2023). PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. ArXiv, abs/2310.00426.

[5] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022). DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 22500-22510.

W2) Analysis of Noise Levels and Iteration Steps

We thank the reviewer for raising this important concern. To address it, we conducted two experiments, varying the noise budget $\eta$ and the number of iteration steps, and summarized the results in the tables below. The updated tables have also been included in Appendix A.6.

1. Noise Levels $\eta$

First, we conducted an experiment with different values of $\eta$, ranging from 0.04 to 0.1. While the PSNR values of adversarial examples decrease as $\eta$ increases, we observed consistent improvements across all evaluation metrics, including FID, precision, and LPIPS. For AdvPaint, we set η to 0.06, as it effectively balances protection against inpainting tasks with the quality of the protected image, achieving a PSNR of approximately 32 dB.

2. Iteration Steps

Second, we experimented with varying iteration steps. We evaluated iteration steps ranging from 50 to 150. Due to memory limitations, we set the default iteration steps to 250 in AdvPaint, as higher iterations result in memory overload. The results show that while there may not be significant improvement for iteration steps around 100 and 150, optimizing for 250 steps consistently outperforms lower iteration counts, validating our choice of 250 steps as the default setting for AdvPaint.

We hope these additional analyses comprehensively address the reviewer’s concern and provide deeper insights into the robustness of AdvPaint under varying adversarial intensities.

Dear Reviewer 37iq,

Thank you for your detailed comments and helpful suggestions regarding our manuscript. We have carefully considered each of your points and provided comprehensive responses below. If there are any aspects we may have inadvertently overlooked, please let us know, and we will gladly address them.

W1) Robustness of AdvPaint Against Custom Mask Configurations

We thank the reviewer for highlighting this important aspect of real-world scenarios.

In the real-world scenario, the purpose of inpainting may differ. To address these concerns, we conducted additional experiments in Fig.24 of Appendix A.8.3 using masks that (a) exceed or (b) overlap with the optimization boundary. Specifically, we visualized inpainting results where foreground masks were applied to regions without objects, simulating adversarial scenarios aimed at generating new objects in the background. These experiments demonstrate that AdvPaint remains robust in such diverse inpainting cases, generating inpainted images with no discernable objects.

Additionally, in Section 5.6, we present the inpainting results for two types of masks: $m^{out}$ and $m^{in}$. Since masks can be hand-crafted by adversaries and may exceed the enlarged bounding box used at the optimization step of AdvPaint, we simulate this scenario by randomly shifting the segmentation mask $m^{seg}$ in four directions (up, down, left, and right) to ensure that at least one boundary of the rectangular optimization box is exceeded. This shifted mask is referred to as $m^{out}$. In contrast, $m^{in}$ is generated by randomly shifting $m^{seg}$ while ensuring the entire mask remains inside the rectangular optimization box, reflecting scenarios where segmentation masks differ but are confined to the bounded region.

Our experiments, as shown in Table 3, demonstrate that AdvPaint maintains robust performance for both $m^{out}$ and $m^{in}$. Specifically, it achieves strong performance across key metrics, including FID, precision, and LPIPS scores, effectively safeguarding images under both the scenarios.

We hope these additional scenarios in evaluating the robustness of AdvPaint addresses the reviewer’s concerns.

The author's response addressed all my concerns. Therefore I will raise the score to 8.

Experimental Settings and Baseline Models

Please note that experiments were conducted under the same settings as in the main paper (i.e., using the Stable Diffusion Inpainting model with a total of 100 images, 50 prompts per image where we use only a single seed for a single prompt), ensuring a fair and consistent comparison.

Additionally, we chose the baseline models—Photoguard, AdvDM, and SDST—that employ one of three existing objective functions (i.e. minimize Eq.2, maximize Eq.2, minimize Eq.4) from prior protection methods mentioned in the main paper.

We present the quantitative results of DreamShaper and Stable Diffusion v2 inpainting model.

Q4) Experimental Details for Figure 7 and 8

We apply AdvPaint to both image-to-image and text-to-image diffusion models to demonstrate the transferability of AdvPaint. For image-to-image tasks, we implement the Stable Diffusion image-to-image pipeline using Diffusers (runwayml/stable-diffusion-v1-5). Specifically, we follow the default settings of the pipeline, where inference steps = 50, strength = 0.8, and guidance scale = 7.5.

For text-to-image tasks, we leverage Textual Inversion [1] and follow the official implementation and settings. Specifically, we set the inference steps to 50 and the guidance scale to 7.5. For the input images, where 3 to 5 images are required, we utilized the official dataset of DreamBooth. For both tasks, we used images of 512×512 size and randomly crafted the conditional prompts.

We have updated the above information in Section A.1.3 to clarify the experimental settings for Figure 10 and 11 (formerly Figure 7 and 8). We hope this additional detail resolves confusion regarding how these experiments were conducted.

W3, Q5) Evaluation on Inpainting Model Variants

We thank the reviewer for highlighting the concern regarding the use of a single inpainting model in the main experiments and the practicality of considering white-box settings. To address this, we conducted additional experiments on multiple inpainting model variants: HD-Painter [2], DreamShaper [3], and the Stable Diffusion v2 inpainting model [4].

HD-Painter proposes a training-free approach to inpainting and replaces the self-attention block in the Stable Diffusion (SD) inpainting model with a block designed to align the prompt more closely with the generated image. DreamShaper is a fine-tuned version of the SD inpainting model, widely available on public platforms. In all experiments, we followed the default settings as in their respective papers.

We evaluated AdvPaint against these variants using FID, precision, and LPIPS metrics and compared it with other baseline protection methods. Even when the architecture differed significantly (i.e., HD-Painter) or when fine-tuning changed the model parameters (i.e., DreamShaper, [4]), AdvPaint consistently outperformed earlier protection methods across all metrics. These results demonstrate the transferability of AdvPaint to different inpainting model variants, providing robust protection against potential attacks. We also provide the corresponding qualitative results in Figure 7,8, and 9 (Appendix A.3.1) for a comprehensive comparison.

We present the quantitative results of HD-Painter in the current comment and both DreamShaper and Stable Diffusion v2 inpainting model in the next comment.

[1] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., & Cohen-Or, D. (2022). An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. ArXiv, abs/2208.01618.

[2] Manukyan, H., Sargsyan, A., Atanyan, B., Wang, Z., Navasardyan, S., & Shi, H. (2023). HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models. ArXiv, abs/2312.14091.

[3] DreamShaper. (2024). DreamShaper: A model available on CivitAI. Retrieved from https://civitai.com/models/4384/dreamshaper

[4] Stability AI. (n.d.). Stable Diffusion 2 Inpainting. Retrieved November 21, 2024, from https://huggingface.co/stabilityai/stable-diffusion-2-inpainting

W2) Experimental Details

Q2) Explanation about $m^{out}$ in Section 5.6

In Section 5.6, we present the inpainting results for two types of masks: $m^{out}$ and $m^{in}$. Since masks can be hand-crafted by adversaries and may exceed the enlarged bounding box used at the optimization step of AdvPaint, we simulate this scenario by randomly shifting the segmentation mask $m^{seg}$ in four directions (up, down, left, and right) to ensure that at least one boundary of the rectangular optimization box is exceeded. This shifted mask is referred to as $m^{out}$. In contrast, $m^{in}$ is generated by randomly shifting $m^{seg}$ while ensuring the entire mask remains inside the rectangular optimization box, reflecting scenarios where segmentation masks differ but are confined to the bounded region.

Our experiments, as shown in Table 3, demonstrate that AdvPaint maintains robust performance for both $m^{out}$ and $m^{in}$. Specifically, it achieves strong performance across key metrics, including FID, precision, and LPIPS scores, effectively safeguarding images under both the scenarios.

Evaluation on Masks Exceeding or Overlapping the Optimization Boundary

To address concerns of real-world inpainting scenarios where masks vary in sizes and shapes, we conducted additional experiments in Fig.24 of Appendix A.8.3 using masks that (a) exceed or (b) overlap with the optimization boundary. Specifically, we visualized inpainting results where foreground masks were applied to regions without objects, simulating adversarial scenarios aimed at generating new objects in the background. These experiments demonstrate that AdvPaint remains robust in such diverse inpainting cases, generating inpainted images with no discernable objects. We hope this additional analysis effectively addresses the reviewer’s concerns.

Q3) Clarification for Section A.2.2 and Table 4

1. Section A.2.2

AdvPaint aims to protect images from potential malicious abuse of various inpainting tasks, where there exists a typical architecture (depicted in Figure 6 in Section A.2.2) that takes three types of input: original image, mask, and the masked image. This model also serves as the baseline for other inpainting models [1, 2, 3, 4]. Thus, to clarify, we utilize the same inpainting model described in Section A.2.2 to perform both the optimization and inpainting tasks in AdvPaint.

2. Table 4

In Table 15 (formerly Table 4 in the pre-updated version), we conduct an experiment to evaluate the impact of replacing the objective functions in our baseline models. Specifically, we use the same Latent Diffusion Model (LDM) as the baselines but substitute their objective functions—replacing Eq. (2) (e.g., AdvDM, CAAT) and Eq. (4) (e.g., Photoguard) with our proposed attention loss (Eq. 8). In other words, the only difference between the two rows of Table 15 is the targeting model (LDM (top) vs. SD Inpainting model (bottom). The objective function stays the same. Since attention blocks are also present in this default LDM, our attention loss is directly applicable. After optimizing perturbations targeting this LDM, we generate inpainted results using the same Stable Diffusion inpainting model referenced in our main paper.

The results indicate that, while trained with our proposed objective, the perturbations fail to provide effective protection against inpainting tasks, underperforming compared to AdvPaint. Furthermore, compared to rows 2–6 in Table 1 (i.e., baseline models), replacing the formal objective functions with our attention loss does not result in a significant improvement in performance.

We attribute this to the lack of direct targeting of inpainting models, which limits their ability to counter inpainting-specific attacks. This highlights a key limitation of current protection methods that rely on the default LDM for inpainting tasks and underscores the critical importance of designing defensive methods specifically tailored for such tasks.

To alleviate any confusion caused by the experimental settings in this section, we have updated the explanations in Appendix A.7 to provide clarity.

[1] Manukyan, H., Sargsyan, A., Atanyan, B., Wang, Z., Navasardyan, S., & Shi, H. (2023). HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models. ArXiv, abs/2312.14091.

[2] DreamShaper. (2024). DreamShaper: A model available on CivitAI. Retrieved from https://civitai.com/models/4384/dreamshaper

[3] Xie, S., Zhang, Z., Lin, Z., Hinz, T., & Zhang, K. (2022). SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 22428-22437.

[4] Tang, L., Ruiz, N., Chu, Q., Li, Y., Holynski, A., Jacobs, D.E., Hariharan, B., Pritch, Y., Wadhwa, N., Aberman, K., & Rubinstein, M. (2023). RealFill: Reference-Driven Generation for Authentic Image Completion. ACM Trans. Graph., 43, 135:1-135:12.

Here, we present the FID, precision, and LPIPS metrics evaluated after applying purification methods of Gaussian noise, upscaling, and JPEG compression.

Gaussian Noise, Honig et al., and JPEG Compression

We calculate the FID, precision, and LPIPS metrics evaluated after applying purification processes, including Gaussian noise addition, upscaling, and JPEG compression. Tables are demonstrated in the next comment.

We observed that both AdvPaint and the previous methods lose their ability to protect images when subjected to Gaussian noise addition, upscaling, and JPEG compression. As shown in the below tables, the FID, Precision, and LPIPS scores indicate significant degradation in protection under these conditions.

However, as depicted in the Fig.17 (a) Gaussian noise addition (c) JPEG compression, the inpainted results are noisy and blurry (e.g. noisy backgrounds for (a) “sunflower” and (c) “bicycle” images) compared to images generated from non-protected input. This raises concerns about their visual quality. It calls into question the practicality of noise-erasing methods, as the generated images often fail to meet acceptable quality standards.

Additionally, we observed a critical drawback in the existing purification methods: they tend to degrade the quality of the purified image itself. For instance, AdvPaint and baseline methods in our experiments leveraged PGD with $\eta$=0.06, ensuring adversarial examples retained a PSNR around 32 dB (Table 12 in the Appendix). On the other hand, after purification (e.g., via upscaling), we observed a PSNR drop of approximately 2.5 dB for AdvPaint, with similar reductions observed for other methods. This decline highlights a significant trade-off between the purification effectiveness and input image quality. Additional qualitative results and quantitative evaluations have been provided in Fig.17 and Table 9, 10, and 11 of Appendix A.5.

Based on these findings, we argue that while establishing the robustness against purification methods is indeed critical, the quality degradation of purified images should also be considered. Therefore, the protection capability of AdvPaint is valuable since it increases the adversary’s cost in abusing AdvPaint-perturbed images. We hope this addresses the reviewer’s concern and emphasizes the practical significance of our work.

Dear Reviewer 5BhD,

We are truly grateful for your thoughtful comments and constructive feedback on our manuscript. Below, we have addressed each of your questions and concerns in detail. Should there be any additional points or suggestions that we may have missed, we would be happy to address them. Please do not hesitate to share further feedback with us.

W1, Q1) Resistance against purification methods

We thank the reviewer for raising this concern regarding the robustness of AdvPaint against the purification methods designed to remove adversarial perturbations. To address this, we conducted experiments to evaluate the robustness of AdvPaint against the recent purification techniques, including IMPRESS [1] and Honig et al. [2]. Please note that among the four suggested methods in [2], we evaluated two methods for which official code is available—Gaussian noise addition and upscaling—while the others could not be tested due to the lack of accessible implementations. For the purification methods, we follow the Pytorch implementation for JPEG compression with quality 15 and official codes for other methods where Gaussian noise strength is set to 0.05.

IMPRESS

Our results show that AdvPaint retains its protective ability even against IMPRESS, outperforming baseline methods in terms of FID, Precision, and LPIPS. Since IMPRESS uses LPIPS loss to ensure the purified image remains visually close to the perturbed image, we believe this objective inadvertently preserves a part of the adversarial perturbation. While the protection effectiveness of AdvPaint has decreased compared to Table 1 in the main paper, the residual perturbation appears to maintain sufficient protection against malicious inpainting, as evidenced by the continued high FID values. Additional qualitative results and quantitative evaluations have been provided in Fig. 16 and Table 8 of Appendix A.5.

[1] Cao, B., Li, C., Wang, T., Jia, J., Li, B., & Chen, J. (2023). IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI. ArXiv, abs/2310.19248.

[2] Honig, R., Rando, J., Carlini, N., & Tramèr, F. (2024). Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI. ArXiv, abs/2406.12027.

Reminder for Discussions

Dear Reviewer 5BhD,

We would like to remind you that we are open to further discussions on our submission and additional experiments. Considering the deadline for uploading the revised supplementary materials is tomorrow, we ask if you have any additional questions or concerns that we should address.

Thank you for your time and effort in reviewing our work.

Experimental Settings and Baseline Models

Please note that experiments were conducted under the same settings as in the main paper (i.e., using the Stable Diffusion Inpainting model with a total of 100 images, 50 prompts per image where we use only a single seed for a single prompt), ensuring a fair and consistent comparison.

Additionally, we chose the baseline models—Photoguard, AdvDM, and SDST—that employ one of three existing objective functions (i.e. minimize Eq.2, maximize Eq.2, minimize Eq.4) from prior protection methods mentioned in the main paper.

Here, we present the FID, precision, and LPIPS metrics evaluated after applying purification methods of Gaussian noise, upscaling, and JPEG compression.

Gaussian Noise, Honig et al., and JPEG Compression

We calculate the FID, precision, and LPIPS metrics evaluated after applying purification processes, including Gaussian noise addition, upscaling, and JPEG compression. Tables are demonstrated in the next comment.

We observed that both AdvPaint and the previous methods lose their ability to protect images when subjected to Gaussian noise addition, upscaling, and JPEG compression. As shown in the below tables, the FID, Precision, and LPIPS scores indicate significant degradation in protection under these conditions.

However, as depicted in the Fig.17 (a) Gaussian noise addition (c) JPEG compression, the inpainted results are noisy and blurry (e.g. noisy backgrounds for (a) “sunflower” and (c) “bicycle” images) compared to images generated from non-protected input. This raises concerns about their visual quality. It calls into question the practicality of noise-erasing methods, as the generated images often fail to meet acceptable quality standards.

Additionally, we observed a critical drawback in the existing purification methods: they tend to degrade the quality of the purified image itself. For instance, AdvPaint and baseline methods in our experiments leveraged PGD with $\eta$=0.06, ensuring adversarial examples retained a PSNR around 32 dB (Table 12 in the Appendix). On the other hand, after purification (e.g., via upscaling), we observed a PSNR drop of approximately 2.5 dB for AdvPaint, with similar reductions observed for other methods. This decline highlights a significant trade-off between the purification effectiveness and input image quality. Additional qualitative results and quantitative evaluations have been provided in Fig.17 and Table 9, 10, and 11 of Appendix A.5.

Based on these findings, we argue that while establishing the robustness against purification methods is indeed critical, the quality degradation of purified images should also be considered. Therefore, the protection capability of AdvPaint is valuable since it increases the adversary’s cost in abusing AdvPaint-perturbed images. We hope this addresses the reviewer’s concern and emphasizes the practical significance of our work.

W4) Resistance Against Purification Methods

We thank the reviewer for raising this concern regarding the robustness of AdvPaint against the purification methods designed to remove adversarial perturbations. To address this, we conducted experiments to evaluate the robustness of AdvPaint against the recent purification techniques, including IMPRESS [1] and Honig et al. [2]. Please note that among the four suggested methods in [2], we evaluated two methods for which official code is available—Gaussian noise addition and upscaling—while the others could not be tested due to the lack of accessible implementations. For the purification methods, we follow the Pytorch implementation for JPEG compression with quality 15 and official codes for other methods where Gaussian noise strength is set to 0.05.

IMPRESS

Our results show that AdvPaint retains its protective ability even against IMPRESS, outperforming baseline methods in terms of FID, Precision, and LPIPS. Since IMPRESS uses LPIPS loss to ensure the purified image remains visually close to the perturbed image, we believe this objective inadvertently preserves a part of the adversarial perturbation. While the protection effectiveness of AdvPaint has decreased compared to Table 1 in the main paper, the residual perturbation appears to maintain sufficient protection against malicious inpainting, as evidenced by the continued high FID values. Additional qualitative results and quantitative evaluations have been provided in Fig. 16 and Table 8 of Appendix A.5.

[1] Cao, B., Li, C., Wang, T., Jia, J., Li, B., & Chen, J. (2023). IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI. ArXiv, abs/2310.19248.

[2] Honig, R., Rando, J., Carlini, N., & Tramèr, F. (2024). Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI. ArXiv, abs/2406.12027.

W3) Explanations of AdvPaint and Multi-Object Images

We appreciate your insightful comments. We will closely explain more about the optimization process of multi-object images as the extension of Section 5.7 and the effectiveness of AdvPaint regardless of which masks are used for the multi-object images.

For the optimization process of multi-object images, we first position ourselves as content owners and select the objects that may be at risk of malicious inpainting modifications. Then, AdvPaint performs PGD optimization for each object using enlarged bounding box masks generated by Grounded SAM. After optimizing for each object, the leftover background regions, where objects potentially at risk do not exist, are also optimized. We have updated Fig. 20 in the Appendix to clarify the masks used to optimize multi-object images, aiding comprehension.

After securing each object, we conducted experiments with a variety of mask types, including single-object masks, masks for other objects, combined-object masks, and their inverted versions. Figure 21 demonstrates the robustness of AdvPaint for multi-object images. For example, since AdvPaint optimizes each object individually, it ensures protection for each object, resulting in inpainted images that lack discernable objects in the foreground. Furthermore, AdvPaint is robust to masks that encompass all objects, as shown by the absence of "two cameras" replacing "two dogs". Additionally, the method effectively secures background regions when inverted masks are used for inpainting tasks. These results substantiate the effectiveness of AdvPaint’s per-object protection method, even for complex multi-object scenarios.

We have updated the explanations of multi-object images in Appendix A.8.1. We hope this clarifies our approach and demonstrates AdvPaint’s real-world applicability to multi-object image scenarios.

Evaluation on Masks Exceeding or Overlapping the Optimization Boundary

In the real-world scenario, the purpose of inpainting may differ. To address these concerns, we conducted additional experiments in Fig.24 of Appendix A.8.3 using masks that (a) exceed or (b) overlap with the optimization boundary. Specifically, we visualized inpainting results where foreground masks were applied to regions without objects, simulating adversarial scenarios aimed at generating new objects in the background. These experiments demonstrate that AdvPaint remains robust in such diverse inpainting cases, generating inpainted images with no discernable objects. We hope this additional analysis effectively addresses the reviewer’s concerns.

W2) Prompts and masks from different models other than ChatGPT and Grounded SAM

We appreciate the reviewer’s insightful feedback on the preprocessing steps used in our method. To address these concerns, we conducted additional experiments employing alternative resources for prompt generation and mask creation to evaluate the robustness and generalizability of AdvPaint’s protection performance.

For prompt generation, in addition to ChatGPT, we utilized Claude 3.5 Sonnet to generate diverse prompts. Specifically, for foreground inpainting, we requested prompts consisting of {noun} phrases (e.g., “a jellyfish,” “a mushroom”), while for background inpainting, we generated prompts with {preposition, location} phrases (e.g., “at a cherry blossom grove,” “at a volcanic crater”) combined with a brief description of the input image (e.g., “A deer at a cherry blossom grove,” “A wolf at a volcanic crater”). This prompt structure is consistent with the main paper.

For mask generation, we replaced Grounded SAM with the zero-shot segmentation method proposed by Yu et al. [1], which employs CLIP [2] to create object masks based on the given prompt. Using the same input images as in the main paper, we ran [1] to generate segmentation masks, selecting one mask for foreground inpainting and using its inverted counterpart for background inpainting. These newly generated prompts and masks were then used with the same Stable Diffusion inpainting model as in the main paper.

In Fig. 22 of Appendix A.8.2, we added qualitative results using this alternative setup. These experiments demonstrate that AdvPaint retains its protection performance for inpainting tasks, comparable to its performance when using ChatGPT and Grounded SAM. Additionally, we observed that while the exact outputs might vary due to differences in prompt wording or mask generation, the relative protective effectiveness of AdvPaint against earlier protection methods remains consistent. This highlights the robustness of AdvPaint to diverse inpainting scenarios, even when the preprocessing steps involve varying methods.

Note that, as updated in Fig. 23, we observed that the segmentation results from [1] were generally less accurate compared to those generated by Grounded SAM. Masks often fail to fully cover the object, leading to incorrect inpainting results. We report that Grounded SAM may be more suitable for inpainting tasks, given its demonstrated effectiveness and its adoption in related works such as SDST [3] for generating masks in similar contexts. This reinforces our choice of Grounded SAM as the primary segmentation tool in the main paper, while also validating AdvPaint’s adaptability to alternative segmentation approaches.

[1] Yu, S., Seo, P.H., & Son, J. (2023). Zero-shot Referring Image Segmentation with Global-Local Context Features. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19456-19465.

[2] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. International Conference on Machine Learning.

[3] Xue, H., Liang, C., Wu, X., & Chen, Y. (2023). Toward effective protection against diffusion based mimicry through score distillation. ArXiv, abs/2311.12832.

Dear Reviewer BUBL,

We sincerely appreciate your valuable comments and insightful suggestions regarding our manuscript. Below, we provide detailed responses to each of your questions and concerns. If there are any additional points or feedback that we may have overlooked, please feel free to let us know.

W1) Technical Comparison with CAAT

We appreciate this insightful feedback and clarify the technical differences between AdvPaint and CAAT [1].

CAAT first optimizes its perturbation using PGD by maximizing the objective function of Eq.2 in the main paper, which does not directly target the attention mechanism of a target LDM. Then, within the same iteration step, it fine-tines the parameters of key and value of the cross-attention layer of the denoiser. The fundamental motivation behind fine-tuning cross-attention parameters is to make the diffusion model more robust, thereby making the perturbation harder to attack and enabling the generation of a more effective perturbation in the subsequent iteration steps.

However, we question whether fine-tuning the parameters of the denoiser U-Net genuinely disrupts the mapping of text and image features. While CAAT does not explore this, AdvDM [2], which utilizes the same objective function of maximizing Eq.2 but omits the fine-tuning process, shows visually similar results. Specifically, in Figures 10, 11, and 18 of the Appendix, the outcomes of image-to-image, text-to-image, and inpainting tasks for both CAAT and AdvDM exhibit similar trends, often manifesting as noisy and plaid patterns. We believe these similar patterns are primarily driven by the shared loss function, and the fine-tuning process in CAAT has a negligible impact on disrupting the attention mechanism.

In contrast, AdvPaint directly targets both the cross and self-attention layers, impairing their representation capabilities without requiring an additional fine-tuning process. Moreover, AdvPaint introduces a two-stage optimization process (Section 4.2), applying different perturbations to the target mask regions (i.e., foreground) and the remaining regions (i.e., background). Thus, this approach, specifically designed for undermining adversarial inpainting tasks, effectively disrupts the adversary’s tasks more reliably than the previous methods, including CAAT and AdvDM.

[1] Xu, J., Lu, Y., Li, Y., Lu, S., Wang, D., & Wei, X. (2024). Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 24534-24543.

[2] Liang, C., Wu, X., Hua, Y., Zhang, J., Xue, Y., Song, T., Xue, Z., Ma, R., & Guan, H. (2023). Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples. International Conference on Machine Learning.

W2, Q3) FG Inpainting: prompts describing the object itself

We conducted additional experiments using prompts that specify the object in each given scene. For example, we used the prompt “A man” for an input image describing a male and performed an inpainting task to generate another male image.

In all cases, AdvPaint successfully disrupted the adversary's inpainting task, resulting in the generation of an image with no discernible object. The following table describes the foreground inpainting performance of AdvPaint for this set of prompts, outperforming baseline methods in terms of FID, Precision, and LPIPS. This is because the perturbation optimized to disrupt the attention mechanism successfully redirects the attention to other unmasked areas as explained in Section 5.2 and Figure 3.

We included the qualitative examples and experimental results in Appendix A.4.1 (Figure 14) of the revised supplementary material.

W2, Q4) BG Inpainting: prompts without nouns

We experimented with prompts where the noun describing the object is omitted and evaluated their effectiveness in undermining the adversary’s background inpainting task. Specifically, we assumed the adversary might adjust the prompt to exclude the object (e.g., in Figure 4, using "rocky slope" instead of "A monkey on a rocky slope") to mitigate artifacts. In all cases, AdvPaint outperformed all baselines, as demonstrated in the table below and updated in Appendix A.4.2.

Additionally, in the qualitative results included in Fig. 15 of Appendix A.4.2, we observed a consistent similarity across prompts with and without nouns: in both cases for AdvPaint, the inpainting models fail to recognize the object in the unmasked region. When using prompts that contain nouns (originally used prompts), the models generate new objects in the masked area based on the noun in the prompt (e.g., in (b) and (d) of Fig. 1), disregarding the presence of the object in the unmasked region. Similarly, when prompts are provided without nouns (Fig. 15), the inpainting models fail to recognize the object, blending it into the unmasked region and often generating background components based on the prompt description (e.g., adding a "horse" to the background for the prompt "in the countryside").

We believe this similarity arises from AdvPaint's targeted disruption of the self- and cross-attention mechanisms in inpainting models, which impairs their ability to maintain semantic relationships and align the generated content with the condition prompt, regardless of whether the prompt includes a noun describing the object. These findings highlight AdvPaint’s robustness in protecting images, as it consistently prevents recognition of objects in the unmasked region, even when the attacker adjusts the prompt to exclude the object.

Q1) Runtime and Memory Requirements

1. Memory Requirements

The memory usage of AdvPaint primarily depends on the size of an input image, the number of optimization stages, and the number of time-steps in the diffusion models. At the first optimization iteration step, we observe that the minimum memory storage needed is approximately 20GB VRAM, which mainly comes from the Stable Diffusion inpainting model. All the optimization and experiments for AdvPaint were conducted on an NVIDIA GeForce RTX 3090 (24GB VRAM) using a 512x512 image size, with optimization performed for 250 steps per mask at the initial time step T of the sampling process.

2. Runtime

For each applied mask, AdvPaint performs PGD for 250 steps, which takes approximately 2 minutes. With a reasonable number of masks applied to each target image, AdvPaint delivers an efficient runtime, averaging 2 minutes per mask (Min: 110 seconds, Max: 128 seconds).

Q2) Condition Prompt for Crafting Perturbation

We would like to refer the reviewer to Section A.1.4 in our supplementary material. The condition prompts were designed to indicate an object in the scene (e.g., "a woman," "a man," "a dog"). This conditioning format was designed to simplify the optimization process, making it easier for the image owners to protect their images easily using a straightforward prompt.

This approach aligns with the settings used by our baseline models: AdvDM and Mist employ prompts such as "A photo" or "A painting," while CAAT utilizes prompts in the format "A photo of [noun]," where the [noun] specifies the image content (e.g., "person" for human images).

Experimental Settings and Baseline Models

Please note that experiments were conducted under the same settings as in the main paper (i.e., using the Stable Diffusion Inpainting model with a total of 100 images, 50 prompts per image where we use only a single seed for a single prompt), ensuring a fair and consistent comparison.

Additionally, we chose the baseline models—Photoguard, AdvDM, and SDST—that employ one of three existing objective functions (i.e. minimize Eq.2, maximize Eq.2, minimize Eq.4) from prior protection methods mentioned in the main paper.

Dear Reviewer e2Ae,

We sincerely appreciate your thoughtful comments and suggestions in reviewing our manuscript. We address each of your questions and concerns individually below. Please let us know if you have any comments/concerns that we have not addressed.

W1) Application to Diffusion Transformers

As the reviewer suggested, we evaluated the robustness of AdvPaint against an adversary using the inpainting model of Flux[1] and text-to-image model Pixart-$\delta$[2] where both methods leverage diffusion transformers. Unlike [2], we note that models like DiT[3] and Pixart-$\alpha$[4] are designed for generating images solely from text prompts using diffusion transformer architectures, which make them unsuitable for our tasks that require accepting input images.

Flux Inpainting Model

Flux provides an inpainting module based on multimodal and parallel diffusion transformer blocks. We utilized the “black-forest-labs/FLUX.1-schnell” checkpoint, and the image size was set to 512x512 to match our settings.

As shown in Figure 12 in Appendix A.3.3, AdvPaint effectively disrupts the inpainting process by causing misalignment between generated regions and unmasked areas. For example, it generates cartoon-style cows in (a) and adds a new rabbit in (b), while also producing noisy patterns in the unmasked areas of the images.

DreamBooth with DiT

Chen et al. introduced Pixart-$\delta$, which integrates DreamBooth [5] into DiT. We selected this model as the adversary’s generative framework since it supports feeding an input image along with a command prompt for performing generation. Additionally, as discussed in Section A.3.2, we demonstrate the transferability of AdvPaint to text-to-image tasks.

As shown in Fig. 13 (a) in Appendix A.3.3, AdvPaint-generated perturbations consistently undermine the generation ability of Pixart-$\delta$. Furthermore, AdvPaint also renders noise patterns that degrade the image quality on the resulting output images of the diffusion model, which aligns with the behavior of previous methods (i.e. Photoguard, AdvDM, SDST).

Protection for DiT-based Models

AdvPaint also effectively disrupts the original DreamBooth[5], as depicted in Fig. 13 (b). However, our findings indicate that AdvPaint and the previous methods are less effective against Pixart-$\delta$ that leverages DiT (Fig. 13 (a)). Additionally, compared to the results of LDM-based inpainting models in Figure 1, current methods are less effective when applied to DiTs. For example, in AdvPaint, discernible objects appear in the foreground inpainting tasks and new objects are not always generated in the background inpainting tasks. We believe this ineffectiveness stems from the distinct characteristic of DiT, which processes patchified latent representations. AdvPaint and our baselines are specifically designed to target LDMs, which utilize the entire latent representation as input, allowing perturbations to be optimized over the complete latent space. Thus, when latents are patchified in DiTs, perturbations may become less effective at disrupting the model's processing, thereby diminishing their protective capability. This discrepancy necessitates further research to develop protection methods specifically tailored to safeguard images against the adversary misusing DiT-based models. For instance, optimizing perturbations at the patch level rather than across the entire latent representation could prove more effective in countering the unique paradigm of image generation in DiT-based models.

[1] Black Forest Labs. (n.d.). Home. Retrieved November 21, 2024, from https://blackforestlabs.ai/

[2] Chen, J., Wu, Y., Luo, S., Xie, E., Paul, S., Luo, P., Zhao, H., & Li, Z. (2024). PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models. ArXiv, abs/2401.05252.

[3] Peebles, W.S., & Xie, S. (2022). Scalable Diffusion Models with Transformers. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 4172-4182.

[4] Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J.T., Luo, P., Lu, H., & Li, Z. (2023). PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. ArXiv, abs/2310.00426.

[5] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022). DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 22500-22510.