StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations

We sincerely appreciate the reviewer's valuable suggestions and constructive feedback. In this response, we address the reviewer's concerns point by point.

Question 1: StyleGuard assumes a relatively narrow threat model where the attacker mimics an artist’s style using only prompt-based queries to a T2I model. However, it does not consider more advanced or adaptive adversaries, such as those employing prompt inversion techniques, personalized fine-tuning via DreamBooth or LoRA, or combining reference images with prompts for multimodal imitation.

Response: Our method explicitly considers prompt inversion (Table 4), DreamBooth (Table 1), and LoRA fine-tuning (Table 3) in our main paper and appendix. Experimental results demonstrate superior performance over baselines. While previous defenses, such as Mist and AntiDreambooth, focused primarily on DreamBooth attacks, our style loss formulation inherently provides protection against various attack modalities simultaneously.

Question 2: The approach heavily relies on a style classifier trained per artist to guide perturbation optimization.

Response: Our method does not need a style classifier trained per artist to guide perturbation optimization. Our approach uses general pre-trained feature extractors (VGG/VAE) to derive style representations from intermediate layer statistics (mean and variance), eliminating the need for artist-specific classifiers. This ensures scalability without compromising protection effectiveness.

Question 3: Lack of perceptual quality metrics

We did not report perceptual quality metrics in our paper because we use the same attack budget (8/255) compared with the baseline methods. Because we cannot provide images for visualization, to further investigate the perceptual quality under different budgets, we conduct attacks under the same budget (8/255), and then compare the visual quality results with the baselines. We compared them with the original images using two widely adopted metrics: LPIPS [1] and SSIM [2]. LPIPS measures perceptual difference (lower = better quality). SSIM evaluates structural similarity (higher = better detail preservation)

We compared against baseline approaches: Glaze, Mist, Anti-Dreambooth, SimAC, and MetaCloak. Our method achieves the best LPIPS (0.02) and SSIM (0.87), which outperforms SimAC (LPIPS: 0.03, SSIM: 0.82) and MetaCloak (LPIPS: 0.03, SSIM: 0.87).

Table 1: Visual Quality Comparison between Original and Perturbed Images.

Question 4: Need for human evaluation

Response: We evaluated our method using multiple quantitative metrics, including FID, Precision, and CMMD, to rigorously assess its effectiveness. Additionally, we conducted a user study (detailed in Figure 4 of our paper) where participants were asked: "Based on the image style and quality, which image better fits the reference samples?" The results demonstrated that, compared to baseline methods, our approach achieved the lowest success rate in style mimicry, confirming its superior protection capability.

Question 5: Potential interference with legitimate use

Response: Artists can always use unprotected versions of their own work for artwork expansion. Our method specifically targets unauthorized mimicry attempts while preserving legitimate usage rights.

Question 6: Since StyleGuard depends on a style classifier trained per artist to generate perturbations, how well does this classifier generalize across prompts, unseen styles, or different diffusion models? Is there a risk that the protection degrades under cross-model or cross-domain mimicry attempts?

Response: As mentioned earlier, our method does not rely on a specific style feature extractor, but an ensemble of pretrained image encoders, such as VGG VAE and CLIP. This design ensured that the generalizable style features are extracted. Through the style loss, we make it closer to the target style and farther away from the original style. The results showed that our method is generalizable to different models (Table 2 in our paper). Moreover, our method is also generalizable to different styles. Table 2 shows the results of using images with different styles as the target images. Note that we do not need to train a specific style classifier/extractor but use a general style extractor, such as VAE and VGG, to extract the style-related features. It is shown that we can successfully defend against the attacks by using target images from different styles.

Table 2: Evaluating the influence of the target styles

To evaluate the cross-domain mimicry attacks, we conduct experiments on the CelebA dataset, which contains faces from different identities. The results show that our method can successfully prevent identity customization attacks, as shown in Table 5 and Figure 7 in our paper.

Question 7: The paper does not include experiments on adaptive attacks.

Response: Thank you for inspiring further ideas for our experimental design. We admit that our present work did not consider the adaptive attacks because all of the current purifier-based attacks use pretrained diffusion models. We agree that an adaptive attacker that has defense knowledge poses more of a threat to previous methods and to ours. We will explore the feasibility of introducing adaptive attackers in experimental settings to test the limits and robustness of our method in future work.

Question 8: There is no accompanying description or discussion for Table 2, which makes it difficult to interpret its relevance and implications. Furthermore, the reference in Line 283 appears to mistakenly point to Table 3 instead.

Response: A comprehensive analysis of Table 2 has been provided in lines 283–288 of our paper. The citation in line 283 correctly refers to Table 2 (verified upon careful re-examination). Line 294 corresponds to the reference for Table 3. Tables 2 and 3 are on the same row, which may be the reason for misleading the reviewer.

Response to Q9: Thanks for your suggestion. We will revise these in our final version.

We hope our explanations address the reviewers' questions. We welcome any further questions.

We propose a simple adaptive attack method in which the attacker is aware of the purifier model used to generate the perturbation but employs a different purifier model to remove the noise. The results are summarized in the table below. The terms X2 and X4 refer to different stable-diffusion-based upscale models used to compute the upscale loss, while SinSR denotes the purifier employed by the attacker. Our findings indicate that utilizing a purifier with a different architecture can reduce the effectiveness of the protection to some extent. However, by leveraging an ensemble of purifiers when calculating the upscale loss, the overall protection performance is significantly enhanced.

To further investigate the threat of adaptive attacks, we design a more advanced adaptive attack where the adversary has access to both clean and perturbed images. Using these paired images, the adversary fine-tunes a purifier model to effectively remove protective perturbations. Specifically, we first employ X2 and X4 as shadow purifiers to generate protective perturbations on 200 images. Then, following the fine-tuning method from SinSR, we fine-tune a ResShift [1] purifier. After training, the purifier is used during the test phase to remove the protective noise, and subsequently, DreamBooth is applied to fine-tune a Stable Diffusion model. As shown in the table below, experimental results demonstrate that the fine-tuned ResShift purifier significantly degrades the effectiveness of the protection, highlighting the practical threat posed by adaptive attacks. However, such an adaptive attack relies on the attacker being able to obtain a large number of clean and perturbed corresponding samples. In the case that the attacker cannot obtain a large number of paired samples, our attack still has a strong protective effect.

[1] ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting, NIPS 2023

According to the review guideline， reviewers are subject to Responsible Reviewing initiative, including the desk rejection of his/her co-authored papers for grossly irresponsible behaviors. We hope the reviewer rV8Y could read our paper carefully instead using AI to generate content. Our paper are acknowledged by the other two reviewers.

Hi, authors. I would like to clarify that I have carefully read your paper and taken time to understand its design and contributions. If I have misunderstood any part of your work, I sincerely welcome clarification and further discussion. I believe some of our disagreement stems from differing interpretations of what constitutes an appropriate adaptive attack in your setting, so let me explain in more detail.

From my reading, the core of your defense lies in designing a new style loss and leveraging a combination of purifiers and upscalers to generate perturbations. Therefore, an adaptive attacker, in my view, should be modeled based on knowledge of this design. In Section 4.1, the paper mentions that an adversary can access images and finetune a text-to-image generator. I assume your adaptive attack is based on a similar assumption, where the attacker finetunes a purifier using original images and perturbed images to weaken the protection.

Where I disagree is the assumption that an attacker would have access to a large set of perturbed images generated by your defense. If the attacker only knows the defense structure (the use of a style loss or ensemble of purifiers) but does not have access to the actual perturbed outputs, how would they obtain the data needed to train a purifier?

Additionally, I find the attack approach of training a purifier itself somewhat confusing, as it does not appear to directly exploit the core weakness of the defense. In contrast, I believe a more meaningful adaptive attack should test whether, given core knowledge of the defense (e.g., the design of style loss), the attacker can manipulate the input image (via adding perturbation or somehow) in such a way that the defense system still outputs an unprotected image.

If I have misunderstood your design choices, I would be glad to hear your perspective.

Thanks for your further comments. We will clarify the concerns. We have designed two different adaptive attacks. In the simple adaptive attacks, the attacker only knows the ensemble purifiers used for training and uses a different upscaler in the test phase. In this case, the attacker uses a different upscaler, as shown in the table below. The terms X2 and X4 refer to different stable-diffusion-based upscale models used to compute the upscale loss, while SinSR [1] denotes the purifier employed by the attacker. Our findings indicate that utilizing a purifier with a different architecture can reduce the effectiveness of the protection to some extent. However, by leveraging an ensemble of purifiers when calculating the upscale loss, the overall protection performance is significantly enhanced.

For the style loss, we propose an adaptive attack scenario in which the attacker is assumed to have knowledge of the style encoder and the target image. The attacker optimizes the input image’s features to closely match those of the original image while simultaneously distancing them from the target image’s features. We conducted experiments under this scenario, where no purifier was applied during testing; instead, the image was re-optimized by maximizing the style loss. Results show that this adaptive attack reduces the effectiveness of our protection. However, due to the presence of the denoise loss and upscale loss components, our method still provides a certain level of defense.

We thank the reviewer for suggesting the possibility that the attacker pre-perturbs the image. We should note that in our attack scenario, the images are first posted online by the artist and then downloaded by users. A malicious user could download these images, remove the protective noise, and then perform the attack. Therefore, we did not consider the possibility that the attacker pre-perturbs the image.

In the strong adaptive attacks, we suppose the attacker can download hundreds of both clean and perturbed images online. We admit that this is a strong assumption, but it also shows that our defense will not be easily breached.

[1] SinSR: diffusion-based image super-resolution in a single step, CVPR2024

We sincerely thank the reviewers for the constructive comments. In response to Q1, we add ablation studies and detailed explanations of the metrics. In response to Q2, we compare the time cost of our method with the baseline method.

Response to Q1

We conducted ablation studies on the style loss weight $\lambda$, the attack budget ($B_{\inf}$ in Eq.8), upscale loss weight $\eta$, $K_1$, and $K_2$ to explore their influence on the attack effect. To assess the generated image qualities of the fine-tuned SD model, we adopt two additional metrics, in addition to FID and Precision, including LPIPS[1] and CMMD[2]. CMMD is an improvement metric for FID, which can better measure the image distortions. The higher the value, the greater the gap with the training dataset.

We first generate perturbations on the SD1.4 and Upscaler x4. The training dataset includes 15 images from Van Gogh. Then we upscale the images using a different upscaler x2, and fine-tune the SD1.4 and SD1.5, respectively, using DreamBooth. In all experiments, the first stage (generating protection noise) trains for 10 epochs. The second stage (Dreambooth fine-tuning) trains for 200 epochs. We also fine-tune an SD1.4 and an SD1.5 with clean images as the references. Next, we use the reference models and adversarially fine-tuned models to generate the reference image set $I_{ref}$ and the attacked image set $I_{attack}$ under the same prompt (such as "an <sks> painting of a house). Finally, we compute the above metrics using $I_{ref}$ and $I_{attack}$. The results are shown in Table 1 and Table 2.

Table 1: Ablation Study on the Style Loss Weight (Dreambooth Finetuning on SD14)

Table 2: Ablation Study on the Style Loss Weight (Dreambooth Finetuning on SD15)

Experiments show that as $\lambda$ increases, FID, LPIPS, and CMMD first increase and then decrease, and the Precision first decreases and then increases. This indicates that there is a balance between denoising loss and style loss. This could be attributed to the fact that the two different targets of the denoising loss and style loss may not always be consistent and might be partially contradictory to each other. The aim of the denoising loss is to pull the representation of the image out of the feature space of the diffusion model, and the aim of the style loss is to make the style features of the generated image closer to the target image and further from the original image.

Table 3 displays an ablation study on the upscale loss weight, $\eta$. The findings indicate that a low weight of 0.1 results in the least effective protection, evidenced by a low FID score of 135.466 and a precision of 0.625. In contrast, higher weights, such as 10, offer better protection. However, as the value of $\eta$ increases from 10 to 100, the FID and the CMMD decrease. This trade-off implies the existence of a balanced weight that maximizes performance across the metrics.

Table 3: Ablation Study on the Upscale Loss Weight

The ablation study of the attack budget is shown in Table 4. It is shown that as the budget increases, FID and CMMD gradually increase, while Precision gradually decreases. When the budget is higher than 0.063, increasing the budget does not significantly improve the protection effect. This is because the difference in the generated images is already large enough. In addition, a too high budget will introduce visible protection noise, which will damage the quality of the protected image.

Table 4: Ablation Study on the Attack Budget (Dreambooth Finetuning on SD15)

Table 5 shows the ablation study on the $K_1$ and $K_2$. We highlight the top two best values for each metric in bold. It can be observed that when $K_1$=10 and $K_2$=5, better protection effects are achieved across four metrics. The next best results are achieved when $K_1$=5, $K_2$=5, with good protection effects across three metrics. The results also show that the protection effect is significantly weakened when the difference between K1 and K2 is large (for example, when K1=5, K2=20 or K1=20, K2=5, the CMMD values are only 0.674 and 0.704). We think this may be because when the difference between K1 and K2 is large, the noise is more likely to fall into overfitting.

Table5: Ablation Study on $K_1$ and $K_2$ (Dreambooth Finetuning on SD15)

Response to Q2

Table 6 details the computational time costs associated with different loss functions, while Table 7 provides a comparison between our approach and the baseline method, SimAC. Both methods were evaluated on two RTX 3090 GPUs with a batch size of 4, using parameters K1=10 and K2=5. The tables present the average time cost for various stages, the total time per epoch, and the total attack duration for each method. Our method incurs a 36% higher time cost per epoch compared to SimAC. This is due to the need to compute three different losses during the PGD attack stage, whereas SimAC computes only two losses (feature loss and denoising loss). Despite the increased time per epoch, our method achieves a ~50% reduction in overall attack time compared to SimAC. This advantage arises because SimAC requires an additional 10 minutes for diffusion step selection before initiating the attack, which our method does not require.

Table 6: Time Cost for Different Losses

Table 7: Comparing the Time Cost with the Baseline

Response to Q3

Thank you for inspiring further ideas for our experimental design. This is a very important suggestion. While current purifier-based attacks mainly focus on using pretrained diffusion models because of the high computation cost of training a new diffusion-based purifier, we agree that an adaptive attacker that can fine-tune a purifier on protected images poses more of a threat to previous methods and to ours. We will explore the feasibility of introducing adaptive attackers in experimental settings to test the limits and robustness of our method in future work.

To explore the defense bound, we use a novel purifier, SinSR, in the attack phase, which is unseen in the training. The result is shown in Table 5 in the response to Reviewer 1. We compute the upscale loss on three different settings (X2, X4, and X2+X4), and then use three different purifiers in the attack phase, including X2, X4, and SinSR [4]. The result shows that SinSR is able to achieve a certain attack effect. However, when both X2 and X4 are used for the upscale loss, the protection performance on the SinSR slightly improved. This shows that using an ensemble of purifiers can improve the protection strength. We also conducted the ablation study on the budget and show the results in Table 4.

Table 5: Ablation Study on the Attack Budget (Dreambooth Finetuning on SD15)

Finally, thank you again for your in-depth analysis and valuable suggestions regarding our work.

Reference [1] The unreasonable effectiveness of deep features as a perceptual metric. CVPR, 2018. [2] Rethinking FID: Towards a Better Evaluation Metric for Image Generation. Arkiv2401.09603, 2024 [3] Improved Precision and Recall Metric for Assessing Generative Models. NIPS, 2019 [4] SinSR: diffusion-based image super-resolution in a single step, CVPR2024

Dear Reviewer bWmD,

We hope this message finds you well.

We have done all the experiments according to your review. As the rebuttal period deadline is approaching, we would greatly appreciate it if you could kindly respond to our rebuttal. We are looking forward to receiving any further questions or suggestions for improvement. Your support is important to our work. We thank you sincerely for reading our paper carefully, especially considering that the reviewer rV8Y used AI to generate all the comments, and most do not reflect the facts in the paper.

Best regards,

Authors of NeurIPS 13927

To further investigate the threat of adaptive attacks, we design a more advanced adaptive attack where the adversary has access to both clean and perturbed images. Using these paired images, the adversary fine-tunes a purifier model to effectively remove protective perturbations. Specifically, we first employ X2 and X4 as shadow purifiers to generate protective perturbations on 200 images. Then, following the fine-tuning method from SinSR, we fine-tune a ResShift [1] purifier. After training, the purifier is used during the test phase to remove the protective noise, and subsequently, DreamBooth is applied to fine-tune a Stable Diffusion model. As shown in the table below, experimental results demonstrate that the fine-tuned ResShift purifier significantly degrades the effectiveness of the protection, highlighting the practical threat posed by adaptive attacks. However, such an adaptive attack relies on the attacker being able to obtain a large number of clean and perturbed corresponding samples. In the case that the attacker cannot obtain a large number of paired samples, our attack still has a strong protective effect.

[1] ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting, NIPS 2023

We sincerely thank the reviewer for insightful comments. Below, we respond to the key points raised.

Response to Q1

To assess the impact of different target styles on our method, we selected ten distinct art genres, each represented by 30 images. We conducted simplified attacks while omitting the upscale loss to focus on style influence. To measure style variation, we utilized CMMD [1], an enhanced metric over FID, where a higher CMMD value indicates a greater divergence from the training dataset. The second row in the results reflects the CMMD distance of target style images compared to the training dataset. The third to the fifth rows are the FID, CMMD, and Precision scores evaluated on the generated images. Here, we computed the style loss using the average mean and variance of style features on images in the same genre. We find that a larger CMMD distance between the target style dataset and the training dataset correlates with a significant decline in generated image quality, as evidenced by higher FID and CMMD values and lower Precision scores.

Table 1: Evaluating the influence of the target styles

Inspired by this, we introduce a straightforward method for selecting the target image. This method evaluates their effectiveness using CMMD distances and FID scores. The complete process can be executed in under an hour utilizing 8 3090 GPUs.

• Step 1: Begin by selecting 10 distinct art genres from an open-source dataset such as WikiArt.
• Step 2: Calculate the CMMD distance between each image in these styles and the training dataset.
• Step 3: Identify and select the image with the highest CMMD distance in each style.
• Step 4: Perform a Simplified StyleGuard Attack
  • For each of the 10 selected images:
    • Train for 10 epochs using the selected target images while omitting the upscale loss.
    • Using the generated images with protective perturbations to fine-tune the SD model for 200 epochs using DreamBooth.
    • Record both FID (Fréchet Inception Distance), Precision, and CMMD values at the end.
• Step 5: Choose the target image with the largest CMMD with FID>200 and Precision < 0.2 as the final target image for further optimization.

Response to Q2

Thanks for your valuable suggestions. We conducted ablation studies on the style loss weight $\lambda$, the attack budget ($B_{\inf}$ in Eq.8), upscale loss weight $\eta$, $K_1$, and $K_2$ to explore their influence on the attack effect. To assess the generated image qualities of the fine-tuned SD model, we adopt two additional metrics except FID and Precision, including LPIPS[1], and CMMD[2].

We first generate perturbations on the SD1.4 and upscaler x4. Then we upscale the images using a different upscaler x2, and fine-tune the SD1.4 and SD1.5, respectively, using DreamBooth. In all experiments, the first stage (generating protection noise) trains for 10 epochs. The second stage (Dreambooth fine-tuning) trains for 200 epochs. We also fine-tune an SD1.4 and an SD1.5 model with clean paintings as the reference models. Next, we generate the reference image set $I_{ref}$ and the attacked image set $I_{attack}$ under the same prompt. Finally, we compute the above metrics using $I_{ref}$ and $I_{attack}$. The results are shown in Table 2 and Table 3.

Experiments show that as $\lambda$ increases, FID, LPIPS, and CMMD first increase and then decrease, and the Precision first decreases and then increases. This indicates that there is a balance between denosing loss and style loss. This could be attributed to the fact that the two different targets of the denosing loss and style loss may not be always consistent and might partially controdictary with each other, in which the aim of the denosing loss is to pull the representation of the image out of the feature space of the diffusion model, and the aim of style loss is to make the style features of the generated image close to the target image and away from the original image.

Table 2: Ablation Study on the Style Loss Weight (Dreambooth Finetuning on SD14)

Table 3: Ablation Study on the Style Loss Weight (Dreambooth Finetuning on SD15)

The ablation study results of the attack budget are shown in Table 4. It is shown that as the budget increases, FID, LPIPS, and CMMD gradually increase, while Precision gradually decreases. When the budget is higher than 0.063, increasing the budget does not significantly improve the protection effect. This is because the difference in the generated images is already large enough. In addition, a too high budget will introduce visible protection noise, which will damage the quality of the protected image.

Table 4: Ablation Study on the Attack Budget (Dreambooth Finetuning on SD15)

For the ablation study of upscale loss weight $\eta$, $K_1$, and $K_2$, please refer to Response to Q1 of Reviewer 2.

We also evaluate the influence of different purifiers used for the upscale loss. The result is shown in Table 5. We compute the upscale loss on three different settings (X2, X4, and X2+X4), and then use three different purifiers in the attack phase, including X2, X4, and a new purifier, SinSR [4]. The result shows that when both X2 and X4 are used for the upscale loss, the protection performance on the SinSR significantly improved. This shows that using an ensemble of purifiers can improve the protection strength.

Table 5: Evaluate the Different Purifiers for the Upscale Loss

Response to Q3

Thanks for your suggestion. To defend against transformations such as crop and rotation, we utilize Expectation over Transformation (EoT) in our method (L194-195 in our paper). We have evaluated the crop and resize transformation in our paper (Table 1). To further evaluate the robustness of our method to transformation combinations, including crop and resize, rotation, and JPEG compression, we generate perturbations using SD 1.4 and then fine-tune the SD 1.5 model on the protected images. We compare our methods with SimAC. The training epochs for both methods are set to 20. Table 6 shows the results of the different combinations of transformations. The results show that we consistently outperform the SoTA baseline on different transformations, indicating that our method can effectively resist the combination of different transformations.

Table 6: Evaluation on Diverse Transforms

Reference [1] The unreasonable effectiveness of deep features as a perceptual metric. CVPR, 2018. [2] Rethinking FID: Towards a Better Evaluation Metric for Image Generation. Arkiv2401.09603, 2024 [3] Improved Precision and Recall Metric for Assessing Generative Models. NIPS, 2019 [4] SinSR: diffusion-based image super-resolution in a single step, CVPR2024

Dear Reviewer YoSP,

We hope this message finds you well.

As the rebuttal period deadline is approaching, we would greatly appreciate it if you could kindly respond to our rebuttal. We are looking forward to receiving any further questions or suggestions for improvement. We thank you sincerely for reading our paper carefully.

Best regards,

Authors of NeurIPS 13927

Dear Reviewer YoSP,

I hope this message finds you well! I would like to take this opportunity to thank you once again for your thorough review and valuable comments on our paper. We have diligently addressed all of your concerns in our rebuttal submission.

Considering the rebuttal deadline is approaching, I would kindly like to inquire if you have had the time to review our rebuttal.

Thank you for your support and understanding. I look forward to your response!

Best wishes,

Authors of Paper 13927