CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models

We sincerely thank the reviewer for the valuable comments (Q). We hope that our responses (A) have fully addressed the concerns, and remain committed to clarifying any further questions that may arise during the discussion period.

Q1: It is good to try training-free DM customization methods (e.g., IP-Adapter).

A1: We appreciate this insightful suggestion. Indeed, training-free customization methods such as IP-Adapter represent a promising direction for low-cost and scalable diffusion model customization. While our current study focuses on training-based customization approaches (e.g., DreamBooth and LoRA) due to their widespread usage and controllability in evaluating robustness, extending CAT to training-free settings is an exciting future direction. We plan to incorporate IP-Adapter into future evaluations to further evaluate the generalization of CAT beyond current training-based frameworks.

Table 3. Quantitative results for object-driven image synthesis using CAT methods customized in DreamBooth for the CelebA-HQ dataset compared to the Noisy-Upscaling (NU) and Gaussian Filtering (GF) methods.

Table 4. Quantitative results for object-driven image synthesis using CAT methods customized in DreamBooth for the VGGFace2 dataset compared to the Noisy-Upscaling (NU) and Gaussian Filtering (GF) methods.

We sincerely thank the reviewer for the valuable comments (Q). We hope that our responses (A) have fully addressed the concerns, and remain committed to clarifying any further questions that may arise during the discussion period.

Q1: Yes, but could be ... using the fidelity metrics, such as FID.

A1: We thank the reviewer for pointing this out. In the evaluation of Table. 1 below, we use the FID metric to assess generation quality. Specifically, for each identity, we generate 30 images per prompt (two prompts in total), and compute FID by comparing the results to images generated by a customization model trained on clean samples. We observe that both CAT-both and CAT-en consistently achieve lower FID scores than the baseline across both datasets and all evaluated protections. These results demonstrate the effectiveness of our CAT under the evaluated settings.

Table 1. Quantitative results for object-driven image synthesis using CAT methods customized in DreamBooth for CelebA-HQ and VGGFace2 datasets compared to the Noisy-Upscaling (NU) and Gaussian Filtering (GF) methods.

Q2: The core rationale ... proposed approach unconvincing.

A2: We sincerely thank the reviewer for the valuable suggestion. To the best of our knowledge, this is the first work that systematically evaluates the robustness of nine existing protective perturbation methods across two downstream tasks, which requires careful data preparation and repeated experiments. That said, we totally agree that for the style mimicry task, only qualitative results were provided in the main paper due to space limitations. To address this, we include quantitative results in Table 2 below in CLIP-IQA score [r2] for both CAT-both and CAT-en settings. We observe that in the style mimicry task, CAT consistently outperforms the baseline across all evaluated protection methods, further demonstrating the effectiveness of our approach.

Table 2. Quantitative results for style mimicry using CAT methods customized in DreamBooth for the WikiArt dataset.

Q3: I am curious about the ... and helpful to the future reader.

A3: Thank you for the insightful comment. We fully agree that while adaptive attacks reveal current vulnerabilities in protective perturbations, their broader value lies in informing the development of more effective defenses. In this regard, we would like to emphasize that our CAT method not only exposes the limitations of existing protections but also offers potential insights for future defense strategies. In particular, our findings suggest that effective protection for LDMs may benefit from incorporating defense mechanisms that explicitly consider the diffusion process, especially in end-to-end optimized frameworks, rather than relying solely on latent autoencoders, which can be more easily compromised by adaptive attacks. We believe this perspective highlights a promising direction for designing more robust protective perturbations to better safeguard the IP rights of data owners. A more detailed discussion will be added in the revised version.

[r2] Wang, J., Chan, K. C., & Loy, C. C. (2023, June). Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 2, pp. 2555-2563).

We sincerely thank the reviewer for the valuable comments (Q). We will correct the identified typos and incorporate the suggested references you mentioned. We hope that our responses (A) have fully addressed the concerns, and remain committed to clarifying any further questions that may arise during the discussion period.

Q1: The claim ... other potential factors.

A1: We would like to clarify that the observed strong correlation refers to the effectiveness of adversarial noise as protective perturbations and the distortion in their latent representations. This conclusion is supported by the following observations:

1. Latent representation distortion: As shown in Fig. 3, adversarial noise (red dots) causes significantly more distortion in the latent space compared to random perturbations (yellow dots) under the same budget. After applying CAT, the adversarial samples (green dots) are noticeably re-aligned with the clean samples (blue dots). This observation is quantitatively verified in Fig. 4.
2. Learnability of perturbed latent representations: Fig. 5 shows that adversarially perturbed latent representations can still be effectively learned by the diffusion model, with learnability comparable to clean and randomly perturbed samples.
3. Degradation on generation quality: Fig. 6 and Table 1 in the manuscript present results when fine-tuning on adversarially perturbed data. Without CAT (baseline), generation quality drops significantly, while applying CAT improves quality to a large extent.

Together, these obervations and experimental results support our conclusion that the unlearnability of adversarially perturbed samples primarily comes from their latent representation distortion. That said, we fully agree with the reviewer that, even in the presence of a strong correlation, causality should be claimed with caution, as other factors may also play a role. We thank the reviewer for this important insight, and will emphasize that this conclusion is mainly based on empirical experimental observations and there are other potential factors.

Q2: Lack of baseline ... methods.

A2: We have compared our proposed CAT with baseline adaptive attacks against protective perturbations: Noisy-Upscaling [r1] (optimization-based) and Gaussian Filtering (low-pass filtering-based). IMPRESS++ is not open-sourced yet, so we instead adopt Noisy-Upscaling, the superior adaptive attack from the same paper. Due to space limitations, experimental details are provided in our response to Reviewer GQca, with results on the CelebA-HQ dataset shown in Table 3, and on the VGGFace2 dataset in Table 4 (both in response to Reviewer jvVV). We apologize for any inconvenience this may cause and appreciate your understanding. We can observe that our proposed CAT (CAT-both and CAT-en) consistently achieves comparable or superior performance to both Noisy-Upscaling and Gaussian Filtering across all protective perturbations, in terms of both FQS and FSS. These results highlight the competitive effectiveness of CAT compared to existing purification-based methods.

Q3: The evaluation ... generated images.

A3: We appreciate the reviewer for bringing this to our attention. We will report the results for Retina-FDR and ISM separately, as well as those for TOPIQ-FDR and FIQ, in the supplementary materials. Due to space limitations, we present the traditional evaluation metric FID to assess the quality of generated images in Table 1 (in response to Reviewer 6VvR). It can be observed that our proposed CAT (either CAT-both or CAT-en) consistently improves generation quality in terms of FID across both datasets and all evaluated protection methods. These results demonstrate the effectiveness of our approach.

Q4: According to Table 1 ... behind this?

A4: The potential reason behind this that CAT-de only adds adapters to the VAE decoder, which has limited impact on realigning latent representations. Fine-tuning only the decoder is more challenging, as the diffusion model learns from distorted latents that are highly diverse, making accurate reconstruction difficult. The weaker performance of CAT-de compared to CAT-en and CAT-both further supports our observation that latent distortion is the key factor behind the effectiveness of adversarial noise as a protective perturbation.

Q5: Since “CAT-both” ... use in practice?

A5: In our experiments, we keep the parameter size the same for CAT-both and CAT-en. Specifically, CAT-both uses half the adapter rank of CAT-en, as it adds adapters to both the encoder and decoder. While their performance varies across tasks, datasets, and metrics, the overall results are comparable. That said, we speculate that CAT-en may perform better on tasks requiring more specific or dense latent representations, such as the style mimicry task in Table 2 (in response to Reviewer 6VvR), as it enables stronger latent alignment during customization.

We sincerely thank the reviewer for the valuable comments (Q). We will incorporate the suggested references you mentioned. We hope that our responses (A) have fully addressed the concerns, and remain committed to clarifying any further questions that may arise during the discussion period.

Q1: I think the authors ... various low-pass filters.

A1: We have compared our proposed CAT with two image augmentation-based adaptive attacks towards the protective perturbation: Noisy-Upscaling [r1] (based on optimization) and Gaussian filtering (based on low-pass filters). The evaluation is conducted on both the CelebA-HQ and VGGFace2 datasets using CAT-both and CAT-en, following the same experimental setting. For Noisy-Upscaling, we adopt default configurations provided in the original paper, and for Gaussian Filtering, we use a Gaussian kernel with size equal to 5 and $\sigma = 1.0$. Due to space limitations, we present the results on the CelebA-HQ in Table 3 and on the VGGFace2 dataset in Table 4 (both in response to Reviewer jvVV). We apologize for any inconvenience this may cause and appreciate your understanding. We can observe that our proposed CAT (CAT-both or CAT-en) consistently achieves comparable or superior performance to both Noisy-Upscaling and Gaussian Filtering across all protective perturbations, in terms of both FQS and FSS. These results highlight the competitive effectiveness of CAT compared to existing purification-based methods.

Q2: Two major drawbacks... for the specified defensive noise.

A2: We would like to clarify two key aspects of our threat model: the data owner has the clean data $x_c$ and intends to share it publicly. To prevent unauthorized customization, the owner applies a perturbation to generate the protected data $x_a$, which is then released. In this setting, the adversary only has access to the protected data $x_a$ and is unaware of the specific protection method used by the data owner. This protected data already contains the defensive noise, and no additional generation is required by the adversary. We want to demonstrate that, even without knowledge of the protection technique and with access only to the protected data, the adversary can still effectively learn using our proposed CAT. This is achieved by using our proposed contrastive adversarial loss during customization. We will include a more detailed explanation of the threat model in the revised version.

Q3: I think the AT method from RobustCLIP [r1] ... for new defensive noise.

A3: We fully agree that RobustCLIP presents an effective adversarial training (AT) framework for enhancing the robustness of CLIP-based vision encoders, particularly against perturbations that disrupt text-image semantic alignment. That said, we would like to clarify key differences between RobustCLIP and our proposed CAT:

1. Optimization objective: RobustCLIP aims to preserve semantic alignment between text and image representations. In contrast, CAT defends against protective perturbations that distort the latent representation in the autoencoder, using a contrastive adversarial loss to explicitly enforce latent space alignment.
2. Implementation framework: While it is possible to apply RobustCLIP to the VAE encoder in LDMs, this would involve fine-tuning the VAE on self-generated adversarial-clean pairs, which is the same idea aligned with CAT’s motivation. However, CAT achieves this via lightweight adapters applied during customization on protected data. These adapters are detachable at inference time, leaving the original model performance unaffected. In contrast, fine-tuning of RobustCLIP would require significantly more data and adversarial examples, and would modify the entire encoder, potentially degrading performance on the original task.

Despite these differences, we greatly appreciate the reviewer’s suggestion and will discuss it in the related work section to highlight these differences.

Q4: I suggest the authors add a small section ... of this work.

A4: We totally agree that a brief explanation of how Stable Diffusion operates as an LDM with the latent autoencoder would improve the paper’s readability for those unfamiliar with this background. We will add a dedicated section to clarify this in the related work.

Q5: What do those blue points $z_0$ mean in Fig.3? It seems that they are never explained in the paper.

A5: We apologize for the confusion caused by the incorrect labeling in the original Fig. 3. The blue points were intended to represent $z_c$， which is the latent embeddings of clean samples as correctly noted in the caption. We have updated both the figure and its caption to reflect this correction. Thank you for pointing this out.

[r1] Hönig, R., Rando, J., Carlini, N., & Tramèr, F. Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI. In The Thirteenth International Conference on Learning Representations.