Generalizability Across T2I Model Variants

We evaluated our method on other widely used T2I models, SD 1.5 and SD 2.1, to assess the generalizability of our method. Due to time constraints, we used the UCE unlearned model with "nudity" removed, keeping $\sigma_0$ and $k$ consistent with version 1.4. The table below shows results using four attack methods: I2P, P4D, RAB, and QF-PGD, with average attack success rate (ASR) for robustness and FID and CLIP scores for utility.

Our findings indicate that our method remains effective across different T2I model variants, enhancing the adversarial robustness of unlearned models without significantly affecting their generation capabilities. We will expand the analysis to include additional baselines in our paper.

Discussion on Failure Cases

Thank you for your insightful feedback. We have provided some qualitative results here. These results show that with the default hyper-parameter value of $\sigma_0$ (as seen in the second and fifth columns), certain adversarial prompts can still restore the nudity concept. In our implementation, $\sigma_0$ is fixed for each unlearned model, but there are challenging cases that require a larger $\sigma_0$ to address effectively. While increasing $\sigma_0$ can help defend against these prompts, it may also degrade model performance with benign inputs. Therefore, it is worthwhile to explore how to dynamically adjust $\sigma_0$ based on the input prompt in future work. We will include this discussion in our paper.

Multiple Concept Unlearning

Thank you for the insightful comment. We conducted a preliminary exploration on the UCE unlearned model with nudity and violence concepts erased simultaneously. For this multi-concept setting, we computed each token's relevance score by taking the maximum of its similarity with both the nudity concept direction and the violence concept direction. The results are presented in the table below, where for each concept, the average attack success rate (ASR) is computed across three attack methods (I2P, RAB, and QF-PGD).

Our method is effective in concurrently handling multiple concepts. Compared to the single-concept case, the additional computation involves calculating similarities with more concepts and performing a subsequent max operation. However, when the number of concepts is large, the efficiency of our method may be affected. We will include this analysis in our paper.

Constraints on Adversarial Perturbation

The perturbation is constrained within a norm-bounded set $\\mathcal{B}$, specifically $\||\delta\||\_2<\rho$ in the isotropic case (Lemma 3.1) and $\||\delta\||_{\Sigma,2}<\rho\prime$ in the anisotropic case (Theorem 3.2). If the upper bound of the perturbation norm is large, users can increase noise intensity in median smoothing. A detailed discussion on handling large perturbations is available in the Theoretical Tightness under Large Perturbation section of our response to Reviewer CcZX.

Justification for Considering Intermediate Steps in Formulation

Our formulation follows the DDPM [1] framework, modeling generation as Markov chains. While we aim to constrain $x_0$, the dependency structure ($x_0$ depends on $x_1$, $x_1$ on $x_2$, and so forth) means perturbations propagate through the entire sequence. The formulation ensures robustness at each step, preventing cascading errors and ultimately safeguarding the final output. Notably, related works, such as [2], also incorporate intermediate steps in their formulations.

In practical applications, efficient sampling techniques enable us to perform smoothing over a limited number of time steps, often just tens or even fewer, thus maintaining computational efficiency.

[1] Denoising Diffusion Probabilistic Models, NeurIPS 2020.

[2] Ablating Concepts in Text-to-Image Diffusion Models, ICCV 2023.

Collection of Prompt Pairs

Positive and negative prompts are sourced from the ViSU dataset, as explained in the Implementation Details (Section 4.1), requiring no additional computation. We also explored generating prompt pairs using a large language model (LLM) interface. More details are in the Clarifying the Use of Prompt Pairs from ViSU section of our response to Reviewer 9A3r.

Additional Comments

We will shorten the abstract and number all the equations in our paper. Thanks for your advice.

Thank you for your insightful comment regarding the relationship between our method and AdvUnlearn [1]. Our response is elaborated on the following three aspects:

Methodological Differences

AdvUnlearn [1] is a pre-inference method that fine-tunes the text encoder. It falls into the category of adversarial erasing-based defenses, similar to the RACE and RECE baselines evaluated in our manuscript. While AdvUnlearn operates in the text-embedding space as our method does, several key differences exist:

• Efficiency: AdvUnlearn requires relatively large computational resources for the fine-tuning process, as acknowledged by the authors [1]. In contrast, our approach is training-free and operates at inference time, offering greater efficiency.
• Theoretical Guarantees: Unlike AdvUnlearn’s empirical defense, we provide theoretical guarantees through generalized median smoothing (Theorem 3.2), which could potentially offer new insights to this field.

Comparative Performance Evaluation

We implemented AdvUnlearn using its official codebase to compare it with our method, targeting the concept of "nudity". For our approach, we integrated our method with ESD, setting $\sigma_0$ to 0.012 and $k$ to 9. We evaluated adversarial robustness through four attacks (i.e., I2P, RAB, MMA, and QF-PGD) and calculated the average attack success rate (ASR), while assessing model utility through FID and CLIP scores. The results are shown in the table below:

The results indicate that our method achieves adversarial robustness comparable to AdvUnlearn. In terms of model utility, AdvUnlearn maintains superior FID metrics (image quality) because it preserves the U-Net weights. However, its CLIP score (image-text alignment) is lower due to modifications in the text encoder. Our defense employs adaptive median smoothing without altering text encoder parameters, better preserving image-text alignment capabilities.

Compatibility and Future Work

We found that directly applying our method to an unlearned text encoder presents challenges. Our approach requires calculating each token's relevance to unsafe concepts, but AdvUnlearn's fine-tuning process maps unsafe token representations to benign ones, making it difficult to distinguish between them based on textual representations. This results in inaccurate relevance scores.

Currently, our proposed method serves as an effective complement to unlearning approaches that modify U-Net parameters, which constitute the majority of diffusion model unlearning techniques. In future work, we plan to explore adaptations to our method to enhance compatibility with unlearned text encoders.

We will incorporate a citation to AdvUnlearn and include the above analysis in our paper.

[1] Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models, NeurIPS 2024.

Evaluation Across Object and Style Concepts

We would like to clarify that our original manuscript includes experiments not only on the concept of nudity but also on violence. Following your suggestion, we expanded to include object and style concepts.

• For the object concept, we targeted "gun" and used three attack methods (UDA, RAB, QF-PGD) to assess adversarial robustness, reporting the mean attack success rate (ASR). Due to time constraints, we compared only with SLD-Medium and SLD-Max baselines. The results, presented in the table below, show our method achieves a superior balance between robustness and utility. | Unlearned Model | Defense | ASR $\downarrow$ | FID $\downarrow$ | CLIP $\uparrow$ | | --------------- | ----------- | ---------------- | ---------------- | --------------- | | ESD | w/o defense | 56.11 | 6.595 | 30.06 | | ESD | SLD-Medium | 28.44 | 9.624 | 29.45 | | ESD | SLD-Max | 18.11 | 16.630 | 28.26 | | ESD | Ours | 26.07 | 7.959 | 29.66 | | UCE | w/o defense | 58.56 | 5.281 | 31.03 | | UCE | SLD-Medium | 40.11 | 8.201 | 30.65 | | UCE | SLD-Max | 21.00 | 14.223 | 29.25 | | UCE | Ours | 29.85 | 7.142 | 30.29 |

• For the style concept, we targeted "Van Gogh" using three attack methods (UDA, RAB, QF-PGD), with results shown in the table below. These results demonstrate the method's effectiveness in addressing style concepts. | Unlearned Model | Defense | ASR $\downarrow$ | FID $\downarrow$ | CLIP $\uparrow$ | | --------------- | ----------- | ---------------- | ---------------- | --------------- | | ESD | w/o defense | 11.33 | 5.612 | 30.38 | | ESD | SLD-Medium | 1.33 | 7.929 | 30.03 | | ESD | SLD-Max | 0.00 | 15.364 | 28.70 | | ESD | Ours | 0.00 | 6.181 | 30.16 | | UCE | w/o defense | 39.83 | 2.024 | 31.11 | | UCE | SLD-Medium | 1.67 | 6.569 | 30.55 | | UCE | SLD-Max | 0.00 | 17.322 | 28.30 | | UCE | Ours | 0.00 | 3.843 | 30.94 |

We acknowledge the limited baselines for object and style concepts and will include more in our final paper.

Clarifying Median Estimation Independence Across Timesteps

We want to clarify that the conditional text embeddings at each timestep are independent of one another. For each timestep, we sample noise from Gaussian distribution and apply adaptive median smoothing to obtain the conditional text embedding. This smoothed text embedding is then input into the U-Net for the current denoising step. Although the computation of the smoothed text embedding occurs at each timestep, it can be performed in parallel, making it more efficient than computing the median U-Net noise prediction, as shown in Table 2 of our manuscript.

Theoretical Tightness under Large Perturbation

The bound in Equation (2) remains theoretically meaningful even for large adversarial perturbation, as the bound explicitly depends on the interplay between perturbation norm upper bound $\rho$ and the noise magnitude $\sigma$ used in median smoothing.

According to Lemma 3.1, the perturbed output $\mathcal{E}\_{0.5}(y+\delta)$ is bounded by $\underline{\mathcal{E}}\_{\underline{p}}(y)$ and $\overline{\mathcal{E}}_{\overline{p}}(y)$. Tighter bounds are achieved when $\underline{p}$ and $\overline{p}$ are closer to 0.5. The probabilities $\underline{p}:=\Phi\left(-\frac{\rho}{\sigma}\right)$ and $\overline{p}:=\Phi\left(\frac{\rho}{\sigma}\right)$ suggest that increasing $\sigma$ can tighten the bound when $\rho$ is large. Thus, theoretically, the bound remains effective even with large perturbations.

In practice, while $\rho$ may be large, increasing $\sigma$ helps maintain adversarial robustness. However, excessive $\sigma$ may harm the model utility, so it requires careful tuning.

Reasons Behind Choosing ESD and UCE as Base Unlearned Models

We selected ESD and UCE as the base unlearned models for the following reasons:

• Widely Recognized Baselines: ESD and UCE are frequently used in the field of unlearning for diffusion models.
• Distinct Paradigms: They represent two primary diffusion unlearning paradigms: ESD is associated with fine-tuning, while UCE pertains to model editing.
• Ensuring Fair Comparison: The adversarial erasing-based baselines, RACE and RECE, are implemented using ESD and UCE, respectively. Therefore, selecting ESD and UCE as the base unlearned models ensures a fair comparison between our method and adversarial erasing-based approaches.

Discussion on AdvUnlearn

We will cite and discuss AdvUnlearn [1] in our paper. The detailed discussion is provided in our response to Reviewer ZpJv.

[1] Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models, NeurIPS 2024.

Notation Section

Thanks for your advice. We will include a notation section in our paper.

Analysis of Vague Adversarial Prompts

Thanks for your insightful comment. Our method’s robustness against vague adversarial prompts is validated through evaluations on the MMA attack [1], which constructs adversarial prompts avoiding sensitive words while inducing unsafe generations. As shown in Table 1 of our manuscript, our method reduces the attack success rate (ASR) for MMA from $7.50\\%$ to $4.23\\%$ (ESD model) and from $39.30\\%$ to $17.53\\%$ (UCE model), demonstrating effectiveness against such vague adversarial prompts.

[1] MMA-Diffusion: MultiModal Attack on Diffusion Models, CVPR 2024.

Ablation Study of Hyper-Parameters

In response, we have conducted an ablation study on hyper-parameters using the other unlearned model, UCE. The updated results, provided here, indicate that moderately increasing $\sigma_0$ and $k$ enhances the adversarial robustness of unlearned models without significantly compromising their utility.

Discussion on Failure Cases

We have provided qualitative results here. These results reveal that the framework occasionally fails on "hard cases" where adversarial prompts bypass default hyper-parameters ($\sigma_0$), allowing unsafe concept restoration (columns 2 & 5). While increasing $\sigma_0$ mitigates such cases, it risks degrading benign-generation quality. This highlights a trade-off between robustness and utility under fixed hyper-parameters. Future work will explore dynamic $\sigma_0$-adjustment based on input prompt to better balance these objectives.

Clarifying the Use of Prompt Pairs from ViSU

Thank you for your valuable feedback. Here is our response:

• Details on Prompts: As reported in [2], the authors fine-tuned a large language model (LLM), Llama 2, to generate unsafe sentences from safe ones collected from COCO captions. The training set contains 7,863 prompt pairs for the nudity concept.
• ViSU for Baseline Methods: Most defense baselines in our manuscript do not require prompt pairs. Specifically, pre-inference defenses (RACE and RECE) optimize adversarial prompts and subsequently erase them. For training-free defenses, SLD uses inappropriate keywords to calculate unsafe guidance, while Blacklist uses NSFW words for filtering.
• Ablation Without ViSU: To further demonstrate the effectiveness of our proposed method, we implemented "nudity" unlearning without using prompt pairs from ViSU. Specifically, we first used an uncensored LLM interface on Hugging Face to generate 100 nudity-related prompts (positive prompts) and then used another LLM interface (e.g., Kimi) to transform them into safe versions (i.e., negative prompts). We then used these 100 prompt pairs (denoted as Self-Gen) in our experiments, with results shown in the table below (a detailed table with attack success rates for each attack is provided here). Overall, our method remains effective even with just 100 collected prompt pairs. In the detailed table, we observe that adversarial robustness against vague adversarial prompts (i.e., MMA) degrades, suggesting that collecting more prompt pairs further enhances defense against vague adversarial prompts. | Unlearned Model | Defense Methods | ASR $\downarrow$ | FID $\downarrow$ | CLIP $\uparrow$ | | --------------- | --------------- | ---------------- | ---------------- | --------------- | | ESD | w/o defense | 25.42 | 7.161 | 30.18 | | ESD | Ours (ViSU) | 6.80 | 7.365 | 30.10 | | ESD | Ours (Self-Gen) | 5.33 | 7.436 | 30.09 | | UCE | w/o defense | 40.34 | 4.379 | 31.02 | | UCE | Ours (ViSU) | 11.02 | 5.343 | 30.48 | | UCE | Ours (Self-Gen) | 15.93 | 5.526 | 30.54 |
• Generalization to Other Concepts: ViSU primarily contains unsafe prompts (e.g., nudity and violence) and is not suitable for other types of concepts (e.g., objects and styles). For these concept types, we first used an LLM interface to generate 100 positive prompts depicting the target concept, and then instructed the LLM to remove target concept-related elements to create negative prompts. The experimental results for these additional concept types are provided in the Evaluation Across Object and Style Concepts section in our response to Reviewer CcZX.

[2] Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models, ECCV 2024.

Typos & Grammar Errors

Thank you for your careful review. We have corrected the identified typos/grammar errors and will perform a thorough proofread of the entire manuscript to ensure clarity and correctness.