When Are Concepts Erased From Diffusion Models?

We thank the reviewer for their feedback. We are glad you found our conceptual framework intuitive. We address specific concerns below:

"trajectory probing method … does not cover vulnerabilities beyond gradient-based and inpainting-based attacks” "using classifiers/conditions to guide the generation/searching process"

We appreciate this thoughtful suggestion. One of the key strengths of the noising probe is its simplicity and training-free nature, and it can outperform other evaluation methods in specific cases (see Figure 7 for example, or the NSFW study in response to Reviewer jc67).

That said, your comment inspired us to explore classifier-guided generation as a complementary approach. Following the method outlined in “Diffusion Models Beat GANs on Image Synthesis” (Dhariwal & Nichol), we implemented classifier guidance during generation. In addition to using their pre-trained classifier, we also trained a custom binary classifier for garbage truck on noised latents from Stable Diffusion 1.4 using the ImageNet-1k dataset.

While classifier guidance alone provided some benefit, combining it with the noise-based probe significantly boosted performance: even surpassing UDA on the garbage truck class.

“tension between erasure effect and side effect is well-known”

While we acknowledge that such tradeoffs have been observed, in our paper we further investigate this tension. To emphasize our findings, we report each method’s performance along three key axes: erasure effectiveness, FID score, and unrelated concept preservation.

We define a Combined Method Score, which is an average of the method’s normalized erasing performance (across the different probes), unrelated concept preservation, and FID score. Our score reveals several interesting trends: STEREO outperforms ESD-U, the method it builds upon; and RECE similarly improves on its base method, UCE. Although both GA and STEREO achieve near-perfect erasure scores, their overall performance diverges: STEREO ranks highest overall, while GA ranks lowest.

“Are the classifier/CLIP highly correlated with human perception?” “Should we introduce non-neural network based evaluations”

Thank you for the questions. Yes, classifiers are correlated with human perception and are often used for similar purposes [1,2,3,4]. Furthermore, to complement our automated evaluation metrics (e.g., CLIP and classifier accuracy), we conducted a small-scale human evaluation to assess whether erased concepts were still recognizable to non-expert observers.

We recruited human subjects to manually classify generated images for ESDX, UCE, and GA, and 3 concepts (English Springer Spaniel, Cassette Player, Van Gogh) across 5 evaluation settings:

• Standard Prompt

• Inpainting

• Noise-Based Probe

• Textual Inversion

• UnlearnDiffAtk

Each setting included 20 images per concept, for a total of 900 images (3 models x 3 concepts × 5 evaluations × 20 images).

Below, we report the average human classification accuracies (in %) alongside the automated classifier results from the main paper for comparison:

The human evaluation results generally aligned with our automated metrics. Across all five evaluation settings and four models, the relative performance ordering was preserved: GA produced the lowest recognition rates (e.g., 1.3% human vs. 0.6% automated on standard prompts), while human classification of inpainting results for ESD-x and GA averaged 67.5% and 75.6%, closely tracking automated results in 69.1% for ESD-X and 61.7% for GA.

These observations suggest that while human and automated judgments mostly agree, human evaluation provides complementary insights, particularly in borderline cases where visual quality or conceptual ambiguity may affect recognition.

[1] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[2] Kumar, Manoj, et al. "Do better ImageNet classifiers assess perceptual similarity better?." Transactions on Machine Learning Research.

[3] Gandikota, Rohit, et al. "Unified concept editing in diffusion models." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

[4] Gong, Chao, et al. "Reliable and efficient concept erasure of text-to-image diffusion models." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

“Does it make sense to use an ensemble of classifiers to improve accuracy?”

While we believe different classifiers tell a similar story, we are happy to report results with other classifiers as well. For example, see the table below, where we calculated the pearson correlation between other classifiers trained on ImageNet and the resnet-50 classifier we used:

We find overall pearson correlations of ~0.8, which suggest strong alignment between the classifiers.

“The concept map is not rigorously verified”

The concept map in Fig. 1 is intended to illustrate the two extremes of concept erasure we consider: guidance-based avoidance and destruction-based erasure. While the probing techniques we propose offer empirical insights into the erasing mechanisms of different methods, we acknowledge that a more rigorous theoretical foundation is needed. We view this work as a first step and hope it inspires further research from the community.

Our motivation behind guidance vs destruction categorization is based on the resultant distribution of the erased concept post erasure. With guidance based methods, the unlearning algorithm pushes the conditional probability $P(X | c)$, where $c$ is the erased concept, to another known distribution. For example, with UCE we can write the erased distribution $P(X | c*)$ where $c*$ is the anchor concept, which comes from the following loss function:

$$\mathcal{L}(W) = \sum_{c_i \in E} \left\| W c_i - v_i^* \right\|^2 + \sum_{c_j \in P} \left\| W c_j - W_{\text{old}} c_j \right\|^2$$

Specifically, UCE learns a new attention projection matrix $W$ that maps erased concept prompts $c_i$ to substitute vectors $v^*_i$, while preserving the responses to protected concepts $c_j$.

On the other hand, other methods (e.g., Gradient Ascent, TV) use alternative objectives such as increasing the diffusion loss. Even if we suspect that the erasing mechanism removes explicit knowledge of the target concept from the model, the resultant concept distribution is difficult to explicitly compute and categorize. To tackle this challenge, we have proposed our suite of evaluations described in the paper as an empirical tool for analyzing erased models.

Thank you very much for your comments. If our clarifications address your concerns, we would greatly appreciate an updated assessment. If not, please let us know what can be further improved; we are happy to continue the discussion any time until the end of the discussion period. Thank you!

Thank you for giving us the opportunity to clarify about the main point of our paper:

Our work is not a survey paper, rather we propose to robustly evaluate erasure method from a multiple-perspective lens. Most of the erasure methods in diffusion models, evaluate their erasure robustness through a single or narrow lens [1,2,3]. We found that every erasure method can be circumvented with one or more of our suggested evaluations (e.g. in-context probing, gradient-based probing, diffusion trajectory, etc.). We believe our work will shift the current norm of evaluating robustness, from prompting based evaluation or textual inversion attacks, to a more holistic approach.

[1] Schwinn, Leo, et al. "Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space." Advances in Neural Information Processing Systems 37 (2024): 9086-9116.

[2] Pham, Minh, et al. "Circumventing Concept Erasure Methods For Text-To-Image Generative Models." The Twelfth International Conference on Learning Representations.

[3] Rusanovsky, Matan, et al. "Memories of forgotten concepts." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

We thank the reviewer for their thoughtful summary and positive feedback. We appreciate that the reviewer found the motivation clear and the problem well framed, and the probing methods effective. We are especially encouraged by your comment that the evaluation framework may serve as a useful tool for future work in this area.

“what is the essential difference between guidance-based avoidance and destruction-based removal?”

To clarify, we suggest that the key difference between guidance-based avoidance and destruction-based removal lies in the resulting distribution of the erased concept after erasure. With guidance based methods, the unlearning algorithm pushes the conditional probability $P(X | c)$, where $c$ is the erased concept, to another known distribution. For example, with UCE we can write the erased distribution $P(X | c*)$ where $c*$ is the anchor concept, which comes from the following loss function:

$$\mathcal{L}(W) = \sum_{c_i \in E} \left\| W c_i - v_i^* \right\|^2 + \sum_{c_j \in P} \left\| W c_j - W_{\text{old}} c_j \right\|^2$$

Specifically, UCE learns a new attention projection matrix $ W $ that maps erased concept prompts $ c_i $ to substitute vectors $ v^*_i $, while preserving the responses to protected concepts $ c_j $.

On the other hand, destruction based methods fundamentally remove the knowledge of the concept, reducing the overall likelihood that samples from $P(X)$ exhibit the erased concept. A canonical example of this would be retraining the model on data that excludes the target concept. However, even if we suspect that the erasing mechanism removes explicit knowledge of the target concept from the model, the resultant concept distribution is difficult to explicitly compute and categorize. To tackle this challenge, we propose our suite of evaluations described in the paper as an empirical tool for analyzing these erased models. We will update the final draft to make this more clear.

"This paper stops by identifying limitations of existing methods"

While our paper focuses on identifying the limitations of existing erasure methods rather than proposing a new one, similar studies have often paved the way for developing stronger approaches [1,2,3,4]. For instance, the STEREO method (the most robust and highest ranked model in our evaluation framework) was developed in response to Pham et al.'s [1] finding that textual inversion can circumvent erasure in ESD. We hope our probing insights inspire more robust erasure strategies, such as integrating context-aware probes during training or leveraging the noise-based probe scheduling setup to better capture diverse generation paths.

[1] Pham, Minh, et al. "Circumventing Concept Erasure Methods For Text-To-Image Generative Models." The Twelfth International Conference on Learning Representations.‏ [2] Srivatsan, Koushik, et al. "Stereo: A two-stage framework for adversarially robust concept erasing from text-to-image diffusion models." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.‏ [3] Robey, Alexander, et al. "Smoothllm: Defending large language models against jailbreaking attacks." arXiv preprint arXiv:2310.03684 (2023). [4] George, Naveen, et al. "The Illusion of Unlearning: The Unstable Nature of Machine Unlearning in Text-to-Image Diffusion Models." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

Thank you very much for your comments. We will include all the revisions in the final version of the paper. We are happy to continue the discussion any time until the end of the discussion period. Thank you!

Thank you for engaging in the discussion! We appreciate the further enquiry on "resultant distribution" and we would like to further clarify our earlier response:

The "resulting distribution" can in some cases, be in-theory estimated based on the loss function of the erasure method. For example, most of the erasure methods aims to change the conditional distribution of the diffusion model $P(\mathbf{X}_t|c)$ where $\mathbf{X}_t$ is the noisy image and $c$ is the prompt for the erased concept (e.g. "Van Gogh").

• In the case of ESD [1], the method aims to change the conditional distribution to $\frac{P(\mathbf{X}_t|c)}{(P(c/\mathbf{X}_t))^\eta}$ (please refer to equation 4 in their paper [1]).

• In the case of UCE [2], the method aims to change the conditional distribution to $P(\mathbf{X}_t|c*)$ where $c*$ is the anchor concept prompt (e.g. "art")

However, in other cases the resulting distribution is hard to determine. For example, Gradient Ascent aims to increase the diffusion loss under the conditional constraints (i.e. increase the diffusion loss when the model is prompted with the erased concept prompt). This makes it difficult to estimate the "resultant distribution". For this case, as the reviewer correctly pointed out, the way to estimate the distribution is post-hoc, by generating images using the erased model. Yet, generating images without any conditioning is unlikely to produce the target concept, even with the original model. As a result, verifying whether the concept has been erased is inherently challenging. This observation motivates the suite of evaluations presented in this paper.

We hope this provides better clarity.

Regarding the paper's motivation: We would also like to provide some more clarity on our main motivation. Most of the erasure methods in diffusion models, evaluate their erasure robustness through a single or narrow lens [3,4,5]. We found that every erasure method can be circumvented with either or multiple of our suggested evaluations (e.g. in context lens, gradient based lens, diffusion trajectory, etc.). We believe our work would change the current norm of evaluating robustness by simple prompting or textual inversion attacks, but rather take a holistic approach.

[1] Gandikota, R., Materzynska, J., Fiotto-Kaufman, J., & Bau, D. (2023). Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 2426-2436).

[2] Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. (2024). Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 5111-5120).

[3] Schwinn, Leo, et al. "Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space." Advances in Neural Information Processing Systems 37 (2024): 9086-9116.

[4] Pham, Minh, et al. "Circumventing Concept Erasure Methods For Text-To-Image Generative Models." The Twelfth International Conference on Learning Representations.

[5] Rusanovsky, Matan, et al. "Memories of forgotten concepts." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

We would also like to clarify the notation of the UCE loss function:

$$\mathcal{L}(W) = \sum_{c_i \in E} \left\| W c_i - v_i^* \right\|^2 + \sum_{c_j \in P} \left\| W c_j - W_{\text{old}} c_j \right\|^2$$

In this loss:

• $c_i \in E$ a token embedding in the set of erased concepts.
• $c_j \in P$ a token embedding in the set of protected (non-erased) concepts.
• $W$ is the new projection matrix of the cross attention projection layers learned during erasure.
• $W_{\text{old}}$ is the original (pre-erasure) projection matrix.
• $v_i^*$ is an "anchor" or substitute vector representing the target distribution for the erased concept $( c_i )$.

The first term encourages the projection of the erased concept $( W c_i )$ to match the substitute vector $( v_i^* )$, pushing the generation with the erased concept toward a substitute target distribution. The second term ensures that projections of protected concepts remain close to their original representations, preserving unrelated knowledge.

We present this example as a special case where erasure is expected to be guidance-based rather than destruction-based. That is, the diffusion model may still assign high likelihood to the erased concept, even if it fails to generate it from simple prompts, as demonstrated in the paper. However, many robust erasure methods rely on objectives that make it difficult to determine whether the concept is truly removed or merely avoided in specific contexts. Our empirical evaluation aims to distinguish between these two scenarios.

We thank the reviewer for acknowledging the value of the guidance-based avoidance and destruction-based removal concept erasing framework. As you point out, results emphasize the importance of testing erasure methods across multiple probing types.

“Limited Scope of Concepts”, “Scalability to Harmful Content”

We thank the reviewer for raising this point and agree that expanding to other concepts would strengthen our results. In the initial study, we deliberately chose a focused set to control for confounding factors such as prompt ambiguity, visual diversity, and domain shifts.

To assess the scalability of our framework to safety-relevant domains, we evaluated it on NSFW content, specifically nudity. We use 100 prompts from the I2P dataset and evaluate nudity scores using the NudeNet classifier. As shown in the table below, our framework successfully captures differences in guidance vs destruction-based erasing behavior across the methods. Interestingly, the Noise-Based probe (Sec.3.3) was surprisingly effective at revealing residual NSFW features in erased models. For example, using this probe, ESD-u's classification rate increased from 23% to 79%. This reinforces the utility of our multi-probe evaluation in these settings as well.

“more support of this categorization”

Our motivation behind guidance vs destruction categorization is based on the resultant distribution of the erased concept post erasure. With guidance based methods, the unlearning algorithm pushes the conditional probability $P(X | c)$, where $c$ is the erased concept, to another known distribution. For example, with UCE we can write the erased distribution $P(X | c*)$ where $c*$ is the anchor concept, which comes from the following loss function:

$$\mathcal{L}(W) = \sum_{c_i \in E} \left\| W c_i - v_i^* \right\|^2 + \sum_{c_j \in P} \left\| W c_j - W_{\text{old}} c_j \right\|^2$$

Specifically, UCE learns a new attention projection matrix $ W $ that maps erased concept prompts $ c_i $ to substitute vectors $ v^*_i $, while preserving the responses to protected concepts $ c_j $.

On the other hand, other methods (e.g., Gradient Ascent, TV) use alternative objectives such as increasing the diffusion loss. Even if we suspect that the erasing mechanism removes explicit knowledge of the target concept from the model, the resultant concept distribution is difficult to explicitly compute and categorize. To tackle this challenge, we have thus proposed our suite of evaluations described in the paper as an empirical tool for analyzing erased models.

“Trade-off … more quantitative analysis”

We thank the reviewer for their suggestion. To address it, we define a Combined Method Score, which is an average of the method’s normalized erasing performance (across the different probes), unrelated concept preservation, and FID score. We calculated the Fréchet Inception Distance (FID) on the MS-COCO30k dataset for each model to assess the "quality" of images generated by these erased models compared to the original model as yet another axis of evaluation.

This Combined Method Score reveals several interesting trends: STEREO outperforms ESD-U, the method it builds upon; and RECE similarly improves on its base method, UCE. Although both GA and STEREO achieve near-perfect erasure scores, their overall performance diverges: STEREO ranks highest overall, while GA ranks lowest. This disparity is driven by GA's poor FID and preservation scores, which offset its strong erasure.

“theoretical justification for the noise-based probe (app B) relies on some strong assumptions”

We thank the reviewer for raising this concern. While our analysis does rely on assumptions such as partial latent retention and the SDE-based formulation of diffusion, we emphasize that this is not intended as a formal proof characterizing the Noise-Based probe. In particular, we used it as a motivation for why introducing noise into the diffusion process would expand the space of “useful” trajectories for recovering images of erased concepts. We have now made this clearer in the revised manuscript.

“Prior Work”

Thank you for these helpful suggestions. We agree these works provide meaning context to our work and we incorporated these citations into the manuscript.

“discussing the computational overhead … potential limitations in scalability”

Thank you for this thoughtful suggestion. In Appendix C.2, we report that running the full evaluation suite (spanning 7 erasure methods and 13 concepts across all 6 probes) took approximately two weeks on two NVIDIA A6000 GPUs, or roughly one day per method per concept. We agree that as production systems begin to scale verification across hundreds of concepts and models, this level of computational cost could pose practical limitations.

While our framework is designed for comprehensive academic benchmarking, we recognize that production settings may benefit from more lightweight diagnostics. However, several of our introduced probes, such as the noise-based probe and inpainting-based context probe, are training-free and are in fact faster to run than other approaches.

To illustrate, we provide a rough breakdown of compute cost by probe (as a percentage of total runtime):

1. Textual Inversion: 39%
2. UnlearnDiffAtk (UDA): 37%
3. Inpainting: 3%
4. Diffusion Completion: 4%
5. Noise-Based Probe: 7%
6. Dynamic Concept Tracing: 10%

Taken together, the training-free probes (inpainting, noise, diffusion completion, and concept tracing) require less than 15% of the total runtime.

Thank you again for your excellent comments. We will revise the final version of the manuscript to include your kind suggestions. We are happy to continue the discussion at any time until the end of the discussion period. Thank you!

We thank Reviewer r56k for their positive assessment of our work and for highlighting the clarity of our writing and the novelty of the in-context probing evaluation. We are especially glad that you found Section 3's evaluation framework comprehensive and well motivated.

“it is unclear or unconvincing why the redirection arrow (black arrow in the middle and right subfigures) should necessarily lead to a nearby concept.”

Thank you for your question. The arrows on Fig.1 come to illustrate the idea of an alternative generation, “chosen” by a model that underwent a destruction based erasure. As shown in Fig.5 and Fig.6, the alternative generation leads to a nearby concept to the original one when the erasure strength is small (erasure strength being the method “strength” parameter, e.g., fine-tuning epochs). Even and especially in models that do not explicitly define “neutral concepts”. Yet, as the strength parameter increases, the alternative generation is indeed driven to be further away from the original concept.

In any case, we would like to clarify Fig.1 comes to illustrate the idea of the alternative concept distance from the original one, and not necessarily to claim it is nearby (or faraway) for any specific method. We have now further clarified this in the caption.

“The use of Textual Inversion (TI) can be highly sensitive to the amount and type of fine-tuning data”

We thank the reviewer for their comment. We report below further experiments to explore the sensitivity of the Textual inversion attacks on our chosen erased models, by training TI embeddings on 1, 10, 100, and 300 images for five representative erasure methods.

With limited reference images, TI may indeed struggle to recover erased concepts in some cases And as shown in the table below, increasing the number of TI training images significantly improves both CLIP scores and classification accuracy for certain models, indicating that erased concepts can often be relearned through TI if sufficient data is provided.

Notably, models like UCE and ESD-X showed substantial increases in classification accuracy as the number of used images increased, whereas methods such as GA and TV remained more resistant to TI attacks.

"I believe that achieving complete destruction (as claimed in destruction-based removal) may be unrealistic."

We agree that achieving complete destruction of a concept may be difficult in practice, or even impossible particularly when powerful recovery strategies like Textual Inversion (TI).

Our intention is not to claim that current methods fully realize the destruction based ideal case, but rather to organize the space of existing approaches and motivate future work that pushes closer toward true destruction.

That said, we appreciate the reviewer’s broader point: while stronger methods such as GA are better at thoroughly removing concepts, we acknowledge they may still not achieve a complete destruction of the concept. While developing fully robust methods may remain an open challenge, our paper aims to examine existing approaches and motivate future work that pushes closer toward true destruction.

“Does this mean the authors applied unlearning to the inpainting model, rather than to the standard text-to-image SD model?”

Thank you for the question. The unlearning was applied only to SD v1.4 models. We found that using a base SD v1.5 inpainting pipeline with an SD v1.4 erased UNet produced more stable and visually coherent outputs than using the SD v1.4 inpainting pipeline.

Regarding Figure.6: Consistent generations

In line 196, with the claim: “Gradient Ascent and Task Vector often converge on a consistent alternative concept”, our intention was to divert the reader’s attention to the evolution of the concept through the unlearning process. This can be seen in row 2, GA’s outputs throughout the unlearning training smoothly converges to the same ‘mushy’ concept whereas ESD has a more ‘erratic’ convergence to its final concept. We have clarified this better in our text.

Regarding Figure.5: Centroid plots

In Fig. 5, we do not plot standard deviation across generations. Rather, we show the average distance between the original and alternative generation embeddings. GA and TV show steady increases, indicating a smooth drift away from the original concept. ESD-U and ESD-X, however, exhibit sharp spikes in this metric: corresponding to the abrupt visual transitions seen in Fig. 6. We will clarify it in the final version of the manuscript and add the distribution images to show the variations of each model.

What is the target concept used in the experiments in Section 3.4 for the ESD-x and ESD-u methods?

We thank the reviewer for their observation and would like to clarify. The target concept used for these methods was "vincent van gogh self portrait." Accordingly, the trajectories converged to "a man", diverging away from Van Gogh as the erasure progressed. In accordance with the reviewer's comment about the target concept having a strong influence on the replacement output, we have run ESD-u and ESD-x with the target concept "a painting in the style of van gogh," and the resultant trajectories converged to images of a painting.

“What would happen if noise were added to the guidance signal instead”

We thank the reviewer for their insightful suggestion. To investigate this, we conducted a detailed analysis of perturbing different components of the classifier-free guidance expression:

$$\epsilon_{\text{guided}} = \epsilon_u + s \cdot (\epsilon_c - \epsilon_u)$$

We find that perturbing the conditional prediction ($\epsilon_c \rightarrow \epsilon_c + \sigma z$) or the guidance vector ($( \epsilon_c - \epsilon_u ) \rightarrow ( \epsilon_c - \epsilon_u + \sigma z )$) results in the same final expression:

$$\epsilon_{\text{guided}} = \epsilon_u + s \cdot (\epsilon_c - \epsilon_u) + s \sigma z$$

However, perturbing the unconditional prediction ($\epsilon_u \rightarrow \epsilon_u + \sigma z$) yields a subtly different form:

$$\epsilon_{\text{guided}} = (\epsilon_u + \sigma z) + s \cdot (\epsilon_c - (\epsilon_u + \sigma z)) \\ = \epsilon_u + s \cdot (\epsilon_c - \epsilon_u) + (1 - s) \sigma z$$

Notably, the noise term here is scaled by $(1 - s)$, which becomes negative when $s > 1$. This introduces an anti-aligned perturbation relative to the guidance direction, effectively suppressing or counteracting the text guidance. To explore this effect, we now include both conditional and unconditional perturbation variants in the table below:

Interestingly, we observed that perturbing the unconditional signal occasionally led to out-of-distribution generations, often producing semantically ambiguous or hallucinatory outputs. Despite this, concepts that were difficult to recover using the standard noising probe (like English Springer Spaniel), were better recovered using the unconditional perturbation. This suggests that such perturbations may expose residual concept traces in the models learned prior that are not typically activated during normal guidance.

We sincerely thank the reviewer for their thoughtful suggestions and questions. We will revise the final version of the manuscript to improve clarity, and we welcome the opportunity to continue this discussion.

Thank you for engaging in the discussion! Please find our further clarifications below:

Definition of Mechanisms

Figure 1 (Z-axis) indeed visualizes P(x). While serving illustrative purposes, we fully agree that this should be made clearer and appreciate the reviewer’s attention to this point.

Potentially Misleading Conclusion

Thank you for this opportunity to clarify. We refer to methods such as GA as “destruction-based” as we claim the tend to edit the model unconditional likelihood ($P(x)$) rather than mostly preserving $P(x)$ and editing $P(x|c)$. Yet, in practice, $P(x)$ for the target concept is reduced albeit not necessarily to zero.

We will make it clearer in the final version of the manuscript.

Figure 5 – Long-Term Behavior

We would like to clarify that for ever-increasing values of edit strengths (e.g., number of epochs for GA or vector magnitude for TV), as the target concept becomes more thoroughly erased, the ability of the model to generate unrelated concepts is destroyed as well. Yet, per the reviewer request, we are happy to report the consistency of alternative-generation for larger edit strength:

We observed that as edit strengths increased beyond typical thresholds, the CLIP centroid deviations rose gradually before plateauing, which aligns with the expected behavior of GA and TV. However, the generated images at these extreme edit levels often appeared broken or degraded, reflecting behavior well beyond that of standard erasure methods.

Inpainting-based Probe with an Erased UNet:

Thank you for this question. To clarify the mechanics: we used the Hugging Face StableDiffusionInpaintPipeline as a convenient wrapper to handle the masked latent processing, but we did not use a specifically fine-tuned inpainting model. The inpainting pipeline essentially handles the mechanics of: (1) encoding the masked image into latent space, (2) adding noise to the masked regions, and (3) running the diffusion process where our erased UNet must complete the masked areas. While Hugging Face recommends using specifically fine-tuned models for inpainting, their documentation also states that standard models like SD 1.5 can be used effectively. Since SD 1.4 and SD 1.5 share identical UNet architectures, this replacement ensures architectural compatibility and consistency with the rest of our experiments.

Our goal was not to unlearn an inpainting model, but to use the inpainting setup as a unique probe for evaluating concept erasure in a “base” diffusion model.

Perturbing the Conditional Signal τ(y):

Thank you for the clarification! We conducted a follow-up experiment where we added a Gaussian noise vector to the text embedding (conditional) signal across a range of 25 noise strengths (0.0 to 5.0) and across four concepts (“church”, “airliner”, “golf ball”, “Van Gogh”). The results are shown below:

Conditional Signal Noising – Evaluation Table

We found that methods focusing on editing the cross-attention layers (such as UCE and ESD-X, which directly interact with the prompt encoding) were particularly vulnerable to this type of perturbation. We also found that approximately 80% of successful generations occurred at higher noise levels (ε > 3.0), but beyond ε = 5.0, the images became less recognizable.

Thank you again for your insightful suggestions and follow-up comments. We would be glad to continue the discussion further during the remainder of the review period.

We would like to thank the reviewer for the thoughtful discussion and for recognizing the contributions of our work.

We agree it is important to be more cautious and precise in our claims about robustness, and we will revise the manuscript accordingly. We will ensure that our language clearly distinguishes between our conceptual categorization and empirical claims about specific methods, which may not align perfectly with the categorization. Additionally, we will use more precise language to describe alternative generations and the inpainting process.

Thank you again for your constructive feedback!