Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models

We sincerely thank you for the exceptionally detailed and highly constructive review. Your comments have provided a clear roadmap for improving our paper, and we are especially grateful for your offer to raise your score. We are pleased to report that we have conducted four new sets of experiments during the rebuttal period that specifically address every one of your concerns. We are confident that these new results and clarifications will resolve your questions.

Weakness 1: Clarity, LCP process, inference time, and C_vis construction.

Thank you for these detailed questions. We apologize for the lack of clarity in our initial submission and have worked to significantly improve the presentation.

For Visual Explanation: We have prepared a new diagram that visually explains our entire workflow. This has been included in Section 3 of the final paper to make our method much more intuitive.

For LCP Process & Detailed Inference Time Analysis: To clarify, the LCP feedback loop is an on-demand, two-pass process. A second, time-consuming pass is only triggered if residual concepts are detected. To quantify this overhead, we have conducted a new, detailed timing experiment:

This analysis reveals three key points:

(1) Our core method adds negligible overhead (3.21s vs 3.11s).

(2) The "doubled time" is a worst-case scenario.

(3) In a practical safety task (I2P), our average time (4.09s) is comparable to other training-free methods like SLD and SAFREE, while achieving vastly superior safety (as shown in our new I2P comparison). The LCP module is an optional(only in high demand when security requirements are extremely high), highly efficient trade-off for critical safety needs.

This complete timing analysis has been prepared for inclusion in the final paper.

For C_vis Construction: To improve clarity, we have expanded the description in Section 3.3. The revised text now explicitly states that C_vis is formed by selecting concepts {c_i} where the vision detector's confidence score surpasses a threshold τ_vis, and we emphasize that this is an efficient, single-batch operation. This clarification will be present in the final revised version.

Weakness 2: Missing comparison with baselines like SLD or SAFREE.

This is a very important point, and we thank you for raising it. Our initial experiments focused on comparing against strong parameter-modifying baselines (like MACE and Receler). The rationale was based on the general understanding in the field that inference-time methods, while flexible, typically do not match the performance of methods that directly edit model weights. Our primary goal was therefore attempting to demonstrate that our inference-time approach could not only compete with but surpass these powerful, heavyweight baselines, which would be a significant finding. However, to validate your point and provide a complete picture of our method's standing among its direct, training-free peers, we have conducted a new, head-to-head comparison with SLD and SAFREE on the challenging I2P safety task. The results, broken down by category, are as follows:

New Comparison on I2P (Unsafe Instances Count ↓):

This detailed breakdown shows that our method achieves near-perfect erasure across almost all categories, reducing the total number of unsafe generations by over 98% compared to the next best training-free method, SAFREE(from 82 instances to 1). This confirms that our method holds a substantial advantage within its category on safety-critical tasks. We have already drafted the analysis for this new comparison and will ensure the table and discussion are included in the final paper.

Weakness 3: Lack of robustness evaluation against adversarial attacks.

We agree that adversarial robustness is a critical concern. Our initial work focused on establishing a general-purpose framework for high-fidelity erasure under standard conditions, similar to other general-purpose methods like MACE (CVPR 2024) and AdaVD (CVPR 2025) that didn't add the adversarial experiment as well. However, to address your valid concern, we conducted a new, targeted experiment using the Ring-A-Bell (RAB) benchmark you referenced. We created a test set of 95 adversarial prompts using methodology provided by ring-a-bell to evaluate the Attack Success Rate (ASR). The results of this preliminary study were highly encouraging:

New Adversarial Robustness Study (ASR on 95 RAB prompts):

It is noteworthy that this 0.0% Attack Success Rate was achieved by our core semantic module alone, without the LCP detector. While not an exhaustive study, this strong initial evidence suggests our approach has inherent advantages in resisting prompt-level attacks. We have prepared this new experiment and its analysis for inclusion in the final revised version.

Weakness 4: Ablation studies are lacking.

Thank you for pushing for a more thorough analysis. We agree this is essential and address each point:

a. Calibrated vs. Naive Addition: This comparison is provided qualitatively in the supplementary material (Figure 1), which clearly shows naive addition fails catastrophically in multi-concept scenarios. We will highlight this in the main text.

b. Impact of LCP: This ablation is presented in the main paper's appendix (Table 5). It shows the LCP module is critical for safety, reducing unsafe images on I2P from 47 to 1.

c. Influence of threshold τ: You are correct, this was an omission. We apologize. We have now conducted a new ablation study on τ of Object Erasure(CIFAR10). The results are as follows:

The results show that performance is highly stable within a wide range [0.05, 0.5], confirming the robustness of our choice. We have prepared this new ablation study and its analysis for inclusion in the final revised version.

We hope our detailed responses, backed by these extensive new experiments, have fully addressed your concerns. We are deeply grateful for your insightful guidance and look forward to your further feedback.

We sincerely thank the reviewer for appreciating the conceptual novelty and strong performance of our work. We are also very grateful for the detailed, constructive feedback on improving the paper's presentation and experimental interpretation. We have carefully addressed all your points and conducted new experiments, which we detail below.

Weakness 1: Connection between formal definitions and the method.

This is an excellent point regarding the paper's structure and logical flow. We agree completely that the connection between our problem formulation and methodology needs to be stronger and clearer. To address the three specific issues you raised, we have implemented the following targeted revisions:

1. Placement and Completeness of Definitions: We followed your suggestion and move the problem statement into its own dedicated "3. Problem Formulation" section. In this section, we formally defined the three key objectives, building upon the definitions already in our paper. Here is our revised and expanded formulation:

• Completeness: As defined in our paper (Eq. 1), the operator must effectively remove the target concepts C_erase. Intuitively, this means the probability of C_erase appearing in a generated image I' should be below a safety threshold ϵ_safe.
• Locality (Fidelity): As defined in our paper (Eq. 2), the modification should minimally affect non-target concepts. This means the change in the presence probability of any non-target concept c should be limited by a tolerance ϵ_tol.
• Robustness (New Definition): Building on this, we formally defined Robustness as maintaining Completeness even when the prompt p is paraphrased or adversarially modified to p_adv. This frames Robustness as a crucial stress test for the erasure's reliability.

2. Explicit Link to Method: To bridge the gap between these definitions and our method, we added concise, explicit statements:

• In the Methodology Section: We added a short "bridging" paragraph after the new Problem Formulation. It will state: "To achieve these objectives, Semantic Surgery is designed with two core principles. We tackle Completeness and Locality through a precisely calibrated vector subtraction that neutralizes semantics at their source. To ensure Robustness, our method operates on the global text embedding and "LCP Visual Feedback", making it inherently resilient to token-level paraphrasing, unlike methods that target specific words."

• In the Experiments Section: When introducing the evaluation metrics (ACC_E, ACC_R, ACC_L), we explicitly "called back" to these intuitive definitions, reinforcing the link right where the results are presented.

We believe this structured approach—with complete, upfront definitions and explicit links in both the methodology and experiments—will create a much clearer and more coherent narrative. Thank you for this deeply insightful suggestion; it has been instrumental in helping us refine the core logical structure of our paper.

Weakness 2: Unclear experimental setup for CIFAR-10.

Thank you for asking for this clarification. We apologize for the ambiguity. We followed the standard evaluation protocol established by prior works in this area (e.g., ESD, MACE, Receler). To be explicit:

• We do not use the 32x32 CIFAR-10 images for any training or fine-tuning. Our method is entirely training-free.
• We use the CIFAR-10 class names (e.g., "dog", "airplane") to form text prompts, such as "a photo of a {class}".
• These prompts are used to generate high-resolution images with the standard SD v1.4 model and the concept-eliminated SD v1.4 models(with different erasing methods).
• Finally, these generated images are evaluated by an object detector to calculate the metrics. We have revised Section 4.2 to make this standard process presented in a clearer manner.

Weakness 3: Interpretation of results could be misleading (H-score vs. Robustness).

This is a great observation, which we fully agree. A key strength of our method lies in its superior robustness (ACCR). Your point is further validated by our new adversarial robustness experiments conducted and presented in our response to reviewer moxY. In these tests against the challenging RAB benchmark, our method significantly outperformed all baselines.

New Adversarial Robustness Study (ASR on RAB prompts):

It is noteworthy that this remarkable 0.0% Attack Success Rate was achieved using our core semantic module alone, without the LCP detector.

In the final version, following your suggestion, we have revised our results analysis. We first highlighted the significant gains in both standard robustness (ACCR) and this newly-demonstrated adversarial robustness. We then presented the H-score as an overall summary metric, providing a much clearer picture of our method's distinct advantages.

Weakness 4 & Minor Weakness 1: Hyper-parameter discussion and lack of a visual diagram.

We thank you for these important points.

• Hyper-parameters: We apologize for not highlighting these analyses sufficiently in the main part of the submitted version. We have conducted extensive studies on our hyperparameters. The analysis for β and γ can be found in the Supplementary material (Figures 2 & 3). We further conducted a new ablation study on the threshold τ, which confirmed our method's stability (H-score > 92.0 for τ in [0.05, 0.5] with a fixed beta(-0.12) and gamma(0.02)). We have created a new subsection in the main paper to summarize these crucial findings.
• Visual Diagram: We agree that a clear visual diagram is essential. We have prepared a new figure that illustrates the entire workflow of Semantic Surgery. This has been added to Section 3 in the final revised version to make our method's intuition much more accessible.

Question 1. How can comma-separated string concatenation be an effective operator?

This is a good question about the underlying mechanism. The effectiveness of comma-separated concatenation stems from the powerful compositional understanding of the pre-trained CLIP text encoder. CLIP was trained on vast amounts of web text, where concepts listed with commas typically represent a collection of distinct entities within a shared context. When CLIP processes a string like "dog, cat, playing together", it generates a composite embedding that captures the individual concepts ("dog", "cat") while preserving the shared context ("playing together"). Our qualitative results in the Supplementary material (Figure 1) provide strong empirical evidence for this: our method successfully removes both animals while keeping the "playing" scene intact, whereas a naive arithmetic approach fails. This demonstrates that leveraging CLIP's inherent compositional ability is a simple yet highly effective strategy.

Question 2. Why use the maximum estimated score (p_joint = max{p_i}) rather than an average?

This is another great question about our design choice. We chose to use the maximum score based on a conservative, 'safety-first' principle. In multi-concept erasure, if even one concept is detected with very high confidence (a high ρ_i), it implies a strong user intent for erasure related to that entire conceptual group. An average score could be diluted by other, lower-confidence concepts in the prompt, potentially leading to an under-powered erasure strength and incomplete removal of the most prominent target concept. The max operator ensures that our erasure strength is always driven by the strongest signal of user intent, prioritizing erasure completeness, which is often crucial in safety-related applications.

We hope our detailed responses, supported by the new experiments, have fully addressed your concerns. We are very grateful for your meticulous review, which has provided invaluable guidance for improving the clarity and depth of our paper.

We sincerely thank the reviewer for the positive feedback on our technical formulation and novelty. We also appreciate the insightful comments on potential weaknesses, which have pushed us to conduct several new, comprehensive experiments. We are pleased to report that the new results not only address your concerns but also reveal surprising new strengths of our method.

Weakness 1: Essential calibration details in the Appendix.

Thank you for the suggestion. We agree that moving key calibration details to the main text would improve readability. In the final version, we will condense our method section and move a concise summary of the key calibration details from the appendix to the main text, ensuring the paper is more self-contained and easier to read.

Weakness 2: Possibility of conflating semantically overlapping concepts with Co-Occurrence Encoding.

We strongly agree with the reviewer that handling semantic overlaps is indeed crucial. This is precisely why we designed the Co-Occurrence Encoding. As demonstrated in our Supplementary material (Fig. 1), our comma-concatenation approach effectively preserves the semantic context (e.g., 'playing together'), while a naive vector summation leads to significant image degradation. This qualitative result demonstrates that our method successfully avoids the conflation issue in multi-concept erasure.

The effectiveness of comma-separated concatenation stems from the powerful compositional understanding of the pre-trained CLIP text encoder. CLIP was trained on vast amounts of web text, where concepts listed with commas typically represent a collection of distinct entities within a shared context. When CLIP processes a string such as "dog, cat, playing together", it generates a composite embedding that captures the individual concepts ("dog", "cat") while preserving the shared context ("playing together").

Weakness 3: Missing common robustness evaluation through adversarial prompt attacks.

We sincerely thank the reviewer for this critical feedback and agree that evaluating adversarial robustness is an important aspect. Our initial work focused on establishing a general-purpose framework for high-fidelity zero-shot concept erasure under standard conditions, aiming to first achieve state-of-the-art performance in completeness and locality, similar to other general-purpose methods including MACE (CVPR 2024) and AdaVD (CVPR 2025). This was the primary reason for not including adversarial evaluations in the initial submission.

However, we fully acknowledge the importance of this dimension. To address this valid concern, we have conducted a new, targeted experiment to probe our method's robustness. For this study, we chose the Ring-A-Bell (RAB) benchmark [1]. Similar to the UnlearndiffAtk framework you mentioned, RAB represents a state-of-the-art approach for generating adversarial prompts. We selected RAB for two principled reasons: (1) Its prompt-based nature makes it model-agnostic, ensuring a fair, end-to-end comparison across all baselines; and (2) its focus on linguistic manipulation provides a direct stress test of our core semantic-level intervention.

We created a test set of 95 adversarial prompts from RAB to evaluate the Attack Success Rate (ASR). The results are as follows:

New Adversarial Robustness Study (ASR on 95 RAB prompts):

The results show that on this test set, our method achieved a 0.0% Attack Success Rate. It is particularly noteworthy that this result was achieved using our core semantic module alone, without relying on the LCP visual feedback loop. This demonstrates the powerful, inherent resilience of our semantic-level manipulation against this type of prompt-based attack.

While we acknowledge this is not an exhaustive study covering all attack vectors, this strong initial evidence suggests our core mechanism possesses significant advantages. We have added this new experiment and discussion to the final paper. We are grateful for your suggestion, as it has helped us uncover this important strength.

We hope our responses and the new experimental results have fully addressed your concerns.

[1] Tsai et al. "Ring-A-Bell! How Reliable are Concept Removal Methods For Diffusion Models?." The Twelfth International Conference on Learning Representations. 2024.

We are deeply grateful to the reviewer for their strong support and for the detailed, insightful feedback. Your positive assessment of our work is incredibly encouraging. We have carefully considered all your suggestions, especially the critical point about evaluation bias, and have conducted new experiments to address them. We believe these additions have made our paper significantly stronger.

Weakness 1: Lack of visualization and discussion on limitations

Thank you for these valuable suggestions.

For Visuals: You are correct that qualitative examples are very important. While we included several in the appendix due to space constraints, we agree that a key figure should be in the main text. To give you a preview of what we will add, our appendix already contains extensive qualitative results:

• Figure 3 demonstrates the effectiveness of our LCP module in mitigating latent concept persistence, correcting errors caused by implicit concept association.

• Figures 4 & 6 provide qualitative evidence for erasure Completeness and Locality on the CIFAR-10 benchmark.

• Figures 5, 7, & 8 showcase the versatility of our method across diverse tasks, including object erasure, multi-celebrity erasure, and artistic style removal.

• In our supplementary materials, Figure 1 gives a qualitative comparison between our Co-Occurrence Encoding and Naive Approach, showcasing the effectiveness of our proposed Co-Occurrence Encoding mechanism on multi-concept erasure.

To better showcase this in the main paper, we have designed a new, consolidated figure. The design features the most compelling examples, mostly from Figure 5, to provide an at-a-glance demonstration of our method's wide-ranging capabilities. We have prepared this new figure for inclusion in our final revised version.

For discussion on limitations: To demonstrate our commitment to a balanced presentation, here is the detailed summary of the expanded limitations section we will add to the paper. We believe a thorough analysis of these points is crucial for contextualizing our work and guiding future research.

---

Limitations and Future Work

Our method, while effective, has several limitations that open avenues for future work:

1. Reliance on Encoder's Linear Structure: Our approach's applicability is fundamentally conditioned on the text encoder itself exhibiting a near-linear structure where concepts are semantically separable. This assumption, while holding for powerful encoders like CLIP and many common concepts, may not be universally true. It might be less effective for: (a) future, potentially non-linear encoder architectures, or (b) highly entangled, abstract, or metaphorical concepts where a clean vector representation is ill-defined.

2. Sources of Non-local Effects on Locality (ACC_L): Although our method shows strong locality, it is not entirely free from non-local effects, as reflected by the ACC_L metric. We identify three primary sources for this:

• Imperfect Semantic Thresholding: The decision boundary defined by our β parameter, while effective, may not be perfectly sharp for every conceivable prompt-concept pair. In ambiguous cases, our method might perform a semantic operation on a prompt where the concept is only weakly present, thus affecting related but untargeted semantics and impacting locality.
• Visual Detector Errors (in LCP): This is a significant factor. For the optional LCP module, false positives from the vision detector can trigger an unnecessary second-stage erasure, which can inadvertently alter related visual elements. We observed a concrete example of this during our experiments: when erasing the concept "automobile", the AOD detector used in our LCP loop frequently misclassifies "trucks" as "automobiles". This causes our LCP module to incorrectly trigger a stronger erasure on prompts that should have generated a "truck", leading to a noticeable drop in locality for the "truck" class. This highlights the critical dependence of the LCP's locality on the detector's precision.
• Inherent Model Stochasticity: Some locality degradation is attributable to the inherent stochasticity of the base generative model itself. As seen in our baseline results (SDv1.4 with No Erase), the model sometimes fails to generate concepts perfectly aligned with the prompt, which contributes to a baseline level of locality "noise".

3. Detector Dependency for LCP Module: The performance of our optional LCP module is inherently tied to the availability and accuracy of an external vision detector. The lack of reliable detectors for abstract concepts (e.g., 'loneliness', 'artistic style') limits the module's applicability to more concrete, visually verifiable concepts. Furthermore, the detector's performance directly creates a trade-off: false negatives (missing a concept) can lead to incomplete erasure, while false positives (detecting something that isn't there) can harm locality by triggering incorrect feedback, as discussed in point 2.

---

We have already included this detailed discussion in our paper to provide a clearer context for our work and to chart clear paths for future improvements in the field.

Weakness 2: SD v1.4 may be outdated.

We thank the reviewer for his idea and agree that this is a thoughtful point. We chose SD v1.4 primarily to ensure a fair and direct comparison with the extensive list of prior works in concept erasure (e.g., ESD, MACE, Receler), which almost exclusively use SD v1.4 as the standard benchmark model. This allows for a clear assessment of our method's advancement over the existing literature. We agree that testing on newer models like SDXL or SD 3 is an important next step, which we would include in our discussion of our method's current limitation.

Question 1: Did we use the AOD detector for both LCP and model evaluation?

This is an extremely insightful and critical question. We sincerely thank you for identifying this potential evaluation bias in our original submission. We initially did use the same AOD detector for both the LCP feedback and the final evaluation.

We agree that this setup is not ideal and may introduce a bias. To rectify this, we have conducted a new, more rigorous evaluation. While keeping AOD as the feedback detector within our LCP loop (to maintain the method's original mechanism), we now use a completely independent and powerful open-vocabulary detector, OWL-ViT[1] , as the unbiased "referee". To ensure a fair comparison under this new, stricter protocol, we re-evaluated our method and a representative set of strong and foundational baselines.

New Evaluation on CIFAR-10 (Avg, Evaluated by the independent OWL-ViT):

This new, unbiased evaluation reveals several key observations and findings:

1. Maintained Superiority: Our method maintains its significant lead in the overall H-score (93.60) over the strongest baselines, confirming our reported superiority is robust and not an artifact of evaluation bias.
2. Dominance in Robustness: As pointed out by another reviewer, a key advantage of our method is its robustness. This is starkly evident here, with our method achieving an ACC_R of 2.00, an order of magnitude better than the next best parameter-modifying method (Receler at 10.00).
3. Unbiased High Performance: The results confirm our method's exceptional balance between Completeness (ACC_E) and Robustness (ACC_R) without overly sacrificing Locality (ACC_L).

We are deeply grateful for this crucial suggestion, as it has substantially improved the scientific rigor of our work. We will replace the original evaluation tables with these new results in the final paper.

Paper Formatting Concerns

1. Typo "desiderata": Thank you for pointing out the typo and we will correct it in the final version. We have also gone through the paper and make sure typos are corrected before the final version.

Once again, thank you for your strong endorsement and for providing feedback that has been instrumental in strengthening our paper. We hope our responses and the new, more rigorous experiments have further solidified your confidence in our work.

[1] Minderer, Matthias, et al. "Simple open-vocabulary object detection." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.