One Image is Worth a Thousand Words: A Usability Preservable Text-Image Collaborative Erasing Framework

Thank you for your valuable comments! Due to space constraints, we include additional_tables.pdf and figures at the following link: https://anonymous.4open.science/r/icml25_rebuttal-8608. References to $\textcolor{blue}{\text{Table}}$ and $\textcolor{blue}{\text{Figure}}$ in our responses correspond to the tables provided in this link.

Q1 Feature or statistically analysis to support how text-guided works.

A1: We conduct a feature space analysis to provide deeper insight into how the proposed text-guided refinement works at the embedding level. Concretely, we extract CLIP embeddings of images and visualize their movement in the latent space before and after refinement using PCA projection. This visualization clearly shows that the refined features consistently move closer to the text embedding, indicating alignment with the semantic concept described in the prompt.

***p < 0.001 (paired t-test between w/o and with text-guided)

Q2 How the image modality helps?

A2: The discrete nature of the text modality often leads to ambiguity or under-specification, especially in generative tasks. In contrast, images fill in semantic gaps left by sparse or ambiguous text. To verity this, we use LLM to generate an expression set (including words and phrases) related to the concept nudity. With a technique similar to Textual-Inversion, we use model-generated images to retrieve top-5 related expressions from the set. As shown in the examples below, the retrieved phrases from images (in $\textcolor{blue}{\text{Figure 2}}$) often reflect a broader and more nuanced conceptual space than the original seed terms (e.g., “nudity”, “sexy”). This suggests that image embeddings encode richer semantic representations than their originating textual prompts.

This demonstrates that visual representations enable the model to recover latent semantics that go beyond literal textual expansions.

Q3 Merge the framework and text-guided refinement as one contribution, and figure 7 lacks information.

A3: The refinement module is a core component of Co-Erasing, therefore we now describe the framework and its refinement module as a unified contribution, which can further reflect the essential of our approach more accurately. In the final version, we will also make sure to clarify and enrich all figures, including Figure 7, to better convey the key information.

A summary of responses:

• Additional Experiments:
  1. Integration with MACE for multi-concept erasure (a8WR)
  2. Comparison with other methods (a8WR)
  3. Defense against Ring-A-Bell attack (a8WR)
  4. Exploration of optimal fusion of text and image vectors (a8WR)
  5. Applying LoRA to reduce cost (uRE6, GDyf)
  6. Erasure of more harmful concepts (GDyf)
  7. Erasure of similar but untargeted concepts (uRE6)
• Explanations of Method:
  1. Comprehensive analysis of limitations (a8WR)
  2. Potential bias by different images of the same concept (a8WR)
  3. Computational cost analysis (uRE6, GDyf)
  4. Potential drawbacks and advantages of using model-generated images (a8WR, uRE6, VY4k)
  5. Explanation of text-guided functionality (VY4k)
  6. Explanation of image functionality (VY4k)
• Organization:
  1. Reorganization of tables for clarity (a8WR)
  2. Refinement of contributions and figures (VY4k)

Thank you for your valuable comments! Due to space constraints, we include additional_tables.pdf and figures at: https://anonymous.4open.science/r/icml25_rebuttal-8608. References to $\textcolor{blue}{\text{Table}}$ and $\textcolor{blue}{\text{Figure}}$ correspond to those provided in this link. A summary of all responses can be found at the end of the response to Reviewer VY4k.

Q1 Additional concepts including violence, hate speech.

A1: To address this concern, we conduct additional experiments including violence, illegal activity, hate speech. $\textcolor{blue}{\text{Table 18}}$ shows that Co-Erasing maintains strong performance, which confirms the generalizability of our framework and its potential applicability to a wider set of real-world moderation scenarios.

Q2 Extra computational cost of generating image set.

A2: We address the concerns as follows:

(1) Acceptable Cost of Image Generation: While Co-Erasing introduces an additional step to generate concept images, this is done prior to training, incurring no extra generation cost during training or inference. The peak memory usage during image generation never exceeds that of model deployment. Therefore, as long as the model can be deployed, generating the image set remains feasible within the same resource constraints.

(2) Plug-and-Play Design: Co-Erasing is inherently modular and can be applied to different baseline methods with minimal overhead. For instance, when used with SLD (Safe Latent Diffusion), a training-free baseline, it introduces virtually no additional computational cost.

(3) Low-Rank Adaptation (LoRA) for Training-Based Methods: For fine-tuning-based methods like ESD, we additionally incorporate LoRA to reduce training overhead. $\textcolor{blue}{\text{Table 16}}$ shows that LoRA significantly lowers memory consumption while maintaining erasure performance. Specifically, memory consumption drops from ~17K MB to 8.16K MB, with minimal impact on effectiveness.

Q3 Usability Trade-offs

A3: We agree that artifact-free erasure with perfect semantic alignment remains a challenging goal. As observed in prior works, there is often a trade-off between erasure strength and usability preservation.

In our experiments, we demonstrate that Co-Erasing achieves a significant reduction in Attack Success Rate (ASR) while maintaining competitive or improved FID and CLIP scores, indicating better preservation of overall generation quality. Although occasional artifacts may occur, our method outperforms prior approaches in balancing effective concept removal with minimal impact on usability as illustrated by Figure 2 in the paper, where Co-Erasing approaches the Pareto frontier.

Q4 Failure Cases Exist.

A4: Some failure cases do remain, particularly when the target concept shares visual features with benign or overlapping concepts. In such scenarios, adversarial prompts may still trigger partial or unintended reappearance of erased content, as illustrated in Appendix C. Nonetheless, Co-Erasing significantly reduces the model’s ability to generate the target concept across a wide range of prompts.

Thank you for your valuable comments! Due to space constraints, we include additional_tables.pdf and figures at: https://anonymous.4open.science/r/icml25_rebuttal-8608. References to $\textcolor{blue}{\text{Table}}$ and $\textcolor{blue}{\text{Figure}}$ correspond to those provided in this link. A summary of all responses can be found at the end of the response to Reviewer VY4k.

Q1 Potential drawbacks of using model-generated images instead of real images.

A1: Model-generated images offer several advantages over real images in the context of concept erasure, and potential drawbacks can be effectively mitigated:

(1) Better Alignment with Model Knowledge: Generated images are produced by the model itself and therefore directly reflect the internal knowledge and distribution we aim to erase. In contrast, real images may not align as well with how the model represents the concept internally, making them less effective for guiding targeted erasure.

(2) Control and Filtering: Prompt templates allow for precise control over the appearance and context of generated images, helping to exclude the untargeted concept. Furthermore, automatic filtering tools (e.g., NudeNet, CLIP, etc.) are applied to filter out low-quality or inappropriate images, ensuring the set remains clean and relevant.

Q2 Generating images introduces extra computational cost compared to text-only methods. Also, any optimizations to reduce the computational overhead?

A2: We address the concerns as follows:

(1) Acceptable Cost of Image Generation: While Co-Erasing introduces an additional step to generate concept images, this is done prior to training, incurring no extra generation cost during training or inference. The peak memory usage during image generation never exceeds that of model deployment. Therefore, as long as the model can be deployed, generating the image set remains feasible within the same resource constraints.

(2) Plug-and-Play Design: Co-Erasing is inherently modular and can be applied to different baseline methods with minimal overhead. For instance, when used with SLD (Safe Latent Diffusion), a training-free baseline, it introduces virtually no additional computational cost.

(3) Low-Rank Adaptation (LoRA) for Training-Based Methods: For fine-tuning-based methods like ESD, we additionally incorporate LoRA to reduce training overhead. $\textcolor{blue}{\text{Table 16}}$ shows that LoRA significantly lowers memory consumption while maintaining erasure performance. Specifically, memory consumption drops from ~17K MB to 6.18K MB, with minimal impact on effectiveness.

Q3 Potential influence on visually similar but benign content.

A3: We address the risk of over-erasure from both theoretical and practical perspectives:

(1) Conceptual Justification: Co-Erasing operates under this assumption: if the diffusion model to be erased can distinguish between two similar concepts, it can be fine-tuned to suppress one (the target) while preserving the other. By guiding the model to move away from the distribution of the target concept, we aim to minimize unintended effects on adjacent but benign concepts.

(2) Empirical Validation: To verify this in practice, we further conduct experiments on visually similar yet untargeted concepts in $\textcolor{blue}{\text{Table 17}}$. These tests evaluate whether the method preserves non-target content that shares overlapping features with the erased concept. Our results confirm that Co-Erasing maintains semantics compared to the baseline, avoiding significant degradation in generation quality for related but non-targeted categories.

Q4: Erase misinformation or deepfake elements?

A4: Unfortunately, Co-Erasing is not designed to erase abstract or evolving concepts such as misinformation or deepfake elements. These types of content often lack a consistent or well-defined visual representation, making them fundamentally different from the specific, visually grounded concepts that Co-Erasing targets. In fact, addressing misinformation or deepfakes aligns more closely with the goals of forgery detection or adversarial content analysis, which typically involve classification or verification tasks rather than generative model editing.

Thank you for your valuable comments! Due to space constraints, we include additional_tables.pdf and figures at: https://anonymous.4open.science/r/icml25_rebuttal-8608. References to $\textcolor{blue}{\text{Table}}$ and $\textcolor{blue}{\text{Figure}}$ correspond to those provided in this link. A summary of all responses can be found at the end of the response to Reviewer VY4k.

Q1 Vulnerability from text-image gap or model incapability?

A1: We believe the text-image gap remains a key factor to the vulnerability of diffusion models. Our reasoning is threefold:

(1) Scaling doesn’t close the gap: We further conduct erasure on Stable Diffusion XL, a stronger model with improved text-image alignment. $\textcolor{blue}{\text{Figure 1}}$ shows similar vulnerabilities, indicating that the gap is ubiquitous regardless of scale.

(2) Attacks often rely on visual cues. CCE[1] uses Textual Inversion from concept embeddings, and UnlearnDiff[2] optimizes prompts with target concept images, highlighting reliance on cross-modal cues.

(3) Prior work on modality misalignment. Studies[3] support that modality gaps in CLIP (used in Stable Diffusion) impacts downstream tasks.

[1]Circumventing Concept Erasure Methods For Text-to-Image Generative Models

[2]To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy to Generate Unsafe Images ... For Now

[3]Mind the Gap: Understanding the Modality Gap in Multi-Modal Contrastive Representation Learning

Q2 Limitation 2 only shows ESD, and text augmentation is too simple.

A2: We extend analysis to SLD (Safe Latent Diffusion) and MACE and apply LLM-based text augmentation[4]. $\textcolor{blue}{\text{Table 1}}$ shows persistent text-based limitations under more models and richer prompts.

[4]Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate

Q3 Potential disadvantages on multiple concept erasing task.

A3: We agree multi-concept erasure is important. Since Co-Erasing is plug-and-play, we additionally integrate it with MACE and evaluate on multi-concept erasure. $\textcolor{blue}{\text{Table 2}}$ shows that Co-Erasing can complement MACE effectively.

Q4 Differences with other works involving images.

A4: While other methods use images, Co-Erasing differs in key ways:

(1) Data Efficiency: Unlike [5], which uses external real images, Co-Erasing uses self-generated images, reducing cost and aligning better with model's internal knowledge. See A1 (uRE6) and A2 (VY4K) for details.

(2) Preservation of Untargeted Concepts:[5] does not explicitly preserve unrelated concepts, whereas Co-Erasing incorporates a text-guided refinement module to avoid unintended erasure.

(3) Modularity: Co-Erasing is plug-and-play with methods like MACE and SLD, whereas [5] lacks flexibility.

Additionally as shown in $\textcolor{blue}{\text{Table 3}}$, Co-Erasing outperforms[5] with a better trade-off.

[5]Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearning

Q5 Potential bias when images of the same concept differ significantly and using only one image.

A5: Co-Erasing is robust to visual diversity across images of the same concept:

(1) We further analyze the distribution of image variations across concepts in $\textcolor{blue}{\text{Table 4}}$ and found no strong relation between visual diversity and performance drop, indicating Co-Erasing is robust to image-level differences within a concept.

(2) At each training step, one image is randomly selected from the generated set (typically 50+), which mitigates overfitting to any single image and helps capture a common representation of the target.

(3) Text-guided refinement module focuses on the semantic core of target concept, reducing impact from untargeted visual elements.

Q6 Evaluation against red-team attack: Ring-A-Bell.

A6: We already evaluate against UnlearnDiff (UDA) in the paper. To further address this concern, we include Ring-A-Bell in $\textcolor{blue}{\text{Table 5}}$, which shows Co-Erasing can improve resistance against this attack.

Q7 The reason why generated images perform better.

A7: Please see A1 (uRE6) and A2 (VY4K).

Q8 Other potential merging methods of text and image latent vectors.

A8: Summation of text and image latent vectors follows IP-Adapter, widely adopted for integrating images into the generation process. To explore alternatives, we run experiments with different schedulers. $\textcolor{blue}{\text{Table 6}}$ shows slight performance gains. In future work we will further investigate optimal fusion designs.

Q9 More informative result comparison.

A9: We show $\textcolor{blue}{\text{Table 7-15}}$, following MACE and AdvUnlearn[6], explicitly presenting quantitative results.

[6]Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models