Minimalist Concept Erasure in Generative Models

Thank you for recognizing the strengths of our work. We’re glad you found the minimalist objective conceptually appealing, and we appreciate your acknowledgement of the practicality and scalability of our framework. We reply to your concerns below:

“While metrics such as FID and SSIM scores are informative, they do not comprehensively capture all facets of model performance.”

The use of ACC, CLIP, FID, and SSIM is well justified, as these metrics are commonly used for evaluation in prior unlearning works. Please refer to our Related Works section for supporting literature.

“Essential dimensions like diversity, novelty, and controllability remain insufficiently evaluated. For instance”

Can you elaborate on evaluation metrics for diversity, novelty, and controllability and provide concrete metrics?

“The paper tests only a few adversarial prompts (e.g., Ring-A-Bell, MMA-Diffusion, P4D, I2P), leaving uncertainty about performance under more diverse or sophisticated adversarial scenarios”

Adversarial attack on unlearning is a newly field emerged just recently. There are only a few related works available. In this work, we selected SOTA adversarial attack methods (Ring-A-Bell from ICLR2024, MMA-Diffusion from CVPR2024, P4D from ICML2024, and I2P from CVPR2023). We believe our choices are representative. We welcome any specific suggestions you may have.

“The paper lacks evidence demonstrating effectiveness across fundamentally different generative architectures, such as GANs or VAEs”

Our method can be easily extended to GANs and VAEs, since they both perform single-step generation. However, study on GANs and VAEs have limited importance due to their limited generative ability. We choose to evaluate our method on SOTA models. This is also acknowledged by reviewer B5EZ and BukH.

“The current description of the algorithm lacks clarity on key details. How is the binary mask initialized and optimized? Are additional loss functions beyond distributional distance utilized? How is the trade-off between concept erasure effectiveness and performance preservation explicitly managed during optimization?”

Our final loss is precisely described in L172, i.e. no other loss is utilized. In addition, Appendix H provides details and training configuration. Trade-off between concept erasure effectiveness and performance preservation is discussed in L370-375 and Figure 5 in section Ablating $\beta$.

“The paper currently lacks explicit theoretical claims and formal proofs.”

Our loss is rigorously derived from KL divergence between the generative distributions of a model. The problem formulation and core derivation are all presented in Section 3. Appendix A shows full derivation of preservation loss. Appendix B shows full derivation of erasure loss. Appendix C shows the full loss derivation for diffusion models. Appendix D shows the connection between step-wise loss by analyzing a different KL divergence setup.

“The presented ablation studies lack sufficient depth”

We present 4 other ablation studies besides module ablation. They are closely related to the algorithmic choice mentioned in this work.

“However, due to the absence of theoretical derivations or mathematical proofs and since the provided code was not practically executed during this review, the clarity and reliability of the algorithmic details remain uncertain.”

We want to emphasize that Appendix A,B,C, and D provides sufficient mathematical proofs. All these proofs are appended to the maintext rather than separately in the supplementary materials. Regarding the issue with executing our code. We would like to know more details so we can help you reproduce our results.

“The paper emphasizes quantitative metrics with limited qualitative visual examples. Conducting more thorough error analyses, examining failure cases … ”

Besides Figure 3-6 in the main text, we present Figure 7-14 in Appendix, which include some failure cases in these qualitative examples due to incorrect setups or hyperparameter changes.

“Is there a theoretical or empirical analysis that supports your claim that neuron masking provides inherent robustness advantages over weight-tuning methods?”

As stated in L201, we build on the finding that masking leads to improved robustness performance [1].

We hope our response has addressed your concerns, and we’re happy to answer any further questions you may have. If there are no additional concerns, we would be grateful if you would consider further supporting this work by raising your rating.

[1]: Pruning for robust concept erasing in diffusion models

Thank you for the thoughtful and thorough review. We greatly appreciate your acknowledgment that our claim is well-supported by the evaluation with conventional settings as well as adversarial settings. We also appreciate your review of our theoretical derivations and acknowledgement of their soundness. We address your concern below:

1. Additional concept erasure works:

We appreciate your comment regarding additional baselines. From your reference, Recler edit features after cross-attention modules, MACE finetunes $W_v$ and $W_k$ in cross-attention modules, and Forget-Me-Not performs attention-resteering on cross attention modules. Since all three methods rely on cross-attention, they are not directly applicable to FLUX, which uses MM-attention instead. As for One-dimensional Adapter, SA, and Scissorhands, including them as baselines poses significant challenges, as they do not provide official implementation for FLUX, and adapting them to FLUX requires substantial effort due to their complexity. We will include these baselines and discuss them in our related works.

2. Training on separate models or a monolithic model

In our evaluation, we train separate models for each category that contains multiple concepts—an evaluation setting commonly adopted in prior unlearning works. Additionally, we present a result where a single monolithic model is trained to unlearn concepts across multiple categories. As shown in the table, unlearning multiple concepts across different categories is more challenging compared to unlearning concepts in a single category.

Table: COCO performance as the number of unlearned concepts increases. Arrows indicate preferred direction.

3. Details on how baselines are compared:

Since many unlearning works are implemented on SD1.4 and lack official support for FLUX, we reimplemented all baselines used in this work. For a fair comparison (hyperparameters are originally proposed for SD1.4), we performed hyperparameter search for each baselines to find the best-performing hyperparameter, which we used in our experiments. Details of baseline implementation are shown in Appendix H.2. One qualitative example of ESD hyperparameter performance on FLUX is shown in Figure 12. We will update the appendix to include the full set of baseline hyperparameters. Additionally, we plan to release our FLUX implementations of these baselines upon acceptance to support reproducibility.

4. Do the authors have any insight into which attention blocks (joint attention/self-attention) in FLUX play a role in generating these concepts?

On FLUX, we observe behavior that differs significantly from U-Net-based SD models (e.g., SD1.4). Unlike SD models, which generate concepts in cross-attention layers, we observe that normalization layers and FFN layers play a critical role in generating concepts. This is supported by our ablation study in Table 4. We also present additional qualitative results in Figure 11 in Appendix K.2. These findings motivate our decision to exclude attention modules from FLUX unlearning, as noted in Line 382.

We hope our response has addressed your concerns, and we’re happy to answer any further questions you may have. We sincerely appreciate your recognition of our work as a meaningful contribution to the current unlearning research with SOTA FLUX model. We would be grateful if you would consider further supporting this work by raising your rating.

Thank you for the thoughtful and thorough review. We’re glad you found our approach to minimalist concept erasure well-motivated and effective across benchmarks. We appreciate your feedback and the recognition of our theoretical and empirical contributions.

Regarding your concern, we acknowledge the efficiency trade-off introduced by complete gradient trajectories across all generation steps. Nevertheless, since unlearning only requires one training run, improving performance with a trade-off on runtime is acceptable. Besides, we show that concept removal requires only a few GPU hours.

We’re happy to answer any further questions. If you don’t have any additional concerns and find that this work complements the current unlearning research, we would greatly appreciate it if you could raise your score to further support it.

Thank you for the thoughtful and thorough review. We address your concern below:

Claims:

1. As stated in L201, prior masking-based unlearning methods achieve accurate and robust results, which we build on due to their strong performance. Regarding neuron masking for strongly correlated concepts, Table 2 shows that model performance degrades minimally on neutral prompts (see LAION dataset). Visual examples in L228 further illustrate that our method does not unlearn a concept by pushing it toward a predefined anchor or a contrastive concept (our result is neither nude nor heavily dressed). Further understanding how the model functions internally falls under the domain of mechanistic interpretability research and is beyond the scope of this work.

2. Our derivation in Appendix A shows how we derive from the KL divergence between the original and unlearned models to our preservation loss. L263 explains how the mask gradient is composed of the gradients of all intermediate masks, following the chain rule.

3. We conduct extensive evaluations of model performance post-unlearning. For target concepts, we report CLIP (text alignment), FID (image quality), and SSIM (structural similarity between original and unlearned outputs), with results in Table 1 showing our method outperforms baselines. For neutral prompts, we evaluate 5,000 samples from the LAION dataset (Table 2), demonstrating that our unlearned model maintains strong performance.

Methods and Evaluation Criteria:

1. We response in Claims 3.

2. Our method fundamentally differs from CA: while CA applies per-step losses on intermediate variables, our loss operates only on the final output. Additionally, CA requires an anchor concept paired with each target concept, whereas our approach does not rely on any anchor.

3. In our evaluation, we train separate models for each category that contains multiple concepts. This evaluation setting is adopted in almost all unlearing works. We present an additional result to unlearn one monolithic model across multiple categories. | Number of Unlearned Concepts | CLIP ↑ | FID ↓ | SSIM ↑ | |----------------------------------|--------|--------|--------| | FLUX (Original) | 0.31 | 40.4 | - | | 10 (IP + Styles) | 0.29 | 44.4 | 0.55 | | 20 (IP + Styles + Nudity) | 0.23 | 49.3 | 0.53 | | 50 | 0.20 | 58.3 | 0.48 | Table: COCO performance as the number of unlearned concepts increases.

4. We do not observe any significant gradient instability issues, as modern model architectures utilize residual connections that help stabilize gradients during training.

5. One cannot conclude that an early step still preserves a concept simply because it is not altered, as it may no longer resembles the concept in the final outcome. Afterall, we concern the presence of a concept in the last step.

6. In response to your feedback, we performed UMAP visualization on the final denoised latent. The results show clear separation between unlearned and original FLUX features using the same prompts. We will include this plot in the revision to further support our claims, alongside detection accuracy results.

Experimental Desings Or Analyses:

1. We apologize for the misunderstanding caused. We include the result on SD-XL below and will add it in Appendix. In addition, Appendix C shows the derivation and final loss for diffusion models, we welcome you to check its correctness. | Category | Method | ACC ↓ | CLIP ↑ | FID ↓ | SSIM ↑ | |----------------|--------|--------|--------|--------|---------| | IP | Ours | 3% | 0.30 | 33.4 | 0.63 | | IP | SDXL | 100% | 0.31 | 35.5 | – | | Art Styles | Ours | 7% | 0.29 | 37.5 | 0.53 | | Art Styles | SDXL | 89% | 0.31 | 35.5 | – | Table: Concept erasure results (IP Characters and Art Styles) on SDXL.

2. We response in Claims 3.

3. In training details in Appendix E (Line 887), we state that neutral prompts are randomly selected from the GCC3M dataset. In addition, we use 100 neutral prompts during unlearning. We appreciate your suggestion on the detailed settings of neutral prompts and will include these details in the main text.

Questions:

1. In Appendix H.1, we state that we train 400 steps on an H100 GPU. As replied above, we randomly select 100 GCC3M prompts as neutral prompts.

2. We include one additional ablation study on the number of neutral prompts. Our method does not require a large number of neutral prompts. | # Images | ACC ↓ | CLIP ↑ | FID ↓ | |----------|--------|---------|--------| | 100 | 4% | 0.31 | 35.4 | | 50 | 4% | 0.31 | 39.7 | | 10 | 12% | 0.27 | 48.2 | Table: Ablation on number of neutral prompts.

We would like to thank you again for your thorough review. We are happy to answer any further questions.