ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning

Thank you for your review. We answer your concerns below.

• Lack of adequate baselines

Please refer to the common response. While we have tried our best to include as many related works as possible in the revised version of the paper, it is unclear while baselines the revieweer has implied. We would appreciate a more specific review.

Robustness of the method is insufficiently studied.

We acknowledge the reviewer’s observation that recent studies, such as [1,2], have highlighted the vulnerability of machine unlearning methods to adversarial prompts. However, we respectfully disagree with the claim that the robustness of our method has not been sufficiently studied. We have conducted an extensive evaluation of ConceptPrune against two black-box and two white-box adversarial attacks across tasks involving artist, nudity, and object erasure. Our results clearly demonstrate that ConceptPrune exhibits superior adversarial robustness compared to its direct competitors.

[1] Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey, Liu et al, arXiv [2] Adversarial attacks and defenses on text-to-image diffusion models: A survey, Zhang et al, arXiv

We thank the reviewer for their constructive feedback and appreciate the effort put into the review. We address the reviewer's concerns below.

• Limited Comparison with Recent SOTA Methods and Insufficient Discussion on Related Concept Pruning Work

We appreciate the reviewer for bringing these two recent works to our attention. Given the numerous recent state-of-the-art (SOTA) methods under consideration, we propose a categorization of related work based on their applicability to real-world scenarios.

We categorized all baselines based on whether they are training-free or lightweight, as these two are important criteria for a real-world application. We direct the reviewer to the common response and Section 2 of the paper for further details. Based on this, the requested baselines - Scissorhands, SalUn, and AdvUnlearn can be considered indirect competitors, as ConceptPrune enables efficient erasing concepts without the need for fine-tuning or backpropagation, unlike these two methods.

We have updated Table 2 (artist erasure), Figure 2 and Table 3 (nudity erasure), and Table 5 (adversarial attacks) to incorporate the new baselines. We observe that while ConceptPrune outperforms SalUn in adversarial robustness (Table 3 and 5), Scissorhands and AdvUnlearn are more adversarially robust than the two. However, ConceptPrune is more adversarially robust than its direct competitors.

• Optimal Pruning ratio for skilled neurons

We direct the reviewer to Section A.2 in the appendix, where we understand the effect of the pruning ratio on the erasure vs retention trade-off. For our experiments, we pick the pruning ratio that offers a good balance of improved erasure with minimal retention loss.

We have addressed all the comments and concerns raised by the reviewer. We sincerely hope that you consider revising the score accordingly.

We thank the reviewer for their constructive feedback and appreciate the effort put into the review. We address the reviewer's concern below.

• Comparison with AdvUnlearn

We appreciate the reviewer for bringing recent works to our attention. Given the numerous recent state-of-the-art (SOTA) methods under consideration, we propose a categorization of related work based on their applicability to real-world scenarios.

We categorized all baselines based on whether they are training-free or lightweight, as these two are important criteria for a real-world application. We direct the reviewer to the common response and Section 2 of the paper for further details. We point out that AdvUnlearn relies on extensive fine-tuning with adversarial prompts, whereas ConceptPrune eliminates concepts without requiring any fine-tuning or backpropagation. From Table 5, although AdvUnlearn is more robust than other baselines, ConceptPrune is more adversarially robust to its direct competitors.

• Visualizations

We also direct the reader to the appendix where we have provided many visualization examples in the appendix on artist erasure (Figures 5, 6, 7, 8, 9), nudity erasure (Figure 12), and object erasure (Figure 13). We also present qualitative results for white-box and black-box attacks (Figure 10).

• In Table 5, the FID metric is missing

Table 5 (now Table 6) on multi-concept erasure already contains the FID on 3k samples in the COCO dataset to demonstrate that we are comparable to UCE in retaining unrelated concepts.

We have addressed all the comments and concerns raised by the reviewer. We sincerely hope that you consider revising the score accordingly. Thank you for your time.

• Generalization to other architectures

To demonstrate Concept-Prune’s generalization to other architectures, we add more results on SD-XL in Table 13 in the Appendix. We erased different artist styles with UCE for comparison and demonstrated that for both SD 2.0 and SD-XL, ConceptPrune not only generalizes to different architectures but also delivers superior erasure performance.

• Aggregation over timesteps

Thank you for the insightful question, and we sincerely appreciate the reviewer highlighting these interesting papers. As noted in Section 4, our work draws inspiration from the study in DiffPrune [9], which utilizes Taylor expansion at pruned timesteps to estimate weight importance. Their findings reveal that earlier timesteps focus on local features like edges and colors, while later timesteps shift attention to broader content, such as objects and shapes. Similar to [9], [10] also shows that properties as background color, object shape, etc generated in the earlier timesteps are carried forward to later in the denoising trajectory. Since our work primarily addresses local properties such as style, color, and object shape, we focus on removing concept-generating neurons in the earlier timesteps, which contribute the most to salient properties in the image.

To determine $\hat{t}$ in Equation 6, we performed a straightforward grid search over timesteps t=1 to t=15, examining erased images to identify the timestep at which the concept was effectively removed from the majority of images. We add more visualizations in the paper in Section A.2 in the Appendix. We demonstrate that focusing exclusively on neurons too early in the denoising trajectory fails to capture all the neurons responsible for generating the target concept. On the other hand, extending beyond 10 timesteps results in a noticeable degradation of image content and quality, striking a delicate balance between effective concept removal and preserving the overall integrity of the image. Therefore, $\hat{t} = 10$ is an optimal point for concept erasure and good retention.

[9] Structural Pruning for Diffusion Models, Fang et al. [10] The journey, not the destination: How data guides diffusion models, Georgiev, Kristian, et al.

• Style classifiers instead of CLIP-based metrics -

We agree with the reviewer that CLIP-based similarity metrics may be considered redundant factors. Initially, we considered utilizing style classifiers, such as those employed in UnlearnDiffAtk or UnlearnCanvas. However, we found that these classifiers often exhibited biases toward specific styles. For instance, the style classifier in UnlearnDiffAtk demonstrated higher accuracy in identifying "Van Gogh" styles but struggled with accurately classifying "Pablo Picasso" styles. As a result, we opted to rely on CLIP similarities, as they are consistently reported across the related works and baselines considered in our study.

We have addressed all the comments and concerns raised by the reviewer. We sincerely hope that you consider revising the score accordingly.

We thank the reviewer for their constructive feedback and appreciate the effort put into the review. We respond to all comments below.

• The method seems relatively straightforward and may be considered incremental in light of existing works [1,2] that also employ pruning techniques for unlearning in diffusion models. Comparison to more SOTA diffusion model unlearning tasks

We appreciate the reviewer for bringing these two recent works to our attention. Given the numerous recent state-of-the-art (SOTA) methods under consideration, we propose a categorization of related work based on their applicability to real-world scenarios.

We categorized all baselines based on whether they are training-free or lightweight, as these two are important criteria for a real-world application. We direct the reviewer to the common response and Section 2 of the paper for further details. Based on this, Scissorhands and SalUn can be considered indirect competitors, as ConceptPrune enables efficient erasing concepts without the need for fine-tuning or backpropagation, unlike these two methods.

We have updated Table 2 (artist erasure), Figure 2 and Table 3 (nudity erasure), and Table 5 (adversarial attacks) to incorporate the new baselines. We observe that while ConceptPrune outperforms SalUn in adversarial robustness (Table 3 and 5), Scissorhands is more adversarially robust than the two. However, ConceptPrune is more adversarially robust than its direct competitors.

We agree that our work is related to [2] as they also employ a pruning method, but we could not perform any comparisons die to lack of open-source code or models.

• Choice of FFN-2

Our work draws inspiration from several papers that have effectively isolated skilled neurons within FFNs [A-C], particularly [B], which demonstrated that FFNs in LLMs implicitly encode mixture models, allowing specific skills or concepts to be activated as components of these mixtures. While we acknowledge the existence of other studies focusing on concept control via attention layers, a potential reason for the observed difference is that skills may be more disentangled in FFN layers [B]. Consequently, while it is possible to densely modulate attention layers, selective pruning for concept control is most effective in FFN layers, where the disentangled structure makes it easier to isolate and prune specific concepts.

[A] Finding Skill Neurons in Pre-trained Transformer-based Language Models, Wang et al, EMNLP 2022 [B] Emergent Modularity in Pre-trained Transformers, Zhang et al, ACL 2023 [C] Knowledge Neurons in Pretrained Transformers, Dai et al, ACL 2022

In our study, we identify pruning candidates for SD1.4/1.5 through a comprehensive ablation analysis. This analysis evaluates various potential candidates, including FFN-2 (as used in the paper), FFN-1 (the first layer of the FFN), the Cross-Attention Value Matrix (CA-Value), and both text encoders within the pipeline (CLIP). To validate our findings, we repeat this ablation study on SD-XL, confirming that FFN-2 consistently emerges as the optimal pruning candidate for effective concept erasure.

Artist Styles - We report CLIP similarity between the generated image and input prompt for different pruning candidates for 5 artist styles. Lower CLIP similarity indicates better concept erasure.

Thank you to the authors for their thoughtful responses and the revisions made to the manuscript. I acknowledge the substantial effort invested in addressing the feedback. Although it is uncommon for significant modifications to occur during the rebuttal phase, given the current circumstances, this is understood as part of the process. I believe that most of my concerns have been adequately addressed, and the authors have effectively defended the advantages and uniqueness of their proposed method. Additionally, the comprehensive survey and elaboration on recent unlearning methods are commendable. Consequently, I am willing to increase my score to a 5, indicating a borderline reject.

However, a key concern remains that prevents me from assigning a more favorable rating: the practicality of the authors' definition of "light-weight." While ConceptPrune avoids the need for backpropagation during model training, determining how to prune the corresponding weights still requires forward inference. Specifically, evaluating the final 10 timesteps is approximately equivalent to performing 10 epochs of inference on the data to be forgotten. Although backpropagation is avoided, the computational cost of 10 epochs of inference is still significant. If the authors could provide an equitable comparison of the unlearning's running time (and memory cost), encompassing both inference and training (where applicable to methods other than ConceptPrune), this would further emphasize the lightweight advantage of ConceptPrune. I would be willing to further increase my score upon receiving such clarifications.

I look forward to the authors' response.

Thanks for the detailed clarifications. I've correspondingly increased my score to 6 - borderline accept. I encourage the authors to incorporate the updates discussed during the review process into the manuscript.

Furthermore, I recommend that the authors revisit the use of classifier-based evaluation for style removal, as Weakness 5 mentioned. Specifically, the manuscript could benefit from including both CLIP-score-based metrics and classifier-based metrics, accompanied by an in-depth discussion of the results. Such additions would provide a more comprehensive understanding of the proposed approach and further strengthen the manuscript.