Unveiling Concept Attribution in Diffusion Models

We really appreciate your insightful comments. Please see our response below.

Q1: While the paper introduces parameter-level attribution and negative components, these contributions are incremental rather than transformative.

A: We'd like to clarify the contributions of our paper compared to the literature. Previous works on localizing knowledge in diffusion [2,3] focus on coarse-grained components such as layers, thus, may over-generalize and misinterpret their roles. For example, [2] concludes that knowledge is distributed amongst UNet and localized in text-encoder. On the other hand, since our framework focuses on fine-grained components, we confirm the localization hypothesis and show that knowledge is localized in a small number of parameters (Section 6.2).

Secondly, although Shah et al. [34] explore the linear counterfactual estimator, they only apply it to classification models. As discussed in Section 4.2, this approach is inefficient and challenging for diffusion models. Our work leverages first-order approximation, showing that it could accurately approximate the objective (Section 6.1) without expensive computational overhead and help us locate positive and negative components.

Finally, the primary goal of our framework is to identify positive and negative components, which is distinctive from methods that erase knowledge from the model, such as ConceptPrune and UCE. In particular, ConceptPrune removes model parameters based on feature norms, while UCE updates the projection matrix with a closed-form solution. In contrast, by computing the attribution score, we show that there are parameters contributing positively or negatively to a knowledge, and ablating them from the model leads to the inverse effect.

Q2: the core idea of attributing model components builds heavily on prior work like Shah et al. [34] and knowledge localization studies [2, 3, 27]. The main score and formulation used (Eq. 4) is not novel [21,27] as acknowledged by the authors.

A: As discussed in the previous answer, our work offers an advanced methodology compared to [34] and provides novel insights compared to [2,3,27].

Although prior works have explored Eq. 4, they directly fine-tune the model to minimize that objective. In contrast, we show that Eq. 4 could be used to identify positive components, which is interesting and hasn't been observed. This property facilitates the proof of the localization hypothesis, which is the main contribution of our paper.

Q3: The discovery of negative components, while interesting, lacks sufficient exploration.

A: We'd like to highlight that the primary objective of our work is seeking a principled framework to answer the question of "how do components in diffusion models contribute to a generated concept?", and the "discovery of negative component" is a significant finding, but not a focus, of the paper. Specifically, we propose the first-order, inexpensive approximation to locate important components (specifically, parameters -- the most fine-grained components), confirm the localization hypothesis, and empirically validate the utility of these components; these contributions are novel and important in understanding diffusion models, and distinct from existing interpretability papers.

We agree with the reviewer that the intensive exploration of negative components would be important, but it is beyond the already extensive scope of our work. For example, one can study how to fine-tune these negative components for better editing results, which can be, by itself, a self-contained work. We thus leave these explorations for future work.

Q4: Weak Theoretical Foundation.

A: In our study, we only focus on a small number of parameters, and ablating them induces an insignificant change in the input $w$ of the objective $J(c,w)$. As discussed in Section 4.2, this assumption makes the first-order approximation theoretically accurate, and we empirically validate that in Section 6.1 as well as extensive experiments where we ablate positive and negative components of several concepts on SD-1.4 and SD-2.1.

While theoretical analysis is desirable, both reliance on the well-established first-order approximation (with valid assumptions) and rigorous empirical confirmation of the approximation would strongly justify the soundness of our framework. Coupled with the extensive empirical observation, the framework is expected to generalize to other models and concepts. Nevertheless, we would be happy to address any specific suggestions the reviewer may have regarding the theoretical analysis that the reviewer believes is necessary.

Q5: Incomplete Analysis of Negative Components.

A: Thank you for your suggestion. As mentioned above, our paper is the first work that points out the existence of negative components and their roles, i.e., ablating them enhances knowledge in diffusion. We believe that theoretically studying their origin during the training process is an interesting direction, yet it deserves a separate intensive study, and we leave it to future work (please also see our response to Q3 on the paper's objective and contributions).

Q6: Narrow Experimental Scope: The evaluation is limited to Stable Diffusion-1.4

A: We'd like to clarify that the experiments in our paper follow standard evaluation in the literature and are extensively conducted not only on SD-1.4 (main paper), but also SD-2.1 (appendix). Meanwhile, existing works also only apply their methods on SD-1.4 [2,11,12,13,19,21,25,43], SD-1.5 [5], and SD-2.1[49]. This makes our evaluation already more extensive than these related works.

Nevertheless, we follow the review's suggestion and report the results on SD-3.5 and SD-XL. We ablate no more than 0.03% of the model's parameters and measure the number of nudity content in images generated from prompts in I2P, Ring-A-Bell, and MMA datasets, as well as CLIP-Score on COCO30k. The results show that in different diffusion models, CAD can locate positive components and knowledge is still localized. In particular, CAD reduces the number of nudity labels by 97.39% and 98.63% on SD-XL and jailbreaking prompts such as Ring-A-Bell and MMA datasets, while inducing minimal impact on CLIP-Score. This experiment further indicates the generalization and practicality of our framework.

Q7: Concept Entanglement Issues.

A: The main contribution of CAD is to confirm the localization hypothesis and the existence and utility of negative components, which is significant and hasn't been explored in prior studies. The effect of different concepts and their entanglement, despite being interesting, is an orthogonal direction to our framework and requires an independent study that is beyond the scope of this work.

Q8: Lack of Statistical Rigor.

A: Our evaluation follows exactly the experimental setup in existing research [5,11,12,13,19,21,25,43,49]. In particular, to evaluate the knowledge of objects and artistic styles, we generate images from several prompts and random seeds provided in the dataset. This evaluation helps reduce the randomness in the result. To evaluate the knowledge of nudity content, we generate with a fixed seed on several datasets (I2P, Ring-A-Bell, MMA) with massive prompts (31285 in total) that assess every aspect of the knowledge. This high number of evaluation prompts provides a reliable verification of our claims.

We also evaluate knowledge of nudity content in the base Stable Diffusion, ConceptPrune, and CAD with 5 different random seeds and provide the mean and standard deviation of the number of nudity labels. The table confirms that ablating positive components found by CAD indeed erases the knowledge from the model.

We are grateful for your helpful comments. Please see our response below.

Q1: No Overhead Analysis

A: We'd like to clarify that CAD as well as other erasing methods edit diffusion models by only updating their weights, and this process is performed a single time only. Therefore, CAD has no impact on inference speed and memory. Furthermore, CAD does not require training, unlike other methods such as ESD; thus, the time and memory overhead for editing is insignificant.

Nevertheless, we follow the review's suggestion and measure the computational cost for editing the model. We conduct experiments on a single A5000 GPU. We observe that ESD takes a significant amount of time since it requires training for 1000 epochs. ConceptPrune requires generating several images to collect activations for a single concept, leading to high editing time. UCE and RECE offer fast editing due to the closed-form update; however, for a certain type of knowledge such as artistic styles, they require additional computation to preserve many unrelated knowledge. Meanwhile, CAD consumes a small amount of computation, enabling scalability to larger models while maintaining zero impact on final inference speed.

Q2: Experiments are confined to Stable Diffusion 1.4(mostly) and 2.1, leaving its effectiveness on other architectures (like SDXL, SD3) unproven. How does the "localization hypothesis" generalize to newer architectures like SD3 or FLUX? We suggest adding theoretical reasoning or a preliminary analysis.

A: We'd like to note that our experimental setup follows existing works that apply their methods on SD-1.4 [2,11,12,13,19,21,25,43], SD-1.5 [5], and SD-2.1 [49], making our evaluation quite extensive compared to these works. Nevertheless, we follow the review's suggestion and report the results on SD-3.5 and SD-XL. We ablate no more than 0.03% of the model's parameters and measure the number of nudity content in images generated from prompts in I2P, Ring-A-Bell, and MMA datasets, as well as CLIP-Score on COCO30k. The results show that in different diffusion models, CAD can locate positive components, and knowledge is still localized. In particular, CAD reduces the number of nudity labels by 97.39% and 98.63% on SD-XL and jailbreaking prompts such as Ring-A-Bell and MMA datasets, while inducing minimal impact of CLIP-Score. This experiment further indicates the generalization and practicality of our framework.

Regarding the theoretical reasoning of the localization hypothesis, we believe that it could be a result of the training process on over-parameterized models. Theoretically studying its origin is an interesting direction, yet it deserves a separate intensive study, and we leave it to future work.

Q3: The approach of ablating parameters by setting them to zero is a blunt instrument. While effective, it might not be optimal. Could the method of directly setting ablating parameters to zero be optimized other than finetuning?

A: Thank you for the interesting suggestion. Our work aims to answer the research question "how do components in diffusion models contribute to a generated concept?" -- thus, we focus on directly ablating the identified components as a means to validate their roles. More specifically, CAD can locate positive and negative components of a knowledge, confirmed by their ability to "edit" knowledge in diffusion models via ablation. Importantly, the implications of our work go beyond ablation; for example, as you suggested, existing finetuning unlearning approaches could potentially be combined with these components to further improve the unlearning ability. We leave these promising applications as future extensions of our work.

Thank you for your insightful feedback. Please see our response below.

Q1: Figure 1 does not effectively illustrate the method itself; it primarily presents a conclusion (attribution scores), offering limited help in understanding the CAD procedure.

A: Fig. 1 illustrates the "problem statement" of our work: there exist negative and positive components for each concept, where their ablations result in "increasing" or "decreasing", respectively, the presence of that concept in the generated image. Thus, it does not illustrate the specific technique used to identify these components; please note that the specific proposed algorithms are detailed in Algorithms 1 and 2. We believe this division is clearer to highlight the studied research problem and the proposed algorithmic solution.

Q2: Experimental validation is notably limited.

Thank you for the comment. However, we'd appreciate it if the reviewer could elaborate the "limited experimental validation", as it'd better help us to address your concerns.

In the paper, we conduct experiments on SD-1.4 and SD-2.1, 10 objects, 5 artist styles, nudity content, 2 black-box attacks, and 2 white-box attacks. Our experimental validation is extensive, following the setting in prior research [5,11,12,13,19,21,25,43].

Q3: Results lack robustness: Table 1 fails to highlight superior metrics effectively (e.g., via bolding).

A: We will highlight the best performance method in Table 1 in the final version. We'd also like to note that the main goal of our work is to study the localization hypothesis in diffusion models, rather than proposing a state-of-the-art knowledge erasing method. Table 1 along with other empirical evidence in the paper successfully verify that hypothesis by showing that ablating a small number of parameters removes the target concept while keeping other concepts intact. We believe that future research could utilize our findings to improve knowledge erasing algorithms.

Q4: Evaluation is primarily conducted on low-resolution images.

A: We'd like to clarify that we generate images using the standard resolution of Stable Diffusion models, including 512x512 and 768x768 for SD-1.4 and SD-2.1, respectively. These are high resolutions and have also been used in several related studies and benchmarks [5,11,12,13,19,21,25,43,49]. Moreover, we also conduct experiments with image size of 1024x1024 on SD-3.5 and SD-XL below, further confirming the utility of our framework.

Q5: Model Scope Limitation: Only Stable Diffusion 1.4 (SD v1.4) is tested

A: We'd like to clarify that the experiments in our paper are extensively conducted not only on SD-1.4 (main paper), but also SD-2.1 (appendix), which is mentioned in the reviewer's comment. Meanwhile, existing works also only apply their methods on SD-1.4 [2,11,12,13,19,21,25,43], SD-1.5 [5], and SD-2.1[49].

We believe these experiments are sufficiently comprehensive to support our claim and demonstrate the utility of our framework.

Nevertheless, we follow the review's suggestion and report the results on SD-3.5 and SD-XL. We ablate no more than 0.03% of the model's parameters and measure the number of nudity content in images generated from prompts in I2P, Ring-A-Bell, and MMA datasets, as well as CLIP-Score on COCO30k. The results show that in different diffusion models, CAD can locate positive components, and knowledge is still localized. In particular, CAD reduces the number of nudity labels by 97.39% and 98.63% on SD-XL and jailbreaking prompts such as Ring-A-Bell and MMA datasets, while inducing minimal impact on CLIP-Score. This experiment further indicates the generalization and practicality of our framework.

Q6: Qualitative results are weak. For instance, Figure 3 (rows 1-2) shows that concept "erase" significantly alters background information, indicating the method lacks practical advantage and suffers from unintended side effects.

A: We'd first like to clarify that the qualitative results are intended to demonstrate the successful removal of the concepts, while showing minimal interference with other concepts. Extensive qualitative results on SD-1.4 (main paper) and SD-2.1 (appendix) verify this property.

Second, as the diffusion process involves multiple sampling steps, slight variations in the generated images (or their backgrounds) are expected in multiple runs (those with or without ablations). This behavior is consistent across all methods, where the backgrounds change in multiple generations. Nevertheless, the key focus is the successful removal (or reservation) of the concept.

Finally, the quantitative results, based on well-established evaluation processes, clearly demonstrate the performance of the baselines and CAD. In other words, the combination of qualitative and quantitative results consistently highlights the effectiveness of CAD.

Q7: Limited Methodological Novelty: The core attribution technique (perturbing parameters and measuring output change via a classifier) is fundamentally simple and bears strong resemblance to well-established gradient/importance-based attribution methods developed over the past decade (e.g., CAM and its numerous derivatives). The novelty contribution is therefore significantly lacking.

A: As discussed in the Related works, our paper attributes model components, which is fundamentally different from prior works such as CAM that attribute the input features. Section 4.2 also highlights the significant challenge in model (component) attribution for diffusion (Section 4.2); our proposed algorithm offers a significantly efficient solution to this problem, making it a valuable contribution to the literature.

More importantly, the primary contribution of this work extends beyond the efficient solution itself -- it provides significant insights into the role of model components in diffusion models. Through CAD, we confirm the localization hypothesis (Section 6.2) and reveal the existence of negative components (Section 6.3), which are missing in prior works.

Q8: Misleading Terminology

A: In this work, we refer to "components" as grouped model parameters [34], such as convolutional filters, attention heads, layers, etc. This definition allows us to work with different levels of granularity, from coarse-grained components such as layers in existing works [2,3] or fine-grained components such as parameters in our paper. Regarding the use of "concept", we follow the widely accepted definition that has been used in prior works [5,11,12,13,19,21,25,43,49].

Thanks for the detailed reply. Most of my concerns have been resolved. However, I still believe the novelty is somewhat limited. Indeed, CAM modifies input features, this paper modifies 'components'. However, both input and parameters are simply nodes on the computational graph. And there are many works have also investigated the change of parameters. On the other hand, I admit that the authors did put effort on empirical evaluations. Maybe there are researchers that can benefit from the results of this work. Considering all the facts, I decide to raise my final score to 4 but not against for rejection.

We are grateful for your insightful comments and for appreciating the effectiveness of our method and the importance of our findings. Please see our response below.

Q1: The writing quality is a bit poorer than typical for a NeurIPS submission.

A: Thank you for your comment. We will update the writing in the camera-ready version according to the discussion during the rebuttal.

Q2: Design decisions are not very well justified, relative to alternatives. For example, the authors make clear that the first-order Taylor series expansion enables an efficient estimation but do not comment on why their chosen approach is likely to do a better job of this than alternatives. More explanation of the reasons for design decisions would improve the paper.

A: Thank you for your comment. The alternatives are actually impractical for diffusion models, as discussed in Section 4.2, due to the data construction process. In particular, creating one training sample for the linear estimator requires computing the objective function on the masked model, and we need a high number of samples to train an accurate estimator. For example, [34] constructs 100,000 samples for ResNet50 with 25M parameters; with 859M parameters of SD-1.4, the computational overhead of training the linear estimator becomes extremely high, if not impractical.

Furthermore, both CAD and the linear estimator approach are an approximation of the true objective function, based on the linearity assumption. Section 6.1 shows that CAD yields a high correlation and can approximate the true objective with high accuracy.

For these reasons, CAD is a much more desirable choice than the existing alternatives.

Q3: If the contributions were revised to focus on the specific novel contributions of the paper, what would they look like?

Do you claim to be the first to discover that a very small percent of DNN components have nearly all the influence over the expression of a concept?

Is the key novel idea for the counterfactual predictor to use the Taylor series expansion to enable efficient prediction, or is there another primary novel contribution related to the predictor?

Do you claim to be the first to find that there are some components for which higher activations repress a concept?

Are there other novel ideas or discoveries claimed?

A: We'd like to clarify our contributions as follows. First, we propose a framework that helps identify the contribution of model components to generated data. Although the idea of using counterfactual estimator and Taylor approximation has been explored in classification and language models [34,38], applying them to diffusion is challenging due to the reverse process. Our work shows that we can design a suitable objective function such that the approximation is accurate and becomes beneficial for the study in next sections.

Secondly, we utilize our framework to confirm the localization hypothesis. Prior study focuses on coarse-grained components such as layers and conclude that knowledge is distributed amongst UNet and localized in the text encoder. In contrast, we study fine-grained components and pinpoint a small number of parameters that highly contribute to a knowledge, indicating that knowledge is localized.

Finally, we are the first that observe and verify the existence of negative components in diffusion models. These components contribute negatively to a knowledge; in particular, ablating them leads to high generation probability.