Emergence and Evolution of Interpretable Concepts in Diffusion Models

We appreciate the reviewer’s time and thoughtful feedback. We're especially glad that the reviewer found the early emergence of the final composition of the scene surprising and noteworthy. Please find below our point-by-point responses to the reviewers’ comments.

• Re weakness 1: We thank the reviewer for this helpful suggestion. We agree the background should better support readers unfamiliar with SAEs and diffusion. We will expand it and relocate definitions (e.g., classifier-free guidance, encoder/decoder) from the methods section as appropriate.

• Re weakness 2: We thank the reviewer for raising this question. As state-of-the-art image diffusion pipelines (e.g., Stable Diffusion, Imagen, DALL·E) rely on a multi-step denoising process to synthesize images, we believe it is crucial to understand how semantic information emerges and evolves over these steps. Our claim that temporal analysis is critical stems from a broader goal: to demystify why the diffusion process is so effective as a generative modeling framework. Specifically, breaking down the mapping from noise to natural images into a sequence of intermediate steps is a deliberate design choice. Hence, understanding what is gained from this decomposition can help answer fundamental questions: Are there early commitments to layout or object identity? When does texture or fine detail emerge? Which steps are most causally influential in determining the final output? By analyzing the time-evolution of interpretable concepts, our work sheds light on these dynamics. Such insights cannot be recovered from single-step or few-step diffusion models that bypass these. We believe such understanding is essential not only for interpreting current models, but also for guiding the development of more efficient, controllable, or interpretable generative systems. We will revise the manuscript to better articulate this motivation and clearly differentiate our approach from prior work focused on single-step analyses.

  Regarding the number of forwards steps: we collect the activations across 50 DDIM steps. We will clarify this in the revised manuscript.

• Re weakness 3: Thank you for pointing this out. Collecting activations for millions of text prompts is costly time and storage wise. We did our best to perform a comprehensive set of experiments on a typical and popular architecture and provide the first SAE based temporal analysis of the evolution of interpretable concepts. In future work, we are interested in exploring further architectures, including Diffusion Transformers, where we expect our causal interventions to be even more effective.

• Re weakness 4: We would like to first clarify that we do not report stronger results than the existing work that utilize vision-language models. In fact, our work is aimed at analyzing the time-evolution of concepts during the diffusion process, and is not a particular method where concrete numerical results of performance can be reported. We acknowledge that vision-language models can be also leveraged for concept discovery, however our design choice is to opt for an alternative approach due to the differences highlighted in our work. Replacing the concept discovery pipeline in our framework with vision-language models is not straightforward, as it is unclear how to build the concept dictionary in this case, and thus it is not a simple ablation study we could perform. With respect to the usage of Word2Vec embeddings: we agree with the reviewer that richer semantic embeddings could be used. A key benefit of our pipeline is its modularity: we can replace individual components of the annotation pipeline, or the semantic embeddings used to generate the concept dictionary. We used Word2Vec embeddings for the sake of simplicity. In fact, we believe that using embeddings from LLMs would be an interesting direction to further improve the granularity of our analysis.

• Re weakness 5: We understand the reviewers’ concerns. We conducted two sets of experiments to bolster our claims quantitatively. First, we run experiments to quantitatively measure the purity and separation of the concepts that we discover and how it temporally evolves across diffusion iterates. We observe that as we progress along the reverse diffusion process, concepts in the dictionary cluster around the mean better (increased concept cohesion) and that the concepts are better separated (more distinct). We refer the reader to our response to “question 1 and weakness 1” of Reviewer HyPM for the full experimental details and results table.

  Next, to evaluate the effectiveness of our intervention method, we introduce a proxy metric based on CLIP similarity. Specifically, we compute the CLIP distance between a generated image and the mean CLIP embedding of the target concept (obtained by averaging embeddings of all associated dictionary terms). An edit is considered successful if this similarity increases after intervention. We conducted this experiment on 100 prompts from the validation set and applied our editing method at three timesteps (early, middle, late), focusing on concept indices corresponding to cartoon, sea & sand, and painting (as shown in Figures 4, 5, and 6).

  Results are as follows:

  Timestep	Concept ID	Before Edit	After Edit	Δ (After - Before)	Edit Success
  t = 0.0	1722	0.198	0.206	+0.008	0.82
  	2349	0.190	0.193	+0.003	0.69
  	3593	0.198	0.204	+0.006	0.79
  Average		0.195	0.201	+0.006	0.77
  t = 0.5	1722	0.198	0.222	+0.024	0.97
  	524	0.190	0.220	+0.030	0.95
  	2137	0.198	0.219	+0.021	0.93
  Average		0.195	0.220	+0.025	0.95
  t = 1.0	1722	0.198	0.211	+0.013	0.84
  	2366	0.190	0.193	+0.003	0.56
  	2929	0.198	0.211	+0.013	0.78
  Average		0.195	0.205	+0.010	0.73

  We find that edits are most successful at the middle timestep (t = 0.5), with a 95% success rate and the highest average ΔCLIP similarity (+0.025), compared to 77% (+0.006) and 73% (+0.010) at early and late stages, respectively. This aligns well with our qualitative observations and supports our main insight: intervening during the middle stages of diffusion offers the most effective control over image style while preserving content.

  Regarding the IoU scores at $t = 0$, an IoU of $0.26$ is not particularly high for a dedicated segmentation model, however our SAE has not been trained on any objective even remotely related to image segmentation, and thus the results are non-trivial. In fact, some zero-shot segmentation methods report comparable scores (albeit on a different dataset) [1]. We view this not as a shortcoming of our method (as it is not a segmentation model), but rather as an insight into the diffusion process itself: while a coarse layout (hence fairly low IoU) can already be predicted at $t = 1.0$ (also depicted in Figure 1), the final image composition is not yet fully determined at this early stage.

  [1] Lüddecke, Timo, and Alexander Ecker. "Image segmentation using text and image prompts." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

• Re weakness 6: Related work in Section 2 overviews some prior works in a similar spirit to ours, however none of them are directly comparable to ours due to the lack of focus on time-dependency. Furthermore, we would like to emphasize again that we do not present this work as a competitive controlled generation framework, but as a tool to better understand diffusion as a phenomenon. Our aim is to demystify the evolution of concepts. Being able to causally intervene in an interpretable way is a side-effect of the framework.

• Re weakness 7: We pick the binarization threshold of 0.1 to filter out low-magnitude, noisy activations that we observe on most samples. We haven’t tuned this parameter extensively, however it is possible that further tuning could further improve the purity and localization of discovered concepts.

  We determine the intervention strengths for both global and spatially targeted interventions by qualitatively inspecting generated images across a range of values (see Figure 6). The higher variance in the spatial intervention strengths reported in Table 2 likely stems from slight differences in normalization. For example, using strengths significantly above 400 (e.g., 1000 or 4000) during mid-stage spatial interventions often introduced visual artifacts.

• Re weakness 8: We do mention two key limitations of the current work: first the discovered concepts and their spatial resolution is upper bounded by the accuracy/segmentation capability of the zero-shot pipeline we use. In fact, we visually observe that the image segmentation predicted from our concepts close to t=1.0 are more accurate than the segmentation maps provided by the zero-shot pipeline! Second, due to skip connections in the architecture, there is a “leakage” effect when attempting to intervene on specific concepts. We believe that this can be addressed by exploring further architectures such as Diffusion Transformers.

We appreciate your time and are happy to engage further should any questions remain during the discussion phase.

Hello, a gentle reminder for discussions. Authors have provided a rebuttal, and it would be great if you could acknowledge it and comment whether it addresses your concerns!

Lack of justification for temporal analysis.

• Regarding the initial review. You are claiming that per-step analysis is « critical ». In your response, I do not see what kind of new and critical insight the temporal analysis brings you, compared to a single-step analysis. Hence, I do not feel like my question has been answered. For example, you claim that the concepts can already be predicted at $t=1$. Since one-step diffusion models map noise ($t=1$) to data, shouldn't they also have the same property?

• You mention that your results help to devise better « more efficient, controllable or interpretable generative models ». Which part of this work would you say can be used to improve the efficiency, controllability or interpretability of future generative models?

Limited model scope an generalizability:

We expect our causal interventions to be even more effective with diffusion transformers

• Why would your causal interventions be even more effective than for a U-net?
• I still believe that analyzing a single model is too little. I understand that the cost is high, however, you claim « collecting activations for millions of text prompts ». Given the large number of text prompts collected, I assume you could have split them in 3, and assigned them to 3 different models, instead of allocating them to a single one. This should have retained the overall collection cost similar.
• The fact that prior work focuses on single-step generation does not make it unsuited for comparison

Interpretation pipeline

• On lines 111-113, you claim that previous work « struggles with scale and inherits biases and hallucinations of the foundation models », while your method is a «  scalable pipeline that builds a flexible concept dictionary ». The meaning is relatively clear: you claim your method is more scalable, yet this is not tested, as noted in my original review.

For these reasons, I still believe my original review is fair, and currently intend to retain my score.

Thank you for taking the time to review our manuscript and provide valuable feedback. We are glad to hear that the reviewer appreciates our analysis on interventions, found it easy to follow the paper and that the reviewer believes our examples validate the proposed approach. Please find below our responses to the concerns.

• Re question 1: With respect to novel concepts, by construction, our framework can discover concepts based on natural language annotations of image regions. Thus, given such annotations for a novel concept, we can incorporate it into the concept dictionary in the first step of our method by observing which features fire for the given annotated samples. With respect to the granularity of discovered concepts, we refer the reviewer to figures 14,15,16 in the Appendix where we provide snippets from our concept dictionaries. Based on the recommendation of Reviewer HyPM, we run experiments to quantitatively measure the purity and separation of the concepts that we discover and how it temporally evolves across diffusion iterates. We observe that the objects in the dictionary cluster around the concept center better (increased concept cohesion) and that the concepts are separated better (more distinct) as we progress along the reverse diffusion iterates. We refer the reader to our response to “question 1 and weakness 1” of Reviewer HyPM for the full experimental details and results table.
• Re question 2 and weakness 1: We are unaware of prior work that highlights the presence of high-level concepts in the early stages of diffusion with the granularity that our technique provides, namely decoding the final generated image layout before the first diffusion step. Even though there is existing work that shows that image layout can be manipulated in early timesteps, our findings are stronger: we can localize and label the concepts, which is a key difference. Moreover, our SAE-based technique provides a more structured analysis on the emergence of visual features and their granularity that provides further explanation and corroboration of phenomena observed empirically in existing work.
• Re question 3 and weakness 3: We would like to clarify that we do not manually inspect the diffusion timesteps to select an intervention time. A priori, we split the diffusion iterates into 3 stages: early (t ∈ [0.6, 1.0]), middle (t ∈ [0.2, 0.6]) and final (t ∈ [0, 0.2]). We use these fixed intervals for all of our experiments and in Figures 4,5,6.
• Re questions 4: We define the image style of a concept in the dictionary as the shared visual characteristic among its top-activating examples from the dataset. Indeed, if the shared characteristics are low-level (such as texture), we define the style accordingly.
• Re question 5: One can still examine the top-activating examples from the dataset for a given concept to get a sense of its visual characteristics without needing to assign a specific label (e.g., "anime style" or "cartoon style"). In this way, domain knowledge is not required to visually compare a candidate image to the set of images associated with a concept. That said, domain knowledge may be necessary to accurately name or describe the style represented by that set.
• Re weakness 2: Great question. We use the set of prompts from the LAION-COCO dataset to generate images, which are then annotated by our zero-shot pipeline, thus we are not leveraging LAION-COCO images directly at all when building the concept dictionary. As we have explained earlier, given samples of a novel concept and annotated image regions, we can incorporate such concepts into the concept dictionary.
• Re weakness 4: Great point. We hypothesize that early stage concepts do not have enough granularity to encode different dog breeds as separate concepts. Instead, a unified “dog-ness” is encoded in the concept dictionary. Therefore, manipulations to such a concept do indeed change the location of the “dog” to the target spatial location but cannot give guarantees to preserve high granularity details.
• Re weakness 5: We refer to our interventions in Section 3.5 as causal because our goal is to verify that the discovered concepts have a visual impact when we manipulate them through targeted interventions. We apply slightly different normalizations for spatially targeted and global interventions. We will clarify this better in the revised manuscript.

Please let us know if you have any unanswered questions or concerns with respect to our work. We are happy to discuss further in the discussion period.

Hello, a gentle reminder for discussions. Authors have provided a rebuttal, and it would be great if you could acknowledge it and comment whether it addresses your concerns!

We appreciate the reviewer’s time and thoughtful feedback on our manuscript. We're glad the reviewer found the use of SAEs to analyze the diffusion process “interesting” and that the experiments demonstrate the effectiveness of our method.

• Re weakness 1: As we have explained in our response to Reviewer Pskz, we are unaware of prior work that highlights the presence of high-level concepts in the early stages of diffusion with the granularity that our technique provides, namely decoding the final generated image layout before the first diffusion step. Even though there is existing work that shows that image layout can be manipulated in early timesteps, our findings are stronger: we can localize and label the concepts, which is a key difference. Moreover, our SAE-based technique provides a more structured analysis on the emergence of visual features and their granularity that provides further explanation and corroboration of phenomena observed empirically in existing work.

• Re weakness 2: We appreciate the reviewer’s suggestion and agree that visualizing generated images across timesteps can offer insight into the evolution of image structure during the diffusion process. However, we would like to clarify an important distinction: our work uses a latent diffusion model (LDM), in which the diffusion process occurs in a latent space rather than pixel space. As shown in Figure 1 of our paper, the latent representations at early timesteps (e.g., t = 0.0) resemble noise and do not convey any obvious spatial composition when visualized via the decoder's posterior mean. This makes it much harder—if not impossible—to directly infer spatial structure at these stages using standard visualization. Even if it is possible to obtain low-quality representation of the diffusion model’s output at different time steps, it is unclear how to leverage them for fine-grained concept discovery, as out-of-the-box annotation tools typically only work on clean images.

• Re weakness 3: We thank the reviewer for highlighting the importance of including quantitative results to support our qualitative findings. In response (echoing suggestions from other reviewers as well) we conducted two sets of new experiments to strengthen our analysis.

  Quantitative Characterization of Concept Dictionaries

  To quantify the structure of concept representations discovered at different timesteps, we define two metrics: concept cohesion and concept separability.

  • Concept cohesion measures how tightly clustered the items in a concept dictionary are around their respective concept centers.
  • Concept separability quantifies how distinct the concepts are from each other.

  To compute cohesion, we take the mean Word2Vec embedding of the dictionary items for each concept, then calculate the average cosine distance between individual items and the concept center. The final cohesion score is the average of these values across all concepts.

  For separability, we compute the pairwise cosine similarity matrix of the concept centers (shape $n \times n$ where $n$ is the number of concepts). We then report the average of the off-diagonal similarities, representing the average similarity between different concepts. The results for each timestep are summarized below:

  t	concept cohesion	concept separability
  0	0.664	0.344
  0.5	0.639	0.363
  1.0	0.588	0.433

  We observe that, as we move along the reverse diffusion process (from $t=1.0$ to $t=0.0$), concepts become more coherent and less similar to each other—indicating that they become purer and more distinguishable (that is more specific) as generation progresses toward the clean data distribution. These results support our interpretation of the evolving semantic structure of concepts over time.

  Quantitative Evaluation of Editing Performance

  To evaluate the effectiveness of our intervention method, we introduce a proxy metric based on CLIP similarity. Specifically, we compute the CLIP distance between a generated image and the mean CLIP embedding of the target concept (obtained by averaging embeddings of all associated dictionary terms). An edit is considered successful if this similarity increases after intervention.

  We conducted this experiment on 100 prompts from the validation set and applied our editing method at three timesteps (early, middle, late), focusing on concept indices corresponding to cartoon, sea & sand, and painting (as shown in Figures 4, 5, and 6). Results are as follows:

  Timestep	Concept ID	Before Edit	After Edit	Δ (After - Before)	Edit Success
  t = 0.0	1722	0.198	0.206	+0.008	0.82
  	2349	0.190	0.193	+0.003	0.69
  	3593	0.198	0.204	+0.006	0.79
  Average		0.195	0.201	+0.006	0.77
  t = 0.5	1722	0.198	0.222	+0.024	0.97
  	524	0.190	0.220	+0.030	0.95
  	2137	0.198	0.219	+0.021	0.93
  Average		0.195	0.220	+0.025	0.95
  t = 1.0	1722	0.198	0.211	+0.013	0.84
  	2366	0.190	0.193	+0.003	0.56
  	2929	0.198	0.211	+0.013	0.78
  Average		0.195	0.205	+0.010	0.73

  We find that edits are most successful at the middle timestep (t = 0.5), with a 95% success rate and the highest average ΔCLIP similarity (+0.025), compared to 77% (+0.006) and 73% (+0.010) at early and late stages, respectively. This aligns well with our qualitative observations and supports our main insight: intervening during the middle stages of diffusion offers the most effective control over image style while preserving content. Moreover, we would like to emphasize that our framework is not specifically designed for image editing. Our work is aimed at uncovering the evolution of interpretable concepts in the representation space of the diffusion model, incorporating analysis of interventions to demonstrate the causal effect of manipulating concepts on the model’s output. That said, our framework opens the door for extremely fine-grained image manipulation through natural language instructions. In particular, the concept dictionary can be viewed as a library of image editing knobs expressed in natural language. As an exciting future direction, an LLM can act as a controller that maps editing instructions from natural language to interventions based on the concept dictionary (similarly how a human would browse the concept dictionary to implement their edit).

  We thank the reviewer for prompting these additions and will incorporate both sets of results into the revised manuscript to clarify and reinforce our contributions.

If there are any aspects of our response that remain unclear, we welcome further discussion and would be glad to address them.

Hello, a gentle reminder for discussions. Authors have provided a rebuttal, and it would be great if you could acknowledge it and comment whether it addresses your concerns!

Thank you to the authors for their response. However, several of my core concerns remain unresolved, as summarized below. I believe there is still a considerable gap between the current version and the quality expected for publication. Therefore, I will maintain my original score.

1. Lack of critical new findings compared to prior studies

I appreciate the authors' effort in discussing the difference from [1] in the rebuttal, and I noted their claim that the work highlights "decoding the final generated image layout before the first diffusion step". However, I remain unconvinced that this is a critical finding for highlighting the first diffusion step, instead of early denosing steps in prior studies[1,2,3]. In fact, this work itself performs interventions across a range of early steps (see line 275), which further weakens the argument that the first step is uniquely important.

The temporal analysis of diffusion models has already been widely discussed prior studies, including but not limited to the works listed below. For instance, Prompt-to-Prompt (with over 2,000 citations) shows in Figure 4 that object layouts is decided in early timesteps. SDEdit [3] discusses how selecting different starting timesteps $t$ influences image layout and style. Another work [4] investigates how the editing strength varies with different diffusion timeeps.

[1] Perception Prioritized Training of Diffusion Models, CVPR 2022

[2] Prompt-to-Prompt: Image Editing with Cross Attention Control

[3] SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, ICLR 2022

[4] Diffusion Models Already Have a Semantic Latent Space, ICLR 2023

2. Lack of systematic and solid evaluations

I share the concerns raised by Reviewer HyPM and Reviewer ZsBz regarding the evaluation. I appreciate the authors' effort in answering this point, however, my concerns remain. Although this work claims the focus is to show the phenomenon, more comprehensive and quantitative evaluations on larger and more diverse datasets are essential to support the conclusions.

Some of my specific questions were not fully addressed. For example, my earlier concern that “it is not convincing enough to define an edit as “successful if this similarity increases after intervention.” Could a trivial increase qualify as success? ” was not answered. In addition, my question regarding the quality of the generated images also remains unresolved. Specifically, I was asking for an evaluation of image fidelity. Moreover, the use of only 100 prompts for evaluation is insufficient to convincingly demonstrate the robustness or generality of the proposed intervention method.

Considering these points, I will maintain my original score. I sincerely thank the authors for their efforts in responding to my questions, and I hope these comments will be helpful in improving the work.

Thanks to the authors for the response.

Regarding point 1: I understand that this work does not specifically focus on editing. However, the reason I mentioned related work[1,2,3,4] is to highlight that prior studies have already analyzed and drawn important conclusions about the temporal behavior of diffusion models. These studies make the findings of temporal analysis in this work less surprising and critical. I quote my previous discussions on these related work as follows.

I appreciate the authors' effort in discussing the difference from [1] in the rebuttal, and I noted their claim that the work highlights "decoding the final generated image layout before the first diffusion step". However, I remain unconvinced that this is a critical finding for highlighting the first diffusion step, instead of early denosing steps in prior studies[1,2,3]. In fact, this work itself performs interventions across a range of early steps (see line 275), which further weakens the argument that the first step is uniquely important.

The temporal analysis of diffusion models has already been widely discussed prior studies, including but not limited to the works listed below.

Regarding point 2: Thanks for providing numbers with 1000 prompts. However, my previous questions are still not answered:

For example, my earlier concern that “it is not convincing enough to define an edit as “successful if this similarity increases after intervention.” Could a trivial increase qualify as success? ” was not answered.

We appreciate the time and effort the reviewer put into reviewing our manuscript. We are glad to hear that the reviewer thinks our work addresses “an important and increasingly relevant problem” and that the paper “stands out by emphasizing the temporal evolution of representations”. Please find below our responses to the concerns and questions raised by the reviewer.

• Re question 1 and weakness 1: We thank the reviewer for raising this important question. We agree with the reviewer that a quantitative and temporal analysis of concept purity would be beneficial to understand how much each SAE feature aligns with a particular concept beyond the qualitative results we have. To this end, we introduce two metrics: concept cohesion and concept separability. We refer to concept cohesion as a measure of how tightly clustered the items in the concept dictionary are around their concept center. Similarly, we calculate concept separability as a measure of how distinct concepts are from one another. Next, we talk about in detail how we calculate these scores. To calculate concept cohesion, for each concept, we calculate the mean word2vec embedding of the items in the concept dictionary. Then, for each item in the dictionary entry, we calculate the cosine distance to the mean embedding and take the average. The final score is calculated by taking the average across concepts. As for the concept separability, we first calculate the concept centers as described before. Note that, we have an $n \times d$ tensor where $n$ is the number of concepts in our dictionary and $d$ is the word2vec embedding dimension. We then calculate the pairwise cosine similarity matrix (of shape $n \times n$). Then we denote the final score as the average of non-diagonal entries. We calculate these for our concept dictionaries for each timestep (t=0.0, 0.5, 1.0). The results are presented in the following tables:

  t	concept cohesion	concept separability
  0	0.664	0.344
  0.5	0.639	0.363
  1.0	0.588	0.433

  We observe that as we move along the reverse diffusion iterates (t=1.0 to t=0.0), the concepts that we have identified become purer (as measured by the increase in concept cohesion from 0.588 to 0.664) Moreover, average pair-wise similarity of concepts decrease from 0.433 to 0.344. This indicates that concepts are becoming more distinct and separated as generation progresses, in other words more specific. We thank the reviewer for the suggestion. We believe that these numbers complement our analysis well and we will incorporate these results in the revised manuscript. With respect to the superposition hypothesis, the high explained variance we observe during training combined with the enforced sparsity underpins the existence of an overcomplete basis where model activations can be decomposed into a few high-dimensional feature vectors. Our framework then labels these feature vectors (or concept vectors) with human-interpretable concepts.

• Re question 2 and weakness 2: Great point. We agree with the reviewer that analysis on more models would be beneficial. Collecting activations for millions of text prompts is costly time and storage wise. We did our best to perform a comprehensive set of experiments on the Stable Diffusion v1.4 architecture and provide the first SAE based temporal analysis of the evolution of concepts. We would like to explore other architectures such as FLUX in future work.

• Re question 3 and weakness 2: Let us clarify the main takeaways from our experiments based on the temporal stages:

  • Early stage of diffusion: coarse image composition emerges as early as during the very first diffusion step. At this stage, we are able to approximately identify where prominent objects will be placed in the final generated image. Moreover, image composition is still subject to change: we can manipulate the generated scene by spatially targeted interventions that amplify the desired concept in some regions and dampens it in others. However, we are unable to steer image style at this stage using our global intervention technique. Instead of high-level stylistic edits, these interventions result in major changes in image composition.
  • Middle stage of diffusion: image composition has been finalized at this stage and we are able to predict the location of various objects in the final generated image with high accuracy. Moreover, our spatially targeted intervention technique fails to meaningfully change image composition at this stage. On the other hand, through global interventions we can effectively control image style while preserving image composition, in stark contrast to the early stages.
  • Final stage of diffusion: Image composition can be predicted from internal representations to very high accuracy (empirically, often higher than our pre-trained segmentation pipeline), however manipulating image composition through our spatially localized interventions fails. Our global intervention technique only results in minor textural changes without meaningfully changing image style.

  The practical implications of this work are far-reaching. First, it informs practitioners when to intervene effectively to manipulate the behavior of diffusion models, not only for image editing but also for generated content moderation. For instance, to avoid copyright infringements, interventions on filtering protected artistic style is most effective in the middle timesteps. Second, our framework opens the door for extremely fine-grained image manipulation through natural language instructions. In particular, the concept dictionary is a library of tuning knobs expressed in natural language. As future direction, an LLM can act as a controller that maps editing instructions from natural language to interventions based on the concept dictionary (similarly how a human would browse the concept dictionary to implement their edit).

• Re question 4: We conducted a new experiment to quantitatively evaluate the success of our editing method. Since no ground truth images are available, we adopt a proxy metric: CLIP similarity between the generated image and the mean CLIP embedding of the target concept (computed as the average embedding of all dictionary words associated with that concept). We measure this similarity both before and after the intervention. An edit is considered successful if the similarity increases post-intervention. To perform this analysis, we sample 100 prompts from the validation set and apply our editing method at three different diffusion timesteps (early, middle, late) for the concept indices highlighted in Figures 4, 5, and 6 (these correspond to the concepts cartoon, sea & sand, and painting, respectively). The results are summarized in the following table:

  Timestep	Concept ID	Before Edit	After Edit	Δ (After - Before)	Edit Success
  t = 0.0	1722	0.198	0.206	+0.008	0.82
  	2349	0.190	0.193	+0.003	0.69
  	3593	0.198	0.204	+0.006	0.79
  Average		0.195	0.201	+0.006	0.77
  t = 0.5	1722	0.198	0.222	+0.024	0.97
  	524	0.190	0.220	+0.030	0.95
  	2137	0.198	0.219	+0.021	0.93
  Average		0.195	0.220	+0.025	0.95
  t = 1.0	1722	0.198	0.211	+0.013	0.84
  	2366	0.190	0.193	+0.003	0.56
  	2929	0.198	0.211	+0.013	0.78
  Average		0.195	0.205	+0.010	0.73

  We observe that stylistic edits are most successful when applied in the middle timestep (t = 0.5), achieving a 95% success rate, compared to 77% and 73% at early and late stages, respectively. This is also reflected in the highest mean ΔCLIP similarity (+0.025), compared to +0.006 and +0.010 at the other timesteps. These findings align with our qualitative observations and reinforce our key insight regarding the middle stage of diffusion: global interventions at this point enable effective control over image style. Combined with our additional experiments on concept cohesion and separability, we believe this quantitative evidence strengthens our analysis which we will incorporate into the revised manuscript.

We hope our responses have adequately addressed the reviewer’s concerns. We welcome further discussion during the rebuttal period should any issues remain.

Hello, a gentle reminder for discussions. Authors have provided a rebuttal, and it would be great if you could acknowledge it and comment whether it addresses your concerns!