Energy-Based Conceptual Diffusion Model

Q4. What distinguishes the generation process described in Equations (12) and (13) from that of a standard conditional diffusion model? It seems the only change is replacing the conditioning input y with the processed conditioning input $\mathbf{c}$ obtained from $\mathbf{y}$.

This is a good question. After training with the joint energy-based objective, our model performs concept-based joint generation by minimizing the joint energy. As clarified in W2.1, this joint sampling process can be further simplified for computational efficiency and ease of implementation.

Note that Eqn. (12) and (13) are only part of the generation process. As mentioned in the response to W2.1 above, in practice, one can alternate between interpretation $p(\mathbf{c}|\mathbf{x})$ (using Eqn. (15) or (17) in the paper) and generation $p(\mathbf{c},\mathbf{x}|\mathbf{y})$ (using Eqns. (11)-(13) in the paper) until convergence. This is the key difference between standard conditional diffusion models and our ECDM's generation.

The key distinction between our ECDM's formulation (Eqn. (6), (10)-(13), (17)-(19)) and a standard conditional diffusion model is that the variable $\mathbf{c}$ in ECDM is not merely a processed condition.

Note that concept-based generation is only a small part of our ECDM's contribution. Our ECDM goes far beyond generation and unifies five different tasks, i.e., concept-based generation, conditional interpretation, concept debugging, intervention, and imputation, in a single probabilistic framework. For example:

• Intervention. During generation, one can easily intervene on the concepts $\mathbf{c}$ to fix any incorrect generation.
• Interpretation. Given a generated image, one can infer the concepts $\mathbf{c}$ to check what concepts are expressed in the image, thereby interpreting the generation process. One can then perform intervention (mentioned above) based on the inferred $\mathbf{c}$.
• Debugging. Given the input $\mathbf{y}$ and the generated image $\mathbf{x}$, one can debug what concepts are generated incorrectly by comparing the what concepts are generated (i.e., $p(\mathbf{c}|\mathbf{x})$) and what concepts should be generated (i.e., $p(\mathbf{c}|y)$). One can then perform intervention (mentioned above) based on the debugging results.

In the tasks above, the concept vector $\mathbf{c}$ is not merely a "processed conditioning input"; it is also an interpretable variable that enables flexible compatibility measurement and joint modeling of the instruction, concept, and generation within our energy-based framework.

W2.1. The concept-based generation method described in (11)-(13) resembles a Gibbs sampling or coordinate-wise algorithm, but equation (11) focuses on maximizing mapping energy rather than the entire joint energy...the rationale behind this approach...as $E^{joint}=E^{map}+E^{concept}$ incorporates dependencies on the concept in both terms...

We are sorry for the confusion. In the concept-based joint generation, we perform concept inference and image generation by minimizing the joint energy $E_{\mathbf{\psi}}^{joint}({\mathbf{x}},{\mathbf{c}},{\mathbf{y}}) \triangleq E_{\mathbf{\psi}}^{concept}({\mathbf{x}},{\mathbf{c}}) + \lambda_m E_{\mathbf{\psi}}^{map}({\mathbf{c}},{\mathbf{y}})$. This process entails the minimization of both the mapping energy $E^{map}$ and the concept energy $E^{concept}$.

In the full model, to minimize the joint energy $E_{\mathbf{\psi}}^{joint}({\mathbf{x}},{\mathbf{c}},{\mathbf{y}}) \triangleq E_{\mathbf{\psi}}^{concept}({\mathbf{x}},{\mathbf{c}}) + \lambda_m E_{\mathbf{\psi}}^{map}({\mathbf{c}},{\mathbf{y}})$, we alternate between

• Eqn. (17) to infer $p(\hat{\mathbf{c}} | \mathbf{x})$,
• Eqn. (11) to infer $p(\hat{\mathbf{c}} | \mathbf{y})$, and
• Eqn. (12)-(13) to infer $p(\mathbf{x}|\hat{\mathbf{c}} )$ until convergence. This is indeed similar to Gibbs sampling, but it is slightly different. For example, Gibbs sampling involves computing the conditional probability of one variable given all other variables. By contrast, in Eqn. (11), we infer $p(\hat{\mathbf{c}} | \mathbf{y})$ rather than $p(\hat{\mathbf{c}} | \mathbf{x}, \mathbf{y})$.

In practice, we find that simply alternating between minimizing Eqn. (11) (mapping energy) and Eqn. (13) (concept energy) can already provide satisfactory results with improved computational efficiency.

Note that in Eqns. (12)-(13), the diffusion-style sampling dynamics of the image $\mathbf{x}_t$ necessitate computing the gradient with respect to $\mathbf{x}$ for each sampling step. Here the mapping energy $E^{map}(\mathbf{y},\mathbf{c})$ in $E^{joint}$ is not relevant to $\mathbf{x}$, and therefore is not involved in Eqn. (13).

W2.2. Additionally, maximizing $E^{map}$ with respect to the binary vector suggests an integer programming problem, which the paper does not sufficiently address regarding efficiency.

This is a good question. Note that while the concept label $\mathbf{c}$ is binary, our ECDM's predicted concept probability $\hat{\mathbf{c}}$ is a real value in the range $[0, 1]$. Therefore, we can use gradient descent to compute the gradient of $E^{joint}$ (including $E^{map}$) w.r.t. $\mathbf{c}$ and update $\mathbf{c}$ iteratively to infer $\mathbf{c}$. In this case, it is not an integer programming problem and is therefore very efficient.

For Questions:

Q1. Code Availability

Thank you for your interest in our code. We assure you that we will make our code open-source and available to the wider research community after acceptance, thereby facilitating the reproducibility of our results. So far, we have finished cleaning up the source code and will release it if the paper is accepted.

Q2. How is the number of concepts $K$ in the concept vector $\mathbf{c}$ determined? Is $K$ fixed?

The number of concepts $K$ is determined by the concept annotations provided by the specific dataset. The overall concept number $K$ is fixed during training and inference, but can be adjusted according to different datasets. We also include more details on the Datasets in the Experiments Setup Section (Section 4.1) and Appendix C.

Q3. How is the concept embedding $\mathbf{v}_k$ modeled?

Thanks for mentioning this. For a given concept phrase (e.g., "black bird wings"), we first extract the textual embeddings using the text encoder. Subsequently, the textual embedding is projected (through a learnable projection neural network) into a positive embedding $\mathbf{v}_k^{(+)}$ and a negative embedding $\mathbf{v}_k^{(-)}$. The final concept embedding $\mathbf{v}_k$ is then a combination of $\mathbf{v}_k^{(+)}$ and $\mathbf{v}_k^{(-)}$, weighted by the concept probability $c_k$, i.e., $\mathbf{v}_k = c_k \cdot \mathbf{v}_k^{(+)} + (1-c_k) \cdot \mathbf{v}_k^{(-)}$. This combined concept embedding is further utilized in all five unified tasks (i.e., generation, interpretations, debugging, intervention, and imputation).

Thank you for your insightful and constructive feedback. We are glad that you found our proposed framework "versatile"/"applicable to any conditional data generation task"/"enhances the interpretability", our paper "clearly written"/"well-organized", and our experiments "thorough". Below we address your questions one by one in detail.

W1. ... the significance of this work within the broader field of generative models ... a straightforward combination of concept bottleneck models and standard conditional diffusion models.

We would like to clarify that our ECDM's primary contribution lies not only in combining existing models but in developing a unified probabilistic framework that unlocks new capabilities. We provide more details below.

Fundamental Difference between ECDM and Concept Bottleneck Models (CBMs) Combined with Conditional Diffusion Models. Conventional CBMs typically predict a set of concepts from an image input and then predict the class label based on these predicted concepts, i.e., predicting concepts $\mathbf{c}$ and labels $\mathbf{y}$ given an image $\mathbf{x}$. Similarly, conditional diffusion models typically generate an image $\mathbf{x}$ given an input text or class label $\mathbf{y}$. In contrast to such "sequential" modeling, our proposed ECDM jointly models the relationships among class-level instructions $\mathbf{y}$, concept sets $\mathbf{c}$, and the generated image $\mathbf{x}$ within an energy-based framework.

Consequently, our energy-based framework facilitates a more flexible and unified inference process. Given any subset of these three elements (the instruction $\mathbf{y}$, the concepts $\mathbf{c}$, and the generated image $\mathbf{x}$), the joint framework can infer the remaining elements by composing energy functions and deriving conditional probabilities. This unique characteristic allows us to achieve generation, interpretation, and intervention within a unified framework. These capabilities extend beyond what either component could achieve independently. Unifying these tasks under a flexible framework provides the interpretability and generation modeling community with deeper insights into how diffusion models generate images, comprehend, and incorporate concepts in the generation process through a human-understandable probabilistic interpretation, thus holding significant value for the community. Our novelty and contributions include:

• Unifying Five Tasks in a Single Framework. Our ECDM framework unifies concept-based generation, conditional interpretation, concept debugging, intervention, and imputation under a joint energy-based formulation. These five tasks encompass a typical workflow of the text-to-image diffusion model, and unifying them enhances generation quality, boosts interpretability, and enables interpretable intervention. In our proposed energy-based modeling of the diffusion model, we incorporate large-scale pretrained diffusion models (e.g., Stable Diffusion) within our framework to achieve a unified interpretation of these diffusion models, an area that previous methods have less extensively explored.

• Probabilistic Interpretations by Energy Functions. With ECDM’s unified framework, we have developed a set of algorithms to compute various conditional probabilities by composing corresponding energy functions. These conditional probabilities provide theoretical support for concept-based interpretation of the generation process, as opposed to merely visualizations, and enable flexible inference of different elements (as mentioned in point 1) for diverse tasks.

• Better Experimental Results. Empirical results on real-world datasets demonstrate ECDM's state-of-the-art performance in terms of image generation, imputation, and their conceptual interpretations.

Dear Reviewer 5CJZ,

Thank you for your time and effort in reviewing our paper.

We appreciate your valuable comments and suggestions, and we firmly believe that our response and revisions can fully address your concerns. We are open to discussion (before Nov 26 AOE, after which we will not be able to respond to your comments unfortunately) if you have any additional questions or concerns, and if not, we will be immensely grateful if you could reevaluate your score.

Thank you again for your reviews which helped to improve our paper!

Best regards,

ECDM Authors

For Questions:

Q1. Further clarification for "Pivotal Inversion" and "Energy Matching Inference".

We apologize for any unclear parts in our explanation and are pleased to provide more detailed explanations below.

The intuition behind the ECDM's interpretation task is that the diffusion model's sampling trajectory conditioned on the instruction $\mathbf{y}$ and the optimal concept set $\tilde{\mathbf{c}}$ should be alike in our framework, since they are all under our joint energy-based formulation. To substantiate this intuition, two steps are required: (1) pivotal inversion, which aims to simulate the sampling trajectory conditioned on the instruction $\mathbf{y}$; (2) energy matching inference, which optimizes the concept probability $\tilde{\mathbf{c}}$ to find the most compatible concept set (i.e., the concept set that is the most compatible with the generated image) by minimizing the distance between the sampling trajectory conditioned on concepts and the one obtained from pivotal inversion.

We explain these two steps individually below.

Pivotal Inversion: The goal of pivotal inversion is to simulate how the pretrained diffusion model samples an image directly conditioned on the instruction. For instance, consider an image of a "black billed cuckoo" bird generated from the instruction "A photo of the bird black billed cuckoo" using pretrained Stable Diffusion 2.1. Pivotal inversion utilizes reverse DDIM [8] to derive a set of latents that represent how the image is gradually denoised from Gaussian noise conditioned on this instruction. This derived set of latents serves as pivots that illustrate the model's original sampling trajectory and, within our formulation, simulates the energy landscape of the external energy model (pretrained diffusion model). This facilitates the matching process in the subsequent step.

Energy Matching Inference: The goal of this energy-matching-inference step is to determine the most compatible concept set, i.e., the concept set that is the most compatible with the generated image; this is done using the simulated trajectory from pivotal inversion. Specifically, we freeze all learned embeddings as well as the concept energy network, and optimize the concept probability $\tilde{\mathbf{c}}$. The optimization target is to minimize the distance between the sample trajectory conditioned on the concept set and the fixed pivotal trajectory simulated in the previous step. By minimizing this distance, we align the energy landscape of the concept energy network with the external energy network to obtain the most compatible concept set, which is why this process is called energy matching inference. Once the most compatible concept set is found, these concepts can then be used to interpret the generated image.

We hope this further explanation of these two concepts clarifies the overall idea. If you need additional clarification or have further questions, we would be more than happy to provide any additional details required. These further clarifications have been incorporated in Appendix D.1.

[1] Ismail, Aya Abdelsalam, et al. "Concept Bottleneck Generative Models." The Twelfth International Conference on Learning Representations. 2023.

[2] Du, Yilun, et al. "Unsupervised Learning of Compositional Energy Concepts." Advances in Neural Information Processing Systems 34 (2021): 15608-15620.

[3] Liu, Nan, et al. "Unsupervised Compositional Concepts Discovery with Text-to-image Generative Models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[4] Hao, Shaozhe, et al. "ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extraction." ECCV. 2024.

[5] Su, Jocelin, et al. "Compositional Image Decomposition with Diffusion Models." ICML. 2024.

[6] Hertz, Amir, et al. "Prompt-to-prompt Image Editing with Cross Attention Control." arXiv preprint arXiv:2208.01626 (2022).

[7] Xu, Xinyue, et al. "Energy-based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations." The Twelfth International Conference on Learning Representations. 2024.

[8] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising Diffusion Implicit Models." arXiv preprint arXiv:2010.02502 (2020).

W3.Code Availability.

Thank you for your interest in our code. We assure you that we will make our code open-source and available to the wider research community after acceptance, thereby facilitating the reproducibility of our results. So far, we have finished cleaning up the source code and will release it if the paper is accepted.

W4. The experiments rely on a fixed pretrained stable diffusion model, while other models are not explored.

This is a good question. Our model does not assume any specific model architecture and is therefore compatible with any pretrained diffusion model.

We follow the convention in the field to choose Stable Diffusion as the base model because it is the most commonly used pretrained large text-to-image diffusion model. Thousands of works have used Stable Diffusion as their only base model [3, 4, 5]. Note that a lot of open-source pretrained diffusion models are actually finetuned from the Stable Diffusion model. Therefore, we believe the results are representative and generalize across different pretrained models.

Nonetheless, we are happy to further validate our framework if you have any specific open-source pretrained diffusion model in mind. We would be very happy to include additional results before the discussion period ends in late November.

W5. Precise regional control and computational expense.

Thank you for mentioning this. As mentioned in our Conclusion and Limitations section, precise regional control and computational expense are two of our ECDM's limitations.

Precise Regional Control. Enhancing precise regional control in concept-based editing is indeed an intriguing direction for future work. This challenge could potentially be addressed by incorporating attention-based regional editing techniques, such as prompt-to-prompt [6]. This is definitely interesting future work, but is out of the scope of our paper.

Computational Cost. We would like to clarify that our ECDM introduces low computational overhead compare to existing methods. Specifically:

• Mapping Energy Network. We sample from the mapping energy network using the Gradient Inference technique, as outlined in ECBM [7]. This sampling procedure requires approximately 10 to 30 steps, taking around 10 seconds of wall-clock time.
• Concept Energy Network. For the generative concept energy network, we model the diffusion model as an implicit representation of the energy function, making the diffusion model sampling algorithm applicable to our framework. We utilize the standard diffusion sampling algorithm (i.e., DDIM [8]]) to generate an image from the concept energy network. This process involves approximately 50 steps and takes around 3 seconds of wall-clock time when using an NVIDIA RTX 3090. Therefore, the computational overhead remains comparable to that of standard diffusion models.

We agree that accelerating the sampling process of our framework is another promising area for future research and could potentially be addressed by adapting variational inference methods. However, we decided to leave this to future work, as it is beyond the scope of this paper.

Thank you for your valuable comments. We are glad that you found our model "is a valuable tool"/"supports both generative and interpretive tasks" and that our ECDM "shows quantitative improvements in image quality and concept alignment". Below we address your questions one by one.

W1. Comparisons with some related works (COMET or CBGM).

Thank you for mentioning this. We would like to clarify that our setting focuses on concept-based generation and interpretation given a pretrained large diffusion model. Therefore CBGM [1] and COMET [2] (both cited in our paper) are not applicable in this setting. Specifically:

• CBGM [1] involves training a new diffusion model from scratch using a modified Diffusion UNet. In contrast, we focus on augmenting an existing pretrained large diffusion model (e.g., Stable Diffusion) to enable concept-based generation, intervention, and interpretation. Therefore CBGM is not applicable to our setting.
• CBGM [1] is not an energy-based model, which is different from our ECDM.
• In this paper, we focus on the text-to-image generation setting, where the input is free-form text, and the output is an image. CBGM [1] is a conditional diffusion model that takes class labels as input. Therefore CBGM is not applicable to our setting.
• COMET [2] is an unsupervised, unconditional diffusion model that does not take any input (neither class labels nor text). Therefore COMET is not applicable to our setting either.
• Since COMET is an unsupervised learning model, the visual concepts decomposed by COMET do not have ground truth. Therefore, it is not possible to evaluate COMET in our setting.

We have included the discussion above in the revision as suggested (Appendix E.1).

W2. Incorporation of other metrics.

Thank you for your suggestions.

Additional Experiments and Metrics. Following your suggestion, we run additional experiments on another metric, Inception Score (IS). The results are presented in the tables below.

Table A. Results on the CUB dataset.

Table B. Results on the AWA2 dataset.

Table C. Results on the CelebA-HQ dataset.

These tables show that even in terms of IS, our method still outperforms all baseline methods, indicating consistently improved image quality. These results and the necessary discussions have been incorporated into the revised version of our paper in the Experiment section.

FID Already Evaluates Diveristy. We would like to clarify that the metric FID already evaluates the diversity of our generated images. Specifically, FID is computed as the distance between the distribution of generated images and the distribution of real images, using feature vectors from a pre-trained Inception network. Therefore, a small FID means that our ECDM's generated images are as diverse as the real images.

Thank you again for your comments, and we are open to considering additional metrics that might enhance the evaluation process and provide further insights into our model's performance. If you have specific metrics in mind, we would be very happy to explore them further during the discussion period.

Dear Reviewer grAS,

Thank you for your time and effort in reviewing our paper.

We appreciate your valuable comments and suggestions, and we firmly believe that our response and revisions can fully address your concerns. We are open to discussion (before Nov 26 AOE, after which we will not be able to respond to your comments unfortunately) if you have any additional questions or concerns, and if not, we will be immensely grateful if you could reevaluate your score.

Thank you again for your reviews which helped to improve our paper!

Best regards,

ECDM Authors

Q4. How robust is the concept interpretation when handling out-of-distribution samples?

This is a good question. Inspired by your comment, we conducted additional experiments regarding out-of-distribution samples, and the results are included in Figure 7 and Appendix B.2.

Additional Results on Out-of-Distribution Samples. Specifically, the experiments are conducted on the TravelingBirds dataset following the robustness experiments of CBM [7]. We provide the bird image under significant background shift to our models for concept interpretation. In this case study, our model can still accurately infer the corresponding concepts of the bird "Vermilion Flycatcher" (e.g., "all-purpose bill shape" and "solid belly pattern"). These findings demonstrate our model's robustness when facing domain shifts.

Why ECDM Is Robust for Out-of-Distribution Samples. Typical methods tend to suffer from spurious features, e.g., irrelevant backgrounds. In contrast, the concept-based modeling framework of our ECDM ensures the robustness of the interpretations. Specifically, ECDM forces the model to learn concept-specific information and use these concepts to generate images and interpret these images; this way, ECDM focuses more on the genuine attributes of the target object and is less influenced by irrelevant, spurious features, such as irrelevant backgrounds. As a result, our ECDM enjoys robustness when dealing with out-of-distribution samples. For example, when interpreting a water bird with a spurious land background, our ECDM focuses only on the concepts of the water bird in the foreground and, therefore, will not be fooled by the spurious features in the background.

We have incorporated the discussion above in our revised paper (e.g., Appendix B.2) as suggested.

[1] Gao, Ruiqi, et al. "Learning Energy-Based Models by Diffusion Recovery Likelihood." International Conference on Learning Representations. 2021.

[2] Zhu, Yaxuan, et al. "Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood." The Twelfth International Conference on Learning Representations. 2023.

[3] Xie, Jianwen, et al. "A Theory of Generative Convnet." International conference on machine learning. PMLR, 2016.

[4] Du, Yilun, and Igor Mordatch. "Implicit Generation and Modeling with Energy Based Models." Advances in Neural Information Processing Systems 32 (2019).

[5] Xu, Xinyue, et al. "Energy-based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations." The Twelfth International Conference on Learning Representations. 2024.

[6] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising Diffusion Implicit Models." arXiv preprint arXiv:2010.02502 (2020).

[7] Koh, Pang Wei, et al. "Concept Bottleneck Models." International conference on machine learning. PMLR, 2020.

Thank you for your encouraging and valuable comments. We are glad that you found our method "novel" and has "multiple practical applications", our theoretical framework "comprehensive" "with detailed proofs", our empirical results "strong", and our improvement "clear". We will address each of your comments in turn below.

W1. Acknowledging pioneering work on energy-based diffusion models and works using EBM as compositions.

Thank you for pointing us to these interesting papers. Following your suggestions, we have cited and discussed these pioneering papers in our revision (in the related works section).

We recognize the importance of the works [1, 2, 3, 4] you have mentioned. For example, in the pioneering works on energy-based diffusion models, i.e., Diffusion Recovery Likelihood and its extension [1, 2], they trained EBMs on diffusion recovery likelihood to facilitate training and sampling on high-dimensional datasets.

We also note that for these works,
(1) The number of supported concepts is fixed and limited (e.g., only $6$ concepts, compared to $112$ concepts in our ECDM), and hence not sufficiently informative as interpretations.
(2) More importantly, these works aim to compositional generation with deterministic concepts and, therefore, fail to provide probabilistic interpretation, which is the focus of our ECDM.
Therefore, these methods are not applicable to our setting.

In contrast, our ECDM explicitly considers human-understandable probabilistic concept explanations in its design by jointly modeling the input instruction $\mathbf{y}$, associated concepts $\mathbf{c}$, and the generated image $\mathbf{x}$ during the generation process within a unified energy-based framework.

For Questions:

Q1. How does the method scale with increasing number of concepts?

Thank you for mentioning this. Our model is efficient, and scales linearly with the number of concepts in terms of computation and the number of model parameters.

For example, when concept number $K=6$, the parameter size is 27.57 M, excluding all frozen pretrained components, and when $K=112$, the parameter size is 110.99 M. Note that the computational cost and number of parameters for all frozen pretrained components are fixed (i.e., constant).

We have included a scaling analysis in Figure 8 of the updated Appendix D. We can see that the number of model parameters scales linearly with the number of concepts.

Q2. What is the computational overhead compared to standard diffusion models?

Thank you for mentioning this.

Mapping Energy Network. We sample from the mapping energy network using the Gradient Inference technique, as outlined in ECBM [5]. This sampling procedure requires approximately 10 to 30 steps, taking around 10 seconds of wall-clock time.

Concept Energy Network. For the generative concept energy network, we model the diffusion model as an implicit representation of the energy function, making the diffusion model sampling algorithm applicable to our framework. We utilize the standard diffusion sampling algorithm (i.e., DDIM [6]]) to generate an image from the concept energy network. This process involves approximately 50 steps and takes around 3 seconds of wall-clock time when using an NVIDIA RTX 3090. Therefore, the computational overhead remains comparable to that of standard diffusion models.

Q3. Could the framework be extended to handle continuous concept values rather than binary?

This is an insightful question and points to an interesting extension of our ECDM.

• Our framework naturally supports normalized continuous-valued concepts. For example, By normalizing the continuous concept value to the range of $[0,1]$, the concept probability $c_k$, which is already a real (continuous) number in the range of $[0, 1]$, used for mixing the positive/negative concept embedding can be substituted by this value, and further be integrated into our framework.
• Our framework can be further extended to support unnormalized continuous-valued concepts. For example, we can learn a unit concept embedding $\mathbf{e}_k \in \mathbb{R}^d$ that represents the unit value of a certain concept, and a continuous magnitude concept $c_k \in \mathbb{R}$ embedding that represents the actual magnitude of the concept. With $\mathbf{e}_k$ and $c_k$, we can then replace the final concept embedding (in Line 232-238) $\mathbf{v}_k = c_k \cdot \mathbf{v}_k^{(+)} + (1-c_k) \cdot \mathbf{v}_k^{(-)}$ with $\mathbf{v}_k = c_k \cdot \mathbf{e}_k$. All other components of our ECDM can remain unchanged.

We agree that extending our method toward continuous concepts would be an interesting future work, and we have included the discussion above in our revised paper (Appendix E.2).

Dear Reviewer oJRi,

Thank you for your time and effort in reviewing our paper.

We appreciate your valuable comments and suggestions, and we firmly believe that our response and revisions can fully address your concerns. We are open to discussion (before Nov 26 AOE, after which we will not be able to respond to your comments unfortunately) if you have any additional questions or concerns, and if not, we will be immensely grateful if you could reevaluate your score.

Thank you again for your reviews which helped to improve our paper!

Best regards,

ECDM Authors

W2. "Given a binary concept labels set, I am wondering the optimal output of the concept energy model with input y? It seems that a binary output is also expected from y to minimize the loss? Can you provide any explanation why it would not learn a binary vector given the target is binary?"

This is a good question. Our ECDM's formulation and learning process goes beyond a simple binary logical mapping; instead, they involve probabilistic interactions among concepts, instructions, and generated images.

As shown in the response to W1 above, given the same text prompt (e.g., "A photo of the animal Polar Bear"), our ECDM can generate different concept probabilities according to the different generated images. Therefore, it is not a simple binary logical mapping from the text prompt (and its embedding) to the concepts; they also depend on the images.

Formally, we model the joint energy function as:
$E_{\mathbf{\psi}}^{joint}({\mathbf{x}},{\mathbf{c}},{\mathbf{y}}) \triangleq E_{\mathbf{\psi}}^{concept}({\mathbf{x}},{\mathbf{c}}) + \lambda_m E_{\mathbf{\psi}}^{map}({\mathbf{c}},{\mathbf{y}})$,
with the mapping energy function
$E_{\mathbf{\psi}}^{map}(\mathbf{y},\mathbf{c}) = D_{uw}(\mathbf{u},\mathbf{w})$,
and concept energy function
$E_{\mathbf{\psi}}^{concept}(\mathbf{x},\mathbf{c}) \triangleq \mathbb{E}_{\mathbf{x}, \epsilon \sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{I}), t} [ \left\| \epsilon - \epsilon _\theta(D_c(\mathbf{c}),\mathbf{x}_t, t) \right\|^2_2 ]$.

• Therefore, given the input $\mathbf{y}$, the optimal concepts $\mathbf{c}$ depend on not only mapping energy function $E_{\mathbf{\psi}}^{map}(\mathbf{y},\mathbf{c})$ but also the concept energy function $E_{\mathbf{\psi}}^{concept}(\mathbf{x},\mathbf{c})$. Since the inferred concepts $\mathbf{c}$ would also change with respect to the image $\mathbf{x}$, the resulting $\mathbf{c}$ for each concept will be a real value between $0$ and $1$.
• Since different images $\mathbf{x}$ may have different sizes, concepts such as "big" are actually real values from $[0,1]$, as you mentioned. With our ECDM inferring concepts $\mathbf{c}$ from both the input text $\mathbf{y}$ and the image $\mathbf{x}$, the resulting $\mathbf{c}$ for each concept will be a real value between $0$ and $1$.

Last but not least, we would like to thank you again for your insightful comments and for keeping the communication channel open. Please do not hesitate to let us know if you have any follow-up questions. We will be very happy to provide more clarifications.

Thank you for your valuable reviews. We are glad that you found our model "provides probabilistic interpretation" and our paper "well-written and easy to read". Below, we address your questions one by one:

W1. "... The point is that it is a deterministic mapping. ... I am expecting that the concept probability vector changes with generated image (not only the input text prompt)."

Thank you for this insightful comment. We would like to clarify that (1) our ECDM's concept probability does vary with both the text instructions and the generated image and that (2) it provides probabilistic interfaces rather than a deterministic mapping.

Why Our ECDM's Concept Probability Varies with the Generated Image Too. This is actually a key property enabled by our ECDM's joint energy-based modeling. Specifically, we model the joint energy function as:
$E_{\mathbf{\psi}}^{joint}({\mathbf{x}},{\mathbf{c}},{\mathbf{y}}) \triangleq E_{\mathbf{\psi}}^{concept}({\mathbf{x}},{\mathbf{c}}) + \lambda_m E_{\mathbf{\psi}}^{map}({\mathbf{c}},{\mathbf{y}})$,
with the mapping energy function
$E_{\mathbf{\psi}}^{map}(\mathbf{y},\mathbf{c}) = D_{uw}(\mathbf{u},\mathbf{w})$,
and concept energy function
$E_{\mathbf{\psi}}^{concept}(\mathbf{x},\mathbf{c}) \triangleq \mathbb{E}_{\mathbf{x}, \epsilon \sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{I}), t} [ \left\| \epsilon - \epsilon _\theta(D_c(\mathbf{c}),\mathbf{x}_t, t) \right\|^2_2 ]$.

This joint formulation of energy functions, particularly the concept energy function, allows the concept probability ($\tilde{\mathbf{c}}$) in the inference process to change with the generated image in our model's interpretation task, which is the main focus of our paper. Specifically, we optimize the concept probability $\tilde{\mathbf{c}}$ using the concept energy network (Eqn. (17)) given a generated image for interpretation, with a probability range of $\tilde{\mathbf{c}} \in [0,1]$. This non-binary probability indicates "to what degree the diffusion model generates the image based on these specific concepts." For example, given a "polar bear" image generated by Stable Diffusion, our ECDM will infer a high probability for the concept "arctic" and a low probability for "forest" using Eqn. (17), suggesting that the Stable Diffusion model may generate this image based on "arctic" rather than "forest".

Additional Experiments: Concepts "Water" and "Arctic". Inspired by your comments, we conducted additional experiments to observe the changes in this non-binary concept probability (Figure 6 of Appendix B.1). Given the same prompt, "A photo of the animal Polar Bear", the diffusion model generates two different "Polar Bear" images: The top image does not have a "water" and "arctic" background, while the bottom image has a "water" and "arctic" background. Our ECDM correctly infers that the probabilities of the concepts "water" and "arctic" in the top image are 0.1233 and 0.0363, respectively, much smaller than those in the bottom image (0.9543 and 0.8015, respectively).

Additional Experiments: Concept "Big". For the concept "big," we can also see meaningful variation in the inferred probabilities, 0.9067 (top image) versus 0.9922 (bottom image), meaning that our ECDM is more certain that the bottom image is a "big" polar bear, but is less certain about the top image since it only shows the head of the bear.

Therefore, our ECDM's concept probability vector does adjust with the generated image in interpretation. We have incorporated your valuable insight and further discussions into Appendix B.1.

Dear Reviewer AYc9,

Thank you for your time and effort in reviewing our paper.

We appreciate your valuable comments and suggestions, and we firmly believe that our response and revisions can fully address your concerns. We are open to discussion (before Nov 26 AOE, after which we will not be able to respond to your comments unfortunately) if you have any additional questions or concerns, and if not, we will be immensely grateful if you could reevaluate your score.

Thank you again for your reviews which helped to improve our paper!

Best regards,

ECDM Authors