The Hidden Language of Diffusion Models

We thank the reviewer for the comprehensive feedback and for the interesting points for discussion. We are encouraged that the reviewer found our work to be well-motivated and recognized the extensiveness and effectiveness of our experiments. In the following, we provide a response to each of the weaknesses pointed out in the review. Additionally, the modifications in the revision are marked in red, for your convenience.

Re. Related works on image decomposition: We thank the reviewer for bringing this to our attention. We have revised the paper to include the proposed works in the related works section.

To reiterate the difference between the objective of these methods and our method, we enclose a short comparison to the methods indicated above:

As a general note, all methods described below only offer decomposition for a single image and, therefore, fall short of providing concept-level explanations, which is the main objective of concept-based explainability.

FineGAN: disentangles background, foreground, shape, and appearance. This method is not applicable to interpret SD since (1) it’s GAN-based and therefore limited to the training domain, (2) the disentangled information provides little to no interpretable information that is not observable just by looking at the images.

GIRAFFE: similar to FineGAN, GIRAFFE disentangles objects, appearance, shape, and background information, with the objective of obtaining controllable generation. In addition to the points mentioned above, this method incorporates 3D scene representations, which are not applicable in our case.

Slot Attention: learns object-centric representations for complex scenes, and employs them for object discovery and set prediction. The task of object discovery has some similarities to ours but is also very different, see below. Additionally, the datasets used for training (CLEVR6, Multi-dSprites, Tetrominoes) differ significantly from the distribution of images generated by SD.

DTI Sprites: similar to Slot Attention, DTI Sprites decomposes a scene into objects including their shape, size, and color (appearance), which allows improved controllable generation.

Inspired by the review, we conducted an additional experiment to demonstrate the difference between our objective and existing methods for image decomposition. The closest relation we observed between the proposed methods and ours is through the task of unsupervised object discovery (including shapes, colors, sizes, etc.). Importantly, our task largely differs from that of object discovery, since an interpretation should often contain elements beyond objects that are physically present in the image. For example, Fig. 2 shows connections such as “snake” to “gecko” + “twisted”, a “gecko” is not present in the image, and “twisted” is not an object but an adjective. Similarly, “cashmere” is not present in an image of a “camel”, and “winding” is not present in images of “snails”. In contrast to these semantic, profound connections, interpretation using object discovery only provides representations that are based on concrete objects and parts that are visible in the image. Therefore, these methods fall short of discovering deeper connections (e.g., reliance on exemplars, non-trivial biases, etc.).

To empirically demonstrate this point, we added exemplary results using [1], which is the state-of-the-art object discovery method based on SD, and allows to discover concepts from sets of images, as well as single images. The results of this exploration of two exemplary basic concepts, “snail”, “snake” and an additional complex concept “impression of Japanese serenity”, are presented in Appendix G of the revision. This experiment directly demonstrates our intuition above. As can be seen, the method learns to embed the concept in a single token (as expected from its task definition for images of one class) and does not employ any other learned tokens.

[1] “Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models”, Nan Liu, Yilun Du, Shuang Li, Joshua B. Tenenbaum, Antonio Torralba, ICCV’23.

Re. Forming abstract connections beyond objects: Kindly note that our examined concepts include an extensive set of both very complex and abstract concepts (e.g., “elegance on a plate”, “happiness”, “affection”, “impression of Japanese serenity”, etc.). A full list of these concepts can be found in Appendix B (see Emotions, Actions, Complex concepts). Following the review, we have added word cloud visualizations of the learned decompositions for abstract and complex concepts (see Appendix F). The provided examples show that our method can meaningfully decompose abstract concepts and detect relations to other abstract concepts (e.g., linking “happiness” to “dream”, “emotion”, “soul”, “laughter”, “childhood”; or linking “sadness” to “grief”, “depression”, “worried”, “alone” etc.). These results indicate that Conceptor is capable of revealing connections between abstract concepts. Examples of single-image decompositions for such abstract concepts can be found in Fig. 2 (“fear”, “impression of Japanese serenity”) and in Fig. 8 (“happiness”).

Re. The difference with word arithmetics: We thank the reviewer for raising this interesting point for discussion. We wish to emphasize that our method does not perform word embedding decomposition, and in fact, significantly differs from it. In the following, we list the fundamental differences between the two:

1. Our learned pseudo-token $w^{\ast}$ does not decompose the text embedding of the concept $w^c$, i.e.: $w^{\ast}= \sum_i w_i \alpha_i \neq w^c$:

   Kindly note that our method does not decompose the word embedding of the concept token(s) $w^c$. Instead, we propose to decompose the internal representation of the concept by the model. This is done by aggregating the set of features used by the model in the image generation process (see Sec. 3). The text embeddings serve merely as an intermediate language linking us to the uninterpretable inner representations of SD. To empirically prove this point, we calculated the cosine similarity between our learned pseudo-token $w^*$ and the concept token $w^c$ averaged across our entire dataset, and obtained a low similarity score of 0.61 (for reference, “cat” and “car” get a higher similarity score of 0.66). This result, combined with our Token Diversity metric (Tab. 1), our single image decomposition results (Fig. 2) and our CLIP top words ablation (Tab. 3) demonstrate that our method is able to learn connections based on visual semantics, that transcend word arithmetics, as mentioned in the paper (see Sec. 1, 6).

2. Our single-image decomposition results do not suggest word embedding arithmetics:

   Note that the annotations in our single-image decomposition figures (Figs. 2,8) do not indicate word embedding arithmetics. For example: $word\textunderscore embed(painter) \neq word\textunderscore embed(electrician) + word\textunderscore embed(brush) + word\textunderscore embed(Monet)$ (the cosine similarity between “painter” and “electrician + brush + Monet” is merely 0.496). This indicates that the decomposition is not based on textual semantics but on an inner-representation decomposition.

   Additionally, note that a diffusion model generates diverse images for the same concept given different random seeds, and not all learned features are manifested in all of the concept images (e.g., the images of the nurses in Fig. 12 can be in black and white or in color, with scrubs or a white suit, with and without a stethoscope, etc). Our single-image decomposition scheme extracts the specific set of elements from the decomposition learned by Conceptor that are used to generate a single specific image. This is evident when different features appear for single-image decompositions of different images of the same concept, e.g., the “camel” and “snake” examples in Fig. 2 vs. Fig. 8. Thus, the arithmetic annotations in Figs. 2, 8 indicate the addition of the features of the decomposition elements to the output image.

   This ties directly to the question by the reviewer:

   “I'd love to see if u can optimize on electrician image and then try removing the brush concept, does it become a painter?”.

   The answer is that we are not doing word arithmetics and this manipulation is not possible, since applying Conceptor to the concept “electrician” results in a decomposition that does not include the element “brush”. This is logical: SD does not employ the feature “brush” in the generation of images of electricians, unlike the role “brush” plays for the “painter” concept.

Re: Novelty: As mentioned and demonstrated above, our novel decomposition formulation has not been attempted before and differs significantly from other image decomposition objectives such as object/concept discovery. As we established, Conceptor also fundamentally differs from word embedding arithmetics.

Our experiments demonstrate that Conceptor has significant value in exposing mixed visual/textual components in a way that, as far as we can ascertain, has never been shown before. For example, Conceptor exposes reliance on exemplars and mimicking of artistic styles and reveals concepts that are incorporated purely for visual reasons such as shape or texture, non-trivial biases, and the way textual ambiguities are handled.

From a technical point of view, we believe that there is a considerable novelty in our decomposition strategy, e.g., in the combination of Eqs. 3 and 4, in which the coefficients $\alpha$ are modeled as a function of the word embedding $w$ (see Sec. 3). These unique choices are extensively ablated in Sec. 4.2.1, where we empirically demonstrate that they are all crucial to maintaining the faithfulness and meaningfulness of our method. Our single-image decomposition scheme introduces additional innovation by enabling interpretation for a specific generation, in addition to the general concept interpretation provided by our method.

Re. Optimization length per image: Please note that Conceptor does not operate on a single image but on an entire concept (see Sec. 3). The optimization process for the entire concept takes around 6.5 minutes on a single A100 GPU with 40GB (given more memory, one could significantly increase the batch size and speed up the process).

After this optimization, manipulation of the obtained coefficients for a given image is dominated by the time it takes to generate the image (around 4 seconds, again depending on the hardware used).

We are happy to address any other questions.

Thank you for your response to our rebuttal! We appreciate the continued discussion.

In addressing the second point, the literature on word arithmetics shares with our method the use of a vocabulary in which each word is embedded as a vector in some vector space. However, the two lines of research diverge at this point, and it is crucial for us to emphasize the novelty of our method:

1. Unlike the goal of word embedding arithmetics, we do not measure distances in the vector space. Specifically, our decomposition of $w^c$ is not intended to find a weighted sum of dictionary elements equal to it. Instead, we measure the reconstruction loss of generated images.

2. Our method does not optimize the set of coefficients directly. Instead, it operates by learning a mapping function between each word embedding and a corresponding coefficient. In other words, our method can be viewed as a nonlinear mapping of the form: $w^* = \sum_i f(w_i) w_i$, where $f$ is a nonlinear function. Again, this is different from word arithmetics, which considers linear operations in the vector space.

   Our extensive ablations (Sec. 4.2.1, Tab. 3 “without MLP”) show that this novel form of learning the decomposition is critical to the success of the method, as without it (i.e., learning separate coefficients) the decompositions are not faithful. This ablation shows that the task of assigning the appropriate coefficients to word embeddings to reconstruct the learned representation is not trivial.

3. As a result of 1+2, the tokens obtain their meaning from the context of the decomposition and not from their single-word meaning (see also Fig. 5). For example, as a standalone token, “winding” generates an image of winding roads and spirals, however in the context of the decomposition for “snail” its meaning is transformed to the shape of the snail’s shell. “Civilization” paired with “sphere” turns a simple sphere into planet Earth, etc.

   To further substantiate this point empirically, we enclose the results of simple word arithmetics (a+b) for examples of concepts from Fig 2. The connection “reflections of earth = sphere + civilization” is only made possible by the learned coefficients. When considering only the word embeddings, these are the resulting images (which do not resemble planet Earth whatsoever). Similarly, “snake = twisted + gecko” would yield these images without the appropriate coefficients, for the combination “snail = winding + ladybug” these are the resulting images, etc.

4. Finally, we would like to point out that there are many other lines of research, such as dictionary learning, that learn a combination over a set of dictionary vectors. Therefore, we do not believe that the fact that we are optimizing a combination of tokens is enough to classify our work as a form of word embedding arithmetics. As we showed above, our method significantly differs from word embedding arithmetics in its goal, implementation, and objective function.

Overall, we believe that our obtained results are non-trivial and profound, and are entirely different from those obtained by traditional word embedding arithmetics (see our mentioned ablation above). We show that SD can blend concepts in a way in which each concept plays a completely different role that can be image-based (shape, texture, color), semantic (the essence of the concept), or abstract (describing a property of the concept). This is, in our view, surprising and quite remarkable. One may not agree with our conclusions, but given the magnitude of evidence we present, which was increased by the various additional experiments conducted following the reviews (OOD experiments, another text-to-image model, robustness to the loss function, etc.), we find it hard to ignore.

If you found our previous response and this one to be mostly satisfactory, we kindly ask you to consider increasing your rating in recognition of the strengths and extensiveness of our evaluation, and the value of our work to the research community as the first interpretability work for text-to-image generative models.

Thank you for addressing my questions.

I have increased my score from 5 to 6.

Re. Downstream applications (with emphasis on bias mitigation): First, as the reviewer kindly pointed out, Conceptor enables human-understandable interpretations for diffusion models. Other than the useful downstream tasks (described below), we believe that interpreting diffusion models is an imperative, timely, and underexplored task on its own since these models are now being employed widely. The insights obtained by Conceptor can help both end-users and researchers to use these models responsibly while being aware of their shortcomings. Thus, we believe that interpreting generative models is, by itself, a worthy cause with much interest to our community.

With that, Conceptor also enables useful downstream tasks such as concept debiasing via semantic concept editing, that we are happy to elaborate on. As mentioned in the qualitative results section (Sec. 4.1), Conceptor allows one to control each element in the decomposition by manipulating its corresponding coefficient. The magnitude of the coefficient controls the extent to which the element is manifested in the image. Fig. 3 demonstrates such manipulations. For example, one can decouple the two meanings of a dual-meaning concept (see first two rows of Fig. 3) or remove an undesired property from the concept (e.g., “nerdy” for “professor”).

As mentioned in Sec. 4.3, this ability is also useful for bias mitigation. Once a bias is identified by Conceptor, the user can choose to “switch off” the biased elements by simply lowering their corresponding coefficients until an equal representation is achieved. Such examples can be found in Appendix H. For better readability, we have edited the Appendix to include the biased terms of Fig. 13, see Tab. 7. As can be seen, Conceptor enables debiasing while also maintaining the other image features and the original scene, i.e., Conceptor’s decomposition can surgically remove the biased property without harming the other features.

Additionally, following the reviews, we have added an experiment showcasing another downstream task. In Appendix J, we explore Conceptor’s ability to provide insight into the representation of Out of Domain (OOD) concepts, defined by a set of image samples. As we demonstrate, our method is able to extract meaningful and profound decompositions even for completely OOD concepts (e.g., the sloth plush in Fig. 16 is linked to the concepts “plush”, “fluffy”, and “knitted”, the monster toy in Fig. 17 is linked to “woolly” due to its wool-like fur, to the character “Domo” due to the shape of its body, and to a “crab” and a “shrimp” due to its unique limbs). Kindly refer to our answer to Reviewer HDAw for more details.

We thank the reviewer for providing the feedback and especially for aiming to validate our assumptions.

1. Kindly note that our method aims to provide an interpretation of the model’s latent representation of concepts. Therefore, faithfulness is a crucial component of our method (see Sec. 4), i.e., we strive to decompose the concept into elements that can reconstruct it. When applying the logic suggested by the reviewer to decompose concepts, faithfulness is not guaranteed and in fact, in most cases, it does not hold.

   For example, an alternative decomposition for "snail" under this logic could be "table" + "cake" since sim("snail", "table") = 0.7749, sim("snail", "cake") = 0.7847, and sim("snail", "table" + "cake") = 0.8110 > max{sim("snail", "table"), sim("snail", "cake")}. However, the resulting images for the decomposition of "table" + "cake" plugged into SD provide images that are entirely different from snails, such as the two examples in this link.

   Additional such examples can be found very easily, for example, one could replace "cashmere" in the decomposition for "camel" and decompose it into "giraffe" + "book" as follows: sim("camel", "giraffe") = 0.834, sim("camel", "book") = 0.7485, sim("camel", "giraffe"+"book") = 0.8408 > max{sim("camel", "giraffe"), sim("camel", "book")}, see examples of SD generation for this decomposition in this link. Other possibilities to replace "cashmere" in the decomposition include: "technology" (sim("camel", "technology")=0.786, sim("camel", "giraffe" + "technology")=0.866 > max{0.834, 0.786}), "cattle" (sim("camel", "cattle")=0.832, sim("camel", "giraffe" + "cattle")=0.873 > max{0.834, 0.832}) and many other words that are not semantically related to "camel" and do not produce a visual reconstruction.

   Note that our method learns weighted decompositions of elements to reconstruct the concept images (see Tab. 1, Fig. 4 of the paper for example). The examples presented in Fig. 2 are actual decompositions of images generated by SD, and reconstructed by Conceptor. The image reconstruction objective is the component that guarantees a connection between the decomposition and the concept itself, which otherwise would not exist, as demonstrated in the examples above.

2. We note that the inequality suggested by the reviewer, i.e., sim(A+B, C) > max{sim(A, C), sim(B, C)} often holds for any triplet of three words. This may be a result of increased similarity with the “blurred” word obtained by averaging two concepts. To validate this empirically, we selected 10,000 random triplets of words (A, B, C) out of the SD text encoder dictionary. In 85% of the cases, the inequality holds. This indicates that the proposed inequality does not necessarily suggest a better semantic similarity, nor that A+B is a decomposition of C.

3. We would like to point out that the CLIP text-text similarity scores (as calculated by the reviewer using CLIP ViT-B/32) appear to give abnormally high similarities for various words that are unrelated semantically. We drew 10,000 pairs of random words from SD’s vocabulary to demonstrate this point and found that the average similarity between these random words was 0.815. These high scores are somewhat surprising, especially given that a large portion of the tokens in this dictionary are not actual words but rather punctuation marks, emojis, etc. Additionally, the scores are often unintuitive. For example, the CLIP text-text similarity between "cat" and "house" is 0.79, which is higher than the CLIP text-text similarity between “cat” and “poodle” (0.74) (to reiterate the previous point, “cat” to ”house”+“poodle” is 0.81).

4. Finally, please note that the latest version of SD (SD 2.1) employs a different CLIP text encoder, i.e., OpenCLIP ViT-H (see Appendix A). When examining the CLIP text-text similarities in the OpenCLIP ViT-H encoder, the phenomena described in the review do not reproduce. First, we find that the CLIP text-text similarities between the elements are far lower than those indicated by the reviewer (with CLIP ViT-B/32), e.g., the similarity between "reflections of earth" and "civilization" is merely 0.2817, the similarity between "snake" and "twisted" is 0.365, etc. Second, the logic indicated for CLIP ViT-B/32 is not reproduced. In 2/6 triplets mentioned by the reviewer, the sum of the two elements does not increase the similarity:

   • For "snake": sim("snake", "twisted") = 0.365, sim("snake", "gecko") = 0.639, sim("snake", "twisted" + "gecko") = 0.627 < 0.639.
   • For "fear": sim("fear", "wolf") = 0.326, sim("fear", "scream") = 0.638, sim("fear", "wolf" + "scream") = 0.60 < 0.638.

   For this text encoder, the inequality holds for 65% of the random triplets as described in item 2.

We would happily address any other questions.

We appreciate the diligent efforts put into reviewing our work. Like you, we believe that performing sanity checks to validate the underlying assumptions is important.

May we inquire if you have had the opportunity to review our response from November 11?

Thank you,

The authors

Thanks the author response. I've verified and indeed in the CLIP text space, the triangular similarity relationships are very noisy and seem not reflect true semantic similarity. Therefore I'd raise my rating to 6. As a relevant piece of observation, my original comments on the CLIP triangular relationships are kept.

We thank the reviewer for the positive and comprehensive feedback. We are encouraged that the reviewer found our work to be a novel, flexible, and efficient solution for diffusion model interpretability. In the following, we provide a response to each of the questions in the review. Additionally, the modifications in the revision are marked in red, for your convenience.

Re. Size of test dataset: First, kindly note that the size of our dataset (188 concepts from diverse categories) is consistent with, and even significantly bigger than that of related works that tackle concept-based interpretability and concept analysis. For example, one of the most prominent works on concept-based explainability, ACE [1], presents a quantitative evaluation of 100 random ImageNet classes. Similarly, seminal works in concept personalization such as DreamBooth [2] performed evaluation on 30 diverse concepts, etc.

Importantly, note that we formed a diverse dataset including professions (to indicate biases and human-centered concepts), abstract concepts (including emotions and actions), very complex concepts (requiring hierarchical reasoning), and on top of those, we added 100 random concepts from the well-known general concept bank, ConceptNet, to ensure that our produced results are robust across a wide variety of rich and unrelated concepts. Note that ConceptNet contains a wide mixture of concepts from all categories, and is a very widely used knowledge graph. Across all these different types of concepts, Conceptor has consistently demonstrated an ability to produce faithful, meaningful, and robust decompositions.

[1] Towards Automatic Concept-based Explanations, Ghorbani et al. NeurIPS 2019

[2] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, Ruiz et al., CVPR 2023.

Re. Out-of-domain (OOD) images: Thank you for this question. While the goal of our work is to interpret the internal representations of the model (i.e., in-domain concepts) and images generated by the model for textual concepts, this question is indeed very interesting. Following the review, we have added a new Appendix (J) in which we examine three domains out of the DreamBooth dataset [2], with an increasing amount of distance from the training domain of SD. As can be seen, our method is able to extract meaningful and profound decompositions even for OOD concepts (e.g., the sloth plush in Fig. 16 is linked to the concepts “plush”, “fluffy”, and “knitted”, the monster toy in Fig. 17 is linked to “woolly” due to its wool-like fur, to the character “Domo” due to the shape of its body, and to a “crab” and a “shrimp” due to its unique limbs). Moreover, our optimization results in a virtual token $w^*$ that is able to generate new OOD images at least as effectively as unrestricted optimization of the virtual token. We find these results to be extremely reassuring.

Re. Analysis between Conceptor and interpolation methods: Could you please clarify which interpolation method you are referring to? Given more details, we would be more than happy to conduct the appropriate analysis.

We would happily address any other questions.

Thank you for considering our rebuttal! We share your enthusiasm for the OOD results, and deeply value the insightful suggestion that helped enhance our understanding of Conceptor.

Upon reviewing the clarification, we are still not sure we fully understand the proposed interpolation method and how it selects the condition embeddings. We derived two possible interpretations for the question which we address below, and we welcome your corrections if our understanding is not accurate.

1. Embedding interpolation as simple summation of word embeddings (i.e., summation of embeddings in the token space):

   Allow us to emphasize the following:

   • Recall that our method operates by learning a mapping function between each word embedding and a corresponding coefficient. In other words, our method can be viewed as a nonlinear mapping of the form: $w^* = \sum_i f(w_i) w_i$, where $f$ is a nonlinear function.

     Our extensive ablations (refer to Section 4.2.1, Table 3 under "without MLP") demonstrate the critical role of this novel approach in the success of our method. Without it (i.e., learning separate coefficients) the decompositions are not faithful. This ablation highlights the non-trivial nature of assigning appropriate coefficients to word embeddings to reconstruct the learned representation, affirming that a simple summation cannot substitute for Conceptor.

   • To demonstrate the important role of our learned coefficients empirically, we enclose the results of simple word embedding summation (a+b) for examples of concepts from Fig 2. The connection “reflections of earth = sphere + civilization” is only made possible by the learned coefficients. When considering only the word embeddings, these are the resulting images (which do not resemble planet Earth whatsoever). Similarly, “snake = twisted + gecko” would yield these images without the appropriate coefficients, for the combination “snail = winding + ladybug” these are the resulting images, etc. These examples demonstrate that even given the ground truth set of decomposition concepts, a simple summation cannot match our method.

   We believe that the combination of the aforementioned points effectively demonstrates the powerful and non-trivial ability of Conceptor to establish semantic links between concepts through our novel method.

2. Encoding interpolation after applying the text encoder (i.e., summation of embeddings in the conditioning space):

   In exploring this scenario, we observed that such interpolations result in non-meaningful images (perhaps due to OOD encodings). Here’s an example for “ladybug” + “winding”, and “twisted” + “gecko”.

We are happy to address any remaining questions, and appreciate the continued in-depth discussion.