Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function

Firstly, we would like to express our sincere gratitude for reviewing our manuscript and providing valuable feedback. Below are our responses to the weaknesses (W) and Questions (Q).

W1: We acknowledge that attribute binding, as our main focus, is part of compositional generation. However, since Magnet performs outside the U-Net, it is plug-and-play for layout-based methods (see Fig. 2(c), layout-guidance+Magnet) or add spatial control (see Fig. 2(d), ControlNet+Magnet) to address the object relationships. Additionally, we have compared Magnet with GORS [1] (see Fig. 2(e) in PDF). Whether trained on color or complex datasets, GORS cannot demonstrate the anti-prior ability and disentangle concepts like Magnet. Meanwhile, it fine-tunes both the CLIP text encoder and the U-Net with LoRA is time-consuming to adapt to different T2I models, while Magnet is plug-and-play. Although the focus of this work is on attribute understanding, we sincerely hope that our work can provide new insights to the community and motivate further work investigating compositional understanding of T2I models.

W2: We have applied Magnet to SD 2.1 (another version of the CLIP text encoder), SDXL [2] (multiple text encoders), and PixArt [3] (T5 model) in Fig. 2(a)(b) in the attached PDF. Magnet also improves text alignment and image quality, showing the anti-prior ability. We will add these results to our final version.

W3: Magnet is completely automatic during evaluation. Though we describe that we gathered 641 nouns "manually", this procedure is one-for-all. We compute 641 embeddings once for each new text encoder and save them to the local path. In practice, given any prompt, only need to load this local file to search neighbors and encode a few new prompts (e.g., a prompt with 2 concepts, $\sim5\times2\times2$ new prompts are automatically constructed and need to be encoded if $K=5$). This process is fast as shown in Tab. 2 where Magnet has a negligible increase in runtime and memory usage compared to StructureDiffusion and Attend-and-Excite.

Q1: Your careful review is much appreciated. We will cite Stanza correctly in the main paper, the same as line 466.

Q2: Here is our justification for Magnet different from other works: [4], [6] rely on the layout constraints, which is an additional condition and not included in the vanilla diffusion models, while Magnet only requires the input prompt. We cited two similar works in lines 252-253. [5] controls objects in prompts, cannot improve the text alignment for attributes like Magnet. [7] composes different noise latents and needs multiple diffusion processes to obtain the target latents. Magnet operates the text embedding outside the U-Net and performs only one diffusion process, which adds negligible cost to the original model and is more efficient. We will discuss these mentioned works in related works in the final version.

Q3: We have evaluated on FID for two SD versions (v1.4 and v2.1). We follow the standard evaluation process and generate 10k images from randomly sampled MSCOCO captions. The result is SD v1.4 (19.04), +Magnet (18.92); SD v2.1 (19.76), +Magnet (19.20). This shows that Magnet will not deteriorate the image quality while improving the text alignment.

Q4: Thanks for recommending this impressive work. We found some ideas in this paper aligned with ours. But here are some major differences to point out:

1. It lacks interpretability. In our paper, we analyze the CLIP text encoder and the diffusion model by the 4-case experiment in Fig. 2(a), pointing out the entanglement of the padding embeddings. This has motivated our method and explains why Magnet works (i.e., using the binding vector to enhance the distinction between concepts and improve disentanglement);

2. It defines "positive and negative prompts" that are semantically contrastive, e.g., "young" vs. "old" for the "age" attribute (may need to manually specify per attribute). In Magnet, positive and negative attributes are obtained from the given prompt automatically, and no need to be semantically opposite. We also introduce unconditional prompts as pivots, which is suitable for attributes that are hard to define their opposite attributes (e.g., what is the contrastive adjective of the color "yellow"?);

3. Its direction vector is learned with a loss function to ensure robustness. According to the experiment setting, it uses fixed strength (edit delta). Differently, we propose the neighbor strategy and adaptive strength to improve the vector estimation. Encountered with a new attribute, [7] needs a training process to seek this direction, while Magnet can be directly applied.

In the final version, we will discuss this concurrent work [8] in more detail.

Your positive comment on the contribution of this work is much appreciated. We apologize for any difficulty you may have experienced in following this paper. Hope the following example and the illustration in Fig. 3 in the attached PDF will help you to understand these $\mathcal{P}$-terms and our method.

Consider an input prompt $\mathcal{P}=$"a blue book and a red cup", we use an off-the-shelf parser to extract concepts "blue book"(A_1\\&E_1) and "red cup"(A_2\\&E_2). The term $\mathcal{\tilde{P}}$ refers to a new prompt that involves only one object (in line 110, described as "out of the current context of $\mathcal{P}$"). For instance, when operating the object "cup" (i.e., $i=2$), we define positive concept $\mathcal{\tilde{P}}^{pos}_2$="red cup"(A_2\\&E_2), negative concept $\mathcal{\tilde{P}}^{neg}_2$="blue cup"(A_1\\&E_2), and unconditional concept $\mathcal{\tilde{P}}^{uc}_2$="cup"(\varnothing\\&E_2). The function $\mathcal{F}$ extracts the embedding w.r.t. a word in one specific prompt (illustrated by the red box). Notice that the same word in different prompts with varied contexts will produce different word embeddings, i.e., $\mathcal{F}(E_2,\mathcal{\tilde{P}}^{pos}_2)\neq \mathcal{F}(E_2,\mathcal{P})$. Based on this fact, we introduce Eq. (1)(2) to estimate the binding vectors $v^{pos}_2,v^{neg}_2$ for "cup". The vector $v^{pos}_2$ identifies the direction towards "red", while $v^{neg}_2$ is the direction towards "blue", specifically for "cup". Similarly, we obtain the binding vectors $v^{pos}_1,v^{neg}_1$ specifically for the object "book"($E_1$).

When involving neighbors, we compute $K$ objects close to the "cup" embedding in the feature space (lines 129-132), denoted $\\{B^{(2)}\_k\\}^{K}\_{k=1}$, where superscript $(2)$ refers to the current object "cup" $i=2$. Next, we replace "cup" in three constructed prompts with each neighbor object. For instance, the second neighbor "mug" ($B^{(2)}_2$, where subscript refers to $r=2$) will produce the positive concept $\mathcal{\tilde{P}}^{pos}_2$="red mug"(A_2\\&B^{(2)}_2), negative concept $\mathcal{\tilde{P}}^{neg}_2$="blue mug"(A_1\\&B^{(2)}_2) and unconditional concept $\mathcal{\tilde{P}}^{uc}_2$="mug"(\varnothing\\&B^{(2)}_2). The same process is for the remaining neighbors to obtain $K$ positive and $K$ negative vectors. The final binding vectors $v^{pos}_2,v^{neg}_2$ used to modify "cup" are averaged as Eq. (5)(6).

The adaptive strength in Eq. (3) improves robustness for practical use. For the object "cup" ($i=2$), we extract the first [EOT], i.e., $\mathcal{G}(\mathcal{\tilde{P}}^{pos}_2)$, and the last padding embedding, i.e., $\mathcal{H}(\mathcal{\tilde{P}}^{pos}_2)$, to compute $\omega_2$ and two strengths $\alpha_2,\beta_2$. Note the used prompt to obtain $\omega_2$ is not the input prompt $\mathcal{P}$ but the constructed positive concept $\mathcal{\tilde{P}}^{pos}_2$. Our motivation for this strategy is described in lines 116-119. Similarly, $\omega_1, \alpha_1,\beta_1$ are calculated for the object "book" ($i=1$).

Finally, we modulate the original object embedding $c\_{E\_i}$ using the adaptive strengths $\alpha_i,\beta_i$ and the estimated positive and negative vectors $v^{pos}_i,v^{neg}_i$, denoted $\hat{c}\_{E\_i}$ as Eq. (4). Note Magnet does not manipulate the cross-attentional activations as existing works do. As described in Section 3.3, Magnet only modifies the embedding of each object word in the input prompt and treats the denoising U-Net as a black box to generate the image.

Magnet provides a simple but effective way to identify the attribute direction for each object, as we said in line 98, to attract the target attribute (i.e., positive vector) and repulse other attributes (i.e., negative vector). Experiments show Magnet disentangling different concepts and improving attribute alignment (see Fig. 4,5 in the main paper).

In Fig. 2(b) in the attached PDF, we show Magnet is not limited to the CLIP text encoder, improving the text alignment and synthesis quality of PixArt [1], which uses T5 as the text encoder.

We sincerely hope that the above explanation will clear up the confusion. We will improve the method section to make this paper more readable.

[1] PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis

We sincerely appreciate your thorough review of our manuscript and the insightful comments provided. Here are our detailed responses to the identified Weaknesses (W) and Questions (Q).

W1: Our comparison experiment has been designed to evaluate whether each method achieved binding. In Tab. 1, attribute disentanglement is assessed by asking "Which image shows different attributes more clearly?". Only the method achieving the best binding will be voted, otherwise evaluators tend to choose no winner. In Tab. 2 attribute alignment (Attr.), human annotators annotate a precise number of successful bindings (whether one object presents the desired attribute). This is done separately for each method, without comparing the others. Lines 161-178 have explained each criterion in detail.

W2: Automatic results in Tab. 2 only assess object existence (see Q3). Magnet is inferior to Attend-and-Excite (Attend) since it optimizes the noisy latent to encourage the model to attend to all objects. As discussed in limitations, we acknowledge Magnet suffering from missing objects. Though designed to enhance attribute alignment (has outperformed all baselines in Tab. 2), Magnet shows improvement in object existence (Det. $+5$ on SD), which is a bonus of disentangling different concepts (see Fig. 6(b)). This still holds when integrating Magnet with Attend. We tested Attend+Magnet on CC-500 via GroundingDINO. The Det.($\uparrow$) result is 87.7, compared to 84.3 w.r.t the original Attend. Attend+Magnet can encourage all objects and improve attributes simultaneously (see Fig. 8). Since there is no available API at this moment, we cannot evaluate Magnet with GPT-4o. But we will definitely try out GPT-4o whenever possible.

Q1: The 4-case experiment is deeply connected to our method. Lines 64-78 and Fig. 2(a) focus on comparing how the context information in the word embedding affects generation (i.e., from case 1 to 2, or case 3 to 4), and how paddings' context information affects (i.e., from case 1 to 3, or case 2 to 4). Appendix A.2 provides a detailed analysis of these fine-grained cases. We discern that the attribute information is rich in word embeddings but entangled in padding embeddings, and modifying these word embeddings will not change the image layout as significantly as modifying padding embeddings. These have motivated us to introduce the binding vector, adaptive strength, and neighbor strategy (see lines 439-443).

Q2: We did consider modifying the padding tokens, but for the following reasons it didn't work:

1. These padding embeddings strongly entangle different concepts, especially given complex prompts. It is hard to manipulate one specific object from these paddings;
2. As shown in Fig. 2 and Fig. 12 in the main paper, modifying the padding (from cases 1 to 3) will change the image significantly compared to modifying the word embedding (from cases 1 to 2). We consider not changing the non-attribute part can better maintain the pre-trained model's capability;
3. As shown in Fig. 1 in the attached PDF, the padding embeddings are less activated than word embeddings, especially in U-Net's last two upsampling blocks and the later diffusion steps, which are more crucial for generating semantic features.

Q3: In our comparison experiment, we input all object words in the prompt to GroundingDINO. For example, given the prompt "a blue apple and a green vase" with two objects, GroundingDINO's input is "apple . vase ." without any adjective. We have considered inputting adjectives, i.e., "blue apple . green vase .". However, no matter what color the apple is (e.g., blue, red, or green), GroundingDINO in both settings will detect it. That's why we use GroundingDINO to assess object existence only, but not attribute alignment (see lines 177-178). Appendix Fig. 15(c) presents GroundingDINO's failure cases. We further conducted the following experiment to make our judgment sound. Given 50 images (generated by the prompt "a blue apple and a green vase"), GroundingDINO detected $38$ (w/o adjective), $38$ (w/ adjective), human annotated $\sim 20$ "blue apple", maximum is 1 per image. This comparison shows that GroundingDINO is not reliable for measuring attribute alignment.

Q4: We agree with your idea that the CLIP's embedding does not effectively bind positional information to the desired token. However, this makes an analysis like Fig. 1 difficult - we don't know which token to identify. We conjecture that the positional information is used in the early diffusion steps, where all tokens have relatively small attention values (see Fig. 1 in the attached PDF). In this case, the entangled padding embeddings may affect generation. This also explains why Magnet modifying the word embeddings cannot address the positional problem. However, we can integrate Magnet with layout-based methods to handle position (see Fig. 2(c) in the PDF). Except for colors, Magnet is capable of binding other attributes (see Fig. 2(a) row 3 and 2(b) row 2 in the PDF). The final version will include a detailed discussion of positional relationships and more examples.

Finally, the two textual-based image editing works are impressive. We will cite the given papers and discuss the differences in the final version.

[1] T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation

Thank you to the authors for their detailed response. I have carefully considered the points raised.

[W1] "Our comparison experiment has been designed to evaluate whether each method achieved binding. In Table 1, attribute disentanglement is assessed by asking, 'Which image shows different attributes more clearly?' Only the method achieving the best binding will be voted for; otherwise, evaluators tend to choose no winner."

While I understand the authors' approach, I still believe that this method inherently involves a comparison. Evaluating A vs. B seems likely to yield different insights than evaluating A and B separately. The authors suggest that both approaches could lead to similar results, but I respectfully disagree, as I believe pairwise comparisons and individual method evaluations may not produce the same outcomes.

[W2] "Since there is no available API at this moment, we cannot evaluate Magnet with GPT-4o. But we will definitely try out GPT-4o whenever possible."

I appreciate the authors' intent to evaluate with GPT-4o when possible. However, isn't the GPT-4o API already available?

[Q3] "This comparison shows that GroundingDINO is not reliable for measuring attribute alignment."

I understand this concern, and I recognize the challenges it presents. It is unfortunate that there currently isn't a more robust evaluation method beyond manual inspection. I had hoped that GPT-4o might offer some supplementary capabilities in this area.

Given these unresolved concerns regarding the evaluation, I will maintain my score as a weak accept.

We sincerely appreciate your careful review and the valuable suggestions provided. Our responses to the Weaknesses (W) and Questions (Q) are outlined below.

W1: Magnet is fully automatic during evaluation and does not require manual definition of these positive/negative attributes. We have introduced these components to ensure robustness. Various ablation experiments have proved the effectiveness of each component, including the parameter $\lambda$ (see Appendix Fig. 17), the neighbor strategy (see Fig. 18), and the positive and negative vectors (see Fig. 19). Meanwhile, our method can be simplified by using fixed strengths and estimating the binding vector without neighbors (i.e., by the object itself). Furthermore, as you pointed out, it is not expensive at all.

W2&Q1&Q2: Your comments on our abstract are very valuable to us. The following is our explanation of the terms used in the abstract:

1. the term "blended text embeddings" in other words means "text embeddings are blended", with reference to [1] "tokens in the later part of a sequence are blended with the token semantics before them";
2. the term "attribute bias phenomenon" appears in line 60, i.e., our discovery based on the analysis of the CLIP text encoder, rather than the "attribute binding problem".

We will check all the terms in the abstract, and improve it to be clear and professional.

W3: This sentence can be explained with an example in Fig. 1(b) (apology for its blur, we have provided a high-resolution one in Fig. 1 in the attached PDF). For the object "banana", both word and EOT embeddings show high similarity on the color "yellow" but low similarity on "blue", as if "banana" prefers the attribute "yellow" but dislikes "blue". We will improve the analysis section and clarify all definitions in the final version.

W4: The 4-case experiment is deeply connected to our method. As discussed in lines 439-443, our motivation is based on several observations between 4 cases. Specifically, case 1 is a reference case w.r.t. standard generation, case 2 shows how the context information of word embeddings affects generation, case 3 shows how padding embeddings' context information affects generation, and case 4 tests whether the model can capture attribute information in adjectives (described in lines 428-429). Using all 4 cases would be more convincing to introduce our proposed binding vector and adaptive strength. In addition, we add 3 new cases in the Appendix to justify the information-forgotten problem in the latter padding embeddings (see Fig. 12, cases A-C). We will add more descriptions of this experiment to the main paper and improve readability.

Q3: We define "negative attributes" as adjectives in the given prompt that do not belong to the current object. For instance, given the prompt "a green apple on a wooden table and a red chair" with three concepts, when operating the object "apple", its positive attribute is "green", then negative attributes are "wooden" and "red" (belong to "table" and "chair", respectively). It should be noted that negative and positive attributes do not need to be semantically opposed to each other. Magnet is not limited to colors and can work on prompts with extensive attributes (see Fig. 2(a) row 3 and (b) row 2 in the attached PDF). We will add more non-color examples in the final version.

Q4: We assure that our experiments are completely fair through consistent control of all settings (e.g., the same SD&CLIP version and seeds). Its poor performance can be explained by our discovery in the text encoder: the improper binding is caused not only by the word embeddings, but also by the entangled padding embeddings. StructureDiffusion simply separating the word embeddings of different concepts is insufficient to offset the entanglement in the padding embeddings. Conversely, we introduce binding vectors to reinforce the difference between each concept. This improves disentanglement. Additionally, StructureDiffusion also performed poorly in [2]'s comparison experiments - Structure Diffusion may even get worse scores than Stable Diffusion.

Finally, we sincerely apologize for Fig. 1 and will fix it in the final version. Your feedback is very helpful to us. We will improve our paper to strengthen our contribution, reducing visual examples and adding quantitative results (e.g., FID-10k: SD 19.04, Magnet 18.92).

[1] Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis

[2] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models