Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis

We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.

W1: Related work

These two methods generally fall under optimization-based approaches. They primarily adjust noisy signals to enhance attention maps and strengthen semantic binding. Attend-and-Excite improves object presence by increasing the attention score for each object. SynGen performs a syntactic analysis of the prompt to identify entities and their modifiers, using attention loss functions to ensure that cross-attention maps align with the linguistic binding reflected in the syntax. These methods have indeed provided a solid foundation for future semantic binding work. Different from these two works, our training-free method ToMe is based on the text embedding additivity property. Instead of updating the latent space, we update the composite token embeddings. We further introduce two auxiliary losses, i.e. the entropy loss and semantic binding loss, to augment the ToMe performance. We will include more detailed introductions to these two methods in the introduction and related work sections in any future version.

W2: Syntactic information

In Section 4.1 on the implementation details, we mention that we use SpaCy[27] to detect relationships, following the approach outlined in the SynGen. More specifically, we use SpaCy’s transformer-based dependency parser to analyze the prompt and identify all entity nouns, including both proper nouns and common nouns, that are not serving as direct modifiers of other nouns. This process helps pinpoint entity nouns and their corresponding modifiers. We will further detail this process in the method and implementation details section.

W3: User study

As discussed in the General Response 1. In this paper, we use the ImageReward[72] model to evaluate human preference scores, which measures image quality and prompt alignment. However, since quantitative results may not fully capture these aspects, we now conduct a user study with 20 participants to enrich the evaluation. Here we compare our method ToMe with SDXL, SynGen, Ranni and ELLA. Each participant is questioned with 100 images, each generated from five methods and twenty prompts. The results are presented in Fig.16 of the rebuttal file. In the user study, we ask the participants to rate the semantic binding into 4 levels. We then calculate the distribution of each method over the four diverse levels. We can observe that ToMe better achieve the semantic binding performance by mainly distribute in the highest level 1, while the other methods struggle to obtain satisfactory results.

W4&Q4: Complex prompts generation

As illustrated in General Response 2, ToMe is not limited to only simple cases. In the main paper Table 2, we show experiments on the T2I-CompBench benchmark, where there are long captions with various prompt templates. There are also complex prompts T2I generations in Fig.12. To further show that our method ToMe can be applied to the T2I generation with complex prompts. We adopt the suggestions from the reviewer to generate with more complicated prompt templates, that includes longer prompts with multiple objects, multiple attributes or even multiple sentences. The generation results are shown in Fig.13 in the rebuttal file.

Q1: Global encoding text

SDXL is using a conditioning concatenation for text embeddings with two prompts. To keep our method ToMe applicable and generalizable to other T2I generative models, we keep both prompts the same. For example, with prompt “A blue cat and a green dog”, we assign this prompt as both the local and global encoding prompt and input it to the two text encoders and concatenate the text embeddings along the channels. And in our method, ToMe is updating the concatenated text embedding conditions with our proposed losses. We will add this to the implementation details.

Q2: Generalizability to other T2I models

As claimed in General Response 4, T2I models are facing generation limitations for various specific cases. We aim, as a future goal, to generalize our method to various generative models (such as DeepFloyd-IF, SD3, PixArt, etc.) to counter this issue. Current T2I generation models are built on various language models and vision-language models, such as CLIP and T5, and incorporate diverse architectures, including UNet, GAN, and DiT. We are also curious to see if the additivity property truly exists across all these cases, and whether this property is a generalizable feature.

Q3: Latent Distribution Drifts

You are correct; large gradients can cause significant changes in the latent space, which is a common issue in training-free semantic binding methods[7,41,56]. The primary goal of these methods is to update the latent or text embeddings to align the semantics between texts and images, often resulting in varying degrees of distribution change. To mitigate this issue, we only update the composite token representations during the first $T_{\text{opt}} = 0.2T$ time steps. This technique helps our method ToMe experience fewer distribution changes from the base model SDXL, as demonstrated in multiple cases in Fig. 5, Fig. 12 and Fig.13.

Q5: SynGen reproduction details

The original SynGen paper is based on the SD1.5 model. To adapt it to use SDXL as the base model, we utilize the 32×32 cross-attention map from the first three layers of the UNet-decoder, which serves as the most semantic layer, similar to the 16×16 cross-attention in SD1.5. We also performed a grid search to determine the optimal learning rate for SynGen in this context. All other details remain consistent with the original paper. After tuning the hyperparameters, the SDXL-based SynGen outperforms the SD1.5-based SynGen, as shown in Table 1 and Table 4. We will release the SynGen adapted code in the future for reproduction.

We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.

W1&Q1&L1: Complex prompts generation

As demonstrate in General Response 2, our method ToMe is not limited to only simple cases. In the main paper Table 2, we show experiments on the T2I-CompBench benchmark, where there are long captions with various prompt templates. There are also complex prompts T2I generations in Fig.12 in the Appendix. To further show that our method ToMe can be applied to the T2I generation with complex prompts. We adopt the complicated long caption templates suggested from the reviewer. The generation results are shown in Fig.13 in the rebuttal file.

In this case, we use our method ToMe in an iterative updating way. As an instance, we generate the fourth image example in Fig.13 (left down) with the prompt “a blue cat wearing sunglasses and a yellow dog wearing a pink hat”. To apply ToMe, we merge ‘yellow dog’ into token1 (dog*) and ‘pink hat’ into token2 (hat*). Following that, we merge token1 and token2 into token3. The noun phrases for computing the semantic binding loss corresponding to these three tokens are ‘a yellow dog’, ‘a pink hat’ and ‘a dog wearing a hat’, respectively. Then we update token1 and token2 by our losses proposed in Section 3.2.2.

W2: Inference time cost

As detailed in the General Response 3, we present a time cost comparison in Table 3 in Appendix C.4, where we compare performance using 50 inference steps using the SDXL model with float32 version on an A40 GPU. We demonstrate that our method does not significantly increase inference time while improving semantic binding performance. We further extend this analysis by measuring the time cost with 20 inference steps and various ToMe configurations, as shown in the Table below. We report the time cost (by seconds) along with BLIP-VQA scores across the color, texture, and shape attribute binding subsets. From this table, we can observe that using the token merging (ToMe) technique and entropy loss (Config.C in Table 2), our method achieves excellent performance with minimal additional time cost. Additionally, even with only 20 inference steps, our method, ToMe, maintains high performance with very little degradation.

W3: User study

Please also refer to the General Response 1. In this paper, we use the ImageReward[72] model to evaluate human preference scores, which comprehensively measures image quality and prompt alignment. However, since quantitative results may not fully capture these aspects, we now also conduct a user study with 20 participants to enrich the evaluation. Here we compare our method ToMe with SDXL[51], SynGen[56], Ranni[19] and ELLA[28]. Each participant is questioned with 100 images, each generated from these five methods and twenty prompts.

The results are presented in Fig.16 of the rebuttal file. Similar to our proposed GPT-4o benchmark (detailed in Appendix C.5 and Fig.10), we ask the participants to rate the semantic binding into 4 levels and calculate the distribution of each comparison method over these four diverse levels. We can observe that our method better achieve the semantic binding performance by mainly distribute in the highest level 1, while the other methods struggle to obtain user satisfactory results.

Q2: Mechanism to avoid semantic misalignment

In this paper, we propose two losses to avoid the semantic misalignment and cross-attention leakage among objects and attributes.

First, the semantic binding loss assist in purifying the token semantics. Building on the ToMe technique, we decrease the semantic binding loss to enhance the cross-attention. It encourages the newly learned token to infer the same noise prediction as the original corresponding phrase, reinforcing the semantic coherence between the text and the generated image. To ensure that the semantics of the composite tokens accurately correspond to the noun phrases they represent, we use a clean prompt as a supervisory signal. The semantic binding loss is computed separately, ensuring no interference among noun words.

In addition, we reduce the entropy loss to help ensure that tokens focus exclusively on their designated regions, preventing the cross-attention maps from becoming overly divergent. The experimental results comparing scenarios with and without entropy loss are presented in Table 2, Fig. 6, and Fig. 7. Both losses are functioning together to improve the semantic binding.

We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.

W1&Q1: Entropy loss

In Appendix C.3, we demonstrate that the information coupling of token embeddings is also reflected in the entropy of cross-attention for each token. In Fig. 9-(c), we calculated the entropy of cross-attention maps for each token and found that tokens appearing later in the sequence generally have higher entropy, indicating that their cross-attention maps are more dispersed.

Based on this observations, in this paper, we combine the entropy loss with our proposed ToMe approach. Reducing the entropy of these cross-attention maps of composite tokens helps to ensure that these tokens focus exclusively on their designated regions, preventing the cross-attention maps from becoming overly divergent, thus avoiding cross-attention leakage [64]. The experimental results comparing scenarios with and without entropy loss are presented in Table 2, Fig.6, Fig.7 and Fig.14 (in the rebuttal file).

As an example in Fig.14, the original SDXL (Config A) suffered from attribute binding errors due to divergent cross-attention maps. When only applying token merging (Config B), the co-expression of entities and attributes resulted in a dog wearing a hat in the image, but the attribute leakage issue remained due to the divergent cross-attention maps. When only applying the entropy loss (Config E), although the cross-attention maps corresponding to each token are more concentrated, they may focus on wrong regions. Only by applying both token merging and $\mathcal{L}_{ent}$ techniques (Config C), the cross-attention map of the composite token becomes better concentrated on the correct areas and thus leading to more satisfactory semantic binding of entities and attributes.

W2&Q2: EOT ablation study

The end token substitution (ETS) technique is proposed to address potential semantic misalignment in the final tokens of long sequences. As the [EOT] token interacts with all tokens, it often encapsulates the entire semantic information, as shown in Fig. 2. Therefore, the semantic information in [EOT] can interfere with attribute expressions, we mitigate this by replacing [EOT] to remove the attribute information it contains from the original prompts, retaining only the semantic information for each subject.

For example, as the cross-attention maps and T2I generation performance shown in Fig.15 (in the rebuttal file), when ToMe is not combined with the EST technique, the ‘sunglasses’ semantics contained in the EOT token cause the boy to incorrectly wear sunglasses. However, when combined with ETS, the unwanted semantic binding is relieved.

Q3: Efficiency

As detailed in the General Response 3, we present a time cost comparison in Table 3 in Appendix C.4, where we compare performance using 50 inference steps using the SDXL model with float32 version on an A40 GPU. We demonstrate that our method does not significantly increase inference time while improving semantic binding performance. We further extend this analysis by measuring the time cost with 20 inference steps and various ToMe configurations, as shown in the Table below. We report the time cost (by seconds) along with BLIP-VQA scores across the color, texture, and shape attribute binding subsets. From this table, we can observe that using the token merging (ToMe) technique and entropy loss (Config.C in Table 2), our method achieves excellent performance with minimal additional time cost. Additionally, even with only 20 inference steps, our method, ToMe, maintains high performance with very little degradation.

Dear Reviewer mo93:

Thank you for your valuable feedback on our work. Your constructive comments on our work are invaluable, and we genuinely hope to get feedback from you.

Regarding the weaknesses you mentioned, we include the corresponding experiments and discussion as follows:

1. Entropy Loss & Token Merging: Our experiments (Fig. 14 in the rebuttal file) show that combining these techniques effectively focuses cross-attention maps, improving semantic binding.

2. End Token Substitution (ETS): As illustrated in Fig. 15 in the rebuttal file, ETS is also can prevent unintended semantic interference, leading to more accurate attribute expressions.

3. Efficiency Compare: We've provided a time cost analysis , showing that our method improves performance with minimal additional time.

Your feedback is incredibly important to us, and we sincerely thank you for considering our rebuttal. We are more than happy to discuss them if you have any further concerns or questions.

Thank you again for your time and effort to review our work and looking forward to your response.

Best Regards, Authors of submission 1763

We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.

W1&Q2: Token Update

By training-free, we mean that our method ToMe does not involve training over datasets, unlike finetuning-based methods such as Ranni[19], ELLA[28], and CoMat[32]. Like other works (AnE[7], D&B[41], SynGen[56], etc.), methods that only backpropagate during inference time (on the generation of a single image) are considered to be training-free.

ToMe distinguishes itself from other optimization-based methods by updating token embeddings, whereas most existing methods (e.g., AnE[7], D&B[41], SynGen[56]) update the latent space using designed loss functions. Instead of back-propagating the gradients to the T2I model parameters or the latent variables of the previous timestep latent $z_t$, we compute the loss and apply the gradients to the composite token embeddings in each inference step during the first $T_{opt} = 0.2T$ time steps.

More specifically, at time step $t$, the latent variable $z_t$ and the text embedding $\mathcal{C}$ are fed into the diffusion model. After computing the loss, the gradient is back propagated to update $\mathcal{C}$. The latent variable $z_t$ and the updated text embedding $\mathcal{C}$ are then fed into the diffusion model again to predict the noise and obtain $z_{t-1}$.

Q1: Complex prompts generation

As we have demonstrated in the General Response 2, our method ToMe is not limited to only simple cases. In the main paper Table 2, we show experiments on the T2I-CompBench benchmark, where there are long captions with various complex prompts. There are also complex prompts T2I generations in Fig.12 in the Appendix.

To further show that our method ToMe can be applied to the T2I generation with complex prompts. We adopt the suggestions from the reviewer. That is to achieve T2I generation with prompts including multiple objects or backgrounds with multiple attributes. The first row of T2I generation results shown in Fig.13 (in the rebuttal file) demonstrate the effectiveness of our method ToMe under complex scenarios, where more objects and background with various properties are demanded.

L1: Generalizability

As we discuss in the General Response 4, we agree with you on this point. T2I models are facing generation limitations for various specific cases. We aim, as a future goal, to generalize our method to various generative models to counter this issue. Current T2I generation models are built on various language models and vision-language models, such as CLIP and T5, and incorporate diverse architectures, including UNet, GAN, and DiT. We are also curious to see if the additivity property truly exists across all these cases, and whether this property is a generalizable feature.