Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models

The paper addresses the challenges of using large language models (LLMs) as prompt encoders in text-to-image diffusion models. It identifies two primary issues: the misalignment between LLM training objectives and the requirements of discriminative prompt features in diffusion models, and the positional bias introduced by the decoder-only architecture of LLMs. To tackle these issues, the authors propose a novel framework called LLM-infused Diffuser, which leverages human instructions and linguistic token refiners to enhance text representation capabilities. They design an LLM-Infused Diffusion Transformer (LI-DiT) and demonstrate its superior performance over state-of-the-art models in both open-source and commercial systems.

This paper presents an investigation into the integration of Large Language Models (LLMs) into text-to-image diffusion models. It identifies issues with using LLMs as prompt encoders, namely misalignment between next-token prediction training in LLMs and the need for discriminative prompt features in diffusion models, as well as positional bias introduced by the decoder-only architecture. The authors propose a new framework to overcome these challenges and introduce an LLM-Infused Diffusion Transformer (LI-DiT) to leverage LLMs effectively in image generation tasks. The paper also discusses broader societal impacts and adheres to ethical guidelines.

In the context of text-to-image (T2I) generation, this work addresses the problem of leveraging representations from state-of-the-art decoder-based LLMs for conditioning image generation. The authors highlight challenges of leveraging existing LLMs, namely - misalignment in representations due to differing training objectives (autoregressive v/s non-autoregressive) and the effects of positional bias. The authors then propose ways to mitigate this misalignment via prompt changes (leveraging instruction tuning of LLMs), embedding ensembling techniques (for positional bias), and a new DiT based on this ensembled conditioning mechanism. The proposed observations and method show consistent qualitative and quantitative improvements on benchmarks across the board.

This work identifies two main reasons for degraded prompt-following ability in image generation with decoder-only transformers: the misalignment between pretraining objective and diffusion's need of discriminative prompt feature, as well as the intrinsic positional bias for decoder-only transformer. The solution is to enhance text representation capability and remove positional bias. This work also proposes a diffusion architecture conditioned on multiple LLMs. The proposed LLM-Infused Diffusion Transformer (LI-DiT) surpasses previous state-of-the-art open-source models in prompt understanding.

We sincerely appreciate the valuable time and effort all reviewers have dedicated to review our work. We are pleased to learn that the reviewers generally acknowledge and commend our contributions, including:

• The importance of LLM-infused diffuser in integrating decoder-only LLMs into the diffusion framework. (yHpu, NTY6, nE1N)

• The insightful analysis of the poor performance when adopting decoder-only LLMs as text encoders. (Nr61, yHpu)

• State-of-the-art performance. (Nr61, yHpu, nE1N)

• Solid ablation studies. (Nr61, yHpu).

We express our gratitude for the insightful and constructive suggestions provided by all the reviewers. We also show more high-quality generated images in the attached file. We will provide the link to our online platform in the final version. Here, we address the common concerns raised by the reviewers.

Q1: The contribution and novelty of our work. (Nr61, NTY6)

The core contribution and novelty of our paper is addressing the issue of poor performance when using powerful decoder-only LLMs as diffusion text encoders. We put more emphasis on investigating the inherent properties of T5-like encoder-decoder models and GPT-like decoder-only models serving as text encoders (paragraph starts with line41), and the reasons behind the poor performance of decoder-only LLMs (Section 2). The simple and effective LLM-infused Diffuser is designed based on novel and contributional observations. Similarly, the key discovery of Imagen[1] is that T5 series models are surprisingly effective, which leads subsequent advanced works to adopt the T5 series models. Our work further advances the application of LLMs within the diffusion framework, enabling the continual development of LLMs to better enhance the text understanding capabilities.

Q2: Comparing with other methods adopting LLMs. (Nr61, NTY6)

We argue that our approach has significant differences compared to the existing methods that utilize LLMs for prompt encoding.

Current works that utilize LLMs can be categorized into: 1) LLMs first generate the image layout based on the prompt, then the diffusion model completes the image based on this layout[2,3,4] mentioned by reviewer Nr61. 2) Training an extra adapter to align LLM with frozen diffusion models like SD1.5[5] and SDXL[6] for better prompt comprehension capabilities[7,8] mentioned by reviewer NTY6. 3) Leveraging LLMs as text encoders without specific design[9]. UNIMO-G[10] mentioned by reviewer NTY6 adopts MLLMs to take extra images as conditional information, which is not the same task as text-to-image generation.

The contribution of the LLM-infused diffuser does not conflict with the layout approach. The layout methods are usually adopted as the controllable plugin in specific areas like visual composition and number-sensitive tasks. They need to be used in conjunction with a powerful diffusion model. However, the generation quality of each object in the layout still relies on the prompt understanding capability of the diffusion model. When generating a single object with a complex description, the layout approach essentially falls back to directly using the diffusion model for generation. Meanwhile, the layout can only provide the spatial relationship of objects but can not guide the generation of complex object relationships such as a boy sitting on the shoulder of a man, while the LLM-infused diffuser can easily deal with it.

The adapter-based methods have not addressed the issues we observed. LLM4GEN[7], which was submitted to arxiv on 30 Jun 2024 after the submission of NeurIPS, also observed that the performance when adopting T5-XL can also easily outperform using larger 13B decoder-only LLMs. However, they did not provide any further analysis and directly used T5-XL as the final text encoder. The following table is the Table 4 in LLM4GEN.

Meanwhile, LI-DiT-1B also shows outstanding performance compared with LLM4GEN with fewer parameters.

Lumina-T2X directly introduces decoder-only LLMs into diffusion transformers, which is similar to our baseline setting.

Compared with the related works above, our method specifically identifies the issues and provides effective solutions. We will include these analyses in the final version.

• [1] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• [2] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models.
• [3] LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
• [4] Grounded Text-to-Image Synthesis with Attention Refocusing.
• [5] High-Resolution Image Synthesis with Latent Diffusion Models.
• [6] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• [7] LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation
• [8] SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models
• [9] Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers
• [10] UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion

This paper explores the role of LLMs in prompt encoding for diffusion models, especially in terms of the learned representation and effect of positional bias. The paper received 2 weak accepts, 1 borderline reject, and 1 borderline accept recommendations from reviewers. Positive points included interesting analysis, clear writing, good results, and extensive ablation studies. Negative points included limited novelty, some missing related work, some unclear comparisons, and lacking discussion on training and inference costs and scalability. Most of these concerns were addressed by the rebuttal. The reviewer who gave the borderline reject rating did not participate in the post-rebuttal discussions; however, the ACs felt that the reviewer’s major concerns were addressed by the rebuttal. Overall, after carefully considering the paper, rebuttal, and discussions, the ACs feel that the paper makes a positive contribution and recommend accept. Please incorporate the rebuttal points into the camera ready version.

Hi, thanks for the great work. I notice the cases shown in Figure 4, where the causal LLM generates the ongoing text while T5 repeats the text.

Denote the text as $X$, when we use LLM as a textual information extractor, we usually directly treat the hidden state of $X$ from the last layer of LLM (denote as $f_\theta (X)$) as the representation, and feed $f_\theta (X)$ to DiT model for image generation.

I have a question, the case in Figure 4 shows the continual generation process of LLM (given $X$ -> predict $Y$) rather than the feature of the representation $f_\theta (X)$. We just need to use $f_\theta (X)$ for DiT rather than $f_\theta (Y)$. What's the connection between the cases shown in Figure 4 and the hidden state we actually need?

I hope the authors can answer this question, many thanks!