LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts

We thank reviewer for providing insightful comments. Please find our responses addressing the specific queries below.

1. Missing information on human study.

Our approach is designed to overcome the limitations observed in diffusion models when confronted with lengthy and intricate textual prompts that involve multiple objects. Given the absence of an efficient metric tailored for this specific challenge, we conducted a thorough human study to offer a more comprehensive evaluation, as outlined in the main paper. The subjects for human study were recruited from a pool of academic students with backgrounds in computer science and engineering. To ensure an unbiased evaluation and maintain objectivity, the subjects participating in the evaluation remained anonymous with no conflicts of interest with the authors. Furthermore, we prioritized user confidentiality by refraining from collecting personal details such as names, addresses etc.

The entire survey process was anonymized, ensuring that there was no leakage of data to any user. Following a 2-AFC (two-alternative forced choice) design, each participant was tasked with selecting one image out of a pair of images, which best aligns with the given textual prompt and faithfully captures all the objects. Each pair of images was randomly sampled from a pool containing images from both baselines as well as our approach. This system mitigates biases which can originate from user-curated surveys. Further, the pool of images was shuffled before sampling each time. The system was further equipped with the feature of randomly flipping the position of the images, to remove the user-specific position bias in case any pair of images is repeated. Additionally, a 2-second delay between questions was introduced to facilitate more thoughtful decision-making.

The system yielded approximately 350 responses from 50 subjects, with each user answering an average of 7 questions. The results, as presented in Fig. 4 of the main paper, indicate that users predominantly selected images generated from our approach. This observation is further supported by qualitative analyses (Fig. 5 of the main paper and Figs. 9 and 10 of the appendix) and quantitative comparisons in Section 4 of the main paper and Table 1 of the revised appendix. We believe that this comprehensive approach and the subsequent results contribute significantly to the robustness and reliability of our outcomes.

2. This framework relies on recently proposed models like a strong LLM, CLIP, and an image composition model. The collection of previous works provides shallow techniques compared with them.

As per our understanding of the question, our methodology leverages the complementary strengths of diverse existing approaches, enhancing and refining each component in the process. Furthermore, our approach incorporates specific modifications to optimize the performance of each individual element. To prevent the Large Language Model (LLM) from generating results that lack coherence, we implement an interpolation mechanism. Additionally, we address potential issues in the output of the first-stage model through an iterative refinement scheme. In the context of the composition model, we introduce multi-modal guidance. The the impact of these components is visually demonstrated in Figures 6 and 7 of the main paper. Thus, our proposed layout generation, iterative refinement and multi-modal guidance solves for the weaknesses of the existing components and enhances the overall generalization towards complex textual prompts.

3. Comparison with DenseDiffusion Please refer to Fig. 10 of revised appendix for qualitative comparisons with DenseDiffusion [1]. Additionally, refer to the Table 4 below for quantitative comparisons. Our method compares favorably well against DenseDiffusion both qualitatively and quantitatively (in terms of PAR score).

Table4. Quantitative comparison with DenseDiffusion in terms of proposed PAR score.

4. Use of chatGPT logo.

We understand the reviewer's concerns. Thank you for pointing it out. We have removed the ChaTGPT logo in the new revised version of the paper. We have further removed the period from Sec. 3.1 title.

[1] Kim, Y. et al. (2023). Dense Text-to-Image Generation with Attention Modulation. http://arxiv.org/abs/2308.12964

We thank reviewer for providing insightful comments. Please find our responses addressing the specific queries below.

1. On efficiency of the proposed approach.

Our approach operates as a two-stage framework, relying significantly on the efficacy of the underlying models it employs. While some existing state-of-the-art methods exhibit faster performance, they encounter challenges in faithfully capturing every detail of lengthy and intricate textual prompts. Notably, these methods often fall short in generating all the objects specified in the prompt, as illustrated qualitatively in (Figs. 1 and 5) of main paper and (Figs. 9 and 10) of appendix. Our approach demonstrates a high abjectness score expressed as Prompt Adherence Recall (PAR) rate, indicating a notable alignment between the number of objects mentioned in the prompt and those actually present in the generated image (please see Sec. 4 in the main paper). Therefore, a discernible trade-off emerges between addressing complexity and ensuring the faithful reproduction of images. We provide a comparison of our method with existing approaches in terms of time efficiency and PAR score below.

Table 2. Quantitative comparison of our approach with baselines in terms of PAR score and inference times. Kindly note that all inference times are measured on a single Nvidia A100 GPU.

Recent works, such as those introduced by [1] and [2], have presented methods capable of scaling up diffusion inferences. Looking ahead, we anticipate incorporating these advancements in speed-boosting techniques into our ongoing research as a future work.

2. Unlike objects, bounding box locations cannot be refined. If the LLM predicts unreasonable bounding box layouts at the first stage, it cannot be corrected. Have the authors think of introducing the refinement of bounding box locations into the pipeline?

Our method is based on two stage process of layout generation and bounding box refinement. The spatial faithfulness of final generated image depends on the quality of layouts generated in the first stage. We observe that ChatGPT is effective in generating faithful layouts (please refer to Sec. A.7 of appendix) which exactly follow the details from the textual prompt. [3] further verifies this finding. However, in extreme cases, such as in Fig. 6 of the main paper and Fig. 12 of the revised appendix where object positions are not explicitly defined in the prompt, ChatGPT produces bounding boxes that do not coherently align with the prompts. We tackle such cases by generating multiple layouts, selecting the relevant ones via clustering and then interpolating them to a single optimal layout to lessen the influence of any outlier bounding boxes. We further provide a user-specific hyperparameter $\eta$ (refer to Fig. 3 of main paper) which can be controlled to vary the location of bounding boxes. Refinement of bounding boxes along with object refinement is an interesting future research idea, but is likely to further scale up the compute cost which may not be ideal for user experience.

3. How many layouts are needed to interpolate? How does the number of layouts for interpolation affect the results?

While there is no restriction on the number of layouts needed for interpolation, our final results are generated with the interpolation of 3 layouts. Please refer to Figures 11 and 12 (Sec A.6) of the revised appendix for a visualization of generated images with different number of layouts. We conclude that while interpolation plays a significant role on the final generated image in extreme cases where spatial locations are not clear from the prompt, it does not have a drastic impact on the final generated image when object locations are well-defined.

[1] Sylvain Gugger et al. "Accelerate: Training and inference at scale made simple, efficient and adaptable". https://github.com/huggingface/accelerate

[2] Zhang, Kexun, et al. "ReDi: Efficient Learning-Free Diffusion Inference via Trajectory Retrieval." ICML 2023.

[3] Lian, L., Li, B., Yala, A., & Darrell, T. (2023). LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models. arXiv preprint arXiv:2305.13655.

We thank reviewer for providing insightful comments. Please find our responses addressing the specific queries below.

1. Information on how Eq. (5) is used to guide the sampling process and clarity in writing.

We kindly refer the Reviewer to section A.8 of revised appendix for a detailed mathematical explanation on sampling process using guidance.

We will refine the writing for better clarity and ease of understanding in our final manuscript.

2. More intuition on the interpolation step.

We observed that proprietary LLMs such as ChatGPT (used in our approach) are good at following the details from the prompts and generating image blueprint in the form of box proposals as per the prompt. The same has been verified in [1]. However, we observed that in the multi-object settings where the spatial position of objects in unclear, ChatGPT has certain inherent bias for such cases. We conducted an experiment where we instructed ChatGPT to provide boxes for the prompt "In a living room with a dog and a cat sitting one on each side". Note that, in this prompt, the relative position of cat and dog is not clear. We observed that 60% of the times ChatGPT generated bounding boxes towards the left for the dog and only 30% times the dog was on the right. For the remaining 10% of the times, it had arbitrary position. To avoid such errors and to account for this bias, we instead prompt ChatGPT to output $K$ bounding boxes per object proposal. To exploit the above bias, we perform density clustering and choose the cluster containing the majority of the boxes. After this, we introduce the interpolation step to merge these filtered bounding boxes into a single optimal box. This potentially removes the effect of the outlier boxes if present. We have further presented a visual demonstration of the effect of such irrelevant boxes on the final generated image in Fig. 6 of the main paper and Fig. 12 of the revised appendix. Kindly note that this is the extreme case where the position of the objects is not clear from the text. So to further facilitate the user-specific control, we provide a hyperparameter $\eta$ which can be controlled to adjust the position of the boxes. Kindly refer to Sec. 3.3 of the main paper for the description of $\eta$ and Fig. 3 (main paper) for qualitative visualization of the effect of $\eta$.

3. Comparison with DeepFloyd.

As recommended, we compare our method with the DeepFloyd and present visual comparisons in Fig. 10 (section A.5) of the revised appendix. Additionally, we also present quantitative comparisons in terms of Prompt Adherence Recall (PAR) below in Table 1. We observe that even with a strong T5 text encoder, DeepFloyd struggles to generate coherent images with complex compositional prompts. We conclude that our scene blueprint augmented with iterative refinement is effective for generating coherent images from complex compositional prompts.

Table 1. Quantitative comparison of our approach with DeepFloyd.

4. Potential reasons why these text-to-image models are not able to generate images corresponding to simple compositional or complex prompts?

We agree with the reviewer's comments that current text-to-image diffusion models struggle to generate the image content even for non-complex compositional prompts. Experiments on DeepFloyd with a strong text encoder show that even stronger models struggle to generate coherent images. As per our intuition, there could be several factors that could contribute to inconsistencies in text-to-image diffusion models to generate images corresponding to simple compositional or complex prompts.

• Lack of generalizability towards diverse textual prompts can lead to images with missing details.

• The limitation could also stem from the lack of compositional prompts in the training dataset of text-to-image diffusion model inhibiting it from generalizing and generating diverse and accurate images for those prompts.

• Lack of location-aware training of diffusion models on complex compositional prompts can limit their ability to capture intricate relationships between objects, scenes, or attributes, leading to incoherent images.

[1] Lian, L., Li, B., Yala, A., & Darrell, T. (2023). LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models. arXiv preprint arXiv:2305.13655.