Fantasy: Transformer Meets Transformer in Text-to-Image Generation

We appreciate for the valuable feedback and address the concerns as follows.

W1: Explanation for Chosen Benchmarks.

This is a good question. Several articles have noted that FID often misaligns with human evaluations, has limitations in assessing model quality, and is affected by factors such as image resolution and style. Moreover, since we use some generated data as a part of training source, there is a domain gap between Fantasy-generated images and the ground truth for FID-30K. To more thoroughly assess the images generated by Fantasy, we additionally choose HPSv2 as an evaluation metric (Table 1 in the paper), which aligns better with human evaluations.

W2: Scaling Laws for Fantasy.

We appreciate these concerns. Due to limitations in training data and computational resources, Fantasy’s training is currently limited to a 22-layer transformer-based MIM. Despite these constraints, Section 4.2 explores Fantasy’s scale-down performance, reducing from 22 layers to 6 layers. Additionally, Section 4.2 provides preliminary evidence that scaling laws validated for diffusion-based T2I models also apply to MIM-based T2I models. While diffusion models based on UNet face challenges with scaling, the scalability of Transformers has been well demonstrated in ViTs. Further exploration of scaling up the depth of MIM-based T2I models will be pursued in future work.

Dear Reviewer aiiy,

We sincerely thank you for your insightful and constructive feedback. We have incorporated your suggestions and responded comprehensively to in the comments.

We highlight our core contributions and distinguish our method from other two-stage training models for all reviewers. We address the Scaling Study’s significance and future directions, and clarify our benchmark selection as recommended.

We hope that our study will be well-received as a valuable contribution to the NeurIPS' focus on theory & application. We are available for any further discussions or inquiries the reviewers may have during the reviewer-author discussion period.

Best regards,

Authors

We appreciate the valuable feedback and address the concerns as follows.

W1: Discussion about the generated images.

There are currently many powerful diffusion-based T2I methods (e.g., pixart-$\alpha$, SDXL) that generate images with excellent visual appeal and details. However, our goal is not to develop a powerful MIM-based T2I method integrated with LLMs to produce highly detailed images that surpass diffusion models. Instead, we aim to create a lightweight T2I network with good text faithfulness by combining LLMs and MIM, and explore the limits of compressing such a T2I network.

Due to the different methods of image generation, Fantsy’s visual appeal is limited by the image resolution. We constrain the image resolution to 256×256 pixels to ensure efficient computation, preventing the generation of highly detailed images. Our hierarchical training strategy prioritizes text understanding and alignment over visual detail, a deliberate trade-off to achieve efficient training with limited resources. preventing the generation of highly detailed images. The task of marrying LLM with MIM to produce images with rich detail and visual appeal will be left for future work.

Additionally, Figure 4 in the attachment provides more examples of images generated by Fantasy, where we have highlighted the corresponding nouns and their attributes. These images demonstrate strong text faithfulness, proving that Fantasy, as a lightweight T2I network, is effective. Furthermore, there is potential for developing a larger-scale T2I network that combines MIM and LLM to generate richer and more text-aligned images in future.

W2: Clearer Description of Fantasy

We appreciate the careful review and will revise in the updated manuscript according to the feedback.

W3: Details about Efficient Training

We totally agree that faster training convergence can be influenced by various factors, such as pre-encoded data and training precision. Currently, we only test the training efficiency of different model precision and find that training speed is fastest with bf16 compared to fp16 and fp32. In the future, we will conduct more related experiments and include them in the updated manuscript.

W4: Supplementary Information for Generated Images.

We really appreciate this advice. In Figure 4 of the attachment, we provide more generated images with long text inputs. For better visualization, we highlight multiple entities and color attribute descriptions in the text inputs that appear in the generated images.

Q1: Explanation of Figure 1.

Thank you for pointing this out. We apologize for the confusion with the previous version and redraw Figure 1 as shown in the attachment. Figure 1 compares training costs and generation quality of different models. Circle sizes indicate image quality improvements. We categorize FID into three levels: FID < 7.5 as level one (Pixart-$\alpha$), 7.5 < FID < 15 as level two (ParaDiffusion, DALLE2, SDv1.5), and FID > 15 as level three (Fantasy, WÜRSTCHEN).

Q2: Why use Phi-2?

This is a good question. Phi-2 is a lightweight decoder-only LLM, it cannot be replaced by CLIP or T5, but can be optimized by using Phi-3 or other lightweight decoder-only LLMs in the future work.

CLIP’s text encoder employs an absolute positional embedding limited to 77 tokens, and LongCLIP reveals that the actual effective text length for CLIP is only 20 tokens, which significantly limits CLIP’s text understanding capabilities. While encoder-decoder LLMs have been explored in various works, such as Muse and Pixart-$\alpha$, decoder-only LLMs have proven to perform better in text understanding tasks. As mentioned in Section 2.2.2, fine-tuning LLMs is crucial to leverage their enhanced semantic comprehension and generalization potential. Due to constraints in training data and computational resources, only a lightweight LLM like Phi-2 allows us to efficiently perform full fine-tuning. During the fine-tuning stage, we compare full fine-tuning with LoRA fine-tuning (Figure 3 in the attachment). The full fine-tuning approach results in lower loss in both training and evaluation compared to LoRA tuning. This is likely because direct LoRA tuning can diminish the capabilities of LLMs in the T2I process, and larger-scale LLMs like T5 do not support efficient full fine-tuning within our resource constraints.

Q3: Inference speed of Fantasy.

Fantasy has a significant inference speed advantage over diffusion-based models and is currently comparable to existing MIM-based methods. In specific, Fantasy requires 1.2 seconds to infer a single image with 32 sampling steps, similar to Muse’s inference speed of 1.3 seconds, and 2 times faster than Stable Diffusion v1.4. Due to the use of discrete tokens and parallel decoding, Fantasy is more efficient during inference. However, inference speed is influenced by various factors, including sampling steps and optimization of inference code. In the current work, we aim to optimize the memory requirements of Fantasy, leaving the optimization for accelerating inference speed for future work.

Q4: Supplementary for Generated Images.

We really appreciate this advice. In Figure 4 of the attachment, we provide more generated images with long text inputs. For better visualization, we highlight multiple entities and color attribute descriptions in the text inputs that appear in the generated images.

Dear Reviewer soBM,

We sincerely thank you for your insightful and constructive feedback. We have incorporated your suggestions and responded comprehensively to in the comments.

We highlight our core contributions for all reviewers. We explain our use of Phi-2, image detail limitations. We also describe the Figure 1 in the paper, and compare the inference speed with others. Following your advice, we detail additional training steps and include more generated images with long texts in the PDF.

We hope that our study will be well-received as a valuable contribution to the NeurIPS' focus on theory & application. We are available for any further discussions or inquiries the reviewers may have during the reviewer-author discussion period.

Best regards,

Authors

We appreciate the valuable feedback and address the concerns as follows.

W1: Core Contributions of Fantasy.

We propose a novel T2I framework by combining decoder-only LLM with transformer-based image generators to achieve the balance of effectiveness and efficiency. Our approach aims to empower smaller models with stronger generative capabilities. We describe the details of our designs in the author rebuttal and will revise in the updated manuscript.

W2: Explanation for Two-stage Training.

This is a good question. Our two-stage training is quite different from others used in image synthesis. The two-stage approach not only aims for improved aesthetics but also enhances text faithfulness by utilizing different data and training components in each stage. We are the first to fully fine-tune LLMs using only a small amount of high-quality real and synthetic data during T2I training. Thanks to our proposed hierarchical training strategy, we achieve state-of-the-art results among other open-source transformer-based models and rank above average compared to diffusion models (Table 1 in paper). For more details, please see the Q2 response of the rebuttal for all reviewers. We will include the details in our revised version.

W3: Why aren’t the generated images more impressive than those from diffusion models?

While diffusion-based models are known for producing high-quality images with impressive detail and visual appeal, our focus with Fantasy is on developing a lightweight, efficient T2I network that integrates a lightweight decoder-only LLM effectively with a Transformer-based MIM for efficient training and long-form text alignment, rather than maximizing visual details. We achieve state-of-the-art results among other open-source transformer-based models and rank above average compared to diffusion models (Table 1 in the paper).

Fantsy’s visual appeal is mainly limited by the image resolution. We constrain the image resolution to 256×256 pixels to ensure efficient computation, which inherently reduces details. Our hierarchical training strategy prioritizes text understanding and alignment over visual details, a deliberate trade-off to achieve efficient training with limited resources. Future work will focus on images with higher resolutions and appealing details.

Figure 4 in the attachment provides more examples of images generated by Fantasy, where we have highlighted the corresponding nouns and their attributes. These images demonstrate semantic correctness and strong text faithfulness, proving that Fantasy, as a lightweight T2I network, is effective. We believe that our approach has significant potential for future enhancement.

Q1: Why use Phi-2?

We choose the lightweight Phi-2 for both efficiency and better image-text alignment. As mentioned in Section 2.2.2, fine-tuning LLMs is necessary to capitalize on their enhanced semantic comprehension and generalization potential. Due to constraints in training data and computational resources, only a lightweight LLM, Phi-2, allows us to efficiently perform full fine-tuning. We initially experiment with using LLaMA as the text encoder in the pre-training stage, but due to high costs and minimal loss reduction (Figure 2 in the attachment), we suspended this approach. Moreover, during the fine-tuning stage, we compare full fine-tuning with LoRA fine-tuning (Figure 3 in the attachment). The full fine-tuning approach results in lower loss in both training and evaluation compared to LoRA tuning. This is likely because direct LoRA tuning can diminish the capabilities of LLMs in the T2I process.

Dear Reviewer AAcU,

We sincerely thank you for your insightful and constructive feedback. We have incorporated your suggestions and responded comprehensively to in the comments.

We highlight our core contributions and distinguish our strategy from other hierarchical training strategy both in data and components for all reviewers. We explain our use of Phi-2 and image detail limitations as recommended.

We hope that our study will be well-received as a valuable contribution to the NeurIPS' focus on theory & application. We are available for any further discussions or inquiries the reviewers may have during the reviewer-author discussion period.

Best regards,

Authors

Dear Reviewer AAcU,

As the discussion phase ends today, we will not be able to further clarify potential additional concerns. We would be very grateful if you could respond to our further comment and offer us an opportunity to address any questions you might have!

Thank you again for your time and feedback!

Best,

Authors

We appreciate the valuable feedback and address the concerns as follows.

W1: Contributions Key Points.

Our goal is to investigate whether combining LLM with transformer-based generators can enhance generative models by achieving a balance between effectiveness and efficiency. We also explore the scaling-down performance of our model which aims to empower smaller models with stronger generative capabilities. We emphasize the core designs as detailed in the author rebuttal and will revise in the updated manuscript.

W2: Efficient T2I network.

Thanks for highlighting this point. We aim to integrate an LLM with transformer-based MIM for T2I generation and use the lightweight Phi-2 given limited resource. Fantasy is efficient in both framework design and training strategy, optimizing computation and resource use.

• Structure. Fantasy employ transformer-based generator and discrete tokens, which aligns with LLMs. By using the same structure for visual and text inputs, Fantasy enhances the effectiveness of the alignment between text and visual contents, so that smaller models can also achieve strong performance. Meanwhile, non-autoregressive Fantasy allows simultaneous computation of all positions during decoding, shortening dependency chains and speeding up image generation with better GPU utilization. More importantly, transformers’ scalable multi-layer structure enables efficient scaling, allowing for more compact image generators compared to diffusion models.
• Training. By using MIM and fully fine-tuning the LLM, Fantasy increases the efficiency of data usage, allowing training with minimal high-quality data. Our hierarchical training strategy accelerates model fitting: the first stage achieves concept alignment, and the second mixes real and synthetic images with instructions to fine-tune the LLM for long text understanding. Compared to other models, Fantasy requires less training data and time (Figure 1 in the paper).

W3: Difference of Hierarchical Training Strategy.

While multi-stage training is common in T2I models, Fantasy’s strategy is unique in its components and data. We are the first to fully fine-tune LLMs with minimal high-quality real and synthetic data for T2I training. Our hierarchical strategy achieves state-of-the-art results among open-source transformer-based models and ranks above average compared to diffusion models (Table 1 in the paper). For more details, see the Q2 in the author rebuttal.

W4: High-Quality Data.

Our data comes from 3 sources: filtered Laion-2B with strategies like the CLIP score for pre-training, and higher-quality real and filtered synthesized data for the second stage. During fine-tuning, we use data from SAM-LLaVA without blurred human faces and synthetic images from JourneyDB and internal collections. To learn object relationships, we use NLTK to filter out noun phrase prompts and retain captions over 30 characters to enhance understanding.

W5: HPSv2 Values for SDXL

We agree that more baseline values are necessary. As shown in Table 1 of the attachment, we add HPSv2 values for SDXL, and Fantasy outperforms SDXL on nearly all metrics despite having fewer parameters and lower training costs. Note that for fair comparison, we set the image resolution to 512x512. We will include the result in the revised version.

W6: Further Solutions for Scalability

This is a good question. Although there is a gap in visual appeal compared to diffusion models, Fantasy effectively generates objects with their attributes and relationships. We freeze the VQGAN due to limited resources but plan to scale up by expanding its codebook when resources allow. Magvit-v2 shows that a larger vocabulary improves generative quality by allowing more diverse visual tokens for images and videos. The accuracy increasing as the codebook grows from 1K to 16K (ours is 8K). We will include these description in our revised version.

W7: Meaning of Scaling Study

We acknowledge that exploring scaling-up limits is restricted by training resource. Our experiments focus on finding the smallest effective scale for a T2I generation model, which we find to be 6 layers in our framework. This setup can still represent objects, attributes, and relationships in captions. As research advances, replacing Phi-2 with a more capable lightweight LLM and using higher-quality data could enhance Fantasy’s visual appeal and text faithfulness at this scale. Although limited to a 22-layer transformer, our findings suggest that scaling laws for diffusion-based T2I models also apply to MIM-based models. Further exploration of these scaling limits will be addressed in future work.

Q1: Difference with MIM Methods

The differences between Fantasy and the leading MIM-based T2I model, Muse, are as follows:

• Motivation. Muse is the first to integrate MIM and LLM for T2I, while Fantasy aims at exploring a lightweight but effective T2I network.
• Model Components. Muse includes a frozen encoder-decoder LLM, a trainable VQ, and a MIM generator, along with a MIM-based super-resolution module. Fantasy utilizes a frozen VQGAN, a tunable decoder-only LLM, and a lightweight transformer-based MIM.
• Data Strategy. Muse is trained on Imagen with 460M text-image pairs in a single stage, and Fantasy is only trained on 16M text-image pairs with the mixture of real and synthetic data for two stage.
• Training Strategy. Fantasy freeze the VQGAN and is the first to fully fine-tune LLM with a two-stage training strategy. Muse train the VQ and freeze the LLM, which is suboptimal for merely aligning text embeddings with visual features.
• Model Scale. Fantasy targets a lightweight, efficient T2I network with models from 0.25B to 0.6B, while Muse offers larger models (0.6B and 3B) with examples only for the 3B model.

Dear Reviewer 5cW5,

We sincerely thank you for your insightful and constructive feedback. We have incorporated your suggestions and responded comprehensively to in the comments.

We highlight our core contributions and distinguish our strategy from other hierarchical training strategy both in data and components for all reviewers. We refine our efficiency definition, contrast our approach with MUSE, and incorporate baseline results as recommended. We also address the Scaling Study’s significance and future directions.

We hope that our study will be well-received as a valuable contribution to the NeurIPS' focus on theory & application. We are available for any further discussions or inquiries the reviewers may have during the reviewer-author discussion period.

Best regards,

Authors

Dear Reviewer 5cW5,

As the discussion phase ends today, we will not be able to further clarify potential additional concerns. We would be very grateful if you could respond to our further comment and offer us an opportunity to address any questions you might have!

Thank you again for your time and feedback!

Best,

Authors