Denoising Autoregressive Transformers for Scalable Text-to-Image Generation

1. Q: My main concern is that whether the proposed method can truly scale to higher-resolution images. So, for resolutions like 512 or 1024, can it really be trained and sampled efficiently using this framework? Although VAR also adopts a similar progressive generation approach, it uses different resolutions in its generation trajectory, whereas DART’s generation trajectory consists of full-sized, noisy images. I would like to see the experiments on higher resolutions (512 and 1024).

   • We thank the reviewers for bringing up this valuable question. Our DART can scale to high-resolution images of resolution 512 and 1024 efficiently. Instead of learning a fixed resolution of images, one can learn a joint distribution of $p_\theta(\{x^i_0\}_{i=1}^N)$ where $x^i_0\in \mathbb{R}^{K_i\times C}$ is $x_0$ with a different resolution. Following this formulation, a single DART can model multiple resolutions by representing each image with its corresponding noisy sequence $\{x^k_t\}_t$ separately, then flattening and concatenating these sequences for sequential prediction. Fig. 12 illustrates the denoising generative process of images with both resolutions $256 \times 256$ and $512 \times 512$.
   • By this means, DART also benefits from progressive generation. Vanilla DART on resolution $256 \times 256$ uses 16 denoising steps. Since we follow this progressive generation paradigm, higher-resolution image generation is conditioned on generated $256 \times 256$ image, thus less denoising steps are needed, which greatly saves computation. Therefore, DART may be even more scalable to higher-resolution generation than standard diffusion models, as DART can generate realistic images in very limited denoising steps. In particular, we apply 4 denoising steps for image of $512 \times 512$ followed by 2 denoising steps for $1024 \times 1024$ image. We follow the default setting and set the patch size as 2 for $512 \times 512$ and $1024 \times 1024$ images, which ends up containing 1024 and 4096 tokens for each image separately. We finetuned a pretrained $256 \times 256$ DART on high resolution data. Fig. 14 and 15 show the text-to-image generation results of DART on resolutions of $512 \times 512$ and $1024 \times 1024$.

2. Q: The paper presents many different models (DART, DART-AR, DART-FM, Kaleido-DART), without providing a sufficient explanation of these models, such as their strengths and weaknesses. After reading this paper, I do not know how to choose between them since there is no consistent winner in the provided experiments. I would like to see a table to clearly describe differences, strengths, and weaknesses of each DART variant.

   • We have included Appendix F in updated manuscript to further clarify the differences and connections between the variants of DART. Table below lists the major comparison between DART, DART-AR, and DART-FM. DART-AR applies a token-wise autoregression instead of block-wise autoregressive in vanilla DART, which conduct denoising generation in a more fine-grained granularity. DART-FM, on the other hand, keeps the block-wise autoregression while introduces an additional flow network to conduct flow-mating-based refinement for generated tokens. Both DART-AR and DART-FM improve the performance over vanilla DART. In general, DART-FM demonstrates a better tradeoff between generation quality and inference efficiency.

     Model	Attn Mask	#AR steps	#FM steps
     DART	Block-wise	16	0
     DART-AR	Causal	4096	0
     DART-FM	Block-wise	16	1600

   • Kaleido-DART is for multimodal text-image generation, which integrates next-token prediction for text and next-denoising prediction for images (proposed in our DART). In image generation, it can seamlessly adapt all three variants of proposed methods: DART, DART-AR, DART-FM. Similarly, Matryoshka-DART, which enables multi-resolution generation, also adapts to all the three DART variants. Since in Matryoshka-DART, one can simply concatenate high-resolution image tokens after the low-resolution ones, which doesn’t affect the denoising modeling in these variants.

We thank the reviewer for detailed comments.

We understand your major concern is on the capability of scalable generation of our proposed DART. We want to highlight that, from the perspective of this work, the scalability of a visual generative model contains two aspects: (1) use compute in an efficient way especially as the model size grows, and (2) have the ability to easily generate images at any resolution.

For (1), DART is designed to leverage the whole denoising trajectory and the pre-computed hidden states, so that it can generate realistic results with less steps than standard diffusion model. Also, as shown in Figure 9b, DART benefits trivially from scaling up the model size, which demonstrates the scalability of proposed framework. For (2), we have further showcased the capability of DART in generating images at high-resolution of $512 \times 512$ and $1024 \times 1024$ by trivially connecting the denoising sequence of different resolutions without additional architectural/model change.

We respectfully notice that we may have discrepancy regarding the extensive terminology of scalable generation. And if this helps, we would be willing to change the title to “DART: Denoising Autoregressive Transformers for Visual Generation” to avoid the ambiguity of “scalable” in different meanings. We believe this would not diminish our core contribution, our proposed method paves a way for a novel generative model with a more unified framework beyond diffusion and autoregressive models.

Please find below point-by-point response to your comments.

• "It does not convincingly demonstrate that the DART framework itself can directly model higher resolutions. In fact, it appears that the model is essentially performing simple super-resolution..."

  • We want to highlight that our Matryoshka-DART for high-resolution generation is different from super resolution. Since the generation of high-resolution images are not dependent on the generated clean low-resolution image but the whole trajectory of noisy images. The low-resolution clean image is the output which is never fed back to DART. By this means, Matryoshka-DART naturally mitigates potential error propagation in conventional super-resolution models.
  • Also, Matryoshka-DART models different resolutions by concatenating the tokens into one sequence, which doesn’t require extra efforts in handling shape change at boundaries as in traditional cascaded generation. Such a strategy further offers flexibility of modeling images of any ratios. We showcase results on squared images for simplicity purposes. But the model can be easily extended to other ratios by modeling the sequence of tokens. This model is also not restricted to model the same image of different resolutions.
  • We also want to kindly point out that it’s a widely adapted paradigm for visual generation that the model is first trained on low-resolution (e.g., $256 \times 256$) and then tuned on high-resolution images (e.g., $512 \times 512$ and $1024 \times 1024$), like DALLE [1], SD3 [2], PixArt [3]. Here, we follow the mainstream of scalable generation to high-resolution. We adapt the pretraining on $256 \times 256$ and finetuning on higher-resolution strategy for training efficiency considerations.
  • The model leverages its pretrained capability for low-resolution generation and the denoising trajectory of low-resolution images, enabling efficient training. However, it can also be trained from scratch for higher-resolution generation. Notably, after fine-tuning on higher-resolution data, DART retains its ability to generate low-resolution images, unlike conventional diffusion models, which are typically limited to high-resolution generation after fine-tuning.
  • Finally, the high-resolution generation are not of low quality; in fact, they are relatively sharp compared to some existing works. While certain images may appear to lack fine details, this is primarily due to insufficient training during the rebuttal phase. In the next version, we will include a detailed comparison on ImageNet 512 to further demonstrate the model’s capabilities.

• "...This suggests that the quality of higher-resolution generation is not on par with using a dedicated super-resolution model, such as RealESRGAN, to upsample the images directly..."

  • We would like to clarify that our model, DART, is not a super-resolution model, as discussed earlier. While models like RealESRGAN are designed specifically for super-resolution tasks, generating high-resolution outputs from given low-resolution images, DART is fundamentally a generative model. Consequently, comparing DART to models like RealESRGAN may not be entirely appropriate. Additionally, DART is currently trained using only an L2 loss function. Incorporating GAN-based loss during training could further enhance its ability to produce sharper and more visually compelling results, which we plan to explore in future work.

• "Finally, the authors admit that DART cannot directly adopt existing techniques from diffusion or flow-based models, such as DPM-Solver, nor do they compare against such methods. This further limits DART’s potential as a scalable framework for the community to build upon."
  • We acknowledge that the current implementation of DART is not directly compatible with solvers developed for diffusion and flow-based models. However, as demonstrated in Proposition 1, there exists a bijective mapping between the non-Markovian diffusion process (DART) and the Markovian diffusion process (conventional diffusion). This mapping opens up the possibility of adapting diffusion solvers by transitioning the DART denoising process to a Markovian framework. Far from limiting DART’s potential for community-driven development, this adaptability presents opportunities to further enhance the framework, fostering a deeper unification of diffusion and autoregressive models.
  • Unlike conventional diffusion models, DART features a consistent training and sampling denoising trajectory, enabling the optimization of the generation process in a single forward pass. Exploring the development of a learnable scheduler or integrating advanced solvers into the denoising trajectory presents valuable opportunities for the community to further enhance and build upon this framework.
  • DART introduces a novel class of generative models that can be built on any non-Markovian process. In this paper, we focused on the basic setting of independent noise. Extending DART to incorporate more general non-Markovian processes represents a promising direction for future research, with the potential to significantly enhance model performance.

References: [1] Ramesh, Aditya, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. "Hierarchical text-conditional image generation with clip latents." arXiv preprint arXiv:2204.06125 1, no. 2 (2022): 3.

[2] Esser, Patrick, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi et al. "Scaling rectified flow transformers for high-resolution image synthesis." In Forty-first International Conference on Machine Learning. 2024.

[3] Chen, Junsong, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang et al. "Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis." arXiv preprint arXiv:2310.00426 (2023).

[4] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205. 2023.

[5] Ma, Nanye, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. "Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers." arXiv preprint arXiv:2401.08740 (2024).

[6] Li, Tianhong, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. "Autoregressive Image Generation without Vector Quantization." arXiv preprint arXiv:2406.11838 (2024).

[7] Zhao, Wenliang, Minglei Shi, Xumin Yu, Jie Zhou, and Jiwen Lu. "FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner." arXiv preprint arXiv:2409.18128 (2024).

[8] Tian, Keyu, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. "Visual autoregressive modeling: Scalable image generation via next-scale prediction." NeurIPS 2024.

1. Q: The comparison is not comprehensive. DART integrates diffusion in the AR model, using the AR to model the diffusion. MAR uses diffusion for the AR model, using the diffusion to model the tokenization. Even though their two different approaches, the comparison should include MAR, as MAR has already shown good generalization ability on multimodal generation. It is becoming a trend. With this comparison, we can see if DART has this potential.

   • DART is connected with MAR as both methods integrate diffusion and autoregressive models. MAR autoregressively predicts a latent for each patch and applies an additional diffusion model to generate the image tokens conditioned on the predicted latent. Whereas our DART integrates the denoising process into the autoregressive modeling and doesn’t require extra diffusion module. Also, the best performing setting of MAR resembles masked modeling more instead of conventional autoregressive (AR) modeling. As shown in the MAR paper, with conventional AR, the performance greatly falls behind masked modeling setting.
   • We agree that it would be beneficial to include compare DART with MAR. We have added comparison to MAR and other baselines in Table 2 of the updated manuscript. Admittedly, MAR achieves better performance than DART. However, MAR uses more sampling steps than vanilla DART. We also include comparison to reproduced MAR with 16 autoregressive steps in inference which follows a more similar setting as DART. As shown, with limited inference steps, DART achieves competitive performance with FID 3.98 vs MAR-16 with FID 6.37. This demonstrates the effectiveness of DART in leveraging the whole denoising trajectory.

2. Q: In the inference stage, does the DART-FM also show the short-step sampling property? How to realize short-step sampling with DART-FM.

   • If we understand correctly, “short-step sampling property” refers to DART can generate realistic images in limited number of denoising steps. DART-FM uses the same number of autoregressive steps as vanilla DART (i.e., 16 steps). Additionally, DART-FM includes a small flow-matching module which refines the prediction at each denoising step. We implement a small flow network which only adds about 1% of total parameters. To further increase the efficiency of DART-FM, one can further decrease the sampling steps for flow-matching module. One strategy could be adaptive module where more sampling budget is applied when closer to clean data. Also, distillation methods can help significantly reduce the sampling steps. We believe these are valuable directions to explore as future works.

3. Q: I still have no idea why DART-AR is better than DART. For me, the whole image processing mechanism should work better. Could the authors give me more detailed explanation?

   • Both DART and DART-AR have access to all the previous image patches through the denoising process. Therefore, both strategies enjoy the benefit of “whole image processing”. We attribute the better performance of DART-AR over DART to the more refined generative granularity. DART outputs a whole image at once which leaves no room for the model to refine the prediction of individual patches in the image. On the other hand, DART-AR outputs one patch as a time which allows the model to correct the imperfect prediction of previous patches. DART-AR conducts denoising in a more refined granularity than DART (patch v.s. image) which leads to better generative results.

1. Q: The paper's narrative structure prioritizes theoretical formulation over an intuitive explanation of the approach. The image is first transformed into several patches in the latent space through a Variational Autoencoder (VAE). The basic version of DART processes these patches simultaneously, predicting their denoised mean values, and then continues denoising in a manner similar to traditional diffusion models. The authors can include a more accessible explanation of how DART works in practice.

   • We have added the following descriptions to Appendix F. “Conceptually, DART predicts the denoised value and adds independent noise to acquire a less noisy image at each step. It conducts this denoising process autoregressively until the clean image is generated.”

2. Q: I am not sure whether this AR+diffusion method become new trend for future visual generation. Is it better than transfusion, show-o, and mar manner? (I recognize those should be concurrent work. This is just for curiosity, and I will not penalize the paper in terms of scoring).

   • We thank the reviewer for bringing up this valuable discussion. There are definitely works in attempt to integrating autoregressive (AR) and diffusion models. Transfusion and Show-O develop multimodal vision-language models (VLMs) but they apply AR for text while applying diffusion for image generation separately. Such that they integrate AR and diffusion models in a brutal way. Namely different losses are used to train the two aspects. In DART, image generation is also modeled as an autoregressive denoising which can be integrated seamlessly with AR text generation as shown in preliminary results of Kaleido-DART.
   • On the other hand, MAR autoregressively predicts a latent for each patch and applies an additional diffusion model to generate the image tokens conditioned on the predicted latent. DART integrates the denoising process into the autoregressive modeling and doesn’t require extra diffusion module. By this means, DART enables a more unified framework that combines AR and diffusion models.
   • It is still uncertain which model will be the best way to integrate AR and diffusion models. As this area is still a growing rapidly. However, the results shown in the paper already demonstrate the capabilities of DART that conventional diffusion or AR model cannot achieve trivially. We believe DART paves a way for developing novel generative models beyond AR and diffusion models and may lead to a more unified framework. And we would like to further improve the framework in future works.

1. Q: Lack of comparison to SOTA approach, either in the form of diffusion models or auto-regressive models, for image generation tasks.

   • We have added Table 2 to the updated manuscript to include recent baselines of diffusion and autoregressive models on class-conditioned image generation of ImageNet-256. Compared with diffusion models like LDM [1] and autoregressive (AR) models like VQGAN [2] and RQ-Transformer [3], DART achieves competitive performance. Admittedly, there is a gap between DART and SOTA visual generative models (like VAR [4] and MAR [5]). However, we want to point out that many baselines are trained with significantly more FLOPs. For example, DiT is trained for 7M iterations whereas DART is only trained for 500k iterations. Also, baselines like VAR and MAR employ larger models than our DART. In particular, VAR deploys a 2B model while the largest DART model is approximately 800M. Besides, in our reproduced results, when using only 16 sampling steps as the setting of our vanilla DART, our model show better performance than DiT [6], SiT [7] and MAR. Also, we report MAR-AR, a variant of MAR which generate tokens in an autoregressive manner instead of masked modeling which is applied in standard MAR models. DART which generates samples through autoregressive denoising shows better performance than MAR-AR. It further validates the effectiveness of leveraging the whole denoising trajectory. More details can be found in Response to All point 3 and Appendix C in updated manuscript.

2. Q: The efficiency of the model is a big concern. As shown in Fig 7 and Fig. 9(a), although a finer grain modeling by DART-AR could bring a marginal improvement, it would introduce 100x more latency, making it hard to justify whether this efficiency-performance tradeoff is ideal.

   • We agree with the reviewer that it’s valuable to explore the tradeoff between performance improvement and the sampling efficiency. One way is to develop an adaptive sampling strategy where more tokens are output at once in the beginning while less tokens are output later. Since we may want to allocate more FLOPs to better refine the generative results that are closer to clean data. Also, efficient architecture like state-space models may increase the efficiency over Transformer in modeling the denoising process. We believe these are valuable directions to explore as future works.

References:

[1] Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695. 2022.

[2] Esser, Patrick, Robin Rombach, and Bjorn Ommer. "Taming transformers for high-resolution image synthesis." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883. 2021.

[3] Lee, Doyup, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. "Autoregressive image generation using residual quantization." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11532. 2022.

[4] Tian, Keyu, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. "Visual autoregressive modeling: Scalable image generation via next-scale prediction." arXiv preprint arXiv:2404.02905 (2024).

[5] Li, Tianhong, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. "Autoregressive Image Generation without Vector Quantization." arXiv preprint arXiv:2406.11838 (2024).

[6] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205. 2023.

[7] Ma, Nanye, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. "Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers." arXiv preprint arXiv:2401.08740 (2024).

1. Q: Comparison to existing work is not comprehensive

   • We have added Table 2 to the updated manuscript to compare DART and recent baselines on class-conditioned image generation of ImageNet-256. Compared with diffusion models like LDM [1] and autoregressive (AR) models like VQGAN [2] and RQ-Transformer [3], DART achieves competitive performance. Admittedly, there is a gap between DART and SOTA visual generative models (like VAR [4] and MAR [5]). However, we want to point out that many baselines are trained with significantly more FLOPs. For example, DiT is trained for 7M iterations whereas DART is only trained for 500k iterations. Also, baselines like VAR and MAR employ larger models than our DART. In particular, VAR deploys a 2B model while the largest DART model is approximately 800M. Besides, in our reproduced results, when using only 16 sampling steps as the setting of our vanilla DART, our model show better performance than DiT [6], SiT [7] and MAR. Also, we report MAR-AR, a variant of MAR which generate tokens in an autoregressive manner instead of masked modeling which is applied in standard MAR models. DART which generates samples through autoregressive denoising shows better performance than MAR-AR. More details can be found in Response to All point 3 and Appendix C in updated manuscript.

2. Q: How the autoregressive generation is the cause of improvements. Can authors run any experiments supporting the fact that autoregressive generation, where attention happens with all previous tokens, has a huge role?

   • We thank the reviewer for bringing up this interesting discussion. In fact, DART won’t work if only tokens of one previous denoising step are given as standard diffusion model. This is because DART follows a non-Markovian formulation where the current state $x_t$ relies on all the previous states $x_{t+1:T}$. In practice, at each denoising step, individual noise is re-sampled to corrupt the data following the noise schedule. Whereas in standard Markovian diffusion process, the previous state $x_{t+1}$ alone is assumed to contain all the information needed to predict next state. In our early experiments, we found that if whole previous trajectory is not provided, the model failed to generate meaningful results. Such design is one of the major novelties in proposed DART which allows leverage of whole denoising trajectory.

References:

[1] Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695. 2022.

[2] Esser, Patrick, Robin Rombach, and Bjorn Ommer. "Taming transformers for high-resolution image synthesis." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883. 2021.

[3] Lee, Doyup, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. "Autoregressive image generation using residual quantization." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11532. 2022.

[4] Tian, Keyu, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. "Visual autoregressive modeling: Scalable image generation via next-scale prediction." arXiv preprint arXiv:2404.02905 (2024).

[5] Li, Tianhong, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. "Autoregressive Image Generation without Vector Quantization." arXiv preprint arXiv:2406.11838 (2024).

[6] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205. 2023.

[7] Ma, Nanye, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. "Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers." arXiv preprint arXiv:2401.08740 (2024).

3. The reproduced results use the official codebase and checkpoint and we follow the best performing cfg scale reported in the original papers. We report reproduced results of DiT [5], SiT [6], and MAR [1] with 16 sampling times which is the same as our vanilla DART. Our model achieves competitive performance when compared with diffusion models like LDM [7] and AR models like VQGAN [8] and RQ-Transformer [9]. Admittedly, there is a gap between DART and SOTA visual generative models (like VAR [2] and MAR [1]). However, we want to point out that due to the limited computation resources, DART is trained with less FLOPs than many baselines. For example, DiT is trained for 7M iterations whereas DART is only trained for 500k iterations. Also, baselines like VAR and MAR employ larger models than our DART. In particular, VAR deploys a 2B model while the largest DART model is approximately 800M. Besides, in our reproduced results, when using only 16 sampling steps as the setting of our vanilla DART, our model show significantly better performance than DiT, SiT and MAR. Also, we report MAR-AR, a variant of MAR which generate tokens in an autoregressive manner instead of masked modeling which is applied in standard MAR models. DART which generates samples through autoregressive denoising shows better performance than MAR-AR. These results further validate the effectiveness of leveraging the whole denoising trajectory.

   Type	Model	FID	IS	#params	Steps
   Diff.	ADM	10.94	101.0	554M	250
   	CDM	4.88	158.7	-	8100
   	LDM	3.60	247.7	400M	250
   	DiT	2.27	278.2	675M	250
   	SiT	2.06	277.5	675M	250
   AR	VQGAN	15.78	74.3	1.4B	256
   	RQTran	3.80	323.7	3.8B	68
   	MAR-AR	4.69	244.6	479M	256
   	MAR	1.55	303.7	943M	256
   	VAR	1.73	350.2	2.0B	10
   Reprod.	DiT	19.52	125.9	675M	16
   	SiT*	6.98	122.9	675M	16
   	MAR*	6.37	221.29	943M	16
   Ours	DART	5.62	231.7	812M	16
   	DART-AR	3.98	256.8	812M	4096
   	DART-FM	3.82	263.8	820M	16

4. DART can also be scaled to higher-resolution generation in an efficient way. We have added the results of DART on image generation of resolutions $512 \times 512$ and $1024 \times 1024$ to Appendix D. In particular, we further extend DART to multi-resolution generation (i.e., Matryoshka-DART). Image patches of higher resolutions are concatenated after low resolution patches ($256 \times 256$) and the model progressive generates images from low to high resolutions. Figure 12 illustrates the process of progressively generating multi-resolution images. This paradigm also allows efficient generation for high-resolution images than standard diffusion model like DiT. Since the model can make full use of the generative trajectory of low-resolution samples, it can be directly finetuned from a pretrained $256 \times 256$ model by concatenating the higher-resolution tokens afterwards. Besides, the model requires less denoising steps for high-resolution generations since it is naturally conditioned on generated low-resolution images. Samples of high resolution results of resolutions $512 \times 512$ and $1024 \times 1024$ from text-to-image generation can be found in Figure 1, 14 and 15.