Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding

3. About the evaluation details about image quality

   To obtain the FID score on the MSCOCO 2017 validation set (comprising 5,000 ground-truth images), we first run each decoding method to generate 5,000 images using the captions from this dataset. We then calculate the FID scores between the images generated by each method and the ground-truth images by using the official API from 'pytorch_fid'. For the CLIP-Score evaluation, we utilize the CLIP-ViT-B pre-trained by OpenAI following [A]. We employ the official API from 'torchmetrics' to compute the CLIP-Score. Specifically, for each decoding method, we send each pair of generated image and its corresponding prompt to the API. After processing all pairs, the API can be used to compute the CLIP-Score. The evaluation follows the settings of the mainstream text-to-image generation works.

   [A] Hessel J, Holtzman A, Forbes M, et al. Clipscore: A reference-free evaluation metric for image captioning[J]. arXiv preprint arXiv:2104.08718, 2021.

4. The good FID scores may come from worse quality and better diversity.

   We would like to emphasize that the image quality of our method remains comparable to that of the original auto-regressive decoding. First, besides FID, we also report the CLIP-Score metric, which does not account for image diversity. As shown in Table 1, our CLIP-Scores are consistently around 31.3, indicating that our method does not pose much impact on image quality. Moreover, we employ the human preference score (HPSv2) to further compare the decoding methods. Please refer to the table below. The different decoding methods exhibit similar human preference scores, further demonstrating the preservation of image quality. Additionally, our SJD method preserves image diversity of standard auto-reregressive decoding, but not amplify the diversity. As shown in Figure 17, comparing SJD and standard AR, the images generated with the same prompt exhibit similar behaviors with SJD and standard AR. Given the maintenance of both image quality and diversity, the FID scores are similar for the original auto-regressive decoding and our SJD.

   Decoding Methods	HPSv2
   Auto-regression	0.2965
   Jacobi Decoding	0.3103
   Ours	0.3020

5. About the number and size of tested models

   Actually, auto-regressive text-to-image generation with discrete image tokens is a relatively new research field, so there are few open-source models available. We have found all the performant open-sourced models before the main submission deadline (Lumina-mGPT, Anole, LlamaGen-Stage1/Stage2, and Emu3) to verify the effectiveness of our SJD. Among these models, LlamaGen is the smallest with 775M parameters, while the others range from 7 to 8 billion parameters. The results for these models have been included in our paper:

   • The results for Lumina-mGPT are presented in Table 1 and Table 2. The revision further adds Figure 5 and Figure 16.
   • The results for Anole can be found in Table 1 and Table 2. The revision further adds Table 4, Table 5, and Figure 17.
   • The results for LlamaGen-Stage1/Stage2 can be found in Table 3.
   • The results for Emu3 can be found in Figure 11. Since Emu3 has the quantitative results on GenEval in its original paper, we perform the following experiment to demonstrate that SJD does not influence too much image quality. Please refer to the table below:

   Decoding Methods	GenEval
   Auto-regression	0.64
   SJD	0.63

6. About the perplexity

   We compare the perplexities between SJD and other decoding methods on Lumina-mGPT in Table 6 of the supplementary material of the revised paper. You can also refer to the table below. Since the perplexities are influenced by the sampling strategies, we report the perplexities under various top-$K$ sampling values. Given an identical $K$ value, the perplexities between our method and other decoding methods are close.

   Configuration	Top-10 Sampling	Top-100 Sampling	Top-2000 Sampling
   A. Lumina-mGPT	7.31	43.37	204.06
   B. w. JD	7.20	43.85	197.64
   C. w. SJD	7.34	43.87	217.96
   D. w. SJD (ISP)	7.26	44.03	199.70

7. About the image quantity for FID on SJD and other compared methods.

   Our evaluation does NOT involve any tricks. For each decoding method, we have utilized the official API from 'pytorch_fid' to calculate the FID scores. Each method in these tables uses 5,000 image samples for calculating FID on the MSCOCO 2017 validation set. For further details regarding our evaluation, please kindly refer to our answer to question 3 above.

We are grateful to the reviewer for the review of our submission. Below, we provide detailed responses to the questions and concerns raised.

1. SJD relaxes the greedy acceptance criterion of JD, which might lead to suboptimal performance; About the contribution and novelty of our paper.

   This work is not a simple trick or naive improvement over Jacobi Decoding. Instead, we conduct a thorough analysis and provide deep insights into a problem that has been previously overlooked, proposing an effective solution to this problem. We begin by analyzing an intriguing phenomenon: Jacobi Decoding does not effectively accelerate auto-regressive text-to-image generation models. Through further analysis, we identified the key problem: the deterministic criterion of Jacobi Decoding is only effective with greedy decoding and is incompatible with the random sampling (e.g., top-$K$ sampling) used in text-to-image models. This random sampling is essential for generating diverse and high-quality images with sufficient details (as depicted in Figure 2). To address this problem, we introduce the probabilistic acceptance criterion into Jacobi Decoding. Our SJD does not lead to suboptimal performance compared to JD (Please refer to the table of human preference score below, where the scores on the decoding methods are similar), but rather it solves the problem that JD cannot accelerate auto-regressive text-to-image generation models under the common random sampling settings. We would also like to emphasize that our SJD is a simple yet effective method for accelerating auto-regressive text-to-image generation. This training-free acceleration algorithm has been deployed on various auto-regressive text-to-image generation models, including Lumina-mGPT, Anole, LlamaGen, and Emu3, achieving notable acceleration ratios with minimal impact on image quality. To thoroughly discuss and analyze the acceleration and visual quality preservation of SJD across different models and settings, we have conducted extensive experiments to verify the effectiveness and generalization ability of our paper, encompassing 10 tables and 13 figures in the revised version of this paper.

   Decoding Methods	HPSv2
   Auto-regression	0.2965
   Jacobi Decoding	0.3103
   Ours	0.3020

2. About the performance of the original Jacobi Decoding.

   Our experiments have demonstrated that the original Jacobi Decoding is NOT suitable for the current auto-regressive text-to-image generation. The original Jacobi Decoding paper demonstrated its success on MNIST and CIFAR-10 class-conditioned image generation, but these datasets are much smaller, and the class-conditioned image generation on these datasets requires far less image diversity and details compared to current text-to-image generation. On these small datasets, models can use greedy decoding to generate images, so Jacobi Decoding works well. However, the current text-to-image generation task requires higher diversity and more details in the generated images, and the current auto-regressive models exhibit different behaviors from the previous class-conditional auto-regressive models that Jacobi Decoding experimented on. Current auto-regressive models require top-$K$ sampling rather than greedy decoding, but Jacobi Decoding fails to perform well on top-$K$ sampling. As shown in Figure 6, our ablation study demonstrates that the acceleration ratio of the original Jacobi Decoding rapidly decreases from about 2.5 to 1.0 as the $K$ values in top-$K$ sampling increase from 1 to 2000. This phenomenon illustrates that Jacobi Decoding is effective only with greedy sampling, which is not practical for current auto-regressive text-to-image generation (as shown in Figure 2, we explained why greedy sampling is unsuitable for current auto-regressive text-to-image generation). We are confident of our implementation and experimental results, and we will open-source our code for the community.

3. Explanation on why Jacobi Decoding does not work for current text-to-image auto-regressive models:

   • Difference between previous-generation auto-regressive models and the current-generation text-to-image auto-regressive models. We would like to first clarify the significant differences between previous-generation auto-regressive image generation models such as PixelCNN (2016) and PixelCNN++ (2017) on MNIST and CIFAR-10, and current-generation auto-regressive text-to-image generation models such as LlamaGen (2024), Lumina-mGPT (2024), and Emu3 (2024). These differences can be summarized from two perspectives: (1) Data perspective: Current text-to-image generation tasks on diverse, open-domain images are much more complex than previous image generation tasks on small-scale, low-diversity datasets like MNIST and CIFAR-10. Current text-to-image models require higher diversity and more detailed image generation. (2) Model perspective: Previous auto-regressive image generation models operate in pixel space and leverage small codebook sizes (e.g., PixelCNN has 256 codes), while current-generation autoregressive image generation models operate in latent space with large codebooks (8,000 to 16,000 codes) to sample from.

   • The current-generation text-to-image auto-regressive models require higher randomness in sampling, compared with previous-generation auto-regressive models. Due to (1) and (2), previous-generation auto-regressive image generation models (e.g., PixelCNN, PixelCNN++) require limited randomness during inference, while current-generation auto-regressive text-to-image generation models (e.g., LlamaGen, Lumina-mGPT, Emu3) require higher randomness during inference. For example, LlamaGen leverages top-$1000$ sampling.

   • Jacobi Decoding, designed for the previous-generation auto-regressive models, is not compatible with high randomness in sampling. At the time Jacobi Decoding was proposed, algorithms such as PixelCNN and PixelCNN++ required limited randomness for sampling. Designed for these algorithms, Jacobi Decoding effectively accelerates these models on datasets like MNIST and CIFAR. However, Jacobi Decoding is not compatible with highly random sampling such as top-$1000$ sampling in LlamaGen. This explains our motivation for why naive Jacobi Decoding does not work for auto-regressive text-to-image generation models and why it is necessary and non-trivial to design new algorithms to accelerate these models.

   • Summarization. We summarize our above analysis as follows: Current auto-regressive text-to-image models are trained on large and diverse datasets with a large codebook size. In these models, token sampling with high randomness (e.g., top-$2000$ sampling) is typically used to generate diverse images. With such highly random sampling, the original Jacobi Decoding is not effective because it is rare to encounter identical model outputs between two Jacobi iterations. As evidenced by Figure 6, Jacobi Decoding can accelerate text-to-image generation with top-$1$ sampling (easy to get identical tokens between two iterations) but not with top-$2000$ sampling (hard to get identical tokens between two iterations).

4. About testing small models for demonstrating the generization ability of SJD

   Please refer to question 2. We use a small text-to-image generation model, LlamaGen-XL with only 775M parameters. This model can generate images quickly and requires a maximum of only 5GB of GPU memory, making it even feasible for personal computers.

Thank your for your replies!

1. I still doubt about the performance of JD. I think JD should at least accelerate 1.5 or faster. But the authors' result show that JD is invalid while their JD variant with randomness lead to 2.0 faster. (the authors argue that this phenomenon is caused by the size of codebook. If this is true, the variation should be contiguous rathor than no acceleration of original JD). If the authors could provide the acceleration rate variaiton according to codebook size, it will be persuasive.
2. Comparison to recent work Break the Sequential Dependency of LLM Inference Using LOOKAHEAD DECODING(icml 2024); This work also use SPECULATIVE sampling similar to this paper.
3. Comparisons to pixelCNN is strange in their reply. The authors should compare to Autoregesive models with small codebook size. In conclusion, this works look not solid enough. It seems an application of jacobian decoding to text-to-image generation(using reject sampling to bring randomness into JD).

I only have several papers about text-to-image with diffusion models and I hope AC to add a more prefessional reviewer. I doesn't prepare to give too much comments but the other two reviews also did not give prefesional comments on this paper(they only raise some corncerns on the experimental results). If I did not understand this paper well, forgive me (:

4. Comparing SJD with ISP to other decoding methods on Anole.

   Thank you for your suggestion. We have conducted experiments comparing SJD with ISP to other decoding methods on Anole. Please refer to the tables below. These quantitative results are also presented as Table 4 and Table 5 in the supplementary material of the revised paper:

   COCO:

   Configuration	Average Latency $(\downarrow)$	Acceleration Latency $(\uparrow)$	Acceleration Step $(\uparrow)$	FID $(\downarrow)$	CLIP-Score $(\uparrow)$
   A. Anole	48.96s	$1.00 \times$	$1.00 \times$	$28.87$	$30.59$
   B. w. JD	47.60s	$1.03 \times$	$1.06 \times$	$29.34$	$30.64$
   C. w. SJD	27.08s	$1.81 \times$	$1.94 \times$	$29.04$	$30.54$
   D. SJD (ISP)	26.18s	$\mathbf{1.87} \times$	$\mathbf{1.97} \times$	$29.14$	$30.61$

   Parti:

   Configuration	Average Latency $(\downarrow)$	Acceleration Latency $(\uparrow)$	Acceleration Step $(\uparrow)$	CLIP-Score $(\uparrow)$
   A. Anole	48.24s	$1.00 \times$	$1.00 \times$	$30.46$
   B. w. JD	44.65s	$1.08 \times$	$1.14 \times$	$30.57$
   C. w. SJD	26.77s	$1.80 \times$	$2.00 \times$	$30.55$
   D. SJD (ISP)	26.12s	$\mathbf{1.92} \times$	$\mathbf{2.11} \times$	$30.48$

5. Typo fix.

   Thank you for pointing out the typo, and we have corrected it.

6. Discussion on inaccurate token acceptance when tokens have low conditional probability but slightly higher than previous iteration.

   We acknowledge that the mentioned corner case might happen in theory, where the draft token is sampled with very low conditional probability (confidence), yet the verification accepts it because the conditional probability (confidence) is higher than that of the previous iteration. However, we find that the tokens with low conditional probability (confidence) would be normally rejected by our method, meaning that such corner case rarely happens. We compute the statistics of the logarithm of the token probability for both auto-regressive decoding and our SJD. We find that the conditional probability (confidence) of image tokens accepted by our method exhibits similar statistics to those accepted by the original auto-regressive decoding. Consequently, our method generally does not mistakenly accept tokens with lower probabilities. The statistics are in the table below:

   Decoding Methods	Average of Logarithm of Token Probability	Standard Deviation of Logarithm of Token Probability
   Auto-regression	-4.8950	2.3457
   Ours	-4.9007	2.3275

   It is worth noting that even for standard auto-regressive decoding without SJD, the conditional probabilities of sampled tokens are not very high. This is because, in visual auto-regressive generation models, the conditional distribution for the next token is not an unimodal distribution (i.e., there could be more than one possibility for the next token given the previous ones). Moreover, visual generation requires diversity, so instead of sampling the next token with the highest probability, we add randomness by using top-$K$ sampling, further decreasing the conditional probability of the sampled tokens. Nonetheless, we did not observe a severe decrease in the conditional probability of the accepted tokens in SJD, compared with the sampled tokens in standard auto-regressive decoding.

We are grateful to the reviewer for their insightful review of our submission and for providing valuable suggestions. Below, we provide detailed responses to the questions and concerns raised.

1. The effectiveness of reusing unaccepted tokens as new draft tokens.

   We conduct empirical analysis and find that reusing unaccepted tokens as draft tokens in the next iterations is critical to our method. To demonstrate this, we performed the following experiments, with quantitative and qualitative results presented in Figure 14 of the supplementary material of the revised paper: When we deactivate the refinement (i.e., using newly initialized tokens to replace the unaccepted tokens as draft tokens in the next iteration), we observe that the model requires over two thousand forward passes to generate images, rather than one thousand. Although our token initializations with spatial prior (e.g., horizontal repeat) are slightly better than random token initialization, their acceleration is still significantly worse compared to directly refining the unaccepted tokens. You can also refer to the quantitative results below:

   Draft Token	Average Steps	Average Latency	Step Compression Ratio
   Not Reusing Unaccepted Tokens + Random Token Initialization	2356	107.65s	$1.00 \times$
   Not Reusing Unaccepted Tokens + Horizontal Repeat Token Initialization	2281	95.85s	$1.03 \times$
   Reusing Unaccepted Tokens	1005	42.13s	$2.35 \times$

2. About the initialization with spatial prior (ISP).

   We acknowledge that this design is not our major contribution. It is specifically designed to accelerate images in extreme cases, such as the images containing simple and repeating patterns. The acceleration ratios and generated images are showcased in our ablation studies (Figure 9, more than $3 \times$ step compression) and teaser (Figure 1, the ''Corgi dog'' case, $2.8 \times$ step compression). Although it has not achieved significant acceleration across all cases, this represents a preliminary exploration of using spatial priors for accelerating auto-regressive text-to-image generation, which would probably inspire future research to leverage spatial prior for the acceleration of auto-regressive text-to-image generation.

3. Adding quantitative performance of image quality to Figures 6,7,8 and 9.

   Thank you for your feedback. We utilize the CLIP-Score and the human preference score (HPSv2) as the metrics for evaluating the visual quality of the images generated in our ablation studies. The results are presented in Table 7, Table 8, Table 9, and Table 10 of the supplementary material of the revised paper. From the results in these tables, we find that the change in visual quality is within a small range. Please refer to the tables below (We have bolded the adopted settings for qualitative results in Figure 15):

   Table 7: Visual quality for Figure 6:

   Decoding Methods	Sampling	CLIP-Score	HPSv2
   Auto-regression	Top-1 Sampling	26.40	0.1976
   Auto-regression	Top-10 Sampling	32.83	0.2950
   Auto-regression	Top-100 Sampling	32.41	0.3020
   Auto-regression	Top-2000 Sampling	32.00	0.2965
   Jacobi Decoding	Top-1 Sampling	26.34	0.1413
   Jacobi Decoding	Top-10 Sampling	32.75	0.2960
   Jacobi Decoding	Top-100 Sampling	32.46	0.3089
   Jacobi Decoding	Top-2000 Sampling	31.68	0.3103
   Ours	Top-1 Sampling	26.16	0.1695
   Ours	Top-10 Sampling	32.27	0.2942
   Ours	Top-100 Sampling	32.65	0.2977
   Ours	Top-2000 Sampling	31.83	0.3020

   Table 8: Visual quality for Figure 7:

   Decoding Methods	Resolutions	CLIP-Score	HPSv2
   Auto-regression	512	29.49	0.2503
   Auto-regression	768	32.00	0.2965
   Auto-regression	1024	31.41	0.2961
   Ours	512	29.69	0.2558
   Ours	768	31.83	0.3020
   Ours	1024	31.11	0.2935

   Table 9: Visual quality for Figure 8:

   Jacobi Window Size	CLIP-Score	HPSv2
   1	32.00	0.2965
   4	31.91	0.3046
   16	31.83	0.3020
   32	31.55	0.3045

   Table 10: Visual quality for Figure 9 (evaluation on images with extremely simple patterns):

   Token Initialization	CLIP-Score	HPSv2
   Horizontal Sample	31.52	0.2567
   Vertical Sample	30.91	0.2622
   Horizontal Repeat	31.17	0.2616
   Vertical Repeat	31.15	0.2651
   Random	31.37	0.2681

We thank the reviewer for taking the time to read our submission and provide constructive feedback. We will now respond to the highlighted questions and concerns.

1. Analysis on the image randomness in SJD.

   Thank you for your insightful suggestion. Randomness is essitial for all visual generative models to achieve diversity in generated images. For auto-regressive text-to-image models without SJD, randomness is introduced from the sampling strategy (e.g., top-K sampling), resulting in diverse generated images. Our SJD is not designed to amplify such randomness, but to preserve this randomness. To compare the randomness of SJD and tranditional auto-regressive decoding, we have conducted a qualitative analysis to examine the image randomness, with the results presented in Figure 17 of the supplementary material of the revised paper. Firstly, SJD indeed introduce some randomness into image generation (as represented by the random variable $r$ in our Equation 1). Consequently, the images generated using SJD cannot be exactly identical to those generated through auto-regression, even when the random seed is fixed. For example, in Figure 17, two images with the same $K$ (i.e., the value from top-$K$ sampling) and the same random seed (i.e., within the same column) cannot be exactly identical. Secondly, comparing the images generated by SJD and standard auto-regressive decoding, SJD has a negligible impact on the diversity of images. For instance, when $K=2000$, comparing SJD and standard auto-regressive decoding, the images generated with the same prompt exhibit similar behaviors in the diversity of generated images. Thirdly, the diversity of generated images is mainly determined by the sampling strategies rather than the decoding methods. In Figure 17, as the $K$ values in top-$K$ sampling increase, the details related to textures, colors, and local structures consistently become more diverse in SJD and standard auto-regressive decoding.

2. The quantitative results of LlamaGen.

   The results for LlamaGen are presented in the table below. These results have been included in Table 3 of our supplementary material.

   Dataset	Configuration	Acceleration Latency $(\uparrow)$	Acceleration Step $(\uparrow)$	FID $(\downarrow)$	CLIP-Score $(\uparrow)$
   COCOval2017	LlamaGen-stage1	$1.00 \times$	$1.00 \times$	$28.54$	$30.87$
   COCOval2017	LlamaGen-stage1 + Ours	$\mathbf{1.56} \times$	$\mathbf{1.63} \times$	$29.00$	$30.82$
   COCOval2017	LlamaGen-stage2	$1.00 \times$	$1.00 \times$	$56.21$	$28.26$
   COCOval2017	LlamaGen-stage2 + Ours	$\mathbf{1.54} \times$	$\mathbf{1.63} \times$	$57.02$	$28.33$
   Parti-Prompt	LlamaGen-stage1	$1.00 \times$	$1.00\times$	-	$30.22$
   Parti-Prompt	LlamaGen-stage1 + Ours	$\mathbf{1.57} \times$	$\mathbf{1.73} \times$	-	$30.29$
   Parti-Prompt	LlamaGen-stage2	$1.00 \times$	$1.00 \times$	-	$28.14$
   Parti-Prompt	LlamaGen-stage2 + Ours	$\mathbf{1.62} \times$	$\mathbf{1.69} \times$	-	$28.16$

   Note that the FID score for LlamaGen-stage2 is high because this model typically generates images with various styles that are distant from the distribution of COCO images. This is a property of the original LlamaGen-stage2 model [A] which is not related to our SJD algorithm.

   [A] Sun P, Jiang Y, Chen S, et al. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation[J]. arXiv preprint arXiv:2406.06525, 2024.

3. Additional qualitative evaluation

   We have included additional qualitative results for Lumina-mGPT and Anole in the supplementary material of the revised paper, specifically in Figure 15 and Figure 16, and we report both the steps and latency. According to the reported latency and step compression in these figures, our SJD outperforms other decoding methods while maintaining visual quality.

4. Discussion on failure cases and their causes.

   Thank you for your valuable suggestion. We observe that the SJD method can introduce artifacts when generating images with intricate details, as shown in Figure 18 of the supplementary material of the revised paper. The pre-trained auto-regressive model is not sufficiently robust to handle such complex images. Consequently, it may mistakenly accept a sequence of draft tokens that contain artifacts.

Thank you for conducting additional experiments and providing a detailed response.

I still believe that this paper requires further analysis or discussion regarding the randomness in speculative jacobi decoding for visual AR models. However, as this work represents an early attempt to apply speculative decoding to visual AR models, including the current version of the discussion on the randomness in the main paper could be appropriate and provide valuable insights for future researchers. Considering the strengths of this work and its potential impact, I will maintain my score as borderline accept.