Elucidating the design space of language models for image generation

Thanks a lot for your valuable comments. Below, we will address your concern in detail.

Discuss with LlamaGen and MAR

We agree that both [1]LlamaGen(Sun et al.) and [2]MAR(Li et al.) have made important contributions to advancing autoregressive image generation. We would like to clarify that we do compare against both works in Table 1, and further discuss MAR in the appendix (A.7 Limitation section) as a parallel study that proposes new modeling approaches and domain-specifc design for AR to effectively work in the continuous image domains.

Compared to MAR, our work is more aligned with LlamaGen, as we both follow the same direction of directly adapting language models for image generation to explore their modeling potential. However, our work differs in that we conduct a more comprehensive and systematic analysis of the image-as-language modeling framework. In particular, we study:

• The effect of tokenizer choice (VQGAN vs. BAE),
• Modeling paradigms (AR vs. MLM),
• Detailed learning behavior reflected by attention pattern and loss dynamics,
• Vocabulary design through scalable codebooks under BAE,
• And an in-depth ablation and analysis of sampling strategies.

We believe our findings provide a complementary perspective to LlamaGen and MAR, and offer insights that could be valuable when combined with more advanced objectives or tokenization schemes in future work. We appreciate the suggestion and will incorporate the detailed discussions more clearly in the revised version.

W1. technical novelty

We believe that our work makes a meaningful and timely contribution by systematically analyzing the design space of applying language models in the image domain, providing insights into adopting LLMs as a unified method and guiding future methodological innovations, especially in light of recent developments of GPT-4o, a powerful natively AR large multimodal model. You may find our detailed response to W1 from Reviewer xmQD.

W2. Comparison with MAR

Thank you for pointing out that our conclusion on the choice of generation paradigm differs from MAR. However, given the differences in our methods, we believe this divergence is entirely reasonable.

• We adopt a standard language modeling view, treating images as discrete token sequences with no image-specific modifications. Under this setup, we systematically compare AR and MLM using standard tokenizers and sampling, finding AR consistently superior in quality and scalability. Besides, this discrete token framework can integrate well with LLM-based multimodal extensions.
• MAR presents a new generative modeling approach tailored for continuous image data with diffusion loss and no quantization. Their finding that MLM outperforms AR is valid within this setting but reflects a fundamentally different modeling goal and assumptions from ours.

We view these works as complementary: MAR explores domain-specific continuous modeling, while we investigate how standard language modeling paradigms transfer to vision. Based on the different tokenizers and training objectives, we come to different conclusion according to the generation modeling method, which totally makes sense. We have acknowledged this in our appendix and will add the discussion to the main paper.

W3. qualitative study and empirical discussion on top-k and cfg

We agree that understanding top-k sampling and classifier-free guidance (CFG) is key to characterizing image generation behavior, especially in contrast to standard LLM decoding. We have included an extensive analysis of both strategies (Figure 6, Table 12). We further provide qualitative comparisons (linked here). We find that image generation requires larger top-k values and moderate CFG weights to achieve a better FID score. However, there is a trade-off between FID score and visual quality. A large k (close to the vocab size) introduces more texture details and diversity, making the outputs closer to the real distribution and thus yielding lower FID. In contrast, a moderate k often produces visually more appealing results, with cleaner and smoother images.

While CFG is rarely applied in LLM-based text generation, top-k sampling is widely adopted in both domains:

• In image generation, small k (less than 100) values lead to overly smooth, low-diversity images, lacking texture richness. Conversely, image tokens require broader sampling (e.g., k ≈ 0.5 * vocab size) to maintain realism.
• In language generation, a typical k lies in the 20–100 range with a vocab size of ~50,000 or 128,000, and can produce diverse and grammatical fluency texts (Holtzman et al., 2020; Fan et al., 2018).

ref: Holtzman et al, 2020, The Curious Case of Neural Text Degeneration.

Fan et al, 2018, Hierarchical Neural Story Generation.

Thanks for your valuable comment! Below, we will address your concern in detail.

W1. methodological innovation

We acknowledge that our work does not center around proposing a new model architecture, but rather focuses on systematically analyzing and understanding how existing design components interact within the context of LLM-based image generation. We believe this perspective is important and timely, especially in light of recent developments such as GPT-4o, which has been introduced as a natively AR large multimodal model capable of cross-modal reasoning and generation, demonstrating impressive performance across tasks. These advances further support the need for a deeper understanding of AR modeling as a unified generative paradigm, and our study lays the empirical foundation for such future multimodal research. In particular, our contributions lie in:

• Providing a fair comparison between AR and MLM under standard image tokenization;
• Demonstrating how tokenization and dictionary structure affects model performance and interacts with model scaling;
• Revealing surprising robustness of AR LMs to image token randomness and their ability to generate high-fidelity outputs;
• Offering design principles that can guide future LLM-based multimodal generation systems.

We view this paper as an analytical and foundational contribution, aiming to guide future methodological innovations. We appreciate the reviewer’s suggestion and will clarify this positioning in the revised version.

W2. baselines compared on the ImageNet-512

On the ImageNet-512 benchmark, we did not re-train a model from scratch at 512×512 resolution. Instead, we performed a lightweight fine-tuning of our model that was originally trained on 256×256 images, using only a few additional training steps. Our primary goal in this experiment is to demonstrate the resolution scalability and efficiency of ELM, so we only choose some typical methods to compare with. Training directly on 512-resolution images from scratch would require approximately 4× more compute compared to 256-resolution training. However, with a token-prediction-based architecture, we find that simple fine-tuning can achieve strong high-resolution performance with significantly reduced cost—a major advantage over diffusion models, which typically require retraining and redesigning the entire pipeline for higher resolutions.

As a reference baseline, we chose to compare against DiT, a strong class-conditional diffusion model trained natively on 512×512 images. We also included MaskGIT as a discrete-token-based generation model for completeness. We acknowledge that VAR has also reported ImageNet-512 results. Below, we include a comparison with VAR.

Although our results are slightly behind those of VAR, it is important to note that VAR uses more than 2× the number of parameters and was trained for a significantly longer time.

W3. scaling law for FID

Thank you for the helpful suggestion. While Figure 7 primarily illustrates visual improvements across scales, we would like to clarify that we do provide quantitative evidence of a scaling law for FID throughout the paper:

• In Table 1, we report FID scores across models with increasing capacities, showing a clear performance improvement as model size increases.
• In Figure 2, we plot FID curves over training epochs for AR and MLM models of different sizes, which further demonstrates the impact of scaling on convergence and generation quality.
• In the Appendix (Table 11), we present a comprehensive breakdown of FID scores for AR models across multiple scales—from L, XL, XXL to 2B—under different vocabulary designs, further reinforcing the consistency of the observed scaling trend.

Taken together, these results clearly support a scaling law for FID in AR-based image generation under the image-as-language paradigm, similar in spirit to what is shown in the VAR paper. We will make this connection more explicit in the revised version, and thank you again for encouraging us to clarify this aspect.

Thanks for your valuable comments! We will address your concerns in detail.

Limit 1. no results on text-to-image generation

Our main goal is to assess the effectiveness of LLMs as a unified generation paradigm, especially when applied to images. To isolate modeling factors like tokenizers, AR vs. MLM, and sampling, we use class-conditional generation for its simplicity and control. While our findings (e.g., on scalable tokenization and AR modeling) are relevant to text-to-image generation, we consider T2I a distinct challenge—requiring further study on alignment, prompt structure, and retrieval—which requires delicate future work.

Limit 2 & W1. Efficiency analysis

Our work focuses on understanding the modeling behavior and asses the effectiveness of LLMs for image generation. As LLMs acceleration techniques (e.g., KV-cache, FlashAttention, quantization) are well-developed and directly transferable to our setting, we do not emphasize efficiency in this paper. To address your concern, we summarize key efficiency insights below:

• Training efficiency: AR is more efficient than MLM and diffusion. During training, each AR step predicts all tokens, while MLM predicts only a subset. As shown in Figure 2 in our main paper, AR converges in ~100 epochs, MLM in ~200, and DiT needs ~160 (refer to [Scalable Diffusion Models with Transformers]).
• Inference speed: Vanilla AR is slower due to sequential decoding (e.g., 256 steps), but KV-cache accelerates it by ~4×. Summary: MLM > AR (with KV-cache) > Diffusion (DiT).
• Memory usage: All models are similar in training. During inference, AR’s KV-cache increases memory usage with sequence length; MLM and diffusion require fewer steps and less memory per sample.

We summarize FLOPs, training, and inference speed across AR, MLM, and DiT with matched model scales:

Issue1. rigorous proof on AR&MLM in image domain

While we do not provide formal theoretical proof that AR is superior to MLM for image generation, our systematic empirical study offers meaningful evidence supporting this claim. To the best of our knowledge, no existing work in language or vision has established a rigorous theoretical justification. However, under our image-as-language framework, some NLP arguments for AR over MLM are partially transferable—e.g., AR benefits from training-inference consistency, while MLM suffers from exposure bias [1], and AR models the full joint distribution, unlike MLM [2].

Moreover, the lack of standardized image tokenization adds complexity, making theoretical analysis more difficult and beyond the scope of this paper. We believe that our findings provide a strong foundation and motivating evidence for such future theoretical work.

Issue 2. The impact of KL-divergence analysis

We included the KL-divergence analysis as an additional diagnostic tool to investigate why, during next-token prediction on tokenized images, the training loss remains high—despite the model generating high-quality images, unlike in typical language modeling. Actually, we conduct some analysis of the impact on model optimization in our paper.

• Training loss curve plateaus rather than fully converging, as shown in Figure 14 & 16 in Appendix. The KL-divergence analysis shows that image tokens are more random and less sequentially structured than language tokens, leading to less accurate predictions.
• Attention Analysis: Attention maps analysis in Sec. 3.4 reveals strong local structure, and larger models capture more global dependencies, indicating effective learning and scaling despite the high loss.

W2. Why choose VQGAN

We acknowledge the emergence of stronger tokenizers beyond VQGAN (e.g., hierarchical, residual, multi-scale) with improved utilization and reconstruction. However, we chose VQGAN as our baseline due to its widespread use, reproducibility, and alignment with prior works (e.g., MaskGIT, LDM, LlamaGen), enabling controlled comparisons. We include BAE as a inherently different tokenizer: unlike VQGAN’s vector-level quantization, BAE performs bitwise scalar quantization, allowing us to isolate the effect of granularity on the quantization. Its design is also compatible with other advanced quantization methods, suggesting potential for future integration.

[1] Song et al., 2019. MASS: Masked Sequence to Sequence Pre-training for Language Generation.

[2] Ghazvininejad et al., 2019. Mask-Predict: Parallel Decoding of Conditional Masked Language Models.

Thanks for your valuable comments! Below, we will answer your concern in detail.

W1: Difference performance of VQGAN from LlamaGen

To ensure a controlled comparison between structurally different quantization methods, we use the original VQGAN from Taming Transformers [Esser et al., 2021], downloaded directly with rFID from the official repository （taming-transformers-github).

LlamaGen tends to show that discrete image tokenization can achieve a reconstruction ability close to continuous ones. They re-train VQGAN with L2-normalization on latent codes refer to [1] to improve codebook usage and representation smoothness, which likely accounts for their stronger rFID results.

To further address the reviewer’s concern, below table shows the comparison between LlamaGen's VQGAN, the original Taming VQGAN, and BAE (all with a down-sampling rate of 16 on 256px ImageNet):

This comparison shows that BAE can still outperform both, ensuring 100% code utilization, and the key advantage of supporting larger codebooks with better generation performance remains valid.

W2. Discuss with Infinity paper - bitwise self-correction method.

Both our method and Infinity∞ highlight that introducing stochasticity or 'error' during training —via Bernoulli sampling or bitwise self-correction—can improve autoregressive image generation by enhancing robustness to prediction errors. This suggests that stochastic modeling is a valuable direction for AR-based synthesis.

The key difference lies in the series modeling formulation:

• Infinity∞ builds on domain-specific serialization and next-scale prediction, where self-correction mitigates error accumulation across scales.
• Our approach adopts standard image tokenization and treats images as language; Bernoulli sampling helps address the inherent randomness of image token sequences, improving tolerance to uncertainty.

As Infinity∞ and our work were developed concurrently, we were not aware of it at the time of writing and therefore did not include a comparison. We appreciate the reviewer’s suggestion, and we will incorporate a discussion of this work in the revised version of our paper.

W3. We will re-design the Figures to improve the readability

W4. Review on Scan Pattern Choice section

We study scan patterns to evaluate whether generation order impacts performance in the image-as-language setting, where token sequences lack inherent structure. Zigzag, commonly used in tasks like image compression and video coding, serves as a meaningful alternative to raster.

Our results provide empirical grounding for choosing a raster scan in AR image generation, and the minor performance differences across scan patterns further suggest that large language models are robust to token order variations.

W5. Discuss with the scaling law in LlamaGen and VAR

We agree that works like LlamaGen and VAR have demonstrated the scalability of AR models for image generation. Different from ours, VAR provides a new image autoregressive modeling mechanism. Our study is in line with LlamaGen, adopting a language-centric approach without domain-specific design, and offers complementary insights. Beyond generation quality scaling in LlamaGen, we further analyze training dynamics, attention behavior, AR vs. MLM scaling trends, and the interactions between vocabulary and model capacity, offering a broader perspective on what enables scalable image generation with LLMs. We believe our findings complement and extend those of LlamaGen and VAR and help build a more principled foundation for scaling image-as-language models.

W6. question about the claim in Line 243, about the vocabulary threshold

We appologize that the statement in Line 243 is imprecise. Our intention was to highlight that as the vocabulary size approaches or exceeds 2^20, the memory required for the output projection (softmax layer) in next-token prediction becomes a major bottleneck—especially in image generation, where larger vocabularies are often needed for higher fidelity and resolution.

While LLMs like LLaMA handle vocab sizes up to ~200k (2^17) without issue, our use of 2^16 was meant as an illustrative threshold, not a hard limit. We will revise the wording for clarity in the final version.

[1] Yu et al. 2021, Vector-quantized image modeling with improved vqgan