Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation

We truly thank the reviewer for your detailed review and thoughtful comments in helping us enhance the quality of our paper. Below, we provide detailed point-to-point responses addressing each of your valuable suggestions:

Q1: Training overhead of ST-AR.

In Table D1, we quantify the training overhead introduced by ST-AR. All experiments are conducted using 8 NVIDIA A100 GPUs (80GB each), with a global batch size of 256 and an input image resolution of $256 \times 256$. We report the average training time per iteration as well as the peak memory consumption during training. The training overhead of ST-AR is higher than the baseline LlamaGen but comparable to that of REPA. However, ST-AR achieves significantly better performance gains. Reducing the training cost of ST-AR will be a key direction for our future work.

Table D1: Comparison of training costs.

Q2: Effect of Batch Size

We evaluate the impact of batch size on generation performance and representation quality. As shown in Table D2, a larger batch size is more beneficial for semantic information learning, resulting in higher linear probing accuracy. However, in terms of generation quality, a batch size of 512 performs worse than 256. We attribute this to the reduced number of training iterations when using a larger batch size.

Table D2: Effect of batch size.

Q3: Extending ST-AR to other AR domains.

One of the key contributions of our work is establishing a methodology for analyzing autoregressive generation models. By quantifying the scope of the attention map and evaluating linear probing performance, we gain insights into the intrinsic mechanisms of autoregressive models.

To extend ST-AR to other domains, such as video generation, conducting similar analyses is a crucial first step. Intuitively, for video generation, we would need to further examine the attention weights between frames and assess the semantic consistency across frames. Based on these insights, we could then introduce additional high-level inter-frame representation alignment losses to ST-AR, aiming to enhance the video generation performance of baseline models.

Q4: High norm issue on the first patch

The input at the first time step is a conditional vector that guides the generation process. It is reasonable for this vector to always have a high norm.

To validate the effect of register tokens, we inserted 4 register tokens between the first token (conditional vector) and the second token (the first image patch). The image generation process starts from the last register token. As shown in Table D3, the introduction of register tokens leads to a slight performance improvement. However, visualizations of the attention maps reveal that the conditional vector still exhibits a high norm.

Table D3: Effect of register tokens.

Thank you for reviewing our work and providing positive feedback. We are glad to address any further questions or comments.

We truly thank the reviewer for your detailed review and thoughtful comments in helping us enhance the quality of our paper. Below, we provide detailed point-to-point responses addressing each of your valuable suggestions:

Q1: Training cost of ST-AR.

In Table C1, we provide a fair comparison of the training costs for LlamaGen, REPA [1], and ST-AR. All experiments are conducted under the same training settings, using 8 A100 GPUs with a global batch size of $256$ and an input image resolution of $256\times256$. We report the average runtime per training iteration and peak memory usage. Compared to the baseline LlamaGen and the REPA (which requires a pre-trained image encoder), ST-AR incurs slightly higher training costs but delivers significant performance improvements. Reducing the training cost of ST-AR will be an important focus of our future work.

Table C1: Comparison of training costs and comparison between ST-AR and pretraining-based methods.

Q2: Comparison with methods requiring a pre-trained image encoder.

We follow REPA[1] to align the intermediate features of LlamaGen-B with the representations from DINOv2-B and MAE-B, respectively. As shown in Table C1, LlamaGen-B with REPA demonstrates better generation performance compared to the baseline but falls short of ST-AR.

We analyze the pros and cons of ST-AR and pretraining-based methods from two perspectives:

1. Spatial misalignment when aligning intermediate features from autoregressive models with pre-trained visual representations. For position $i$, an autoregressive model involves two adjacent tokens: the $(i-1)$-th token as the input and the $i$-th token as the predicted output. In contrast, visual encoders like DINOv2 extract features from the $i$-th image patch. Forcing alignment may introduce inductive bias into the autoregressive model. In comparison, ST-AR does not rely on pretrained visual encoders, thereby avoiding the spatial misalignment.
2. Representation quality. While visual encoders pretrained on larger-scale datasets may learn better semantic representations, these representations do not necessarily contribute to learning for image generation. In contrast, ST-AR unifies representation learning and image generation, allowing it to directly learn features that are more beneficial for autoregressive image generation.

Q3: The effect of $\alpha$ and $\beta$ and the balance between AR loss and self-supervised losses.

In Table C2, we examine the effect of $\alpha$ and $\beta$ on generative performance. When training for 50 epochs:

• ST-AR performs best when $\alpha=1.0$ and $\beta=0.5$.
• $\alpha$ is the weight of the MIM loss. Reducing the value of $\alpha$ decreases the model's effective receptive field, leading to poorer generation quality.
• $\beta$ is the weight of the two contrastive losses. A lower value ($0.25$) makes it difficult to learn semantic information, while values greater than $0.5$ might hinder the learning of the autoregressive loss.

For training dynamics, we evaluate the performance of ST-AR with different $\alpha$ and $\beta$ at $20$ epochs and $50$ epochs. As in Table C2, the best performance is achieved by set $\alpha=1.0$ and $\beta=1.0$ at $20$ epochs. This indicates that larger $\alpha$ and $\beta$ contribute to learning better semantic representations, which accelerates the convergence of the autoregressive model. However, larger contrastive loss may hinder the learning of autoregressive loss, leading to inferior performance of setting $\{\alpha=1.0, \beta=1.0\}$ at $50$ epochs.

Table C2: Ablation study for $\alpha$ and $\beta$.

Q4: The interaction between the reconstruction objective and the contrastive objectives.

The interaction between the reconstruction objective and contrastive objectives exhibits both conflict and mutual benefits.

Conflict. MIM and contrastive learning capture different levels of semantic information, resulting in a unique challenge of representation conflict. The inter-step and inter-view contrastive losses are designed to address semantic inconsistencies across steps and views, respectively, promoting the learning of global, image-level semantics. In contrast, the MIM loss trains the autoregressive (AR) model by reconstructing the correct semantics of masked tokens, emphasizing token-level semantics. To mitigate this conflict, we compute the contrastive losses on intermediate-layer features and the MIM loss on final-layer features. As in the Table 5 of our manuscript, this design can bring the best performance.

Mutual Benefits. As shown in Figure A1 of the supplementary material, contrastive objectives face limitations in expanding the effective receptive field, which restricts the learning of wide-range high-level semantics (e.g., objects). The reconstruction objective effectively addresses this issue. On the other hand, the reconstruction objective alone struggles to capture precise semantic entities, whereas the contrastive objectives complement and expand the capabilities of MIM, leading to more robust learning of semantic representations.

Q5: Applying contrastive losses to the middle layer.

The contrastive losses in ST-AR aim to align high-level semantic information across different timesteps and views. Notably, the most semantically meaningful features exist in the middle layer of the autoregressive model.

The autoregressive model processes preceding tokens as input to predict the next token, which can conceptually be divided into two stages:

1. The shallow subnetwork maps the preceding tokens into a high-level representation.
2. The deeper subnetwork decodes this high-level representation into the next token.

The most semantically meaningful features are produced in the middle layer rather than the final layer. This claim is supported by the linear probing results shown in Table C3.

Table C3: Linear probing results of different layers in LlamaGen-B.

[1] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. ICLR 2025

Thank you for your effort in reviewing our work and for your positive feedback. We are happy to address any further questions or comments.

We truly thank the reviewer for your detailed review and thoughtful comments in helping us enhance the quality of our paper. Below, we provide detailed point-to-point responses addressing each of your valuable suggestions:

Q1: Comparison on other baselines and datasets.

In the supplementary material, we provide the results of applying ST-AR on the vanilla Transformer, also in Table A1. Besides, we provide the text-to-image generation results of ST-AR and LlamaGen in the supplementary material, also in Table A2.

For the vanilla autoregressive Transformer, we follow the training setup of LlamaGen and conduct comparative experiments on the ImageNet-256x256 benchmark. As shown in Table A1, ST-AR consistently and significantly improves the image generation performance of the vanilla Transformer.

Table A1: Results of vanilla Transformers on ImageNet-256x256 Benchmark.

For text-to-image generation experiments, we train LlamaGen-XL and ST-AR on a 2M subset of the SAM-11M dataset with an image resolution of $256 \times 256$. We use a pretrained FLAN-T5-XL to extract text embeddings, where the maximum embedding length is set to 120. We use FID and CLIP Score to evaluate image quality and text-image alignment on the COCO-val-30K benchmark. As shown in Table A2, ST-AR also achieves significant improvements on text-conditional generation, further demonstrating the generalizability of ST-AR.

Table A2: Results of LlamaGen-XL on the text-conditional COCO-val benchmark.

Q2: Comparison with pretraining-based methods

Aligning the intermediate features of autoregressive models with pretrained representations introduces spatial misalignment. Specifically, for position $i$, pretrained vision encoders extract features of the $i$-th image patch, whereas autoregressive models take the $(i-1)$-th image token as input to predict the $i$-th token. Such behavior is fundamentally distinct from diffusion models. Therefore, directly applying representation alignment methods, such as REPA[1], to autoregressive models is suboptimal. To validate this, we apply REPA to LlamaGen, strictly following the settings of REPA, aligning the intermediate features of LlamaGen-B with representations from DINOv2-B or MAE-B. As shown in Table A3, while REPA achieves better generation performance compared to the baseline, it is still inferior to ST-AR.

Table A3: Comparison between ST-AR and pretraining-based methods.

Q3: Applying ST-AR to pretrained representations.

We also apply ST-AR to pretrained representations (the last row of Table A3). We align the intermediate features from the $6$-th layer of LlamaGen-B with the visual representations extracted by pretrained DINOv2. We retain the three losses of ST-AR and set the weight of the alignment loss to $0.5$. As shown in Table A3, by incorporating additional pretrained representations, the performance of ST-AR is further improved. We attribute this improvement to the richer semantic information learned by DINOv2 on a larger-scale dataset.

Q4: Choice of contrastive loss depth.

The contrastive losses in ST-AR are designed to align the high-level semantic information across different timesteps and views. In Table 5 of our manuscript, we explore the impact of contrastive loss depth and demonstrate that applying the losses at half of the model depth achieves the best performance for ST-AR. Here we provide more insights about this design. REPA [1] and REPA-E [2] propose to divide a diffusion model into an encoder (shallow layers) and a decoder (deeper layers), where the encoder implicitly learns a representation that reconstructs the target. We find that the above perspective also holds true in autoregressive models. Table A4 shows the linear probing performance of features extracted from different layers of LlamaGen-B, where the $6$-th layer achieves the best representation quality. This demonstrates that autoregressive models exhibit behavior similar to diffusion models. Specifically, for step $i$, the shallow subnetwork (encoder) of the autoregressive model maps preceding tokens into a high-level representation, which is then decoded by the deeper subnetwork (decoder) to predict the $i$-th token. Therefore, applying contrastive losses ($L_{step}$ and $L_{view}$) to the intermediate features is an effective strategy.

Table A4: Linear probing results of different layers in LlamaGen-B.

Reference

[1] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, ICLR 2025

[2] REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers, ICCV 2025

We truly thank the reviewer for your detailed review and thoughtful comments in helping us enhance the quality of our paper. Below, we provide detailed point-to-point responses addressing each of your valuable suggestions:

Q1: Innovations of ST-AR

To the best of our knowledge, we are the first to explore the representation learning capabilities of AR models and to identify the challenges of transferring next-token prediction to the visual domain. The innovations of ST-AR can be summarized in two main aspects:

• We introduce an effective methodology to analyze and identify the limitations of autoregressive models in visual representation learning by evaluating attention maps and linear probing performance.
• We demonstrate that by appropriately incorporating well-established self-supervised learning techniques, the representation quality of AR models can be significantly improved, which subsequently enhances generation performance.

Furthermore, the application of established self-supervised approaches to AR generation has not been explored before. ST-AR is the first unified framework that bridges representation learning and AR image generation.

Notably, ST-AR outperforms methods that align AR features with pre-trained representations (e.g., REPA). In Table C1, we align the intermediate features of LlamaGen-B to those of either DINOv2-B or MAE-B, but both approaches perform worse than ST-AR. We attribute this to two reasons:

• Spatial misalignment between AR features and pre-trained representations. In AR models, each time step uses the preceding token as input and predicts the next token as output, while pre-trained visual encoders focus on a single token. Directly aligning representations from these two paradigms is suboptimal. For more details, please refer to our response to Reviewer rKee Q2.
• Uncertainty in desired representations for AR models. Different pre-trained visual encoders learn different types of representations, and it remains unclear what specific kinds of representations are optimal for AR models. ST-AR avoids introducing inductive bias by employing a self-guided approach that jointly trains representations and generation within a unified framework.

Table C1: Comparison between ST-AR and pretraining-based methods.

Q2: The importance of pure visual representation learning.

The effectiveness of incorporating high-level visual representations into generative models has been well demonstrated[1][2]. Notably, REPA conducts the comparison between CLIP features and pure visual representations (e.g., DINOv2), showing that DINOv2 provides a stronger boost to generation performance than CLIP. This suggests that the potential of pure visual representation learning is not inferior to vision-language alignment.

Pure visual representation learning and vision-language alignment (e.g., CLIP) each have their own strengths and weaknesses, and they can be combined to obtain richer and more comprehensive representations. Enhancing image understanding through pure visual representation learning remains necessary and valuable.

• Representation quality. Pure visual representation learning excels at capturing high-quality features of diverse objects. For example, DINO features have been effectively utilized for unsupervised instance segmentation[3][4]. In contrast, CLIP focuses on aligning image content with text embeddings, which can sometimes result in slightly inferior features.
• Data Requirements. Pure visual representation learning requires only a large collection of images as training data, making it less resource-intensive in comparison to vision-language alignment methods like CLIP, which depend on costly textual annotations paired with images.

In practice, combining pure visual representation learning with vision-language alignment can significantly enhance visual understanding at various levels. For instance, SPHINX-X[5] leverages both DINOv2 and CLIP as its visual encoders to achieve superior performance.

Reference

[1] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, ICLR 2025

[2] REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers, ICCV 2025

[3] Cut and Learn for Unsupervised Object Detection and Instance Segmentation, CVPR 2023

[4] Freesolo: Learning to segment objects without annotations. CVPR 2022

[5] SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models. ICML 2024