Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

I would like to thank the authors for taking the time to answer my comments and for the clarifications. I believe that VAR is a strong work that should be highlighted at the conference. Hence, I would like to keep my initial score.

[Q1] Does VAR perform next-scale prediction completely in the latent space? For example, if the image resolution is 256 and VQ-VAE has a latent size of 32, does VAR operate on resolutions up to 32, or does it extend up to the image resolution?

[A4] Yes, VAR is completely done in the latent space. As mentioned in line307, our VQVAE uses a downsampling ratio of 16. So the largest latet token map $r_K$ of 256px|512px image would have a size of 16|32.

[Q2] Is there a loss term missing in equation (5)? VQ-GAN models typically include a commitment loss and a codebook loss, but it appears VAR's quantizer only uses one of these losses. Could you provide more details on this part?

[A5] The equation (5) is a simplified version of the loss function in VQ-GAN. We use the same VQVAE training loss as VQGAN so we also have a commitment loss and a codebook loss too. We'll update equation (5) to clarify this.

[Q3] Can you provide some intuition on why the transformer part can handle this prediction task effectively? Is it due to the strong conditioning from the lower scales?

[A6] Recall our motivation in line41: "Our work reconsiders how to order an image. Humans typically perceive or create images in a hierachical manner, first capturing the global structure and then local details. This multi-scale, coarse-to-fine nature suggests an order for images". If this is acknowledged, then predicting multiple tokens at the same time is natural: it just mimics how human understands or creates images.

On the other hand, the model and computation scaling is also vital to VAR's good performance. As continuing to scale-up the size of transformer and improving the expressive power of model, the model will have stronger capability to solve difficult task like VAR pretraining.

[Q4] Are the features computed by the VAR quantizer also useful for image recognition tasks, or do they perform best in image generation?

[A7] Thanks for such an interesting and enlightening question. Exploring whether VARs can improve image understanding similar to previous image pre-training work (e.g., contrastive learning and masked modelling) is a highly valuable direction. We have also included this in the Future Work section and will actively explore it in the future!

Dear Reviewer dFai,

Many thanks to your professional, detailed, and valuable reviews. We're going to response to your concerns one by one.

[W1] I suggest that the authors add more details and clarification on VAR's training process, the residual tokenization, and the workings of the transformer part. Currently, these details are somewhat obscured in Algorithm 1 and 2, and Figure 4.

[A1] Thank you for your thorough review and suggestion. We've rearranged our Method part, adding more details on the two different training stages, and make them more closed to each other.

[W2] It would be better to rewrite such claims in relation to more realistic baselines, such as RQ-VAE models, which are closer to VAR's methodology.

[A2] We agree that comparing RQ-VAE with VAR can also demonstrates the effectiveness and efficiency of VAR and without any potential overclaims. Also, this would not devalue VAR's novelty and technical contributions. We have updated these according descriptions.

[W3] Therefore, it would be more appropriate to state that VAR performs "competitively" with diffusion models rather than significantly outperforms them.

[A3] Thanks for this professional advice. We'd first explain why models like MDTv2 was not inclued in table 1: Since our table 1 mainly focuses on Generative model family comparison on class-conditional ImageNet 256x256, we did not include some latest powerful models in it. As also mentioned in line 308, our main focus was on VAR algorithm and we used a plain GPT-2 transformer without SwiGLU, RMSNorm, or Rotary position embedding. So there is still large room to boost VAR. Comparing VAR with other long-optimized diffusion model can be a bit unfair. We have added some explanations and used "performs competitively" to describe our VAR.

Nonetheless, we added an extra comparison which focuses on comparing VAR with latest, state-of-the-art model to the Appendix. We also updated the Future Work section to see if we can integrate more advanced technique like in MDTv2 or LLaMa to further upgrade VAR.

Thanks for your comments and suggestions, we will add these experiments to our revision. Feel free to let us know if you have any further questions or concerns :-).

Dear Reviewer yx9H,

[W1 Q1, about VQVAE]

[A1] We appreciate your detailed comments and will address them one by one.

1. VQVAE rFID: please see the overall Author Rebuttal part of "VQVAE concerns"
2. pre-training cost: please see the overall Author Rebuttal part of "VQVAE concerns"
3. why we didn't choose other multi-scale quantization ways: if the residual quantization is removed and independently quantization is used (e.t., independently downsampling VAE features, or independently encoding image in 16x16, 32x32, ..., 256x256), the mathematical premise of unidirectional dependency would be broken. The details can be found in line140 of our paper. To ensure that property, we have to use the residual quantization way. We'll add these explanations to our manuscript.
4. Lack of Ablation Study on VQVAE: please see the overall Author Rebuttal part of "VQVAE concerns".

[W2] Resolution Flexibility

[A2] Thanks for this detailed comment. We've observed that the multi-scale VQVAE pretrained only on 256px images, can easily generalize to higher resolutions like 512 and 1024. We have added visualizations of this to a new Appendix part.

[Q2] Question on Figure 7

[A3] Thank you for this professional question. In practice, we: 1) use a fixed sampling seed for every step, and 2) also fixed the first token which intuitively determined the global structure of the image. We found both of them are crucial for the generation consistency. We'll add these details to our manuscript.

[Q3] Question on zero-shot generation

[A4] For the details on our zero-shot generalisation algorithm, please see the overall Author Rebuttal part of "Zero-shot generalisation algorithm". For your second question, we actually allows full, bidirectional attention among tokens in the same scale $r_k$. So the model can learn inter-token dependence.

Lastly, thank you so much for helping us improve the paper and appreciate your open discussions! Please let us know if you have any further questions. We are actively available until the end of this rebuttal period. Looking forward to hearing back from you!

Thanks for this question. Unlike VQGAN or other AR image generators, where the number of steps is usually fixed, the inference steps in VAR (also the scale shapes in its VQVAE) can be slightly adjusted, e.g., replacing (1,2,3,4,5,6,8,10,13,16) with (1,2,4,6,8,10,13,16) or (1,2,3,4,5,6,8,10,12,14,16). This is possible because VAR treats a scale, rather than a single token, as an autoregressive unit, and operations based on the unit (interpolations) can generalize to different spatial resolutions. In other words, the reason is that we chose the scales in the visual signal for autoregressive generation, which is also mentioned as the Strengths1 and Strengths2 in the review comment.

Another natural question is how diffusion transformers will perform when the number of steps is small enough to approach VAR's 10 steps. We investigated this by using some ODE solvers like DDIM and DPM-Solver to reduce the diffusion inference steps. The results are as below.

It can be seen that while diffusion can be accelerated many times by the ODE solver, the sacrifice of FID becomes non-trivial when the number of steps reach 10.

When considered more broadly, we feel there's a common limitation of all Diffusion, AR, and VAR models: it is not possible to automatically determine the number of steps based on the input. For example, generating a blank blackboard clearly requires a different number of steps than a blackboard filled with math formulas. AR/VAR are expected to solve this with some modifications, like introducing an [EOS] token to allow the model itself to predict when to stop. We believe this is a valuable direction to be explored and will add it into our Limitation or Future Work section.

Regarding [A1]

Yes, we found the causal dependency (which is brought by the residual quantization) is critical to VAR's success, just as the left-to-right linguistic order is important to LLM's success. Earlier in the VAR research, we had tried the two independent encoding ways you mentioned and compared them to the causal one:

• Independently encoding images of different resolutions was nearly impossible because a 16x downsampled VQVAE could barely reconstruct an image with <=64 resolution. Huge color differences and shape deformation.

• Independent quantization can be seen as the ablation on the "residual quantization". We had tried this idea and while the reconstruction quality of the VQVAE did not change much, the VAR's validation token accuracy decreased a lot (tested on an ImageNet subset with 250 classes):

  Method	Model	#Para	Accuracy$\uparrow$
  Residual quantization	VAR-$d24$	1.0B	3.92%
  Independently quantization	VAR-$d24$	1.0B	3.01%

• Empirically, a 0.9% accuracy drop implies a huge performance degradation when the model size was close to 1B (can be sensed in Figure 5(c)). Therefore, we abandoned this independent scheme early in our research.

We do agree with you that "coarse-to-fine scaled images may indeed not strictly have causal dependency". If they don't, the whole generation process is more like a series of super-resolutions; but if they do, it's much more similar to the way human paintings work: first the whole, then the details, with each step being a refinement to all the past steps (due to the residual).

We highly believe the refinement is a key to VAR's success since it allows VAR to fix past mistakes like a diffusion model. Among AR methods, this is an unique advantage of VAR because it is even impossible for language AR models to do this -- they can't correct historical mistakes via some "residual" mechanism. We'll add these extra ablation studies and discussions to our paper and thanks for your insightful questions.

Regarding [A2]

The tokenizer consists of three parts, the CNN encoder, the quantizer, and the CNN decoder. Multi-scale exists only in the quantizer, so we just need to change the resolution hyperparameters $h_1, h_2, \dots, h_K$ in the multiscale quantizer, detailed in Algorithm 1 and 2. Since the operations in it are interpolations and convolutions which can generalize to any resolution, no additional operations are needed anymore. In other words, we only need to set $h_1, h_2, \dots, h_K$ from (1,2,3,4,5,6,8,10,13,16) to (1,2,3,4,6,9,13,18,24,32) and Algorithm 1 and 2 will still work. We'll update our paper to make this more clear.

Regarding [A4]

Yes, we agree that "predicting some spatial positions based on others" is not direct learned through VAR (which may be learned through BERT, MAE, etc.). But we checked the VAR's self-attention score when it generated an image, and observed that many tokens on a certain object's body would still show relatively high attention scores. Taking this into account, and the fact that VAR can fully utilize information from all previous scales, it still has the ability to do tasks like inpainting (Figure 8). We'll add these explanations to our paper for better presentation.

Thank you again for your insightful and professional comment, which made our work more complete and solid! If there're any further questions, please let us know. If you feel all questions have been addressed, you can kindly consider re-rating our work. Thank you!

Regarding [A1]

1. What I'm concerned is exactly this premise. From my perspective, coarse-to-fine scaled images may indeed not strictly have causal dependency if not constructed in your way. However, is this strict causal dependency fundamental to the success of your method? Or is the core idea that predicting the coarse-scale image first, followed by the fine-scale image, the key to your success (even if the strict causal dependency doesn't hold, as in the example I provided)? I believe this is a critical factor that warrants an ablation study.

2. It appears that there is no ablation study on the "residual quantization" technique. Specifically, I am referring to the case where you still construct a multi-scale token map in your VQ-tokenizer, but without applying residual quantization.

Regarding [A2]

Could you provide more details on how the 256x256 tokenizer can be directly applied to tokenize a 512x512 image?

Regarding [A4]

I understand that bidirectional attention is allowed during training. However, the model’s objective is to use this bidirectional attention to learn how to predict the next scale, rather than predicting some spatial positions based on others. My point is that the second objective was not explicitly optimized during training—can you confirm if this is accurate?

Dear Reviewer Hisd,

Many thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.

[W1] The second stage of training VAR transformers is too short and is hard for me to fully understand how it works...

[A2] Thanks for pointing this and we'll detail more on the 2nd stage of VAR method in our manuscript. To generate $h_k\times w_k$ tokens $r_k$ in $k$-th autoregressive step (like to generate $r_2$ in Figure 4), the input is the previous token map $r_1$. It'll be reshaped to 2D ($1\times 1$ here), embedded to a 2D feature map, upsampled to $r_2$'s shape $2 \times 2$, projected to the VAR transformer's hidden dimension, and added by a 2D positional embedding to get $e_1$. So the actual input to the VAR transformer is $e_1$ in Figure 4, and these steps are the details of "word embedding and up-interpolation" noted in Figure 4. The detailed implementation can also be found in the code "codes/models/var.py autoregressive_infer_cfg" attached in supplementary material.

To make sure all tokens in $r_k$ are correlated to each other, we don't apply attention mask within $r_k$ to keep a bidirectional attention on them. In other words, $r_k$ can attend to all tokens of $r_{\le k}$

[W2] there are 10 scales (1,2,3,4,5,6,8,10,13,16). I wonder if there is any motivation to choose these scales.

[A2] Yes the chose of scales is a key design of VAR. We choose an exponential function $h_k=w_k=\lfloor a\cdot b^k\rfloor$ to get these scales because:

1. As discussed in appendix F, we can reach a total complexity of $\mathcal{O}(n^4)$ via an exponential schedule.
2. We want to increase the number of steps to reach $16\times 16$ for image quality. An exponential function grows slowly in the early stages and quickly in the later ones. So it allows us to increase steps (mainly in the early stages) without a significant increase in total sequence length.

In practice, we use $a=1.36$ and $b=1.28$ to get that (1,2,3,4,5,6,8,10,13,16).

[W3] I think the paper should include the sampling algorithm of autoregressive model with hyper-parameter details such as temperature, top-k, top-p and CFG sampling.

[A3] We use temperature of $1.0$, top-k of $k=900$, top-p of $p=0.96$, and CFG of $1.25$ (VAR-$d16$) or $1.5$ (the others). We'll add these to our manuscript.

[W4] The zero-shot generalisation algorithm should be included in the paper for clarity.

[A4] Thanks for this valuable advice. We'll add a pseudo code for detailing the algorithm of zero-shot generalisation in our manuscript. The algorithms of inpainting, outpainting, and class-condition editing are basically the same.

Specifically, we would mask out the according region at each scale of (1,2,3,4,5,6,8,10,13,16) given the task, i.e., masking the inner area for inpainting, the outer area for outpainting, and the area we want to edit for editing. During VAR generation, ground-truth non-masked tokens are maintained (like teacher forcing) and we only collect VAR's predictions on those masked regions.

Thank you again for helping us improve the paper and hope our response can resolve your concerns! Please let us know if you have any further questions. We will be actively available until the end of rebuttal period. If you feel your concerns are addressed, please consider reevaluating our work. Looking forward to hearing from you :-) !

We're glad to hear your concerns have been addressed. We'll be active till the end of the discussion period. If you have more questions, please let us know. Thank you!

Dear Reviewer oCJa,

Many thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.

[W1 Q1, about the efficiency evaluation] I'd like to see a throughput comparison in img/s as the batch size is increased. I'd also like to see some DiT results where a more advanced sampler such as DPM-solver or SA-solver is used.

[A1] Thank you for your suggestions to make our efficiency evaluation more complete and thoughtful!

First, we add the throughput comparison with batched inference to our manuscript. All models are tested on a single A100 GPU. VQGAN's results are quoted from [1]. From the results we can see that VAR also gets a big speedup from batch processing, similarly to vanilla autoregressive models. We attribute this to the short sequence length of the first few VAR steps (like generating $1\times 1$ and $2\times 2$ tokens). After batching, VAR reaches a throughput of 0.04 $\text{img/s}$, which is still 4 times faster than batched VQGAN, even if VAR has a larger model size.

Second, we add the results of DiT and U-ViT (another transformer-based diffusion model) with fewer diffusion steps ($<50$) to our manuscript. The results are quoted from [2, 3]. From the table one can see that:

1. Reducing the number of diffusion steps via ODE sampler (like DDIM, DPM-Solver) will result in a large FID rise, especially when reduced to near 10 steps.
2. Both using 10 steps, VAR-1B is still faster and better than DiT-675M. This is also because VAR has a shorter sequence length in the early steps, e.g., the first three VAR steps only generate $1\times 1 + 2\times 2 + 3\times 3 = 14$ tokens.
3. The Diffusion community has been building up for a long time for efficiency boosting. In contrast, VAR, as a newly proposed method, promises to see more ways to accelerate or distill VAR in the future. We'd like to add this to our Future Work section.

[W2 Q2, details on multi-scale VQVAE]

[A2] We'd like to respond to your queries mentioned in Weakness 2 one by one, and add all the details below to Appendix section. Please see the overall Author Rebuttal part of "VQVAE concerns" for specific responses.

[W3 Q3, The use of "zero-shot" to refer to the model's editing ability is different in nature to zero-shot generalisation in LLMs; doesn't give details on how the editing is performed]

[A3] Thanks for your insightful comments. In the field of language processing, it has been verified that every task can be formulated to an autoregressive generation task. This allows a pretrained LLM can generalize to many tasks different to its pretraining task without any finetuning. In our model, we also use the term "zero-shot" to emphasize that our model can also do tasks that are different to our pretraining task. We have updated some according descriptions in our paper to make it more clear and accurate.

We added enough details on how the inpainting/outpainting/editing is performed to our manuscript. You can also find them in the overall Author Rebuttal part.

Many thanks to Reviewer oCJa for their professional, detailed, and valuable reviews! We have done our best to address each of your concerns and hope our response can resolve them. Please let us know if you have any other questions. We will actively join the discussion until the end of the rebuttal period. We are looking forward to hearing from you :-) !

Thanks for the insightful question on the throughput improvement. We further investigate the batched inference using batch size of 150, 200, and 250:

Observation. When the batch size gets larger than 100, the throughput improvements of both VQGAN and VAR becomes marginal. Upon reaching the largest batch size without out of memory issue, VAR still shows 3.6x throughput, though it was larger and has more tokens to infer.

Analysis. To understand why VQGAN does not present higher throughput than VAR when larger batch sizes are used, we further check the behaviors of VAR and VQGAN when they do autoregressive inference -- we plot their attention masks. (since NeurIPS does not allow authors to upload an image or provide an external link, we put the python code here and it'll plot the masks)

import torch
import matplotlib.pyplot as plt

patch_nums = [1, 2, 3, 4, 5, 6, 8, 10, 13, 16]
L = sum(pn**2 for pn in patch_nums)
d = torch.cat([torch.full((pn * pn,), i) for i, pn in enumerate(patch_nums)]).view(1, L, 1)
dT = d.transpose(1, 2)
mask_VAR = (d >= dT).reshape(L, L).numpy()

mask_AR = torch.tril(torch.ones(patch_nums[-1]**2, patch_nums[-1]**2)).numpy()

fig, axs = plt.subplots(1, 2, figsize=(10, 5))
axs[0].imshow(mask_VAR), axs[0].set_title('mask_VAR'), axs[0].axis('off')
axs[1].imshow(mask_AR), axs[1].set_title('mask_AR'), axs[1].axis('off')

plt.show()

From the figure one can see that:

• VAR's block-wise causal mask and VQGAN's standard causal mask look very close.
• Or in other words, VAR still maintains many AR properties.
• So both VQGAN and VAR can benifit from the batched inference, and their batch size sweet-spots (when the throughput starts saturating) can be close to each other.

We will add all of the above supplementary results and analysis to a new Appendix section named throughput benchmark using batched inference. Thank you again for your constructive and detailed response.

Thank you again for your professional review which did strengthen our paper. For the two more questions:

• Yes it's a typo. All "img/s" should be "s/img". We've corrected this in our manuscript.

• Yes, when the spatial resolution is too low, some interpolations could be inaccurate. But the smallest token map would be 2x2 because during inference the 1x1 was the start token. We've added this to our limitation.

• Here we provide more details on that "interpolation": Considering an inpainting case on a 256x256 image where the upper left NxN pixels are masked. To mask the smallest 2x2 token map, the binary mask $M$ will be interpolated to 2x2 $M_2$. The upper left token on the 2x2 token map will be masked only if $M_2^{(0,0)} \ge 0.5$. We've added these to the Appendix to make it more clear.

It seems that the manuscript does not contain your added information. Does NeurIPS allow revising manuscript during rebuttal period?

Dear Reviewer yx9H,

1. In the "Author responses" part at https://neurips.cc/Conferences/2024/CallForPapers , it is said that "Authors may not submit revisions of their paper or supplemental material, but may post their responses as a discussion in OpenReview." This policy does cause some inconvenience, but we have done our best to describe all revisions and copied all new experimental results to our responses.

2. We went through each of your questions and our responses again, and found that only one revision ([W2] Resolution Flexibility) was not copied to our response. So we provide more evidence for you:

   • The VAR transformers generating 256 and 512 images both use the same multi-scale VQVAE that is trained only on 256 resolution. The good performance of the VAR on the 512 generation task shows the generalizability of VQVAE.

   • This 256-trained multi-scale VQVAE has a good reconstruction FID of 2.28 on the 512 ImageNet validation set.

3. For your convenience, here we also provide a complete response to your [Q1] question about VQVAE, so you don't need to search them in the Overall Author Rebuttal:

• VQVAE rFID: We added an evaluation of rFID to a new Appendix section of "Comparisons on VAEs" and also pasted it here. It evaluated different VAEs on the 256px ImageNet validation set. Generally, the number of tokens is crucial to a VAE's performance, which shows a trade-off between sequence length and reconstruction FID (rFID). VAR-VQVAE with 680 tokens performs similarly to VQGAN's with 1024 tokens. | Model | downsampling| num of tokens | rFID$\downarrow$| |:-|:-|:-|:-| | VQGAN-VQVAE | 16 | 256 | 4.90 | | VQGAN-VQVAE | 8 | 1024 | 1.14 | | MaskGIT-VQVAE | 16 | 256 | 2.28 | | VAR-VQVAE | 16 | 680 | 1.00 | | SDXL-VAE | 8 | 1024 | 0.68 |

• pre-training cost: we've added more hyperparameters and training costs in Appendix B: Implementation details, which are: 16 epochs training on OpenImages, batch size 768, fp16 precision by torch.cuda.amp, standard AdamW optimizer with betas (0.5, 0.9), learning rate 2e-4 and weight decay 0.005. The loss weight of L2 reconstruction, L1 reconstruction, codebook loss, commitment loss, LPIPS and discriminator are 1.0, 0.2, 1.0, 0.25, 1.0, 0.4. This training will take ~60h on 16 A100 80G GPUs.

• why we didn't choose other multi-scale quantization ways: if the residual quantization is removed and independently quantization is used (e.g., independently downsampling VAE features, or independently encoding images in 16x16, 32x32, ..., 256x256), the unidirectional dependency of VAR algorithm would be broken. The details can be found in line140 of our paper. To ensure that property, we have to use the residual quantization. We'll add these explanations to our manuscript.

• Lack of Ablation Study on VQVAE: Since this work mainly aims to explore a new VAR algorithm, we keep the VQVAE structure as simple as possible. Specifically in line173, we use the same architecture of VQGAN's single-scale VQVAE and the multi-scale modules (several convolutions) only contain 0.03M parameters. So comparing VQGAN-VQVAE and our VQVAE can be seen as an ablation of this VQVAE. We've added these to our paper.

We hope the above response can help solve your questions. Thanks again for your thorough review and looking forward to your reply!