Autoregressive Image Generation without Vector Quantization

We thank the reviewer for appreciating our clean and neat approach as well as our solid empirical results. Here, we address the weaknesses (W) and questions (Q) raised by the reviewer.

Inference speed (W1)

We agree that DiffLoss can introduce overhead to the inference process, and there is always a trade-off between a larger MLP with more capacity and increased computation overhead. In our ImageNet experiment, a small MLP with a depth of 3 and a width of 1024 achieved favorable performance (1.97 FID) with marginal computational overhead during inference. For large text-to-image datasets, we assume that increasing the capacity of the autoregressive transformer will be more important than increasing the capacity of the MLP in Diffusion Loss. This is because the role of Diffusion Loss is to model a single token with a few dimensions (e.g., 16), which remains a relatively easy task even on large-scale datasets.

More experiments (W2)

We have provided an additional experiment on ImageNet 512x512 in the general response to validate our model’s ability to generalize across different resolutions. Due to resource and time constraints, we are unable to conduct extensive experiments on large-scale text-to-image datasets. However, similar to reviewer xuXX, we believe that the simplicity and standard implementation of the proposed Diffusion Loss makes it easy to be generalized to other datasets and tasks.

Other choices to model $p(x|z)$ (Q1)

This is an excellent question. Conceptually, $p(x|z)$ can be modeled by any conditional generative model and is not limited to diffusion models. We chose diffusion models due to their simplicity and superior performance, as demonstrated in recent literature. We believe that other generative models, such as VAE, GAN, and consistency models, also have the potential to model $p(x|z)$. Exploring these alternatives would be a very interesting future direction.

We thank the reviewer for appreciating our approach and the comprehensive experimental results. It seems there is a key misunderstanding about our paper that “the proposed method still needs a discrete tokenizer”. In fact, ALL experimental results in our paper using Diffusion Loss are conducted with continuous KL-based tokenizers, except for the first row in Table 2, where we show that Diffusion Loss can also work well with a discrete VQ-based tokenizer. We believe this misunderstanding may be the primary reason the reviewer questions our motivation and method. We hope the reviewer will reconsider the rating in light of this clarification. Below, we address the specific weaknesses (W) and questions (Q) raised.

Motivation (W1)

The reviewer questions the motivation of the paper, stating that "the authors neither provide sufficient examples supporting 'conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens,' nor demonstrate the unnecessity of vector quantization.” We respectfully disagree with these two statements. As stated in L21-25 and 68-70, almost all recent works on autoregressive image generation rely on vector-quantized tokens, starting from VQGAN [1] and DALL-E [2] to MaskGIT [3] and MAGE [4], and more recently, VAR [5] and TiTok [6]. As a side proof, Reviewer xuXX mentioned, "I always believed that sticking to discrete tokens was necessary to train AR models," and Reviewer V3y3 noted, "the paper introduces a novel approach of using continuous-valued tokens in autoregressive models for image generation, challenging the conventional wisdom of discrete vector-quantized tokens." These comments strongly support the notion that "conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens."

Moreover, as mentioned earlier, all major experimental results in our paper using Diffusion Loss DO NOT use VQ-based tokenizers. In Table 1, we show that Diffusion Loss, when used with a continuous KL-16 tokenizer, outperforms its cross-entropy counterpart with a discrete VQ-16 tokenizer, consistently observed across all variants of AR and MAR. This result clearly demonstrates the unnecessity of vector quantization and the superior advantage of using Diffusion Loss to model continuous-valued tokens.

Complication of Diffusion Loss (W2)

We respectfully disagree with the statement that “the diffusion process is complicated and hard to converge.” The intuition behind the diffusion model is quite simple: adding and removing noise. The mathematical formulation is also straightforward, involving a standard SDE equation. Many generative modeling works in recent years have empirically demonstrated that diffusion models are not hard to train and converge, and they have been successfully applied to various domains and problems.

We also respectfully disagree with the reviewer’s statement that “the authors just replace a complicated method with another complicated method.” We use a simple MLP controlled by only two parameters, depth and width, as the denoising network. We employ a very standard diffusion process from iDDPM [7], which is widely used by common diffusion models such as ADM [7] and DiT [8]. The simplicity of Diffusion Loss is also highly appreciated by other reviewers. Reviewer xuXX states that “the architecture does not contain any specialized components or complex diffusion loss.” Reviewer oAwj notes that “the idea is simple, clean, and neat.” Thus, we believe that the proposed Diffusion Loss is a straightforward yet highly effective module that enables autoregressive image generation on continuous-valued tokens, achieving superior performance compared to traditional autoregressive models on discrete-valued tokens.

Unification of autoregressive and masked generative models (W3)

First, we want to clarify that the unification of autoregressive (AR) and masked generative (MAR) models is not intended as a novel technical contribution, as both approaches have been thoroughly explored in the literature [1, 3, 4, 9]. Instead, as stated in L47-51, the unification aims to broaden the scope where DiffLoss can be applied and can be helpful.

Second, the two works [10, 11] mentioned by the reviewer focus on self-supervised representation learning rather than generative models. It's important to note that within the visual generative modeling community, masked generative models are not commonly recognized as autoregressive models. In fact, many of these models are referred to as “non-autoregressive models” in the literature [3, 9]. One goal of this paper is to show that these models still possess the autoregressive characteristic of “predicting next tokens based on known ones,” allowing them to seamlessly use Diffusion Loss. As noted by Reviewer V3y3, “the unification of autoregressive and masked generative models under a generalized framework is a great contribution to the field.”

Position ambiguity (Q1)

Similar to MAE [12], we add a learnable positional embedding to each masked token to indicate its position in the sequence.

[1] Taming Transformers for High-Resolution Image Synthesis

[2] Zero-Shot Text-to-Image Generation

[3] MaskGIT: Masked Generative Image Transformer

[4] MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis

[5] Scalable Image Generation via Next-Scale Prediction

[6] An Image is Worth 32 Tokens for Reconstruction and Generation

[7] Diffusion Models Beat GANs on Image Synthesis

[8] Scalable Diffusion Models with Transformers

[9] Muse: Text-To-Image Generation via Masked Generative Transformers

[10] Bridging Autoregressive and Masked Modeling for Enhanced Visual Representation Learning

[11] Self-supervision through Random Segments with Autoregressive Coding (RandSAC)

[12] Masked Autoencoders Are Scalable Vision Learners

We thank the reviewer for the appreciation of the clear motivations, original contributions, and significant benefits of our work. Here we address the weaknesses (W) and questions (Q) raised by the reviewer. We are also happy to discuss further if you have any additional questions or confusion.

Analysis of Information Loss (W1)

As pointed out by the reviewer, our experiment results in Table 2 and the attached PDF in the general response demonstrate that the reconstruction quality of VQ-based tokenizers is significantly worse than that of KL-based tokenizers. We believe this is because VQ-based tokenizers experience much more information loss due to their higher compression ratio.

Here, we provide an informal reasoning about the information loss by analyzing the compression ratio of VQ-based and KL-based tokenizers. For example, a VQ-16 tokenizer with a codebook size of 1024 tokenizes a 256x256 image into a 16x16 sequence of discrete indices, each of which can be represented by 10 bits. In contrast, a KL-16 tokenizer encodes the same image into a 16x16x16 sequence of float numbers. Thus, the compression ratio of the VQ-16 tokenizer is much higher than that of the KL-16 tokenizer, resulting in significantly more information loss. Therefore, the lower bound of information loss by modeling KL-16 continuous tokens using Diffusion Loss is lower compared to modeling VQ-16 discrete token indices using cross-entropy loss.

We also believe a formal analysis could further strengthen the theoretical foundation of Diffusion Loss. However, due to the significant efforts required, we leave this for future work.

Explicit prior or posterior for the whole AR system (W2)

One advantage of using a categorical distribution is that it can explicitly compute the probability of each sampled discrete token, which can then be used to compute the posterior of the entire AR system. While diffusion models are typically implicit generative models, [1] has demonstrated that it is possible to compute the exact likelihood of each continuous token using a probability flow ODE. This could potentially enable computing the posterior of the AR+DiffLoss system, offering an intriguing direction for future exploration.

[1] Score-Based Generative Modeling through Stochastic Differential Equations

Directly modeling the pixel space (W3)

This is an excellent question. Diffusion Loss is independent of the tokenizer and can be directly applied in the pixel space. Given the limited time for the rebuttal, we conduct a preliminary experiment on ImageNet 64x64, where we group every 4x4 pixels into one token for Diffusion Loss to model. A MAR-L+DiffLoss model trained for 400 epochs achieved an FID of 2.93, demonstrating the potential to eliminate the need for tokenizers in autoregressive image generation with diffusion loss. However, as commonly observed in the literature on diffusion models, directly modeling the pixel space is much more computationally expensive than using a tokenizer [2]. For MAR+DiffLoss, directly modeling pixels at a higher resolution might require either a much longer sequence length for the autoregressive transformer or a much larger network for Diffusion Loss to handle larger patches.

[2] High-Resolution Image Synthesis with Latent Diffusion Models

Is AR still necessary with diffusion loss (Q1)

We believe that one major advantage of AR/MAR models is their ability to decompose a complex joint probability distribution $p(x_1, \cdots, x_n)$ into the product of multiple simpler conditional distributions $\prod^n_{i=1} p(x^i~ |~ x^1, ..., x^{i-1})$. Therefore, we believe AR/MAR models will continue to play an important role in modeling complex data distributions, such as high-resolution images or videos. Our paper aims to pave the way for this by eliminating the need for quantization and enabling the autoregressvie modeling of continuous distributions.

End-to-end training of decoder (Q2)

We thank the reviewer for pointing out this possibility. Since Diffusion Loss eliminates the need for vector quantization, the reconstruction loss on pixels can be directly back-propagated to the continuous token output by the AR+DiffLoss part. This makes it possible to train the entire framework in an end-to-end manner. We believe exploring this possibility would be a very interesting future direction.

Direct regression using MSE loss (Suggestion)

We thank the reviewer for the suggestion. A naive baseline to model continuous-valued tokens is to compute the Mean Squared Error (MSE, i.e., L2) loss directly between the predictions and the target tokens. In the case of a raster-order AR model, using the L2 loss introduces no randomness and thus cannot generate diverse samples. In the case of MAR models with L2 loss, the only randomness is the sequence order; the prediction at a location is deterministic for a given order. In our experiment, we trained an MAR model with L2 loss, which, as expected, led to a disastrous FID score ($>$100). We will include this in the revision.

We thank the reviewer for the appreciation of our motivation and proposed method. Here we address the weaknesses (W) and questions (Q) raised by the reviewer.

Scope of experiments (W1)

We have provided an additional experiment on ImageNet 512x512 in the general response to validate our model’s ability to generalize across different resolutions. Due to resource and time constraints, we are unable to conduct extensive experiments on large-scale text-to-image datasets. However, we agree with the reviewer that the simplicity and standard implementation of the proposed Diffusion Loss make it easy to be generalized to other datasets and tasks, including text-to-image generation.

Adding CLS tokens (Q1)

If no additional padding is used (i.e., only one [CLS] token), the encoder's input sequence can become very short at high masking ratios, sometimes as short as one token when the masking ratio is 100%. This can make the training unstable and hurt the generation performance due to the limited amount of computation spent in the encoder when the masking ratio is high. To address this, we pad the encoder sequence with 64 [CLS] tokens at the beginning, ensuring a minimum sequence length of 64. This padding stabilizes training and enhances the encoder's computation, leading to improved generation performance.

The difficulty in training VQ-based tokenizers (Q2)

We apologize for the confusion. By “difficult to train,” we mean that VQ-based tokenizers typically require much more specialized techniques compared to standard KL-based tokenizers to achieve good reconstruction performance. These techniques include replacing dead codes, L2 normalization on the encoded latent variables, logit-Laplace loss, affine reparameterization of the code vectors, synchronized and alternating training, etc [1, 2]. This also introduces many additional hyper-parameters to tune. With similar specialized techniques and hyper-parameter tuning, the reconstruction quality of VQ-based tokenizers can lag significantly behind their continuous KL-based counterparts, as shown in the attached PDF in the general response. We will clarify this point in the revision.

[1] Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks.

[2] Vector-quantized Image Modeling with Improved VQGAN

Tokenizer stride (Q3)

Yes, tokenizer stride means the downsampling factor of the tokenizer.

The gap between CrossEnt and Diff Loss is smaller for the AR model than the MAR model (Q4)

We speculate that the generative power of the AR model is limited by its raster order and causal attention mechanism, which are not as well-suited for image generation as random order and bidirectional attention. Thus, the bottleneck for the AR model may lie in the generative capacity of the backbone transformer architecture rather than the nature of the data distribution, whether continuous or discrete. In contrast, the MAR model has sufficient capacity to effectively model the data distribution, with the primary bottleneck being the information loss caused by the VQ tokenizer. This might explain why Diffusion Loss sees larger gains in MAR models than in AR models. A definitive answer to this question requires further extensive experiments, which we leave as future work.

Tokenizers with mismatched strides (Q5)

The purpose of the experiments with KL-16 and KL-8 is to demonstrate the flexibility provided by Diffusion Loss, which decouples the downsampling factor of the tokenizer from the sequence length of the autoregressive transformer. This flexibility eliminates the need to retrain the tokenizer for each sequence length. More importantly, similar to what is seen in DiT, it allows for a customizable choice of the transformer's sequence length, enabling a tailored trade-off between computational costs and performance.

Random order vs. raster order (Q6)

We note that random order does not necessarily improve fidelity when comparing the performance of raster-order and random-order AR models with classifier-free guidance. While the random-order AR models achieve a better FID, they have a worse IS. IS primarily measures fidelity, whereas FID measures both fidelity and diversity. The improvement in FID is likely due to the global randomness introduced by the random order, which enhances the diversity of the generated images.

We thank the reviewer for appreciating our novel approach, significant contribution, and comprehensive empirical results. Here, we address the weaknesses (W) and questions (Q) raised by the reviewer.

Computational complexity (W1)

The table above compares the training times of different methods on our cluster of 128 V100 GPUs. While the diffusion loss introduces some training overhead compared to cross-entropy, this overhead is relatively marginal to the computational cost of the backbone autoregressive transformer, resulting in a total training cost increase of only 14%. Additionally, the training cost of our model remains significantly lower than that of representative diffusion-based models such as DiT-XL/2 and LDM-4.

Performance vs. #Parameters (W2)

In Table 4 of the paper, we evaluated MAR's performance with different numbers of parameters and demonstrated promising scaling signals up to 1 billion parameters. Due to resource constraints, we are unable to conduct an extensive systematic evaluation of the scaling behavior and will leave this for future work. MAR-L+DiffLoss achieves an FID of 1.78 with 479M parameters, while a representative diffusion model, DiT-XL/2, achieves an FID of 2.27 with 675M parameters. This demonstrates that MAR+DiffLoss can be more parameter efficient than common diffusion models. This is likely due to the performance improvement brought by DiffLoss, as shown in Table 1 of the paper.

Continuous tokens vs. discrete tokens (W3, Q3)

We provide visualizations of the reconstruction performance using VQ-based and KL-based tokenizers in the attached PDF in the general response. The results clearly demonstrate that the reconstruction quality of VQ-based tokenizers is significantly worse than that of KL-based tokenizers. The KL-16 tokenizer can reconstruct far more details from the original image. For example, the face of the Mona Lisa reconstructed by the KL-16 tokenizer is much better than that reconstructed by the VQ-16 tokenizer.

This is because VQ-based tokenizers experience much more information loss due to the higher compression ratio of the quantized tokens: for example, a VQ-16 tokenizer with a codebook size of 1024 tokenizes a 256x256 image into a 16x16 sequence of discrete indices, each of which can be represented by 10 bits. In contrast, a KL-16 tokenizer encodes the same image into a 16x16x16 sequence of float numbers. Thus, the compression ratio of the VQ-16 tokenizer is much higher than that of the KL-16 tokenizer, resulting in significantly more information loss. Therefore, the lower bound of information loss by modeling KL-16 continuous tokens using Diffusion Loss is lower compared to modeling VQ-16 discrete token indices using cross-entropy loss. We believe this may explain why Diffusion Loss on continuous-valued tokens can significantly outperform cross-entropy on discrete tokens.

On the other hand, one possible advantage of discrete-valued tokens is their ability to explicitly compute the probability of each sampled discrete token, which is important for certain sampling methods such as beam search. Although prior works have shown that diffusion models can also compute the explicit probability density on the continuous distribution using a probability-flow ODE [1], such computation is relatively less straightforward.

The rationale behind using a small MLP conditioning on latent code (Q1)

We use a small MLP conditioned on the latent code $z$ to model the probability distribution $p(x|z)$ through a diffusion process. Since the dimension of $x$ is small (16 for KL-16), the MLP can be small yet still accurately model $p(x|z)$. Such conditioning is also key to training the autoregressive transformer, as the diffusion loss is backpropagated to the large autoregressive transformer through $z$. Without conditioning on $z$, the MLP cannot effectively model $p(x|z)$, nor can the autoregressive transformer be properly trained.

Directly modeling the pixel space (Q2)

This is an excellent question, as also pointed out by reviewer mV98. Diffusion Loss does not depend on the tokenizer and can thus be applied directly to pixel space. Given the limited time for the rebuttal, we conducted a preliminary experiment on ImageNet 64x64, grouping every 4x4 pixels into one token for Diffusion Loss to model. A MAR-L+DiffLoss model trained for 400 epochs achieved an FID of 2.93, demonstrating the potential to eliminate the need for tokenizers in autoregressive image generation. However, as commonly observed in the literature on diffusion models, directly modeling the pixel space is much more computationally expensive than using a tokenizer [2]. For MAR+DiffLoss, directly modeling pixels at a higher resolution might require either a much longer sequence length for the autoregressive transformer or a much larger network for Diffusion Loss to handle larger patches.

Artifact (Q4)

We believe this issue is not primarily due to the KL-based continuous image tokenizer or the diffusion process. Instead, we think it is mostly due to the dataset size. The artifact problem is commonly observed when models are trained only on ImageNet (DiT also exhibits artifacts). Similar tokenizers and diffusion models are widely used in commercial models like Stable Diffusion, which experience far fewer artifacts, likely because they are trained on the much larger LAION-2B dataset. A concrete example is that when trained on LAION-400M, LDM [2] achieves significantly better performance compared to training on ImageNet, even though the model architecture and tokenizer remain almost the same.

[1] Score-Based Generative Modeling through Stochastic Differential Equations

[2] High-Resolution Image Synthesis with Latent Diffusion Models