Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens

Thank you for your insightful comments. Besides the general response, we have also incorporated your suggestions into the revised manuscript and provided detailed explanations for each question below. We hope that our revisions and rebuttal could address your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results.

Novelty

Please see the general response.

256x256 resolution

Thank you for the observation. To address this, we trained a 3B Fluid model capable of generating 512x512 resolution images by fine-tuning the 256x256 model. Specifically, we interpolated the position embeddings and fine-tuned the model with 512x512 images for an additional 500k iterations using a constant learning rate of 1e-5. A detailed comparison of the results is provided in Section C.3 and Figure A10 and A11 in the updated supplementary material.

Our findings show that the 512x512 images capture finer details and exhibit an overall improvement in visual quality. However, this comes with a trade-off between resolution, sequence length, and computational cost. Higher resolution requires increased sequence length, leading to greater computational demands. By balancing these factors, we demonstrate the feasibility of scaling to higher resolutions while highlighting the associated trade-offs.

Different resolution for eval metric

Thank you for raising this concern. Regarding evaluation metrics, differences in resolution should not affect the fairness of the comparison. We adhere to the community standard for computing zero-shot FID on MSCOCO. Specifically, all generated and real images are resized to 299x299 before feature extraction using the Inception-V3 network and subsequent distance calculation. This standardization ensures that resolution discrepancies do not bias the evaluation. We will clarify this in the revised version

LlamaGen as baseline

Thank you for pointing out LlamaGen[1] as a potential baseline. LlamaGen (0.7B) only scores 0.32 on the GenEval benchmark, while our smallest Fluid 0.3B model achieves 0.62, and the largest 10.5B model achieves 0.69.

We also used the same set of prompts provided by LlamaGen and conducted a visual comparison. The detailed results are included in Section C.4 in the updated supplementary material, Figure A13 and A14. Overall, our findings indicate that Fluid generates images of higher quality, with more intricate details and better text alignment, attributable to the advantages of continuous tokens.

Qualitative comparison

Thank you for your feedback. We respectfully note that the primary objective of our work is not to build the best-performing text-to-image model, but rather to conduct the first systematic study on the scaling behavior of autoregressive vision models.

We are the first to scale an autoregressive text-to-image model to the 10.5B parameter level, demonstrating consistent performance improvements as model size increases. Additionally, we identified key design principles—such as using continuous tokens and random-order generation—that enable autoregressive models to scale more effectively.

While qualitative and efficiency comparisons with state-of-the-art models are valuable, we believe our findings make a significant contribution by advancing the understanding of scaling in autoregressive vision models and providing actionable insights to guide future research in this area.

[1] Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation, Sun et al., 2024

Thank you for your insightful comments. Besides the general response, we have also incorporated your suggestions into the revised manuscript and provided detailed explanations for each question below. We hope that our revisions and rebuttal could address your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results.

Novelty

Please see the general response for a detailed response. We believe novelty not only comes from the technical side, e.g., proposing a new model, but also comes from interesting findings that can inspire the future direction. The latter can sometimes be more important.

Comparison Fairness

Thank you for raising this concern. We took careful steps to ensure that our comparisons are both plausible and fair, despite the inherent complexity of autoregressive image generation. Specifically:

1. Token Representations:

   • For discrete tokens, we used state-of-the-art VQ tokenizers, as employed in MUSE (but we re-trained it with larger datasets and improved recipe).
   • For continuous tokens, we adopted the Stable Diffusion tokenizer, which we found through ablations to produce the best FID scores.

2. Inference Hyperparameters: We conducted extensive sweeps over inference hyperparameters (e.g., temperature and classifier-free guidance) for all configurations, reporting the best results for each model to ensure fairness.

3. Training Hyperparameters:

   • All training hyperparameters, such as dataset, learning rate, and number of iterations, were kept consistent across configurations.
   • The only exception was the learning rate schedule: we used a cosine schedule for MUSE as it led to better results compared to a constant schedule.

While we acknowledge that achieving absolute fairness in such comparisons is inherently challenging, these measures were taken to minimize confounding factors as much as possible. We believe this approach provides a robust foundation for evaluating the roles of token representation and generation order in scaling autoregressive models.

Q1 Discrete tokens are better in small models

Thank you for pointing this out. This observation aligns with our expectations. The categorical distribution used in discrete tokens is inherently simpler and easier to learn compared to the continuous distribution. While this simplicity allows smaller models, which have limited capacity, to perform better with discrete tokens, it comes at the cost of being lossy and less capable of modeling complex distributions.

In contrast, continuous tokens offer greater expressive power and flexibility, but their complexity makes it harder for smaller models to optimize effectively. This trade-off highlights why discrete tokens are better suited for smaller models, while continuous tokens exhibit their advantages as model capacity increases.

Ablation on diffusion head

Thank you for raising this point. The following Table S1 presents a pilot study on the effect of the diffusion head for continuous tokens. We trained the smaller model with 277M parameters using random order continuous tokens with a T5-XL text encoder for 500k steps. The FID score shows consistent improvements as we enlarge the depth/width of the diffusion head. We choose to use 6 layers with the same width as the backbone transformer in our Fluid models to balance performance and efficiency.

<Table S1. FID vs. different diffusion head sizes.>

Incorporating a diffusion head for discrete tokens is inherently challenging because the diffusion process is designed to model continuous distributions. As a result, applying this structure to discrete tokens would require significant modifications, and it would diverge from the current architecture's principles.

Q3 Inference efficiency comparison

We provide a detailed comparison of inference speeds across different configurations in Table S2. The key details of the setup are as follows:

Random-Order Generation:

• Images are generated in 64 steps.
• The inference speed is measured on a single H100 PCIe GPU, using the largest possible batch size (128).

Raster-Order Generation:

• Images are generated with KV-cache enabled.
• The inference speed is also measured on a single H100 PCIe GPU, with the largest batch size supported (64). The largest batch size is smaller than the random-order models because KV-cache takes more GPU memory.
• The continuous token + raster order setting is particularly slow because the DiffLoss module can only use a batch size of 64 at each autoregressive step, which largely wastes the capacity of the GPU.

<Table S2. Inference speed comparison.>

Dear Reviewer QGyS,

Thanks again for your insightful suggestions and comments. Since the deadline of discussion is approaching, we are happy to provide any additional clarification that you may need.

In our previous response, we have carefully studied your comments and made detailed responses summarized below:

• Provided further explanation of our empirical novelty and contributions to the community.
• Offered additional discussion on ensuring fairness in quantitative comparisons.
• Conducted new experiments to study the effect of diffusion head sizes.
• Compared inference efficiency across different configurations through additional experiments.

We hope that the provided new experiments and the additional explanation on our empirical novelty have convinced you of the contributions of our submission.

Please do not hesitate to contact us if there's additional clarification or experiments we can offer. Thanks!

Thank you for your time!

Best, Authors

Thank you for your insightful comments. Besides the general response, we have also incorporated your suggestions into the revised manuscript and provided detailed explanations for each question below. We hope that our revisions and rebuttal could address your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results.

Q1-1 Validation loss & metric

Correlation Between Validation Loss and Metrics:

Validation loss directly reflects the training target, while metrics like FID and GenEval capture different aspects of generation quality and alignment. In NLP, this gap is smaller because validation metrics (e.g., next-token prediction) are the same as the training objective. In contrast, for autoregressive vision models, the discrepancy is larger because:

• All autoregressive steps occur in the latent space, and validation loss is measured there.
• The final image is decoded from the latent space and then evaluated using metrics, introducing additional complexity.

This separation between latent-space training and pixel-space evaluation may drive the observed differences, and we highlight this as a key finding in our study.

A linear relationship between validation loss and metrics:

They are correlated because they all measure the generation capability (but from different perspectives). The relationship is highly linear under the regime we evaluated, which suggests the model is highly scalable (at least to the extent we explored). As pointed out in line 485, such a linear relationship will be broken at some point.

Q1-2 Human-aligned evaluation criterion

Identifying a universally human-aligned evaluation metric is a broader challenge not specific to this paper. While benchmarks like GenEval are promising, they are limited in scope. A potential direction for future work is creating more complex prompts and datasets to evaluate nuanced relationships and subject matter. Such efforts could pave the way for better automatic evaluation methods aligned with human perception.

Q1-3 Why random order is better with higher loss

Thank you for your thoughtful question. Random-order generation introduces additional complexity to the image generation task by using a masked version of the original image as input. This makes the model's task inherently harder, as it must predict tokens in arbitrary orders without relying on positional continuity. While this results in a larger training loss, it encourages the model to learn more robust representations for the image-text alignment.

During scaling, this robustness translates into better generalization and higher visual quality, as the model can generate coherent and detailed images even in the presence of challenging input scenarios. Additionally, random-order generation reduces the positional bias that sequential generation might introduce, leading to improved flexibility and accuracy in token prediction.

The confusion may come from directly comparing the loss of random order with the loss of raster order. We emphasize that such validation loss is not directly comparable across different training setups. A simple demonstration is that we can train the 1.1B model with continuous tokens and random order for 500k iterations, using different mask ratios. As the masking ratio increases, the loss gets consistently higher but the FID first increases then decreases.

<Table S1. Validation loss and FID with different masking ratios.>

Q2 Ablation on number of training samples

Thank you for highlighting this important aspect of scaling. To address this, we trained a 1.1B Fluid model with different amounts of image-text pairs for 500k iterations, and summarized the results in the table below.

<Table S2. Scaling behavior of Fluid 1.1B model on different data sizes>

Our findings show that FID improves consistently as the dataset size increases, up to approximately 5-10% of the total training samples. Beyond this point, the improvements on FID plateau, likely because larger datasets require more training iterations and larger models to showcase its benefits.

Q3 Emergence phenomena

Thank you for raising this interesting point. We have added additional qualitative comparisons between the images generated by the 3.1B model and the 10.5B model in Section C.2 in the updated supplementary material, Figure A8 and A9. From these comparisons, we observe an interesting phenomenon in visual quality: the 10.5B model demonstrates the ability to accurately capture and represent numbers and characters, such as correctly generating a “stop” sign with legible text. In contrast, the 3.1B model lacks this capability, failing to generate plausible or coherent characters. However, this ability is not reflected in the FID and GenEval evaluation metrics, as they do not explicitly measure it. Nevertheless, this suggests that as the model scales, it develops a deeper understanding of complex patterns, such as textual and numerical information, which were not evident in smaller models. These emergent abilities highlight the advantages of scaling autoregressive models for text-to-image generation.

Dear Reviewer y1Ue,

Thanks again for your insightful suggestions and comments. Since the deadline of discussion is approaching, we are happy to provide any additional clarification that you may need.

In the original review, you expressed the willingness to turn to the positive side if additional insights on our results and experiments examining scaling behavior with respect to data sizes were provided. We hope that the new experiments and the additional explanation on validation loss and our empirical contribution have convinced you of the contributions of our submission.

A key takeaway from our work is continuous tokens with random order scale significantly better than other configurations and work best for large autoregressive text-to-image models. Based on this, we recommend adopting continuous tokens with random order for large models, and we believe these findings offer valuable empirical guidance for future research in autoregressive vision-language and video generative models.

Please do not hesitate to contact us if there's additional clarification or experiments we can offer. Thanks!

Thank you for your time!

Best, Authors

Thank you for your positive feedback. We are glad to hear that we have addressed your concerns. In addition to the positive aspects, we have provided failure cases and analysis of different design choices (please see Section D in the supplementary material). In Figure 5, we demonstrate that the validation loss follows a clear and predictable scaling law w.r.t. the number of parameters, which can potentially serve as a practical guidance for training large-scale generative models. However, we also notice that the transfer from validation loss to evaluation metrics remains unclear. The exact scaling law on evaluation metrics is a very interesting and important topic, and we leave it for future work to explore.

As you pointed out, for large-scale models, a less constrained prior could potentially yield better results. We believe this is likely the reason why both continuous tokens and random order perform better, as they represent "harder" tasks and are less constrained.

We have also included more "text-on-image" generation results, which are provided in Figure A12 in the updated supplementary material.

We truly appreciate that you agree to raise your score. Please let us know if anything remains unclear or if you would like to see additional results or explorations.

Thank you for your insightful comments. Besides the general response, we have also incorporated your suggestions into the revised manuscript and provided detailed explanations for each question below. We hope that our revisions and rebuttal could address your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results

Q2. Explanation of why continuous is better

We agree that more theoretical reasoning would strengthen our findings. Here, we provide additional quantitative and qualitative analysis to explain why continuous tokens and random-order generation outperform discrete tokens and raster-order generation:

Why continuous tokens perform better than discrete tokens

• Compression Ratio Analysis: Continuous tokens preserve significantly more information than discrete tokens due to differences in compression ratios:
  • Continuous tokenizers map a 256x256 image to a 32×32×4 latent space (float32), yielding a compression ratio of approximately 2 bits per pixel.
  • Discrete tokenizers map the same image to a 16×16 latent space with a codebook of size 8192 ($\log⁡_2(8192)=13$ bits per token), resulting in a much lower compression ratio of approximately 0.05 bits per pixel.
  • This compression ratio for discrete tokens is even significantly lower than that of standard JPEG (typically 0.25–2 bits per pixel), making discrete tokens overly lossy and unable to retain sufficient image details.
• Reconstruction Quality: As shown in Figure 4 of the main paper, reconstruction results using discrete tokens fail to preserve fine details, such as the identity and texture in the Mona Lisa example, leading to poorer visual fidelity and lower PSNR values. Continuous tokens, by contrast, provide much better reconstructions, retaining finer details and textures, which is beneficial for higher-quality image generation.

Gradient flow and expressive capacity

• Continuous tokens benefit from smoother gradient flow due to their continuous-valued representation, enabling more precise optimization during training. Discrete tokens, on the other hand, rely on categorical representations, which can introduce challenges in optimization, such as difficulty in capturing nuanced details and gradients being limited by discrete choices.

We believe this analysis complements the empirical findings presented in the paper and adds depth to our understanding of why continuous tokens and random-order generation yield better performance. Thank you for encouraging us to provide this additional reasoning.

Impact of dataset size scaling

Thank you for highlighting this important aspect of scaling. To address this, we trained a 1.1B Fluid model with different amounts of image-text pairs for 500k iterations, and summarized the results in the table below.

<Table S1. Scaling behavior of Fluid 1.1B model on different data sizes>

Our findings show that FID improves consistently as the dataset size increases, up to approximately 5-10% of the total training samples. Beyond this point, the improvements on FID plateau, likely because larger datasets require more training iterations and larger models to showcase its benefits.

Q3: Training time comparison

We perform a comparison of the training FLOPs using different configurations. Models with continuous tokens require slightly more training FLOPs because of the DiffLoss MLP training:

<Table S2. Training efficiency comparison on the 3.1B model across different setups.>

Q4: Inference step effect

For raster-order generation, inference steps are fixed since tokens must be generated one at a time with causal attention, resulting in 256 inference steps. For random-order generation, we conducted an analysis of FID versus inference steps for continuous tokens on the 3.1B model, as shown in the following table.

<Table S3. FID vs inference steps on 3.1B model with continuous tokens>

The results indicate that FID consistently decreases with an increasing number of inference steps, up to 64 steps. Beyond 64 steps, further increases do not improve FID.

Q5: Finetune for higher resolution

Fluid models are indeed capable of generating 512x512 resolution images by fine-tuning the 256x256 checkpoints through position embedding interpolation. For 256x256 images, the token length is 256, whereas for 512x512 images, the token length increases to 1024. Due to the limited time during the discussion period, we focused on training a 3.1B Fluid model with continuous tokens and random-order generation at 512x512 resolution. We replace the trained position embedding with 2D rotary position embedding to support variable sequence length. Starting from the 256x256 checkpoint, we interpolated the positional embeddings to support the increased token length and fine-tuned the model for an additional 500k steps on 512x512 resolution images. As detailed in Section C.3 in the updated supplementary material, Figure A10 and A11, the 512x512 images demonstrate significant improvements in detail and sharpness compared to 256x256 images. This shows that Fluid models scale effectively with resolution, making them suitable for applications requiring higher-resolution image generation.

Q6: Compare scaling with pure diffusion

The scaling of diffusion-based models, and later flow-based models, has been studied in several works in the literature. Most notably, in SD3 [1] (one of the current state-of-the-art flow-based models), they scaled the model from 0.8B to 8B parameters and observed consistent scaling behavior. In Figure A5 in the updated supplementary material and Table S4 below, we compare the scaling behavior of SD3 and Fluid in terms of the GenEval score. Surprisingly, despite various differences in model architecture, training data, and implementations, the scaling rates of these two models are quite similar within the same GenEval score range (0.6–0.7). Moreover, Fluid performs even better than SD3 at the same number of parameters. However, we also observe that the GenEval scores plateau for the 10.5B Fluid model compared to the 3.1B Fluid model. We hypothesize that this is due to the inconsistency between the scaling of the validation loss and the validation metrics. The scaling behavior of SD3 beyond 10B parameters would also be interesting to explore, and we leave it for future work.

<Table S4. Scaling comparison between Fluid and SD3.>

Q7: Failure cases

We have included examples of failure cases for both continuous tokens with random-order and raster-order generation Section D, Figure A15 and A16 in the updated supplementary material.

• Random-Order Generation:
  Occasionally, a bright spot appears in the generated image, rendering other parts unrecognizable. We hypothesize that this issue stems from insufficient diffusion steps in the diffusion head. Increasing the number of diffusion steps could potentially alleviate this problem.
• Raster-Order Generation:
  In some cases, the generated images collapse into a gray pattern in the lower part, particularly at transitions between lines. We believe this occurs because the learned position embeddings struggle to capture the abrupt changes between lines. Incorporating 2D Rotary position embeddings may address this issue by better modeling spatial relationships.

These observations provide valuable insights for future improvements and highlight areas for further exploration.

Q8: Reproducibility

Thank you for highlighting the importance of code availability. We strictly adhere to the code implementation of MAR [2] (for continuous tokens) and MaskGIT [3] (for discrete tokens), and we have provided all additional hyperparameters and training details in the paper to ensure reproducibility. If there are specific details that you feel are missing or unclear, please let us know, and we would be happy to provide further clarifications or supplementary materials to support reproduction.

[1] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, Esser et al., 2024

[2] Autoregressive Image Generation without Vector Quantization, Li et al., 2024.

[3] MaskGIT: Masked Generative Image Transformer, Chang et al., 2022

Dear Reviewer UT5L,

Thanks again for your insightful suggestions and comments. Since the deadline of discussion is approaching, we are happy to provide any additional clarification that you may need.

In our previous response, we carefully addressed your feedback and made the following updates and additions: Provided additional explanation of our empirical novelty and contributions to the community.

• Offered detailed reasoning on why continuous tokens outperform discrete tokens.
• Included a comparison of the scaling behavior of our Fluid model with diffusion models.
• Conducted additional experiments to examine the effect of inference steps.
• Conducted additional experiments to evaluate the performance of higher-resolution image generation (512x512).
• Compared training efficiency across different configurations.
• Discussed failure cases in detail.

We hope these new experiments and explanations have clarified the contributions and strengthened our submission.

Please do not hesitate to reach out if there are any further clarifications or analyses we can provide.

Thank you for your time and thoughtful feedback!

Best, Authors