Reference

[1]Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis[J]. Advances in neural information processing systems, 2021, 34: 8780-8794.

[2] Ho J, Saharia C, Chan W, et al. Cascaded diffusion models for high fidelity image generation[J]. Journal of Machine Learning Research, 2022, 23(47): 1-33.

[3] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.

[4] Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4195-4205.

[5] Gao S, Zhou P, Cheng M M, et al. Masked diffusion transformer is a strong image synthesizer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 23164-23173.

[6] Liu Q, Zeng Z, He J, et al. Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models[J]. arXiv preprint arXiv:2406.09416, 2024.

[7] Gu S, Chen D, Bao J, et al. Vector quantized diffusion model for text-to-image synthesis[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10696-10706.

[8] Tang Z, Gu S, Bao J, et al. Improved vector quantized diffusion models[J]. arXiv preprint arXiv:2205.16007, 2022.

[9] Chang H, Zhang H, Jiang L, et al. Maskgit: Masked generative image transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 11315-11325.

[10] Li T, Katabi D, He K. Self-conditioned image generation via generating representations[J]. arXiv preprint arXiv:2312.03701, 2023.

[11] Yu L, Lezama J, Gundavarapu N B, et al. Language Model Beats Diffusion--Tokenizer is Key to Visual Generation[J]. arXiv preprint arXiv:2310.05737, 2023.

[12] Yu Q, Weber M, Deng X, et al. An Image is Worth 32 Tokens for Reconstruction and Generation[J]. arXiv preprint arXiv:2406.07550, 2024.

[13] Weber M, Yu L, Yu Q, et al. Maskbit: Embedding-free image generation via bit tokens[J]. arXiv preprint arXiv:2409.16211, 2024.

[14] Razavi A, Van den Oord A, Vinyals O. Generating diverse high-fidelity images with vq-vae-2[J]. Advances in neural information processing systems, 2019, 32.

[15] Esser P, Rombach R, Ommer B. Taming transformers for high-resolution image synthesis[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 12873-12883.

[16] Lee D, Kim C, Kim S, et al. Autoregressive image generation using residual quantization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 11523-11532.

[17] Yu J, Li X, Koh J Y, et al. Vector-quantized image modeling with improved vqgan[J]. arXiv preprint arXiv:2110.04627, 2021.

[18] Tian K, Jiang Y, Yuan Z, et al. Visual autoregressive modeling: Scalable image generation via next-scale prediction[J]. arXiv preprint arXiv:2404.02905, 2024.

[19] Sun P, Jiang Y, Chen S, et al. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation[J]. arXiv preprint arXiv:2406.06525, 2024.

[20] Chung Y A, Tang H, Glass J. Vector-quantized autoregressive predictive coding[J]. arXiv preprint arXiv:2005.08392, 2020.

Weakness 2

Thank you very much for your suggestion. To address your concerns regarding the model, we conducted the same experiments on other models, as detailed in Appendix C. It can be observed that due to the influence of the continuous representation space, MAR, despite exhibiting excellent scaling laws, do not demonstrate superior bit-level scaling laws,similar to DiT. In contrast, LLaMaGen, which shares the discrete representation space with VAR, exhibits outstanding bit-level scaling laws.

Additionally, we provide an explanation of the impact of model size on bit-level scaling laws based on their underlying principles. Firstly, Our research focuses on analyzing the differences in scaling trends between models that already exhibit superior scaling laws, rather than being influenced by specific model sizes. To ensure clarity, we have aligned the initial total bits in Figure 1 of paper, providing you with a clearer understanding. To assess whether a model exhibits strong bit-level scaling laws, one must compare the internal change trends of the model (e.g., comparing 8-bit VAR vs. 16-bit VAR). As shown in Figure 1 in paper, we can observe that, regardless of the quantization method, when the full-precision VAR is quantized to lower bit precision, the overall scaling law of the model shifts towards the lower-left corner. However, DiT does not exhibit this behavior. This outstanding characteristic displayed by the discrete model enables us to increase the model parameters through quantization under limited resources, leading to better generative performance, which is not possible for continuous models. This is precisely what bit-level scaling laws aim to demonstrate.

More importantly, our work shows that quantization is no longer just about reducing model size. By optimizing both model design and quantization techniques to achieve superior bit-level scaling laws, we can obtain better generative performance under the same resource constraints. This outstanding feature is something that researchers should pay more attention to.

Weakness 3

We apologize for any confusion caused by the phrasing in the paper. We hope that the explanation of Figure 1 in the main text helps clarify the concept of bit-level scaling laws and this interesting phenomenon.. As shown in the figure 1, quantized VAR, after being quantized to lower bit precision, demonstrates a shift in its scaling law curve towards the lower-left region, which exhibits its superior bit-level scaling laws. By leveraging this outstanding feature, we can increase the model parameters under limited resources (e.g., in specific deployment scenarios such as mobile devices or edge computing) while maintaining efficiency, ultimately improving generative capabilities. In contrast, for continuous diffusion-style models, regardless of the quantization method used, the quantized model shows "almost" no improvement compared to full precision. Bit-level scaling laws serve as a strong predictor of model performance.

This paper indicates that achieving optimal bit-level scaling behavior requires a synergistic interaction between model design and quantization algorithms. Our study is an essential step towards understanding how various models and quantization methods influence bit-level scaling behavior, and it also provides valuable recommendations for future work.

Weakness 1

We greatly appreciate your valuable review comments. We apologize for any confusion caused by the phrasing in the paper and hope our response can clarify your concerns regarding the statement: "The paper is more like an experimental report than a research paper." Furthermore, TopKLD is just one part of our research. The goal of this paper is not merely to propose an improvement to existing methods, but rather to conduct an in-depth study of the bit-level scaling laws in vision generative models, addressing the "What," "Why," and "How" from the perspective of extensive experimental design.

Exploration of bit-level scaling laws must be based on the internal patterns derived from a large number of experiments. These patterns can guide future research, which is why we conducted numerous experiments, as it is essential for uncovering these insights.

The analysis of VAR and DiT represents research into two mainstream development directions in the vision generative model field. As shown in Table 1 above, there has been ongoing debate regarding the use of discrete versus continuous representation spaces (e.g., [17,18,19,20]). Both approaches have shown strong performance in terms of scaling laws. This work, however, takes a different perspective by investigating the impact of these representation spaces on the scaling laws in quantized models. We find that, despite achieving comparable performance at full precision, discrete autoregressive models consistently outperform continuous models across various quantization settings. To validate the effectiveness and broad applicability of our conclusions for you, we conducted the same experiments on other models, as detailed in Appendix C. This indicates that our work provides general guidance for subsequent model design and applications in specific deployment scenarios (e.g., mobile devices, edge computing).

Secondly, while low-bit precision representation often focuses on trading performance for efficiency, this work demonstrates that by optimizing either the model or quantization algorithm, models can achieve superior bit-level scaling laws. This outstanding characteristic enables the use of lower bit precision to increase model parameters, ultimately enhancing generative capability without sacrificing efficiency — is a key feature that we hope researchers will pay particular attention to.

As such, you can see the tremendous potential of low-bit precision in the context of bit-level scaling laws. However, existing methods fail to further improve the bit-level scaling laws of models. To address this, we introduced TopKLD, which enhances the bit-level scaling behaviors of language-style models by one level.

Our study is an essential step toward understanding how various models and quantization methods influence bit-level scaling behavior, and it also provides the following recommendations for future work: We hope the reviewer will take into account the contributions of this work to model design and the application of quantization algorithms. Thank you again!

Rebuttal Revision Paper Modifications

We greatly appreciate your valuable review comments. We have revised the paper according to your suggestions and submitted the rebuttal version. For detailed modifications, please refer to the rebuttal version PDF and appendix C: Supplementary materials for rebuttal. Below, we address your identified weaknesses and questions, hoping to resolve your concerns and improve our score.

Table 1

Dear reviewer:

Thanks you for your great efforts in reviewing out paper and providing constructive suggestions/comments. To address the weaknesses you raised, we have conducted extensive experiments in appendix C and figure 1 of main paper to alleviate concerns regarding the size of the VAR model. Additionally, this work focuses on investigating the impact of whether the representation space in vision generation models is continuous or discrete. Furthermore, we propose strategies to optimize bit-level scaling laws under various quantization scenarios. Our exploration of model design and quantization methods provides significant insights for guiding future applications in specific deployment scenarios, such as mobile devices and edge computing. If our rebuttal does not address your concerns, you are warmly wecomed to raise questions. If our responses have addressed your concerns, we sincerely request that you consider raising our score.

Best Wishes!

Authors

Dear reviewer:

Thanks you for your great efforts in reviewing out paper and providing constructive suggestions/comments. To address the weaknesses you raised, we have conducted extensive experiments in appendix C and figure 1 of main paper to alleviate concerns. If our rebuttal does not address your concerns, you are warmly wecomed to raise questions. If our responses have addressed your concerns, we sincerely request that you consider raising our score.

Best Wishes!

Authors

Dear reviewer:

Thanks you for your great efforts in reviewing out paper and providing constructive suggestions/comments. To address the weaknesses you raised, we have conducted extensive experiments in appendix C and figure 1 of main paper to alleviate concerns. If our rebuttal does not address your concerns, you are warmly wecomed to raise questions. If our responses have addressed your concerns, we sincerely request that you consider raising our score.

Best Wishes!

Authors

Weakness1

We greatly appreciate your valuable review comments and hope that our response addresses your concerns regarding the statement: "Bit-level scaling laws and the robustness of discrete auto-regressive models seem to be intuitive and straightforward."

Firstly, as shown in table 1 above, in the field of vision generative models, there has been ongoing debate regarding the use of discrete versus continuous representation spaces (e.g., [17,18,19,20]). Both approaches have shown strong performance in terms of scaling laws. This work, however, takes a different perspective by investigating the impact of these representation spaces on the scaling laws in quantized models. We find that, despite achieving comparable performance at full precision, discrete autoregressive models consistently outperform continuous models across various quantization settings. To validate the effectiveness and broad applicability of our conclusions for you, we conducted the same experiments on other models, as detailed in Appendix C. This indicates that our work provides general guidance for subsequent model design and applications in specific deployment scenarios (e.g., mobile devices, edge computing).

Secondly, while low-bit precision representation often focuses on trading performance for efficiency, this work demonstrates that by optimizing either the model or quantization algorithm, models can achieve superior bit-level scaling laws. This outstanding characteristic enables the use of lower bit precision to increase model parameters, ultimately enhancing generative capability without sacrificing efficiency.

To validate the effectiveness and broad applicability of our conclusions for you, we conducted the same experiments on other models, as detailed in Appendix C. It can be observed that due to the influence of the continuous representation space, MAR, despite exhibiting excellent scaling laws,similar to DiT, do not demonstrate superior bit-level scaling laws. In contrast, LLaMaGen, which shares the discrete representation space with VAR, exhibits outstanding bit-level scaling laws.

This work provides a deeper, foundational understanding of bit-level scaling laws in visual generative models, from both the model design and quantization algorithm perspectives, supported by rigorous experimental design.

Our study is an essential step toward understanding how various models and quantization methods influence bit-level scaling behavior, and it also provides the following recommendations for future work:

We hope the reviewer will take into account the contributions of this work to model design and the application of quantization algorithms. Thank you again!

Reference

[1]Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis[J]. Advances in neural information processing systems, 2021, 34: 8780-8794.

[2] Ho J, Saharia C, Chan W, et al. Cascaded diffusion models for high fidelity image generation[J]. Journal of Machine Learning Research, 2022, 23(47): 1-33.

[3] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.

[4] Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4195-4205.

[5] Gao S, Zhou P, Cheng M M, et al. Masked diffusion transformer is a strong image synthesizer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 23164-23173.

[6] Liu Q, Zeng Z, He J, et al. Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models[J]. arXiv preprint arXiv:2406.09416, 2024.

[7] Gu S, Chen D, Bao J, et al. Vector quantized diffusion model for text-to-image synthesis[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10696-10706.

[8] Tang Z, Gu S, Bao J, et al. Improved vector quantized diffusion models[J]. arXiv preprint arXiv:2205.16007, 2022.

[9] Chang H, Zhang H, Jiang L, et al. Maskgit: Masked generative image transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 11315-11325.

[10] Li T, Katabi D, He K. Self-conditioned image generation via generating representations[J]. arXiv preprint arXiv:2312.03701, 2023.

[11] Yu L, Lezama J, Gundavarapu N B, et al. Language Model Beats Diffusion--Tokenizer is Key to Visual Generation[J]. arXiv preprint arXiv:2310.05737, 2023.

[12] Yu Q, Weber M, Deng X, et al. An Image is Worth 32 Tokens for Reconstruction and Generation[J]. arXiv preprint arXiv:2406.07550, 2024.

[13] Weber M, Yu L, Yu Q, et al. Maskbit: Embedding-free image generation via bit tokens[J]. arXiv preprint arXiv:2409.16211, 2024.

[14] Razavi A, Van den Oord A, Vinyals O. Generating diverse high-fidelity images with vq-vae-2[J]. Advances in neural information processing systems, 2019, 32.

[15] Esser P, Rombach R, Ommer B. Taming transformers for high-resolution image synthesis[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 12873-12883.

[16] Lee D, Kim C, Kim S, et al. Autoregressive image generation using residual quantization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 11523-11532.

[17] Yu J, Li X, Koh J Y, et al. Vector-quantized image modeling with improved vqgan[J]. arXiv preprint arXiv:2110.04627, 2021.

[18] Tian K, Jiang Y, Yuan Z, et al. Visual autoregressive modeling: Scalable image generation via next-scale prediction[J]. arXiv preprint arXiv:2404.02905, 2024.

[19] Sun P, Jiang Y, Chen S, et al. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation[J]. arXiv preprint arXiv:2406.06525, 2024.

[20] Li, Tianhong, et al. "Autoregressive Image Generation without Vector Quantization." arXiv preprint arXiv:2406.11838 (2024).

Rebuttal Revision Paper Modifications

We greatly appreciate your valuable review comments. We have revised the paper according to your suggestions and submitted the rebuttal version. For detailed modifications, please refer to the rebuttal version PDF and appendix C: Supplementary materials for rebuttal. Below, we address your identified weaknesses and questions, hoping to resolve your concerns and improve our score.

Table 1

Question 1

Thank you very much for your suggestion. In this paper, we have provided additional experiments to demonstrate the effectiveness of TopKLD, as shown in the table below.

Question 2

Thank you very much for your suggestion. To investigate the impact of top-k sampling on the bit-level scaling behavior of the model, we performed ablation experiments using different values of K. The results in the table below show that:

1.While the choice of K does affect the final generation results to some extent, it does not influence the overall trend of the bit-level scaling laws.

2.The best performance is achieved when the value of K matches the K used in the Top-K sampling during the model's image generation process.

Question 3

Thank you very much for your correction. We will revise the error accordingly.

Dear reviewer:

Thanks you for your great efforts in reviewing out paper and providing constructive suggestions/comments. This work focuses on investigating the impact of whether the representation space in vision generation models is continuous or discrete. Furthermore, we propose strategies to optimize bit-level scaling laws under various quantization scenarios. To address the weaknesses you raised, we have provided extensive examples related to the recent debate on continuous versus discrete representation spaces in vision generation models, as shown in Tab.1. Through numerous experiments presented in Appendix C and Figure 1 in the main text, we demonstrate the significant impact and unique differences of this feature on vision generation models. These findings provide valuable insights into model design and quantization methods, offering guidance for future applications in specific deployment scenarios, such as mobile devices and edge computing. Additionally, to address your questions 1 and 2 and demonstrate the advantages of TopKLD, we have conducted extensive comparisons with current state-of-the-art methods across various settings, as per your suggestion. If our rebuttal does not address your concerns, you are warmly wecomed to raise questions. If our responses have addressed your concerns, we sincerely request that you consider raising our score.

Best Wishes!

Authors

Thanks for the author's supplementary experiments. The current results are adequate to illustrate the advantage of TopKLD.

While "Knowledge Distillation in Quantization-Aware Training" (KD-QAT) can enhance VAR's scaling ability at low bits, I am wondering whether KD-QAT also works for DiT.

Thank you very much for your response!!!

The following experiments demonstrate that KD-QAT is also effective for DiT. Additionally, we have validated its effectiveness for LlamaGen in Appendix C.

However, it is important to note that, compared to discrete representation space models like LlamaGen and VAR, the bit-level scaling laws of DiT are inherently limited by its continuous representation space. If our rebuttal does not address your concerns, you are warmly wecomed to raise questions. If our responses have addressed your concerns, we sincerely request that you consider raising our score.

Dear reviewer:

Thanks you for providing constructive suggestions. We would like to kindly ask if our responses and additional experiments have addressed all your concerns. If so, we would greatly appreciate it if you could reconsider the score in light of the clarifications and new evidence provided.

Best Wishes!

Authors

Dear reviewer:

Thanks you for your great efforts in reviewing out paper and providing constructive suggestions/comments. To address the weaknesses you raised, we have conducted extensive experiments in appendix C and figure 1 of main paper to alleviate concerns. If our rebuttal does not address your concerns, you are warmly wecomed to raise questions. If our responses have addressed your concerns, we sincerely request that you consider raising our score.

Best Wishes!

Authors

Weakness 3

Thank you very much for your suggestion. We have provided a comparison of TopKLD with the current mainstream quantization methods. Please refer to the results in Weakness2 for further details.

Weakness 4

Thank you very much for your suggestion. TopKLD is an optimization of current mainstream distillation loss functions. It balances the "implicit knowledge" and "explicit knowledge" derived from full-precision models, thereby enhancing the bit-level scaling behaviors of language-style models by one level. As a result, it does not incur any additional resource overhead compared to methods like ForwardKLD. For your reference, we have provided the training times for TopKLD on A100 GPU below:

Rebuttal Revision Paper Modifications

We greatly appreciate your valuable review comments. We have revised the paper according to your suggestions and submitted the rebuttal version. For detailed modifications, please refer to the rebuttal version PDF and appendix C: Supplementary materials for rebuttal. Below, we address your identified weaknesses and questions, hoping to resolve your concerns and improve our score.

Weakness 1

Thank you very much for your valuable reminder. We align the initial bit settings to better compare the bit-level scaling laws of language-style models and diffusion-style models. We have modified Figure 1 in the main paper, as shown in the rebuttal version of the PDF.

To determine whether a model exhibits superior bit-level scaling laws, we compare the internal trends of the model (e.g., 8-bit VAR vs. 16-bit VAR), rather than making a direct comparison of generative quality between two types of models at the same total bit precision (e.g., 8-bit DiT vs. 8-bit VAR). As shown in Figure 1, regardless of the quantization method, when full-precision VAR is quantized to lower bit precision, its scaling law shifts towards the lower-left region. In contrast, DiT does not show such a shift. This is precisely what we mentioned in the caption: "Quantized VAR exhibits better bit-level scaling laws than full-precision VAR, while Quantized DiT shows almost no improvement compared to full precision."

The characteristics demonstrated by the discrete language-style model allow us to increase the model's parameters through quantization under limited resources, thereby achieving better generative capability. This is a feature that continuous diffusion-style models lack, and it is precisely what bit-level scaling laws aim to showcase.

Weakness 2

Thank you for your suggestion. We provide a more comprehensive evaluation against other mainstream quantization methods for its practical effectiveness: GPTQ, GPTVQ, SmoothQuant, OmniQuant, and TopKLD. The experiments below demonstrate our superior performance across various bit precisions.

As shown, whether in high-bit or low-bit settings, and whether quantizing only weights or both weights and activations, TopKLD consistently exhibits superior performance. Even at higher bit precision, TopKLD still leads to noticeable improvements in model accuracy.

Regarding your comment on the “Limited Advantage of TopKLD in High-Bit Settings,” the reason for this is that our focus is not solely on improving model accuracy but also on scaling laws. At high precision levels, models retain sufficient precision, resulting in minimal degradation compared to full-precision models. Thus, there is no significant enhancement in bit-level scaling in these settings.

Through the explanation in Weakness1, we believe you can see that models with excellent bit-level scaling laws demonstrate enhanced capabilities at low-bit conditions. The goal of TopKLD is to improve the scaling ability of models under low-bit conditions. As shown in the results in Section 3.3 of the paper, TopKLD enhances the bit-level scaling behaviors of language-style models by one level.

Questinon 1

Thank you very much for your reminder. We have used similar initial bit allocations to strengthen the comparison. Please refer to the details in Weakness1.

Question 2

When a model exhibits excellent bit-level scaling laws, by leveraging this outstanding feature, we can increase the model parameters under limited resources (e.g., in specific deployment scenarios such as mobile devices or edge computing) while maintaining efficiency, ultimately improving generative capabilities. However, as shown in Figure 5 of the main text, existing methods fail to achieve better bit-level scaling laws under low-bit settings, which hinders further enhancement of model capabilities in specific deployment scenarios. If you wish to further improve model generation quality under limited resource conditions, TopKLD is an excellent choice. Although it incurs some additional computational cost, it results in a significant improvement in model performance.

Question 3

Thank you for your valuable suggestion. We have provided a comparison with existing mainstream quantization methods in Weakness2. As shown in Figure 5 of the main text, existing methods fail to achieve better bit-level scaling laws under low-bit settings, which hinders further enhancement of model capabilities in specific deployment scenarios (such as mobile devices or edge computing). TopKLD addresses this issue by balancing the "implicit knowledge" and "explicit knowledge" derived from full-precision models, enhancing the bit-level scaling behaviors of language-style models by one level.

Question 4

1.Potential of Bit Scaling Laws: As shown in Weakness 1, if a model or quantization algorithm is optimized to achieve excellent bit-level scaling laws, it is possible to increase model parameters using lower bit precision while maintaining better generative capability under current resource constraints. This outstanding feature plays a significant role in specific deployment scenarios, such as mobile devices and edge computing.

2.Insights for Model Design: In the field of vision generative models, there has been ongoing debate regarding the use of discrete versus continuous representation spaces (e.g., [1,2,3,4]). Both approaches have shown strong performance in terms of scaling laws. This work, however, takes a different perspective by investigating the impact of these representation spaces on the scaling laws in quantized models. We find that, despite achieving comparable performance at full precision, discrete autoregressive models consistently outperform continuous models across various quantization settings.

3.Introduction of a New method: We introduced the TopKLD method, which enhances knowledge transfer from full-precision models by effectively balancing explicit and implicit knowledge, thereby improving the bit-level scaling performance of language-style models.

[1]Li T, Tian Y, Li H, et al. Autoregressive Image Generation without Vector Quantization[J]. arXiv preprint arXiv:2406.11838, 2024.

[2]Tian K, Jiang Y, Yuan Z, et al. Visual autoregressive modeling: Scalable image generation via next-scale prediction[J]. arXiv preprint arXiv:2404.02905, 2024.

[3]Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4195-4205.

[4]Sun P, Jiang Y, Chen S, et al. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation[J]. arXiv preprint arXiv:2406.06525, 2024.

Reference

[1]Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis[J]. Advances in neural information processing systems, 2021, 34: 8780-8794.

[2] Ho J, Saharia C, Chan W, et al. Cascaded diffusion models for high fidelity image generation[J]. Journal of Machine Learning Research, 2022, 23(47): 1-33.

[3] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.

[4] Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4195-4205.

[5] Gao S, Zhou P, Cheng M M, et al. Masked diffusion transformer is a strong image synthesizer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 23164-23173.

[6] Liu Q, Zeng Z, He J, et al. Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models[J]. arXiv preprint arXiv:2406.09416, 2024.

[7] Gu S, Chen D, Bao J, et al. Vector quantized diffusion model for text-to-image synthesis[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10696-10706.

[8] Tang Z, Gu S, Bao J, et al. Improved vector quantized diffusion models[J]. arXiv preprint arXiv:2205.16007, 2022.

[9] Chang H, Zhang H, Jiang L, et al. Maskgit: Masked generative image transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 11315-11325.

[10] Li T, Katabi D, He K. Self-conditioned image generation via generating representations[J]. arXiv preprint arXiv:2312.03701, 2023.

[11] Yu L, Lezama J, Gundavarapu N B, et al. Language Model Beats Diffusion--Tokenizer is Key to Visual Generation[J]. arXiv preprint arXiv:2310.05737, 2023.

[12] Yu Q, Weber M, Deng X, et al. An Image is Worth 32 Tokens for Reconstruction and Generation[J]. arXiv preprint arXiv:2406.07550, 2024.

[13] Weber M, Yu L, Yu Q, et al. Maskbit: Embedding-free image generation via bit tokens[J]. arXiv preprint arXiv:2409.16211, 2024.

[14] Razavi A, Van den Oord A, Vinyals O. Generating diverse high-fidelity images with vq-vae-2[J]. Advances in neural information processing systems, 2019, 32.

[15] Esser P, Rombach R, Ommer B. Taming transformers for high-resolution image synthesis[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 12873-12883.

[16] Lee D, Kim C, Kim S, et al. Autoregressive image generation using residual quantization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 11523-11532.

[17] Yu J, Li X, Koh J Y, et al. Vector-quantized image modeling with improved vqgan[J]. arXiv preprint arXiv:2110.04627, 2021.

[18] Tian K, Jiang Y, Yuan Z, et al. Visual autoregressive modeling: Scalable image generation via next-scale prediction[J]. arXiv preprint arXiv:2404.02905, 2024.

[19] Sun P, Jiang Y, Chen S, et al. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation[J]. arXiv preprint arXiv:2406.06525, 2024.

[20] Li, Tianhong, et al. "Autoregressive Image Generation without Vector Quantization." arXiv preprint arXiv:2406.11838 (2024).

Weakness 1

We greatly appreciate your valuable review comments. As shown in Table 1, the field of vision generation models currently has two main development paths, and some of these paths exhibit excellent scaling laws. This paper is based on models that, at the time of our research, had already demonstrated scaling laws, which are DiT and VAR, for further exploration.

Due to the chronological order of submissions, scaling laws have also recently been observed in the continuous autoregressive model domain, specifically in MAR. Therefore, following your suggestion, we conducted the same experiments to verify the correctness of our conclusions. Additionally, we validated our findings on LlamaGen, a model similar to VAR, to further enhance the generalizability of our conclusions.

The results of these experiments can be found in Appendix C2 of the rebuttal revision, titled "Empirical Validation Through Additional Models."

Weakness 2

From Table 1, we can observe that in the field of vision generative models, there has been ongoing debate regarding the use of discrete versus continuous representation spaces.[20] Both approaches have shown strong performance in terms of scaling laws. This work, however, takes a different perspective by investigating the impact of these representation spaces on the scaling laws in quantized models. We find that, despite achieving comparable performance at full precision, discrete models consistently outperform continuous models across various quantization settings. Additionally, through our additional experiments on continuous autoregressive models and discrete AR models in Appendix C, as well as the analysis in Section 3.2, it becomes evident that the nature of the representation space—discrete or continuous—has a significant impact on determining whether AR and diffusion models exhibit superior bit-level scaling laws.

Question1

We conducted a statistical summary of vision generation models and, based on this analysis, selected models that have already reported scaling laws for further exploration: MAR[20], VAR[18], DiT[4], and LlamaGen[19]. The details of this overview can be found in Appendix C1 of the rebuttal revision, titled "Overview."

Question 2

As shown in Table 1, current discrete diffusion models do not exhibit scaling laws, making it impossible to explore their bit-level scaling laws. Therefore, we focused on supplementary research into continuous autoregressive models. The results show that, due to their continuous representation space, the bit-level scaling laws of continuous autoregressive models are not as strong as those observed in discrete models, which aligns with our conclusions.

At the same time, we observed that LlamaGen, a discrete autoregressive model, demonstrates the same excellent bit-level scaling laws. This suggests that the observed scaling behavior is not specific to VAR but is instead a result of the discrete representation space, as discussed in Section 3.2. Since the representation space has been abstracted, this characteristic holds universally across various discrete models, as detailed in Section 3.2.

Question 3

Thank you very much for your valuable suggestions. In the field of vision generative models, there has been ongoing debate regarding the use of discrete versus continuous representation spaces (e.g., [17,18,19,20]). Both approaches have shown strong performance in terms of scaling laws. This work, however, takes a different perspective by investigating the impact of these representation spaces on the scaling laws in quantized models. We find that, despite achieving comparable performance at full precision, discrete autoregressive models consistently outperform continuous models across various quantization settings.

Our study is an essential step toward understanding how various models and quantization methods influence bit-level scaling behavior, and it also provides the following recommendations for future work:

From our exploration, we can conclude that discrete representation space reconstruction offers a more stable foundation for scaling at low bit precision. Moreover, we introduced the TopKLD method, which enhances knowledge transfer from full-precision models by effectively balancing explicit and implicit knowledge, thereby improving bit-level scaling performance. This study indicates that achieving optimal bit-level scaling behavior requires a synergistic interaction between model design and quantization algorithms.

Rebuttal Revision Paper Modifications

We greatly appreciate your valuable review comments. We have revised the paper according to your suggestions and submitted the rebuttal version. For detailed modifications, please refer to the rebuttal version PDF and appendix C: Supplementary materials for rebuttal. Below, we address your identified weaknesses and questions, hoping to resolve your concerns and improve our score.

Table 1

Dear reviewer:

Thanks you for your great efforts in reviewing out paper and providing constructive suggestions/comments. To address the weakness you raised regarding the limited scope, we have provided extensive examples of the recent debate on continuous versus discrete representation spaces in vision generation models, as presented in Table 1. Additionally, following your suggestion, we conducted numerous experiments detailed in Appendix C and Figure 1 of the main text to demonstrate the validity and generalizability of our conclusions on bit-level scaling laws for vision generation models. These experiments cover various mainstream research directions in vision generation models. If our rebuttal does not address your concerns, you are warmly wecomed to raise questions. If our responses have addressed your concerns, we sincerely request that you consider raising our score.

Best Wishes!

Authors

Dear reviewer:

Thanks you for your great efforts in reviewing out paper and providing constructive suggestions/comments. To address the weaknesses you raised, we have conducted extensive experiments in appendix C and figure 1 of main paper to alleviate concerns. If our rebuttal does not address your concerns, you are warmly wecomed to raise questions. If our responses have addressed your concerns, we sincerely request that you consider raising our score.

Best Wishes!

Authors

Dear reviewer:

Thanks you for your great efforts in reviewing out paper and providing constructive suggestions/comments. To address the weaknesses you raised, we have conducted extensive experiments in appendix C and figure 1 of main paper to alleviate concerns. If our rebuttal does not address your concerns, you are warmly wecomed to raise questions. If our responses have addressed your concerns, we sincerely request that you consider raising our score.

Best Wishes!

Authors