About Novelty

We acknowledge that our work is not driven by particularly complex or hard-to-conceive ideas. However, we firmly believe that it addresses previously unresolved problems and carries significant value for the field. We are delighted to see that you recognize the value of this type of work. For your information, given that several other reviewers pointed out that our initial submission lacked a clear summarization of contributions, we have clearly stated our contributions in the abstract and the introduction parts in the revision, and the relevant sections have been highlighted in red for easier review. The summarized key contributions are as follows:

1. We are the first (especially in the open-source domain) to demonstrate that a decoder-only AR model can achieve image generation performance comparable to modern diffusion models. Furthermore, by initializing the model with multi-modal generative pretraining, this capability can be achieved with relatively low computational resources (32 A100 GPUs * 7 days for the complete 7B FP-SFT).
2. We propose UniRep, an image representation that empowers decoder-only AR models with the ability to flexibly generate images of varying aspect ratios.
3. Building on the strong image generation capabilities, we further explore Omni-SFT, an initial attempt to elevate the model into a unified, omni-potent model. Experimental results highlight the immense potential of decoder-only AR models in this direction.
4. We open-source the entire pipeline to encourage the community's further exploration of this topic.

About the phrase zero-shot

Thank you so much for pointing out this abuse of terms, and we totally agree with your opinion. We have carefully revised the expressions to avoid improper claims and potential misunderstandings, and the contents are now arranged in Appendix.B. In fact, by mentioning zero-shot in this manuscript, we referred to a specific phenomenon that we have observed:

given two data flows:

1. Image $\xrightarrow{\text{VQVAE Encoder}}$ latent$\xrightarrow{\text{VQVAE Decoder}}$ Reconstruction1
2. Image $\xrightarrow{\text{VQVAE Encoder}}$ latent$\xrightarrow{\text{Lumina-mGPT using editing system prompt with instruction "no-edit"}}$ latent2$\xrightarrow{\text{VQVAE Decoder}}$ Reconstruction2

We surprisingly find that the quality of Reconstruction2 sometimes surpasses that of Reconstruction1. Note that, while we train on image editing tasks, the model is not trained on the "no-edit" instruction, which led us to originally use the term "zero-shot". This phenomenon is counter-intuitive, as Lumina-mGPT only has access to VQVAE latents during training and cannot access original image, so intuitively the quality of VQVAE reconstruction should somehow build an upper bound for Lumina-mGPT's image generation quality, which contradicts our observations. We find this observation intriguing and believe it may provide some insights (e.g. it may indicate that the latents encoded by VQVAE encoder could contain noises of certain pattern that can be learned and even correctly by Lumima-mGPT after training), so we added relevant contents to the manuscript. Overall, we believe that it would much more proper to claim it as an interesting observation instead of a zero-shot ability.

Improvement from data instead of better training method

Thank you for this comment! We basically agree with the statement about much of the performance boost appears to come from ..., rather than advancements in the model’s actual ability to learn distributions. However, we respectfully clarify that our goal in this work is not to demonstrate that decoder-only autoregressive (AR) modeling has better learning ability than other learning methods. Instead, we aim to show that this modeling approach is at least competent in tasks like image generation, thereby enabling the research community to further explore its true advantage—an easy and elegant unification of various tasks.

Specifically, we believe that the value of decoder-only AR models with discrete media representations lies not in outperforming specialized methods on individual tasks but in their potential to provide a simple, elegant, and extensible framework to unify various understanding and generation tasks across different modalities. This characteristic makes them a promising approach for achieving true unification. Given this innate strength, as long as their task-specific performance approaches that of specially designed methods, such models deserve a distinct place in the field. However, prior to our work, the performance gap between decoder-only AR models and mainstream methods in image generation is kind of too large, which largely hindered progress toward the ultimate goal. In contrast, our work is the first in the community to elevate the image generation performance of decoder-only models to a modern diffusion model level, establishing a critical foundation for their further development.

Dear Reviewer c5qN,

Considering the discussion period is about to end, we sincerely look forward to your replies. We hope that we have addressed your concerns; if not, we are happy to listen to the follow-up questions. Thanks!

Bests, Lumina-mGPT authors

Dear Reviewer c5qN,

Thank you very much for your thoughtful response. After carefully reading your feedback, we believe we have gained a deeper understanding of your concern. If we interpret your concern correctly, you are suggesting that the observed improvement in reconstruction quality should primarily be attributed to the inherent properties of VQ-VAE's discrete representation, and similar phenomena can be also observed in various generative models (e.g., autoregressive models, discrete diffusion models, etc.); in contrast, the improvement is not likely to stem from any particular advantage or uniqueness of (the components of) Lumina-mGPT—especially those claimed as contributions. Thus, we should avoid implying that this phenomenon constitutes an advantage or contribution specific to our work (Lumina-mGPT).

Upon reflection, we completely agree with this concern.

In fact, during writing the previous revision, our understanding was already basically aligned. We simply intended to emphasize that we have observed the phenomenon itself, rather than to claim a causal relationship between this phenomenon and any specific component or contribution of Lumina-mGPT. However, after reviewing your response, we re-examined our related statements and realized that our revisions were not thorough enough. As a result, we have submitted a new revision with updates to Appendix B, and you may review it. The main changes include the following:

1. We updated the section title from "Lumina-mGPT as VQ Codes Refiner" to "Discussions on Reconstruction Quality" to further avoid the aforementioned issue.
2. At the end of the section, we explicitly added the following statement: "It is worth noting that we believe this phenomenon should mainly be attributed to the intrinsic properties of VQ-VAE's discrete representation. We hypothesize that similar effects might also be observed in various generative models (e.g., autoregressive models, discrete diffusion models, etc.) and not due to any specific advantage or uniqueness of the decoder-only autoregressive model or other components of Lumina-mGPT, especially those claimed as contributions."
3. We changed several occurrences of the wording "Lumina-mGPT" to more general expressions, such as "Generative models such as Lumina-mGPT," to emphasize generality.
4. We moved this section from the main text to the appendix, as it is not directly related to the primary focus of the paper. (In fact, it is already done in the last revision).

We would greatly appreciate your further feedback on these improvements.

Once again, thank you for your time and efforts in helping us improve this work!

Sincerely,

The Lumina-mGPT Authors

About Contribution

Thank you very much for the comments, and we apologize for the lack of clarity in our initial submission. In the revision, we have clearly stated our contributions in the abstract and the introduction parts, and the relevant sections have been highlighted in red for easier review.

Overall, our contributions are outlined as follows:

1. We are the first (especially in the open-source domain) to demonstrate that a decoder-only AR model can achieve image generation performance comparable to modern diffusion models. Furthermore, by initializing the model with multi-modal generative pretraining, this capability can be achieved with relatively low computational resources (32 A100 GPUs * 7 days for a 7B FP-SFT).
2. We propose UniRep, an image representation that empowers decoder-only AR models with the ability to flexibly generate images of varying aspect ratios.
3. Building on the strong image generation capabilities, we further explore Omni-SFT, an initial attempt to elevate the model into a unified, omni-potent model. Experimental results highlight the immense potential of decoder-only AR models in this direction.
4. We open-source the entire pipeline to encourage the community's further exploration of this topic.

Why claiming Decoder-only Architecture as a contribution:

First, we totally agree that the original expression was unclear, and we have carefully refined them in the revision. Meanwhile, we hope to kind note that in the initial submission, decoder-only architecture is considered as a feature, instead of contribution. Below is a more precise explanation of what we intended to convey:

Compared to more complex designs like encoder-decoder architectures, decoder-only models have a significant innate advantage: they provide a simple, elegant, and extensible framework to unify various understanding and generation tasks across different modalities. This makes them a promising candidate for achieving true unification. However, the image generation capabilities of existing works in this paradigm—especially in the open-source community—remain limited, raising doubts about whether high-quality image generation is feasible for decoder-only models. This limitation casts a shadow over such models' potential as an all-encompassing solution. In contrast, our work is the first in the community to elevate the image generation performance of decoder-only models to a modern diffusion model level, establishing a critical foundation for their further development.

About the configuration of FP-SFT

Thank you for the question! The progressive training strategy is a widely adopted approach in training diffusion models $1$

$2$ . Due to the quadratic complexity of transformers with respect to sequence length, directly training on high-resolution images results in low throughput and high resource costs. In contrast, a progressive strategy often achieves better visual quality at a fixed training cost.

The specific resolution choices—512 → 768 → 1024—are based on various considerations. For instance, given that we consistently use 32 A100 GPUs for this project, lowering the resolution to 256 would decrease the computation-to-communication ratio and GPU utilization. Additionally, visualization revealed that the reconstruction quality at 256 resolution is relatively poor due to distortion introduced by VQVAE, so we chose 512 as the starting resolution. We selected 1024 as the final resolution because it is a standard in the field for high-resolution image generation. To bridge the gap between the starting and final resolutions, we introduced an intermediate stage at 768 resolution. Figure 4 in the manuscript illustrates the impact of each stage when using FP-SFT.

Overall, we acknowledge that the choice of training configurations lacks rigorous validation. However, conducting extensive ablation studies is challenging for us due to the associated resource costs.

[1] Zhuo, Le, et al. "Lumina-next: Making lumina-t2x stronger and faster with next-dit.", 2024

[2] PKU-Yuan Lab and Tuzhan AI etc. "Open-Sora-Plan", 2024

Dear Reviewer uRrH,

Considering the discussion period is about to end, we sincerely look forward to your replies. We hope that we have addressed your concerns; if not, we are happy to listen to the follow-up questions. Thanks!

Bests, Lumina-mGPT authors

Dear Reviewer uRrH,

As the extended discussion period is coming to an end, we would like to kindly remind you that we sincerely appreciate your further feedback. We hope that we have addressed your concerns and questions. We would also like to know if you have any other concerns or suggestions.

Best regards, Lumina-mGPT Authors

Thank you for your comments and question!

How is Lumina-mGPT different from other works?

First, from a broader perspective, most vision-language models that pivot around large language models (LLMs), often referred to as Multimodal LLMs (MLLMs), rely on pre-trained vision encoders like CLIP for visual understanding and pre-trained vision generators like Stable Diffusion for visual generation. While such approaches achieve good performance on specific tasks, they expose two key gaps: the gap between visual generation and understanding representations, and the gap between learning in the text and vision modalities.

In contrast, Lumina-mGPT follows a different research trajectory. It adopts decoder-only autoregressive (AR) LLMs combined with discrete modality tokenization to unify all modalities into a shared discrete token space. The most significant advantage of this approach is its promising potential to provide an elegant, extensible, and unified framework for both understanding and generation across modalities, making it a compelling direction toward true multimodal unification. Against this backdrop, the differences between Lumina-mGPT and the two works mentioned—Chameleon and Bai et al. (2024)[1]—become evident:

1. Chameleon shares a similar goal to Lumina-mGPT. However, its value is limited by the absence of released image-generation capabilities. The defining strength of decoder-only AR models lies in their ability to unify both understanding and generation across modalities, but without generation capabilities, an understanding-only version cannot meet the broader expectations of such models. In contrast, Lumina-mGPT not only proposes but also open-sources its capabilities, providing the community with a foundation to further explore this domain. Moreover, based on the presented demos and reported quantitative metrics, Chameleon’s image-generation capabilities significantly lag behind those of Lumina-mGPT (Chameleon attains a score of 0.39 on the GenEval benchmark, while Lumina-mGPT attains 0.56).

2. Bai et al. (2024), while undoubtedly an excellent work, focuses solely on the image modality without incorporating natural language. Natural language, however, is the most intuitive and user-friendly way for humans to interact with models. Its absence complicates tasks like generating content tailored to user needs, and often requires lengthy image prompts to specify target tasks. More importantly, language serves as the ideal "glue" for integrating diverse modalities within a shared context, which makes it an important part in fulfilling the primary advantage of decoder-only AR models in achieving broad unification.

Furthermore, none of the existing works have demonstrated that decoder-only AR models can generate photo-realistic images at the level of modern diffusion models. Meanwhile, despite the simplicity of the proposed UniRep, it fills the blank in existing related works regarding variable aspect ratio image processing, and we believe that simplicity is usually not a bad thing in research.

For your information, our contributions are outlined as follows:

1. We are the first (especially in the open-source domain) to demonstrate that a decoder-only AR model can achieve image generation performance comparable to modern diffusion models. Furthermore, by initializing the model with multi-modal generative pretraining, this capability can be achieved with relatively low computational resources (32 A100 GPUs * 7 days for 7B FP-SFT).
2. We propose UniRep, an image representation that empowers decoder-only AR models with the ability to flexibly generate images of varying aspect ratios.
3. Building on the strong image generation capabilities, we further explore Omni-SFT, an initial attempt to elevate the model into a unified, omni-potent model. Experimental results highlight the immense potential of decoder-only AR models in this direction.
4. We open-source the entire pipeline to encourage the community's further exploration of this topic.

[1] Bai, Yutong, et al. "Sequential modeling enables scalable learning for large vision models." 2024.

Dear Reviewer UuZu,

Considering the discussion period is about to end, we sincerely look forward to your replies. We hope that we have addressed your concerns; if not, we are happy to listen to the follow-up questions. Thanks!

Bests, Lumina-mGPT authors

Dear Reviewer UuZu,

As the extended discussion period is coming to an end, we would like to kindly remind you that we sincerely appreciate your further feedback. We hope that we have addressed your concerns regarding the clarity of novelty and the differences from existing works. We would also like to know if you have any other concerns or suggestions.

Best regards, Lumina-mGPT Authors

About quantitative evaluation

Thank you very much for your valuable feedback! We fully agree that adding numerical metrics would help clarify the performance and positioning of Lumina-mGPT. In response, we have tested Lumina-mGPT on popular T2I benchmarks, and the results have been added to the revision (Sec. 3.1). For your convenience, we also show the results here:

As shown in the table, Lumina-mGPT demonstrates a significant improvement over Chameleon on the GenEval benchmark. Furthermore, Lumina-mGPT outperforms Lumina-Next, a modern diffusion transformer trained using the same text-to-image data as Lumina-mGPT, and we infer that generative pre-training may be the reason behind the performance gap. Additionally, Lumina-mGPT shows competitive performance compared to SDXL, while maintaining a stable advantage over SDv2.1, providing an intuitive picture of Lumina-mGPT’s position in the field.

About the zero-shot capability in enhancing visual details

As also pointed out by reviewer c5qN, we believe that our previous statement of zero-shot is inappropriate and we have changed the statement from a zero-shot ability to an interesting observation, please see Appendix.B for details. In fact, the zero-shot visual quality enhancement in the initial submission refers to a phenomenon we observed during our experiments:

given two data flows:

1. Image $\xrightarrow{\text{VQVAE Encoder}}$ latent$\xrightarrow{\text{VQVAE Decoder}}$ Reconstruction1
2. Image $\xrightarrow{\text{VQVAE Encoder}}$ latent$\xrightarrow{\text{Lumina-mGPT using editing system prompt with instruction "no-edit"}}$ latent2$\xrightarrow{\text{VQVAE Decoder}}$ Reconstruction2

We surprisingly find that the quality of Reconstruction2 sometimes surpasses that of Reconstruction1. Note that, while we train on image editing tasks, the model is not trained on the "no-edit" instruction, which led us to originally use the term "zero-shot". This phenomenon is counter-intuitive, as Lumina-mGPT only has access to VQVAE latents during training and cannot access original image, so intuitively the quality of VQVAE reconstruction should somehow build an upper bound for Lumina-mGPT's image generation quality, which contradicts our observations. We find this observation intriguing and believe it may provide some insights (e.g. it may indicate that the latents encoded by VQVAE encoder could contain noises of certain pattern that can be learned and even correctly by Lumima-mGPT after training), so we added the contents to the manuscript. However, since the model is not explicitly trained on the "no-edit" instruction, the process is difficult to control, and quantitative evaluation in a controlled setting is challenging. Besides, our purpose is to present this intriguing observation, rather than claim that Lumina-mGPT was already a good vq code refiner. in the revision, we have carefully refined the expressions for clarity.

About quantitative results of Omni-SFT

Thank you very much for your comment. We fully understand that the true advantage of decoder-only AR models lies in their ability to unify different tasks and modalities. We also agree that performance across a wide range of omnipotent tasks is crucial for evaluating such models. However, considering the current progress in this field—the quality of image generation, which is a prerequisite for achieving omni-potency, is still very limited—in this work we have focused primarily on improving image generation quality, as well as (building on this improved quality) on conducting a preliminary exploration of omni-potency. We are excited to see the initial ability of the model to address various tasks after Omni-SFT. However, we must acknowledge that the current capabilities are still far from being on a level that allows for a reasonable comparison with task-specific models. We look forward to further exploring this topic in future work.

About initializing Omni-SFT from initial mGPT

We would kindly note that the mGPT we use, namely the publicly-released Chameleon model, does not have image generation capabilities. Therefore, we believe that it is quite reasonable and straightforward to first build the image generation ability (with flexible height-width ratio) through FP-SFT, and subsequently apply Omni-SFT to learn tasks that are essentially framed as conditional image generation problems.

About the trend of image quality variation with respect to CFG

Thank you very much for pointing this out. We sincerely apologize for the lack of rigor in the initial submission. In Fig.7 of the original submission, the image quality indeed shows an improving trend as the CFG value increases; however, this conclusion is limited within a small interval (1.0 to 8.0). Our previous description may have mistakenly implied that this trend holds universally across the entire domain. In the revision, see Appendix.E and Fig.8 in the revision, we have included test results over a broader range (1-400), which demonstrate that, overall, the image generation quality follows a trend similar to diffusion models: it initially improves as CFG increases but then deteriorates. We have also revised the relevant phrasing to eliminate any ambiguity.

Dear Reviewer AimJ,

Considering the discussion period is about to end, we sincerely look forward to your replies. We hope that we have addressed your concerns; if not, we are happy to listen to the follow-up questions. Thanks!

Bests, Lumina-mGPT authors

About contribution

Thank you very much for the comment. We acknowledge that our initial submission lacked a clear enumeration of our contributions. Following your comment, in the revision, we have clarified our contributions in the abstract and the introduction parts. For your convenience, the outlined contributions are as follows:

1. We are the first (especially in the open-source domain) to demonstrate that a decoder-only AR model can achieve image generation performance comparable to modern diffusion models. Furthermore, by initializing the model with multi-modal generative pretraining, this capability can be achieved with relatively low computational resources (32 A100 GPUs * 7 days for 7B FP-SFT).
2. We propose UniRep, an image representation that empowers decoder-only AR models with the ability to flexibly generate images of varying aspect ratios.
3. Building on the strong image generation capabilities, we further explore Omni-SFT, an initial attempt to elevate the model into a unified generalist. Experimental results underscore the promising potential of this direction.
4. We open-source the entire pipeline to encourage the community's further exploration of this topic.

We also agree that, though multi-modal generative pre-training is an important design choice, claiming it as the core insight is improper, and we have also modified relevant statements in the revision.

About the gap introduced by UniRep

Thank you very much for your comment on the gap introduced by UniRep. While we acknowledge the gap mentioned does exist to some extent, we sincerely believe that this may not significantly affect our model for the following reasons:

1. First, please kindly note that the image generation capability of Chameleon has not been released yet, so we are currently initializing from a model without any image generation capability. In this context, the gap introduced by UniRep is relatively small when compared to the much larger gap between a model with no image generation ability and one that can generate images.

2. As mentioned in Sec.2.2.2 of the manuscript, UniRep is indeed a must instead of a decoration if we are to support the model with variable height/width ratios. This makes the gap something that we cannot avoid. We fully agree that it makes more sense to apply UniRep also to the pre-training stage. However, pre-training comes with enormous resource costs (e.g., Chameleon 7B used 856,481 GPU hours), which are unfortunately beyond the reach of most research labs.

3. On the other hand, despite the existence of this gap, there is no evidence suggesting that it significantly impacts the generation quality. Both the qualitative examples provided in our initial submission, as well as the quantitative results presented in this rebuttal (shown below), demonstrate the superior image generation quality of Lumina-mGPT.

About evaluations on image generation

Thank you so much for this valuable feedback! We sincerely agree with your opinion that quantitative results are necessary to support the following claims:

1. Chameleon demonstrates degraded image generation quality, while Lumina-mGPT performs significantly better.
2. Lumina-mGPT exhibits competitive capabilities relative to diffusion models.

We have since conducted evaluations on popular text-to-image benchmarks, and the results are as follows:

As shown, Lumina-mGPT demonstrates a significant improvement over Chameleon on the GenEval benchmark. Furthermore, Lumina-mGPT outperforms Lumina-Next, a modern diffusion transformer trained using the same text-to-image data as Lumina-mGPT, and we guess that generative pre-training may be the reason behind this performance gap. Additionally, Lumina-mGPT shows competitive performance compared to SDXL, while maintaining a stable advantage over SDv2.1, providing an intuitive picture of Lumina-mGPT’s position in the field.

The results have also been added to the revision (Sec.3.1).

About text-only capability

We fully acknowledge the importance of text-only capabilities. However, our work primarily focuses on the model's performance in image-text-related tasks, particularly in image generation. The text-only performance depends on factors such as the quality of the pretrained model and the pure text data used in supervised fine-tuning (SFT), which are orthogonal to the focus of this work.

Dear Reviewer nbPV,

Considering the discussion period is about to end, we sincerely look forward to your replies. We hope that we have addressed your concerns; if not, we are happy to listen to the follow-up questions. Thanks!

Bests, Lumina-mGPT authors

Dear Reviewer nbPV,

As the extended discussion period is coming to an end, we would like to kindly remind you that we sincerely appreciate your further feedback. We hope that we have addressed your concerns regarding evaluation, the clarity of novelty, etc. We would also like to know if you have any other concerns or suggestions.

Best regards, Lumina-mGPT Authors