Thank you for your thoughtful review and for highlighting the strengths of our work, including the unified architecture and the proposed UniGRPO algorithm. We appreciate your recognition, and below we address your concerns in detail.

Q1: The paper lacks an in-depth discussion of the unique advantages offered by the proposed diffusion-based architecture compared to AR-based architectures

A1: We appreciate your insightful suggestion. In the revised manuscript, we will include an expanded discussion on the distinct advantages of our diffusion-based architecture over autoregressive (AR) models. While both unified AR and diffusion models share a probabilistic generative formulation, unified diffusion models offer several unique advantages:

1. Substantially Imporved Sampling Efficiency: A core limitation of unified AR-based models lies in the inference latency for image generation. When designing a unified architecture, the goal is to approach the quality, fidelity, and efficiency of modality-specific models. However, AR models decode sequences in a left-to-right (raster-scan) fashion, leading to an O(N) inference cost for sequence length N. This results in significant delays during image generation. In contrast, diffusion models involve O(T) iterative denoising steps, where T ≪ N, making them notably faster in generation. We compare generation time on an A100 GPU across two state-of-the-art AR-based unified models, Emu3 [1] and Lumina-mGPT [2], against our diffusion-based MMaDA:
   Tab.A: Comparison of Image Generation Time Cost

   Model	Generation Time	Resolusion
   Emu3	~600 seconds	720p
   Lumina-mGPT	~78 seconds	512p
   MMaDA	~15 seconds	512p

   This efficiency gap makes AR-based unified models less practical for real-world deployment, particularly in latency-sensitive applications.

2. Empirical Performance Advantages: To isolate the impact of our diffusion-based formulation, we conduct a direct comparison with an AR baseline using LLaMA-3 8B as the shared backbone. The only difference lies in the training objective (AR likelihood vs. diffusion objective) and inference method (greedy sampling vs. diffusion decoding). Both models are trained on the same dataset with identical compute (64×A100 GPUs for 100K steps). We evaluate them on key generative benchmarks and also report image generation latency:
   Tab.B: Quantitative Comparison of Unified AR and Diffusion Architecture. | Model | PPL ↓ | Image-Caption CLIP Score ↑ | FID ↓ | Image Generation Latency↓ | | ------------ | ---- | -------------------------- | ------- |-| | AR (LLaMA-3 8B, AR loss) | 23.1 | 14.5 | 16.2 |~120s| | MMaDA (Unified Diffusion) | 25.3 | 16.9 | 12.4 | ~7s|

   These results demonstrate that MMaDA outperforms the AR baseline on image generation and multimodal alignment, while maintaining competitive language modeling performance. Most notably, MMaDA achieves 17× speedup in image generation latency, highlighting the efficiency advantage of diffusion-based decoding. These results demonstrate that the diffusion-based decoding avoids the raster-order biases and error accumulation inherent to AR models, especially in vision tasks, and largely improved efficiency. While the PPL is slightly higher, we attribute this to the relatively underexplored nature of diffusion decoding for text generation.

   Notably, recent work such as UniDisc [3] also explores unified modeling using discrete diffusion and reports similar trends: notable improvements in image generation and comparable results in text generation compared to AR-based models. This convergence further supports our design choice and conclusions.

3. Extensions of Capabilities: As detailed in Task Extension part of our Section 5, diffusion models inherently support inpainting, outpainting, and text/image editing without retraining, due to their denoising-based iterative structure. In contrast, AR models typically lack such capabilities without additional architectural or training modifications, limiting their flexibility in multimodal scenarios.

We thank the reviewer again for prompting this deeper analysis. The above discussion and results will be added to the revised version to clarify the distinct advantages of our diffusion-based unified modeling approach.

Q2: The comparison with SOTA models is incomplete in Tables 2 and 3.

A2: Thank you for pointing this out. We acknowledge that due to the rapid progress in this area, some relevant baselines were inadvertently omitted from the original submission. We will include these models—VILA-U, SynerGen-VL, Liquid, and MetaMorph—in our revised tables and update the corresponding discussions to ensure a more comprehensive and fair comparison. Their metrics are presented as follows:

Tab.C: Additional Multimodal Understanding and Image Generation Results

Q3: The writing could be improved. The manuscript relies heavily on bullet points and bolded phrases, which disrupts the overall flow and coherence of the narrative. This formatting choice makes the paper read more like a collection of notes rather than a cohesive essay, ultimately detracting from the clarity and readability of the work.

A3: We appreciate your detailed observations about the use of bullet points and bolded phrases. We would like to clarify that our initial intention in adopting this structured format was to allow readers to more clearly and quickly identify and follow the key innovations and contributions of our work. However, we fully acknowledge your valid concern that this formatting approach may have compromised the narrative flow and overall coherence of the paper. In response to your feedback, we will make a comprehensive revision in the final version of our manuscript to convert bullet-pointed content into well-structured paragraphs and reduce the use of bolded phrases while ensuring that the innovative aspects of our work are clearly presented within a more traditional academic writing style.

Q4: To what extent does UniGRPO improve the model’s understanding and generation performance? A comparison with diff-GRPO would also be helpful.

A4: Thank you for the thoughtful suggestion. We would like to clarify that our UniGRPO framework is designed to demonstrate the effectiveness of reinforcement learning tailored for diffusion models. As such, we focused our evaluation on several representative tasks—including math reasoning, geometry reasoning, and text-to-image generation—which are particularly sensitive to reward-driven optimization in Table 6 of Appendix E.1.

Following your suggestion, we have now extended our ablation to include a broader set of metrics aligned with those in Tables 2 and 3 of manuscript. The ablation results are summarized below and will be included in full detail in the revised manuscript.

Tab.D: Additional Multimodal Understanding Results after UniGRPO

Tab.E: Additional Image Generation Results after UniGRPO

Additionally, we provide a direct comparison with diff-GRPO under matched conditions. While Appendix E.1 originally included the training curve, we now also report metrics for math reasoning, geometry, and image generation after the same 4000 steps training on 64X A100 to more concretely demonstrate the advantages of UniGRPO.

Tab.F: Additional Comparison with diff-GRPO

These results show that UniGRPO offers more stable convergence, higher reward alignment, and improved generalization across modalities.

We appreciate the reviewer’s insightful suggestion and will incorporate these expanded analyses and results into the revised version to more clearly illustrate the advantages of UniGRPO.

[1] Emu3: Next-token prediction is all you need
[2] Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining.
[3] Unified multimodal discrete diffusion

We sincerely thank the reviewer for your valuable feedback, and we address your concerns as follows:

Q1: Overstated claims of MMaDA's performance.

A1:Thank you for the constructive feedback. We provide the following clarifications and revisions:

1. Our core contribution lies not in surpassing all representative models, but in proposing a unified diffusion-based backbone that supports three mainstream tasks including text reasoning, multimodal understanding and image generation, along with a fully open-sourced training and data curation pipeline. This design is intended to promote reproducibility and further progress in unified multimodal modeling. To better reflect this intention, we will revise our claim to clarify that "our model achieves highly competitive performance, and state-of-the-art on some benchmarks, particularly when compared against models trained on publicly available data." We deeply regret any unintentional misinterpretation our initial wording may have caused.
2. We will also include missing models such as Dream-7B (for only text generation) and Janus-Pro-7B (for multimodal understanding and generation) in our final version, and report their parameter counts. Regarding these comparisons, we want to respectfully highlight some crucial factors. Many leading models, including Janus-Pro, are trained on private and internally-curated datasets that are not available to the public. Similarly, Dream-7B is initialized from Qwen2.5-7B, which is substantially stronger than our backbone LLaDA-8B. The pretraining data and process of Qwen2.5 are also undisclosed and are widely believed to be of significantly higher quality. The performance gap resulting from the quality difference between private and publicly available datasets can be considerable. We primarily aim to provide a transparent, fully open-source alternative for a new line of backbone for the community.

Q2: Inconsistent benchmarking results of LLM.

A2: Thank you for your detailed observation and for pointing this out. In Tab.4 of our manuscript, we directly cited the numbers from LLaDA's Table 1, which reflect the base pretraining versions of the models, including Qwen2-7B. This is also consistent with our reported results for LLaDA-8B-base and our own MMaDA-8B. As you rightly mentioned in Weakness 4, our comparisons are made against pretrained LLMs, not instruction-tuned ones. The GSM8K score of 80.2 for Qwen2-7B-base is correct and consistent with other literature; it corresponds to the pretrained base version, not the instruction-tuned variant (which reaches 85.7). To clarify this and provide a more comprehensive view, we have now prepared a more detailed comparison that distinguishes between base models and instruction-tuned models, as shown below:
Table a: Revised LLM Comparison

We will revise the manuscript to clearly separate base and instruction-tuned model results to avoid further misunderstanding and ensure fair comparison across all baselines.

Q3: Comparison of Fairness of Table 4's LLM benchmarking.

A3: Following Q2 and A2, we would like to further clarify the fairness concern raised in Table 4's LLM comparison:

1. First, our core contribution lies in the exploration of a unified diffusion-based training paradigm that supports three mainstream tasks, including text reasoning, multimodal understanding, and image generation, using only open-sourced data and limited computational resources. The goal of our work is not to claim absolute superiority over all existing models, but to demonstrate that it is possible to build a competitive unified model under constrained conditions, and to open-source the entire training pipeline to benefit the community.

2. Regarding fairness, we respectfully argue that while our model includes multi-stage training such as mixed long-CoT and UniGRPO, the comparison is actually unfair to MMaDA. We humbly argue that to focus solely on our post-training stages overlooks the most critical factor in a large model's capabilities: the pre-training phase. As evidenced by previous works [1][2], the large volume and high quality of pre-training data determine a model's foundational intelligence. In this regard, there is a staggering disparity: Qwen2-7B was pretrained on 7 T tokens, LLaMA 3 was pretrained on 15 T tokens. In contrast, the first pre-training stage of our MMaDA used less than 0.1 T tokens from RefinedWeb. Although our model was initialized from LLaDA, integrating a new modality (vision) is not a trivial step. This process fundamentally perturbs the model's existing linguistic knowledge base, necessitating a significant relearning phase. This is quantitatively demonstrated in our ablation studies (Table 6), where our model's GSM8k score drops from LLaDA's 70+ to just 17.4 after our initial multimodal pre-training. Furthermore, since we do not have access to LLaDA's proprietary training data, our newly introduced open-source datasets cannot seamlessly leverage the model's original knowledge.

3. Regarding post-training, our instruction tuning relies entirely on open-source data, totaling fewer than 200K pairs. This stands in stark contrast to LLaDA’s 4.5M meticulously curated, internal data pairs. The difference in both quantity and, crucially, quality is immense. Moreover, models like Qwen2-7B incorporate RL into their training but do not release the specifics or volume of their preference data, making a truly fair comparison with their internal alignment process impossible.

Therefore, we respectfully suggest that the real unfairness lies not in our post-training, but in the resource and data gap between MMaDA and other stronger baselines. Our training pipeline is designed not to gain an edge, but to bridge this gap through carefully staged training. We hope the community sees this as an effort to make unified modeling more accessible and reproducible, despite the inherent challenges.

Q4: Refining related work.

A4: Thanks for your constructive suggestion. We plan to expand our Related Work section in main text as follows (abbreviated version due to rebuttal limits):

Multimodal Understanding: ....... Early MLLMs such as LLaVA, MiniGPT-4 ...... primarily focused on understanding tasks by projecting features from pre-trained modality-specific encoders (e.g., CLIP) into LLM input spaces. These models demonstrated impressive capabilities in multimodal reasoning but were limited in their generation abilities.

Visual Generation: Diffusion models have established themselves as the dominant paradigm........ These approaches excel at high-quality visual synthesis. Meanwhile, purely autoregressive methods, such as LlamaGen and VAR, have attempted to model visual content through sequential prediction but generally achieve lower generation quality than diffusion models......

Unified Vision-Language Foundation Models: Recent unified models like SEED-X, DreamLLM, and Emu3 have demonstrated varying degrees of success....... However, a critical gap remains in the post-training optimization of these unified models.......This work addresses this limitation by developing novel post-training techniques that optimize unified multimodal foundation models for three generative tasks within a single, efficient framework.

Q5: In Table 5, the metrics for each task are limited to only one.

A5: Due to layout and space constraints, we only reported a single representative metric per task in Table 5. In the revised manuscript, we will include a more comprehensive set of metrics and expand the corresponding discussion accordingly. To briefly illustrate, we present the full set of text task results below, with different denoising steps:

We observe that performance degrades across all tasks as denoising steps decrease, with math-related tasks exhibiting the most pronounced drop. This suggests that mathematical reasoning is particularly sensitive to step-wise denoising quality—errors introduced early in parallel decoding are more likely to compound. We plan to investigate more stable denoising strategies in future work to improve robustness on such tasks. And we will revise our manuscript to include a more thorough discussion on sampling efficiency and openly acknowledge the current limitations. We sincerely appreciate your thoughtful feedback and for bringing this important point to our attention.

Q6: Typos in the text

A6: Thank you for your careful observation. We will correct these typos accordingly.

Q7: What do "higher-order solvers" refer to in line 294?

A7: This refers to higher-order denoisers, such as DPM-Solver[3], which leverages high-order numerical integration for efficient sampling. It prioritizes sampling on uncertain tokens with greater precision, potentially enabling faster decoding.

Q8: Sampling algorithm for MMaDA performance evaluation?

A8: We use semi-AR for sampling.

[1] Scaling laws for neural language models.
[2] LLM post-training: A deep dive into reasoning large language models.
[3] Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Neurips 2022.

Dear Reviewer:

Thank you for your constructive and detailed feedback. We would like to begin with a brief self-reflection on our paper’s contributions and limitations, followed by clarification and our revision plan.

The core contribution of our work lies in proposing a new foundational architecture along with new post-training algorithms that successfully unifies multiple modalities and tasks, and demonstrating its effectiveness and competitive performance compared to existing approaches. Our focus has been on exploring this new architectural paradigm, rather than aggressively optimizing for benchmark scores. We acknowledge that in certain benchmarks—such as those reported for Janus-Pro and Dream-7B—our model currently lags behind the state of the art. We believe this performance gap may stem from several factors: (1) the limited quality and quantity of the publicly available training data, (2) the potential representational bottleneck introduced by our reconstruction-only VQ tokenizer, and 3) the disruption of pretrained language capabilities after introducing new modalities. Moving forward, we plan to make further engineering improvements to help close the gap with top-performing models.

Below we provide more clarifications to your questions:

1. Missing results of certain models: We would like to clarify that our selection of models for the quantitative results table was not intended to mislead but rather to demonstrate our model's performance against comparable work. The results for Janus-Pro and Dream were omitted solely due to rebuttal word count constraints and will be included in the revised manuscript. For instance, we provide detailed results for additional models in our response to Reviewer fZ2o.

2. Comparison of LLM models: Similar to the above, we report base model performance because they are the most comparable to ours. And will include a detailed version in the revision.

3. Absence of limitation: Thank you for this valuable reminder. While we included a brief discussion of limitations in the "Conclusion and Limitations" section of the original manuscript, we agree that this deserves further elaboration. During the informative rebuttal period, we also discovered additional limitations, such as recognizing that the vision encoder poses a performance bottleneck. We discussed its strengths and limitations in response A4 to Reviewer gskN and A5 to Reviewer Xen8. In future work, we aim to improve the tokenizer towards a unified one, jointly trained with alignment and reconstruction losses, which may enhance both image understanding and generation capabilities.

4. Further clarification of high-order solvers: You are correct that DPM-Solver is specific to continuous-time diffusion. In our context, “higher-order solver” refers to drawing inspiration from second-order numerical integration to improve discrete diffusion inference. We may extend confidence-based sampling by incorporating second-order confidence dynamics, such as evaluating changes in confidence over iterations to better assess token stability. We mention this as a potential future direction to highlight how inference strategies from continuous diffusion may inspire innovations in the discrete domain.

Dear Reviewer 7Uo4:

We sincerely appreciate the time and effort you dedicated to reviewing our paper. In response to your concerns, we have conducted additional experiments and provided an in-depth analysis on our method.

As the discussion period concludes in two days, we kindly request, if possible, that you review our rebuttal at your convenience. Should there be any further points requiring clarification or improvement, please know that we are fully committed to addressing them promptly. Thank you once again for your invaluable contribution to our research.

Warm regards,

The Authors

Dear Reviewer 7Uo4,

We would like to kindly ask whether our clarification and revision plan address your concerns adequately.

If there are any further issues or additional points you would like us to address, we are more than happy to provide any necessary explanations.

Best Regards,

The Authors

Dear Reviewer 7Uo4,

Thank you again for your feedback. We have provided detailed clarifications and revision plan addressing all your concerns in previous Official Comments. We believe the issues you raised are well-addressable through our proposed revisions.

If you have any additional questions about our clarifications or revision plan, we would be happy to address them. Looking forward to your continued engagement.

Best regards,

The Authors

We sincerely thank you for your time and efforts in reviewing our paper. We are glad to see your acknowledgement of our model's single framework across modalities, state-of-the-art unified performance, and the clear clarification of our three stages of training and thorough ablations. We provide responses for your concerns below:

Q1: Comprehensive ablation studies of UniGRPO process.

A1: Thank you for the insightful suggestion. We agree that a more fine-grained ablation of the UniGRPO components is important for understanding their individual contributions and interactions. While our primary contribution lies in the unified diffusion framework and three-stage training, UniGRPO is implemented with minimal deviations from the original GRPO to ensure reproducibility. We retain KL regularization, use a uniform-ratio mask scheduler, and adopt equal reward weights across modalities for clarity.
In response to your suggestion, we have conducted a more detailed ablation study starting from our stage-2 checkpoint. We independently modified each of the following components: (1) removing the KL constraint, (2) replacing uniform-ratio masking with random masking, and (3) varying the weighting of the reward heads across modalities. Each variant was trained for 4,000 steps on 64 A100 GPUs, and we report the performance on the representative benchmarks for three tasks in the table below:

Table a: Ablations on Components of UniGRPO.

From the results above, we observe the following key findings: (1) The KL constraint has minimal impact on overall performance. It brings negligible improvement in text generation and does not significantly affect the multimodal components, aligning with recent GRPO-based works such as DanceGRPO [1] (2) Using a uniform-ratio mask scheduler consistently improves model performance, by better simulating the progressive noise levels in diffusion training. Please kindly refer to our Appendix E.2, where we reported our training curves using a uniform masking scheduler, further validating these quantitative results. (3) Varying reward weights across modalities leads to task-specific performance trade-offs. We use 1:1:1 for balance and simplicity, deferring optimization to future work. We will incorporate these findings in the final version to provide a more comprehensive analysis of UniGRPO and its effectiveness in training MMaDA. Thank you again for your valuable feedback.

Q2:How sensitive is performance to the relative weights of correctness, CLIP, and ImageReward signals?

A2: We thank the reviewer for raising this important point. We further analyze modality-specific reward weights.

1. Correctness vs. Format Reward We first examine how the balance between correctness reward and format reward affects performance in text reasoning and multimodal reasoning tasks. We conduct UniGRPO training for 4,000 steps from the same stage-2 checkpoint, varying the ratio between correctness reward and format reward across 1:1, 4:1, and 10:1 settings over 3 random seeds. As shown in the table below, the 4:1 setting offers the best trade-off, preserving both performance and output consistency, and best stability across seeds with reduced variance.
   Table b: Ablations on Correctness vs Format Ratio of UniGRPO. | Correctness:Format Ratio | GSM8K | MATH500 | GeoQA | Formatting Acc. |Std. | |------|-------|-----|--------|----------|----| | 1 : 1 | 65.1 | 26.1 | 14.0| 84.1% |0.71%| | 4 : 1 (default) | 66.5 | 28.5 | 19.5|83.6% | 0.35%| | 10 : 1 | 66.3 | 28.5 |19.2| 73.9% |0.64%|

2. For image generation, we varied the ratio between CLIP Score and ImageReward across 1:3, 1:1, and 3:1, also over 3 random seeds, and reported our ablation results as below:
   Table c: Ablations on CLIP vs ImageReward Ratio of UniGRPO. |CLIP : ImageReward Ratio|CLIPScore(×0.1)|ImageReward|Avg.|Std.| |-|-|-|-|-| |1:3|3.01|0.97|1.99|0.62%| |1:1(default)|3.04|0.95|2.00|0.58%| |3:1|3.05|0.91|1.98|0.63%|

   All three reward ratios consistently lead to performance improvements, indicating that the image generation branch is relatively robust to moderate changes in reward composition. Among them, the 1:1 setting achieves the most balanced results and exhibits the lowest variance across training seeds.

Q3: Adaptation for weaker LLM/VLM

A3: Thank you for this insightful question. While we use a strong LLM to ensure high-quality reasoning, we would like to clarify that the Mixed Long-CoT stage is model-agnostic, and our framework is designed to be compatible with any LLM/VLM capable of producing CoT traces. To evaluate its scalability and robustness in lower-resource or domain-specific scenarios, we conducted additional experiments using two alternative curation models: (1) LLaMA3-8B, representing a relatively weaker and smaller model compared to GPT4.1 in low-resource settings. (2) Qwen2.5-Coder-7B, a domain-specialized model for code generation. We used these models to generate CoT traces for corresponding tasks (general reasoning or coding), and performed Mixed Long-CoT fine-tuning for 80,000 steps starting from our stage-2 checkpoint. Results are shown below:

Table d: Ablations on Different LLMs for CoT Traces Curation.

As seen, while the weaker model (LLaMA3-8B) leads to a moderate drop in absolute performance, it still enables improvement of alignment over the baseline. Meanwhile, the domain-specific model Qwen2.5-Coder-7B yields stronger alignment in its respective coding domain, highlighting the adaptability of our framework to different model sources and domains. We believe these results demonstrate that our training mechanism of MMaDA remains effective and extensible even in scenarios where state-of-the-art curation models are unavailable.

Q4: Any plans for model-generated self-CoT data?.
A4: Thank you for the insightful suggestion. Given MMaDA’s unified understanding and generation capabilities, it naturally supports self-generated CoT data and self-evaluation.. After training, the model can generate its own reasoning traces, which can be filtered via (1) external validation (e.g., GPT-4.1), or (2) internal consistency checks—for instance, generating a caption from an image and then regenerating the image from that caption, with CLIP similarity as a self-evaluation signal.

This self-bootstrapping approach is inspired by HermesFlow [2], showing that unified models can iteratively improve via reinforcement and self-generated data. We view this as a promising future direction and plan to explore it in subsequent work.

Q5: trade-off of retaining the MAGVIT-v2 VQ tokenizer versus adopting higher-resolution or learned continuous encoders.

A5: Thank you for pointing out this important consideration. Indeed, the VQ tokenizer presents limitations, and we elaborate on the trade-offs below:

1. First, our primary goal is to establish a unified diffusion backbone as a proof-of-concept for future multimodal models. Using a fixed-resolution VQ tokenizer (f = 16) enables faster convergence and more efficient training, making it ideal for validating our framework and its effectiveness across modalities.

2. For compression f=16, it's actually very common in image generation, for example, SANA[3] compresses by 32X, but is still able to generate high-quality 4K images, and not a limiting factor for visual quality in generative tasks.

3. For image understanding, we agree that continuous encoders may preserve finer visual details for understanding. Prior work, such as Show-o [4] demonstrates this advantage. However, we note two key challenges: (i) The performance gap is partly due to data scale and training paradigm, not just architecture—SigLIP benefits from massive image-text pretraining, while current VQ tokenizers are trained solely on reconstruction. (ii) More importantly, using continuous encoders for understanding but discrete encoders for generation introduces a modality gap in visual representations, potentially degrading consistency in tasks like multi-turn dialogue.

   To address this, we plan to explore unified discrete tokenizers such as UniTok [5], which integrate alignment objectives during tokenizer training. This direction offers a promising path toward consistent, scalable multimodal representation learning beyond MAGVIT-v2.

[1] DanceGRPO: Unleashing GRPO on Visual Generation.
[2] Hermesflow: Seamlessly closing the gap in multimodal understanding and generation.
[3] Sana: Efficient high-resolution image synthesis with linear diffusion transformers.
[4] Show-o: One single transformer to unify multimodal understanding and generation.
[5] Unitok: A unified tokenizer for visual generation and understanding.

We sincerely thank you for your thorough evaluation and constructive feedback. We are grateful for your recognizing the technical soundness of our UniGRPO method, the motivation of our unified discrete formulation across modalities, the strength of our empirical results, and the significance of our research direction. We address the specific comments and questions below.

Q1:Lack of comprehensive ablation studies

A1: We thank the reviewer for this insightful suggestion. We agree that a comprehensive ablation study is crucial for understanding the individual contributions of our key components. However, conducting a full factorial ablation on an 8B-parameter model is computationally prohibitive. Within the limited time of the rebuttal period, we have performed as many ablations as possible and report preliminary results to isolate the contributions of each major component under constrained training budgets.

1. Ablations of the part of the unified architecture.
   To examine the core contribution of MMaDA’s unified probabilistic formulation based on discrete diffusion, we conducted a comparison experiment with an autoregressive (AR) baseline using LLaMA-3 8B as the backbone. The only difference lies in the training loss and inference method: the AR baseline uses AR-style likelihood loss and greedy sampling, while MMaDA employs diffusion-based objectives and decoding. This baseline resembles Emu3 [1], an AR-based unified multimodal model. Both models were trained under identical settings (data, compute, batch size) for 100K steps using 64 A100 GPUs. For both pre-trained models, we evaluate on foundational generative tasks: conditional text generation using Perplexity (PPL), image captioning using CLIP Score, and class-conditioned image generation using FID on ImageNet. We also report image generation latency to assess practical efficiency. The results are reported as follows:
   Table a: Comparison with AR architectures | Model | PPL ↓ | Image-Caption CLIP Score ↑ | FID ↓ | Image Generation Latency↓ | | ------------------------- | ---- | -------------------------- | ------- |-| | AR (LLaMA-3 8B, AR loss) | 23.1 | 14.5 | 16.2 |~120s| | MMaDA (Unified Diffusion) | 25.3 | 16.9 | 12.4 | ~7s|

   These results show that, under matched conditions, MMaDA outperforms the AR-based baseline on image generation and multimodal understanding tasks, while maintaining competitive performance on language modeling (PPL). Most notably, MMaDA achieves a 17× speedup in image generation latency, highlighting the efficiency advantage of diffusion-based decoding. The improvements stem from the structural benefits of diffusion models for vision tasks: diffusion avoids the unnatural raster-order decoding and error accumulation observed in AR models, which results in both lower fidelity and substantially higher latency. While the AR baseline achieves slightly better PPL, we attribute this to the relatively underexplored decoding strategies for diffusion-based language generation. To further validate this conclusion, we note that a recent work, UniDisc [2], independently explores unified modeling using discrete diffusion and compares it with autoregressive (AR) approaches on joint text-image generation and inpainting tasks. Their findings are consistent with ours: discrete diffusion models offer substantial advantages in image generation, while achieving comparable performance in text generation. These aligned observations across independent studies strengthen our claim that discrete diffusion serves as a more natural and effective generative mechanism for multimodal modeling compared to AR-based approaches.

2. Ablations on mixed long-CoT fine-tuning, and the UniGRPO
   For these two parts of MMaDA, we included our evaluation results of MMaDA across stages in Appendix E.1. As shown in Table 6, we report model performance after each training stage across a range of reasoning and generation benchmarks. After Mixed Long-CoT fine-tuning, the model shows substantial gains in tasks requiring multi-step reasoning, particularly in mathematics (GSM8K: +47.8, MATH500: +22.3) and geometric understanding. UniGRPO further boosts performance across all benchmarks. These results clearly demonstrate the complementary benefits of each component: Mixed Long-CoT fine-tuning primarily enhances reasoning abilities through multi-hop instruction tuning, while UniGRPO improves both understanding and generation through reinforcement-based optimization. Together, they enable our model to match or exceed the performance of strong task-specific baselines in a unified manner.

Q2: Scalabilities of our model?

A2: The principle that performance scales with model size is well-documented for autoregressive LLMs. Encouragingly, this trend appears to hold for diffusion-based and unified models as well. The performance leap from MDLM [3] to LLaDA [4] in diffusion LLMs, and the significant gains observed when scaling from Janus-1.5B [5] to Janus-pro-7B [6] in unified MLLMs, strongly suggest that our framework would similarly benefit from larger model sizes.

Q3: Fundamental barriers of scaling?

A3: The primary barrier to scaling at present is not architectural but rather the availability of powerful, open-source base diffusion language models. Our unified probabilistic formulation requires a diffusion-based text model as its foundation. The current open-source ecosystem is largely limited to models in the ~8B parameter range (e.g., LLaDA-8B [4], Dream-7B [7]). The future development of larger and more capable base diffusion LLMs would directly enable us to scale our training paradigm and likely unlock substantial further performance gains.

Q4: Conditions for Poor Performance?

A4: Our model currently faces challenges in tasks that demand extremely fine-grained visual understanding, as well as OCR tasks. This limitation stems from two main factors related to our vision encoder. We use a MAGViT-v2-based VQ encoder. While effective for general scene comprehension, its reconstructive fidelity for intricate patterns like text is limited. The publicly available weights are not sufficiently trained for these specialized, fine-grained tasks.

Besides, the visual encoder was primarily trained on a reconstruction objective. This is a gap compared to state-of-the-art approaches that use vision encoders like SigLIP, which are pre-trained on massive datasets for deep semantic understanding. We identify this as a key area for future work. In the future, we plan to integrate a more powerful and unified tokenizer, such as UniTok[8], into our framework. We believe this will significantly enhance the model's perceptual capabilities.

[1] Emu3: Next-token prediction is all you need
[2] Unified multimodal discrete diffusion
[3] Simple and effective masked diffusion language models. Neurips 2024
[4] Large language diffusion models.
[5] Janus: Decoupling visual encoding for unified multimodal understanding and generation. CVPR 2025.
[6] Janus-pro: Unified multimodal understanding and generation with data and model scaling.
[7] Dream-7B.
[8] Unitok: A unified tokenizer for visual generation and understanding.