We would like to sincerely thank the Reviewer for the feedback.

W1: The main difference between the proposed method and the one in Fig. 1(b)

We sincerely thank the reviewer for raising this important comparison with the related works in Fig.1 (b). Below are several key differences between our method and the methods under Fig.1 (b):

• Image Prior Injection Strategy: The key difference between our method and those in Fig. 1(b) is that we directly inject 2D image tokens from the MLLM into DiT latents via a ControlNet-based architecture, enabling stronger and more explicit spatial conditioning. In contrast, Fig. 1(b) methods use image/text tokens (e.g., MetaMorph, MetaQuery, DreamLLM) or bounding box layouts (e.g., Layout-GPT, VPGen), typically injecting information via cross-attention, which offers weaker spatial guidance and lower training efficiency.

To support our claim that Latent ControlNet is more efficient than cross-attention, we conducted an additional experiment comparing our default ControlNet-based injection with a cross-attention-based variant using the same 64 MLLM-generated image tokens. The cross-attention setup unfreezes all modules in the conditioning branch, resulting in 21% trainable parameters in FLUX.1-dev, compared to only 13% in our design. Both variants were trained on ImageNet for 7 epochs under the same compute budget. As shown below, the ControlNet-based approach achieves better image generation quality (FID, sFID, IS), demonstrating its superior efficiency and effectiveness.

• Training Objective: Some methods in Fig. 1(b), such as DreamLLM and MetaMorph, generate visual tokens autoregressively (AR) and visualize them using a diffusion head. In contrast, we adopt a discrete diffusion / MAR [1] decoding strategy, where masked tokens are iteratively decoded into clean tokens. During inference, we randomly sample a generation order over image patches and follow a MaskGIT-style [2] cosine schedule to progressively unmask tokens. This schedule, as visualized in Figure 2 of the MaskGIT paper, allocates fewer unmasked tokens in the early stages (when uncertainty is high), and progressively more in later steps as more tokens become available. As shown in Table 1 of the MAR paper, MAR is significantly more efficient and parallelizable than standard AR, supporting our design decision to prioritize training efficiency.

• MLLM-aligned encoder (Ours) vs. No pre-aligned visual encoder: Our method uses the MLLM’s native CLIP visual encoder for visual generation, while most methods in Fig. 1(b) use non–pre-aligned visual encoders such as SigLIP or VAE. Our approach offers three key advantages:

  • (1) Since we initialize the visual generation branch with the same parameters as the original MLLM, its features are natively aligned with the MLLM’s CLIP visual encoder, which significantly accelerates training.

  • (2) Unlike methods that adopt new visual encoders (e.g., SigLIP or VAE), our approach does not require additional projection layers to connect the MLLM with the visual encoder, simplifying the architecture and improving efficiency.

  • (3) The MLLM’s native CLIP encoder encodes a 256×256 image into just 64 patch tokens. In contrast, non–pre-aligned encoders like SigLIP (e.g., ViT-SO400M-14-SigLIP-384) upscale the image to 384×384 and produce 729 patch tokens, requiring further downsampling or compression. This adds complexity and may lead to information loss.

To validate our claims, we compare our default strategy (using the MLLM’s native CLIP visual encoder) with the SigLIP encoder under the same compute budget (7 epochs for the MLLM vision branch and 1 epoch for ControlNet) on ImageNet at 256×256 resolution. Our method consistently outperforms the SigLIP-based variant in image quality. For completeness, we also include results with the VAE encoder from FLUX.1-dev, as shown in Table 1 of our main paper, which further highlights our approach’s efficiency and performance advantage.

• Parameter-efficient training (Ours) vs. full-finetuning: We insert lightweight ControlNet blocks (13% of FLUX.1-dev) into a frozen DiT, keeping all pretrained weights untouched. In contrast, methods in Fig. 1(b) often fully fine-tune DiT to adapt to new visual tokens, which becomes especially challenging for video generation tasks requiring high fidelity and temporal consistency.

• MLLM Freezing Strategy: We keep the MLLM fully frozen to preserve its reasoning and alignment abilities. In contrast, methods like MetaMorph and JanusPro [3] fine-tune the MLLM to enable token generation, risking degraded performance without massive, high-quality data. As shown in Table 3 of the JanusPro paper, their multimodal understanding performance is much lower than that of MetaQuery and BLIP3o, both of which keep the backbone frozen. This supports our design choice to freeze the MLLM.

We hope this helps clarify our technical distinctions from Fig. 1(b). We’re happy to provide further clarification if needed. Thank you again!

[1] Autoregressive image generation without vector quantization, NeurIPS, 2024

[2] Maskgit: Masked generative image transformer, CVPR, 2022.

[3] Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv, 2025.

---

W2: The proposed method and MetaQuery-L seem to have comparable performance on multimodal understanding tasks

Thank you for this insightful observation! Indeed, the concurrent work MetaQuery and our method achieve the same scores on multimodal understanding tasks, as both use Qwen2.5-VL as the base model and keep it frozen during training. Specifically, MetaQuery uses query tokens to extract semantic information from the frozen multimodal LLM, while our method creates a trainable copy as the vision branch, sharing the same visual encoder. As a result, both approaches are expected to perform similarly on understanding tasks. We will clarify this point in the camera-ready version. Thank you again for highlighting this important detail.

---

Q1: Clarification on the experiments in sections 5.1-5.3

Thank you very much for raising this question. Sections 5.1 and 5.2 are conducted on the ImageNet dataset at 256×256 resolution, which contains approximately 1.2M images across 1,000 classes. Specifically, Section 5.1 focuses on text-to-image generation, while Section 5.2 presents image reconstruction experiments. All experiments in these sections are trained on a single GB200 GPU.

In Section 5.3, we scale up to open-world image-text pairs, using a total of 5M samples. Accordingly, we increase the training resources to 8 H100 GPUs instead of a single GB200.

Additionally, we observe that our latent ControlNet typically converges much faster than the vision branch in multimodal LLMs due to its lightweight design. As a result, we allocate a smaller training budget to ControlNet compared to the MLLM vision branch.

---

Q2: Additional scale-up results

Thank you very much for the valuable question regarding training cost. In addition to the comparison of different architectures under the same compute budget in Table 1 of our main paper, we would like to highlight that training cost comparisons with other existing baselines are also provided in Appendix Table 1 (available in the PDF of the supplementary materials, as mentioned in L284-285 of our main paper). In Appendix Table 1, training cost is measured by the total amount of data used, calculated as the product of the number of training steps and the batch size per step. As shown, our method achieves performance comparable to baselines such as Janus and EMU, while requiring significantly less training data (9M for Ours vs. 100M for Janus).

Beyond the results presented in the appendix, we obtained additional GPU resources after the submission deadline and conducted larger-scale experiments using the same training dataset as BLIP3-o [4], which is a recent SoTA method that connects MLLMs with diffusion models for image generation (released on May 14, 2025). Specifically, we use 27 million image-text pairs from the BLIP3o-Long-Caption dataset and train our model for 1.7 epochs, followed by fine-tuning on 60K high-quality images from the BLIP3o-60K dataset. In contrast, BLIP3-o is trained on the same 27M dataset for over 10 epochs. A comparison is summarized in the table below:

As shown, the GenEval score improves from 61% to 81%, and DPGBench increases from 76.41% to 77.67%, demonstrating that better annotations and scaled-up training lead to meaningful gains. Notably, even with this larger-scale setup, our training cost remains much lower than prior works, while still outperforming existing approaches under the same compute budget.

[4] Blip3-o: A family of fully open unified multimodal models—architecture, training and dataset, arXiv, 2025.

---

Dear Reviewer XePf,

We sincerely thank you for all the comments and valuable feedback. We believe that we have addressed all the questions in depth. If you have any additional questions, please feel free to let us know, and we would be more than happy to address them.

Dear Reviewer XePf,

As today marks the end of the author-reviewer discussion period, we would like to express our heartfelt thanks for your valuable feedback and thoughtful suggestions throughout the rebuttal process.

We sincerely appreciate your recognition of our work, particularly our proposed architecture design that injects image generation capabilities with a relatively small training cost while maintaining the MLLM inference ability.

It’s also glad to know that our previous response could be helpful to address your concerns. If you have any remaining questions, please don’t hesitate to let us know, and we would be more than happy to address them by the end of today.

Thank you again for your time and effort!

We would like to sincerely thank the Reviewer for all the valuable feedback. In the rebuttal below, we use “W1/W2/W3” to represent the answer to the bullet points in weaknesses, and “Q1” to represent the answer to the questions.

W1: Could an alternative have been to let the MLLM generate a modality (e.g., depth, sketch) that could be consumed by an existing pretrained ControlNet? While this might compromise reconstruction quality, how would it affect generation fidelity

Thank you very much for bringing up this interesting question! Indeed, one could allow the MLLM to generate an intermediate modality such as depth or sketch, and then reuse a pretrained ControlNet together with an image generation model to render the final image. However, this approach may have several limitations:

• First, the information encoded in a depth or sketch image may be less comprehensive than what is captured by MLLM-generated latents (e.g., the 2048-dimensional CLIP latents from Qwen2.5-VL 3B). Such MLLM-aligned CLIP visual encoder makes it easier to capture richer patch-level image semantics compared to the structural cues available in depth or sketch images. For example, the latents can encode more detailed semantics, including object layouts, colors, textures, and shapes—going beyond simple depth or edge information.

• Second, using a pretrained depth/sketch ControlNet can be more error-prone than training a latent-based ControlNet tailored to the MLLM outputs. This is because the depth or sketch images generated by the MLLM might be imperfect. As a result, strictly following such noisy guidance using a pretrained ControlNet can lead to suboptimal image quality. In contrast, training a ControlNet directly on MLLM-generated latents offers the flexibility to learn how to interpret and leverage those more-or-less noisy latents more effectively than relying on fixed modality-specific guidance. In fact, this is precisely one of the main reasons why we connect the MLLM with DiT through ControlNets. We aim to reduce the burden on the MLLM to generate highly accurate images, as high-level semantics or “blurry” latents are already sufficient to guide the image generation model.

---

W2: Quantitative comparison of training costs with existing baselines

Thank you very much for the valuable question regarding training cost. In addition to the comparison of different architectures under the same compute budget in Table 1 of our main paper, we would like to highlight that training cost comparisons with other existing baselines are also provided in Appendix Table 1 (available in the PDF of the supplementary materials, as mentioned in L284-285 of our main paper). In Appendix Table 1, training cost is measured by the total amount of data used, calculated as the product of the number of training steps and the batch size per step. As shown, our method achieves performance comparable to baselines such as Janus and EMU, while requiring significantly less training data (9M for Ours vs. 100M for Janus).

Beyond the results presented in the appendix, we gained more GPU support after the paper submission deadline, and here we include additional larger-scale experiments trained on the same training dataset from BLIP3-o [1], one of the latest works that connects MLLM with diffusion models for image generation released on May 14, 2025. Specifically, we use 27 million image-text pairs from BLIP3o-Long-Caption dataset, and train the model for 1.7 epochs, followed by fine-tuning on 60k high-quality images in the BLIP3o-60K dataset. In contrast, BLIP3-o is trained on the same 27M dataset for over 10 epochs. We summarize the comparison in the following table:

As shown, the GenEval score improves significantly from 61% to 81%, and the DPGBench score increases from 76.41% to 77.67%. This demonstrates that higher-quality training data with better annotations, as well as scaled-up training, can lead to meaningful performance gains. Lastly, we would like to emphasize that even with this larger-scale training, our training cost remains much lower than that of prior works, and our method still outperforms previous approaches under the same training budget.

[1] Chen, Jiuhai, et al. "Blip3-o: A family of fully open unified multimodal models—architecture, training and dataset." arXiv preprint arXiv:2505.09568 (2025).

---

W3: A more detailed architectural description

Thanks for your question. Here we explain our architecture in more detail.

• Adding a vision generation branch to the MLLM: Starting from a pretrained MLLM (e.g., Qwen2.5-VL 3B), we create a copy of its parameters (e.g., attention layers, FFN layers) to serve as the visual generation branch. This branch is initialized with the same pretrained weights as the original model. The original MLLM parameters are kept fully frozen, and only the new vision generation branch is trainable.

• Encoding images using the MLLM’s native visual encoder: Unlike prior works that use a VAE or CLIP/SigLIP-based encoder to process images, we utilize the MLLM’s native visual encoder during training to encode the images. As shown in Table 1 of our main paper, using the MLLM’s native visual encoder, rather than a VAE as the image encoder, results in significantly better image generation quality under the same computation budget.

• Masked autoregression (MAR) for visual token prediction: During training, we apply MAR to teach the model to predict masked visual tokens. Specifically, the training image is first encoded using the MLLM’s visual encoder. Then, a random value α is sampled, and α × (number of visual tokens) are randomly masked. The text condition is passed into the frozen MLLM branch to guide visual token prediction. The model is trained to reconstruct the masked tokens, with the training objective being the mean squared error (MSE) loss between the predicted and ground truth visual tokens in the masked regions.

• Latent ControlNet for image decoding: The predicted visual tokens are treated as 2D visual cues or image sketches, which are then passed to a latent ControlNet to generate the final image. The overall architecture closely follows the FLUX.1-Dev ControlNet design, but we make it more lightweight by using only 4 MM-DiT (DoubleStream) blocks and 1 Single-DiT (SingleStream) block (i.e., num_double_layers=4, num_single_layers=1). The training objective matches that of FLUX DiT, using a flow-matching noise scheduler to sample timesteps and predict the flow direction from noise to data.

• Inference process: During inference, we begin with a fully masked 2D token grid, which is fed into the MLLM’s vision generation branch. The accompanying text condition is provided to the frozen MLLM branch to guide the visual token prediction. Prior to decoding, we randomly sample a generation order over the image patch tokens, which determines the sequence in which masked tokens are decoded into clean ones. Following MaskGIT [2], we adopt a cosine schedule to determine how many tokens are unmasked at each decoding step. This schedule, as visualized in Figure 2 of the MaskGIT paper, allocates fewer unmasked tokens in the early stages (when uncertainty is high), and progressively more in later steps as more tokens become available. Finally, these decoded tokens are then passed to the latent ControlNet as visual cues to guide final image generation.

[2] Chang, Huiwen, et al. "Maskgit: Masked generative image transformer." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

---

Q1: Generation fidelity and diversity as the number of CLIP tokens increases

Thank you again for raising this interesting question! In our current implementation, we do not use additional modules or loss functions to explicitly encourage diversity in the generated objects. Therefore, the observation from VAEs, where diversity decreases as the number of tokens increases, may also apply to our case.

While this paper primarily focuses on how to effectively connect MLLMs with diffusion models, we fully agree that promoting diversity in the generated outputs is an important and orthogonal direction worth exploring. We are aware of existing works that address this issue [3], and it would be interesting to investigate whether such methods can be adapted to our architecture. We leave this for future work.

[3] Gu, Jiatao, et al. "Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling." Advances in Neural Information Processing Systems 37 (2024): 5498–5527.

---

Dear Reviewer k2H1,

Once again, we sincerely thank you for all the comments and valuable feedback. We believe that we have addressed all the questions in depth, including (1) explanation of our design choice of generating image latents and the use of latent ControlNet, (2) quantitative comparison of training costs with existing baselines, (3) a more detailed architectural description, and (4) discussion of the influence of generation fidelity and diversity as the number of CLIP tokens increases. If the reviewer has any additional questions, please feel free to let us know, and we would be more than happy to address them.

Dear Reviewer k2H1,

As today marks the end of the author-reviewer discussion period, we would like to express our heartfelt thanks for your valuable feedback and thoughtful suggestions throughout the rebuttal process.

We sincerely appreciate your recognition of the strengths of our work. We're grateful for your positive assessment of the overall performance and the well-structured experimental setup, as well as the well-justified design choices, especially the comparisons between CLIP latents and alternatives such as VAE or learnable 2D query tokens.

If you have any remaining questions or concerns, please don’t hesitate to let us know, and we would be more than happy to address them by the end of today.

Thank you again for your time and effort!

We would like to sincerely thank the Reviewer for all the valuable feedback. In the rebuttal below, we use “W1/W2” to represent the answer to the bullet points in weaknesses, and “Q1/Q2” to represent the answer to the questions.

W1/Q1: Comparison with MetaMorph

We sincerely thank the reviewer for raising this important comparison with the related work, MetaMorph. We fully agree that both MetaMorph and our method share the high-level goal of developing a unified framework that decodes LLMs/VLMs outputs with diffusion models. However, we would like to kindly emphasize several key differences between the two approaches:

• Image Prior Injection Strategy: Our method directly injects 2D image tokens generated by the MLLM into the DiT latents via a ControlNet-based architecture. By adding 2D ControlNet latents on top of the noisy DiT latents, our method can enforce spatial structures more explicitly and effectively. In contrast, MetaMorph uses cross-attention to condition on image tokens. which behaves more like a global reference image. However, such cross-attention conditioning typically provides weaker spatial guidance than ControlNet and results in lower training efficiency.

To support our claim that Latent ControlNet is more efficient than cross-attention, we conducted an additional experiment comparing our default ControlNet-based injection with a cross-attention-based variant using the same 64 MLLM-generated image tokens. The cross-attention setup unfreezes all modules in the conditioning branch, resulting in 21% trainable parameters in FLUX.1-dev, compared to only 13% in our design. Both variants were trained on ImageNet for 7 epochs under the same compute budget. As shown below, the ControlNet-based approach achieves better image generation quality (FID, sFID, IS), demonstrating its superior efficiency and effectiveness.

• Training Objective: MetaMorph generates visual tokens autoregressively (AR) and then visualizes them using a diffusion head. In contrast, we adopt a discrete-diffusion/MAR [1] decoding strategy to decode from masked tokens into clean visual tokens. Specifically, during inference, we randomly sample a generation order over the image patch tokens, which determines the sequence in which masked tokens are decoded into clean ones. Following MaskGIT [2], we adopt a cosine schedule to determine how many tokens are unmasked at each decoding step. This schedule, as visualized in Figure 2 of the MaskGIT paper, allocates fewer unmasked tokens in the early stages (when uncertainty is high), and progressively more in later steps as more tokens become available. As shown in Table 1 of the MAR paper, MAR is significantly more efficient and parallelizable than standard AR, supporting our design decision to prioritize training efficiency. This training objective difference, together with the cross-attention image token injection strategy, made us to classify MetaMorph into Fig1(b). This being said, we fully agree with the reviewer's suggestion that classifying MetaMorph as a 1D image token may be inaccurate. We will update the label of Fig. 1(b) to 'Bridge w/ 1D/2D Image/Text Tokens' in the camera-ready version. Thank you again for pointing out this inaccuracy.

• MLLM-aligned encoder (Ours) vs. No pre-aligned visual encoder (MetaMorph): Our method uses the MLLM’s native CLIP visual encoder for visual generation, while MetaMorph uses SigLIP (i.e., ViT-SO400M-14-SigLIP-384) to encode images. Our approach offers three key advantages:

  • (1) Since we initialize the visual generation branch with the same parameters as the original MLLM, the features in the visual generation branch are natively aligned with the MLLM’s CLIP visual encoder.

  • (2) Compared to using new visual encoders such as SigLIP or VAE, our method does not require learning additional projection layers to connect the MLLM with the visual encoder, simplifying the architecture and improving efficiency.

  • (3) The MLLM’s native CLIP encoder encodes a 256×256 image into just 64 patch tokens. In contrast, ViT-SO400M-14-SigLIP-384 first upscales the image to 384×384 and encodes it into 729 patch tokens, requiring additional downsampling or compression to reduce the number of image tokens. MetaMorph uses vanilla bilinear interpolation to compress the features down to 64 tokens, which may lead to information loss. Alternatively, one would need to use learnable downsampling blocks to better preserve image information, adding architectural complexity.

To support these claims, we provide a new quantitative comparison between our default strategy (MLLM’s native CLIP visual encoder) and the SigLIP encoder. We train both variants with the same compute: 7 epochs for the MLLM vision branch and 1 epoch for ControlNet, on the ImageNet dataset at 256×256 resolution. As shown, our default strategy achieves much better image generation quality compared to the SigLIP-based approach. Additionally, Table 1 in our main paper compares our default strategy with the VAE encoder used in FLUX.1-dev. We also show the numbers for VAE encoder here for completeness.

• Parameter-efficient training (Ours) vs. full-finetuning (MetaMorph): We insert lightweight ControlNet blocks (13% of FLUX.1-dev) into a frozen DiT, keeping all pretrained weights untouched. In contrast, MetaMorph needs to fine-tune DiT to adapt to new visual tokens, which becomes especially challenging for video generation tasks requiring high fidelity and temporal consistency.

• MLLM Freezing Strategy: We keep the MLLM fully frozen to preserve its reasoning and alignment abilities. In contrast, methods like MetaMorph and JanusPro [3] fine-tune the MLLM to enable token generation, risking degraded performance without massive, high-quality data. As shown in Table 3 of the JanusPro paper, their multimodal understanding performance is much lower than that of MetaQuery and BLIP3o, both of which keep the backbone frozen. This supports our design choice to freeze the MLLM.

• Performance & Reproducibility: In addition to the attached code in the supplementary material, we have made our model design, training data, and code fully transparent, and will open-source all components upon acceptance. In contrast, MetaMorph is still under legal review, has not released model weights, and lacks evaluation results on standard benchmarks like GenEval or DPGBench, making direct comparisons difficult. We also highlight that our method achieves performance close to SoTA models like MetaQuery and BLIP3o, both generally stronger baselines than MetaMorph, while requiring much lower training cost.

Overall, we have done our best to objectively clarify the differences between our method and MetaMorph, providing detailed explanations and additional experiments to support our claims. We also highly value the contributions of MetaMorph. We hope this explanation helps illustrate the key technical and practical advantages of our method. If the reviewer has any further questions or suggestions, we would be more than happy to address them. Thanks again!

[1] Autoregressive image generation without vector quantization, NeurIPS, 2024

[2] Maskgit: Masked generative image transformer, CVPR, 2022

[3] Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv, 2025

---

W2: Additional quantitative scale-up results

Beyond the results presented in the appendix, we obtained additional GPU resources after the submission deadline and conducted larger-scale experiments using the same training dataset as BLIP3-o [4], which is a recent SoTA method that connects MLLMs with diffusion models for image generation (released on May 14, 2025). Specifically, we use 27 million image-text pairs from the BLIP3o-Long-Caption dataset and train our model for 1.7 epochs, followed by fine-tuning on 60K high-quality images from the BLIP3o-60K dataset. In contrast, BLIP3-o is trained on the same 27M dataset for over 10 epochs. A comparison is summarized in the table below:

As shown, the GenEval score improves from 61% to 81%, and DPGBench increases from 76.41% to 77.67%, demonstrating that better annotations and scaled-up training lead to meaningful gains. Notably, even with this larger-scale setup, our training cost remains much lower than prior works, while still outperforming existing approaches under the same compute budget.

[4] Blip3-o: A family of fully open unified multimodal models—architecture, training and dataset, arXiv, 2025.

---

Q2: Paper restructure suggestion

Thank you very much for the suggestion. We will move the generation benchmark table from the appendix, along with the new scaled-up experiment results, into the main paper to better emphasize their importance in the camera-ready version.

---

Dear Reviewer 98F3,

Once again, we sincerely thank you for all the comments and valuable feedback. We believe that we have addressed all the questions in depth. If there’s any additional questions, please feel free to let us know, and we would be more than happy to address them.

We sincerely thank the reviewer for the feedback. The following are our responses to the reviewer's corresponding questions:

Clarification of the reviewer’s previous comments

First, we would like to kindly clarify that our paper does not claim MAR as a novelty or contribution (see L72–L79). In our response, we mentioned MAR only because the reviewer explicitly asked us to clarify the significant differences between our method and MetaMorph. In doing so, we outlined our high-level motivation and corresponding design differences. We explicitly stated that MAR is not a novel idea proposed by us, nor is it a primary contribution of our work. Rather, we use it to further reduce training cost.

New experimental results

Regarding this second point, we never claimed that using an MLLM to assist image generation is our novelty. As clearly stated in our response, our contribution lies in using an MLLM to generate implicit 2D spatial image priors to guide downstream image generation. To the best of our knowledge, this specific design has not been explored in previous works.

As supported by our empirical results, we show that spatial tokens injected via ControlNet, added element-wise to the diffusion latents, offer more effective spatial conditioning than the cross-attention mechanism used in MetaMorph. In this sense, our approach complements the findings in MetaMorph (which suggest that strong understanding capabilities can enhance generation), and can be applied as a more effective model architecture compared with the cross-attention used in MetaMorph.

We believe our work offers a valuable new perspective: that VLM-generated image tokens can serve as effective priors for guiding diffusion models. We respectfully suggest that the reviewer may have misunderstood our core idea, which was clearly recognized by other reviewers as noted in our earlier responses.

Repetition of the reviewer’s concern about claimed contributions

As for the third point, we believe our previous response has already answered the reviewer’s question. If there are specific additional experiments that the reviewer believes essential to support our core claims, we are more than happy to conduct them and discuss further.

MAR and prior works

Regarding our response to the reviewer’s original question about differences from MetaMorph, we did our best to answer thoroughly, including modeling differences such as AR vs. MAR. Highlighting such differences is a direct response to the reviewer’s request, and we do not believe doing so should be considered a weakness.

The performance and comparison on DPGBenceh

For efficiency, we presented two sets of experiments.

• First, in Table 1 and additional rebuttal experiments on ImageNet, we compared various architectural designs under the same data and compute budgets. These results demonstrate that our architecture is more efficient than alternatives under a fair and strictly controlled setting.

• Second, on GenEval and DPGBench, a fair apples-to-apples comparison is challenging due to the varying training data, compute resources, and model designs used by different baseline methods. Therefore, our experiments in the paper along with the additional scaling experiments, are intended to demonstrate that our method achieves performance almost comparable to existing baselines while using significantly fewer resources.

Reviewing fairness

Lastly, we would like to clarify that we never intended to suggest the reviewer was being unfair. Our comments referred solely to specific statements that we believe may not fully align with the content of our paper or our responses. If our wording gave the wrong impression, we sincerely apologize. We have great respect for all reviewers and deeply value the peer review process.

---

Final Notes

We thank the reviewer for their continued feedback and greatly appreciate the time and effort spent engaging with our work. As our shared goal is to explore open research questions and contribute good ideas to the community, we remain happy to provide further clarifications or conduct additional experiments if needed. While we appreciate the reviewer’s perspective and will keep their suggestions in mind for future extensions, we believe that the current version, together with the additional experiments included during the rebuttal, already provides strong support for our claims.

Thank you to the authors for the new response. I would like to clarify my position and provide my updated comments below.

1. Clarification of my previous comments.

When I stated that “Overall, the paper largely claims contributions from existing methods”, I was referring to both the paper and the responses. In the authors' earlier response ("Clarifying our contributions: more than just patch-level CLIP latents"), the MAR is explicitly listed. This is why I interpreted it as being claimed as a contribution or a difference from other methods.

2. New experimental results.

For Table 2 in the response, my understanding is that the new experiments replace the VLM's original visual encoder with SigLIP or VAE. It is surely natural that these two variants perform worse when trained with only ImageNet with 7 epochs for the MLLM vision branch and 1 epoch for ControlNet. This is because the features are not aligned, and the LLM cannot understand the new image features. While these experiments show a performance gap, I do not find that they provide new insights. The authors' explanation, "We choose to use a VLM rather than an LLM for a straightforward reason: LLMs do not come with natively aligned visual encoders." is already well known and has been explicitly discussed in Kosmos-G. Kosmos-G highlights that using MLLMs for generation benefits from:

1. It capitalizes on the inherent vision-language alignment within the MLLM.
2. The MLLM architecture naturally supports interleaved interleaved multi-image and text input.
3. The pre-trained MLLM can effectively model multimodal input in context.

These reasons appear very similar to the authors' current explanation.

That is also why MetaMorph needs to perform training of vision understanding jointly with the generation. This paper says, "(1) visual generation ability emerges as a natural byproduct of improved visual understanding, and can be unlocked efficiently with a small amount of generation data;". Therefore, I am not convinced that this new evaluation offers fundamentally new insights.

3. Repetition of my concern about claimed contributions.

I repeat the same claim of "Clarifying our contributions: more than just patch-level CLIP latents." because I remain unconvinced by the new claims in this section, including ControlNet-conditioning, VLM, and MAR, etc. My concerns are stated in this and previous responses.

4. MAR and prior works.

I would like to clarify that I don't quite agree that "that prior works employing the MAR strategy, such as Show-o and NOVA, also did not re-run comparisons between AR and MAR, yet were still well-received at top conferences (e.g., ICLR’25). We believe that building upon established methods and findings from prior studies should not be viewed as a limitation.". Citing that prior works such as Show-o were accepted without re-running AR vs. MAR comparisons does not address the core issues:

1. My concern is always not about using them. It is about claim them as a difference with other methods. In the authors' response: "Clarifying our contributions: more than just patch-level CLIP latents.", the MAR is listed, and this is why I think it is claimed as a difference or a contribution.

2. The acceptance of previous papers is not a sufficient reason to justify this paper's contribution under NeurIPS standards.

5. The performance and comparison on DPGBenceh.

Results that rely on untested experiments due to GPU limitations should remain unknown, and should not be implicitly interpreted as supporting evidence for efficiency or performance claims. From the current results and my above analysis, I am not convinced that this paper gives a notable improvement.

6. Reviewing fairness.

I want to emphasize that my continued discussion is not because I want to be an unfair reviewer. This is because I am not convinced and would like to make my concerns and questions as clear as possible. I think this is normal in a reviewing process. I am also confident that I have a deep understanding of this area, which is why my review and responses are much detailed than other reviewers, and I aim to ensure my response is both clear, reasonable, and well-supported by related works.

We would like to sincerely thank the Reviewer for the feedback. Since the weakness and questions of the Reviewer are closely related, we use “W/Q” to represent the answer to the corresponding weakness and questions.

W1/Q1: Standard quantitative metrics (e.g., GenEval and DPGBench) comparing BIFROST-1 with other leading T2I generation models.

Thank you very much for the useful comment. We would like to first kindly point out that we have included standard metrics from GenEval and DPGBench in Appendix Table 1 (in the PDF of the supplementary materials, as mentioned in L284-285 of our main paper). In Appendix Table 1, we measure the training cost by the total number of training images passed to the model, calculated as the product of the number of training steps and the batch size per step. As we can see, our method achieves performance comparable to baselines such as Janus and EMU, while significantly lower training cost (training data 9M from Ours v.s. 100M from Janus).

Beyond the results presented in the appendix, we gained more GPU support after the paper submission deadline, and here we include additional larger-scale experiments trained on the same training dataset from BLIP3-o [1], one of the latest works that connects MLLM with diffusion models for image generation released on May 14, 2025. Specifically, we use 27 million image-text pairs from BLIP3o-Long-Caption dataset, and train the model for 1.7 epochs, followed by fine-tuning on 60k high-quality images in the BLIP3o-60K dataset. In contrast, BLIP3-o is trained on the same 27M dataset for over 10 epochs. We summarize the comparison in the following table:

As shown, the GenEval score improves significantly from 61% to 81%, and the DPGBench score increases from 76.41% to 77.67%. This demonstrates that higher-quality training data with better annotations, as well as scaled-up training, can lead to meaningful performance gains. Lastly, we would like to emphasize that even with this larger-scale training, our training cost remains much lower than that of prior works, and our method still outperforms previous approaches under the same training budget.

[1] Chen, Jiuhai, et al. "Blip3-o: A family of fully open unified multimodal models—architecture, training and dataset." arXiv preprint arXiv:2505.09568 (2025).

---

W2/Q2: Quantitative results or qualitative examples demonstrating the model's ability to handle complex compositional prompts that require precise spatial reasoning and object attribute binding

Thank you very much for this question regarding the model’s ability to handle complex prompts. First, we would like to apologize that, due to the NeurIPS rebuttal policy, we are not allowed to upload any qualitative examples or visualizations. However, we believe the benchmarks used in our evaluation on benchmarks including GenEval and DPGBench already include prompts that assess various capabilities such as object count, spatial and non-spatial relationships, colors, shapes, textures, and attribute binding between objects and their properties.

We compared our method (Bifrost-1) with BLIP3-o on several out-of-distribution prompts involving rare color–attribute bindings, spatial relationships, and object counts. These include examples such as: “A photo of a blue pizza and a yellow baseball glove”, “A photo of four giraffes”, “A photo of a skateboard above a person”, “A purple computer keyboard and a red chair”. Our results demonstrate that Bifrost-1 generates more accurate and faithful images compared to BLIP3-o. We promise to include additional visualizations and comparisons with such complex prompts in our camera-ready version.

Additionally, we would like to highlight that due to our design choice of keeping the MLLM fully frozen, the model is able to preserve its strong reasoning capabilities for understanding complex prompts. For example, given a prompt requiring implicit reasoning such as “The national flag of the country where Yellowstone National Park is located,” the MLLM can infer that the intended flag is that of the United States. It can then rewrite the prompt as “The national flag of the United States” before passing it to the visual generation branch. This ability to transform complex or implicit prompts into explicit and visual-friendly ones greatly enhances the model’s overall effectiveness in handling sophisticated user instructions.

---

W3/Q3: Inference speed for autoregressive masking process

Thank you very much for raising this valuable point regarding the inference speed of our model, particularly the autoregressive masking process. We agree that inference speed is an important consideration for practical applications, and that balancing efficiency and accuracy is a critical design trade-off. Below, we first outline the details of our inference procedure and then present new quantitative results using different numbers of decoding steps in the MLLM.

• Detailed inference procedure: During inference, we begin with a fully masked 2D token grid, which is fed into the MLLM’s vision generation branch. The accompanying text condition is provided to the frozen MLLM branch to guide the visual token prediction. Prior to decoding, we randomly sample a generation order over the image patch tokens, which determines the sequence in which masked tokens are decoded into clean ones. Following MaskGIT [2], we adopt a cosine schedule to determine how many tokens are unmasked at each decoding step. This schedule, as visualized in Figure 2 of the MaskGIT paper, allocates fewer unmasked tokens in the early stages (when uncertainty is high), and progressively more in later steps as more tokens become available. In our default implementation, we set the number of decoding steps equal to the number of visual tokens for simplicity.

• Single step decoding v.s. multistep decoding: In Table 1 of our main paper, we present an ablation study in which patch-level CLIP latents are replaced by 2D learnable query tokens, predicted in a single forward pass. This setup effectively performs a single-step autoregressive prediction and leads to significantly worse generation quality, with an FID of 118.69, compared to our default approach which achieves 25.77.

• Quantitative Results with Varying Decoding Steps: To further assess the effect of decoding steps, we conducted experiments with varying numbers of decoding passes on ImageNet. The results, summarized in the table below, show image quality metrics including FID, sFID, and IS. As demonstrated in the table below, our method remains robust as long as the number of decoding steps is greater than 8. There is, however, a clear trade-off between inference speed and image quality when using fewer decoding steps. Importantly, we also highlight that the MLLM decoding time is significantly smaller than the runtime of the diffusion-based image generation model. For instance, FLUX, a 12B parameter model, takes 14.79 seconds to generate a single image with the default 28 denoising steps. Therefore, the MLLM decoding time is not a major bottleneck. From a practical standpoint, users can flexibly select the number of decoding steps based on their specific priorities—whether they favor faster inference or higher image quality for their particular application.

[2] Chang, Huiwen, et al. "Maskgit: Masked generative image transformer." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

---

Dear Reviewer NRRC,

Once again, we sincerely thank you for all the comments and valuable feedback. We believe that we have addressed all the questions in depth, in particular: (1) additional experimental results comparing Bifrost-1 with other T2I models on standard quantitative metrics, (2) evaluation of the model’s ability to handle complex prompts beyond GenEval and DPGBench, and (3) inference speed of the autoregressive masking process, including experiments with fewer decoding steps. If the reviewer has any additional questions, please feel free to let us know, and we would be more than happy to address them.

Dear Reviewer NRRC,

As today marks the end of the author-reviewer discussion period, we would like to express our heartfelt thanks for your valuable feedback and thoughtful suggestions throughout the rebuttal process.

We sincerely appreciate your recognition of our work, particularly the novel and elegant bridging mechanism using patch-level CLIP latents to connect MLLMs and diffusion models, the training efficiency achieved by freezing the main backbones and updating only lightweight modules, and the preservation of the MLLM’s original understanding and reasoning capabilities.

If you have any remaining questions or concerns, please don’t hesitate to let us know, and we would be more than happy to address them by the end of today.

Thank you again for your time and effort!