Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads

Thank you for your attentive comments! We are glad you thought our paper was well-written and easy to follow. We address your concerns point by point below.

W1: Orthus does not perform exceptionally well in multimodal comprehension, which may be limited by its relatively weak vision embedding module.

Yes. The features encoded by VAE are suited for visual generation, but they can be suboptimal for multimodal comprehension compared to CLIP-ViT due to the lack of semantic information. However, in this paper, we focus on building a unified model capable of interleaved image-text understanding and generation, where images need to be understood and generated simultaneously. Therefore, we embed images using VAE instead of CLIP-ViT. A possible improvement is to incorporate semantic loss during the training of the VAE. Additionally, we train solely on the LlaVa-v1.5-mix-665k dataset for multimodal comprehension currently, demonstrating the advantages of the improved soft vision embedding module and the continuous treatment of visual signals. In the future, we plan to incorporate more high-quality visual understanding data for further improvement.

W2: The authors build Orthus based on Chameleon and substitute the VQ operation with a soft alternative and only tune the parameters of vision embedding module and diffusion head. Have the author tried using a pre-trained multimodal LLM based on a continuous ViT representation, with an added diffusion head to achieve unified multimodal understanding and generation?

Thank you for this interesting suggestion. It is indeed possible to train an additional diffusion head to predict representations encoded by CLIP-ViT autoregressively. However, decoding features encoded by CLIP-ViT back into the original image is challenging. Previous work like VQKD[1] has attempted this but the reconstructed images suffered from significant blurring and an evident loss of high-frequency details, which is attributed to the emphasis on encoding semantic information at the expense of fine-grained details.

Alternatively, prior studies [2] have successfully reconstructed images using a CLIP encoder and a diffusion-based DiT decoder. However, this design introduces redundancy by modeling the correlation between image patches both in the transformer backbone and diffusion decoder, and the inclusion of two separate diffusion processes significantly slows down image generation.

[1] Beit v2: Masked image modeling with vector-quantized visual tokenizers, arXiv:2208.06366.

[2] 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities, NIPS 2024

Question: Why does unified training perform better than understanding training only on the multimodal comprehension benchmark as listed in Tab 4? This seems rarely seen in unified multimodal understanding and generation work. For example, the Emu series conducts separate instruction tuning for understanding and generation.

The synergy may stem from unified cross-modal learning, which renders the characterization of the correlation between modalities. Moreover, both generation and understanding tasks contribute to the alignment of text and image in a shared representation space. On the other hand, previous work DreamLLM [1] also observes a synergy between generation and understanding. It suggests that training LLMs with stronger visual comprehension capabilities leads to more fine-grained encoding of text, which in turn improves text-to-image tasks.

[1] DreamLLM: Synergistic Multimodal Comprehension and Creation, ICLR 2024.

We hope these reclarifications and explanations of our method in response to your initial concerns can convince you to increase your score.

We sincerely thank the reviewer for the time to read our paper. We address your feedback point by point below.

W1: Overclaim of lossless.

By "lossless", we primarily emphasize the continuous treatment of visual signals compared to discrete tokens. We will clarify this in the revised version of the paper.

W2&Q1: Training efficiency.

Pretraining a unified multimodal model from scratch is highly computationally intensive. For example, Show-o-1.3B-base uses 35M image-text pairs and requires training on 48 A100 GPUs for 500k steps, consuming thousands of GPU hours. In contrast, through adaptation of the pre-trained Chameleon, Orthus-7B-base only requires 72 GPU hours, showcasing its super efficiency.

W3&Q3: Quantitative evaluations for interleaved image-text generation. Thank you for your suggestion. For quantitative analysis, we use GPT-4V to compare the storybook generation quality of Orthus and MM-Interleaved[1] and report the win rate of Orthus in the table below. Results show that Orthus excels in generating logically coherent interleaved image-text with high relevance and we will add this table in the revised version.

[1] MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer

Here's the prompt we use for evaluation:

"Please act as an impartial judge and evaluate the quality of the generation story contents provided by two AI assistants. Your job is to evaluate which assistant's generation is better. Your evaluation should consider image quality and consistency. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. After providing your explanation, output your final verdict by strictly following this format: \"[[A]]\" if assistant A is better, \"[[B]]\" if assistant B is better."

W4&Q2: Temperature sensitivity and noise management in the diffusion head.

To analyze the sensitivity of temperature, we ablate different choices and compare their performance in the table below. Results indicate that the model performance is robust to variations in this hyperparameter. We adopt the typical DDPM training SNR schedule[1] in the diffusion head without any tuning. More studies on these will be added to the revision.

[1] Denoising Diffusion Probabilistic Models, NIPS 2020.

S3: Detailed information on the dataset selection.

The details of training data can be found in line322, line355-358, and line622.

S4: Explain for d=30.

This is the model depth of the baseline method SD3.

Q4: Robustness in imbalanced or partially missing interleaved inputs.

Orthus demonstrates generalization ability for unseen interleaved inputs. As shown in Figure 3, it successfully completes the task even when provided with unseen interleaved formats during training.

Q5: Potential limitations.

Orthus exhibits slightly lower image generation efficiency than Chameleon due to the introduction of the diffusion head. Potential solutions include adopting a faster diffusion sampler or consistency distillation.

Essential References Not Discussed: Latest benchmarks including MMIE and OpenING.

Thanks for your suggestion. We primarily focus on fine-tuning Orthus on downstream domain-specific tasks such as image editing and storybook generation to demonstrate its effectiveness in modeling interleaved image-text. We also finetune Orthus on the open-source WebSight dataset to generate webpages (HTML code equipped with images) based on text prompts. Here is an anonymous link showcasing results on this downstream application for your reference. Regarding benchmarks such as MMIE and OpenING, they primarily evaluate interleaved image-text understanding and generation capabilities in more general domains, which are not our main focus. Nevertheless, we evaluate Orthus on MMIE and present the results in the table below.

Despite demonstrating reasonable performance, Orthus sometimes generates text-only responses or replies with "Sorry, I cannot assist with that." due to the lack of instruction tuning or downstream SFT. We plan to finetune Orthus with more diverse and complex interleaved datasets (e.g., MMC4) for further evaluation afterward. Moreover, OpenING evaluates interleaved generation methods through pair-wise comparisons. Since other models' results are not yet open-source, evaluation is infeasible now. We will conduct tests once they are released.

Given these improvements and additional experiments to your initial concerns, we hope you would like to raise your score.

Thank you for the positive feedback and useful suggestions! We are glad you thought our proposed design achieved impressive results. We address your concerns point by point below.

W1&Q1: Numerical comparisons on the inference latencies of the proposed approach and other baseline methods.

Thanks for your suggestion. We estimate the practical inference time (sec/image) for generating images at the resolution of 256 with Orthus-7B and Chameleon-7B under the hugging face inference framework. Transfusion-7B employs bidirectional attention and functions as a DiT for image generation. This design prevents the use of kv-cache optimization, resulting in high latency at each diffusion sampling step. However, when the number of sampling steps is significantly lower than the number of generated image tokens, acceleration over AR frameworks could be possible. We will test the exact inference time of Transfusion-7B and provide a detailed comparison once it is open-sourced.

As shown in the table above, Orthus exhibits slightly higher latency than Chameleon, due to the diffusion head (41M parameters) requiring ~0.001s per forward pass. However, this latency can be significantly reduced by decreasing the number of forward passes with advanced diffusion samplers or consistency distillation (requiring only 1–4 steps for sampling [1]) and by reducing the per-step computation cost with a more lightweight MLP. It is also worth noting that the transformer backbone of Orthus and Chameleon shares the same architecture as LLaMA, which is already supported by the vLLM serving framework (achieving up to 1k tokens/s, enabling image generation in less than 1s). This allows for further acceleration of inference speed. We will include these analyses in the revised version and will actively implement these improvements in the future.

[1]Song, Yang, et al. Consistency Models. ICML 2023.

W2&Q2: The ability to scale to higher resolution.

Thanks for pointing this out. Currently, Orthus supports image generation up to a resolution of 512, which is already advanced among unified multimodal models (e.g., 256 for Transfusion, 256 and 512 for Show-o, and 384 for Janus).

For even higher resolutions, such as 1024×1024, we have checked that our used VAE is capable of encoding and decoding them. Thus, the main challenge lies in the increased sequence length required for higher-resolution images. To address this, we can interpolate the RoPE positional embeddings of our transformer, following [1], and fine-tune the model to extend its capability for processing and generating longer sequences. We can compress the sequence length in attention computation with KV token compression. Alternatively, we can use another VAE with a higher spatial compression rate, such as a partitioned VAE[2], to reduce sequence length. We will explore this in future work.

[1] Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis. CVPR 2025.

[2] Diffusion-4K: Ultra-High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2503.18352.

Q3: Following fine-tuning for interleaved text-image generation tasks (e.g., image editing, storytelling), is it possible for the model’s visual understanding and text-to-image generation performance to remain at the same level as before fine-tuning?

It is challenging to maintain the same level of performance after downstream SFT due to the common issue of catastrophic forgetting[1,2]. We conduct a toy experiment to mitigate this by incorporating the original instruction-tuning data during storytelling finetuning, yet still observe a performance drop of approximately 10% on the POPE benchmark. This can possibly be alleviated by better data mixture and regularization strategies.

[1] An empirical investigation of catastrophic forgetting in gradient-based neural networks. ICLR 2014.

[2] Overcoming catastrophic forgetting in neural networks. PNAS 2017.

Given these clarifications and improvements to your initial concerns, we hope for your reconsideration in raising your score.

Thank you for the positive feedback and useful suggestions! We address your concerns point by point below.

Weakness: The idea is relatively not novel enough, just combining diffusion head and lm head, which shows limited insight into unified image understanding and generation.

• We would like to emphasize that our combination of diffusion head and lm head leads to a unified model for flexible interleaved image-text modeling, mitigating the issues of existing fully AR and AR-diffusion mixed models. Such abilities are important for downstream tasks like in-context editing and storybook generation, as proven by the recent GPT-4o, but have been inadequately explored in research community.
• The incorporation of diffusion head enables the processing of continuous visual signals. And, experiment results in Table 2&3 show that lossless continuous visual signal benefits both image understanding and generation. Namely, we yield the insight that the diffusion head can be essential when we need a unified multimodal generation model.

Missing baselines: Missing VILA-U and Janus as baselines.

Sorry for the missing comparisons and we include them below:

• We first clarify that VILA-U and Janus focus solely on image understanding and generation, lacking support for interleaved image-text generation, where Orthus demonstrates strong capabilities. Additionally, Janus decouples visual understanding and generation by using separate encoders, limiting its flexibility in handling interleaved data—where images must be generated and understood simultaneously.
• We add a comparison with VILA-U and Janus on visual generation and understanding in table below. As shown, Orthus generates images with higher human preference scores at a resolution of 512. For visual understanding, while Orthus still lags, this may be attributed to training solely on the LlaVa-v1.5-mix-665k dataset, whereas the baselines leverage much more high-quality instruction-tuning data [1,2,3]. We plan to incorporate these datasets to further enhance performance.

We will include these comparisons in the revised version. Thank you for your valuable suggestions.

[1] Kvqa: Knowledge-aware visual question answering. AAAI 2019.

[2] Llava-onevision: Easy visual task transfer. TMLR 2025.

[3] Screenqa: Large-scale question-answer pairs over mobile app screenshots. arXiv:2209.08199.

Question: In the ablation study, are the data all the same for understanding-only and generation-only tasks compared with unified training?

Yes, for understanding-only and generation-only tasks, we use the same understanding and generation data as in the unified training setting.

Given these reclarifications and improvements in response to your initial concerns, we hope you would like to raise your score.