Show-o2: Improved Native Unified Multimodal Models

1. Why use flow matching rather than DDPM or autoregressive for generation?

We choose flow matching over DDPM because numerous studies [1,2] have demonstrated that flow matching converges faster and achieves better performance than DDPM. Additionally, flow matching has become the mainstream modeling paradigm for video generation [2,3,4] compared to pure autoregressive modeling. To further support this, we supplement the experimental results showing that flow matching converges more rapidly than DDPM on ImageNet.

Based on these observations and experiments, we adopt flow matching for image and video generation.

[1] SD3: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. 2024.
[2] Goku: Flow Based Video Generative Foundation Models, CVPR 2025.
[3] HunyuanVideo: A Systematic Framework For Large Video Generative Models. 2024.
[4] Wan: Open and Advanced Large-Scale Video Generative Models. 2024.

2. The advantages and comparisons of 3D VAE encoder over other unified tokenizers.

The inherent advantages of the 3D VAE lie in its flexible support for both image and video encoding and decoding, as well as its ability to achieve high-quality reconstruction for both modalities. To push the upper bound of unified multimodal models in image and video generation, we adopt the 3D VAE—a technique widely used in state-of-the-art commercial and open-source video generation models such as WAN2.1.

While UniTok, UniToken, and TokenFlow offer elegant unified discrete token representations for image modalities, they do not extend their solutions to video modalities. Moreover, discrete token-based video tokenizers often lag behind their continuous counterparts, limiting the potential performance ceiling.

Based on these considerations, we propose constructing unified visual representations within the 3D VAE latent space, providing a robust and effective approach to unifying both image and video modalities.

Thanks for pointing it out. We’d like to add these discussions to the related work section of our paper.

We also supplement our models’ evaluation on video generation and understanding benchmarks below (**We omit the reference information and only select some of the metrics to present due to the limited space).

3. Data filtering strategies on interleaved datasets.

We did simple data filtering only based on the image resolution, and whether the image link is legacy to filter out those sequences with missing image files. However, we unfortunately find that the data quality is still relatively low, so we add two datasets, including CoMM and VIST as high-quality downstream data to further fine-tune the model. We will add these details to our paper.

4. Related work section.

Thanks for your suggestion. We plan to add more descriptive comparisons between our method and other related works and add our response to question 2 to the related work section.

5. Do you use the same image-text data for image understanding task and image generation task in stage 1?

Yes. In stage 1, we sample from the same data source to construct sequences for both image understanding and generation learning.

6. OCR and Doc understanding ability. What is the performance of unified model on textvqa, chartqa, docvqa, infovqa datasets?

Since our model was trained using a fixed image resolution of 432×432 (resulting in 729 tokens per image), we make a fair comparison between LLaVA-OneVision and our models under the same inference settings.

As shown above, our model achieves comparable performance to LLaVA-OV-7B on DocVQA, InfoVQA, and TextVQA validation sets. However, it underperforms on the ChartQA task. We believe this is due to insufficient training data for chart types during our semantic layer distillation, which limits the model’s ability to maintain such fine-grained information effectively. We are confident that incorporating more OCR and document data into the distillation process will further enhance the unified model’s OCR and document understanding capabilities.

Additionally, an interesting observation is that although our model was trained with a fixed image resolution, increasing the number of image tokens using the AnyRes strategy during inference consistently improves performance.

These results demonstrate that employing high-resolution images during training would potentially further improve the OCR and document understanding capabilities of our models.

1. Video benchmarks.

We supplement the quantitative evaluation of video understanding as follows, as well as the text-to-video generation on VBench:

We follow the settings of LLaVA-OneVision to enhance our models with video understanding capabilities. As shown above, our model demonstrates comparable performance to LLaVA-OneVision and outperforms VideoLLaMA2 as well as the much larger LLaVA-N-Video-32B on several metrics.

As shown above, our models can achieve competitive video generation performance compared to specialized video generation methods, and achieve better performance than unified multimodal models like Emu3 and HaploOmni.

2. Regarding text corpus.

Publicly available text corpora, such as RefinedWeb, tend to be of relatively low quality and insufficient for reproducing strong large language models such as the Qwen-2.5, which rely on extremely high-quality and diverse proprietary text data that is prohibitively expensive for most institutions to collect. Consequently, unified co-training with image-text pairs and such low-quality publicly available text corpora may potentially degrade the original language knowledge. We provide the evaluation results on text-only benchmarks using the lm-evaluation-harness tool [1] below for demonstration.

Directly co-training with all model parameters in one stage using both image-text pairs and textual data, such as RefinedWeb, leads to significant performance degradation, underscoring the necessity of our two-stage training approach when high-quality corpora are unavailable.

[1] github repository: lm-evaluation-harness

3. Synergies.

Thank you for sharing your valuable comments. We are also interested in exploring the mutual effects within unified learning. Our observations and insights can be categorized into three aspects:

i) Representation Level: Typically, multimodal understanding and visual generation rely on semantic and low-level (fine-grained) features, respectively. Based on our pilot study, we found that constructing unified visual representations by combining these two types of features leads to improvements in both understanding and generation, as demonstrated in the tables below.

ii) Objective Level: We conducted controlled experiments to investigate whether combining understanding and generation objectives leads to conflicts or mutual benefits. Our findings indicate no conflicts in multimodal understanding, as the model trained with unified learning achieved nearly identical performance on understanding benchmarks compared to the model trained solely with the understanding loss. Encouragingly, for visual generation, we observed improvements on the Dense Prompt Graph Benchmark (DPG-Bench), particularly in metrics related to color and relational accuracy, as shown below.

iii) Complex Task Level: With the advancement of multimodal chain-of-thought techniques, there is a growing demand for generating intermediate or auxiliary images to assist in solving complex problems such as mazes or mathematical reasoning. We believe that the native unified multimodal model is well-suited to meet these challenges.

We believe our models provide a strong baseline, and by open-sourcing our code, we hope to inspire further exploration in these three areas to continuously explore the mutual benefits and push the boundaries of unified multimodal models.

4. Comparison to LLaVA-OneVision.

Thanks for your suggestion. We supplement our latest comparison below:

As shown above, for most multimodal understanding metrics, our model can achieve competitive performance with LLaVA-OneVision. We will add these results to the main table.

5. Flow matching vs (discrete) diffusion.

We choose flow matching over (discrete) diffusion because numerous studies [1,2] have demonstrated that flow matching converges faster and achieves better performance than diffusion. Additionally, flow matching has become the mainstream modeling paradigm for video generation [2,3,4]. Furthermore, continuous 3D VAE has better video reconstruction capability than discrete ones. So, we adopt flow matching with a continuous 3D causal VAE. We provide proof-of-concept experimental results showing that flow matching converges faster than diffusion on ImageNet.

[1] SD3: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
[2] Goku: Flow Based Video Generative Foundation Models
[3] HunyuanVideo: A Systematic Framework For Large Video Generative Models
[4] Wan: Open and Advanced Large-Scale Video Generative Models

6. Visual comparisons.

Thanks for your suggestion. We will include visual comparisons among different unified multimodal models.

Apologies for the earlier mistakes. We have revised the table as shown above, indicating that improvements in metrics (DPG-Bench) related to color and relation accuracy have been observed in unified learning with the understanding loss.

1. Missing details for the ablation study settings on spatial fusion.

Sorry for the missing details. We reorganized the table as follows for clarity:

For multimodal understanding, the baseline model utilizes only semantic features (known to be more effective for understanding) extracted by the semantic layers, while for visual generation, the baseline relies solely on VAE features (recognized as crucial for generation) processed through the projector. As demonstrated above, combining these two types of features into a unified visual representation leads to improved performance across both multimodal understanding and generation metrics.

Our primary goal is to construct a unified visual representation. At a minimum, we expect not to degrade the original performance, and surprisingly, the combination yields mutually beneficial improvements, even if they are not very significant. I believe this observation is very valuable and will inspire further research to explore this property in greater depth.

We will update Table 6 to avoid misunderstanding.

2. Video generation quantitative results.

We supplement the quantitative evaluation of text-to-video generation and image-to-video generation on the widely adopted VBench as follows, as well as the video understanding evaluation (We omit the reference information and only select some of the metrics to present due to the limited space):

As shown above, our models can achieve competitive video generation performance compared to specialized video generation models such as CogVideo-X and Gen-3, and achieve decent performance better than unified multimodal models like Emu3, VILA-U, and HaploOmni. Besides, following the LLaVA-OneVision training settings, we also attach the video understanding performance. From the table, we can see our models can achieve competitive performance compared to those understanding only models in terms of Activitynet-QA, Next-QA, PercepTest, and VideoMME metrics.

3. Camera motion and human body motion results.

The quality and motion dynamics of video generation heavily depend on the datasets used. In the submitted version, our model was fine-tuned on a relatively small set of publicly available data, which may have limited its performance. However, as shown in the tables above, our updated model achieves competitive results across multiple metrics, demonstrating the effectiveness of our proposed framework. Additionally, the above supplementary text-to-video evaluation includes a metric for ‘human action,’ which assesses the quality of human body action. According to the table, our updated model outperforms video generation models such as Step-T2V and Kling regarding this metric.

4. Language-only benchmarks.

We evaluated the Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, the model co-trained with both image-text and text-only data i.e., RefinedWeb, in one stage full-training, and our own 1.5B and 7B models using the lm-evaluation-harness tool [1] under identical evaluation settings. The results are presented below.

It can be observed that our models effectively preserve language knowledge and achieve performance comparable to the original Qwen2.5-1.5B and Qwen2.5-7B Instruct models. In contrast, directly co-training in one stage using textual data such as RefinedWeb leads to significant performance degradation, underscoring the necessity of our two-stage training approach when high-quality corpora are unavailable.

[1] github repository: lm-evaluation-harness

5. Comparison with very recent works.

Thanks for the suggestion. We attach the comparison below.

One should notice that these recent works may leverage larger training datasets, bigger models, and powerful pretrained vision-language models such as Qwen2.5-VL, as well as pretrained text-to-image generation models like SANA and FLUX. In contrast, our model builds upon a purely language-pretrained models and shows promising capabilities and potential with a limited amount of training data.

Thank you for recognizing our paper as solid and sound, and I apologize for any misunderstandings.

Regarding the video generation, we have added an additional 1.5 million high-quality samples along with the originally used OpenVid-1M dataset. The performance of text-to-video and image-to-video generation reported in our response reflects this update. Compared to other video generation models that may utilize hundreds of millions of video samples, our dataset scale remains very small. However, this incorporation of high-quality data significantly demonstrates the effectiveness and potential of our proposed framework, helping to achieve competitive performance and enabling a relatively fair comparison with specialized video generation models that have larger model sizes and are trained on more extensive datasets. We will include these results and discussions in our paper.

We would like to clarify that the term "updated model" in the "Comparison with very recent work" table refers to our re-trained 7B model using cleaned-up code prepared for future open-source release. This re-training follows the same training settings as the submission version. As we found that there are minor variations in some metrics (such as MME 1623.5 --> 1620.5 and SEED-Bench 69.4 --> 69.8) when evaluating this model, compared to the original submission (but the overall performance remains largely consistent), we use the term "updated model" to avoid confusion regarding these slight differences in our response. We confirm that “the updated model” means the re-training with the same settings rather than the other update, ensuring consistency with the submission version.

We directly evaluate the trained models to obtain text-only performance, which directly demonstrates the excellent capability of our models to retain language knowledge.

I hope this clarifies the confusion.

1. Novelty.

We argue that our work presents significant innovations and contributions, which are also consistently acknowledged by the other three reviewers from different aspects: i) “use a 3D causal VAE to jointly model visual information for both generation and understanding, which is novel.” commented by Reviewer sCtb; ii) “The unified visual representation is novel and intuitive” demonstrated by Reviewer hZso; iii) “The dual-path fusion mechanism is an intelligent way to create a visual representation” noted by Reviewer H9zV.

In addition, we argue that integrating 3D causal VAE, flow matching, and omni-attention is a logical exploration of more flexible architectural designs that unify a broader range of vision modalities, including both images and videos, in which video modalities remain relatively underexplored in the unified model literature. Further, our proposed two-stage training recipe alleviates the need for large-scale, high-quality text corpora, which are prohibitively expensive for most institutions to collect.

While our work may not represent a groundbreaking conceptual breakthrough on the scale of GPT or Diffusion, it provides significant insights and strong experimental results that pave the way for future research in unifying text, images, and videos. We will open-source our training code and model weights to establish a reproducible platform and inspire further exploration in this area.

2. Ablation study on dual-path fusion.
We employ a 1B-parameter LLM as the base model to conduct an efficient ablation study. All experiments are carried out under consistent training settings and datasets. The experimental results are shown in Table 6 and presented below:

For multimodal understanding, the baseline model utilizes only semantic features (known to be more effective for understanding) extracted by the semantic layers, while for visual generation, the baseline relies solely on VAE features (recognized as crucial for generation) processed through the projector. As demonstrated above, combining these two types of features into a unified visual representation leads to improved performance across both multimodal understanding and generation metrics. This highlights the effectiveness and necessity of our proposed fusion strategy.

3. Flow matching vs DDPM.
We choose flow matching over DDPM because numerous studies [1,2] have demonstrated that flow matching converges faster and achieves better performance. Additionally, flow matching has become the mainstream modeling paradigm for video generation [2,3,4]. To further validate this, we present our proof-of-concept experimental results on the ImageNet dataset, which show that flow matching converges more rapidly than DDPM.

[1] SD3: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. 2024.
[2] Goku: Flow Based Video Generative Foundation Models, CVPR 2025.
[3] HunyuanVideo: A Systematic Framework For Large Video Generative Models. 2024.
[4] Wan: Open and Advanced Large-Scale Video Generative Models. 2024.

4. Effectiveness of two-stage training.
We evaluate the Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, the model co-trained with both image-text pairs and text-only data i.e., RefinedWeb, in single-stage joint training, and our own 1.5B and 7B models in two-stage training using the lm-evaluation-harness tool [1] under identical evaluation settings. The results are presented below.

Our models trained in two-stage recipe effectively preserve language knowledge and achieve performance comparable to the original Qwen2.5 Instruct models. In contrast, directly end-to-end training in a single stage leads to significant performance degradation, underscoring the necessity of our two-stage training when high-quality corpora are unavailable.

[1] github repository: lm-evaluation-harness

5. Video benchmarks.
We supplement the quantitative evaluation of text-to-video generation and image-to-video generation on the widely adopted VBench as follows, as well as the video understanding evaluation: (We omit the reference information and only select some of the models and metrics due to the limited space. Complete ones are in our responses to the other reviewers.)

As shown above, our models can achieve competitive video generation performance compared to specialized video generation methods, and achieve performance better than unified multimodal models like Emu3. Besides, we also attach the video understanding performance. From the table, we can see that our models achieve competitive performance compared to those understanding only models in terms of ActivityNet-QA, Next-QA, Perceptual-Test, and Video-Meme metrics.

6. Interleaved generation evaluation.
As demonstrated by our experimental results, this work focuses more on demonstrating the flexible support across text, images, and videos. Due to the limited availability of high-quality interleaved image-text data and the incompleteness of existing evaluation benchmarks, we have not conducted comprehensive evaluations on mixed-modality generation. Instead, this work primarily serves to showcase the potential of our framework in handling mixed-modality generation. We plan to release the source code to the research community to facilitate further exploration in the future.

7. Error bars.
Due to time constraints, we conducted the second-stage experiments three times to estimate the standard deviation of our models' performance. The results are summarized below:

These results demonstrate that our model's performance is consistently stable across all metrics, and the observed standard deviations do not affect our advantage over previous state-of-the-art methods such as Janus-Pro. We will add these statistical analyses to the paper.

8. Ethical Concerns
We follow prior work in collecting data from publicly available datasets for research purposes. Our internal data undergoes some filtering to ensure the exclusion of unsafe content, user information, and copyright.

Dear Reviewer RCaP,

Thank you for your thorough review and valuable feedback. We have carefully addressed your comments with extensive experimental results. As the discussion deadline approaches, please let us know if any points require further clarification.

With sincere appreciation,
Authors of Paper 5552