MIO: A Foundation Model on Multimodal Tokens

Thanks for your kind reviews. We provide our clarifications below:

Q1: qualitative comparisons between MIO and Emu for image generation.

A1: We have provided the qualitative comparison in the supplemental materials: MIO supplemental materials\IMAGES-qualitative comparisions bwteen MIO and Emu for image generation.zip. The comparisons demonstrate the superior image generation abilities of our model.

---

Q2: how does the model output fix-length image tokens, speech tokens, and video tokens? do authors use any strategies to promise the multimodal tokens within their vocabulary?

A2: During training, we use special tokens such as <image>, </image>, <spch>, and </spch> to define the boundaries for each modality data (images, video frames, and speeches). Specifically:

• <image> and </image>: These tokens mark the beginning and end of an image or a video frame.
• <spch> and </spch>: These tokens are used to define the start and end of a speech.

During training, these tokens help the model distinguish between different modalities and learn the patterns, such as how many tokens each image or speech should be represented by. For example, the model may learn that an image is typically represented by 32 tokens, and that the number of speech tokens for each codebook should be consistent. These patterns regarding the token length can be automatically learned by the models.

Moreover, these multimodal tokens are added directly to the model's vocabulary. The embedding matrix and language modeling head are extended to accommodate the increased vocabulary size, supporting these new multimodal tokens.

So during inference, we don’t use any special trick to promise the multimodal tokens. No constraint is leveraged for generation.

---

Q3: The reason for better image understanding? Is DIDO better than CICO considering that MIO > Emu in many benchmarks?

A3: The better performance of MIO on image understanding with fewer parameters can be attributed to more semantically aligned image tokens and cleaner, richer training data.

Although MIO outperforms Emu in several benchmarks, we cannot conclusively claim that DIDO is better than CICO in all scenarios. The VQ-based tokenization in DIDO may incur some information loss, which could affect performance in tasks where fine-grained details matter. It's also worth noting that the latest version of Emu3, which adopts DIDO instead of CICO, has achieved strong results, indicating DIDO's strong potential. However, in image understanding tasks, CI-based approaches still tend to perform better, while in image generation, CO might struggles with the challenges of the mean square error (MSE) loss used for training continuous output representations, as well as with the uni-modal assumption of the Gaussian distribution in the MSE loss, as discussed in our related works.

[1] Emu3: Next-Token Prediction is All You Need

---

Q4: only video keyframe generation, not a real video.

A4: Video keyframe generation is also a vital issue because the keyframes can capture the most important visual moments of a video, serving as a skeleton for further full video generation with richer semantics. Even the most advanced full video generation models face difficulties in planning and generating keyframes, which are the critical points that define the overall structure and narrative of a (long) video. Compared with full video generation, which focuses on the temporal dynamics and frame density, video keyframe generation focuses on generating the representative frames to establish coherent and meaningful transitions between key moments in the video. These keyframes provide a high-level outline of the video’s content, allowing for easier synthesis of more complex narratives while maintaining semantic consistency across frames. We emphasize this capability over full video generation, as it allows for more effective control and planning of video content.

---

Q5: the difference between MIO and NeXT-GPT

A5: The differences between NeXT-GPT and MIO can be summarized as follows:

1. Approach: NeXT-GPT uses a CICO (continuous-in, continuous-out) method, while MIO adopts a more flexible DIDO (discrete-in, discrete-out) approach.
2. Consistency of I/O: In NeXT-GPT, the input and output representations for the same image/audio data are not consistent, whereas MIO ensures consistency across these representations.
3. Multimodal Interleaving: NeXT-GPT does not support multimodal interleaved output, whereas MIO can handle the interleaved generation of multiple modalities more effectively due to the I/O consistency feature and the multimodal interleaved training stage.

Q6: Why is Emu2 missing in the results?

A6: The Emu2 results have been added in the following tables:

• image understanding:

*Note that the reported results of Emu2-Base are evaluated in 16-shot setting.

• image generation

• video understanding

We find that although MIO-Instruct underperforms Emu2 for image understanding, it achieves competitive results in video understanding and image generation with a significantly smaller model size. Moreover, the instruction-tuned variants of Emu2, namely, Emu2-Chat and Emu2-Gen, separate their abilities for image understanding and image generation. That is, Emu2-Chat doesn’t support image generation, and Emu2-Gen doesn’t support image understanding. While our MIO-Instruct integrates the two directions in one single model, even with the support for speech modality which is heterogeneous to the vision modalities.

---

Thanks again for your valuable feedback and suggestions. We are looking forward to your further feedback.

Hi, Reviewer GVGf,

Thank you for sharing your valuable feedback. As the discussion period approaches its conclusion, we would like to know if our responses have effectively addressed your concerns.

Q7: difficult demos, and cannot hear the audio.

A7: For the demonstrations in Figures 4, 5, and 6, there are three elements for each example: instruction (blue, vertical), input (yellow, horizontal), and output (green, horizontal). We replace the detailed instructions with the task names for simplicity. For example, for the bottom-right example in Figure 4, the task is “visual guideline generation”. The model was asked to generate a visual guideline for “how to make a pizza, from preparing the ingredients to baking it at every step.”. And the model generated the images and texts marked with light green background.

As for the speech demonstrations, we have uploaded the audio files in the supplemental materials. See MIO supplemental materials\MIO_anonymous\generated_speeches. You can also refer to MIO supplemental materials\MIO_anonymous\infer.py for the detailed inference examples.

---

Thanks again for your valuable feedback and interesting ideas. Your comments are the most valuable ones that I have ever seen. Hope our responses can address most of your concerns.

Q5: limited ASR performance, and worse for instruction-tuned variant.

A5:

1. The ASR performance of MIO, as shown in Table 5, is competitive for a general-purpose multimodal foundation model. While models like Whisper and Wav2Vec achieve lower WER values due to their specialization in ASR tasks, MIO-Base achieves a WER of 6.3, outperforming AnyGPT’s 8.5. Both MIO and AnyGPT are designed for multiple modalities, including text, images, speech, and video, but MIO demonstrates superior ASR performance, highlighting its balanced optimization across modalities while maintaining strong speech-to-text capabilities.
2. The SpeechTokenizer employs multiple layers of residual vector quantization (RVQ) to discretize speech, with the first layer specifically incorporating a HuBERT-based distillation to enhance the representation of content and semantic information. Subsequent layers are optimized using a reconstruction loss to extract timbre signals, effectively dividing the tokenization process into a content codebook (first layer) and timbre codebooks (subsequent layers). However, despite this design for content-timbre disentangling, the first layer still suffers from some semantic loss. This is partly due to HuBERT's inherent limitations in capturing the full range of speech semantics and partly because the SpeechTokenizer does not achieve complete content-timbre disentanglement since the later RVQ layers, which are primarily focused on timbre, still retain traces of semantic information, leading to a slight degradation in the content representation provided by the first layer.
3. The instruction-tuned variant of MIO performs worse on ASR (WER 10.3 vs. 6.3 for MIO-Base) primarily due to the shift in task distribution during supervised fine-tuning. In the pretraining stage, ASR-related tasks constitute a significant proportion of the training data, allowing the model to develop robust speech-to-text capabilities. However, during SFT, the inclusion of a broader range of multimodal tasks dilutes the ASR-specific focus. Furthermore, we observed that the instruction-tuned variant tends to extend transcription outputs or generate additional content beyond the transcription instruction. This behavior likely arises from the model's exposure to tasks involving contextual continuation, which inadvertently impacts its ability to adhere strictly to ASR instructions. However, the ASR performance of the instruction-tuned model is still comparable to that of AnyGPT.

---

Q6: more metrics and more benchmarks for TTS:

A6: We have selected two additional benchmarks, LibriSpeech test-clean[1] and GLOBE[2], to evaluate the TTS performance between our model and AnyGPT. For fair comparison, we did not specify the input voice prompt during the evaluation of MIO and AnyGPT. WER and speaker similarity are used as the automatic metrics. The results are shown below:

The results show that MIO performs significantly better than AnyGPT on both WER and speaker similarity across all benchmarks. (Note that AnyGPT-Instruct doesn’t support TTS.)

Additionally, we conduct a human evaluation to assess the speech quality of the outputs from MIO and AnyGPT. In this evaluation, we randomly select 100 samples. Participants are provided with the target speech, the speech generated by AnyGPT, and the speech generated by our model. They are tasked with determining which one sounded more natural and closer to the target speech. Evaluators could choose one of the two generated speeches or indicate that they find them equally natural. Each evaluation is rated by three independent human evaluators, and we report the average scores.

MIO significantly outperforms AnyGPT in the human evaluation, consistent with the results from the automatic evaluation.

[1] Panayotov, Vassil, et al. Librispeech: an asr corpus based on public domain audio books. ICASSP 2015.

[2] Wang, Wenbin, Yang Song, and Sanjay Jha. GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech. Interspeech 2024.

Q3.2: loss curves of each modality for each stage? teacher-forcing ppl or other evaluation (e.g., LLaMA3’s) of the pretrained model? Especially for text-only performance.

A3.2: We find that the losses of different modalities decrease alongside the total loss, so we haven’t logged the losses for each modality to reduce the cost for calculating validation losses. However, we can still provide the training losses for each stage. The loss curves are shown in Figure 3. We observe that when a new data type (i.e., image-text interleaved data) is introduced in stage 2, the training loss exhibits a sudden increase. In contrast, the third pretraining stage, which does not introduce novel data types (i.e., the speech-enhancement stage), demonstrates a smoother transition in training loss.

Despite the fluctuations in loss between stages, which do have some impact on downstream performance during the fluctuation periods, we find that with continued training, the model's loss quickly recovers to its previous convergence level and continues optimizing effectively. As noted in A3.1, this staged strategy is both necessary and effective, making such fluctuations negligible in practice. Additionally, we agree that the proposed one-stage pretraining method with continuous speech ratio warmup could potentially avoid these fluctuations and result in more stable training. However, as mentioned in A3.1, we look forward to exploring this promising idea in our future work, once resource limitations and the need for rigorous experimentation can be addressed.

---

Q4: evaluating speech dialogue performance.

A4: Since the speech-to-speech benchmark of LLaMA-Omni is not open-sourced, we randomly sample some conversations from the moss-002-sft dataset and convert them into speech-to-speech format. Following the evaluation procedures outlined for LLaMA-Omni, we use the content score metric obtained from GPT-4o to assess whether the model's response effectively addresses the user's instructions. The results are as follows:

Note that “s2s” means “speech-to-speech”, while “s2t” and “t2s” denote “speech-to-text” and “text-to-speech”, respectively. Though the content score of MIO is slightly lower than LLaMA-Omni and AnyGPT, both LLaMA-Omni and AnyGPT first generate text replies and then convert these into voice. However, our model, MIO, is capable of directly generating speech responses to speech queries.

*note that LLaMA-Omni is also a submission for ICLR 2025 (id=Submission2959), but we are happy to compare with it.

Q3.1: three-stage pretraining v.s. one-stage pretraining with speech ratio warmup (on a subset)

A3.1: Thanks for your insightful suggestions!! This is a good idea. The proposed one-stage pretraining (i.e., continuous speech ratio warmup) is conceptually similar to our three-stage strategy. Both aim to mitigate the imbalance caused by the extremely large number of speech tokens compared to other modalities by increasing the speech data ratio step by step. The core difference is, our three-stage strategy implements a staircase-style warmup, where the speech data ratio increases suddenly after numerous training steps. In contrast, the proposed one-stage pretraining method uses a continuous ratio warmup, where the speech data ratio increases gradually after each training step.

To validate the effectiveness of the three-stage approach, we compare stage 1, stage 2, and stage 3, with each stage being independent and from scratch with the corresponding data mixing ratio (note that “stage 2” is different from “stage 1+2”). After training for 1K to 2K steps, the experimental results are shown below:

Note that the data composition is in format of “(image-text pair data) : (language-only data) : (image-text interleaved data + video data) : (speech-text pair data)”

We can observe that the image-text alignment by “stage 2”-only improves much slower than by “stage 1”. And there is a risk for very early performance saturation for image-text alignment when directly using the data mixture of stage 2. Moreover, since the image-text paired corpus is much larger and higher-quality than the image-text interleaved corpus, to ensure the pretraining can consume most of the image-text paired data, we exclude the image-text interleaved data in the first stage. And to further unlock the models’ abilities for image-text interleaved generation, we add these data back into the pretraining once the training loss of the stage 1 nearly converges. As for the third stage — speech enhancement pretraining, since the speech tokens are much more enormous, we maximize its data ratio to ensure most of the speech data is consumed once the performance for other modalities during stage 2 is saturated.

In conclusion, such staircase-style warmup is designed to avoid early performance saturation for image-text representation alignment and to ensure that most of the training data for each modality is trained. And this staged strategy is effective, which can further demonstrate that the speech ratio warmup strategy is effective in a staircase-style setting.

We are very interested in the idea of “speech ratio warmup in a continuous setting” and are eager to validate this approach in a future version of our work. We believe this represents a novel idea in pretraining and could potentially merit an entirely new paper. However, in the current version, resource and time limitations prevent us from conducting a fair and comprehensive comparison between the staircase-style and continuous warmup strategies. Even when using a subset of the data, such experiments would require approximately one week of pretraining per setup to produce reliable results. Conducting both strategies simultaneously for a fair comparison would further increase resource demands. Moreover, shorter pretraining durations would likely fail to yield meaningful results, as transitioning to the later stage before the early-stage loss has sufficiently converged would significantly increase the risk of training instability and unreliable findings. We look forward to exploring this promising idea in our future work.

Q1: limited by the tokenizers (esp., SpeechTokenizer, which is inefficient).

A1: We appreciate your insightful feedback regarding the tokenization approach in MIO, especially concerning the inefficiencies of the SpeechTokenizer.

First, we would like to highlight that our approach integrates voice (acoustic signals) into the language model, not just the content (semantic signals). By training not only on the semantic aspects of speech but also on its acoustic properties, our model gets rid of the additional needs of a voice cloning model compared with the previous works such as AnyGPT.

Regarding the inefficiency of the SpeechTokenizer, we acknowledge that the current RVQ-based tokenization with multiple codebooks leads to suboptimal performance, particularly in terms of generation speed (200 Hz). However, more efficient and effective speech tokenizers than SpeechTokenizer, such as WavTokenizer[1] and GLM-4-Voice-Tokenizer[2] are our concurrent works. We plan to replace our speech tokenizer by these more efficient ones in the next version of our model.

[1] WavTokenizer: An Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling

[2] GLM-4-Voice-Tokenizer: https://github.com/THUDM/GLM-4-Voice

---

Q2: text-only benchmarks.

A2: We evaluate our models on MMLU, a well-known language-only benchmark. The results are shown below:

We can observe that our models have superior language-only performance compared with all any-to-any MM-LLM baselines and even surpass LLaMA-1-7B-Base, an advanced pure language model.

Hi, Reviewer Uas7,

We sincerely appreciate your valuable feedback. As the discussion period draws to a close, we kindly ask if you could confirm whether our responses have adequately resolved your concerns.

Q2: not open-source

A2: We have open-sourced both the models and codes, but we cannot provide the links due to the anonymity requirements. You can refer to the attached MIO supplemental materials\MIO_anonymous for the anonymous codes. Unfortunately, we cannot upload our models either in the supplemental materials or in the anonymous github repository due to the size limit.

Q3: capabilities beyond existing methods.

A3: We have shown the advanced abilities of our model, such as visual storytelling, chain of visual thought, visual guideline generation, instructional image editing, etc., in Figure 4. Moreover, our model also supports the trimodal understanding of image+text+speech, as illustrated in Table 8.

---

Thanks again for your valuable comments. Hope our clarifications can address most of your concerns, and we are looking forward to further feedback.

Thanks for your kind feedback. Here are some clarifications:

Q1: speech-enhanced pre-training stage is not a solution but a patch.

A1: The speech-enhanced pretraining stage is designed to address the challenge posed by imbalanced token amount across different modalities, specifically for speech data, by increasing the speech data ratio during the whole training process. We can call our strategy as “staircase-style speech ratio warmup” since the speech data ratio is increased stage by stage. This approach ensures that the model can consume the speech data more effectively without compromising the performance of the other modalities, such as image-text alignment.

To validate the effectiveness of the three-stage approach, we compare stage 1, stage 2, and stage 3, with each stage being independent and from scratch with the corresponding data mixing ratio (note that “stage 2” is different from “stage 1+2”). After training for 1K to 2K steps, the experimental results are shown below:

Note that the data composition is in format of “(image-text pair data) : (language-only data) : (image-text interleaved data + video data) : (speech-text pair data)”

We can observe that the image-text alignment by “stage 2”-only improves much slower than by “stage 1”. And there is a risk for very early performance saturation for image-text alignment when directly using the data mixture of stage 2. Moreover, since the image-text paired corpus is much larger and higher-quality than the image-text interleaved corpus, to ensure the pretraining can consume most of the image-text paired data, we exclude the image-text interleaved data in the first stage. And to further unlock the models’ abilities for image-text interleaved generation, we add these data back into the pretraining once the training loss of the stage 1 nearly converges. As for the third stage — speech enhancement pretraining, since the speech tokens are much more enormous, we maximize its data ratio to ensure most of the speech data is consumed once the performance for other modalities during stage 2 is saturated.

In conclusion, such staircase-style warmup is designed to avoid early performance saturation for image-text representation alignment and to ensure that most of the training data for each modality (esp., speech) is trained. Therefore, our staged stragety is not only a patch, but also a solution.

Hi, Reviewer dfgg,

Thank you for your insightful comments. With the discussion period nearing its end, we would be grateful if you could let us know whether our responses have sufficiently addressed your concerns.

Q4: lack of qualitative and quantitative ablations or analysis for tokenizer selection, data composition, and training setup.

A4: For the image tokenizer selection, we compare three tokenizers qualitatively in Figure 3. We can observe that the other two tokenizers demonstrate too low quality to enable us to leverage comparable quantitative evaluation. For the speech tokenizer, we directly leverage the SpeechTokenizer since it has more token efficiency (200 tokens/seconds) compared with SoundStream[1] (600 tokens/seconds) and HiFiCodec[2] (400 tokens/seconds). Some more efficient speech tokenizers such as WavTokenizer[3] and GLM-4-Voice-Tokenizer[4], are our concurrent works.

For the data composition and the training setup, since investigating numerous data compositions is unaffordable due to high training costs, between each stage (stage1 → stage2, stage2 → stage3), we simply maximize the micro batch size for the data type of interest (i.e., image-text paired data for stage1, image-text interleaved data and video data for stage2, and speech data for stage3) while keep the ratio for other data types reasonable (i.e., mostof the training corpora for each modality should be trained on). We compare stage 1, stage 2, and stage 3, with each stage being independent and from scratch with the corresponding data mixing ratio (note that “stage 2” is different from “stage 1+2”). After training for 1K to 2K steps, the experimental results are shown below:

Note that the data composition is in format of “(image-text pair data) : (language-only data) : (image-text interleaved data + video data) : (speech-text pair data)”

We can observe that the image-text alignment by “stage 2”-only improves much slower than by “stage 1”. And there is a risk for very early performance saturation for image-text alignment when directly using the data mixture of stage 2. Moreover, since the image-text paired corpus is much larger and higher-quality than the image-text interleaved corpus, to ensure the pretraining can consume most of the image-text paired data, we exclude the image-text interleaved data in the first stage. And to further unlock the models’ abilities for image-text interleaved generation, we add these data back into the pretraining once the training loss of the stage 1 nearly converges. As for the third stage — speech enhancement pretraining, since the speech tokens are much more enormous, we maximize its data ratio to ensure most of the speech data is consumed once the performance for other modalities during stage 2 is saturated.

[1] Soundstream: An end-to-end neural audio codec

[2] HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec

[3] WavTokenizer: An Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling

[4] GLM-4-Voice-Tokenizer: https://github.com/THUDM/GLM-4-Voice

---

Thanks again for your kind reviews! We are looking forward to your further feedback.

Q3: limited metrics for image generation (structural integrity, aesthetics, human preferences)

A3: We compute two additional automatic metrics, i.e., SSIM and Aesthetic Predictor v2.5 for the evaluation of structural integrity and aesthetics, respectively. SSIM (Structural Similarity Index Measure) evaluates the perceptual similarity between the generated images and the ground-truth images, focusing on luminance, contrast, and structure, with scores ranging from -1 (dissimilar) to 1 (identical). Aesthetic Predictor V2.5 is a SigLIP-based predictor that evaluates the aesthetics of an image on a scale from 1 to 10 (10 is the best).

In addition, we randomly select 100 image descriptions from MS-COCO test set, and use each model to generate images accordingly for human preference evaluation. We ask 3 annotators to rank 3 images generated by the 3 models: “given the image description, which image is preferred?” The average ranking of MIO’s, AnyGPT’s, and Emu’s generated images are 1.2 (MIO), 2.9 (AnyGPT), 1.9 (Emu). MIO aligns the best with the human preference. The percentage agreement between the three annotators (calculated as the number of cases with identical rankings by all annotators divided by 100) is 82.3%, indicating a high consistency in the human evaluation.

The results are as follows:

Accordingly, MIO achieves the best performance in terms of both structural integrity and aesthetics. Moreover, we recognized the importance of introducing human evaluation for image generation. As shown by the metric of human averaged rankings, we observe that MIO is most preferred by the real users compared with AnyGPT and Emu. We promise to add the references regarding the human preference for image generation (i.e., [1-3]) in our revised version.

[1] Kirstain, Yuval, et al. "Pick-a-pic: An open dataset of user preferences for text-to-image generation." Advances in Neural Information Processing Systems 36 (2023): 36652-36663.

[2] Xu, Jiazheng, et al. "Imagereward: Learning and evaluating human preferences for text-to-image generation." Advances in Neural Information Processing Systems 36 (2024).

[3] Wu, Xiaoshi, et al. "Human preference score: Better aligning text-to-image models with human preference." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Q1: limited novelty in comparison to existing work.

A1: While MIO builds upon prior research, we believe it makes meaningful contributions to the field of multimodal foundation models. Specifically, we list the novel features of our work in Table 1.

We highlight the following novel aspects:

1. Support for Multimodal Interleaved Sequence Generation:

   MIO uniquely supports the generation of multimodal interleaved sequences, a capability absent in prior works such as Emu and CM3Leon. This feature enables advanced tasks, including interleaved video-text generation, chain-of-visual-thought reasoning, and instructional image editing, which we demonstrate in Figure 2.

2. Addressing Limitations in Existing Models:

   • Unlike SEED-LLaMA, Emu1, and Emu2, MIO integrates speech input and output.
   • MIO also surpasses the scope of prior works like AnyGPT, which lack advanced capabilities such as voice generation, interleaved video-text generation, etc.

3. Emergent Abilities and Generalist Features:

   As a model that combines four modalities with both understanding and generation capabilities, MIO demonstrates emergent abilities that arise from its any-to-any foundation. Examples include chain-of-visual-thought reasoning and visual guideline generation (Figure 2), showcasing the model's unique strengths.

By addressing these gaps and extending the capabilities of multimodal foundation models, we believe MIO makes a meaningful contribution to the field.

---

Q2: limited performance and outdated baselines for image understanding.

A2: We add two strong baselines, i.e., Qwen-VL-Chat (13B)[1] and LLaVA 1.5 (7B)[2], in the Table 3:

We can observe that MIO-Instruct still demonstrates minor performance gaps despite less image resolution (Qwen-VL-Chat is trained on 448x448 images, LLaVA 1.5 is trained on 336x336 images, while ours is trained on 224x224 images) and the additional support for speeches and image generation.

[1] Qwen-vl: A frontier large vision-language model with versatile abilities

[2] Improved Baselines with Visual Instruction Tuning

Hi, Reviewer MGdu,

Thank you for your valuable comments. As the discussion period is coming to a close, we would appreciate it if you could let us know whether our responses have addressed your concerns.