OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1&Q5: There is little new novelty in this paper. In my opinion, this paper reads.... However, could you highlight the key innovations, novelties, or findings ..

Response:

We sincerely thank the reviewer for their thoughtful feedback. While many multimodal learning works may seem to simply integrate pretrained components and train them jointly, we believe their novelty lies in addressing specific challenges and developing solutions, which ultimately shape the uniqueness of the work. OpenOmni, as an open-source solution, aims to enable efficient training and competitive performance under limited data and computational resources, requiring just 40GB peak GPU memory. By making omnimodal learning accessible to the community, OpenOmni bridges the gap between academic research and proprietary models.

Key Innovations, Novelties, and Findings

1. Dynamic Progressive Alignment Strategy: OpenOmni uses a progressive alignment strategy to optimize computational resources and data use, achieving shallow alignment for text-speech and deep alignment for image-text tasks. Ablation studies in Tables 7 and 8 validate its ability to improve both efficiency and performance compared to joint training.
2. Robust Speech Decoding with TGM Module: The TGM module stabilizes training for the CTC-based speech decoder, enabling real-time multilingual speech synthesis while enhancing quality comparable to autoregressive models like EMOVA and VITA. Figure 4 confirms its effectiveness.
3. Multilingual Support with MOE Module: Conventional CTC-based methods (e.g., LLaMA-Omni) are limited to single-language speech generation. OpenOmni introduces the MOE module, supporting effective multilingual synthesis during CTC training.
4. CTC-DPO Algorithm for Emotional Speech: OpenOmni’s CTC-DPO algorithm generates contextually consistent emotional speech for multi-turn dialogues by leveraging multi-turn datasets, bridging the user experience gap with proprietary models for adaptive interaction.
5. Impact on Subsequent Research: OpenOmni’s innovations have advanced later works:
   • TGM module improves speech decoder performance [1,2].
   • Multi-turn emotional consistency enhances multi-character interactions [3].

These contributions underscore OpenOmni’s progress in omnimodal learning, lowering research barriers and fostering broader community participation to advance the field collaboratively.

Q1: how much data does one need to align omnimodality?

Response:

Thank you for the reviewer’s insightful feedback. Appendix Tables 7 and 8 present preliminary findings from our ablation experiments on the omnimodal alignment strategy:

1. Alignment Order: No significant performance difference exists between text-speech first and image-text first alignments, as speech and image align independently to the text space.

2. Alignment Depth: Shallow alignment (freezing the LLM) is effective for text-speech due to their natural temporal correlation, while deep alignment (unfreezing the LLM) is required for image-text to achieve better results.

We hypothesize that text-speech alignment demands far less data than image-text alignment. To verify this, we conducted experiments using varying training data amounts, as detailed below.

For text-speech alignment, 900 hours of data achieved 97% of the performance obtained with 1,600 hours, showing the efficiency of natural temporal correspondence where data quality outweighs quantity in later stages.

For image-text alignment, performance improves as data increases, with slower gains in later stages but room for growth, suggesting a larger gap between image and text representations where additional data enhances alignment.

These findings highlight differing data requirements for text-speech and image-text alignments, offering valuable insights for future research.

Q2: Does it learn cross-lingual skills?

Response:

To evaluate multilingual generalization during speech-text alignment, we used the Multilingual Speech Commands Dataset validation set, containing speech data in 14 non-English languages. Zero-shot command recognition experiments were conducted on OpenOmni during text-speech alignment, and the accuracy results are shown in the table below.

As shown in the table, OpenOmni’s text-speech alignment with Chinese and English data demonstrates strong cross-lingual generalization, enabling consistent improvements in non-English speech recognition.

Q3: What are the weaknesses of OpenOmni compared to ..,

Response:

We appreciate the reviewer’s detailed feedback and would like to clarify the distinctions between OpenOmni, VITA, and proprietary models like GPT-4o:

1. Architectural Differences: VITA uses a pipeline approach, generating text responses first and then passing them to a TTS model, which risks error accumulation. It also requires duplex communication with two model deployments, significantly increasing GPU memory needs. OpenOmni adopts an end-to-end design, feeding LLM features into a CTC-based decoder for real-time parallel decoding, enabling emotionally consistent speech in multi-turn dialogues—something VITA struggles with.

2. Training Strategies: OpenOmni employs a progressive alignment strategy that reduces tri-modal data dependency and increases efficiency, requiring only 40GB peak GPU memory and 1,400 GPU Hours for strong performance. By contrast, VITA uses joint training, consuming three times more data (5M vs. 1.7M) and 80GB GPU memory, with nearly 10,000 GPU Hours, making it significantly more computationally expensive.

3. Advantages of OpenOmni: OpenOmni is faster, more efficient, and excels at producing emotionally consistent speech for better interaction experiences. Its design lowers research and deployment barriers, making omnimodal research more accessible.

4. Comparison with Proprietary Models: OpenOmni lacks access to large-scale industry datasets or computational resources like GPT-4o and Qwen2.5-Omni. While its CTC-based decoder offers efficient real-time emotional speech generation, speech quality is slightly inferior to the autoregressive methods used in proprietary models.

In summary, OpenOmni optimizes efficiency, accessibility, and performance, contributing significantly to omnimodal research while lowering entry barriers for the academic community.

Q4: Is ... emotion speech (through DPO) good enough .. what might be other options to ..

Response:

Thank you for the reviewer’s feedback. We are pleased to discuss emotion speech generation further.

Although OpenOmni still has gaps compared to proprietary models like GPT-4o, its focus on multi-turn emotional speech generation and strong context-learning capabilities has significantly narrowed the gaps with previous methods (such as EMOVA and Qwen-Audio) and proprietary models.

Recent studies [4,5] show that large-scale datasets of millions of hours enable adaptive, contextually coherent speech generation. However, such data and resources remain beyond the academic community’s reach. Meanwhile, post-training strategies like CTC-DPO in OpenOmni provide a practical, efficient solution for producing reliably consistent emotional speech.

[1] Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

[2] LLaMA-Omni 2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis

[3] OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction

[4] Higgs Audio V2: Redefining Expressiveness in Audio Generation

[5] MOSS: Text to Spoken Dialogue Generation

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1&Q1: progressive alignment (text as pivot) is not novel. The approach...technical differences. How exactly ...? A quantitative or ablation-based comparison...

Response:

We sincerely appreciate the reviewer’s feedback. While most multimodal learning methods [1,2,3] commonly use text as a pivot as a foundational approach, their novelty and innovation often lie in the specific alignment strategies tailored to applications. To highlight OpenOmni’s contributions, we compare its alignment strategy with previous works such as AnyGPT and Qwen-Omni and summarize the differences in the table below.

As shown in the table, AnyGPT and Qwen-Omni rely on tri-modal data and single-stage joint training for all modalities. In contrast, OpenOmni employs a progressive alignment strategy, aligning modalities in separate stages. Each stage involves at most two-modality alignment, reducing GPU memory requirements and lowering the entry barrier for omnimodal learning.

This approach allows flexible training strategies. Ablation studies in Appendix Tables 7 and 8 show that text-speech alignment achieves competitive results by freezing the LLM, improving training efficiency by 300%, while image-text alignment requires unfreezing the LLM due to larger modality gaps for deeper alignment.

By tailoring strategies to specific modality characteristics, OpenOmni’s dynamic training strategy enhances flexibility, improves efficiency, and maintains competitive results. With less data, OpenOmni achieves SOTA performance compared to other open-source models, showcasing its effectiveness and innovation in omnimodal learning.

W2: OpenOmni largely reuses existing components .... The speech decoder architecture is mostly ....

Response:

Thank you for the reviewer’s valuable feedback. While the reuse of existing components and standard training techniques is common across omnimodal learning frameworks, OpenOmni sets itself apart through its dynamic and flexible modal progressive alignment strategy, which minimizes dependencies on tri-modal data and GPU requirements—marking a key innovation compared to previous works.

Furthermore, OpenOmni utilizes a lightweight CTC-based speech decoder with notable improvements. It introduces the MOE mechanism to handle multilingual conflicts during CTC training and the TGM module, which stabilizes the CTC training process. These enhancements retain the real-time efficiency of CTC decoders while achieving speech generation quality on par with autoregressive methods.

These contributions highlight OpenOmni's innovations and findings, offering valuable insights to the omnimodal learning community and driving advancements in this field.

W3: For example, the TGM module’s design is described but ...

Response:

Thank you for the reviewer’s feedback. In Appendix Lines 650-654, we qualitatively analyzed Table 4, showing the TGM module significantly stabilizes training for both CTC and AR mode decoders, enhancing speech generation. Additionally, we supplemented this with quantitative experiments for a more comprehensive evaluation.

As shown in the table, the TGM module significantly improves content consistency in both AR and CTC modes, with strong generalization for Chinese and English speech generation. It has also been adopted by subsequent works [4,5] to enhance speech decoder consistency.

W4: The progressive alignment strategy...rather than truly novel. Several prior works (e.g., Qwen2-Audio, EMOVA, AnyGPT) have either done similar ....

Response:

Thank you for the reviewer’s valuable comments. We clarify the differences between our method and previous works in terms of progressive alignment strategy, DPO training for emotional speech, and speech decoders:

1. Progressive Alignment Strategy: OpenOmni employs a flexible modal alignment approach based on ablation experiments, using different strategies for text-speech and image-text alignments. This achieves competitive results with minimal computational resources and training data.
2. Speech Decoders: OpenOmni uses a CTC-based approach for parallel real-time speech generation, unlike autoregressive methods in AnyGPT and EMOVA. We enhance CTC to support bilingual speech and yield results comparable to autoregressive techniques while maintaining efficiency.
3. DPO Training: OpenOmni focuses on emotional consistency in multi-turn dialogues, similar to GPT-4o, allowing the model to learn user intent through context and output appropriate emotional speech for better interaction. This approach has been adopted by subsequent work [6] to improve multi-turn dialogue experiences. In contrast, methods like EMOVA rely on predefined emotion keywords for single-turn tasks, limiting their effectiveness in multi-turn scenarios.

From these perspectives, OpenOmni fundamentally differs from previous methods.

Q2:The paper claims DPO is more stable.... An ablation with RLHF or reward-based ...

Response:

Thank you for the reviewer’s comments. We conducted ablation experiments using the PPO algorithm and Verl framework, with the EO2S-9K dataset to train the reward model for RLHF. The experimental results and computational resource comparisons are shown in the table below.

As shown in the table, while PPO shows no significant performance advantage over DPO, it requires loading and training four model copies, resulting in much higher GPU resource consumption.

Moreover, during training, the EO2S-9K reward model often failed to provide accurate evaluations, causing instability in PPO. Outlier data, where the policy model generated unusual speech and the reward model delivered incorrect emotional judgments, repeatedly led to training failures, requiring three attempts to complete.

In contrast, DPO is unaffected by reward model performance on small datasets, offering more stable training, lower resource consumption, and results comparable to PPO, making it a more efficient and practical choice.

Q3: Although WER and emotion classification .... Have human evaluations been conducted?

Response:

Thank you for the reviewer’s comments. We used the MOS (Mean Opinion Score) metric, with 10 experts evaluating the generated speech for Chinese and English sets and EO2S emotional speech. The experimental results are shown in the table below.

The experimental results indicate that the MOS evaluations from human experts demonstrate that the speech generated by OpenOmni is natural, fluent, and exhibits emotional consistency.

[1] LLaVA: Large Language and Vision Assistant

[2] AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

[3] EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions

[4] Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

[5] LLaMA-Omni 2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis

[6] OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction

Dear Reviewer GJPB,

I hope this message finds you well. As the discussion period is nearing its end with only one day remaining, I wanted to ensure we have addressed all your concerns satisfactorily. We have provided comprehensive responses to all four weaknesses and three questions you raised, including the quantitative comparisons and human evaluations you requested.

If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Sincerely,

Author

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1: The authors adopted CTC loss to train the speech decoder instead of the next-token-prediction paradigm, which is more commonly used, such as Kimi-Audio [1] and Moshi [2]. Please clarify the decision.

Response:

Thank you for the reviewer’s detailed comments. To further explain our choice of using the CTC decoding approach, we conducted the following experiments for comparison.

In addition to the basic metrics for semantic consistency of speech content, such as WER (Word Error Rate) and CER (Character Error Rate), we also employed 10 expert evaluators to assess the MOS (Mean Opinion Score), which evaluates speech naturalness and emotional congruence.

These experiments provide a comprehensive evaluation framework to highlight the advantages of the CTC decoding approach.

As shown in the table, the CTC speech decoder enhanced by the TGM module maintains real-time generation capabilities while using less GPU memory. Compared to autoregressive speech decoders, it produces competitive, fluent, and natural speech with high MOS scores evaluated by human experts. This supports our decision to adopt the CTC speech decoder to improve real-time interaction experiences.

W2: the clarification needs improving.

Response:

Thank you for the reviewer’s careful and thoughtful feedback, which has helped improve the readability of our work. We intend to clarify that the omnimodal learning capability is developed through the first two stages, while the third stage is dedicated to speech generation. These clarifications will significantly enhance the clarity of our paper, and we are committed to implementing these changes thoughtfully in the revised version. Thank you again for your constructive suggestions.

W3: Table 1 shows that the proposed model underperforms significantly in terms of text&symbols, count&quantity. Please analyze and discuss the reasons.

Response:

Thank you for the reviewer’s careful feedback. We hypothesize that this performance discrepancy might stem from OpenOmni’s reliance on the image data distribution of MMEvol, which contains relatively few samples for text & symbol and count & quantity tasks.

To validate this hypothesis, we sampled 50K text & symbol data from the Video-SALMONN training dataset and 50K count & quantity data from the Pixmo-Count dataset. These samples were mixed with the original 1.7M instruction data, forming a 1.8M instruction dataset for use in OpenOmni's second-stage alignment training. The experimental results are reported below.

As shown in the table, after incorporating the additional training data specifically targeting text & symbol and count & quantity, OpenOmni demonstrates significant performance improvements in these two capabilities.

Q1: The two-stage training procedure is well elaborated in the paper and the appendix. However, I'm still curious how the authors dealt with catastrophic forgetting?

Response:

Thank you for the reviewer’s valuable comments. We are happy to further elaborate on our strategies for addressing catastrophic forgetting during the multi-stage progressive omnimodal learning process.

During the text-speech alignment stage, we adopt a shallow alignment strategy by freezing the LLM during training. This approach ensures that catastrophic forgetting does not occur.

In contrast, during the image-text alignment stage, we adopt a deep alignment strategy by unfreezing the LLM for training. While this improves image-text alignment capabilities, it can influence the original knowledge of the LLM and potentially disrupt the text-speech alignment knowledge established in the first stage, leading to catastrophic forgetting. To mitigate catastrophic forgetting for the LLM, we employ a mature approach inspired by MMEvol and LLaVA, mixing 143K high-quality textual instruction data from WizardLM into the training process. This strategy effectively alleviates catastrophic forgetting in the LLM.

As demonstrated in our ablation study in Appendix Table 7, the impact of the image-text alignment stage on the knowledge from the text-speech alignment stage is minimal. Specifically, unfreezing the LLM for training in the second stage causes only a 2.8% performance drop (12.10 vs. 12.45) on AI-Shell and a 2.1% performance drop (11.55 vs. 11.8) on LibriSpeech, with no catastrophic forgetting observed.

During the speech generation stage, the LLM remains frozen, ensuring that catastrophic forgetting does not occur at this stage either.

We hope this clarification highlights the robustness of OpenOmni’s multi-stage learning process and strategies for effectively handling catastrophic forgetting.

Dear Reviewer TCBG:

Thank you again for your time and valuable feedback on our paper. We noticed that your official comment on OpenReview is still pending. Could you please let us know if there’s any additional information needed from our side? We’d greatly appreciate it if you could submit your comment when you have a moment.

Sincerely,

Authors

Dear Reviewer TCBG,

I hope this message finds you well. As the discussion period is approaching its deadline, I wanted to ensure we have adequately addressed all of your comments and concerns raised during the review.

If there are any remaining issues or additional points you’d like us to clarify, please don't hesitate to let us know. Your valuable insights are critical for improving our work, and we're more than willing to engage further to address any outstanding questions.

Thank you very much for your effort and time in reviewing our submission.

Sincerely,

Author

Dear Reviewer TCBG,

Thank you so much for taking the time to share your thoughtful feedback and for recognizing the efforts and innovations presented in our work. We are truly grateful that our responses and additional experiments have addressed most of your concerns, and we deeply appreciate your willingness to update your rating score. Your constructive feedback has significantly helped us refine and clarify our contributions, and for that, we sincerely thank you.

We are especially pleased to see that you recognize the value of our dynamic progressive alignment strategy, which demonstrates how SOTA results in omnimodal learning can be achieved with minimal computational resources and a smaller data scale. This strategy showcases our focus on enabling efficient and accessible training, representing a step forward in addressing challenges such as high GPU memory demands and the need for vast amounts of data.

Additionally, we’re thrilled that the capabilities of our TGM-enhanced CTC-based speech decoder stood out to you. As you observed, the lightweight and efficient design of this approach ensures that speech generation remains both competitive in quality compared to AR-based decoders and fully capable of real-time synthesis. The combination of low latency, reduced computational load, and emotionally coherent generation is pivotal for improving the user experience in interactive multimodal systems, and we’re excited about its potential to further enrich omnimodal dialogue and beyond.

This balanced integration of innovation and practicality—achieving high performance under resource constraints—is a central contribution of OpenOmni that we hope continues to resonate with the omnimodal and AI research communities.

If you have further suggestions or questions, we are always happy to engage in future discussions to further polish this work. We look forward to contributing to both the advancement of scientific research in this field and the strengthening of the open-source community.

Thank you again for your valuable review and kind recognition. Your support inspires us to push forward.

Best regards,

Author

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1: The paper considers dataset construction as one of its contributions; however, the description of the process lacks sufficient detail and fails to clearly demonstrate the advantages of the newly constructed datasets.

Response:

Thank you for the reviewer’s comments. In Section 4.1 of the main text, we provided a detailed explanation of our data sources and construction processes, with a brief introduction in Appendix A. Here, we further clarify and highlight the methodologies and advantages of our datasets.

We sampled 300K multi-turn dialogues from UltralChat (text) and MMEvol (multimodal), selecting longer responses to ensure extended audio durations for better model learning. Half of the data was translated into Chinese to support bilingual generation. Using CosyVoice, speech synthesis was performed on the text data to create O2S-300K.
For EO2S-9K, we intensively filtered dialogues with contextual consistency and strong emotional tendencies in multi-turn interactions. Emotional speech synthesis was conducted using CosyVoice, resulting in a dataset optimized for DPO-based emotional speech generation. A total of 8,000 A100 GPU hours were spent synthesizing O2S-300K and EO2S-9K, with filtering, translation, and selection costing approximately $2,000. These efforts ensure diverse, high-quality datasets.

Advantages:

1. O2S-300K: It is the bilingual multi-turn dataset with tri-modal inputs (text, audio, vision). Unlike prior works like LLaMA-Omni, which use single-turn English-only voice data, O2S-300K enables building multi-turn bilingual interaction models, supporting arbitrary input combinations for enhanced contextual consistency and interaction experience.

2. EO2S-9K: It provides fine-grained emotional speech annotations with contextual consistency, allowing adaptive emotional speech generation after training for superior multi-turn interactions. Previous datasets rely on pre-defined emotion tags in prompts, often leading to inconsistencies between emotion tags and text (e.g., “I feel happy [sad]”), limiting adaptive learning and natural dialogue engagement.

These datasets address key limitations of prior works, advancing bilingual, multi-turn, and emotionally adaptive speech synthesis. If further clarification is needed, we are happy to address additional questions.

W2: The paper lacks comparison and discussion with other methods. For example, it does not clearly outline the improvements and differences in architecture and training strategy between OpenOmni and VITA.

Response:

Thank you for the thoughtful feedback. We would like to clarify the key architectural and training differences between OpenOmni and VITA.

1. Architectural Differences:
   • VITA uses a pipeline-based speech generation method, where text responses are passed to a separate TTS model. To speed up synthesis, it employs a duplex communication strategy that requires two simultaneous model deployments, doubling GPU memory usage. Its non-end-to-end design limits multi-turn dialogue context utilization, making it less effective at capturing emotional consistency.
   • OpenOmni features an end-to-end speech generation design, where LLM outputs are directly fed into a CTC-based speech decoder for real-time parallel decoding. This approach supports context-aware emotional speech generation and ensures consistency across multi-turn interactions.
2. Training Differences:
   • VITA employs large-scale joint training, needing three times more tri-modal data (5M vs. 1.7M), 80GB peak GPU memory, and nearly 10,000 GPU Hours, demanding significantly larger computational resources.
   • OpenOmni adopts a progressive multi-modal alignment strategy, reducing reliance on tri-modal datasets while enhancing efficiency. It requires only 40GB peak GPU memory and 1,400 GPU Hours.

In summary, OpenOmni is faster, more efficient, and context-aware, overcoming VITA’s limitations in emotional speech generation and resource consumption, while enabling smoother and more effective multi-turn interactions.

Q1: In Stage 2 of the framework, which focuses on Image-Text alignment, the training involves fine-tuning the LLM. A potential concern is whether this fine-tuning could undermine the Speech-Text alignment achieved during Stage 1.

Response:

Thank you for the reviewer’s thoughtful feedback. In fact, we have conducted relevant experiments in the ablation study presented in Table 7, and provided detailed discussions in lines 669-676 of the paper.

The results demonstrate that due to the natural temporal alignment between speech and text, we choose to freeze the LLM during the speech-text alignment stage, while opting to unfreeze the LLM during the image-text alignment stage. This dynamic alignment strategy effectively balances efficiency and performance trade-offs.

Additionally, we observe that unfreezing the LLM during the image-text alignment stage has only a minimal impact on the speech-text alignment knowledge from the first stage. Specifically, it results in a very minor performance decline on AI-Shell (2.8%, 12.10 vs. 12.45) and LibriSpeech (2.1%, 11.55 vs. 11.8) evaluation datasets, without causing catastrophic forgetting.

By employing such a progressive alignment strategy, we achieve a nearly 300% improvement in training efficiency, making it a highly practical approach. For these reasons, we adopt this method in our proposed framework.

Q2: Table 4 shows that the improvement from DPO training on Chinese evaluation is significantly more noticeable than on English. Is this primarily due to the data distribution or the model's pre-training?

Response:

We believe this may be related to the varying distribution of emotion-themed dialogue data during pretraining. To investigate, we randomly sampled 10K emotional speech data from the pretraining dataset and annotated it using GPT4O-mini for emotion labeling. The distribution of different emotions in the data is shown below:

From the table, it is evident that the differing proportions of emotions in the pretraining data contribute to the observed performance. Emotions with a higher proportion in the pretraining dataset are more likely to be effectively triggered during subsequent DPO training, thereby leading to a significant improvement in emotional speech generation capabilities.

Q3:For the choice of speech decoder, the paper states that AR and NAR decoder differ in generation quality and speed. Please provide quantitative experimental results for both and analyze them.

Response:

Thank you for the reviewer’s valuable comments. Here, we provide more detailed quantitative experimental results comparing AR (Autoregressive) and NAR (Non-Autoregressive) modes, along with their corresponding analysis.

In addition to the baseline metrics for semantic consistency of speech content, such as WER (Word Error Rate) and CER (Character Error Rate), we also employed 10 expert evaluators to assess the MOS (Mean Opinion Score) metric. The MOS metric provides a comprehensive evaluation of speech naturalness and emotional congruence.

Through this detailed comparison, we aim to better illustrate the strengths and weaknesses of both modes and their applicability to various use cases.

As shown in the table, OpenOmni's NAR mode enables real-time speech synthesis with lower GPU memory usage, while still producing high-quality and fluent synthesized speech as confirmed by human evaluations.

In terms of generation quality, the AR mode achieves slightly higher performance but comes at the cost of significantly greater computational requirements and GPU memory consumption. Additionally, the autoregressive nature of the AR mode limits its ability to perform real-time speech generation. In contrast, the NAR mode strikes a balance between speech generation quality and efficiency, making it a practical trade-off solution for widespread adoption.

To support the open-source community and democratize OmniModal research, OpenOmni adopts the most efficient approach by integrating a progressive omnimodal alignment strategy combined with the NAR-mode speech decoder. This choice ensures that the community can participate in cutting-edge research with minimal computational and data requirements, ultimately advancing the field together.

Dear Reviewer LJos,

I hope this finds you well. As we near the end of the discussion period, we wanted to thank you once again for your thoughtful review. Please let us know if there are additional questions or aspects of the work you’d like us to address.

We highly value your expertise and feedback, and we’re eager to continue discussing any unresolved points that could help refine and improve our research.

Thank you for your time and for contributing to the review process.

Sincerely,

Author