LLaMA-Omni: Seamless Speech Interaction with Large Language Models

Thank the authors for their huge efforts in revising the paper. I appreciate the responses that address most of my concerns. I believe the experiments are generally convincing and comprehensive. More importantly, another notable strength is that this paradigm largely preserves the intelligence of pre-trained LLMs.

My rating leans towards accept.

Thank you for providing your valuable and constructive feedback! Please find below the responses to each comment.

---

[Q1] Losses richer features such as emotional nuance and speaker identity.

Thank you for your question. As you mentioned, our speech is not directly generated by the LLM but instead by an additional speech decoder. While this approach may somewhat weaken the speech modeling capabilities, it preserves the fundamental reasoning abilities to a greater extent, which we believe is a more essential and foundational aspect at this stage.

Additionally, since our speech generation still follows an end-to-end modeling approach, the model retains the potential to perceive information beyond the content of the input speech, such as emotion. However, enabling the model to generate speech with emotions or other richer characteristics requires further exploration, which we plan to address in future work.

---

[Q2] The cascaded use of VITS TTS in SALMONN and Qwen2-Audio may introduce errors affecting S2SIF and alignment due to propagated TTS inaccuracies.

Thank you for your question. To ensure a fairer comparison, we replaced the TTS component of the cascaded systems with a industrial streaming TTS model, Orca, and conducted more extensive comparative experiments across different latency settings. The relevant experimental results have been updated in the revised PDF. If you are interested, please refer to the updated paper for details.

---

[Q3] "Popping" at the boundaries.

Yes, we have also observed that popping artifacts can occur between audio segments synthesized from different unit chunks. We believe that addressing this issue requires enhancing the vocoder's streaming synthesis capabilities, such as employing lookahead strategies during synthesis to prevent discontinuities at the end frames. Additionally, our current approach synthesizes each unit chunk independently before concatenation. Incorporating the preceding unit chunk during synthesis could potentially alleviate this issue as well.

---

[Q4] Language comprehension abilities.

Thank you for your question. Following your suggestion, we tested our base model Llama-3.1-8B-Instruct and LLaMA-Omni on the MMLU benchmark. The results are presented in the table below:

We observed that the trained model exhibited a slight degradation in its language comprehension capabilities, although it retained most of its original abilities. We believe this is primarily because no text instruction data was used during the training phase, leading to reduced performance when evaluated on text instruction datasets. This issue might be addressed using simple data replay techniques—by incorporating some text instructions during the speech instruction fine-tuning process. We plan to explore this approach in future work.

I'm very grateful for the detailed reply.

Q1. Comparison with autoregressive TTS models

The author point out that some traditional TTS models, such as VALL-E, need to wait for all the text before starting to generate speech. However, there are streaming TTS modules, such as Moshi and Mini-Omini used. I'm more curious about what advantages CTC-based NAT modules have over AT modules.

Q2. It is not evident what benefits the end-to-end model offers over a straightforward cascaded solution where the text response is segmented into chunks and then fed into an offline TTS model.

Thanks to the author's response, and this concern has been resolved.

Q3. Lacking speaker generalization.

Thanks to the author's response, and this concern has been resolved.

Q4. Evaluation on human recorded speech.

I don't agree with what the author stated that the current model has no way to evaluate ASR tasks. First of all, if the speech and text are well aligned, then it should be able to generalize the ability of dealing with text instructions + speech input. Secondly, the author can concatenate a speech instruction in front of the test speech segment to guide the model to perform the ASR task. I think the performance on ASR tasks can more intuitively reflect the degree of semantic understanding.

Q5. Unconvincing evaluation metrics and results analysis.

Thanks to the author's hard work. There are many additional experiments in the new version compared to the initial version. However, there are two things I don't understand. First, based on Table 2, why does the S2TIF score of LLAMA-Omini be significantly better than that of Qwen2-Audio in the offline evaluation? We all know that GPTScore has a strong length bias and should prefer longer responses. Secondly, UTMOS is an evaluation metric for TTS models and is generally applicable to evaluations when the text content is the same. For different models, the text responses are different, and this metric may not be accurate. At least for me, I think the offline speech quality of SALMONN + Orca is better than LLAMA-Omini.

Q6. Decoding time.

Thanks to the author's reply. WPT is a more convincing metric.

Q7. Multiturn conversations.

Thanks to the author's reply. Looking forward to future work.

Q8. The contribution of this article.

Thanks for the clarification.

In conclusion, I have increased the soundness score to 3 and increased the OA to 6.

[Q7] Multiturn conversations.

Thank you for your suggestion. While we have currently focused only on the model's performance in single-turn conversations, extending the current model to multi-turn dialogues is straightforward and natural. This can be achieved by constructing corresponding multi-turn dialogue datasets and training the model in a similar manner. We plan to explore this direction in the future.

---

[Q8] The contribution of this article.

We believe our contributions can be summarized as follows:

1. Model Architecture: We proposed an end-to-end model architecture capable of streaming speech response generation, achieving high-quality responses with low latency. Comparisons with prior speech language models such as SpeechGPT and cascaded systems based on streaming TTS validate the effectiveness of our proposed model.
2. Data Construction: We introduced a series of steps to adapt text instruction data to the speech domain, which are essential for speech interaction scenarios. Through extensive experiments and human evaluations, we demonstrated the effectiveness of this approach. In addition, we validated that our model exhibits robustness to voice variations when exposed to multi-speaker data, enabling it to handle real-world applications effectively.

For the above results, we have included line charts in the revised PDF illustrating the variation of each metric across different latency conditions for all models. These charts provide a more intuitive comparison of the performance of different models. Please refer to the updated version of the paper. We have also provided audio samples at https://llama-omni.github.io/, allowing for a more intuitive understanding of the speech generated by different models under various latency conditions through listening.

We also conducted human evaluations to investigate human preferences in real-time speech interaction scenarios. Specifically, we performed side-by-side comparisons between LLaMA-Omni ($\Omega=40$, latency=347ms) and two cascade systems: SALMONN+Orca ($\Theta=3$, latency=352ms) and Qwen2-Audio+Orca ($\Theta=3$, latency=420ms). The comparisons evaluated the systems in terms of the helpfulness of the responses and the naturalness of the generated speech, using a win/tie/lose framework. The evaluation results are as follows:

From the results, we observed that LLaMA-Omni achieved higher win rates in both helpfulness and naturalness, demonstrating that LLaMA-Omni's responses better align with human preferences. More details about the human evaluation can be found in Section 4.6 of the revised PDF.

---

[Q6] Decoding time.

Thank you for your suggestion. As you mentioned, comparing the total decoding time is not meaningful due to differences in the response lengths of different models. Therefore, we compared the average decoding time per token and per word for LLaMA-Omni and Qwen2-Audio+Orca in the streaming scenario. The results are shown in the table below:

Since decoding time consists of two components: the first is the decoding time of LLMs (LLaMA-Omni/Qwen2-Audio), which remains constant regardless of latency, and the second is the time required by the streaming vocoder or TTS, which varies depending on the unit/word chunk size. To provide a clearer understanding of the time allocation, we present the two components separately in the above table. As shown, although LLaMA-Omni’s LLM decoding involves both text and unit decoding, its average decoding time remains lower than the average time required by Qwen2-Audio for text decoding alone. This is attributed to the parallel generation capability of LLaMA-Omni's non-autoregressive speech decoder. When comparing the streaming vocoder to streaming TTS (Orca), we observe that the average decoding time for the vocoder is consistently lower across all latency conditions. These findings demonstrate the superior decoding efficiency of LLaMA-Omni.

The numerical results under different latency conditions for the cascade systems (SALMONN + Orca, Qwen2-Audio + Orca) and LLaMA-Omni are as follows:[SALMONN + Orca]

[Qwen2-Audio + Orca]

[LLaMA-Omni]

Note: The latency of LLaMA-Omni has slightly changed compared to the submitted version due to variations in machine performance. To ensure fairness, we reevaluated the latency of all systems under the same conditions.

From the experimental results, we observe the following:

1. Both the cascade systems and LLaMA-Omni can achieve latencies below 320ms (the average audio latency of GPT-4o) and allow latency control by adjusting hyperparameters. Across all latency conditions, LLaMA-Omni consistently achieves higher ChatGPT Scores compared to the cascade baseline systems.

2. In the offline scenario, the cascade systems achieve lower ASR-WER than LLaMA-Omni, demonstrating the superior performance of industrial TTS models. However, in the streaming scenario, as latency decreases, we observe a significant performance drop in the cascade systems compared to the offline scenario (evidenced by a decrease in ChatGPT Score and an increase in ASR-WER). In contrast, LLaMA-Omni shows minimal changes in ChatGPT Score and ASR-WER as latency varies, and even exhibits performance improvements in some cases.

3. In terms of speech quality, all systems experience a decline in quality as latency decreases, primarily due to an increase in discontinuities within the speech. LLaMA-Omni achieves speech quality comparable to that of SALMONN+Orca.

4. From the WPS metric, LLaMA-Omni's speech rate remains almost unaffected by latency changes. In contrast, the overall speech rate of the cascade systems decreases significantly under low-latency conditions, primarily due to increased pauses between word chunks, which significantly affects the overall rhythm and prosody of the speech. For example, at the lowest latency ($\Theta=1$), the overall speech rate drops to less than 60% of the offline scenario. This indicates that cascade systems face significant challenges under extremely low-latency conditions.

[Q4] Evaluation on human recorded speech.

We did not test on datasets such as LibriSpeech, CoVoST, or AIRBench because our model is designed for scenarios that differ from these tasks. Specifically, our model generates responses based solely on speech instructions, without requiring any text instructions. Tasks such as speech recognition, speech translation, or speech-based question answering, however, necessitate the model to simultaneously process speech input and text instructions, which is not encountered during our model’s training.

To address this, we conducted a fine-tuning experiment on LLaMA-Omni using the LibriSpeech train-clean-100h dataset. With just one epoch of fine-tuning (892 steps), the model achieved Word Error Rates (WER) of 3.29% on the test-clean set and 6.88% on the test-other set. From the experimental results, we conclude that LLaMA-Omni inherently possesses the capability to process various types of speech. Fine-tuning merely serves to teach the model how to follow speech recognition instructions.

---

[Q5] Unconvincing evaluation metrics and results analysis.

We sincerely thank you for pointing out potential issues in our experimental results. Based on your suggestions, we conducted a comprehensive review of our experiments and identified several key issues, which we have thoroughly corrected:

1. ASR-WER/CER: Regarding your observation of unusually high ASR-WER/CER scores of the Qwen2-Audio+TTS Model, we found this was due to an oversight. When calculating ASR-WER/CER, the Whisper model incorrectly truncated the speech to 30 seconds, resulting in the loss of content after the 30-second mark, which led to incorrect ASR-WER/CER scores. We have corrected this issue and conducted a complete re-evaluation.
2. ChatGPT Score: The evaluation prompt we previously used may have introduced bias. We have removed all potentially bias-inducing priors from the prompt and simply instructed the model to give a single score based on helpfulness, relevance, fluency, and suitability for speech interaction, without providing any explanations that might introduce bias (details in Appendix A of the revised PDF). We also conducted human evaluations following your suggestions, and the results are presented below.
3. TTS Model: Apologies for the previous unclear explanation, which may have caused misunderstandings. The VITS model we used is integrated within the Coqui TTS framework, which performs text segmentation before speech synthesis to ensure accurate synthesis even for long texts. In fact, the speech responses in our training dataset were also synthesized using this model. However, to achieve a more comprehensive and meaningful comparison, especially in the streaming scenario, we have replaced the cascaded baseline systems with Qwen2-Audio/SALMONN + Orca, where Orca is an industrial TTS model that supports both streaming and offline speech synthesis. When using Orca for streaming speech synthesis, we need to set a word chunk size $\Theta$, which means that speech synthesis is triggered every time $\Theta$ new words arrive. In our experiments, we varies $\Theta$ within the range of $[1, 3, 5, 7, 9]$ to control the response latency of cascaded systems.

We conducted a comprehensive re-evaluation of all models in both offline and streaming scenarios. Specifically, we evaluate the model using the following metrics:

1. ChatGPT Score [Updated]: We use GPT-4o to rate the model's responses. Following the suggestions from reviewers ppgU and DXz7, we have revised the evaluation prompt (details in Appendix A of the revised PDF) to remove potential prior biases.
2. ASR-WER [Unchanged]: This metric evaluates the alignment between text and speech responses. We have corrected the errors that occurred during the previous ASR process.
3. UTMOS [Unchanged]: This metric assesses the quality of the generated speech.
4. WPS (Words Per Second) [New]: This metric measures the speaking rate of the generated speech, which is calculated as the average number of words per second in the generated speech.
5. Latency [Updated]: Latency refers to the total delay between the user's speech input and the moment they hear the speech response. This can be further divided into the decoding latency of the LLM and the latency introduced by the Vocoder or TTS model.

Thank you for providing your valuable and constructive feedback! Please find below the responses to each comment.

---

[Q1] Comparison with autoregressive TTS models

We respectfully disagree with your statement that "Autoregressive TTS models naturally support streaming decoding." Autoregressive TTS models with decoder-only architectures (e.g., CosyVoice, VALL-E) can perform streaming speech generation when the input text is complete. However, when the input text arrives incrementally, such as in the case of LLM output streams, we argue that autoregressive TTS models cannot inherently support streaming decoding.

For example, when the first word $X_1$ arrives in the text stream, an autoregressive TTS model decodes it into $[Y_1, Y_2, Y_3]$. When the second word $X_2$ arrives, there are two possible strategies:

1. Append $X_2$ after $X_1$, which changes the context and necessitates re-encoding $[Y_1, Y_2, Y_3]$. This results in the need to re-encode all target speech positions each time a new word arrives.
2. Append $X_2$ after $Y_3$, feeding $[X_1, Y_1, Y_2, Y_3, X_2]$ into the model. This introduces a mismatch with the training paradigm, as the model is typically trained with input-output sequences in the form $[X_1, ..., X_n, Y_1, ..., Y_m]$.

Thus, we believe that autoregressive TTS models are inherently unsuited for streaming decoding in scenarios where the input text is incrementally provided. In contrast, our non-autoregressive model decouples inputs and outputs, ensuring that outputs do not need to be re-fed into the model. This design makes it inherently compatible with streaming scenarios.

---

[Q2] It is not evident what benefits the end-to-end model offers over a straightforward cascaded solution where the text response is segmented into chunks and then fed into an offline TTS model.

Thank you for your question. We have updated two cascaded baseline systems with a streaming TTS model, which aligns with your suggestion of splitting LLM's text outputs into chunks and feeding them into a streaming TTS model. The following experimental results will demonstrate the superiority of our method compared to the cascaded system.

---

[Q3] Lacking speaker generalization.

Thank you for your suggestion. As you pointed out, the speech instructions in our training set were synthesized using CosyVoice, which includes only two standard voices (male and female). However, since our speech encoder (whisper) remains frozen throughout training, its inherent ability to handle multi-speaker speech remains intact.

To verify this, we synthesized a new training and evaluation dataset using the fish-speech-1.4 model, with a randomly selected voice for each synthesis. This ensured that all data samples have unique and non-overlapping voices. We then trained a new model on this dataset while keeping all other settings unchanged. Finally, we evaluated the model on two test sets: the single-speaker test set synthesized by CosyVoice (the same test set used in other experiments) and the multi-speaker test set synthesized by fish-speech-1.4. The results are presented in the table below.

Comparing Exp1 and Exp2, we observe that even the model trained on only two standard voices performs reasonably well on the multi-speaker test set, with some improvement observed (possibly due to differences in synthesis accuracy between CosyVoice and FishSpeech). However, comparing Exp2 and Exp4, we find that the model trained on multi-speaker data performs better on the multi-speaker test set (4.07 vs. 4.19). This demonstrates the benefits of incorporating voice diversity during training.

In summary, we agree with your point that training with multi-speaker data can enhance the model's robustness. However, we believe that the current model is already sufficient to handle real-world scenarios effectively.

Thank you for your quick and detailed response! We are delighted to hear that some of the concerns have already been resolved and are very grateful that you raised the score. Next, we would be happy to continue addressing some of your questions:

---

[Q1] Comparison with streaming TTS modules used in Moshi and Mini-Omni.

Thank you for your question. The working principle of Moshi and Mini-Omni involves adding additional heads or a local Transformer after the LLM to generate corresponding speech tokens. Therefore, the model operates by generating one text token and one speech token at each time step (for hierarchical speech encoding, it generates all layer tokens for the frame).

Ideally, we aim for the text and speech generation processes to be approximately synchronized, meaning that as the text is generated, the corresponding speech is also produced. This synchronization facilitates better alignment between the generated speech and text. However, one key factor is that speech sequences are typically much longer than text sequences. For example, let's assume the text length is 10 and the speech length is 50. In this case, Mini-Omni generates the text and the first 10 speech tokens in the first 10 time steps. After that, it still needs to autoregressively generate the remaining 40 speech tokens. During this process, the condition on the text response gradually weakens, which may result in the final speech response misaligning with the text response.

Moshi addresses this issue by adding padding (PAD) to the text during training (aligning text and speech using Whisper to construct text-speech alignment information). By inserting PADs between words, it ensures that the number of text tokens corresponds more closely to the number of speech tokens per word. This enables synchronous generation of text and speech during decoding. However, this approach has two potential drawbacks:

1. During training, extracting text-speech alignment information to construct the dataset increases the complexity of data preprocessing.
2. Forcing the LLM to generate text responses with PADs disrupts its natural text generation mode, which might harm its inherent text generation capability.

In contrast, our approach first upsamples the LLM’s output states to ensure even alignment between text and speech. Then, the model uses CTC to adaptively learn the alignment relationships, naturally supporting synchronized generation of text and speech. We believe this offers an advantage over the aforementioned autoregressive models.

Thank you for providing your valuable and constructive feedback! Please find below the responses to each comment.

---

[Q1] Lack of novelty.

Thank you for your question. Our speech decoder draws inspiration from prior works that use CTC for speech generation, such as CTC-S2UT [1], NAST-S2x [2], and StreamSpeech [3]. Similar to [2] and [3], we also adopt a CTC-based model for streaming discrete unit generation. However, our work still differs from theirs in several key aspects.

1. All previous models utilize an encoder-decoder architecture, whereas we are the first to adopt a decoder-only architecture for CTC-based streaming unit generation. This design is more consistent with the architecture of large language models and is potentially easier to scale up.
2. Prior works have only validated the feasibility of CTC-based models on sentence-level machine translation tasks (average unit sequence length on CVSS Fr-En: 101.3). In contrast, we are the first to demonstrate the feasibility of using a CTC-based non-autoregressive model for generating long-sequence discrete units (average unit sequence length: 553.6).

Besides, to the best of our knowledge, we are the first to successfully integrate this architecture with LLMs and validate its effectiveness. Compared to prior speech-language models, such as SpeechGPT, our framework achieves lower training costs and higher quality. Therefore, while some submodules in our work resemble those in earlier studies, we believe that proposing this framework and demonstrating its effectiveness also represents a meaningful contribution.

---

[Q2] Questionable Experiment Results.

We sincerely thank you for pointing out potential issues in our experimental results. Based on your suggestions, we conducted a comprehensive review of our experiments and identified several key issues, which we have thoroughly corrected:

1. ASR-WER/CER: Regarding your observation of unusually high ASR-WER/CER scores of the Qwen2-Audio+TTS Model, we found this was due to an oversight. When calculating ASR-WER/CER, the Whisper model incorrectly truncated the speech to 30 seconds, resulting in the loss of content after the 30-second mark, which led to incorrect ASR-WER/CER scores. We have corrected this issue and conducted a complete re-evaluation.
2. ChatGPT Score: The evaluation prompt we previously used may have introduced bias. We have removed all potentially bias-inducing priors from the prompt and simply instructed the model to give a single score based on helpfulness, relevance, fluency, and suitability for speech interaction, without providing any explanations that might introduce bias (details in Appendix A of the revised PDF). We also conducted human evaluations following your suggestions, and the results are presented below.
3. TTS Model: Apologies for the previous unclear explanation, which may have caused misunderstandings. The VITS model we used is integrated within the Coqui TTS framework, which performs text segmentation before speech synthesis to ensure accurate synthesis even for long texts. In fact, the speech responses in our training dataset were also synthesized using this model. However, following your suggestion and to achieve a more comprehensive and meaningful comparison, especially in the streaming scenario, we have replaced the cascaded baseline systems with Qwen2-Audio/SALMONN + Orca, where Orca is an industrial TTS model that supports both streaming and offline speech synthesis. When using Orca for streaming speech synthesis, we need to set a word chunk size $\Theta$, which means that speech synthesis is triggered every time $\Theta$ new words arrive. In our experiments, we varies $\Theta$ within the range of $[1, 3, 5, 7, 9]$ to control the response latency of cascaded systems.

I thank the authors for their response.

While the response addressed many of my initial concerns, it also highlights a critical weakness of Llama-Omni.

As shown in Table 2, there is a significant gap (0.52) between the S2TIF and S2SIF scores of Llama-Omni, primarily due to the limitations of the adopted CTC speech decoder. In contrast, Orca demonstrates a decline of less than 0.1 in its ChatGPT Score, with acceptable latency as evidenced by Table 6. This suggests that integrating Llama-Omni (Text) with Orca could potentially yield a much better ChatGPT Score without compromising latency. Compared to this straightforward alternative, Llama-Omni is held back by its CTC speech decoder, rendering the proposed architecture impractical and lacking meaningful application.

Furthermore, another major contribution of the paper—the InstructS2S-200K dataset—also raises concerns. While the goal of creating responses tailored for speech interaction is commendable, the proposed method involves rewriting responses using an LLM. This raises the question: would converting the rewrite prompt into a system prompt and directly utilizing it during generation achieve similar results with the desired characteristics?

We conducted a comprehensive re-evaluation of all models in both offline and streaming scenarios. Specifically, we evaluate the model using the following metrics:

1. ChatGPT Score [Updated]: We use GPT-4o to rate the model's responses. Following the suggestions from reviewers ppgU and DXz7, we have revised the evaluation prompt (details in Appendix A of the revised PDF) to remove potential prior biases.
2. ASR-WER [Unchanged]: This metric evaluates the alignment between text and speech responses. We have corrected the errors that occurred during the previous ASR process.
3. UTMOS [Unchanged]: This metric assesses the quality of the generated speech.
4. WPS (Words Per Second) [New]: This metric measures the speaking rate of the generated speech, which is calculated as the average number of words per second in the generated speech.
5. Latency [Updated]: Latency refers to the total delay between the user's speech input and the moment they hear the speech response. This can be further divided into the decoding latency of the LLM and the latency introduced by the Vocoder or TTS model.

The numerical results under different latency conditions for the cascade systems (SALMONN + Orca, Qwen2-Audio + Orca) and LLaMA-Omni are as follows:

[SALMONN + Orca]

[Qwen2-Audio + Orca]

[LLaMA-Omni]

Note: The latency of LLaMA-Omni has slightly changed compared to the submitted version due to variations in machine performance. To ensure fairness, we reevaluated the latency of all systems under the same conditions.

[Q3] Average decoding time.

Thank you for your suggestion. As you mentioned, comparing the total decoding time is not meaningful due to differences in the response lengths of different models. Following your advice, we compared the average decoding time per token and per word for LLaMA-Omni and Qwen2-Audio+Orca in the streaming scenario. The results are shown in the table below:

Since decoding time consists of two components: the first is the decoding time of LLMs (LLaMA-Omni/Qwen2-Audio), which remains constant regardless of latency, and the second is the time required by the streaming vocoder or TTS, which varies depending on the unit/word chunk size. To provide a clearer understanding of the time allocation, we present the two components separately in the above table. As shown, although LLaMA-Omni’s LLM decoding involves both text and unit decoding, its average decoding time remains lower than the average time required by Qwen2-Audio for text decoding alone. This is attributed to the parallel generation capability of LLaMA-Omni's non-autoregressive speech decoder. When comparing the streaming vocoder to streaming TTS (Orca), we observe that the average decoding time for the vocoder is consistently lower across all latency conditions. These findings demonstrate the superior decoding efficiency of LLaMA-Omni.

---

[Q4] Limited speaker support.

Thank you for your suggestion. As you pointed out, the speech instructions in our training set were synthesized using CosyVoice, which includes only two standard voices (male and female). However, since our speech encoder (whisper) remains frozen throughout training, its inherent ability to handle multi-speaker speech remains intact.

To verify this, we synthesized a new training and evaluation dataset using the fish-speech-1.4 model, with a randomly selected voice for each synthesis. This ensured that all data samples have unique and non-overlapping voices. We then trained a new model on this dataset while keeping all other settings unchanged. Finally, we evaluated the model on two test sets: the single-speaker test set synthesized by CosyVoice (the same test set used in other experiments) and the multi-speaker test set synthesized by fish-speech-1.4. The results are presented in the table below.

Comparing Exp1 and Exp2, we observe that even the model trained on only two standard voices performs reasonably well on the multi-speaker test set, with some improvement observed (possibly due to differences in synthesis accuracy between CosyVoice and FishSpeech). However, comparing Exp2 and Exp4, we find that the model trained on multi-speaker data performs better on the multi-speaker test set (4.07 vs. 4.19). This demonstrates the benefits of incorporating voice diversity during training.

In summary, we agree with your point that training with multi-speaker data can enhance the model's robustness. However, we believe that the current model is already sufficient to handle real-world scenarios effectively.

---

References:

[1] CTC-based Non-autoregressive Textless Speech-to-Speech Translation.

[2] A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation.

[3] StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning.

From the experimental results, we observe the following:

1. Both the cascade systems and LLaMA-Omni can achieve latencies below 320ms (the average audio latency of GPT-4o) and allow latency control by adjusting hyperparameters. Across all latency conditions, LLaMA-Omni consistently achieves higher ChatGPT Scores compared to the cascade baseline systems.

2. In the offline scenario, the cascade systems achieve lower ASR-WER than LLaMA-Omni, demonstrating the superior performance of industrial TTS models. However, in the streaming scenario, as latency decreases, we observe a significant performance drop in the cascade systems compared to the offline scenario (evidenced by a decrease in ChatGPT Score and an increase in ASR-WER). In contrast, LLaMA-Omni shows minimal changes in ChatGPT Score and ASR-WER as latency varies, and even exhibits performance improvements in some cases.

3. In terms of speech quality, all systems experience a decline in quality as latency decreases, primarily due to an increase in discontinuities within the speech. LLaMA-Omni achieves speech quality comparable to that of SALMONN+Orca.

4. From the WPS metric, LLaMA-Omni's speech rate remains almost unaffected by latency changes. In contrast, the overall speech rate of the cascade systems decreases significantly under low-latency conditions, primarily due to increased pauses between word chunks, which significantly affects the overall rhythm and prosody of the speech. For example, at the lowest latency ($\Theta=1$), the overall speech rate drops to less than 60% of the offline scenario. This indicates that cascade systems face significant challenges under extremely low-latency conditions.

For the above results, we have included line charts in the revised PDF illustrating the variation of each metric across different latency conditions for all models. These charts provide a more intuitive comparison of the performance of different models. Please refer to the updated version of the paper. We have also provided audio samples at https://llama-omni.github.io/, allowing for a more intuitive understanding of the speech generated by different models under various latency conditions through listening.

We also conducted human evaluations to investigate human preferences in real-time speech interaction scenarios. Specifically, we performed side-by-side comparisons between LLaMA-Omni ($\Omega=40$, latency=347ms) and two cascade systems: SALMONN+Orca ($\Theta=3$, latency=352ms) and Qwen2-Audio+Orca ($\Theta=3$, latency=420ms). The comparisons evaluated the systems in terms of the helpfulness of the responses and the naturalness of the generated speech, using a win/tie/lose framework. The evaluation results are as follows:

From the results, we observed that LLaMA-Omni achieved higher win rates in both helpfulness and naturalness, demonstrating that LLaMA-Omni's responses better align with human preferences. More details about the human evaluation can be found in Section 4.6 of the revised PDF.

Apologies for the delayed response—I got sick for the past few days.

Regarding the archtecture

After reviewing the new experimental results, I believe they support my concerns that the current architecture generally underperforms compared to the more straightforward pipeline approach. This significantly weakens the overall contribution of the architecture.

Regarding the InstructS2S-200K

I reviewed the paper and noticed that the authors directly prompt the 70B model to generate responses without any filtering. This could lead to undesirable behaviors compared to the preference data provided in [1]. While the authors claim they can distill knowledge from the 70B model to the 8B model, I believe this ‘advantage’ may diminish as the size of the text-LLM scales. For instance, if the backbone LLM were switched to Llama-3-70B, directly using the same prompting approach would likely produce the same results.

Besides, fine-tuning LLMs with data generated by another model may lead to catastrophic forgetting of their ability. While the author conduct experiments in response to Reviewer 4A7w's concern, MMLU is not reliable for evaluating forgetting [2]. A more percise evaluation should be conduct on instruction following benchmarks (eg. AlpacaEval). In contrast, the prompting approach does not disturb the model parameter. This limits the applicability of this dataset on other Speech-LLMs.

Conclusion.

Overall, I think this paper is more engineering and does not bring refreshing new insights. At the same time, its performance does not have advantage compared to the straight forward pipeline approach. Therefore, I think I don't want to push for acceptance and tend to maintain my current score.

BTW, although I acknowledge the efforts in the rebuttal of the authors, I don't think it is a good idea to submit an unfinished manuscript and redo almost all the experiment and analysis during the review stage.

[1] Cho et al. Speechworthy instruction-tuned language models. EMNLP 2024.

[2] Yang, Zhaorui, et al. "Self-distillation bridges distribution gap in language model fine-tuning." arXiv preprint arXiv:2402.13669 (2024).

Thank you for the explaination.

I agree the weird behavior of Cascade model could be caused by the 'unnatural rhythm' of Orca, as the provided cases shows wrong ASR results generally appears at the word chunk boundaries (every 5 words). I believe this help address my concern and demonstrates the advantage of End-to-end approaches in audio generation.

I'll raise my score to 6. Good luck!

Thank you for your quick reply! We are glad that our response has addressed some of your concerns. Regarding why the ASR-WER is higher when $\Theta = 5$ compared to when $\Theta$ is smaller, especially when $\Theta = 1$, we have also noticed this issue. After reviewing some examples, we believe the reason is that when $\Theta = 1$, the cascaded system is essentially "reading word by word", which may reduce the difficulty for ASR and result in a lower WER. In contrast, with $\Theta = 5$, there are instances where incoherence or errors occur between words. Some cases are as follows:

Case 1:

Text: The Northern Lights are caused by charged particles from the sun interacting with the Earth's magnetic field and atmosphere.

$\Theta=1$: The northern lights are caused by charged particles from the sun interacting with the Earth's magnetic field and atmosphere.

$\Theta=5$: The northern lights are Cosby charged particles from Thiessen interacting with the Earth's magnetic field and atmosphere.

$\Theta=9$: The northern lights are caused by charged particles from Sun interacting with the Earth's magnetic field and atmosphere.

Case 2:

Text: Yes, human blood is typically a deep red color, but it can appear more purple or blue depending on the oxygen levels and the individual's health.

$\Theta=1$: Yes, human blood is typically a deep red color, but it can appear more purple or blue depending on the oxygen levels and the individual's health.

$\Theta=5$: Yes, human blood is typicalia deep red color. Budit can appear more purple or blue depending on the oxygen levels and the individual shelf.

$\Theta=9$: Yes, human blood is typically a deep red color, but it can appear more purple or blue depending on the oxygen levels and the individual's health.

An interesting observation is that when $\Theta = 5$, many errors occur at the 5th and 10th words, which coincides with the word chunk boundaries. This suggests that most of the errors in the streaming TTS system are caused by the incoherence between word chunks. When we further increased the word chunk size to $\Theta = 9$, we found that many of the errors were resolved, which aligns with the intuition that larger word chunks lead to better performance.

We also investigated why Qwen2-Audio+Orca did not exhibit similar behavior at $\Theta = 1$. We found that for outputs consisting solely of English words, $\Theta = 1$ can achieve a lower WER than $\Theta = 5$, similar to what we observed with LLaMA-Omni and SALMONN. However, for outputs containing special characters, such as newline characters or Chinese characters, when these are split into individual chunks, the model often generates strange outputs, leading to higher WER. This may explain why Qwen2-Audio+Orca performs worse at $\Theta = 1$ compared to $\Theta = 5$ overall. This also highlights a potential advantage of end-to-end systems over cascade systems: there is no need to process intermediate text outputs to bridge the gap between LLM outputs and TTS inputs.

We hope our response can address your concerns. Once again, thank you for your efforts in helping improve our work.

Thanks to the authors for their quick response.

I think that 'prompting only works for cascaded system' is a good point, which I have missed. And I've listened the demo. Llama-Omni does sound more natural. I'll raise my score accordingly.

Regarding the comparison between Cascade and Llama-Omni, I find one thing confusing—the ASR-WER for Llama-Omni (text) + Orca is highest when $\Theta=5$. The same problem also happens in SALMONN + Orca. The high ASR-WER leads to lower performance and may narrow the gap between Cascade and Llama-Omni. Usually, one would expect a streaming TTS to perform better when the chunk size is larger. Is there any explaination on this phenomenon?

For the above results, we have included line charts in the revised PDF illustrating the variation of each metric across different latency conditions for all models. These charts provide a more intuitive comparison of the performance of different models. Please refer to the updated version of the paper. We have also provided audio samples at https://llama-omni.github.io/, allowing for a more intuitive understanding of the speech generated by different models under various latency conditions through listening.

We also conducted human evaluations to investigate human preferences in real-time speech interaction scenarios. Specifically, we performed side-by-side comparisons between LLaMA-Omni ($\Omega=40$, latency=347ms) and two cascade systems: SALMONN+Orca ($\Theta=3$, latency=352ms) and Qwen2-Audio+Orca ($\Theta=3$, latency=420ms). The comparisons evaluated the systems in terms of the helpfulness of the responses and the naturalness of the generated speech, using a win/tie/lose framework. The evaluation results are as follows:

From the results, we observed that LLaMA-Omni achieved higher win rates in both helpfulness and naturalness, demonstrating that LLaMA-Omni's responses better align with human preferences. More details about the human evaluation can be found in Section 4.6 of the revised PDF.

---

[Q3] The "response latency" used in this paper is misleading.

Apologies for the misunderstanding. In our paper, latency does not only refer to the streaming speech decoder's delay but comprehensively accounts for the total delay from when the user finishes their speech input to when they hear the speech response. This includes the decoding latency of LLaMA-Omni (covering both the LLM's text generation and the speech decoder's speech generation) as well as the latency of the vocoder synthesizing the speech. We have clarified this in the table above and in the revised paper, where we explicitly list the latency for each step.

---

[Q4] The audio quality of the proposed method is not good from the MOS score and is not compared with any external works.

We have added the UTMOS scores for the cascade systems in above tables, whose speech is generated by an industrial TTS model. The results show that LLaMA-Omni achieves UTMOS scores comparable to or slightly higher than SALMONN + Orca across different latency conditions, indicating that the speech quality generated by LLaMA-Omni is satisfactory.

---

[Q5] ASR-WER may not serve well as a way to evaluate the intelligence of the generated speech.

Thank you for your suggestion. ASR-WER is a commonly used metric in TTS that evaluates whether the content of the generated speech aligns with the corresponding text. While speech discontinuities may not be well reflected in the ASR-WER metric, they can be perceived through other metrics such as UTMOS and human evaluations. We believe these metrics together provide a comprehensive evaluation of the generated speech from multiple dimensions, with ASR-WER being one of the most reliable methods for assessing speech content accuracy.

---

References:

[1] Shen et al., 2017. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions.

[2] Ma et al., 2020. Incremental text-to-speech synthesis with prefix-to-prefix framework.

[3] Stephenson et al., 2020. What the future brings: Investigating the impact of lookahead for incremental neural tts.

[4] Saeki et al., 2021. Incremental text-to-speech synthesis using pseudo lookahead with large pretrained language model.

[5] Saeki et al., 2021. Low-latency incremental text-to-speech synthesis with distilled context prediction network.

[6] Liu et al., 2022. From start to finish: Latency reduction strategies for incremental speech synthesis in simultaneous speech-to-speech translation.

[7] Mohan et al., 2020. Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning.

[8] Chen et al., 2021. Speech-T: Transducer for Text to Speech and Beyond.

[9] Dekel et al., 2023. Speak While You Think: Streaming Speech Synthesis During Text Generation.

The numerical results under different latency conditions for the cascade systems (SALMONN + Orca, Qwen2-Audio + Orca) and LLaMA-Omni are as follows:[SALMONN + Orca]

[Qwen2-Audio + Orca]

[LLaMA-Omni]

Note: The latency of LLaMA-Omni has slightly changed compared to the submitted version due to variations in machine performance. To ensure fairness, we reevaluated the latency of all systems under the same conditions.

From the experimental results, we observe the following:

1. Both the cascade systems and LLaMA-Omni can achieve latencies below 320ms (the average audio latency of GPT-4o) and allow latency control by adjusting hyperparameters. Across all latency conditions, LLaMA-Omni consistently achieves higher ChatGPT Scores compared to the cascade baseline systems.

2. In the offline scenario, the cascade systems achieve lower ASR-WER than LLaMA-Omni, demonstrating the superior performance of industrial TTS models. However, in the streaming scenario, as latency decreases, we observe a significant performance drop in the cascade systems compared to the offline scenario (evidenced by a decrease in ChatGPT Score and an increase in ASR-WER). In contrast, LLaMA-Omni shows minimal changes in ChatGPT Score and ASR-WER as latency varies, and even exhibits performance improvements in some cases.

3. In terms of speech quality, all systems experience a decline in quality as latency decreases, primarily due to an increase in discontinuities within the speech. LLaMA-Omni achieves speech quality comparable to that of SALMONN+Orca.

4. From the WPS metric, LLaMA-Omni's speech rate remains almost unaffected by latency changes. In contrast, the overall speech rate of the cascade systems decreases significantly under low-latency conditions, primarily due to increased pauses between word chunks, which significantly affects the overall rhythm and prosody of the speech. For example, at the lowest latency ($\Theta=1$), the overall speech rate drops to less than 60% of the offline scenario. This indicates that cascade systems face significant challenges under extremely low-latency conditions.

Thank you for providing your valuable and constructive feedback! Please find below the responses to each comment.

---

[Q1] Comparison with research related to streaming TTS.

Thank you for pointing this out. As you mentioned, our speech decoder performs the task of streaming TTS. Following your suggestion, we investigate research on streaming TTS (also known as incremental TTS) and find that most incremental TTS approaches adopt model architectures similar to offline TTS models (e.g., Tacotron 2 [1]), and employ a fixed lookahead strategy, such as waiting for a few future words [2-3] or using a language model to predict several upcoming words [4-6]. A few other works have also explored using reinforcement learning [7] or Transducer architectures [8] to implement incremental TTS. Compared to these works, we believe our approach has the following advantages:

1. Previous incremental TTS models typically adopt the traditional encoder-decoder architecture, whereas our model employs a decoder-only Transformer structure, unified with the LLM. Based on the success of such architectures in language modeling, we believe this design is more suitable for scaling up.
2. Unlike the fixed lookahead strategies used in most prior works, we leverage the blank symbol from CTC to represent the "WAIT" action. This enables our model to adopt a more flexible and adaptive strategy for determining the timing of outputs.
3. Compared to methods based on reinforcement learning and Transducer architectures, our model is easier and more stable to train. Reinforcement learning approaches often face challenges with unstable training, while Transducer models require significant GPU memory due to the computational complexity of the loss function. In contrast, CTC-based training is more memory-efficient and stable.
4. Many previous incremental TTS works assume that the input has already been converted into phonemes. However, as pointed out by [9], a practical system also requires a frontend module to streamingly convert text into phonemes. Our work integrates all these steps into a unified end-to-end model, avoiding error propagation introduced by intermediate steps.

We have added discussions on streaming TTS works in the Related Work section of the revised PDF. Additionally, we have replaced the TTS model in the baseline system with a streaming TTS model. The following part will present the corresponding experimental results.

---

[Q2] Comparison with cascaded baseline systems with streaming TTS.

Following your suggestion, we replace the TTS component of cascaded baseline systems with an industrial TTS model, Orca, which supports both streaming and offline speech synthesis. When using Orca for streaming speech synthesis, we need to set a word chunk size $\Theta$, which means that speech synthesis is triggered every time $\Theta$ new words arrive. In our experiments, we varies $\Theta$ within the range of $[1, 3, 5, 7, 9]$ to control the response latency of cascaded systems.

We conducted a comprehensive re-evaluation of all models in both offline and streaming scenarios. Specifically, we evaluate the model using the following metrics:

1. ChatGPT Score [Updated]: We use GPT-4o to rate the model's responses. Following the suggestions from reviewers ppgU and DXz7, we have revised the evaluation prompt (details in Appendix A of the revised PDF) to remove potential prior biases.
2. ASR-WER [Unchanged]: This metric evaluates the alignment between text and speech responses.
3. UTMOS [Unchanged]: This metric assesses the quality of the generated speech.
4. WPS (Words Per Second) [New]: This metric measures the speaking rate of the generated speech, which is calculated as the average number of words per second in the generated speech.
5. Latency [Updated]: Latency refers to the total delay between the user's speech input and the moment they hear the speech response. This can be further divided into the decoding latency of the LLM and the latency introduced by the Vocoder or TTS model.

Dear Reviewer MWRj,

Thank you for raising the score! Once again, we sincerely appreciate the valuable feedback you provided, which has been incredibly helpful in improving our work.

Best,

Authors