Parrot: Seamless Spoken Dialogue Interaction with Double-Channel Large Language Models

1. The writing of this paper is very poor, with some tables and figures not being referenced in the main text, and inconsistent statements in the context, making the entire article very difficult to understand. (see Questions for details)

2. The structure design of the model and "next-token-pair prediction" paradigm are not well-motivated, as there is a significant gap between training and inference stages.

   • The author inputs listening tokens and speaking tokens as pairs into the LLM, which doubles the context length. LSLM [1] has demonstrated the effectiveness of modeling double-channel through embedding fusion and is more efficient in context. Therefore, the author needs to prove the effectiveness of their modeling approach
   • In the training process, the prediction of the next token depends on the previous predicted tokens. But in the inference process, according to section 3.3, listening tokens are sent to LLM in chunks. At this point, predicting the next speaking tokens no longer depends on the previous predicted speaking tokens. This inference method will disrupt the casual modeling during training.

3. The lack of evaluation details.

   • In section 4.5.1, LLAMA-Omini and SpeechGPT do not have the ability to interrupt and pause. How does Reflective Pause and Interruption caculated?

   • In section 4.5.2, all these evaluation metrics cannot reflect the linguistic quality of the model. For an open-ended instruction-following task, there is no evaluation of response quality.

   • In section 4.6, the evaluation of speech-to-text tasks, such as ASR and ST, is discussed. However, this model is a textless speech-to-speech model and does not have text generation capability, so it is unclear how these tasks are evaluated.

4. Some results are very strange, which it‘s hard for me to believe that it's real.

   • According to table 1, the model is only trained on English-Only speech data, but in figure 8(c), the model can perform Chinese ASR task on aishell, SER task on MELD, and sound-related task on AIRBench-Sound. This is very strange.
   • In appendix A.4.2, the larger latency, the worse performance. There is no explanation for this strange results.

• Minimal Novelty: The framework essentially serves as a decoder-only adaptation of dGSLM combined with a textually pre-trained model (TWIST), offering limited technical innovation. This incremental change does not justify the need for a separate model or paper.

• No Human Evaluation: The absence of human assessment significantly limits the validity of the framework's claims about improving conversational fluidity and natural interaction, which are central to spoken dialogue applications.

• Lack of Established Benchmark Comparisons: Despite the existence of standard benchmarks like ZeroSpeech [1] and StoryCloze [2] for textless spoken language models, the paper does not include comparisons with these datasets. This omission raises concerns about the thoroughness of the experimental validation.

• Poorly Defined Evaluation Methodology: The evaluation details, especially for the reflective pause and interruption response accuracy (Section 4.5.1), are incomplete. Key information, such as the evaluation metric definitions like interaction accuracy, is missing, making it hard to verify the claimed improvements.

• Insufficient Explanation of Key Evaluation Components: See the comments below.

1. Zerospeech 2021 benchmark, https://arxiv.org/abs/2011.11588
2. StoryCloze, https://arxiv.org/abs/2305.13009

1. The paper lacks specific details on the cascaded system used for evaluation. DialogGSLM used a relatively weak cascaded system in their work, so it would strengthen this study if the authors evaluated a cascaded system with state-of-the-art ASR, LLM, and TTS models, such as Hugging Face's Speech-to-Speech (https://github.com/huggingface/speech-to-speech), for a fairer comparison. Additionally, latency in cascaded systems can be minimized by parallel threading each module, as demonstrated in the Hugging Face repository.

2. Given the primary objective of this work is to enable engaging and naturally fluid conversations, a human study would be valuable for a thorough evaluation—similar to assessments in text-based dialogue systems. When motivating their approach, the authors point out limitations in cascaded methods; more analysis is needed to clarify if these limitations genuinely impact user experience in human-AI conversations. User study could focus on user satisfaction compared to baseline systems as well as other auxiliary metrics such as naturalness of turn-taking and semantic coherence of response.

3. Although the code is public, it lacks sufficient documentation, and I was unable to run their demo. For example, an inference.py file appears to be missing.

4. The paper would benefit from further discussion on design choices:

a. I was curious about the choice to use a single codebook. Table 4’s synthesized audio quality results lack clarity on the dataset used. Multiple codebooks often improve results—did the authors experiment with this option?

b. The motivation for using next-token pair prediction, rather than multi-stream prediction as seen in Moshi, is also unclear. While streaming audio generation is effective, the paper doesn’t address the length limits for audio modeling. Given the potential length of audio sequences, does the model implement techniques to reduce sequence length?

c. The authors use only human-human conversation data for supervised fine-tuning. Adding human-AI conversation data, even synthetic, could be beneficial as there are nuance differences in how humans communicate with AI versus other humans.

5. Clarity

a. Section 4.5.1 is difficult to follow. My understanding is that the ground truth is generated using GPT-4. Did the authors verify this ground truth with human judgments? I would recommend the authors to clarify their methodology for generating and validating the ground truth data, and include this information in the paper.

b. Section 4.5.3 presents an interesting observation. Do the authors have any intuition as to why this occurs?

6. In addition to evaluating turn-taking properties, the paper could also benefit from evaluating "speaking-while-listening" capabilities (https://arxiv.org/pdf/2408.02622).

7. Did the authors assess whether the LLM undergoes catastrophic forgetting when fine-tuned on audio data? For example, is Parrot still capable of instruction-following or answering factual questions at a level comparable to LLAMA 3.1?

8. In section A.1, the paper makes claims about prior works that lack experimental or discussion-based support. For instance, the statement that RQ transformers reduce model efficiency is not substantiated. Unsubstantiated claims should be avoided, as they could lead to incorrect community conclusions.

Summary Of Weaknesses:

• Novelty and Clarity: In the contribution summary, the authors list (1) the Parrot model and its innovative pre-training and SFT pipeline, (2) the paradigm of double-channel spoken language modeling, and (3) the evaluations. In my opinion, textless spoken language models are not particularly rare, especially in machine translation tasks (e.g., Seamless: Multilingual Expressive and Streaming Speech Translation, UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units). Additionally, the pipeline and the double-channel modeling mechanism appear to be two perspectives on the same concept. It is difficult to consider the potential readiness for future exploration and evaluation as innovative contributions.
• Limitations: Besides the limitation regarding the inability to integrate audio tokens, further analysis of the audio tokenizer should be elaborated on (e.g., how it impacts computational efficiency or downstream inference latency). More case studies should also be included to demonstrate the effectiveness of this work, as A.5 seems unfinished.

1. In a few places, the paper misrepresents related work. Examples:

• "the academic community primarily utilizes open-sourced models (Zhang et al., 2023a; Xie & Wu, 2024; Rubenstein et al., 2023; Huang et al., 2024; Wang et al., 2023a; Nachmani et al., 2024; Wang et al., 2023b) following a cascading approach." For instance, [Rubenstein et al., 2023] and [Nachmani et al., 2024] are not cascaded models. They are also not open-sourced.
• "Much of the prior research has utilized the encoder-decoder architecture to enhance pre-training (Borsos et al., 2023; Lakhotia et al., 2021; Kharitonov et al., 2022; Polyak et al., 2021; Chen et al., 2023; 2022; Hsu et al., 2021; Zeghidour et al., 2022; Defossez et al., 2023; Agostinelli et al., 2023; Ao et al., ´ 2022; Tang et al., 2022; Wu et al., 2023).". ** The first three models are decoder-only LMs, and perhaps many others.

2. The evaluation of the proposed model is limited and is insufficiently described.

• S4.5.1 evaluates some pause-prediction accuracy metrics, but the metrics are not really introduced in the text. These metrics are evaluated on some synthetic data --- is there a reason a hold-out subset of Fisher is not used?
• AudioQA evaluation and comparison to AudioQWEN-2 is only mentioned in a single sentence without any discussion or description. At the same time, it does require some discussion. What are the metrics used? The AudioQA figure indicates some reasonable performance on CoVost2 and FLEURS, I assume better than AudioQWEN-2. However, the training data does not include speech-to-speech translation examples nor contains non-tier1 languages. Should we assume the model somehow picked it up from purely text base model?
• Audio-QWEN2 reports 90+% accuracy on VocalSound. In Figure 8c it is reported to be below 80%. What is the reason for the difference?

3. The paper should at least mention Moshi https://arxiv.org/abs/2410.00037