Q1, Why we classify Moshi as a encoder-decoder model

Thanks for raising this important and insightful question. While Moshi utilizes the Helium LLM as its temporal transformer—a decoder-only architecture—the inclusion of the RQ-Transformer introduces a spatial transformer component, which deviates from a standard decoder-only structure for several reasons (see Figure 3 on page 13 and Figure 1 on page 7 in Moshi’s paper). First, the RQ-Transformer employs a spatial transformer that encodes input structure information before the autoregressive decoding process. This encoding step explicitly transforms the input data, making it functionally similar to an encoder stage. Second, instead of directly processing tokens in a purely decoder-based manner, the RQ-Transformer constructs a context vector through the spatial transformer. This context vector resembles the latent representations typically computed by encoders in encoder-decoder architectures. Overall, Moshi’s architecture can be viewed as “two decoder-only architectures connected via an encoder-decoder module.” Consequently, unlike a purely decoder-only approach like our NTPP, Moshi requires maintaining two separate KV caches for its two decoder-only transformers. Therefore, Moshi is not a standard decoder-only architecture; instead, we classify it as an encoder-decoder model from a dual-channel modeling perspective. We refer the reviewer to our response to Reviewer muk3 for further details on this matter.

Q2:The major weaknesses of Moshi architectures compared to our NTPP and previous methods

We refer this reviewer to read the Experiments Designs or Analyses Q1 response for the Reviewer muk3 for further details about this question.

Q3: Further Clarifications on Table 1

We followed the evaluation settings for quality and statistical analysis of generated dialogues used in Moshi[^1]. Specifically, we selected 1,000 random 10-second prompts from the Fisher dataset and utilized NTPP and other baseline models to generate dual-channel speech continuations. For each prompt of NTPP, we generated 32 continuations across three different temperature settings [0.1, 0.5, 0.9], as temperature significantly impacts the results.

Q4: Inference Latency Explanations

We employ Voice Activity Detection (VAD) as the latency measurement benchmark, calculating system response delays exclusively during effective speech token segments. This approach more accurately simulates real-world conversational user experience. Our experimental results demonstrate that the latency improvement in multi-round dialogue primarily stems from our novel Next-token-pair Prediction mechanism: By independently calculating the attention values of dual-channel tokens, our method achieves fine-grained time alignment. In contrast, Moshi’s strategy of directly summing dual-channel token embeddings risks introducing inter-channel information interference.

Q5: Further Clarifications on Human Evaluation Experiments.

We followed the evaluation settings of Moshi and conducted the evaluation study with 25 annotators who possess native-level English proficiency. These annotators were recruited through a combination of academic networks and online platforms, ensuring a diverse range of backgrounds and experiences. Each annotator was thoroughly briefed on the evaluation criteria and the objectives of the study. We adapted the Mean Opinion Score (MOS) protocol, utilizing a 5-point Likert scale, to assess two key aspects: the Naturalness (N-MOS) of turn-taking and the Meaningfulness (M-MOS) of dialogue content. This approach allowed us to gather quantitative data on the perceived quality of the generated dialogues.

Q6: More Representative Demos.

We provide more dialogue examples at Demo Page, which illustrates the performance of our NTPP model across several key scenarios, including multi-turn dialogues, "Interruptions & Reflective Pause" evaluation, and speech continuation. The audio samples demonstrate how NTPP handles these different situations.

Q7: More Explanations on Two-Stage Training.

We describe our two-stage training process in Section 5.1 (Dataset). The first stage involves training a text-based LLM on single-channel speech token sequences. In the second stage, we introduce our newly proposed NTPP paradigm for dual-channel speech learning, building upon the SpeechLM obtained in the first stage. The ablation results in Figure 9 aim to support a simple claim: the first-stage training is essential before the second-stage NTPP training. We have further clarified this ablation study with a slightly modified figure Figure 9. This updated figure includes three curves: Full Two-Stage NTPP (NTPP), NTPP without the first stage (NTPP w/o-1), and NTPP without the second stage (NTPP w/o-2).

Please feel free to ask if you have any further question.

Claims and Evidence

Q1: The inefficiency of Encoder-decoder architectures

The efficiency of decoder-only model is supported by various literature. For example, FlashAttention (Dao et al., 2022) and Parallelized decoding (Kumar et al., 2020). They show how decoder-only models optimize memory and speed compared to encoder-decoder alternatives.

Q3: StoryCloze linguistic quality evaluation

Experiments Designs or Analyses

Q1: The comparison between Moshi, NTPP our other approaches.

We illustrate the comparison across different approaches through the following table:

In general, only our NTPP simultaneously possess the five important properties compared to other approaches. Moshi is not speaker-independent and not encoder-free. Additionally, it requires two KV caches. (1) This Moshi architecture is not “speaker-independent”. dGSLM indicates that the model should learn the joint distribution $p(s^{b},s^{a})$ instead of any conditional distribution $p(s^{b}|s^{a})$ or $p(s^{a}|s^{b})$. To satisfy this property, dGLSM adopts a two-tower transformer that follows the Siamese encoder-decoder architecture to estimate $p(s^{b},s^{a})$. Moshi and LSLM are only learning the conditional distribution $p(s^{b}|s^{a})$, which inherently has the generalization issue when we just simply switch speaker's roles. Also, this problem cannot be easily solved by training the paired data alternatively since learning $p(s^{b}|s^{a})$ and $p(s^{a}|s^{b})$ sequentially is not equivalent to learning $p(s^{b},s^{a})$ directly. (2) Moshi is not an encoder-free model since it adopts the RQ-transofmer architecture. Further clarifications are in the Reviewer Ubh3 Q1 response. (3) Moshi requires storing two separate KV caches for its two decoder-only transformers, reducing memory efficiency. In contrast, our NTPP, as a typical decoder-only architecture, requires only a single KV cache, making it more memory-efficient.

Q2: Two-Stage Ablation Studies

We refer this reviewer to read Reviewer Ubh3 Q7.

Q3 & 4: Figure 7 judge and more clarifications on baselines and evaluation metrics in Table 1 & Table 2.

To save space, we refer this reviewer to read Reviewer Ubh3 Q3.

Q6: Llama-Omni and SpeechGPT Evaluations.

Llama-Omni and SpeechGPT, as single-turn QA audio model, face incompatibilities with these real-time streaming multi-turn conversational benchmarks. Fisher and CANDOR require real-time handling of dynamic context shifts, interruptions (barge-ins), and mid-conversation pauses—capabilities inherently absent in VAD-dependent architectures.

Supplementary Material & Suggestions

We have updated the demo and code Page. Upon acceptance, we will release the full code and model weight publicly and create a well-organized project page for it. The reported digit for the cascaded model should be 4.3 instead of 1.3 (a typo). We will promptly correct this in the revised version.

Questions

Q1-Q2: RVQ and Vocoder Implementations.

Regarding the RVQ tokenizer and Vocoder, we followed the training settings as specified in Soundstream (Neil et al. 2021) and HifiGAN (Jungil et al. 2020), respectively. Additionally, we have updated the training code with detailed training procedures.

Q3: Cascaded models and Table 1 Explanation

Since the performance of the Cascaded model relies on VAD, we use "Cascaded model" as a general term to represent all non-interactive models, including multi-modal approaches and LLM-based cascading models. This class models do not directly learn the generation of dual-channel speech, so we do not introduce them in details. We will add short descriptions in the revised version. For the interpretation of NTPP subscripts in Table 1, please refer the Q3 response for the Reviewer Ubh3.

Q4-Q6: Dataset details, Moshi Evaluation Details and Tokenizer comparison:

We put the corresponding content in Page.

Q7: Inference Latency Analysis

To save space, we refer this reviewer to read Q4 response for the Reviewer Ubh3.

Q8: start tokens. Instruction format:

<bos><Model_0_0>...<Model_0_k><Human_0_0>...<Human_0_k><eos>

Q9: Figure 6 Discussions & Q10: Further explanations on positional encoding

We put Figure 6 discussions and further explanations on positional encoding in Page.

We hope the above responses address your concerns. If you have further questions, please feel free to ask.

Thank you for recognizing the value of our work. Here are our responses:

Q1: Textless Pretraining.

Thanks for your question. In our paper, "textless pretraining" specifically refers to the fact that neither the first-stage single-channel speech pretraining nor the second-stage dual-channel learning incorporates text alignment, unlike some existing SpeechLM models. We recognize the potential ambiguity and will provide a clearer explanation in the camera-ready version.

Q2: Section 5.5 Further Clarifications.

In our study, we aimed to assess the performance of existing audio models in real-world scenarios, particularly focusing on their ability to discern when to start and stop speaking. This is crucial for improving user experience, as current models often rely on Voice Activity Detection (VAD) to determine speech initiation, which can lead to interruptions whenever any sound is detected.

We generated 200 thoughtful pauses (e.g., reflective pause or conversational hesitations like "Hmm... let me think...") and 200 interruptions (e.g., abrupt interjections: “But wait—[interrupted]”) using GPT-4o, ensuring contextual diversity. Audio was synthesized via ChatTTS with explicit silence annotations. We compared the performance of our model against two other models, Cascaded and Moshi, by measuring the proportion of instances where the VAD model correctly identified whether speech should occur.

To ensure alignment between generated audio and labels, we employed human judge as the "gold standard". The closer this proportion was to human judgment, the better the model's performance. Figure 7 shows that our model is closer to human performance than Cascaded and Moshi.

We have also added a demonstration of the corresponding audio effects on the Demo Page.

Q3: About the two audio samples in demo page.

We provide more dialogue examples at Demo Page, which illustrates the performance of our NTPP model across several key scenarios, including multi-turn dialogues, "Interruptions & Reflective Pause" evaluation, and speech continuation. The audio samples demonstrate how NTPP handles these different situations.

Each audio sample on the demo page provides a clear representation of the output dynamics for each channel at any given moment. This setup allows users to observe how the model manages dual-channel audio, particularly in terms of when to initiate and cease speech. By listening to these samples, users can gain insights into the NTPP's ability to handle interruptions and reflective pauses effectively, showcasing its practical application in real-world interactions.

Q4：About Multi-channel system (more than channels)

Thank you for your suggestions. Expanding our approach to multi-channel conversation modeling is definitely on our roadmap, and our method can naturally be extended to this more complex setting. However, due to the lack of publicly available datasets, we are not yet fully prepared for this extension. Nonetheless, this remains a promising direction for our future work.