Aligning Spoken Dialogue Models from User Interactions

We are thankful for your time and careful feedback. We answer specific points below:

Question

In Equation (2), [...], a sequential relationship.

The model’s text and audio must have some dependency pattern, otherwise the two streams would quickly have different content.

Other patterns have been explored in the literature [1], such as first generating all text tokens, then all audio tokens, however it limits the possibility to interrupt at any time (if the model gets interrupted in the middle of generating the audio, we would need to backtrack to erase the text that was never voiced), and increases latency.

Co-generating both audio and text tokens in an interleaved and auto-regressive fashion has no such limitations. While we only penalize the text contribution to the likelihood of a trajectory in the DPO loss, the model is trained to be fed with both the text and audio part of a trajectory, and would be largely out of domain if only fed with text.

Concerns on Methods & Evaluation

The Methods section introduces [...] Moshi and DPO.

We will move the content on Moshi to a different section, and reorganize the DPO part to focus more on our adaptation to the multistream setting. We hope this revision could make the methods section clearer, while keeping the paper self-contained.

The proposed method has limited innovation [...] preference data pairs.

We believe that the key observations are more nuanced:

• To the best of our knowledge, this is the first work studying how to enhance full-duplex spoken dialogue models with large-scale interaction data.
• How to construct and mix preference pairs from multi-turn dialogues itself is not straightforward, especially given that spoken conversations present unique characteristics (e.g. overlaps and silences), have a different distribution than the written text (e.g., more concise and simple sentences), with a much higher number of turns. We believe that our designs and reported results can be interesting for the community and practitioners.
• Through extensive experiments, we show that using synthetic voice data can still significantly improve model performance. This offers several practical advantages, such as preserving privacy and mitigating the challenges of collecting extensive human speech corrections in real-world applications (much harder than for text).

Although the paper is about aligning full-duplex [...] models.

We agree that full-duplex spoken dialogue models exhibit a wide range of interesting phenomena, from the semantic content to the temporal dynamics, at turn-level and dialogue-level.

As assessing conversational dialogue models is a complex task [2], our work mainly aims to improve the quality of the content (in particular, the turn-level factual correctness, safety, the high-frequency timing problems, and the multi-turn dialogue-level quality) in the spoken dialogue, that we disentangled from other specific phenomena that require more fine-grained and targeted assessments.

Evaluation of full-duplex models is also a very nascent area. We are aware of some very recent work on evaluating the turn-taking behaviour [3, 4], but they are not fully open-sourced yet. We agree that the evaluation and improvement of temporal dynamics for conversational models are an important and very timely direction for future work.

Experimental Designs & Other Comments

Experimental Designs Or Analyses

Thank you for the feedback. We will refine the flow of the experimental section to better link with the results. For the data ratios, we will clearly update them in Table 2.

In particular:

• Type-A corresponds to the 20% with content-only issues.
• Type B+C corresponds to the 57% with timing-only issues.
  • Within the timing-only issues:
    • Type-B: 18% is the model cutting the user.
    • Type-C: 72% is the model not answering within appropriate time.

Line 246: No corresponding content in Appendix C

The reference refers to Appendix C of the cited paper [5] for details on TTS (on page 56). We will revise to make the point clearer.

Equation 1 and Equation 3

Thank you for the feedback. We wanted to emphasize after changing the notation of the policy, but we agree that the equation is redundant and will remove it.

Please let us know if the answers addressed your questions, and if we can address further questions you might have. Thanks again!

[1] Nachmani, E., et al. "Spoken question answering and speech continuation using spectrogram-powered llm." ICLR, 2024.

[2] See, A., et al. "What makes a good conversation? how controllable attributes affect human judgments." ACL, 2019.

[3] Arora, S., et al. "Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics." ICLR, 2025.

[4] Lin, G., et al. "Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities." arXiv, 2025.

[5] Défossez, A., et al. "Moshi: a speech-text foundation model for real-time dialogue." arXiv, 2024.

We thank the reviewer for the thorough feedback, and for acknowledging that our work presents a novel contribution for the alignment of voice interaction models.

Questions & Concerns

Loss of context-based prosody

We agree that discarding the original audio (for privacy) loses some audio information, however:

• we want to clarify that the TTS is done for the user's audio, and for Moshi we used its own generated tokens (except for the generated preferred response, also synthesized);
• as the resynthesized speech retains the original timestamps, some aspects of the prosody are kept, such as the rhythm and speech rate;
• for the axes we focused on in the work, the original audio is not absolutely necessary.

Also the original Moshi paper [1] uses synthetic generation for the instruction stage.

Given that in a number of regions in the world, strong protection laws prevent or limit the recording or storage of private and biometric attributes such as a person's voice, our method presents important applications for improving speech systems in a privacy-preserving manner, while respecting local legislations.

The model choice is because Moshi was the only available open-source full-duplex speech-to-speech model when we conducted the project, and for cost/feasibility reasons. We agree that more work is required to further verify the extension to other models, and we will discuss this point in the limitations. We note that the method for building the dataset mixes from multi-turn dialogues is model-independent.

Effect of reinforcement learning on text-to-speech

We computed the WER (lower is better) between each model’s text tokens and Whisper’s transcriptions, before and after alignment, on our human evaluation dataset.

For Moshi (the "matched" voice), after alignment, we observed a slight improvement. For M-Alt-Vox (the "mismatched" voice, i.e. model with a voice different from our synthesized preference data), WER rose a bit, suggesting that adapting to a voice with different characteristics may show mixed effect.

Note that because how the conversations are conducted, they are not constrained to be the same, so the numbers indicate an aggregated trend which confirms that pronunciation accuracy is overall preserved during the alignment process.

Methodological novelty

For the method, we want to clarify a few aspects:

• How to construct and mix preference pairs from multi-turn dialogues is not a straightforward adaptation of the textual approach, given that spoken conversations present unique characteristics (e.g. overlaps and silences, more concise, more turns). We believe that our designs and results can be interesting for the community and practitioners.
• Our approach shows that using synthetic voices can be effective, offering practical advantages such as preserving privacy and mitigating the challenge of scarcity for speech (feedback) data.

There are several motivations for leveraging live interaction data, instead of directly synthesizing from LLM-written dialog scripts:

• While synthetic data could approximate "typical" timing, real interactions yield more diverse and realistic phenomena: e.g. mid-sentence clarifications and stoppings, hesitations, silences when thinking, abrupt topic shifts, etc. In practice, we observed that LLMs struggle at providing or assessing realistic timings, e.g., for overlapping speech. By preserving this information, we could keep some aspects of the prosody, speech rate and rhythm.
• We observe that current LLMs are better at generating written text than content resembling spoken conversations, that is more concise, with more turns, potentially overlapping. This echoes observation in recent work [2]. We observe it's possible to prompt the model to imitate to some extent, but it's harder to generate long, coherent, diverse and speech-like dialogues.
• Crafting discussion topics and timings can induce biases from the researchers and/or the LLM used. Also, available LLMs will usually refuse to generate synthetic training scripts that would cover adversarial or unsafe topics.

We note that the original Moshi model was trained with LLM-written dialog data, synthetized to speech [1], however our organically collected usage data shows a number of failure points. We show leveraging this data for alignment provides additional gains over the original synthetic instruct approach.

Release plan

Because of privacy concerns, we don't plan to release the materials at this point.

Acoustic aspects

We provide demo samples in an anonymous link for the acoustic aspects.

Please let us know if we addressed your questions. Thanks!

[1] Défossez, A., et al. "Moshi: a speech-text foundation model for real-time dialogue." arXiv, 2024.

[2] Cho, H., et al. "Speechworthy instruction-tuned language models." EMNLP, 2024.

We thank the reviewer for the positive assessment and appreciate the insightful and careful feedback. We respond to the points below.

Questions

section 3.3: [...] how do you derive preference data from this? [...] unclear to me.

Thank you for the question. For content-related problems: we feed the conversation history context, the problematic reply, the LLM judge's feedback and the instructions for proposing a response into Mistral Large 2, which generates the improved reply and becomes the preferred response.

The conversation history context starts from the beginning of the conversation, up to and including the user's last response before the model's problematic reply. The LLM judge's feedback includes identified issues along the axes specified in Subsection 3.3, paragraph "Problematic reply identification".

For timing-related issues: if the problem is the model interrupting the user, then the preferred response will be given after the user finishes their utterance. If the semantic content of the initial response is adequate, we keep the same response; otherwise, the model needs to propose a response. If the problem is the model not answering the user, we similarly generate a proper reply, which we put directly after the user’s request.

We will add more clarifications on this part in the paper.

tab.1. incorporating audio tokens for DPO does not really help [...] discussed

The preferences we curated mainly concern semantic aspects and the model's temporal behaviour (e.g., not answering to the user). The actual acoustic contents are not labeled as "preferred/dispreferred". We conjecture that forcing the alignment objective to consider the full audio token probability could introduce noise, harming the model's performance. Focusing on text tokens alone was more stable.

section 5. Aboutr RQ #3 [...] Why would this be problematic ? Especially if incorporating audio tokens for DPO does not really help alignment [...] to be problematic

The preference data uses TTS to resynthesize the user's voice, and for the model side, we use the audio tokens of Moshi itself (except for the preferred response, because we have no existing audio tokens, we also resynthesized using the model's voice). We use this same dataset and didn't resynthesize the audio data when aligning M-Alt-Vox.

During the alignment stage, even if the final objective optimizes the text tokens, the model is still fed with the audio tokens for context. A different voice would create a shift in the context distribution.

In practice, if we fully resynthesize the data with M-Alt-Vox's voice, then we conjecture that we won't see this issue. But this implies a heavier pipeline, and the transfer experiment was to test to the extent to which we can reuse the already synthesized data, and reduce the cost of transferring across voices.

Concerns

Potential reasons for rejection [...] publicly available.

We thank the reviewer for inquiring about data collection.

We collected aggregated data in a privacy-preserving and minimalist manner from organic users of our deployed system (not from specially recruited participants) over a two-week period, following a standard user agreement. This is similar to protocols in past work [1, 2]. This approach yields more authentic, diverse data that can better reflect the interests and feedback.

To protect privacy, we asked users not to share personal data, and in practice, no personally identifying details were retained. Sensitive voice information was never accessed, and there's no human listening to the recorded conversations. Researchers don't have access to any sensitive and personally identifiable information, including vocal attributes.

We promptly discarded original audio after transcribing them into text and allowing a download period for the user. We may lose paralinguistic information, but for the purpose of this work, we choose to only keep the minimal amount of information needed. Our data practices are for instance GDPR-compliant and adhere to principles of minimal, privacy-preserving collection.

Note that the examples in the Appendix are from our human evaluation study, explicitly consented.

As for the preference pairs, they include multiple flagged points from the original multi-turn dialogues. We detect problematic responses in the dialogue, keep the initial problematic response and sample from the other problematic responses if there are any. We will add more information about the distribution in the Appendix (e.g. number of turns).

Because of privacy concerns, we don't plan to release the data at this point.

Please let us know if the answers addressed your questions, and if we can address further questions you might have. Thanks again!

[1] Ram, A., et al. "Conversational ai: The science behind the alexa prize." arXiv, 2018.

[2] Shuster, K., et al. "Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage." arXiv, 2022.

We appreciate that the reviewer finds our work valuable, and thank the reviewer for the constructive remarks. We answer specific points below.

Question

376: It wasn't clear to me why the preference data doesn't fully transfer between different voices [...] this problem.

The preference data uses TTS to resynthesize the user's voice, and for the model side, we use the audio tokens of Moshi itself (except for the preferred response, because we have no existing audio tokens, we also resynthesized using the model's voice). We use this same dataset and didn't resynthesize the audio data when aligning M-Alt-Vox.

During the alignment stage, even if the final objective optimizes the text tokens, the model is still fed with the audio tokens for context. A different voice would create a shift in the context distribution.

In practice, if we fully resynthesize the data with M-Alt-Vox's voice, then we conjecture that we won't see this issue. But this implies a heavier pipeline, and the transfer experiment was to test to the extent to which we can reuse the already synthesized data, and reduce the cost of transferring across voices.

Concerns

A minor issue is that the evidence is limited to a single model (Moshi) [...] in general.

The model choice is because Moshi was the only available open-source full-duplex speech-to-speech model when we conducted the project, and for cost/feasibility reasons. We agree that more work is required to further verify the extension to other models, and we will discuss this point in the limitations. We note that the method for building the dataset mixes from multi-turn dialogues is model-independent.

The main points of weakness concern the extensive reliance on transcriptions and TTS [...] This likewise likely degrades the usefulness of this data.

We agree that discarding the original audio loses some audio information, however:

• we want to clarify that the TTS is done for the user's audio, and for Moshi we used its own generated tokens (except for the generated preferred response, also synthesized);
• as the resynthesized speech retains the original timestamps, some aspects of the prosody are kept, such as the rhythm and speech rate;
• for the axes we are focused on in this work, the original audio is not absolutely necessary.

Given that in a number of regions in the world, strong protection laws prevent or limit the recording or storage of private and biometric attributes such as a person's voice, our method presents important applications for improving speech systems in a privacy-preserving manner, while respecting local legislations. For more details on data protection, we kindly refer the reviewer to our reply to reviewer 2ifX.

We also want to note that according to the original Moshi paper [1], the instruction stage also uses synthetic generation.

If I understand correctly, human evaluation seems to also rely entirely on transcriptions [...] for this design.

We agree that relying on transcriptions with timestamps for the human evaluation has limitations, and this was mostly a logistical choice, for:

• (1) disentangling and reducing the cognitive load of the annotators;
• (2) focusing on the dialogue-level conversation content.

For (1), having evaluators or speakers score conversations after listening to the full audio (that can be up to 2min+) can introduce confounding factors, such as the memory ability to correctly recall details after the fact. By using transcriptions, we ensured annotators could consistently review entire multi-turn dialogues after the conversation concluded.

For (2), our primary alignment goal was to improve the dialogue content, but the automatic metrics only provide assessment for single-turn conversations. With human evaluation, we wanted to evaluate the dialogue-level quality across multiple turns, including aspects such as consistency and transitions across turns.

We will emphasize this as a limitation, and agree that enriching human evaluation with, for instance, direct listening tests or hybrid assessments could capture a broader spectrum of spoken dialogue characteristics.

Additionally, the preference data is automatically generated using and LLM, which is a practical choice, [...] regards to quality.

For the validation of the LLM-generated preference data, we manually curated a small held-out validation set over the axes we defined to assess the model's responses (e.g., helpfulness, safety, factuality), and inspected the automatically generated preference responses. As a practical choice, the design followed a heuristic approach trying to cover diverse failure modes. We agree that it would be interesting for future work to conduct more fine-grained investigation.

Please let us know if the answers addressed your questions, and if we can address further questions you might have. Thanks again!

[1] Défossez, A., et al. "Moshi: a speech-text foundation model for real-time dialogue." arXiv, 2024.