Reinforcement Learning Enhanced Full-Duplex Spoken Dialogue Language Models for Conversational Interactions

We sincerely appreciate your thorough review and valuable comments. Your expertise has significantly contributed to improving this work. Below we provide detailed responses to address your concerns.

Question1: Introducing human preference annotation in the learning loop
Thanks for your valuable question. We clarify that this work focuses on improving conversatioanl interaction accuracy (good/bad judgments) rather than subjective preference rankings. As evidenced by our listening examples: Human perception shows no significant preference between 0.4s vs 0.6s response latencies, while VAD models already provide precise timing judgments. We contend human annotation’s primary value lies in assessing response content quality, not microsecond-level timing nuances.

Question2: Reward Modeling

• We highlight our intentional avoidance of excessive hyperparameters in reward modeling to ensure reproducibility. The KL coefficient (0.2) is not a sensitive hyperparameter and aligns with standard RLHF practices, serving to prevent overly aggressive policy updates.

• Due to conversational dynamic, reward components exhibit varying effectiveness across different optimization set. On UltraChat (Main Experiments), R1 (Turn Consistency) and R2 (Backchannel/Barge-in) dominate so that we present them in Figure2, while R3 (Audio-Text Alignment) shows minimal impact. However, on less-turn datasets (ASR-QA): R1 becomes negligible since all reponse predict correct turns; on other datasets ( like topic): R2’s backchannel/barge-in rarely triggers with short agent responses. Therefore, our solution aggregates directly measurable rewards through simple summation. While preference-based optimization was explored to eliminate manual weight tuning, Table 4 demonstrates its inferior performance compared to our approach.

We sincerely appreciate your thorough review and valuable comments. Your expertise has significantly contributed to improving this work. We would revise our paper according to your sugeestions and corrections, and we provide detailed responses to address your concerns.

Question1: Data Limitations Thank you for your feedback. We wish to clarify two key points: (1) Reliance on Synthetic Data. Existing conversational datasets can NOT train a helpful spoken agent because neither speaker in these dialogues (e.g., Fisher Corpus conversations) fulfills the role of a qualified agent. This fundamental role mismatch necessitates synthetic data for effective agent training. (2) 0.64s Delay Rationale. The delay is essential because the agent cannot inherently preddict when a user will stop speaking or attempt to interrupt. Our experiments demonstrate that this empirically determined threshold, within reasonable bounds, does not disrupt natural interaction patterns. As evidenced by our provided listening examples, the agent exhibits natural latency behaviors in both turn-taking and barge-in scenarios.

Question2: Reward hacking and reward design.
Thank you for your question. We would like to clarify that the VAD metric in reward modeling does not induce reward hacking. As shown in Table 3, the introduction of IORISE does not degrade response quality (measured by GPT score), while maximizing the reward indeed improves the agent's conversational performance in terms of interaction accuracy. Our reward design prioritizes simplicity to avoid introducing excessive hyperparameter. Experimental results demonstrate that simply summing the reward components achieves good performance.

Question3: Limited Real-world Validation.
We have provided unseen samples (both topic and backgournd noises) on the demo page, including multiple stress tests (e.g., rapidly interrupting the agent multiple times within short intervals). The agent consistently demonstrates polite responses and accurate content in these scenarios. We believe this provides sufficient evidence to demonstrate our system's readiness for real-world deployment.

We sincerely appreciate your thorough review and valuable comments. Your expertise has significantly contributed to improving this work. Below we provide detailed responses to address your concerns.

Question1: Can an independent TTS system not be used in text-speech alignment
Thank you for your question. Two critical factors prevent the use of independent TTS systems in this context: (1) Real-time Requirements. Our real-time duplex design requires token-level streaming speech synthesis—each LLM prediction must generate 80 ms of synthesized speech including silence. This ensures a theoretical system latency of 80 ms, enabling rapid handling of dynamic conversational scenarios. Existing mainstream TTS degins are non-causal, and even causal TTS implementations hardly achieve such low systemic latency. (2) Input Differences. Conventional TTS systems rely on text or semantic tokens for pretraining. In contrast, our system performs audio token prediction directly based on the LLM's hidden representations. The LLM implicitly learns an integrated TTS system while simultaneously predicting text, enabling concurrent "thinking" and "speaking" during responses.

Question2: Why exisiting conversational data is insufficient to train a helpful agent
We appreciate the opportunity to clarify this important distinction. We summarize two reasons: (1) Speaker Role Mismatch in Conversational Datasets. Datasets like Fisher Corpus contain dialogues between peers (e.g., friends), where neither spaker assumes an agent role capable of providing structured assistance. Training on such data produces generic conversational models rather than goal-oriented agents specialized in delivering helpful recommendations. (2) The speaker voice in conversational datasets is diverse while we just need a fixed voice for agent. Though zero-shot TTS techniques could theoretically enable diverse vocal personas, our current priority focuses on establishing core competency in polite and effective duplex interaction.

Question2: Why does silence need to be inserted?
Unlike turn-based models, the full-duplex model operates as a real-time system in continuous prediction mode, as illustrated in Figure 1. Specifically, When the user is speaking, the agent must remain silent (i.e., predict silence tokens) to actively listen. Consequently, silence segments must be explicitly inserted into training data to teach the agent when to withhold speech production.

Question3: Why the need for 2 hyperparameters in EQ2
Thank you for your suggestion. A single parameter is indeed sufficient. We initially included two parameters in the formula to explicitly illustrate the one-to-one correspondence between weights and channels. We will revise this along with your other recommendations in the next version.

We hope these responses have addressed your concerns, and we reiterate our gratitude for your efforts in reviewing this work.

We sincerely appreciate your thorough review and valuable comments. Your expertise has significantly contributed to improving this work. Below we provide detailed responses to address your concerns.

Question1: Clarification on ASR Model
We would like to clarify a misunderstanding here: When calculating the Word Error Rate (WER) by an ASR model, we measure the discrepancy between the predicted speech and predicted text (our duplex model predicts them at the same time), rather than comparing generated speech with ground-truth labels. This process does not rely on any external supervision. Like the VAD model, it serves solely as an unsupervised reward evaluator.

Question2: Low BLEU Score in Table 4
On the UltraChat dataset, a BLEU score of 10 is normal. This dataset was adapted from the text-based UltraChat (https://github.com/thunlp/UltraChat) into a conversational format, with intentionally open-ended and diverse topics. As shown, even text-based GPT responses have a human evaluation score of only 6.4, while our 1B model achieves a GPT evaluation score of 4.0, surpassing the 7B Moshi model's 3.4.

Question3: Emotion in ORISE reward modeling
Yes, emotion is not modeled in current system. To ensure stability in agent speech synthesis, we intentionally used neutral emotional profiles during data synthesis, resulting in stable agent speech generation. However, your insight raises an intriguing point: our subsequent experiments reveal that when provided with emotionally varied speech labels for training, the current codec system can indeed generate agent speech with richer emotional expressions. We would explore this direction in future work.

Question4: Backchanneling type
The current user backchanneling only includes 1-5-word affirmative utterances. If the utterances become too long, the agent would struggle to determine whether the user is providing affirmation or attempting to initiate a new conversation.

Question5: Inluustration of System ID-2, 3, and 4
Table 3 demonstrates our duplex system's capability to function as a real-time agent delivering accurate responses. The GPT score serves as the primary quality metric for answers, while conversational metrics are excluded since they measure response etiquette in coversation. Key Comparisons: (1) ID-2 vs ID-3: Both models were trained with direct SFT on our dataset. This comparison confirms that our duplex system can effectively adapt different text-based LLMs into duplex-capable agents. (2) ID-3 vs ID-4: Since the ORISE-enhanced ID-4 model shows significant improvements in conversational performance metrics in Table 2, in this Table 3, we need to confirm that it still maintains comparable GPT scores, proving that ORISE optimizations preserve answer quality.

Question6: including the score for the model without unseen noise for reference.
Thank you for your suggestion. We wish to clarify a potential misunderstanding: The SFT model in Table 5 was the models trained without unseen noise. However, we note that if a model is trained entirely without noise and encounters sudden noise in the test set, the agent would fail to take the turn, as it would misinterpret noise as the user maintaining a keep-speaking mode. Thus, such a reference lacks meaningful comparative value. To make sure the delpoyment in reality, noise-augmentation is neccessary for dupelx training.

We will incorporate your formatting and typographical suggestions in the next version. Thank you once again for your thorough review efforts.