CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching

Dear Reviewer 49qE,

Thank you for your comprehensive review and valuable feedback. We use C,O,S,Q to abbreviate Clarity, Originality, Significance and Question for our responses.

C1-C3, O2 We will modify the Figure 1 with detailed explanations, and add references [1~5] you pointed out in the corresponding sections.

C4During training, we extract a random segment from each dialogue sample to serve as the speaker prompt. This segment must consist of consecutive frames spoken exclusively by a single speaker. We refer to such segments as "candidates."

C5 Our method processes each speaker's text as a two-stream sequence (z = [$z_1$, $z_2$]), which is first concatenated under the following description, and then encoded.Each stream includes a prompt part and a synthesized part. The prompt part uses [spk1] and [spk2] as indicators (not text tokens) to replace the transcription of the prompt, distinguishing which segments belong to each speaker. The synthesized part contains the actual dialogue, aligned with mel-spectrogram frames at the frame level. For example, in a 6-second dialogue with 100 frames per second, if spk1 speaks 30 characters from 1-3s and spk2 speaks 20 characters from 4-5s, $z_1$ includes 100-130 real characters and 130-300 continuation tokens [P], while $z_2$ includes 400-420 real characters and 420-500 tokens [P]. Other tokens in z are silence tokens.

For the real-world dialogue, we utilize the ASR and diarization tool to get the sentence-level timestamp $t_{start}$ and $t_{end}$. These timestamps are not ground truth and may contain errors. We also use simulated LibriTTS data to provide accurate ground-truth timestamps, even for overlapping speech segments. This approach contrasts with single-stream methods by enabling temporal alignment and features like specific silence insertion and overlapping, as shown in our demo.

C6.1, C6.2 The robustness and diverse means the model is robust given prompts under challenging and noisy acoustic environments. We achieve that by eliminating the dependence of prompt’s transcriptions, while other model face serious performance degradation if the ASR transcription is wrong, demonstrated in Table R1 replied to reviewer W9cW. Additionally, the random chosen of $m_{ctx}$ enables in-context learning by allowing the model to extract speaker prompts directly from training dialogue samples. This design can even improve speaker similarity because the prompts and generated speech share the same acoustic environment.

C7.1 We did not mention the name of the speech enhancement API due to the double-blind review policy. We will cite it in the camera-ready paper if the manuscript is accepted.

C7.2 Our current simulated data did not take contextual coherence into consideration due to its complexity and our goal is to model overlapped speech generation. we recognize the value of contextual coherence and will revisit this consideration in future work.

C8.1 The open-source version of Moshi does not support multi-speaker dialogue generation. Moshi's dialogue capability is primarily focused on human-user and agent interactions, not the generation of speech for multiple speakers. While Moshi's paper mentions a multi-stream TTS model for dialogue synthesis, there are no publicly available codes or data to enable reproduction of their described model.

C8.2: The 15 professional linguistic experts we selected were specifically trained in linguistics and experienced in subjective testing, and they were crowd-sourced for this evaluation. Due to the double-blind review policy, we cannot explicitly name the company platform used but will disclose it upon paper acceptance.

C9.1: The main differences between our input format and the baselines lie in two key perspectives (1) CodiFM employs multiple text streams to distinguish "who speaks when and what," whereas Sesame utilizes only a single text stream with only the content and the speaker id, which can more readily lead to speaker confusion. (2) CodiFM places the prompt at the very beginning of the input sequence and generates the entire dialogue in parallel. Sesame, being an AR model, generates each utterance sequentially. While its prompts are also at the beginning, this can cause confusion because the previously generated audio is often more proximate to the segment being synthesized than the initial prompt.

C9.2: The primary focus of this paper is on generating synthesized dialogue with accurate pronunciation by the correct speaker and appropriate turn-taking, including silences and overlaps. The "stable and reliable output quality" mentioned on Line 275 refers to the enhanced stability of our model in terms of speaker accuracy and content fidelity, minimizing instances where Speaker A utters content intended for Speaker B. In this work, we did not specifically design or optimize for speech expressiveness. As a zero-shot model, the expressiveness should be guided by the given prompts in addition to speaker timbre.

C9.3 and Q6: Thank you for your valuable suggestion. We conducted an additional fluency test using CMOS scores as per your request, and the results are presented in Table C93-1.

Table C93-1: Fluency CMOS

To assess speaker change accuracy, we introduced two metrics: Diff-SC (difference between correct and generated speaker changes, ideally zero) and Speaker Change Error Rate (SCER) (proportion of dialogues with incorrect speaker change counts). Table C93-2 proves that our CodiFM outperformed others on both metrics, proving its superior accuracy in handling speaker changes.

Table C93-2: Evaluation for Speaker Change

Table C93-3 demonstrates that prompt length affects Sesame's performance, with shorter prompts leading to poorer results. In contrast, our CodiFM model is robust to varying prompt lengths, showing minimal impact on performance. MoonCast, however, exhibits less stable performance.

Table C93-3: SA-WER/SA-SIM evaluation of different speaker prompt length

C10.1 Thank you for highlighting the potential for misunderstanding in Figure 4. We acknowledge that Figure 4 was initially generated using statistics from a single dialogue, which may have led to misinterpretations. After averaging across multiple dialogues, our model consistently maintains its leading performance.

C10.2 We will revise our description accordingly.

C11.1, C11.2 and C12 Adding simulated dialogue data can hurt model performance, likely due to noise from concatenating utterances with different acoustic environments. However, simulated LibriTTS data is usable because it it clean. The performance degradation noted in Table 4 is not related to overlapping speech, as the test set does not include overlapping speech and WER was calculated after segmenting sentences with the Deepgram ASR+Diarization API.

S2 We acknowledge that preferences vary and baseline models may excel in expressiveness due to their larger parameters and datasets. We've addressed concerns about evaluation bias by introducing a new, more diverse real-world test set with varied acoustic conditions and conversational prosody (Table R2 for reviewer qZtn), and we've expanded our analysis to include longer dialogue durations (Table R3 for reviewer qZtn).

Q1 The utilization of timely compact codec as an intermediate representation does not always yield performance improvements for dialogue generation task. [9] have demonstrated that using semantic tokens as intermediates could cause instances of words being omitted or duplicated in synthesized dialogues with the Hubert semantic tokens.

Q2 We dropped both the text and audio conditions for $p_{uncond}$. In the remaining cases, we dropped the audio condition with $p_{dropx}$. We will revise the paper accordingly.

Q3 Fine-tuning with dialogue data, led to a slight, but not significant, decrease in UTMOS scores, dropping from 3.48±0.19 (pre-train only) to 3.35±0.17 (pre-train and fine-tune). It is because the dialogue data is not clean even after speech enhancement.

Q4 The 3,000-hour dataset used in this study is internal and consists of two-person conversations. Due to confidentiality constraints, we are unable to disclose specific details regarding data acquisition. However, the dataset has been reviewed by our internal legal department to ensure compliance with copyright and ethical standards.

Q5 Thank you for pointing that out. Indeed, full overlap data is uncommon in real life. Currently, we’ve used the LibriTTS dataset to simulate overlapping data (0%–100%), similar to speech separation methods. In the future, we’ll integrate real-world overlap statistics to improve our simulations.

Q7 Thank you for your interest in extending CodiFM to real-user interactive applications. While this lies beyond the scope of our current study, which focuses on use cases such as video dubbing and two-host podcast generation. We would like to share a few of our initial thoughts. A flow-matching model could potentially support streaming by developing a chunk-aware, causal flow-matching variant [1]. Additionally, techniques such as consistency distillation or leveraging mean flows could enable one-step inference. Therefore, it also has the potential to real-user interactions.

[1] Z. Du, et al, CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

Dear Reviewer 49qE,

We appreciate your detailed feedback and would like to clarify some key points regarding our methodology, results and your concern regarding the legality and licensing of the training dataset. The detailed responses are listed below.

Further Question 1

Thank you for the opportunity to clarify our choice of speaker prompts. We believe our approach of selecting a prompt from the current speech is more beneficial for several key reasons, and we will revise the manuscript to articulate this more clearly.

First, this method aligns with the principles of audio continuation [1,2] or In-context-learning [3,4] demonstrated in recent monologue TTS work. By sampling a prompt from the same dialogue (either at the beginning or randomly chosen), we ensure strong continuity between the prompt’s speaker characteristics (e.g., style, timbre, acoustic environment) and the segment to be generated.

Second, our method avoids this potential train-inference mismatch. Our training procedure, which uses a mask for the prompt segment, showing in Figure 1, ensures that the prompt and the target text are distinct, which perfectly mirrors the inference process. In contrast, using a prompt from a prior speech is feasible but could introduce an acoustic mismatch due to differences in the recording environment.

We acknowledge that in some traditional works, such as FastSpeech2 [5], a separate speech is used as a prompt to prevent information leakage. However, this is primarily a concern when the prompt and target share the same text transcription. In our work, the prompt's transcription is intentionally different from the dialogue text to be synthesized, thus mitigating this risk.

Finally, from a practical standpoint, retrieving a suitable monologue segment from a separate dialogue is computationally inefficient, especially with large-scale datasets. This process would require significant overhead for identifying the target speaker's dialogue and isolating a suitable monologue segment to use as a prompt.

Further Question 2

To simulate dialogue from the LibriTTS datasets, we used the following method, and we will open-source this data-simulation code.

1. We begin by randomly selecting two distinct speakers.

2. Next, we retrieve a list of all available dialogues for each of these two speakers.

3. We then generate a new dialogue with a predetermined maximum duration, such as 30 seconds. We interleave speech segments from Speaker A and Speaker B into this new dialogue until it reaches the maximum duration.

4. Once the dialogue reaches the maximum duration, we save the generated dialogue and begin simulating a new one using any remaining speech segments.

Below is an example of the generated results. Each line follows the format spkid|content|start_time(ms)|end_time(ms):

1789|We wot not, said the fishers, but he keepeth it no counsel but that he is a knight of King Arthur's, and by the mighty lord of this isle he setteth nought.|0|8430
1160|I had my share of it; for, as soon as I got back to my seat in the Assembly, I was put on every committee for answering his speeches and messages, and by the committees always desired to make the drafts.|4430|18150 
1789|Then the lady prayed the fishers to bring him to her place.|12150|15170 

It is important to note that our method does not guarantee contextual consistency between the two speakers. Nevertheless, within each individual speaker’s speech segments, we ensure that the topic remains consistent. We appreciate the suggestion to reconstruct dialogues using original metadata or similar approaches. However, identifying speech segments that are not only contextually aligned but also involve utterances from two speakers is a challenging task.

Further Question 3

Our primary task is dialogue generation, which involves taking two speaker's voices (timbre) and producing a multi-turn conversation. We determined that the open-source Moshi model is not suitable for this purpose for the following reasons:

1. Turn-by-Turn Generation and Uncontrollable Silences: Even if we replace the user's text prompt with a monologue from another speaker, Moshi generates the dialogue on a turn-by-turn basis. This process introduces an uncontrollable silence between each turn, which is not ideal for natural dialogue flow.

2. Inconsistent Speech Synthesis: The "pseudo-user" speech in Moshi is synthesized using a standard Text-to-Speech (TTS) model. This approach is not comparable to our work, as it does not capture the natural back-and-forth of a real dialogue.

Furthermore, the baseline Sesame model we use shares a similar architecture to Moshi but provides superior performance, making it a more robust benchmark for our work.

I appreciate the authors' detailed response. It allowed me to clearly understand your perspectives regarding my questions, as well as pinpoint exactly where we agree and where our opinions diverge. Below are my follow-up comments.

• FQ1: I acknowledge and agree with the authors' explanation regarding the standard usage of prompts in prior in-context learning and continuity scenarios. However, under the current approach—where the prompt is essentially a subset of the same dialogue being synthesized—I still find it difficult to accept the claim that "the prompt and the target text are distinct." More specifically, I might have misunderstood the statement that "the prompt's transcription is intentionally different from the dialogue text to be synthesized." In my view, there still seems to be a training-inference mismatch; fully addressing this issue would require selecting prompts from audio clips separate from the dialogue currently being modeled. I agree this would be practically inefficient, but wouldn't introducing "acoustic mismatch due to differences in the recording environment" actually result in a simulation that is closer to inference-time conditions? Although this issue slightly deviates from the main scope of your paper, I am bringing it up again to fully clarify your perspective.

• FQ2: Thank you for sharing the samples and providing a detailed explanation of your data construction process. Your explanation clearly addresses how the simulated dialogue dataset was constructed, and I found it somewhat surprising that the model performs well even with this kind of data. While your argument—that maintaining speaker consistency enables effective dialogue simulation—is convincing, I still suspect that my initial impression about the reduced naturalness of inter-speaker dialogue might stem partly from this data construction method. Specifically, while individual speakers' utterances are coherent and accurate, the overall turn-taking felt somewhat less expressive and natural.

  Additionally, you mentioned the dataset involved two speakers, but the provided examples contain multiple speakers. Could this be because the examples exceed 30 seconds, resulting in additional speakers appearing within the sample?

• FQ3: Thank you for the clarification. Your explanation effectively addressed why Moshi was inappropriate as a baseline and reinforced my understanding of why Sesame was a more suitable choice.

• FQ4: I understand the internal dependency constraints clearly now. Thank you for the clarification.

• FQ5: Thank you for sharing the additional experimental results. They clearly demonstrate that your method maintains superior performance over baselines even with longer prompts (20s). In my view, including these results explicitly in the paper would significantly highlight the strength of your proposed approach.

• FQ6: Regarding point #1, my understanding is that Sesame utilizes Moshi's mimi, an open-source, multi-speaker codec specifically developed for Moshi. Thus, it seems reasonable to expect that mimi can naturally handle overlapping speech scenarios. Concerning points #2 and #3, I agree there are inherent trade-offs involved; while directly predicting mel-spectrograms may yield better reconstruction and eliminate the need for additional codecs, employing compact latent representations might offer advantages like improved alignment. In my opinion, empirical validation would be necessary to clarify these trade-offs fully.

• FQ7: It is encouraging to see that separately dropping inputs during training leads to performance improvements. Additionally, applying guidance separately during inference also seems promising; I recommend considering this approach in future work.

• FQ8: Thank you for your response. I hope the ethical reviewers carefully assess this aspect.

Thank you for the detailed responses.

• FR1: I now understand that the prompts are excluded from the loss calculation, as illustrated in Figure 1. Previously, I had focused solely on whether the text was included and overlooked this detail. Your explanation clarifies that this approach is effectively equivalent to the span masking technique used in prior literature.
• FR2: Thank you for the additional explanation. Reviewing the ‘spkid|content|start_time(ms)|end_time(ms)’ format helped me fully understand your approach.
• FR3: I appreciate you sharing the L2 loss values. I agree with your reasoning regarding the challenges of codec reconstruction for overlapping speech.

Most of my initial concerns and follow-up questions have been satisfactorily addressed. Therefore, I am inclined to raise my score. However, to fully deserve this score, the final revision should incorporate the discussed clarifications and address the missing citations.

Overall, I recommend increasing the score with the expectation that these improvements will be reflected in the final version.

Dear Reviewer qZtn,

We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. We have carefully considered all your comments and suggestions, which have significantly helped us improve the manuscript. We have conducted additional experiments on realistic scenarios and analyzed the model's performance across different durations, and these results will be incorporated into the revised version of our paper. The detailed responses are listed below.

R1: About similarity to existing works (Weakness 1)

To achieve state-of-the-art (SOTA) performance in dialogue speech synthesis, we have fully leveraged existing SOTA technologies in our model building. While this may give the impression that CodiFM is a mere combination of existing methods, these individual methods, on their own, do not fully address the critical challenges inherent in our specific scenario: generating synthesized dialogue with accurate pronunciation by the correct speaker and appropriate turn-taking, including silences and overlaps.

To clarify our distinct contributions, we have provided a comparative analysis with existing works in Table R1. Our key contributions are as follows:

1. Our work enables the generation of entire dialogues with accurate pronunciation by the accurate speaker, effectively resolving the speaker confusion problem prevalent in prior AR-based models.

2. We employ a novel multi-stream Transcription-Level Speaker Disentanglement strategy to enable the model to precisely understand "who speaks what and when.

3. We propose the Prompt-Level Random Masking strategy to enhance in-context learning capabilities specifically within dialogue scenarios. This strategy eliminates the need for prompt transcription, thereby removing the dependency on ASR tools and simplifying the generation process.

Table R1: System comparison

R2: Generalizability to real-world application (Weakness 2)

We appreciate your concern regarding the generalizability of using LibriSpeech for evaluating real conversation styles. We agree that assessing performance in real-world scenarios is important.

To address this, we have designed an additional test set. The speaker prompts for this new set are derived from 10 distinct real-life dialogues from the NCSSD dataset-CEN, as proposed in Reference [31]. This dataset is collected from the internet and features prompts with natural conversational prosody, including various acoustic scenarios such as background noise. The corresponding text content for this test set is sourced from the DailyDialog dataset [37].

As presented in Table R2, our model, CodiFM, consistently demonstrates superior performance compared to baseline models even under these challenging real-world conditions.

Table R2: System performance comparison under real-world scenarios

R3: About CodiFM’s maximum inference duration (Question 1)

Our training data is limited to durations under 30 seconds due to resource constraints. However, our model is capable of inferring dialogues longer than 30 seconds, and our existing test set includes numerous dialogues exceeding one minute in length. We conducted evaluations on four manually designed dialogues of varying durations, assessing SA-WER and SA-SIM as presented in Table R3. We found that CodiFM's performance begins to degrade after 90 seconds, and this degradation is also observed with MoonCast. Sesame fails to generate dialogues longer than 120 seconds. Furthermore, Sesame exhibited unstable results even under 30 seconds due to speaker confusion issues.

Table R3: SA-WER/SA-SIM evaluation of different duration of synthesized data

It is important to clarify that MoonCast, although designed for long-form generation, employs a chunk-wise Flow-matching detokenizer. For each turn in a long dialogue, it calls the Flow-matching detokenizer, requiring the audio of each speaker prompt to be provided every time. Therefore, it can't be considered a purely long-form generation model because the audio detokenizer still generates and concatenates multiple audio segments. Similarly, Sesame generates each utterance separately, relying on previous history, which means it is not a truly long-form generation model and is consequently less impacted by increased duration.

R4: About the mis-leading visualization of Figure 4a (Question 2)

Thank you for pointing out the potential for misunderstanding in Figure 4. We acknowledge that Figure 4a was indeed generated using statistics from a single dialogue, which may have led to misinterpretations.

After averaging across multiple dialogues, our model consistently maintains its leading performance. In contrast, Sesame and MoonCast exhibit unstable generation, leading to inconsistent results across different scenarios. For instance, in Table R2, we observe that MoonCast's speaker similarity on this realistic test set is slightly higher than Sesame's. Given that both models frequently encounter speaker confusion problems, their performance shows high variance, making it challenging to compare them when both are unstable.

We will update this Figure 4a in the revised manuscript with a new visualization that incorporates multiple speech samples, thereby providing a more representative and clearer illustration. Additionally, we will update these new samples on our revised demo page.

Finally, we would like to express our gratitude again for your time and effort in reviewing our paper. Considering this is the first attempt at non-autoregressive multi-talker dialogue generation and that we have incorporated extensive comparisons under realistic and long duration scenarios, we would appreciate it if you could consider increasing your score. Please do not hesitate to let us know if you have any further concerns or comments. We would be happy to address them.

Dear reviewer zySF,

We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. We have clarified some unclear writing that caused misunderstandings and added extra evaluations, which will appear in the revised version. The detailed responses are listed below.

R1: Importance of introducing duration (Weakness1 and Question1)

We acknowledge your query regarding the importance of introducing duration control. Currently, there are no existing Non-Autoregressive (NAR) dialogue generation models, compelling us to compare our work primarily with Autoregressive (AR) models.

A key distinction of our approach, in contrast to conventional NAR models such as VoiceBox and FastSpeech which necessitate highly accurate duration prediction, is our reduced dependency on precise duration, requiring only utterance-level duration information. During training, durations are extracted using ASR and diarization tools. We recognize that these tools may introduce inaccuracies, meaning our extracted durations are not exact ground truth. Crucially, during inference, our model does not rely on ground truth durations. For instance, for a single dialogue, diverse durations can be employed to generate a variety of dialogues.

We emphasize that the ability to control speaker duration addresses a critical gap in existing AR models. Many real-world scenarios, such as video dubbing or advertisements with strict time limits (e.g., 15-second spots), necessitate precise duration control for each speaker. Furthermore, as illustrated in the first sample on our demo page, our model can generate crucial silences—for example, a waiter instructing someone to wait—which AR-based baseline models are unable to achieve.

It is important to note that CodiFM does not strictly require users to provide duration information. We offer an optional functionality where duration can be roughly estimated by counting syllables in the text.

R2: The non-verbal and long dialogue generation capability (Weakness2 and Question2)

Regarding non-verbal capabilities such as laughter, our model's generation of such behaviors is data-driven, consistent with findings in prior work [9]. While we did not explicitly annotate these behaviors in our dataset, their presence allows our model to generate them, albeit in a non-controllable manner. We observe that Mooncast also exhibits non-controllable laughter generation, sometimes at inappropriate moments, as demonstrated in audio sample 2. We will include additional samples to showcase our model's non-verbal behavior generation capabilities in the revised demo page. We plan to annotate future datasets to enable controllable generation of such functionalities.

Due to resource constraints, our training data currently consists of segments less than 30 seconds in duration. Nevertheless, our model demonstrates the ability to infer dialogues longer than 30 seconds, and our test set includes numerous dialogues exceeding one minute. To further investigate long dialogue generation, we manually designed four dialogues with varying durations. We observed that both CodiFM and Mooncast exhibit degraded performance after 90 seconds, while Sesame fails to generate dialogues longer than 120 seconds. Additionally, Sesame showed unstable results even for dialogues under 30 seconds, primarily due to speaker confusion issues.

Table R2 presents the SA-WER/SA-SIM evaluations for synthesized data of different durations. None of the compared systems support streaming. For long-form generation, AR models typically require significant waiting times. In contrast, our CodiFM can generate each segment in parallel with a Real-Time Factor (RTF) of less than 0.3, significantly reducing waiting times.

While Mooncast is specifically designed for long-form generation, it actually utilizes a chunk-wise Flow-matching detokenizer. For each turn in a long dialogue, it calls the Flow-matching detokenizer, requiring the audio of each speaker prompt to be provided every time. Therefore, it can't be considered a purely long-form generation model because the audio detokenizer still generates and concatenates multiple audio segments. Similarly, Sesame generates each utterance separately, relying on previous history, which means it is not a truly long-form generation model and is consequently less impacted by increased duration.

Table R2: SA-WER/SA-SIM evaluation of different duration of synthesized data

R3: About SA-SIM metric (Weakness3 and Question 3)

We would like to clarify that our proposed SA-SIM metric is designed to measure speaker identity similarity relative to the speaker prompt, rather than prosodic similarity. Even with variations in prosody, the speaker similarity for the same speaker consistently remains high.

We did, however, account for the potential for prosodic variations in speakers over different time intervals. To address this, we separately calculated speaker consistency, as detailed in Figure 4b of Section 5.2. This was achieved by comparing the similarity of multiple segments from the same speaker within the same conversation. Our results demonstrate that CodiFM can better maintain speaker identity regardless of prosodic changes. Our experiments further revealed that Mooncast and Sesame exhibit lower SA-SIM scores because certain speakers' voices undergo complete identity changes, not merely prosodic shifts.

R4: About the choice of NAR instead of AR model

The core of several questions revolves around the fundamental distinctions between NAR and AR models. While all previous research in dialogue generation has focused on AR models, there is a lack of purely NAR dialogue models. Although AR models have demonstrated effectiveness in dialogue generation, they possess several drawbacks:

1. Speaker Confusion: As evidenced by the SA-SIM results, some sentences may be attributed to an incorrect speaker.

2. Slow Inference Speed: AR models typically suffer from slow inference.

3. Uncontrollable Generation: They lack control over crucial aspects such as duration, speaking speed, and silence.

NAR models, on the other hand, can effectively mitigate these limitations. Their efficacy has been established in monologue generation, and they demonstrate strong potential in Text-to-Speech (TTS) generation tasks. This motivated our investigation into a purely NAR model for dialogue generation.

Finally, we would like to express our gratitude again for your time and effort in reviewing our paper. Considering this is the first attempt at non-autoregressive multi-talker dialogue generation and that we have added a comparison with previous work, we would appreciate it if you could consider increasing your score. Please do not hesitate to let us know if you have any further concerns or comments. We would be happy to address them.

Dear Reviewer W9cW,

We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. We have addressed the raised concerns by clarifying certain aspects of our methodology and incorporating additional analyses regarding the influence of prompt transcription and our system's performance in extended and complex scenarios. These revisions will be reflected in the updated version of the paper. Our detailed responses are listed below.

R1: Benefits of not using prompt’s transcription (Weakness 1)

Our decision to not utilize prompt transcription is motivated by two primary factors:

First, this design enhances the model's robustness across diverse speaker prompts. Conventional TTS models typically rely on ASR models or user-provided transcriptions. However, when prompts contain noise or exhibit unconventional speaking styles (e.g., whispering or speech with accent), ASR performance can degrade significantly, leading to inaccurate transcriptions (e.g., WER exceeding 50%). As demonstrated in Table R1, an inaccurate prompt transcription can severely compromise the quality of the generated speech in TTS systems. Therefore, our approach is not aimed at improving baseline performance but rather at mitigating performance degradation when confronted with challenging prompts and eliminating dependency on ASR models.

Second, this approach facilitates in-context learning by enabling the extraction of speaker prompts from the same dialogue samples used during training. Obtaining accurate transcriptions for arbitrary segments of an utterance is challenging. By removing the requirement for prompt transcription, our model can utilize any monologue segment from a training dialogue sample as a speaker prompt during the training phase.

Table R1: WER (%) evaluation for monologue and dialogue models with different prompt’s transcription

R2: Scalability and Robustness of Model (Weakness 2, Question 2)

Our current experiments are constrained by resource and data limitations, specifically a dataset containing only two speakers. Despite these limitations, our proposed framework is language-independent and can be readily extended to accommodate multiple speakers. Future work will focus on developing a multi-speaker, multilingual system with three or more speakers.

Nonetheless, our model exhibits robust performance during inference, even for dialogues three times longer than those in our training data (less than 30s). We conducted evaluations on four manually designed dialogues of varying durations, assessing SA-WER and SA-SIM as presented in Table R3. We found that CodiFM's performance begins to degrade after 90 seconds, a degradation also observed with MoonCast. Sesame fails to generate dialogues longer than 120 seconds. Furthermore, Sesame exhibited unstable results even under 30 seconds due to speaker confusion issues.

It is important to clarify that MoonCast, although designed for long-form generation, employs a chunk-wise Flow-matching detokenizer. For each turn in a long dialogue, it calls the Flow-matching detokenizer, requiring the audio of each speaker prompt to be provided every time. Therefore, it can't be considered a purely long-form generation model because the audio detokenizer still generates and concatenates multiple audio segments. Similarly, Sesame generates each utterance separately, relying on previous history, which means it is not a truly long-form generation model and is consequently less impacted by increased duration.

Table R2: SA-WER/SA-SIM evaluation of different duration of synthesized data

Moreover, we have evaluated our model under extreme multi-turn dialogue generation scenarios. While our training data consists of segments less than 30 seconds, with 81% having fewer than two speaker changes and only 0.5% exceeding six speaker changes (maximum of 12), CodiFM successfully generated dialogues with 20 or more speaker changes, where each speaker speak only single words, without errors. In contrast, baseline models failed in these challenging conditions due to severe speaker confusion. We will incorporate these challenging samples into the revised demo page.

R3: About similarity to existing works (Weakness 3 and Question 1)

To achieve state-of-the-art (SOTA) performance in dialogue speech synthesis, we have fully leveraged existing SOTA technologies in our model building. While this may give the impression that CodiFM is a mere combination of existing methods, these individual methods, on their own, do not fully address the critical challenges inherent in our specific scenario: generating a synthesized dialogue with accurate pronunciation by the correct speaker and appropriate turn-taking, including silences and overlaps.

Let’s clarify our contributions compared to existing work, which can be further illustrated in Table R3:

1. Our work enables the generation of entire dialogues with accurate pronunciation by the accurate speaker, effectively resolving the speaker confusion problem prevalent in prior AR-based models.

2. We employ a multi-stream Transcription-Level Speaker Disentanglement strategy to enable the model to precisely understand "who speaks what and when".

3. We propose the Prompt-Level Random Masking strategy to enhance in-context learning capabilities in dialogue scenarios. This strategy eliminates the need for prompt transcription, thereby removing the dependency on ASR tools.

Table R3: System comparison

R4: About the two Step 3 in Figure 1(Weakness 4)

We appreciate you bringing this point to our attention. You're right to highlight the potential for confusion with "Step 3" in Figure 1. Our intention was to depict the training stage and the inference stage as separate processes, each with three distinct steps. We'll revise Figure 1 to clearly differentiate between the steps of the training process and the steps of the inference process, ensuring there's no ambiguity.

Finally, we would like to express our gratitude again for your time and effort in reviewing our paper. Considering this is the first attempt at non-autoregressive multi-talker dialogue generation and that we have incorporated extensive comparisons under challenging scenarios with previous works, we would appreciate it if you could consider increasing your score. Please do not hesitate to let us know if you have any further concerns or comments. We would be happy to address them.