CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations

We sincerely appreciate your efforts in reviewing our paper and providing valuable and constructive feedback. We have implemented the Soundstorm model for dialogue synthesis as previous work to compare with our CoVoMix model, which will be included in the revised version. The detailed responses are listed below

R1: About comparison with previous work (Weakness 1)

We completely agree with your comments regarding the need to compare our proposed model with previous dialogue papers. We have also dedicated a significant amount of time to surveying this area. However, to the best of our knowledge, there are no suitable baselines available for comparison. Our work generates the entire dialogue at once, incorporating natural turn-taking behaviors such as overlap. In contrast, previous studies generate each utterance separately and then concatenate them, or generate each utterance sequentially without considering overlapping speech during speaker turn changes. The latter type of model, such as Soundstorm, is not officially open-sourced. Public implementations, however, were only evaluated on LibriSpeech, and there is no implementation for dialogue synthesis, which requires a special design for speaker turn changes.

To address your concerns, we implemented a SoundStorm-style baseline using the Fisher dataset based on our understanding of the paper. We utilized the EnCodec model for acoustic tokens (since Soundstream is not publicly available) and HuBERT for semantic tokens (the same in CoVoMix). Soundstorm lacks a mechanism to handle two-channel speech for training and overlapping speech from two speakers. Therefore, we had to reprepare the training and test datasets to ensure the speech was mono-channel with no overlapping parts. To isolate other effects, we used oracle semantic tokens for the model comparison with results shown in Table R1.

Table R1: Objective evaluation results in non-overlapped dialogue test set across models

R2: About extra dataset (Weakness 2)

We are eager to utilize one or two more datasets to verify the effectiveness of the proposed model. However, to the best of our knowledge, there are few multi-speaker conversational datasets available for training. One of our novel contributions is the proposal of a data-processing pipeline to leverage speech recognition datasets, such as the Fisher Dataset. We propose a comprehensive strategy for processing ASR datasets, including both training and evaluation for monologue and dialogue speech generation.

It is beneficial to utilize more datasets for training to improve model generalization. According to our observations, CoVoMix demonstrates a certain generalization capability across different domains. For example, the transcripts used for generating example demo videos are derived from DailyDialog[1], a dialogue dataset. We will consider simulating conversational datasets for training purposes in future work.

Ref: [1] Li, Yanran, et al. "Dailydialog: A manually labelled multi-turn dialogue dataset." arXiv preprint arXiv:1710.03957 (2017).

Finally, we would like to express our gratitude again for your time and effort in reviewing our paper. Considering this is the first attempt at multi-talker conversational mixed speech generation and that we have added a comparison with previous work, we would appreciate it if you could consider increasing your score. Please do not hesitate to let us know if you have any further concerns or comments. We would be happy to address them.

I agree with your statement that there are not enough datasets for evaluating or comparing the models' performances. I also want to know what's the major difference between the proposed model and Voicebox[1] or Audiobox [2] in terms of the model architecture. I believe those two models also utilize the Flow Matching technique for generating speech.

[1] Le, Matthew, et al. "Voicebox: Text-guided multilingual universal speech generation at scale." Advances in neural information processing systems 36 (2024).

[2] Vyas, Apoorv, et al. "Audiobox: Unified audio generation with natural language prompts." arXiv preprint arXiv:2312.15821 (2023).

I appreciate that you pointed out the differences between the proposed model and the previous literature in terms of model architecture. Regarding the critical response to my questions and the potential for development of the dialogue generation, I would raise my rating to borderline accept.

We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. We added a table to highlight the unique aspects of our model in comparison with previous works. Additionally, we implemented the Soundstorm-style baseline for dialogue synthesis to compare with our CoVoMix model. The detailed responses are listed below.

R1: About similarity to existing works (Weakness 1)

To achieve state-of-the-art (SOTA) performance in dialogue speech synthesis, we have fully leveraged existing SOTA technologies in our model building. This is why CoVoMix appears to be a combination of existing methods. However, these methods alone could not completely address the critical issues in our scenario: human dialogues are naturally characterized by turn-taking behaviors, such as overlaps and hesitations and non-verbal behaviors, such as laughter and coughing

Let’s clarify our contributions compared to existing work, which can be further illustrated in Table R1:

1. Our work generates the entire dialogue at once. In contrast, previous studies generate each utterance separately and then concatenate them, or generate each utterance sequentially without considering overlapping speech during speaker turn changes, such as in Soundstorm.
2. We employ simultaneous multi-stream semantic token prediction from dialogue text, with each stream representing an individual talker. We use a multi-talker flow-matching-based acoustic model to generate a mixed mono mel-spectrogram given multiple contexts.
3. We propose a comprehensive strategy for processing ASR datasets, including both training and evaluation for monologue and dialogue speech generation.

Table R1: System comparison

R2: About comparison with SoundStorm (Weakness 2, Question3)

The main difference between Soundstorm and CoVoMix lies in the generation process. Although Soundstorm is a non-autoregressive model, its dialogue generation pipeline relies on an autoregressive text-to-semantic model, which generates speaker A and B's content in a sequential ABABAB way and thus fails to produce overlapping content. In contrast, CoVoMix generates multiple streams of semantic tokens (including silence tokens) in parallel, with each sequence corresponding to one speaker. Since these streams are temporally aligned, multiple streams may speak simultaneously, resulting in a more natural dialogue with such overlapped speech.

Regarding your concerns that spontaneous behaviors might arise from differences in the training data, this is one of the critical issues we have addressed. We have proposed a comprehensive strategy for processing ASR datasets, including both training and evaluation for monologue and dialogue generation.

Since Soundstorm is not officially open-sourced, we have considered comparing CoVoMix with publicly available implementations of Soundstorm. However, these implementations were only evaluated on LibriSpeech, and thus there is no implementation for dialogue synthesis, which requires special design for speaker turn changes. Instead, we implemented a Soundstorm-style baseline using the Fisher dataset based on our understanding of the paper. We utilized the EnCodec model for acoustic tokens (since Soundstream is not publicly available) and HuBERT for semantic tokens (the same in CoVoMix). Since Soundstorm is unable to handle two-channel or overlapping speech for training, we had to reprepare the training and test datasets to ensure the speech was mono-channel with no overlapping. To isolate other effects, we used oracle semantic tokens for the model comparison shown in Table R2.

Table R2: Objective evaluation results in non-overlapped dialogue test set across models

R3: About speaker information in HuBERT tokens (Question 1)

We did notice that HuBERT tokens contain speaker information. As shown in Appendix B.2, there is a larger speaker similarity gap between models using predicted and oracle semantic tokens than phoneme sequences. We have attempted to solve this problem using various methods, such as extracting semantic tokens from voice-converted utterances to remove the original speaker identity during the training stage. However, the performance has not met our expectations and is therefore not reported in the paper. We will continue investigating this issue in the future.

R4: Robustness of Laughter in Speech Tokenizer (Question 2)

We believe that Speech Tokenizer HuBERT trained on the Fisher dataset contains tokens representing laughter. The laughter can be generated by giving a tag in the text or automatically generated by using contextual info. In addition, we observed that CoVoMix shows a certain generalization capability across different domains. For example, the transcripts used for generating demo videos are derived from DailyDialog (a dialogue dataset) with manually annotated positions of laughter.

However, according to References [8], HuBERT trained on Fisher shows degraded performance when generalized to LibriSpeech due to domain mismatch.

Finally, we hope we have addressed all your concerns. We would appreciate it if you could consider increasing your score.

Thank you for your kind response. I have one question. In the official demo of SoundStorm, in the first video, there seems to be overlapping speech, and the second spoken dialog sample it seems there are some nonverbal cues such as laughter. This appears to be slightly different from what you mentioned. Could you please provide an explanation regarding this?

We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. We have clarified some unclear writing that caused misunderstandings and added a baseline system for comparison, as well as a speaker similarity subjective test for dialogue, which will appear in the revised version. The detailed responses are listed below.

R1: About unclear presentation of Line 149-152 (Weakness 1, Question 1)

Lines 149-152 indicate that we need to divide the semantic embedding into two halves. For instance, if there is a semantic embedding of shape [B,T,D], we divide it into [B,T,:D/2] and [B,T,D/2:] to obtain these two embeddings. We have revised the paper to clarify this.

R2: About insufficient baseline comparisons (Weakness 2)

We agree that providing more baseline comparisons would give a better understanding of CoVoMix’s performance. We have also spent a significant amount of time surveying this. However, to the best of our knowledge, there are no suitable baselines available for comparison. For instance, Soundstorm models and codes are not officially open-sourced. The publicly available implementation was only evaluated using LibriSpeech. Therefore, there is no Soundstorm implementation for dialogue synthesis, which requires a special design for speaker turn changes. The original paper also lacks sufficient details on this aspect. Furthermore, GPT-4o was not accessible during our research period.

We have tried to reproduce the dialogue model of Soundstorm based on our understanding of the paper, but the results were not as reported, so we did not put the results into the paper. To address your concerns, we continue implementing a SoundStorm-style baseline using the Fisher dataset. We utilized EnCodec for acoustic tokens (since Soundstream is not publicly available) and HuBERT for semantic tokens (the same in CoVoMix). Soundstorm lacks a mechanism to handle two-channel speech for training, and consequently, it does not model overlapping speech from two speakers effectively. Therefore, we had to reprepare the training and test datasets to ensure the speech was mono-channel with no overlapping parts. To isolate other effects, we used oracle semantic tokens for the model comparison shown in Table R1.

Table R1: Objective evaluation results in non-overlapped dialogue test set across models

R3: About speaker similarity in dialogues (Weakness 3)

To address your concerns, we use oracle diarization results for similarity evaluation shown in Table R1. We also added a subjective evaluation for speaker similarity in a dialogue as in Table R3. Ten dialogues randomly selected were manually segmented into multiple single-speaker utterances to avoid speaker diarization errors. (For turn-taking statistics, we had to leverage an automatic diarization system due to the large size of whole testing set). We had 15 linguistic experts evaluate the speaker similarity of these utterances compared to the prompt speaker.

Table R3: Speaker similarity across models for dialogue generation

R4: About speech consistency and flow of speech (Weakness 4)

We agree that speech consistency should also consider the flow of speech and appropriate laughter, in addition to speaker similarity, which is predominantly used in related papers. In our dialogue naturalness subjective test, we have provided with the following specific guidelines (please refer to Fig.12 in appendix):

...evaluating how closely the dialogue resembles a natural conversation in terms of fluency, the rhythm, the intonation ... Consider how seamlessly the conversation flows from one speaker to the other, the appropriateness of pauses, and how these transitions contribute to a realistic conversational experience

Consequently, the flow of speech and appropriate laughter have already been taken into account in the subjective test scores.

Moreover, we added Table R4 to show the distribution of F0, demonstrating that CoVoMix produces speech more similar to the Ground Truth in multi-turn dialogues.

TableR4: F0 distribution across models for dialogue generation

R5: About the potential errors in speaker diarization (Question 2)

We use an open-source diarization tool, pyannote. It achieves an 11.9% Diarization Error Rate in its original domain (AMI dataset), but it may be less accurate on the Fisher Dataset due to domain mismatch.

The challenge for human evaluation lies in the co-appearance of multiple talkers in a dialogue. An interfering speaker can influence the speaker perception during human evaluation. Moreover, although humans can easily distinguish between two speakers of different genders, more than 60% of the utterances in Fisher dataset are of the same gender. To address this issue, we manually segmented the testing utterances into several single-speaker segments and had linguistic experts evaluate the speaker similarity. The results are shown in Table R3.

R6: About speech consistency of CoVoSingle (Question 3)

Speech consistency involves comparing the speaker similarity among different segments within the same dialogue. Although CoVoSingle demonstrates high zero-shot capability, it is less likely to generate multiple identical utterances with the same prompt due to the sampling of the flow-matching model. In contrast, CoVoMix addresses this consistency issue by generating the entire dialogue at once, rather than through multiple generations and concatenation.

Finally, we would like to express our gratitude once again for your time and effort in reviewing our paper. We would greatly appreciate it if you could consider increasing your score.

Most of my concerns are answered and I've increased my score.