Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis

[About the results on conditioning noisy reference prompts]
Here we conduct zero-shot TTS experiments on LibriSpeech test-other set to verify our model's robustness with noisy reference prompts. From the following table, we can see that Mega-TTS maintains excellent performance with noisy reference prompts. We have added these results to Appendix L in the new version of the paper (marked in red). The generated speech samples can be accessed in the "Additional Examples for Rebuttal" Section of our demo page: https://boostprompt.github.io/boostprompt/.

[Explanations about more cases of robust speech synthesis]
In commercial scenarios, news reporting, and other formal scenarios, robustness is a crucial factor. Just a few repeating or skipping words can have significant negative impacts. These situations are better suited for models with duration models that ensure robustness, such as FastSpeech [4], Glow-TTS [5], and our Mega-TTS. However, for models like tacotron [6], word omissions or repetitions can significantly affect the listening experience. On the other hand, in some scenarios, robustness is relatively less important. For example, occasional missing words or repetitions in dialogue scenes can also be natural. Thanks for your advice. We have added these explanations to Appendix K.

[References]
[1] Ren, Yi, et al. "Prosospeech: Enhancing prosody with quantized vector pre-training in text-to-speech." ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
[2] Huang, Rongjie, et al. "Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech." Findings of the Association for Computational Linguistics: ACL 2023. 2023.
[3] Ye, Zhenhui, et al. "CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training." arXiv preprint arXiv:2305.10763 (2023). [4] Ren, Yi, et al. "Fastspeech: Fast, robust and controllable text to speech." Advances in neural information processing systems 32 (2019).
[5] Kim, Jaehyeon, et al. "Glow-tts: A generative flow for text-to-speech via monotonic alignment search." Advances in Neural Information Processing Systems 33 (2020): 8067-8077. [6] Wang, Yuxuan, et al. "Tacotron: Towards end-to-end speech synthesis." arXiv preprint arXiv:1703.10135 (2017).
[7] Kong, Jungil, et al. "VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design." arXiv preprint arXiv:2307.16430 (2023).
[8] Ren, Yi, et al. "Fastspeech 2: Fast and high-quality end-to-end text to speech." arXiv preprint arXiv:2006.04558 (2020).

The authors addressed my concerns about the issues I raised. Thanks for the detailed explanation. I will raise the score from 5 to 6.

By the way I have a question for prosody modeling. I have noticed the authors utilized a fixed frame information bottleneck like AUTOVC, not phoneme-level modeling so I have a concern about the proper size of information bottleneck. In my experience, this proper size of autovc may differ according to the sampling rate and hop size so this makes it difficult to reproduce the results if the preprocessing methods or dataset are changed. In this paper, they adopt this information bottleneck on the phoneme sequence so it would be better if you could add the results or your experiences for proper information bottleneck size according to different language dataset or type of text sequence (grapheme or phoneme).

We thank the reviewer for the constructive and professional review. We hope our response resolves your concerns fully.

[About the missing statement of ProsoSpeech and the missing ablation study for phoneme-level and word-level prosody modeling]
We are sorry for the missing statement of ProsoSpeech. Here, we provide detailed descriptions of the differences between ProsoSpeech and our method. We also conduct ablation studies for the prosody modeling strategies. These descriptions and results are also added to Appendix H.

Differences: We agree that ProsoSpeech [1] has the same structure as P-LLM proposed in this paper (both of us use the auto-regressive Transformer for prosody modeling). However, ProsoSpeech aims to improve the naturalness of prosody modeling for traditional TTS, which is only conditioned on text and global speaker information. Compared with it, our P-LLM aims at improving the similarity of speaker-relevant prosodic patterns for zero-shot TTS, which extracts fine-grained prosodic information from latent prosodic prompts by leveraging the powerful in-context learning capability of LLM. We also propose a novel training strategy that enables P-LLM to capture the useful prosody-level information contained in the multi-sentence contextual prompts. Moreover, our work mainly aims at boosting the timbre and prosody prompting mechanisms for zero-shot TTS.

Ablation study for prosody modeling strategy: We use stride-8-level prosody modeling (the stride of the pooling layer inside the VQ encoder is set to 8) and ProsoSpeech uses word-level prosody modeling. We agree that the ablation studies in this aspect are very important. Therefore, we conducted the following experiments to verify the performance of the phoneme-level, word-level, and stride-8-level prosody modeling. In this experiment, all of the systems use 3 seconds of prompts. The results are shown in the following table. It can be seen that the stride-8-level prosody modeling (used in our paper) achieves the best performance. Intuitively speaking, the phoneme-level prosody modeling provides finer-grained information for better reconstruction while word-level prosody modeling provides more semantic information. Both of these methods would be easily affected by the alignment accuracy of the speeches in training and inference stages. In order to enhance the stability and performance of the proposed model, we use stride-8-level prosody modeling.

[About the missing comparisons with other prosody modeling methods such as Prosody-TTS and CLAPSpeech]
Although our Mega-TTS, Prosody-TTS, and CLAPSpeech are all prosody modeling methods, the objectives and tasks of our Mega-TTS are different from theirs:

• Prosody-TTS [2], and CLAPSpeech [3] aim at improving the overall prosody naturalness for traditional TTS, while our work focuses more on improving the prosody similarity for Zero-shot TTS. Prosody-TTS [1] measures the MOS-P (naturalness of pitch, energy, and duration) as one of their metrics. While, as illustrated in Section 4.1 and Appendix A.2, we use SMOS to measure the prosody and timbre similarity for each speaker. CLAPSpeech aims to improve the prosody prediction by encouraging the text encoder to extract prosody variance from the text context with contrastive loss in the joint multi-modal space. The text encoder proposed in CLAPSpeech can be seen as speaker-irrelevant. Moreover, the text encoder proposed in CLAPSpeech can be also used in our work as a plugin module.

Since the objectives and tasks of Prosody-TTS and CLAPSpeech are different from the task of our Mega-TTS, we did not make comparisons with Prosody-TTS, and CLAPSpeech in our paper.

We are grateful to receive your feedback. In AutoVC, the fixed frame information bottleneck needs to be carefully adjusted to separate the content information from the source speech, making it difficult to reproduce the results if the preprocessing methods or datasets are changed. However, in this work, we directly provide the text content to our model and we only adopt this information bottleneck for the prosody encoder (not for the content and timbre encoder). The objective for disentanglement is the prosody patterns. As a result, the information bottleneck in our work is comparatively more robust and suitable for various datasets.

Thanks for your response. I agree that the information bottleneck on prosody representation is more robust. I hope to discuss more at Vienna thank you, and this is my last question, do you have a plan to release the source code of your model?

We are grateful for your positive review and valuable feedback, and we hope our response fully resolves your concern.

[About the missing zero-shot TTS categories in the related works]
We are sorry that the attention-base adaptation method is missing in the related works. In the revised version of the paper, we have included detailed discussions on the attention-based adaptation methods and highlighted the main differences between them and our own approach in Section 2 (marked in red). We would like to express our gratitude for your valuable suggestions once again.

[About the performance difference between using long context vs short context during training]
Yes, the ablation studies of the context length during training are essential. We have separately trained our Mega-TTS with different numbers of the contextual VQ codes for P-LLM and trained our Mega-TTS with different numbers of the contextual mel-spectrogram frames for MRTE. After evaluating the reconstruction quality for our compressive acoustic autoencoder and the cross-entropy loss for our P-LLM, we find that the performance of the model significantly improves when we increase the length of context, demonstrating the effectiveness of our multi-reference training strategy. We have added the experimental results and analysis to Appendix J in the new version of the paper (marked in red).

[About the assumption in section 3.1] (Question 1)
Yes, we agree that: 1) some performing skills may change the timbre largely in different sentences; 2) the the global style information can also be shared by different utterances. For case 1), we select $\tilde{y}$ randomly from regions near $y_{t}$ as much as possible, ensuring that the timbre information of $\tilde{y}$ is close to that of $y_{t}$. We have made descriptions of it in Appendix I in the new version of the paper (marked in red). For case 2), under this circumstance, the timbre encoder may capture global style information. However, the local prosodic patterns vary dramatically across different utterances by the same speaker. The P-LLM in our paper mainly aims at capturing the fine-grained (local) prosodic patterns. To enhance the clarity of our paper, we have provided additional clarification for assumption (1) and explained that we only disentangle the fine-grained prosody and timbre information in the new version of the paper (marked in red). Furthermore, our experiments demonstrate the capability of Mega-TTS to independently prompt prosody and timbre in a zero-shot setting. Even in some emotional datasets (the ESD dataset) and in some in-the-wild corpus (as shown in the demo page), our method is also able to clone non-audiobook style voices effectively.

[About the baseline methods with 300s context]
There are two types of baseline methods in our work: 1) VALL-E; 2) a baseline that incorporates the GAN used in our Mega-TTS to the FastSpeech 2 backbone [1].

1. For VALL-E, it is impossible to show the results with 300s context. Due to the low compression rate of the Encodec model, the maximum context is limited (around 30 seconds) during training. When the length exceeds this limit during inference, the generation quality will be very poor. As shown in "long-form zero-shot TTS experiments" on our demo page, VALL-E fails to generate intelligible results with long context.
2. For the baseline 2), it is fine-tuned on the corresponding context. Therefore, for Baseline-300s, it has actually encountered and learned the speech information from each speaker's 300-second context, which is more powerful than directly feeding the 300s context into the reference encoder and encoding it to a time-averaged global vector.

[References]
[1] Ren, Yi, et al. "Fastspeech 2: Fast and high-quality end-to-end text to speech." arXiv preprint arXiv:2006.04558 (2020).

Dear Reviewer,

Please respond to authors rebuttal and see whether they have addressed your concerns. Thanks!

[About the dimension in which the concatenation is performed in the Multi-Reference Timbre Encoder]
The concatenation is performed in the time dimension in the Multi-Reference Timbre Encoder. We are sorry that the corresponding part in Figure 1 is misleading. We have added more descriptions to the title of Figure 1in the rebuttal version of the paper (marked in red).

[About using a different model to measure SIM instead of using WavLM the one utilized by VALL-E [1]]
Yes, we agree that replicating the experimental conditions described in the baseline papers in the evaluation is important. Therefore, we use the WavLM utilized by VALL-E [1] to measure SIM and the results are shown in the following table. It can be seen that the trend and conclusion of the new experimental results are consistent with the results of the speaker verification model used in our paper.
We acknowledge the significance of replicating the experimental conditions outlined in the baseline papers during the evaluation. As a result, we employ the WavLM, which was utilized by VALL-E [1], to assess SIM. The corresponding results are presented in the table below. Moreover, it is evident that the recent experimental findings align with the outcomes of the speaker verification model employed in our study.

[References]
[1] Wang, Chengyi, et al. "Neural codec language models are zero-shot text to speech synthesizers." arXiv preprint arXiv:2301.02111 (2023).
[2] Shen, Kai, et al. "Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers." arXiv preprint arXiv:2304.09116 (2023).
[3] Le, Matthew, et al. "Voicebox: Text-guided multilingual universal speech generation at scale." arXiv preprint arXiv:2306.15687 (2023).
[4] Casanova, Edresson, et al. "Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone." International Conference on Machine Learning. PMLR, 2022.

[About using Montreal Forced Aligner (MFA)]
Yes, Mega-TTS needs an external aligner (Montreal Forced Aligner) to obtain the alignment beforehand. We agree that the data preprocessing pipeline is more complicated than those with end-to-end aligners. However, in our experiments, we find that the external aligner based methods are more robust in large-scale and multi-speaker scenarios. Here we conduct WER evaluations for models based on the external aligner and models based on end-to-end aligners (Like YourTTS [4] and VALL-E [1]). All systems are trained on the large-scale LibriLight dataset and tested on the LibriSpeech test-clean set, which is a zero-shot test setting. For fair comparisons, we keep the number of parameters consistent among all systems. The results are shown in the following table. We can see that the WER of the models based on the external aligner is significantly lower than those based on the end-to-end aligner, demonstrating their effectiveness in practical scenarios.

[About the performance of using only the prosody information from another speaker]
Here we compare the use of prosody information solely from another speaker with the use of the interpolation technique. The results are presented in the table below. It is evident that relying only on prosody information from another speaker significantly increases the word error rate (WER). This phenomenon may be due to the fact that speakers A and B may not have identical vocal ranges. Directly transferring B's prosody to A may cause speech intelligibility issues, especially for some special prosody patterns. Our proposed prosody interpolation technique resolves this issue by transferring the prosody patterns in a soft and fine-grained manner during the autoregressive generation process.

Thanks for your valuable comments and we hope our response fully resolves your concern.

[About the performance of Mega-TTS with 3-second prompts]
Thank you for raising this concern. We have evaluated the performance of Mega-TTS with 3-second prompts in the original version of the paper. In the experiment, for each sample, we randomly choose another utterance of the same speaker and crop a 3-second speech segment as the enrolled speech following VALL-E [1]. The results are shown in the following table. However, due to the limited page space, we removed it from the original paper. We agree that this result is important and have added it to the new version of the paper (marked in red).

[About the absence of comparisons with NaturalSpeech2 and VoiceBox]
We agree that the comparisons with NaturalSpeech 2 and VoiceBox are important for evaluating the proposed model's superiority. However, VALL-E [1], NaturalSpeech 2 [2], and Voice Box [3] have not released their codes or pre-trained models officially. As described in Appendix F, we have also tried to carefully reproduce NaturalSpeech 2 and VoiceBox but failed unfortunately. Here, we make comparisons with demo cases on the LibriSpeech test-clean set from their demo pages. To make fair comparisons, we use 3-second prompts for our Mega-TTS in this experiment. We only show the subjective metrics due to the limited number of demo cases. The results are shown in the following table. We can see that Mega-TTS outperforms VALL-E and VoiceBox and achieves a comparable performance with NaturalSpeech 2 when the length of the speech prompt is 3 seconds. Moreover, the performance can be further improved by the multi-sentence prompting mechanisms proposed in our paper.

[About the assumption (1) ]
Yes, we agree that since we use the audiobook-style training corpus, the global speaking style across different utterances by the same speaker remains unchanged. Under this circumstance, the timbre encoder can capture global style information. On the other hand, the local prosodic patterns vary dramatically across different utterances by the same speaker. The P-LLM in our paper mainly aims at capturing the fine-grained (local) prosodic patterns. To enhance the clarity of our paper, we have provided additional clarification for assumption (1) and explained that we only disentangle the fine-grained prosody and timbre information in the new version of the paper (marked in red). Furthermore, our experiments demonstrate the capability of Mega-TTS to separately control prosody and timbre in a zero-shot setting. Even in some emotional datasets (the ESD dataset) and in some in-the-wild corpus (as shown in the demo page), our method is also able to clone non-audiobook style voices effectively.

[About the detailed descriptions of the information bottleneck]
Thank you for raising this concern. Intuitively, there are four crucial variables for ensuring an appropriate information bottleneck: 1) the number of codebook embedding; 2) the codebook channel size; 3) the compression rate $r$ of the VQ Encoder; 4) and the downsampling rate $d$ of the MRTE. However, the search space of these hyperparameters is too large. Since the settings of r and d also influence the reconstruction quality, the burden of P-LLM, and the computational requirements, we first consider $r$ and $d$, fix them, and find the best setting for the hyperparameters of the codebook. We have introduced the detailed guidelines for setting the information bottleneck and designed experiments for evaluating the disentanglement performance in Appendix F in the new version of the paper (marked in red). Moreover, we also include the results with different sizes of training data to demonstrate that our system performs well with different data volumes consistently.

We are grateful for your positive review and valuable feedback, and we hope our response fully resolves your concern.

[About the performance with different languages] (Question 1)
Here, we train our Mega-TTS model on the multilingual WenetSpeech dataset (Chinese) + LibriLight dataset (English) and use the AISHELL-3 dataset (Chinese) as the test set. We split the dataset and make evaluations according to Section 4.2 in our paper. To evaluate the WER for the Chinese dataset, we fine-tune the wav2vec2-large model. The results are shown in the following table. It can be seen that our Mega-TTS demonstrates superior performance with Chinese datasets. Due to the limited time constraints, we only conduct evaluations on the Chinese test set. Chinese audio examples and cross-lingual examples can be found in the "Additional Examples for Rebuttal" Section of our demo page: https://boostprompt.github.io/boostprompt/.

[About the inference times and computational requirements of Mega-TTS compared to other models] (Question 2)
We agree that the computational requirements are crucial for understanding the trade-offs involved in using Mega-TTS. In Table 1, we have provided the real-time factor (RTF), the time (in seconds) required for the system to synthesize one-second waveform, in the inference stage of Mega-TTS compared to other models. Besides, we have also compared the total number of parameters in Table 1 and provided the computational requirements for training Mega-TTS in Section 4.1.

[About the performance of Mega-TTS scale with different sizes of the training data]
Here we evaluated the performance of our Mega-TTS scale with varying amounts of available data. In this experiment, all of the systems use 3 seconds of speech prompts. The results are shown in the following table. We can see that Mega-TTS performs well with different sizes of the training data, while VALL-E fails to obtain satisfying results when the data is insufficient. We also scale our Mega-TTS with 200K hours of speeches and the results can be found in Appendix B in the original version of the paper.

[About handling non-standard speech patterns]
We have generated some speech samples by directly utilizing non-standard speech voice prompts, such as laughing and other forms of emotional expression. The generated speech samples can be accessed in the "Additional Examples for Rebuttal" Section of our demo page: https://boostprompt.github.io/boostprompt/. From the demo page, we can see that Mega-TTS can accurately capture timbre information even from a short clip of a non-standard speech pattern. This demonstrates its effectiveness in "capturing and reproducing a wider range of human speech nuances". Again, we greatly appreciate your valuable suggestions and assistance.

Dear Reviewer,

Please respond to authors rebuttal and see whether they have addressed your concerns. Thanks!