CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech

We sincerely appreciate your positive review and constructive feedback.

Regarding the tradeoff between efficiency and effectiveness in varying codeword rate

As per your great suggestion, we conducted an ablation study to analyze the impact of varying codeword rates in the proposed system and addressed it in the first section of “General Response to AC and Reviewer”. The experimental results demonstrate a trade-off: reducing the code emitting frequency degrades audio quality while increasing it diminishes language modeling performance. We also reported on the inference speed of the entire system in relation to varying code frequencies. This finding illustrates that our main model operates at a balance point within this trade-off, efficiently managing both aspects.

We would like to thank the reviewer for providing a positive review and spotting unclear points.

Regarding the effect of codeword emitting frequency

Thank you for pointing out the parts in our paper that were not explained clearly. The semantic tokens in AudioLM are a sequence of 50Hz, while our tokens consist of 32 sequences of 10Hz due to the use of 32-depth residual vector quantization. This means that when representing audio, we can achieve a higher "length" compression rate compared to AudioLM by compressing the audio into multiple sequences using RVQ. We have conducted experiments and evaluations regarding the codeword emitting frequency, which we addressed in the first section of “General Response to AC and Reviewer”. These results demonstrate that our main model operates in the sweet spot for the trade-off. We sincerely request you to review them for additional context.

Definitions of $y$ and $\hat{y}$

For the definitions of the variables, please refer to the second paragraph in Sec. 3.2.1. $y$ and $\hat{y}$ denote a mel-spectrogram and its reconstruction from the decoder, respectively.

We would like to thank the reviewer for providing constructive feedback and spotting unclear points.

Contribution of this paper

To address the reviewer's concerns and clarify our methodology, we emphasize the distinctions between our approach and similar existing generative methodologies, VQGAN and RQ-Transformer, among generative models such as VAE and latent diffusion.

1. Introduction and utilization of probabilistic residual vector quantization: This feature not only enhances codebook usage to improve reconstruction quality but also offers a principled approach to generate residual codes from latent language modeling within a variational framework.
2. Employing a single autoregressive transformer: Our method proposes a process where a single autoregressive transformer can generate $N$ tokens at each decoding step by utilizing the probabilistic modeling of the residual vector quantizer, without the need for depth-transformers or non-autoregressive (NAR) transformer decoders. Our method shares similarities with VQGAN in that both involve 1) autoencoding of input data using VQVAE and 2) generation of the quantized latent representation using an autoregressive transformer. However, the vector quantizer in VQGAN employs plain vector quantization, producing a sequence of tokens, thereby enabling the transformer to model the discrete sequence with a softmax output. In contrast, RQ-Transformer and neural audio codec language models, which use $N$-depth residual vector quantization, produce $N$ sequences of tokens. This results in the need for not only an autoregressive transformer to decode along the time axis but also an additional depth-transformer to model $N$ tokens along the depth axis or a NAR transformer decoder to generate a token sequence at each decoding step.

Regarding the performance of the proposed method

Thank you for your constructive criticism, which highlights the weaknesses in our research. We are truly grateful for your thorough review.

We acknowledge the observed performance gap of our method compared to Voicebox, particularly in cross-sentence tasks where models are required to generate outputs from discontinuous texts and partially truncated audio prompts. To address this issue, we have empirically analyzed the performance improvements when incorporating phonemes as inputs (as done in VALL-E, Voicebox, and SPEAR-TTS) and durations (as in Voicebox and YourTTS) into our model. We also discuss methods that directly train on cross-sentence tasks, such as those employed by SPEAR-TTS. These investigations are critical in understanding the reasons behind our model's performance drop in cross-sentence tasks compared to other models. In response to the concerns you raised, we have provided detailed explanations in the second section of our “General Response to AC and Reviewer”.

Nonetheless, it is important to emphasize why our approach uniquely relies on plain text input, unlike other approaches. This is due to our model's ability to seamlessly integrate a pre-trained large language model, which is trained on a diverse array of texts and tasks. This integration allows us to achieve performance comparable to other baselines that employ more complex modeling.

Moreover, as per your great suggestion, we have included samples from other competitive models (VALL-E, NaturalSpeech 2, and Mega-TTS) on our demo page for comparison. These samples were taken from each model's demo pages. We note that the small number of these additional samples is due to the lack of official codes or pre-trained models. We acknowledge that the performance analysis using this limited number of additional samples may not have statistical significance. Nevertheless, we kindly request the reviewer to consider these additional samples in their evaluation.

demo page: https://clam-tts-mos.s3.us-east-2.amazonaws.com/demo/index.html

We would like to express our sincere gratitude for the constructive feedback and detailed suggestions, which helped us improve the manuscript. We provide point-by-point replies below.

Regarding the performance of the proposed method compared to VoiceBox

Thank you for your constructive criticism, which highlights the weaknesses in our research. We are truly grateful for your thorough review. We acknowledge the observed performance gap of our method compared to Voicebox.

In response to the concerns, we would like to clarify that our model can easily incorporate phoneme or duration to improve the performance. We have empirically analyzed the performance improvements when incorporating phonemes as inputs (as done in VALL-E, Voicebox, and SPEAR-TTS) or durations (as in Voicebox and YourTTS) into our model. We also discuss methods that directly train on cross-sentence tasks, such as those employed by SPEAR-TTS. These investigations are critical in understanding the reasons behind our model's performance drop in cross-sentence tasks compared to other models. We provide the empirical results and detailed discussion in the second section of “General Response to AC and Reviewer”.

Nonetheless, it is important to emphasize why our approach uniquely relies on plain text input, unlike other approaches. This is due to our model's ability to seamlessly integrate a pre-trained large language model, which is trained on a diverse array of texts and tasks. This integration allows us to achieve performance comparable to other baselines that employ more complex modeling.

Clarification of the descriptions

Thank you for pointing out the parts of our paper where the explanation was insufficient. We are grateful for your detailed feedback. In our work, the latent language model produces three distinct outputs, representing 1) the mixture weights, 2) the means of the mixture of Gaussian distribution, and 3) the probability of ending the generation. Initially, we described these three components collectively as “parallel predictors”. However, in light of your feedback, we have replaced this term in the text with the aforementioned description. Furthermore, we elucidate the decoding step in which multiple tokens are generated from the output mixture of Gaussians (MoG) distribution, consisting of the mixture weights and the means. During each decoding step, a latent vector sampled from this distribution is transformed into latent codes through the residual vector quantizer of Mel-VAE. The autoregressive decoding step concludes when the probability of ending the generation reaches or exceeds 0.5.

Missing references & typos

In response to your feedback, we have added the missing reference and corrected typos, as well as other areas requiring revision, in the updated version of the paper.

Ablation study regarding the frequency

To address the critical concern you raised, we conducted an ablation study to analyze the impact of varying codeword rates in the proposed system and addressed it in the “General Response to AC and Reviewer”. The experimental results demonstrate a trade-off: reducing the code emitting frequency degrades audio quality while increasing it diminishes language modeling performance. This finding illustrates that our main model operates at a balance point in this trade-off. We sincerely request you to review them for additional context.

Using mel-spectrogram for the desired compression

Our choice to use mel-spectrograms instead of raw waveforms as the input for the VAE is driven by a practical consideration. It requires substantially more memory and trainable parameters to train VAE with raw waveforms due to the extended input length. In contrast, using mel-spectrograms allows us to train VAE with considerably fewer resources as well as to use an off-the-shelf vocoder to convert mel-spectrograms to raw waveforms. Nonetheless, as the reviewer pointed out, pursuing higher reconstructed audio quality with the desired compression in an end-to-end manner with raw waveforms can be a promising direction for future work.

We do appreciate your positive review and valuable feedback, and we hope our response fully addresses your concern.

CLaM-TTS with and without the two-stage system

Thank you for your valuable comment. While our method seamlessly generates from coarse to fine-grained tokens without the necessity for a two-stage pipeline, we do not believe that modeling our approach as a two-stage system would inherently degrade its performances. We want to clarify that, in this work, we provide a method that efficiently addresses the complex modeling challenges inherent in text-to-speech synthesis systems at scale. Our proposed probabilistic residual vector quantization not only enhances the reconstruction of audio quality but also offers a principled way to generate multiple codes within a variational inference framework. This approach effectively allows for large-scale text-to-speech synthesis without adding modeling complexities or compromising inference speed.

scaling up dataset and model sizes

Thank you for highlighting the importance of conducting additional experiments to substantiate our claims. We have carried out an ablation study. Following your recommendation, we will incorporate these results into the revised manuscript.

Scaling up dataset size: The results in the below table show that the performance of our model improves with an increase in the volume of training data. Specifically, we trained ByT5-base models on different sizes of training datasets: 1. (small dataset) 50% of the MLS English subset; 2. (normal dataset) the full MLS English subset; and 3. (large dataset) a comprehensive English dataset that includes MLS English subset, Gigaspeech, LibriTTS-R, VCTK, and LJSpeech. We observed a noticeable degradation in performances (WER, CER, and speaker similarity) when the small dataset was employed. Conversely, training on the large dataset resulted in a performance enhancement in speaker similarity while maintaining WER and CER compared to the model trained on the normal dataset. This evidence strongly indicates that scaling up training data can significantly boost our model's performance.

Scaling up model size: Due to time constraints, we were unable to train ByT5-base model on the large dataset with as many steps as the trained ByT5-large model. Consequently, a precise comparison is challenging at this stage, but we share the current results. We will include the experimental results in the revised manuscript as soon as they become available.

Criticism 2: The performance of the proposed method compared to other methods

We are grateful for the insightful observations made by the reviewers (R2, R3) concerning the comparative performance of our proposed method against other methods. The reviewers pointed out that Voicebox outperforms our method, and noted that the incorporation of phoneme and duration information may not be a significant factor in explaining this performance gap.

In light of these comments, we would like to clarify that our model can easily incorporate duration to improve the performance. We have validated and discussed several methods for performance enhancement in the subsequent paragraphs. Nonetheless, it is important to emphasize why our approach uniquely relies on plain text input, unlike other approaches. This is due to our model's ability to seamlessly integrate a pre-trained large language model, which is trained on a diverse array of texts and tasks. This integration allows us to achieve performance comparable to other baselines that employ more complex modeling.

We incorporated the alignment search method from YourTTS to apply identified input durations and then evaluated its performance. The training involved adding the results of applying strided convolution to expanded character embeddings according to duration, to our decoder input embeddings. It is important to note that the only additional parameters in the resulting model are 164 character embeddings and a strided convolutional layer. The performance in terms of WER, CER, and speaker similarity is shown in the table below. When alignment from the alignment search was provided, we observed no quality gain in the continuation task.

However, we can observe a noticeable performance improvement in all metrics for the cross-sentence task, where our method exhibits greater degradation relative to other models and models are required to generate outputs from discontinuous texts and partially truncated audio prompts.

Note that even with only negligible additional parameters, significant performance enhancement occurs in our method when duration is provided, particularly in cross-sentence cases where the input text is concatenated with partially truncated sentences. This also explains the limitations of our model's performance in the cross-sentence task where continuous sentences are not provided.

One potential solution to address this issue, akin to the approach used by SPEAR-TTS which maintains consistent performance in cross-sentence tasks, could be to explicitly inform the discontinuity between the audio prompt and the decoder output by a prompt separator token. This simple yet auxiliary method might help alleviate the identified problem. We acknowledge an error in the CER score (2.21) for SPEAR-TTS in the cross-sentence task and have corrected it to 1.92 in our manuscript (Table 1).