HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis

Thank you very much for your positive feedback and constructive comments. Below, we respond to each comment.

Comment 1: The concept of time-varying audio codes is not a new development. For reference, please see the work by [Variable-rate Discrete Representation Learning (2021)] [https://arxiv.org/pdf/2103.06089].

Response: We have added the suggested paper to the reference list and updated Sections 2 accordingly.

Comment 2: A notable issue lies in maintaining temporal coherence and acoustic fidelity, as evidenced by the declining performance observed in the SIM score.

Response: We conducted an additional experiment to demonstrate that providing longer audio prompts can mitigate the issue and significantly improve the SIM score. As shown in Table 8, with a prompt length of 60 seconds, HALL-E slightly outperformed VALL-E. Since the reduced frame rate achieved by MReQ enables efficient generation, even with long prompts of 60 seconds, this result demonstrates that HALL-E is capable of achieving high-fidelity synthesis, highlighting its practical value.

Table 8. SIM as a function of audio prompt length.

Comment 3: The model continues to face difficulties in handling spontaneous speech, highlighting an ongoing challenge in this area.

Response: We agree with the reviewer that handling spontaneous speech remains a significant challenge. Improving prosody modeling within the AR models is essential for achieving greater naturalness in synthesized spontaneous speech. We believe that low frame rate tokens will be key to addressing this challenge, and that the MinuteSpeech dataset and HALL-E serve as valuable resources to facilitate research in this direction.

Thank you for the clarifications and revision. I think the authors addressed the major concerns that I had, and I raised the soundness score and overall rating of the paper. I still have a few clarification questions. I’d appreciate it if the authors could answer them.

• QA2: I believe estimated PESQ score (PESQ with \dagger) is no longer necessary in Table 3 and Table 9. Please remove the column unless there’s a certain reason to keep them.
• QA4: I have a few clarification questions.
  • What is “HALL-E” and “HALL-E (GT)” in your explanation? Which is the version with and without the audio prompt?
  • Why the number in the new table is different from the number in the prior version of the paper? For example, HALL-E score for 3s prompt was 0.663 in the original paper, but the result on the new table is 0.685 or 0.692, both does not match the number in the original paper.

Thank you very much for your insightful review and positive feedback. Below, we respond to each question and comment.

Question 1: Why and how can PESQ be used to filter out noisy data in the data curation pipeline?

Response: Following the previous study [1], PESQ scores are estimated using TorchAudio-Squim [2], which provides a reference-less method for predicting speech quality metrics. This is the reason why PESQ can be used in the data curation pipeline. We have clarified this in the revised paper.

The score distribution for the MinutesSpeech training dataset is shown in Figure 7 (Appendix B.3), with an average score of 3.10. Compared to LibriSpeech, which has an average score of 3.56, MinutesSpeech has lower-quality audio data. However, audio reconstructed by the Encodec and SpeechTokenizer models trained on MinutesSpeech achieved scores of 3.45 and 3.52, respectively. This indicates that the dataset possesses sufficient quality for TTS purposes.

[1] A. Vyas, et al., Audiobox: Unified Audio Generation with Natural Language Prompts, arXiv:2312.15821, 2023.

[2] A. Kumar, et al., TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudio, ICASSP, 2023. https://pytorch.org/audio/main/tutorials/squim_tutorial.html

Question 2: Why is the PESQ score for the ground truth so low in Table 3?

Response: The score for the ground truth (3.56) is low because we utilized the estimation method to make it comparable with the scores in the training dataset, as discussed above. Please see our response to Question 6 for the updated Table 3 with UTMOS (suggested criterion), PESQ and STOI scores obtained by reference audio.

Question 3: Zero-shot TTS experiment was conducted by "continual" setting or "cross-utterance" setting?

Response: The experiment was conducted by the continual setting. We have clarified this in Section 7.

Question 4: Did the authors exclude the audio prompt portion of the synthesized audio when computing the speaker SIM?

Response: Yes, we excluded the audio prompt when computing speaker SIM. This process was implemented as follows:

with torch.no_grad():
    emb1 = model(ref_wav[..., prompt_length:])
    emb2 = model(gen_wav[..., prompt_length:])

The table below compares speaker SIM values calculated with and without the audio prompt for HALL-E. Note that the increasing trend was observed with both VALL-E and HALL-E.

Thank you very much for your positive feedback and constructive comments. Below, we respond to each question and comment.

Question 1: Have you tried to train MReQ with a much smaller sampling rate, such as 2 or 4 Hz for the first layer?

Response: Yes, Table 18 in the Appendix shows the results. Reducing the sampling rate below 8 Hz remains challenging. Since the audio token length is comparable to the text token length at a sampling rate of 8 Hz, further reductions would require new architectures and training algorithms. We leave further reduction of the token length for both audio and text streams for future work.

Question 2: What is the accuracy of the sub-encoder E to predict $b_{k+1}$ based on $a_{k+1}$? Will this accuracy affect the final performance a lot?

Response: The sub-encoder is frozen during the training of HALL-E. Because the role of this sub-module is to aggregate predicted $a_{k+1,1}, \cdots, a_{k+1,\alpha_{k+1}}$ into $b_{k+1}$, accuracy cannot be computed for this sub-module.

Question 3: For the $L_{\mathrm{total}}$, are the weights of the three sub losses equal and all set to 1.0 ? Have you tried other settings?

Response: Appendix A.1 provides a more detailed description of $L_{\mathrm{total}}$, and Tables 11 and 12 list the weights for each loss function. The weights shown in Table 11 are from the official implementation of Encodec and have not been modified. Table 12 presents the weights for the FLD loss and HSR loss in our proposed method. These weights were selected after considering equal weighting and other weight values, choosing those that resulted in the greatest reduction in training loss.

Comment 1: Some important codec models are not compared with, such as DAC and Vocos.

Response: In preliminary experiments, we confirmed that SpeechTokenizer outperforms these conventional codec models. While training with various codec models could be interesting, it is not expected to improve the performance.

Comment 2: MReQ is only tested on the speech reconstruction, without evaluated with audio dataset.

Response: Our method relies on the fact that speech retains semantic information primarily in relatively low-frequency components. By incorporating a hierarchical structure, MReQ achieved high reconstruction performance by preserving semantic information in low frame rate tokens while accurately retaining acoustic information in high frame rate tokens. Therefore, extending our method to general audio is non-trivial, and verifying this would require different large-scale downstream tasks, which is beyond the scope of this work. As the reviewer have pointed out, achieving a lower frame rate for general audio, not limited to speech, is an important direction for future research.

We appreciate your positive feedback and detailed comments. Below, we respond to each question and comment.

Question 1: The description of the LRVQ modules appears to omit the use of commitment loss for residual vector quantization.

Response: The commitment loss was not omitted and is included in the loss term $L_{NAC}$. For clarification, we have added the detailed loss definition to the appendix A.1.

Question 2: Providing a detailed description of the pre-trained LLM-based TTS model.

Response: Pre-training was performed without incorporating the sub-modules of MRVQ. Thus, the pre-trained model is identical to VALL-E. We have clarified this in Section 7.1.

Comment 1: Although the MReQ module achieves low-resolution quantization while maintaining sample quality, its design is excessively complicated, posing challenges for future research to build upon.

Response: While the proposed architecture for achieving high zero-shot TTS performance may appear complicated, the core idea to introduce a recursive architecture to RVQ is simple. In particular, our implementation of MReQ in audiocraft/models/encoder.py within the provided code is not complicated, and is expected to facilitate future research and development.

Comment 2: Comparisons with state-of-the-art non-autoregressive zero-shot TTS models would better demonstrate the effectiveness and relevance of the proposed method.

Response: We conducted evaluation experiments using MaskGCT [1] (a state-of-the-art NAR TTS) with the pre-trained model provided by the authors.

Table 24 shows that the performance of MaskGCT significantly decreases as the length of synthesized speech increases. This indicates that even with NAR TTS, generating long speech in a single inference procedure is challenging. In the 30-second test setting in Table 25, the WER of MaskGCT is lower than VALL-E and HALL-E. However, its UTMOS is worse compared to HALL-E. In the 90-second test setting in Table 26, only HALL-E achieves a WER lower than 10.0%, with the highest UTMOS. For generating spontaneous speech, producing more natural durations becomes crucial. AR models excel in this aspect compared to NAR models, which likely contributes to the higher UTMOS scores observed. We have added these results in Appendix E.

The comparison suggested by the reviewer provided a clearer demonstration of the proposed method's effectiveness and relevance. We greatly appreciate the reviewer's insightful suggestion.

[1] Y. Wang, et al., MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer, arXiv:2409.00750.

Table 24. MaskGCT performance by test duration

Table 25. Performance comparison in 30-second test

Table 26. Performance comparison in 90-second test

Thank you very much for your insightful review and positive feedback. Below, we respond to each question and comment.

Question 1.1: This technique seems well suited for LLM-based TTS solutions, similar to VALL-E with AR and NAR components. However, can you also say anything about how this kind of encoding might be used as a general purpose codec for speech in general?

Response: The proposed approach could be advantageous for generative speech tasks, such as speech-to-speech translation beyond TTS, due to its ability to model long-term context using low sampling rate tokens. To reduce the bitrate, it would be preferable to improve the teacher model rather than rely on the distillation framework, as shown in Table 19.

Question 1.2: Furthermore, does this means that the PreQ and PostQ components of your LRVQ blocks would be unnecessary since they are primarily used to train the NAR components of the TTS model?

Response: It depends on the task and downstream architecture. For discriminative tasks, including automatic speech recognition, if learning a discriminator over the features of the first quantization layer is sufficient, the PreQ and PostQ components may no longer be needed. For generative architectures that do not rely on NAR components, such as Fisher and Moshi using the AR+AR combination, components like PreQ and PostQ may be helpful but this requires further investigation.

Question 2: With a VALLE model based on Encodec, once can remove RVQ code sequences from the NAR generation to get faster generation, in exchange for coarser generation. For this MReQ based paradigm, how does the audio quality here change?

Response: If our understanding is correct, the reviewer's intent is that using only the output of the AR model and omitting the NAR model can result in faster generation, albeit with coarser quality.

The table below shows the results when audio is generated using only the AR model, omitting the NAR model in each method. As expected, the overall performance deteriorates; however, trends in metrics such as WER remain similar to those observed in Table 4, indicating that our method (HALL-E) continues to demonstrate effectiveness. Nevertheless, as illustrated in Figure 5, the NAR model does not demand significant computational resources, so it is advisable to employ it to improve quality.

Question 3: For the second ablation in section 7.4, how does this effect appear with the VALLE model that you trained?

Response: In response to the feedback, we have added results for VALL-E (trained on MinutesSpeech-90s) to Table 8, where the audio prompt length was varied from 3 seconds to 60 seconds. The SIM scores are recalculated over audio samples with a minimum utterance length of 65 seconds.

As shown below, a similar effect was observed with VALL-E. However, the gap between HALL-E and VALL-E narrowed as the prompt length increased. Finally, HALL-E slightly outperformed VALL-E at a prompt length of 60 seconds, demonstrating that highly accurate cloning is achievable with HALL-E.

Table 8. SIM as a function of audio prompt length.

Question 4: For the third ablation in section 7.4, it is very surprising that even w/o pretraining, it does not seem like this technique suffers significantly (compared to the performance drop in Table 11 for HALL-E). Why is that?

Response: This is because the MRVQ module involves three 48 Hz quantization layers. It is known that with Encodec, reducing the number of RVQ layers to 6 or 3 degrades performance but does not cause collapsed speech. Therefore, the performance drop was not significant compared to that of HALL-E.