Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models

We thank the reviewer for the positive review and constructive feedback, and we hope our response fully resolves your concerns.

Question 1:

The paper mainly focuses on consistency-based methods and doesn't extensively compare these approaches to other potential techniques for handling the many-to-one mapping problem, such as the simpler methods introduced in the introduction section (e.g., setting the kernel size to 1). Including such baselines would be helpful for comparison in Table 2.

Response 1: We attempted to set the kernel size of the convolutional layers in the codec to 1. This means that each audio sample would be discretized into an audio token without being influenced by its context, thereby completely avoiding the many-to-one mapping issue. However, this approach introduced other challenges:

(1) Extremely low training and inference efficiency. Typically, a speech segment averages 6 seconds in length and, at a sampling rate of 16,000, consists of 96,000 audio samples. Consequently, the audio token sequence would also be 96,000 tokens long, rendering it impossible to encode such a length of speech in a single pass even on an A100 GPU.

(2) The excessively long audio token sequence is unsuitable for downstream neural codec language models. A sequence of 96,000 audio tokens is exceptionally lengthy, making it difficult for the language model to effectively train and perform inference.

(3) The quality of the reconstructed audio deteriorates, as demonstrated in Table 1 in this section.

Our goal is to reduce inconsistency phenomenon in the codec without compromising its ability to effectively reconstruct audio, ensuring that the audio tokens are more suitable for the language model. Setting the kernel size to 1, while eliminating inconsistency, results in subpar audio reconstruction and generates overly long audio token sequences that are impractical for modeling with neural codec language models.

We appreciate your attention to these points and value your feedback, which is instrumental in refining our approach.

Table 1: neural audio codec trained with different configurations.

Question 2

The paper would benefit from a baseline without the consistency constraint. The current baselines are different methods/codecs like Encodec, DAC, etc., which might be trained on different datasets. This raises the possibility that the better reconstruction in the author's proposed method might be due to larger training data, rather than the addition of consistency-based loss. To verify this, it would be helpful to include a baseline RVQ-GAN or a model without the additional losses.

Response 2: Table 1 in our paper may have been misleading. As shown in Table 2 in this section, the total duration of training data for various neural audio codecs is listed. To ensure a fair comparison with other neural audio codecs, we have aligned the bandwidth and training data as closely as possible with theirs. In Table 1 in our paper, the total duration of the training audio for our codec does not exceed that of the other codecs, and in some cases, it is even less. Nevertheless, our codec demonstrates superior reconstruction performance. This highlights that our codec's ability to better reconstruct audio is not due to larger training datasets.

We additionally trained a codec without the consistency loss, and the results are presented in Table 3 in this section. As evidenced by the results, consistency loss does not impact the codec in speech reconstruction. Instead, the consistency loss primarily contributes to enhancing the codec's consistency accuracy.

Table 2: Total duration of training data for various neural audio codecs.

Table 3: The speech reconstruction results on LibriTTS test set.

We greatly value your opinions and hope these revisions adequately address the concerns you previously raised. If you find that our revisions meet your expectations, could you please reassess and update your rating? Thank you for your hard work and assistance.

Thanks for your valuable feedback, and we hope our response fully resolves your concerns.

[About Weakness 1]

Response 1: We have responsed this question in the Answer 2 of the Section to all reviewers. In our research, we obtain a series of codecs with various consistency accuracy, which we then utilized as audio tokenizers to train VALL-E and observe how consistency impacts WER. As shown in Table 1 in this section, there is a positive correlation between consistency accuracy and WER improvement. Specifically, as the consistency accuracy increases, the WER correspondingly decreases. To visually illustrate the relationship between WER and consistency, we also plotted the figurehttps://github.com/ConsistencyInNeuralCodec/consistencyinneuralcodec.github.io/blob/main/fig/consistency_wer/consistency_wer.jpg.

As demonstrated in Table 2 in this section, there is a positive correlation between improvements in WER and consistency accuracy of neural audio codecs. For neural audio codecs with the same architecture, we observed a notable reduction in cross-entropy loss when applying consistency constraints during VALL-E training. Moreover, both the top-1 accuracy and top-10 accuracy in predicting the first layer codebook showed enhancements. This suggests that neural audio codecs employing consistency constraints are more suitable for VALL-E.

Table 1: how different consistency accuracy within neural audio codec affect the WER of speech generated by VALL-E.

Table 2: Analysis of Consistency Constraints in Neural Audio Codec on VALL-E Performance. This table presents comprehensive metrics including: cross-entropy loss for first codebook layer prediction during training, top-1 accuracy for first audio token prediction, top-10 accuracy (whether the correct audio token appears in top-10 candidates), and inference metrics (WER and SIM) of generated speech.

[About Weakness 2]

Response 2: Our study focuses on a novel method utilizing consistency constraints to enhance speech generation by the neural codec language model through the reduction of DRI within the neural audio codec. Our primary objective is specifically targeted at speech generation and does not aim to address challenges related to music and sound effects universally. Our proposed method is designed to be general and applicable across various neural audio codecs. To demonstrate the efficacy of our approach, we conducted validations on the neural codec language model. We emphasize that, theoretically, our method holds applicability across multiple neural codec language models. Additionally, we hope that our findings offer valuable insights and inspiration to other fields.

[About Question 5]

Response 5: (1) Our objective in proposing perturbation consistency is to apply slight perturbations to the audio that constrain the consistency of the audio tokens without altering the auditory perception.

(2) There are other perturbation methods, including white noise, background noise, reverberation, and different audio quantization techniques. The essence of these approaches is data augmentation, aimed at enhancing model stability by generating more data. However, these perturbation methods significantly alter the original audio, resulting in a noticeable change in auditory perception. The neural audio codec, being a lightweight model with approximately 66M parameters, struggles with these changes. In early experiments, we also explored other perturbation methods. As shown in Table 4, the quality of reconstruct speech, as well as consistency are decreased.

Table 4: The impact of different perturbation methods on neural audio codecs.

[About Question 6]

Response 6: As shown in Table 5 in this section, when applying the consistency constraint results, VALL-E generates higher quality speech with a lower WER, with reductions observed in the insertion, substitution, and deletion metrics. Notably, the deletions decrease significantly from 1359 to 405, which might indicate that, without the consistency constraint, VALL-E tends to generate speech with more repeated words.

Table 5: detailed WER information

[About Weakness 3 and Question 1]

Response 3: It is considerable that other recent works that have reproduced the results of VALL-E and improved upon them[1, 2, 3, 4, 5], reporting WER significantly below 10%. We have provided the code and training scripts in the links available to all reviewers https://github.com/ConsistencyInNeuralCodec/ConsistencyInNeuralCodec. We invite you to utilize these resources to replicate our results and inspect our model. Consequently, we assert that re-ranking was not employed in our methods.

We have responsed this question in the Answer 1 of the Section to all reviewers. In our paper, we compared several TTS systems. And these TTS systems were originally evaluated with various testing configurations in their respective paper [1, 3, 4, 6]. To ensure a strictly fair comparison of these TTS systems, we performed an evaluation on a subset from LibriTTS. We utilized Whisper v3 as the ASR model to measure the WER like XTTS[7], and employed 3D-speaker for SIM evaluation. To further substantiate the effectiveness of our model, we conducted additional experiments following the testing configurations in the VoiceCraft[6]. It is worth noting that VoiceCraft did not utilize re-ranking, achieving a WER of 4.5, which is consistent with our results as shown in Table 3 in this section.

Table 3: The speech generation results on LibriTTS test set. Bold means the best result, and underline means the second-best result. Ours and Ours w/o consistency constraint denote the same neural audio codecs with and without consistency constraint. The subscripts of the neural codec language models (e.g., 330M, 44Kh) denote the model size and data scale.

[1] Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

[2] VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

[3] VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment

[4] DiTTo-TTS: Efficient and Scalable Zero-Shot Text-to-Speech with Diffusion Transformer

[5] Autoregressive Speech Synthesis without Vector Quantization

[6] VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

[7] XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model

[About Question 4]

Response 4: We obtained several hard examples from the VALL-E R demo page https://www.microsoft.com/en-us/research/project/vall-e-x/vall-e-r/ and tested them on our VALL-E model, as shown in the Table below.

Initially, because we compare various TTS systems with different testing configurations, we conduct tests on a subset extracted from LibriTTS, utilizing Whisper v3 as the ASR model to evaluate WER[1], and employing the 3D-speaker toolkit to assess SIM[2]. Reviewers mention that our evaluation metrics are not aligned with other papers when presenting our results, which would be better validated under different testing settings as a supplement. Thus, we select a representative work, VoiceCraft[3], and follow its testing configuration to evaluate results on different subsets. Due to the use of different testing settings, including new ASR models (Whisper v2 [3]) and new speaker similarity detection tools(WavLM[4]), it would obviously lead to different results, but our conclusions remain consistent, and the distribution of results for each baseline is also consistent with previous testing settings.

[1] XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model

[2] 3D-Speaker-Toolkit: An Open Source Toolkit for Multi-modal Speaker Verification and Diarization

[3] VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

[4] WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

Dear Reviewer aNo3,

We sincerely thank you for your valuable suggestions on our work. Based on your insightful feedback, we have provided additional clarifications, and hope our responses have fully addressed your concerns.

We hope you had a wonderful Thanksgiving holiday during the past few days. As the rebuttal deadline approaches, we kindly look forward to receiving your further feedback. Once again, we deeply appreciate the time and effort you have dedicated to reviewing our paper.

Best regards,

1. The newly reported WER of 0% for GT appears suspicious, and the results for VALL-E and VoiceCraft contradict my prior experience. I will refrain from commenting further on this matter.

2. The authors did not address my third question. Furthermore, the observed correlation between WER and consistency is measured using VALL-E trained with the proposed codec. Several key variables in the codecs, such as training audio lengths, the receptive field of the audio codec, and codec causality, are under-investigated which undermines the clarity and significance of the paper's key contributions.

3. The authors did not evaluate the proposed method's performance on music audio codecs, despite claiming a focus on speech. However, the paper's title, Discrete Audio Tokens for Neural Codec Language Models, implies a broader scope that extends beyond speech.

4. The PDF does not appear to have been updated during the rebuttal phase. In its current form, I think the manuscript is not ready for publication.

[About Weakness 5]

Response 5: This paper is a research-oriented work, where we focuse on a novel method utilizing consistency constraints to enhance speech generation by the neural codec language model through the reduction of DRI phenomenon within the neural audio codec. Our primary objective is specifically targeted at the analysis and the mitigation of DRI phenomenon, and does not aim to improve the model structure. Our proposed method is designed to be general and applicable across various neural audio codecs. To demonstrate the efficacy of our approach, we conducted validations on the neural codec language model. Additionally, we hope that our findings offer valuable insights and inspiration to other fields.

Thank you for your thorough review and valuable suggestions on our paper. Based on your feedback, we have made the corresponding revisions and improvements.

We greatly value your opinions and hope that these revisions adequately address the issues and concerns you previously raised. If you find that our revisions meet your expectations, could you please reassess and update your rating? Once again, thank you for your hard work and assistance.

We are really grateful for your constructive review and valuable feedback, and we hope our response fully resolves your concerns.

[About Weakness 1]

Response1: Table 1 in our paper may have been misleading. As shown in Table 3 in this section, the total duration of training data for various neural audio codecs is listed. To ensure a fair comparison with other neural audio codecs, we have aligned the bandwidth and training data as closely as possible with theirs. In Table 1, the total duration of the training audio for our codec does not exceed that of the other codecs, and in some cases, it is even less. Nevertheless, our codec demonstrates superior reconstruction performance. This highlights that our codec's ability to better reconstruct audio is not due to larger training datasets.

Additionally, we have trained VALL-E on 960 hours of audio data. To further validate the effectiveness of our approach on large-scale data, we have extended the training duration for VALL-E to 44,000 hours, while codec was still trained on 960h.

Table 3: Total duration of training data for various neural audio codecs.

[About Weakness 2]

Response 2: Thank you very much for your suggestions on writing and illustration. We will revise the paper according to your recommendations.

[About Weakness 3]

Response 3: Our codec utilizes 8 codebooks, each with a vocabulary size of 1024, and a base dimension of 128. It operates at a frame rate of 50Hz for audio sampled at 16kHz, resulting in a total parameter count of 66 M. On average, the speech to be encoded has a duration of 7 seconds, leading to an estimation of FLOPs of approximately 2.57 x 10^12.

For the neural codec language model, both the AR model and NAR model are built upon the same Transformer architecture. This setup includes 12 layers, 16 attention heads, an embedding dimension of 1024, and a feedforward layer dimension of 4096, collectively comprising about 365M parameters. Typically, in a TTS task, the phoneme sequence has a length of 100. The audio tokenizer extracts a sequence of audio tokens with a length of 150 from a 3-second prompt audio, and the generated audio token sequence has a length of 550. This results in a computation of FLOPs approximately equal to 1.66 x 10^11.

We conducted experiments with two sets of codecs and VALL-E models differing in parameter size, as shown in Table 4 and Table 5 in this section. As the parameter size increases, both the codec and VALL-E exhibit enhanced performance.

Table 4: codec with different parameter size

Table 5: VALL-E with different parameter size

[About Weakness 4]

Response 4: In the loss function described in Section 4.2, the parameters are set as $\lambda_{adv} = 0.11$, $\lambda_{fm} = 11.11$, and $\lambda_{rvq} = 1.0$. When applying consistency constraints to the codec, $\lambda_{con} = 10.0$.

We sincerely appreciate your valuable feedback and suggestions regarding the writing and formatting of the paper. We have addressed the identified issues and have uploaded a revised version of the PDF. Our major revisions are outlined as follows:

(1) We have added the training duration for each neural audio codec in Table 1 in the paper of the original paper using subscripts.

(2) We have corrected the format of the abstract, the numbers of the mathematical equations and other formatting issues.

(3) In Section 8.7 in the appendix, we report the total number of parametes and FLOPs for the neural audio codec and VALL-E, respectively. Additionally, we examine how VALL-E's performance varies with different parameter sizes.

(4) In Section 4.1 in the paper, we have included the specific values of the weight $\lambda$ used in the loss function.

We still hope you can avoid being influenced by certain reviewers. Our code has been uploaded to https://github.com/ConsistencyInNeuralCodec/ConsistencyInNeuralCodec, and you are welcome to reproduce and verify our results.

Once again, thank you for your diligent efforts in reviewing our work.

Dear Reviewer eE2Q,

We sincerely thank you for your valuable suggestions on our work. Based on your insightful feedback, we have provided additional clarifications, and hope our responses have fully addressed your concerns.

We hope you had a wonderful Thanksgiving holiday during the past few days. As the rebuttal deadline approaches, we kindly look forward to receiving your further feedback. Once again, we deeply appreciate the time and effort you have dedicated to reviewing our paper.

Best regards,

We would like to express our gratitude again for your time and effort in reviewing our paper. If you find that our revisions meet your expectations or address some of your concerns, could you please reassess and update your rating?

We thank the reviewer for the constructive and professional review.

[About Weakness 1 and Question 1]

Reponse 1: We demonstrate the impact of the codec with consistency constraint and without consistency constraint on downstream VALL-E, as shown in Table 1 in this section, indicating that speech generated by the codec with consistency constraint have reductions in insertion, substitution, and deletion.

To more intuitively illustrate the DRI issue, we trained codec with different configurations of slice perturbation (slice percentages) to obtain codecs with different consistency, and utilized these codecs as audio tokenizers and trained VALL-E to observe how consistency affects the WER, as shown in Table 2 in this section and the figure https://github.com/ConsistencyInNeuralCodec/consistencyinneuralcodec.github.io/blob/main/fig/consistency_wer/consistency_wer.jpg, in the Answer 2 to the Section to all reviewers. Specifically, as the consistency accuracy increases, the WER correspondingly decreases.

Table 2 in our paper demonstrates the pipelines of ours (with or without consistency constraint) + VALL-E, which presents models trained with inconsistent and consistent audio tokens. Compared to the VALL-E model trained on inconsistent audio tokens, VALL-E trained on consistent audio tokens achieves significant improvement in WER and similarity. This indicates that consistent audio tokens leads to better TTS performance.

Table 1: detailed WER information.

Table 2: how different consistency accuracy within neural audio codec affect the WER of speech generated by VALL-E. | Consistency Accuracy | 49.15 | 64.74 | 76.75% |

[About Weakness 2]

Reponse 2: Our goal is to enforce audio token consistency while preserving perceptual audio quality. Phase perturbation achieves this by introducing minimal modifications that are imperceptible to human listeners.

Also, we explored various perturbation techniques, including white noise, reverberation and different audio quantization methods. While these traditional data augmentation approaches aim to improve model robustness, our early experiments (shown in Table 3 in this section) revealed that they significantly degrade audio reconstruction quality and reduce consistency metrics and ViSQOL scores. Data augmentation approaches are particularly challenging for our lightweight neural audio codec (approximately 70M parameters).

In our practice, we found that phase perturbation demonstrates significant advantages over alternative perturbation approaches. The relationship between spectral phase and temporal information is fundamental: phase information determines the temporal arrangement of different frequency components in a signal, thereby affecting the temporal waveform. However, phase modifications have minimal impact on speech perceptual characteristics.

This unique property creates a valuable opportunity: the audio details affected by phase perturbation are largely irrelevant for both human perception and downstream VALL-E modeling. As a result, we can leverage phase perturbation for data augmentation without compromising the codec's reconstruction capabilities.

Table 3: The impact of different perturbation methods on the neural audio codec.

[About Question 2]

Reponse 3:

We have also observed this scenario and have conducted additional experiments to further investigate it. By varying the weight of perturbation consistency, we obtained codecs with various consistency accuracies, allowing us to study the relationship between consistency and WER when only perturbation consistency is considered. Our findings indicate that, compared to slice consistency, perturbation consistency has a smaller impact on consistency accuracy. However, as shown in Table 4, improving consistency through phase perturbation still enhances the performance of downstream VALL-E in generating audio. For a clearer illustration, we have also included the figurehttps://github.com/ConsistencyInNeuralCodec/consistencyinneuralcodec.github.io/blob/main/fig/consistency_wer/consistency_wer_phase_aug.jpg.

We speculate that phase perturbation may partially decouple information within the audio. Specifically, the phase perturbation affect the temporal arrangement of audio signals, altering the structure of some information. This may effectively prevent certain features from overfitting to details not prominent for the model, thereby improving WER performance. At the same time, this suggests that certain phase information of audio signals is not decisive for the neural codec language model's prediction of audio tokens. Therefore, even with lower consistency in some experimental settings, a good WER can still be achieved.

This result highlights that, while increasing consistency generally improves performance of generated speech, we also need to consider the complexity of other audio features, such as phase information, and their potential impact on the neural codec language model. We will update the paper to include these discussions and appreciate you for pointing out this observation, which helps improve our research.

Table 4: how different consistency accuracy under different phase perturbation within neural audio codec affect the WER of speech generated by VALL-E.

[About Question 3]

Reponse 4: In Table 1 in our paper, pipelines of SpeechTokenizer + USLM, SpeechTokenizer + AnyGPT are based on open-source models, and the pipeline of SpeechTokenizer + VALL-E is reproduced by Amphion[1].
The pipelines of ours (with or without consistency constraint) + VALL-E are independently reproduced by us. We strictly adhered to the original VALL-E[1] paper's model architecture and training parameters. Our reproduced VALL-E demonstrated better speech generation metrics compared to the original VALL-E, not inferior results. Therefore, when assessing the consistency constraints, our experiments conducted on our own VALL-E reproduction provide stronger evidence. Additionally, we have included the experimental results of Amphion's[1] reproduced VALL-E for your reference and comparison.

In our study, we compared various TTS systems. And the original papers presenting these TTS systems employed different testing configurations [2, 3, 4]. To provide a fair and unbiased comparison, we conducted tests on a subset extracted from LibriTTS, utilizing the Whisper v3 as the ASR model to evaluate WER like XTTS[4], and employing the 3D-speaker toolkit to assess SIM. Additionally, we have followed the testing configuration outlined in VoiceCraft[2] for further experimentation to validate our model's effectiveness. The results of these experiments are detailed in Table 1 in the Answer 2 to the Section to all reviewers.

Table 5: The speech generation results on LibriTTS test set. Bold means the best result, and underline means the second-best result. Ours and Ours w/o consistency constraint denote the same neural audio codecs with and without consistency constraint. The subscripts of the neural codec language models (e.g., 330M, 44Kh) denote the model size and data scale.

[1] Amphion: An Open-Source Audio, Music, and Speech Generation Toolkit

[2] Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

[3] VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

[4] XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model

Thank you for your thorough review and valuable suggestions on our paper. We greatly value your opinions and hope that these revisions adequately address the issues and concerns you previously raised. If you find that our revisions meet your expectations, could you please reassess and update your rating? Once again, thank you for your hard work and assistance.

Dear Reviewer 2NT9,

We sincerely thank you for your valuable suggestions on our work. Based on your insightful feedback, we have provided additional clarifications, and hope our responses have fully addressed your concerns.

We hope you had a wonderful Thanksgiving holiday during the past few days. As the rebuttal deadline approaches, we kindly look forward to receiving your further feedback. Once again, we deeply appreciate the time and effort you have dedicated to reviewing our paper.

Best regards,

We would like to express our gratitude again for your time and effort in reviewing our paper. If you find that our revisions meet your expectations or address some of your concerns, could you please reassess and update your rating?

[About Weakness 3]

Response 3: In Table 1 of our paper, pipelines of SpeechTokenizer + USLM, SpeechTokenizer + AnyGPT are based on open-source models, and the pipeline of SpeechTokenizer + VALL-E is reproduced by Amphion[1]. The pipelines of ours (with or without consistency constraint) + VALL-E are independently reproduced by us. We strictly adhered to the model configurations outlined in the VALL-E paper[1]. Specifically, both the AR model and NAR model are built upon the same Transformer architecture. This setup includes 12 layers, 16 attention heads, an embedding dimension of 1024, and a feedforward layer dimension of 4096. Our reproduced VALL-E demonstrated better speech generation metrics compared to the original VALL-E, not inferior results. Therefore, when assessing the consistency constraints, our experiments conducted on our own VALL-E reproduction provide stronger evidence. Additionally, we have included the experimental results of Amphion's[1] reproduced VALL-E for your reference and comparison.

[1] Amphion: An Open-Source Audio, Music, and Speech Generation Toolkit

[2] Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

Question 1: Consistency Accuracy and WER Correlation

Reponse 4: In our research, we conducted experiments with altering the consistency perturbation during the training process of the codec, and resulted in a series of codecs with various consistency accuracy, which we then utilized as audio tokenizers to train VALL-E and observe how consistency impacts WER.

As demonstrated in Table 3 in this section, there is a positive correlation between improvements in WER and consistency accuracy of neural audio codecs. For neural audio codecs with the same architecture, we observed a notable reduction in cross-entropy loss when applying consistency constraints during VALL-E training. Moreover, both the top-1 accuracy and top-10 accuracy in predicting the first layer codebook showed enhancements. This suggests that neural audio codecs employing consistency constraints are more suitable for VALL-E.

Table 4 in this section illustrates the effect of various consistency accuracy within neural audio codecs on the WER of VALL-E. As shown Table 4 in this section, there is a positive correlation between consistency accuracy and WER improvement. Specifically, as the consistency accuracy increases, the WER correspondingly decreases. To visually illustrate the relationship between WER and consistency, we also plotted the figure https://github.com/ConsistencyInNeuralCodec/consistencyinneuralcodec.github.io/blob/main/fig/consistency_wer/consistency_wer.jpg.

Table 3: Analysis of Consistency Constraints in Neural Audio Codec on VALL-E Performance. This table presents comprehensive metrics including: cross-entropy loss for first codebook layer prediction during training, top-1 accuracy for first audio token prediction, top-10 accuracy (whether the correct audio token appears in top-10 candidates), and inference metrics (WER and SIM) of generated speech.

Table 4: how different consistency accuracy within neural audio codec affect the WER of speech generated by VALL-E.

Question 5:

In Figure 4, SpeechTokenizer performs worse than EnCodec and DAC in terms of consistency accuracy, which seems counterintuitive. From my experience, when training VALL-E, SpeechTokenizer typically yields better WER results compared to EnCodec and DAC.

Reponse 5: While our research emphasizes the importance of consistency, we acknowledge that other codec characteristics also play crucial roles in the WER of speech generated by neural codec language models. A notable example is SpeechTokenizer, where the first layer of audio tokens primarily encodes semantic information with minimal acoustic content. This semantic-focused encoding enable VALL-E to effectively model the first layer tokens, achieving better performance of generated speech, despite lower consistency accuracy.

We sincerely thank you for your thorough examination of our paper and your valuable feedback.

[About Weakness 1]

Response 1: Our research proposes a method to enhance neural audio codecs, theoretically applicable to various commonly used neural audio codecs. By reducing inconsistencies in the encoding process, we can improve the performance of downstream neural codec language models, such as VALL-E[1] and VALL-E 2[2], rather than being limited to VALL-E alone.

(1) The improved versions of VALL-E[1], such as VALL-E 2[2] and VALL-E R[3], primarily focus on model architecture enhancements. In contrast, our approach targets the improvement of neural audio codecs and is a more general solution that can be broadly applied to various neural codec language models. Our method is orthogonal to the improvements made in VALL-E, meaning that when used together with the improved versions of VALL-E[1, 2, 3], it can bring further advancements in speech generation.

(2) Compared to VALL-E's strategy of modeling based on discrete audio tokens, MELLE[4] employs an auto-regressive approach to model continuous representations and generate Mel-spectrograms. MELLE[4] is fundamentally different from our methodology, as we focus on addressing the inconsistency issues within neural audio codecs.

[1] Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

[2] VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

[3] VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment

[4] Autoregressive Speech Synthesis without Vector Quantization

[About Weakness 2]

Reponse 2: In our study, we compared various TTS systems. And the original papers presenting these TTS systems employed different testing configurations [1, 2, 3]. To provide a fair and unbiased comparison, we conducted tests on a subset extracted from LibriTTS, utilizing the Whisper v3 as the ASR model to evaluate WER like XTTS[3], and employing the 3D-speaker toolkit to assess SIM. Additionally, we have followed the testing configuration outlined in VoiceCraft[2] for further experimentation to validate our model's effectiveness. The results of these experiments are detailed in Table 1 in this section.

Furthermore, to ensure the reproducibility of our results, we have made our code and training scripts publicly available here https://github.com/ConsistencyInNeuralCodec/ConsistencyInNeuralCodec.

Table 1: The speech generation results on LibriTTS test set. Bold means the best result, and underline means the second-best result. Ours and Ours w/o consistency constraint denote the same neural audio codecs with and without consistency constraint. The subscripts of the neural codec language models (e.g., 330M, 44Kh) denote the model size and data scale.

[1] Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

[2] VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

[3] XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model

[4] Amphion: An Open-Source Audio, Music, and Speech Generation Toolkit

Thank you so much for your valuable comments and precious time.

[About Question 1]

Response: We are conducting this experiment, and we will report experimental results before the deadline.

[About Question 2]

Response: Initially, because we compare various TTS systems with different testing configurations, we conduct tests on a subset extracted from LibriTTS, utilizing Whisper v3 as the ASR model to evaluate WER[1], and employing the 3D-speaker toolkit to assess SIM[2]. Reviewers mention that our evaluation metrics are not aligned with other papers when presenting our results, which would be better validated under different testing settings as a supplement. Thus, we select a representative work, VoiceCraft[3], and follow its testing configuration to evaluate results on different subsets. Due to the use of different testing settings, including new ASR models (Whisper v2 [3]) and new speaker similarity detection tools(WavLM[4]), it would obviously lead to different results, but our conclusions remain consistent, and the distribution of results for each baseline is also consistent with previous testing settings.

[1] XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model

[2] 3D-Speaker-Toolkit: An Open Source Toolkit for Multi-modal Speaker Verification and Diarization

[3] VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

[4] WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

[About Question 3]

Response: In the Section to All Reviewers, there are two VALL-E reproductions from different works: SpeechTokenizer+VALL-E reproduced by Amphion[1], and Ours+VALL-E, which is our reproduction. Our reproduced VALL-E demonstrated better speech generation metrics compared to the original VALL-E, not inferior results. Therefore, when assessing the consistency constraints, our experiments conduct on our own VALL-E reproduction provide stronger evidence. And we still report the result of SpeechTokenizer+VALL-E in hope to provide reviewers and readers with a reference.

Although we have provided a detailed explanation of the origins of these two VALL-E models in the last version of the paper's PDF, maybe you are still confused by this comparison between two reproduced VALL-E models. Considering that your suggestion makes sense, as comparing the open-source VALL-E with other models could cause unnecessary misunderstandings, and since the open-source VALL-E does not affect our experimental conclusions, we have removed the open-source VALL-E from the TTS systems to be compared in the new version of the PDF.

[1] Amphion: An Open-Source Audio, Music, and Speech Generation Toolkit

[About Question 1]

Response: In order to reply your concerns, we conducted additional experiments.

We train another neural codec language model, UniAudio, with inconsistent or consistent audio tokens. As shown in Table 1 in this section, neural codec language models with consistency constraint demonstrate better performance than those without consistency constraint, which indicates that the consistency constraint is a general method, and is benificial for neural codec language models, including VALL-E, UniAudio, and others, to achieve improvement in intelligibility (WER) and similarity (SIM).

We hope our experimental results will be helpful for your assessment.

Table 1: The speech generation results on LibriTTS test set.

[About Question 4]

Reponse: Besides ours, we conducted additional experiments on EnCodec models. We obtain a series of EnCodec models with different consistency accuracy by varying different weights of consistency constraint loss, which we then utilized as audio tokenizers to train VALL-E and observe how consistency impacts WER. Table 1 in this section illustrates the effect of various consistency accuracy within EnCodec model on the WER of VALL-E, indicating that there is a positive correlation between consistency accuracy and WER improvement. Specifically, as the consistency accuracy increases, the WER correspondingly decreases.

This reflects the inconsistency phenomenon present in EnCodec models, which can also be mitigated through our proposed method and leads to significant improvements in downstream VALL-E models. This demonstrates that our method is available to applied in different codecs.

Table 1: How different consistency accuracy within EnCodec model affect the WER of speech generated by VALL-E.

Dear Reviewer iYgA,

We sincerely thank you for your valuable suggestions on our work. Based on your insightful feedback, we have provided additional clarifications, and hope our responses have fully addressed your concerns.

We hope you had a wonderful Thanksgiving holiday during the past few days. As the rebuttal deadline approaches, we kindly look forward to receiving your further feedback. Once again, we deeply appreciate the time and effort you have dedicated to reviewing our paper.

Best regards,

We would like to express our gratitude again for your time and effort in reviewing our paper. If you find that our revisions meet your expectations or address some of your concerns, could you please reassess and update your rating?

Thank you for your response.

I observed that the TTS results posted here do not align with those reported in the submission paper, and the current results seem to be worse. As the authors claimed they did not use re-ranking, could you please explain the gap?

For example:

Initially, because we compare various TTS systems with different testing configurations, we conduct tests on a subset extracted from LibriTTS, utilizing Whisper v3 as the ASR model to evaluate WER[1], and employing the 3D-speaker toolkit to assess SIM[2]. Reviewers mention that our evaluation metrics are not aligned with other papers when presenting our results, which would be better validated under different testing settings as a supplement. Thus, we select a representative work, VoiceCraft[3], and follow its testing configuration to evaluate results on different subsets. Due to the use of different testing settings, including new ASR models (Whisper v2 [3]) and new speaker similarity detection tools(WavLM[4]), it would obviously lead to different results, but our conclusions remain consistent, and the distribution of results for each baseline is also consistent with previous testing settings.

[1] XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model

[2] 3D-Speaker-Toolkit: An Open Source Toolkit for Multi-modal Speaker Verification and Diarization

[3] VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

[4] WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing