Generative Pre-Trained Speech Language Model with Efficient Hierarchical Transformer

Q4: The memory issue in long sequence generation.

The memory issue in long sequence generation is one of the motivations for GPST as well. The hierarchical structure effectively reduces the sequence length handled by the global transformer, mitigating the memory pressure with lengthy sequences. By allowing the global transformer to concentrate on high-level or coarse features, GPST delegates fine-grained details to local transformers. In our experiments, GPST notably did not encounter memory issues. The low WER indicates the model's competence in handling lengthy sequences without experiencing memory-related challenges.

[1] Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336–1354, 2021.

[2] Eugene Kharitonov, Damien Vincent, Zala'n Borsos, Raphae'l Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. arXiv preprint arXiv:2302.03540, 2023.

[3] Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, et al. Direct speech-to-speech translation with discrete units. arXiv preprint arXiv:2107.05604, 2021.

[4] Lo'ıc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. Seamlessm4t- massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596, 2023.

[5] Gu, J., Bradbury, J., Xiong, C., Li, V.O. and Socher, R., 2017. Non-autoregressive neural machine translation. ICLR 2018

[6] M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer, “Maskpredict: Parallel decoding of conditional masked language models,” in EMNLP-IJCNLP, 2019, pp. 6112–6121.

[7] J. Gu, C. Wang, and J. Zhao, “Levenshtein transformer,” NeurIPS, vol. 32, pp. 11 181–11 191, 2019

[8] Gu, J. and Kong, X. Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade. ACL 2021.

We sincerely appreciate the valuable comments and constructive suggestions to help improve the paper. We give the response below.

Q1: The evaluation with subjective metrics and the demo page. DNSMOS is not enough to measure the speech quality.

We do want to conduct the subjective comparison such as MOS for all the models, but sadly AudioLM and VALL-E are not open-sourced, we cannot perform a fair and rigorous test of MOS. We have tried our best to conduct fair and rigorous experiments to compare the baselines with our model, which shares the same protocol for all the models. DNSMOS metric is widely accepted in SPEAR-TTS, VALL-E, and SoundStorm, we believe it can reflect the quality of the audio to some extent. We can provide some cases on the anonymous demo page (https://gpsticlr.github.io/GPST/demo) and the code will be available too.

Q2: The question about efficiency and storytelling.

We must clarify that the main contribution of our work is a unified pre-trained speech language model that supports Hi-Res speech generation and spoken multi-lingual. And the efficient hierarchical transformer is the approach to achieve it. What the reviewer states in Q4 also agrees with it. However, there appear to be conflicting assertions in Q2 and Q3, suggesting that the primary emphasis of our work lies in efficiency, which doesn't accurately reflect our main focus. We provide our generation speed test with the fairseq-generate tool. The inference is run on the NVIDIA 3090 GPU in the semantic to acoustic mode with a batch size of 64. However, we cannot provide the speed test results for other baselines since both AudioLM and VALL-E are not open-source.

Q3: The potential ability of auto-regressive hierarchical transformer and the shortcomings of the NAR model SoundStorm.

We emphasize that efficiency is NOT the primary focus of our model GPST, instead, the main contribution is the unified speech LM framework. The design of GPST considers the potential possibility of integrating the speech generation ability into auto-regressive LLMs like LLaMA, which is not possible for NAR models. For the model architecture design, we do not blindly pursue the number of parameters for GPST, instead, we carefully configure the experimental setting of a large global model and a small local model, simulating scenarios where a small model integrates with LLMs.

We provide the reason why SoundStorm is not appropriate for comparison. It utilizes duplicate semantic tokens as a condition to generate acoustic tokens, leading to a leakage of duration information. Moreover, duplicate semantic tokens would cause failures in the generation of semantic tokens [1], which would limit the application in speech generation [2] and resynthesis for speech translation system [3], while our model GPST can easily assimilate into the SOTA multi-lingual speech translation system SeamlessM4T [4]. Although NAR approach such as SoundStorm can accelerate the generation speed, it exhibit performance limitations when compared to AR models [5,6,7,8]. According to their experiment results, the SPK result for SoundStorm is $0.57$ while GPST achieves a better result of $0.605$.

We sincerely appreciate the valuable comments and constructive suggestions to help improve the paper. We give the response below.

Q1: The real-time efficiency.

We provide our generation speed test with the fairseq-generate tool. The inference is run on the NVIDIA 3090 GPU in the semantic to acoustic mode with a batch size 64.

Q2: The reason for choosing WER as the evaluation metric in the semantic-to-acoustic task in Table 1.

WER aims to evaluate the speech synthesis robustness. Neural TTS systems usually suffer from the robustness issue, which sometimes has deletion, insertion, and replacement errors due to wrong attention alignments. We perform ASR on the generated audio and calculate the word error rate (WER) with respect to the original text transcriptions. We have other metrics SPK and DNSMOS to evaluate the speech quality. These metrics focus on different perspectives on the speech generation models. It is hard for us to conduct human evaluation benchmarks since the baselines AudioLM and VALL-E are not open-sourced, we cannot perform a fair and rigorous MOS test for them.

For the semantic-to-acoustic condition in Table 1, all the models except GPST-TTS are conditioned on the golden semantic tokens to generate acoustic tokens/waveforms, followed by an ASR model to calculate WER with respect to the text transcriptions. Only our GPST-TTS model can condition the text inputs to generate speeches. The GPST-TTS model first infers semantic tokens with texts as input, then generates acoustic tokens with the semantic tokens as inputs.

Q3: The meaning of GroundTruth in Table 1 row 1.

The results are directly calculated on the ground truth speech in the LibriSpeech test-clean dataset as the upper bound.

Dear Reviewer BzgH,

Thank you again for your insightful feedback on our submission. These valuable suggestions better strengthen the quality of our work. The deadline for the discussion stage is approaching, and we are looking forward to your further comments. If we have properly addressed your concerns, we would deeply appreciate it if you could kindly re-evaluate our paper. If you have further concerns. please let us know and we remain open and would be more than happy to actively discuss them with you.

Best,

Authors

Dear Reviewer BzgH,

Thank you for your review and for acknowledging the efforts made to address the concerns raised in your evaluation. We appreciate your time and insightful feedback.

While we are pleased that the responses have clarified the points you highlighted, we also note that the score remains unchanged. We understand that you have reservations about altering the score at this stage, and we respect your decision. However, we would like to kindly request your reconsideration of the assigned score.

Should our paper be accepted, we are fully committed to implementing any further clarifications or modifications necessary to accommodate the suggestions provided, as you suggested.

We value your expertise and insights and would greatly appreciate a reconsideration of the score in light of the addressed concerns. Your consideration in this matter is highly valued and will contribute significantly to the final assessment of our submission.

Thank you for your time and consideration.

Warm regards,

We sincerely appreciate the valuable comments and constructive suggestions to help improve the paper. We give the response below.

Q1: The evaluation of speaker identity transfer for GPST-TTS.

We must clarify that our primary focus lies in developing a unified speech language model, while GPST-TTS is only a toy model for inspiration trained on LibriSpeech 960h, which is not comparable with VALL-E trained on Librilight 60k and SPEAR-TTS employing pre-training and back-translation techniques. However, we are still interested in the speaker information leak problem and give the extra experiment results here. Unlike VALL-E that directly trains with text transcriptions. If AudioLM and GPST use the semantic tokens inferred by a text-to-semantic model, it will cause a distribution mismatch with the semantic prompt provided in the speaker identity transfer stage. Because, as you state, semantic tokens may contain subtle speaker-related information such as prosody. Therefore, we propose a more rigorous inference mode for the text-to-semantic task like in-context learning in speaker identity transfer mode. We transcribe the speech prompt with an ASR model and prepend it to the text inputs in the text-to-semantic models, which is similar to the speaker identity transfer. We can see that GPST can get a significant improvement with text prompts from $0.547$ to $0.568$. We believe the performance can get further improvement if trained on Librilight 60k and using the pre-training and back-translation method proposed in SPEAR-TTS.

In addition, we find that the semantic tokens can not control the speaker identity in the synthesized speeches. For example, a semantic sequence said by a man would still synthesize the speech in a woman voice. We provide some cases in the anonymous demo page (https://gpsticlr.github.io/GPST/demo).

Q2: Experiments on the non-hierarchical (flattened) vanilla LM.

It's a natural thought to train a vanilla decoder-only transformer for speech generation. However, it is impractical since the length of the flattened acoustic tokens is too large (6000 tokens for 10s speech) to feed into the vanilla LM and will cause a cuda OOM error. This limitation stands as one of the key motivations driving the development of GPST, aiming to address and overcome these computational constraints in speech generation models.

Q3: Why are DNSMOS not presented in Semantic to Acoustic and Acoustic Continuation tasks?

Since both of AudioLM and VALL-E are not open-sourced, we follow AudioLM's instructions and compute DNSMOS with the examples provided on VALL-E’s demo page for fairness, which is described in Section 4.1.4. Unfortunately, VALL-E did not conduct the Semantic to Acoustic task and AudioLM did not conduct the Acoustic Continuation task. Therefore, the Speaker Identity Transfer task is the only one we can test to compare all the models in fair.

Q4: GPST-Hi-Res has better audio quality but degrades in WER and SIM.

The better audio quality means richer harmonic energy in the high-frequency regions, as shown in Figure 3, which focuses on different perspectives of the speech from WER and SIM. WER focuses on the robustness of speech models, which sometimes has deletion, insertion, and replacement errors due to wrong attention alignments. SIM focuses on speaker similarity to evaluate the voice cloning ability. Better speech quality does not guarantee better WER and SIM, which is the reason we choose three different metrics to evaluate the generated speech. As we mentioned in Section 4.2, the Hi-Res speech generation, with much more quantizers (from 8 quantizers to 16 quantizers), is still a tough task for speech-language models and requires a larger model with more parameters and longer training time. In the experiments, we simply use the same setting as non-Hi-Res models, which is not the best setting for Hi-Res model. Additionally, our model did not reach convergence before cessation of training. We believe the results would get improvement with more training steps.

Q5: Does each residual codebook have its own embedding?

Yes, each codebook has its own embedding and the vocab size for $E_a$ is 8192.

Q6: Is $E_a$ shared between local and global transformers?

No. Since the embedding sizes of the local transformer (512) and global transformer (1024) are different, we do not share the embedding table.

Q7: The clarification of Local-drop.

The Local-drop method is proposed to save the GPU memory, without which the GPST-Hi-Res model would cause a cuda OOM error. The local transformer's input size is $(b \times T_2, D)$, so we can simply drop some samples in the batch dimension $b \times T_2$ since they are i.i.d. for the local transformer. The Local-drop method is only effective for hierarchical transformer because the unfolding operation in $T_2$ breaks the dependency in tokens to i.i.d., and is safe to drop them to save the memory. However, if we drop some tokens in the non-hierarchical (flattened) vanilla LM, we can not save memory since the sequence length and batch size are not changed.

Q8: The settings of $N_g, N_l$.

We ablate different settings of $N_g$ and $N_l$ in Table 3 with the number of total parameters fixed. In Table 1, we use the setting $N_g=9, N_l=12$ for evaluation. We will clarify it in the experimental setting.

Dear Reviewer 7zWQ,

Thank you again for your insightful feedback on our submission. These valuable suggestions better strengthen the quality of our work. The deadline for the discussion stage is approaching, and we are looking forward to your further comments. If we have properly addressed your concerns, we would deeply appreciate it if you could kindly re-evaluate our paper. If you have further concerns. please let us know and we remain open and would be more than happy to actively discuss them with you.

Best,

Authors

We sincerely appreciate the valuable comments and constructive suggestions to help improve the paper. We give the response below.

Q1: The model architecture was introduced before in the NLP community and the novelty is limited compared to previous speech models.

It is true that the hierarchical transformer is not a new architecture in the NLP community. However, our focus lies specifically on the speech generation task, where the utilization of the hierarchical transformer remains unprecedented. Our work pioneers the application of this architecture for speech generation, marking a significant departure from prior endeavors. Notably, our approach involves a novel design of the architecture, encompassing the integration of semantic tokens, acoustic tokens, in-context learning, multilingual aspects, among others, tailored specifically for the speech language model. This task distinctly differs from conventional text models, thereby underscoring its novelty.

While acknowledging the foundational groundwork laid by AudioLM in the realm of speech generation, it is imperative to note that our model, GPST, draws inspiration from AudioLM while innovatively leveraging the hierarchical transformer to address inherent challenges encountered in AudioLM and VALL-E. These preceding models face significant limitations when generating lengthy acoustic sequences, a hurdle meticulously addressed in Section 3.1. Our model not only tackles these challenges but also outperforms them, as substantiated by our empirical findings.

Q2: Unsupported claims in the paper.

We will carefully clarify the unsupported claims as follows:

(1). ``...which limits the performance of fine acoustic token generation..''

We state it in the introduction of VALL-E. As described in the paper, VALL-E conducts non-auto-regressive generation of the acoustic tokens from subsequent quantizers. As investigated in non-auto-regressive generation models [1,2,3,4], the independence assumption in the output space ignores the real dependency between target tokens, and the maximum-likelihood training forces to cover all possible modes, which makes NAR systems perform worse than the AR baselines. VALL-E leverages the vanilla non-auto-regressive generation model for the subsequent acoustic tokens, typically resulting in suboptimal performance compared to auto-regressive generation models.

(2). ``These multi-stage generative models induce significant error propagation issues, which can negatively impact the overall performance.''

It is established in the literature that cascaded multi-stage systems often exhibit inferior performance in comparison to end-to-end systems due to inherent error propagation issues [5,6]. This phenomenon stems from the discrepancy between the training and inference procedures within cascaded systems. During training, all modules receive training signals derived from ground truth labels. However, during inference, these modules are fed outputs from preceding modules, resulting in error outputs that are out of distribution concerning the training data. Consequently, as elaborated in Section 3.1 and illustrated in Figure 1, both AudioLM and VALL-E adopt cascaded architectures, inherently predisposing them to the issue of error propagation. This discrepancy contributes to the amplification of errors throughout the cascaded system's feed-forward flow, thereby adversely affecting performance.

(3). ``Additionally, obstructing the information flow among hierarchical quantizers would degrade the model’s performance, especially in Hi-Res speech generation that requires more residual quantizers.''

The acoustic tokens are obtained with residual quantizers, wherein the subsequent quantizers rely heavily on the results from the previous quantizers. In the Hi-Res speech generation setting, the number of residual quantizers (n=16) is much larger than the normal setting (n=8), which poses more challenges for the model. With the two issues described above, the information flow in VALL-E is obstructed by non-auto-regressive model and cascaded systems, resulting in the degradation of the model's performance.

We will add the support evidence and related citations to the revised version, thanks for helping improve our work.

Q3: Regarding the efficiency analysis.

The efficiency analysis comes from two different perspectives, one is computation complexity and the other is FLOPS. In the computation complexity analysis, if we omit $D$, which implies flattening the acoustic sequence, we will get a cuda OOM error on GPU. Therefore, we keep it in our analysis and ensure a comprehensive evaluation of the computational complexity.

Q4: The details of the WER computation in GSLM.

GSLM uses HuBERT code (semantic tokens) as inputs like AudioLM and reconstructs the waveform with the Tacotron2 model and the WaveGlow vocoder. We run their open-sourced code using the released model and evaluate the results. GSLM can not generate speeches conditioned on texts, and the evaluation is done directly based on golden semantic tokens. In the Semantic to Acoustic part of Table 1, only our GPST-TTS model can generate speeches conditioned on texts.

Q5: The comparison with delay pattern models.

The delay pattern models in [7,8] are tailored for semantic tokens like GSLM and music generation. Designing a dedicated model for speech LM incorporating acoustic tokens in a delay pattern extends beyond the scope of our work. However, we provide an analysis here to illustrate why our model is better than the delay pattern approach. The delay pattern model predicts multiple tokens in parallel with a simple linear classification head, while GPST predicts them with a local transformer in an auto-regressive manner, which means GPST can better fit the distribution. The auto-regressive mechanism and the utilization of a local transformer within GPST afford it a superior capacity to model intricate dependencies among tokens, thereby establishing its theoretical superiority over the delay pattern approach.

Q6: Regarding the length of the paper.

As ICLR 2024 Author Guide states (https://iclr.cc/Conferences/2024/AuthorGuide), the optional ethic statement and reproducibility statement will not count toward the page limit, but should not be more than 1 page. So we believe we didn't break the submission guidelines.

[1] Gu, J., Bradbury, J., Xiong, C., Li, V.O. and Socher, R., 2017. Non-autoregressive neural machine translation. ICLR 2018

[2] M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer, “Maskpredict: Parallel decoding of conditional masked language models,” in EMNLP-IJCNLP, 2019, pp. 6112–6121.

[3] J. Gu, C. Wang, and J. Zhao, “Levenshtein transformer,” NeurIPS, vol. 32, pp. 11 181–11 191, 2019

[4] Gu, J. and Kong, X. Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade. ACL 2021.

[5] Inaguma, H., Popuri, S., Kulikov, I., Chen, P.J., Wang, C., Chung, Y.A., Tang, Y., Lee, A., Watanabe, S. and Pino, J., 2022. UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units. ACL 2022.

[6] Barrault, Loïc, et al. "SeamlessM4T-Massively Multilingual & Multimodal Machine Translation." arXiv preprint arXiv:2308.11596 (2023).

[7] Kharitonov, Eugene, et al. "Text-free prosody-aware generative spoken language modeling." arXiv preprint arXiv:2109.03264 (2021).

[8] Copet, Jade, et al. "Simple and Controllable Music Generation." arXiv preprint arXiv:2306.05284 (2023).