A Variational Framework for Improving Naturalness in Generative Spoken Language Models

We sincerely thank the reviewer for recognizing both the robustness of our experimental design and the novelty of our contribution. Below, we address each concern raised:

Emotion and Speaker Recognition

My main concern is how this method is useful for more practical downstream tasks such as ASR, emotion or speaker recognition, to reflect that capturing the additional (mainly paralinguistic) information is useful. I strongly encourage the authors to conduct at least 2 of the above practical tasks using the variational speech LM and compare it to token-based speech LM to see any potential benefits of using variational methods.

We appreciate this valuable feedback. We would like to direct the reviewer to Appendix H, where we present our comprehensive evaluation results on speech emotion recognition and speaker recognition tasks. In Tables 8 and 9, our results suggest that the variational features encode speaking styles and prosodic patterns there are useful for both tasks. We kindly direct the reviewer to Appendix H for more detailed experimental setup, results and analysis. Additionally, we would like to note that our main objective remains to improve naturalness of speech synthesis, and the side experiments provide additional evidence that our method is capable of achieving that.

Additional References

G. Sun et al. "Generating diverse and natural text-to-speech samples using a quantized fine-grained vae and autoregressive prosody prior", In Proc. ICASSP. 2020. This should be discussed in section 3.1. This paper is the first to apply a trainable auto-regressive prior in speech synthesis.

Thank you for this important suggestion. We will incorporate this reference in Section 3.1 and properly acknowledge its contribution.

Figure Presentation

Actually, Fig. 1 can be improved by unifying the font.

We appreciate this attention and will unify the font in Fig. 1.

Dataset Selection

The experiment was conducted using LibriSpeech and Libri-light, which are datasets with quite small variabilities other than semantic information. I believe the variability remains in the speaker representation space, which is not explicitly reflected in the experimental design.

We thank the reviewer for their valuable feedback. We acknowledge that LibriSpeech and Libri-light are not considered highly expressive. We evaluate on these datasets since they are standard benchmarks in the literature [1, 2, 3]. Importantly, our proposed approach improves synthesis naturalness even on these less expressive datasets. This suggests that our method would likely yield even greater improvements on more expressive datasets, where natural expressive speech synthesis presents additional challenges.

We hope these responses adequately address the reviewer's concerns and help in the evaluation of our work.

References

[1] E. Kharitonov, et al., Text-free prosody-aware generative spoken language modeling, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8666–8681, Dublin, Ireland, May 2022.

[2] Z. Borsos, et al., AudioLM: A language modeling approach to audio generation, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523–2533, 2023.

[3] K. Lakhotia, et al., On Generative Spoken Language Modeling from Raw Audio, Transactions of the Association for Computational Linguistics, 2021.

We sincerely thank the reviewer for the thorough review and acknowledging the strengths of our paper, including the balance between innovation and practicability, the automatic learning of continuous prosodic features, and our rigorous experimental design.

Below, we address the feedback from the reviewer:

End-to-End Design Novelty

This "end-to-end" design idea does not seem to be new in the field of speech synthesis.

We thank the reviewer for this observation. End-to-end is indeed the current trend for building speech synthesis models. Our specific contribution lies in making the prosodic features end-to-end learnable with the speech language model through the integration with VAE. This approach enables automatic learning of continuous prosodic features without manual feature engineering, which we believe represents a meaningful advancement in the field.

Hyperparameter Sensitivity

The performance of the model is highly dependent on the tuning of β and γ, which may limit the robustness of the method under different data distributions.

This is an insightful observation. We agree that hyperparameter sensitivity is an important consideration for most work in this field. In Section 8: Limitations and Future Work, we have acknowledged this challenge and identified exploring automated methods for hyperparameter tuning as an important direction for future research. The current work serves as a first and crucial step in demonstrating that the proposed method effectively improves the prosodic naturalness of speech language models.

Cross-Lingual Adaptability

The experiment only verifies the English dataset, while the prosodic patterns of different languages are quite different, and it is unclear whether the model needs to be retrained or adjusted hyperparameters.

This is an excellent point that highlights an important direction for future research. We hypothesize that our approach of learning continuous prosodic features automatically rather than using hand-designed features offers inherent advantages for cross-lingual adaptation. Unlike rule-based systems that require language-specific expertise, our VAE framework can potentially discover and model language-specific prosodic patterns without human supervision.

We will add this discussion to the limitations section and highlight it as a direction for future work. Our approach, which learns continuous representations rather than relying on predefined features, may be well-suited to adapt to the diverse prosodic patterns across languages, though this remains to be verified through additional research.

We thank the reviewer for the valuable suggestions, which will help direct our ongoing research efforts.

We sincerely thank the reviewer for their constructive feedback. We address your concerns comprehensively below. We present the Tables of new experiment results here.

Comparison with Additional Baselines

Some work such as SpeechTokenizer which uses RVQ-based methods and distills semantic tokens in the first layer can also address the issues: present the paralinguistic features while modeling the semantic features. The author should discuss them.

The author should compare it with SpeechTokenizer; 2) the generative model is also not well designed. I suggest using AR to predict the first semantic tokens and NAR to predict the residual acoustic features which use semantic tokens as conditions.

We appreciate your points about comparing our approach with additional baselines. We acknowledge that our comparison could be more comprehensive and have implemented a SpeechTokenizer-based approach as suggested.

SpeechTokenizer Baseline: We used the official SpeechTokenizer checkpoint for the encoder and decoder components (semantic and acoustic tokens ↔ speech). But we trained our own NAR decoder for predicting acoustic tokens from semantic tokens, as the original implementation requires text input (it was originally designed for TTS). We also re-trained the autoregressive model for semantic token prediction due to token set differences (1024 vs 200 tokens).

As shown in Table 2, our approach still achieves superior naturalness and meaningfulness scores compared to the new established SpeechTokenizer baseline. Perceptual listening revealed that the SpeechTokenizer approach indeed improves upon Token-LM in prosody patterns. However, it introduces more audio quality artifacts compared to Token-LM which uses a diffusion-based decoder to convert semantic tokens to speech. These artifacts offset the benefits of better prosody naturalness in the human evaluation scores. Note that we re-evaluate all methods in the human evaluation for fair comparison.

We will revise Sections 5 and 6 to incorporate these new results.

Roles of VAE and Comparison with Existing Works

Why such VAE is necessary? What benefits can it bring compared with existing works?

Our approach offers distinct advantages over existing methods:

1. Compared to SpeechTokenizer and Token-LM: In these baselines, the AR model accesses only semantic tokens. Our AR model additionally accesses variational features encoding prosody, better utilizing the stronger AR model for improved naturalness.
2. Compared to Token-LM+Acoustic: The acoustic (RVQ) tokens from this baseline is extracted without supervision from the AR model, while our approach jointly optimizes variational features for both reconstruction and AR prediction, where the encoder receives training signal from the AR model.
3. Compared to Token-LM+Pitch: Our approach doesn't require hand-engineered features, as it learns prosodic information in an unsupervised fashion.

The VAE framework enables joint learning of variational features optimized for both reconstruction and AR prediction, yielding superior naturalness and meaningfulness of the syntheses. We'll revise Section 6 to emphasize these advantages.

Additional Metrics

Why use CER instead of WER?

We originally chose CER following prior work [1], which evaluates pronunciation errors more accurately. As requested, we've added WER measurements in Table 1. The WER results follow the same trend as CER.

The speaker similarity metric is not reported.

In Table 1, we add speaker similarity metrics, which is calculated using the cosine similarity of the speaker embeddings extracted from a pre-trained speaker verification model. Our method achieves a speaker similarity score slightly lower compared to the baseline approaches (0.42 v.s. 0.45). However, as the reviewer can verify from our audio samples, the perceptual difference in speaker identity is minimal, while naturalness and intelligibility improvements are noticeable. This represents a reasonable trade-off for most speech applications.

We believe these additions and clarifications address the reviewer's concerns and strengthen our paper.

References

[1] K. Lakhotia, et al., On Generative Spoken Language Modeling from Raw Audio, Transactions of the Association for Computational Linguistics, 2021.

We appreciate both the recognition of our formulation's correctness and the critical feedback from the reviewer. We address the feedback below.

Computational Complexity

The model relies on a diffusion decoder, which increases computational complexity compared to predicting discrete acoustic codecs, making it unsuitable for real-time speech generation applications.

We would like to clarify several points:

• Decoder flexibility: Our variational approach isn't tied to a specific decoder. We chose diffusion for easier training, but our method works with various decoding strategies, including discrete token prediction. There is no constraint on the decoder $\theta$ used to parameterize $p_\theta(\mathbf{X}\mid\mathbf{Z})$. To validate this, we conducted additional experiments replacing the diffusion decoder with a token decoder which converts semantic tokens to SpeechTokenizer tokens [1], then used a pre-trained SpeechTokenizer decoder to generate speech. Table 1 shows that this variant (Proposed + Token Decoder) achieves similar or better performance compared to the diffusion approach.
• Fair comparison: We used the same diffusion decoder architecture across all comparing methods. We ensure a fair comparison where only the modeling approach varies, isolating the impact of our variational method.
• Research focus: Our primary contribution is on improving prosodic naturalness with the variational approach, not on optimizing for real-time applications. The improvement in naturalness MOS validates the effectiveness of our method.
• Reasonable RTF: On a single NVIDIA L40S GPU with batch size 1, our current implementation achieves an RTF of 0.68 (<1). While this is not our focus, it shows that this approach remains practical. Importantly, the diffusion decoder only operates once after the AR model completes its generation, so the additional latency introduced is not as significant as it might initially appear.

We will make adjustments to Sections 3 and 4 to emphasize these points and avoid confusion.

Can the mel-spectrogram be decoded only after all continuous features have been generated?

Yes. Our current implementation with diffusion decoder and HiFi-GAN vocoder is not streamable. However, as mentioned above, our framework is compatible with any type of decoder, including streamable ones.

Use of Re-implemented Models

The baselines used are re-implemented versions of existing models rather than direct comparisons with the original models.

We re-implemented the baseline models following standard scientific methodology to ensure a controlled and fair comparison. This decision was made for several critical reasons:

• Architectural consistency: We needed to maintain consistent architecture and model size across all methods. Our re-implementations use identical decoder and autoregressive model architectures, differing only in the specific modeling approach being evaluated. This isolates the impact of our variational method, which is the primary contribution of our work.
• Data consistency: We ensured all models were trained on exactly the same data. Off-the-shelf implementations typically differ in training datasets, preprocessing pipelines, and model sizes, introducing confounding variables that would make it impossible to attribute performance differences specifically to our methodological innovation.

Comparison with More Recent Approaches

If the model were compared to more recent approaches using more compressed acoustic tokens such as WavTokenizer, it would likely provide better insight into prosody than an LLM trained only on SSL-based semantic tokens.

We include the Token-LM + Acoustic baseline, which employs acoustic tokens from residual vector quantization (RVQ), capturing information beyond what semantic tokens. To compare with recent approaches on acoustic tokenization, we have included an additional comparison with SpeechTokenizer in Table 1 (details in our response to Reviewer XtBM). This expanded analysis further validates the advantages of our variational method across different tokenization techniques.

Token-LM Baseline with Limited Clusters

Token-LM models very small discrete tokens with only 200 clusters, the results are not sufficiently comprehensive.

We specifically chose k=200 for our HuBERT baseline after conducting a sweep across different values (k=50, 200, 1000), finding that k=200 performed the best for language modeling, as detailed in Appendix F. Additionally, the new added SpeechTokenizer baseline has 1024 clusters, which our model still outperforms.

Thank you for your valuable feedback which has helped us articulate our contributions more clearly.

References

[1] X. Zhang et al., SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models, ICLR, 2024