A Variational Approach for Generative Speech Language Modeling

My main concern is how this method is useful for more practical downstream tasks such as ASR, emotion or speaker recognition, to reflect that capturing the additional (mainly paralinguistic) information is useful. I strongly encourage the authors to conduct at least 2 of the above practical tasks using the variational speech LM and compare it to token-based speech LM to see if there are any potential benefits of using variational methods.

We thank the reviewer for their valuable feedback. In our general response, we include two experiments—emotion recognition and speaker identification—to demonstrate the information captured by our learned features. The experiments requested by the reviewer strengthen our claims by showing that the features our model learns encode paralinguistic information.

Experimental is conducted using LibriSpeech and Libri-light, which are datasets with quite small variabilities other than semantic information. I believe the variability remains in the speaker representation space, which is not explicitly reflected in the experimental design.

We thank the reviewer for their comment regarding the evaluation dataset. We agree that the LibriSpeech and Libri-light datasets are not the most expressive in terms of emotion. However, we chose these datasets because they are widely used in the speech continuation literature [1,2,3,4], and prior works [1] have indicated that achieving natural synthesis for speech continuation on these datasets is already a challenging task. Our results demonstrate that our approach enhances the naturalness of the generated continuations for these datasets. Given this, we believe that if our method can improve synthesis on these datasets, it is likely to show even larger improvements on more expressive datasets.

In addition, our new results in the general response above show that training on LibriSpeech and Libri-light yields models whose features are useful for two downstream tasks (emotion recognition and speaker identification) on out-of-domain datasets.

[1] E. Kharitonov, et al., Text-free prosody-aware generative spoken language modeling, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8666–8681, Dublin, Ireland, May 2022.

[2] Z. Borsos, et al., AudioLM: A language modeling approach to audio generation, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523–2533, 2023.

[3] M. Hassid, et al., Textually Pretrained Speech Language Models, ArXiv, abs/2305.13009, 2024.

[4] K. Lakhotia, et al., On Generative Spoken Language Modeling from Raw Audio, Transactions of the Association for Computational Linguistics, 2021.

Why do we need to separate $z^c$ and $z^d$? Are they really independent? I believe knowing $z^c$ will tell you a lot about $z^d$, right? The authors are encouraged to explain this design choice further.

We separate $\mathbf{Z}^c$ and $\mathbf{Z}^d$ to model different aspects of speech. $\mathbf{Z}^d$ models the language content, which is inherently discrete, and we are able to leverage prior works (e.g., Hubert) to use speech tokens that are shown to capture it. On the other hand, we believe that the remaining information in speech is better modeled continuously, and we use a variational approach to learn the variational feature $\mathbf{Z}^c$.

We agree with the reviewer that $\mathbf{Z}^c$ and $\mathbf{Z}^d$ should not be independent; knowing the language content implies its paralinguistic information, and vice versa. However, we would like to clarify that the modeling assumption in our paper is less strict; we assume only conditional independence. The past generations are first passed to the same autoregressive transformer to produce the representation, e.g., we can denote this as $o_t=Transformer_{\phi}(\mathbf{Z}_{1:t-1})$.

Then, we have different heads predicting the next $z^c_t$ and $z^d_t$ conditioned on $o_t$.

Therefore, the assumption of this framework is that the transformer is able to learn $o_t$ from the past generations $\mathbf{Z}_{1:t-1}$ so that the $z^c_t$ and $z^d_t$ are conditionally independent given $o_t$.

Given the modeling power, we believe that it is possible that the transformer can learn to extract shared information ($o_t$) between $z^c_t$ and $z^d_t$ from $\mathbf{Z}_{1:t-1}$ and leave the distinct information to the distinct heads.

We acknowledge that the most rigorous modeling choice would be to first predict $z^d_t$ from the discrete head of the autoregressive transformer and then condition the generation of $z^c_t$ based on the generated $z^d_t$. In this way, we don't need any assumption of independence. While this can be incorporated into the framework, we opted for a less rigorous modeling which requires less engineering effort on both training and inference.

We also would like to note that even with this less rigorous modeling, our proposed approach already outperforms the baselines.

We will modify Section 3.2 accordingly to include this discussion.

We thank the reviewer for their feedback and appreciate their insights. We believe there may be a misunderstanding regarding our proposed approach, and we hope this response helps clarify it.

However, the authors use the intermediate features directly to train ($\phi$) and then expect ($\phi$) to generate ($z_t^{c}$) for computing divergence.

Our method does not use pre-determined intermediate features to train the Encoder $\phi$. Instead, the Encoder $\phi$ directly processes the input speech $\mathbf{X}$ and outputs the distribution of variational posterior for the variational features $\mathbf{Z}^c$.

This does not achieve the traditional effect of KL divergence in regularizing the latent space. Furthermore, it diverges from standard approaches where the prior is parameterized independently from the latent features.

The Encoder $\phi$ is trained to minimize two losses: the reconstruction loss $-E_{q_\phi}[\log p_\theta(\mathbf{X}\mid\mathbf{Z})]$, and the KL divergence loss $E_{\mathbf{X}}[D_{KL}(q_{\phi}(\mathbf{Z}\mid\mathbf{X})||p_\psi(\mathbf{Z}))]$. The KL divergence loss regularizes the variational feature $\mathbf{Z}^c$ to be more predictable by the autoregressive prior $p_\psi(\mathbf{Z})$, as shown in the $\mathcal{L}^c_{kl}$ term in Equation 4 of the paper.

Specifically, minimizing $D_{KL}(q_\phi(z^c_{t}\mid \mathbf{X})||p_\psi(z^c_{t}\mid \mathbf{Z}_{1:t-1}))$

for the encoder $\phi$ encourages the variational posterior $q_\phi(z^c_{t}\mid \mathbf{X})$ to approximate $p_\psi(z^c_{t}\mid \mathbf{Z}_{1:t-1})$, making the variational posterior easier to predict autoregressively for $\psi$.

While this differs from the standard VAE framework, where the variational posterior is regularized toward a standard normal distribution, we believe the KL divergence term in our framework achieves a similar regularizing effect.

Additionally, a similar autoregressive prior modeling approach was seen in S3VAE [1] for a different application, and we will include this reference to strengthen the theoretical foundation of our work.

Have you try train the AR prior post training?

As mentioned above, the encoder is shaped by both the KL divergence loss and the reconstruction loss. Particularly, since the KL divergence loss term involves the continuously-updated autoregressive prior $p_\psi(\mathbf{Z})$, the resulting variational feature $\mathbf{Z}^c$ is inherently influenced by this prior. Therefore, in our framework, it is not feasible to pre-train the encoder-decoder pair independently of the autoregressive prior and then train the autoregressive prior separately. The encoder, decoder, and autoregressive prior are interconnected and optimized together.

[1] Y. Zhu, et al., S3VAE: Self-supervised sequential vae for representation disentanglement and data generation. In IEEE conference on computer vision and pattern recognition, pages 6538–6547, 2020.

Conditioning on semantic information to learn para-linguistic features is not new and has already been investigated in works like [1] (SpeechTokenizer).

First, we would like to clarify to the reviewer that the primary goal of this paper is not to extract para-linguistic features. Rather, our goal is to improve speech continuation naturalness by leveraging VAE to enable better model the para-linguistic aspects for token-based autoregressive models. Our experimental results support that we achieved our objective.

Our approach differs from SpeechTokenizer in several key aspects. First, while SpeechTokenizer uses a purely discrete tokenization, our model represents speech through both discrete and continuous features. Second, unlike SpeechTokenizer, which does not jointly train its encoder/tokenizer with the autoregressive (AR) model, our method jointly trains the encoder alongside the AR model. Third, SpeechTokenizer adopts a pipeline similar to AudioLM, where acoustic tokens serve as prediction targets for a non-autoregressive (NAR) decoder. In contrast, our approach focuses on optimizing the input to the AR model while maintaining a fixed decoder capacity, a distinction further elaborated in Section 6. Finally, SpeechTokenizer addresses text-to-speech (TTS) tasks, whereas our work targets speech continuation, highlighting a fundamental difference in application domains.

Additionally, the audio samples presented by the authors are audibly poor and notably worse than those produced by speech synthesis methods that directly utilize pitch features, as demonstrated in [2].

The work referenced by the reviewer addresses a different task from ours. Specifically, their work evaluates re-synthesis from extracted features of speech (e.g., speech tokens, pitch, and duration), whereas our task focuses on speech continuation—generating future speech based on a given speech prompt. The samples on their demo page (https://speechbot.github.io/resynthesis/) are all re-syntheses (including voice conversion, which is re-synthesis from shuffled features), whereas the samples we provided work on speech continuation with autoregressive modeling. It is expected that speech continuation samples sound perceptually worse than re-synthesis samples, as speech continuation is a significantly more challenging task.

Furthermore, unlike most recent studies, the authors did not provide a web interface for easy access to samples, which makes evaluation cumbersome. If MOS evaluations were already conducted, it would be beneficial for reviewers to have a similar interface for sample evaluation. Why no demo page?

We thought that providing all the samples (100 samples per model) used for human evaluations would offer a comprehensive understanding of the synthesis quality for the reviewers. However, we apologize or any inconvenience caused by requiring the reviewer to manually go through the samples. To make this process easier, we have created a demo page, which can be accessed here: https://anonymous.4open.science/w/demo-12D5/ For compatibility, please use the chrome browser to open the demo page.

Thank you to the authors for their response. To clarify, my criticism is not about using “pre-determined intermediate features”—I made this clear in my original post. My concern lies in using latent features to directly learn the priors, which defeats the purpose of the KL divergence in a VAE. In the original VAE and its variants, the prior is parameterized independently of the latent space. This independence is crucial for the KL divergence to function as intended.

My question remains: how can you regularize the latent space features using a prior when the prior itself is learned based on latent space features?

We sincerely thank the reviewer for the clarification and apologize for any misunderstandings. We hope that this response effectively addresses the concerns raised.

First, we would like to clarify that the concept of a VAE with a parameterized autoregressive prior has been explored in previous works [1, 2, 3] in image, video, and text generation, albeit with slightly different modeling approaches. Our work adopts this concept for speech continuation and integrates it with discrete token-based speech language models to jointly model various speech attributes, thereby enhancing the naturalness of the synthesis. These prior works can offer theoretical grounding for this concept.

Additionally, we would like to highlight that we have derived the ELBO formulation for the VAE with an autoregressive prior in Appendix A.1. This demonstrates that, even with a parameterized prior, maximizing the ELBO continues to maximize the evidence (log-likelihood), which remains the central objective of generative modeling. This also provides theoretical grounding for this approach.

Given the above clarification, we would like to answer the reviewer's question: "how can this framework regularize the latent space features using a prior when the prior itself is learned based on latent space features?"

We agree with the reviewer that the KL divergence in the original VAE regularizes the latent features toward a standard Gaussian. However, in generative modeling, we believe that the primary role of KL divergence is to facilitate accessible sampling. In the original VAE, the KL divergence pushes the variational posterior toward a standard Normal prior, enabling sampling from the Normal prior during inference instead of the variational posterior. The regularization effect arises because the latent distribution that achieves the best reconstruction may not naturally align with a Normal distribution. As a result, the latent distribution adjusts by trading off some reconstruction fidelity to better approximate a Normal distribution.

Similarly, in VAE with an autoregressive prior, the KL divergence serves the same purpose by allowing sampling from a tractable autoregressive prior as it moves closer to the variational posterior. There is also a regularization effect, as the latent distribution that provides the best reconstruction may not be perfectly modeled by the parameterized autoregressive prior due to the limited capacity of the autoregressive model. As a result, the latent distribution also needs to adjust by trading off some reconstruction fidelity to better minimize the KL divergence.

[1] A. Vahdat and J. Kautz, NVAE: A Deep Hierarchical Variational Autoencoder, Advances in Neural Information Processing Systems (NeurIPS), 2020.

[2] Y. Zhu, et al., S3VAE: Self-supervised sequential vae for representation disentanglement and data generation. In IEEE conference on computer vision and pattern recognition, pages 6538–6547, 2020.

[3] X. Fang, et al., Discrete Auto-regressive Variational Attention Models for Text Modeling, 2021 International Joint Conference on Neural Networks (IJCNN), 2021.

As the discussion period concludes on December 2, we warmly invite the reviewer to share any remaining concerns or feedback. Your valuable insights are greatly appreciated and will help us further refine our work. If possible, we would be grateful if the reviewer could consider revisiting your score, taking the points we address during the discussion into account.

We would like to provide a follow-up on our earlier response regarding the VAE with parameterized priors, as we believe this is also related to the reviewer's concern about regularization.

As previously suggested, the key difference from the original VAE lies in whether the prior is parameterized. Specifically, the regularizing KL divergence terms are: $D_{KL}(q_\phi(\mathbf{Z}\mid \mathbf{X}))||p_\psi(\mathbf{Z}))$ v.s. $D_{KL}(q_\phi(\mathbf{Z}\mid \mathbf{X})||p(\mathbf{Z}))$. A notable distinction between these two terms is that the parameterized prior is more susceptible to KL Vanishing/Posterior Collapse. This is because $D_{KL}(q_\phi(\mathbf{Z}\mid \mathbf{X}))||p_\psi(\mathbf{Z}))$ can more easily be minimized to zero. For instance, $p_\psi(\mathbf{Z})$ could become any trivial distribution—not necessarily Gaussian—while $q_\phi(\mathbf{Z}\mid \mathbf{X})$ could learn to ignore $\mathbf{X}$ and approximate that distribution. As a result, similar to posterior collapse in VAE, the decoder will ignore $\mathbf{Z}$ as it does not provide any information about $\mathbf{X}$, leading to sub-optimal reconstruction loss. Therefore, this method is actually more prone to be overly regularized, as the regularization loss here is more likely to be minimized toward zero.

Regarding this challenge, we highlight that, as mentioned in the last paragraph of Section 3.3, we employ a linear warm-up strategy for $\beta$, inspired by previous studies (referenced in our paper), to effectively mitigate the KL Vanishing/posterior collapse problem. With this strategy, we generally do not observe KL vanishing, even up to $\beta=0.08$ (but in this case the reconstruction is distorted). However, without the linear warm-up, KL vanishing could occur as early as $\beta=0.05$.

It would be useful to have some confidence measure for the sWUGGY, sBLIMP and Perplexity values. It is unclear how much the 61.75 $\rightarrow$ 60.48 sWUGGY regression means. Is this statistical noise, or an issue that should be addressed.

Multiple training runs are needed to provide confidence measures for sWUGGY and sBLIMP. While it is computationally infeasible for us to perform multiple runs of training within the rebuttal period. The observed trends in our performance suggest that they are not due to statistical noise. For instance, in Table 2, we observe a trend that having more information (Token-LM $\rightarrow$ Token-LM + Pitch $\rightarrow$ Token-LM + Variational Features) consistently decreases both sWUGGY and sBLIMP. Also, in Table 4, there is a trend that a higher value of $\gamma$ results in a lower sWUGGY score. It is unlikely that the score differences are statistical noises. Otherwise, there will be no trend from the pure randomness.

While the variational augmentation adds fewer than 1% of parameters, it would be useful to know if this meaningfully adds to inference latency.

We thank the reviewer for their feedback. Here, we provide the inference time comparison of different methods:

We report the latency of generating a 10-second continuation out of a 3-second prompt. We run the inferences on a single L40S GPU with a batch size equal to 1. We generate the 10-second continuation with different prompts 50 times and report the average time elapsed in seconds. As shown in the table, our proposed method results in only a 1.3% latency increase.

In section 3.2, it is mentioned that "By using these tokens, the model no longer needs to encode as much phonetic information in $Z^c$, allowing $Z^c$ to focus on other continuous speech attributes. ". To strengthen this argument, it would be more convincing to include some analytical experiments. For instance, demonstrating that $Z^c$ excels in speaker verification or emotion recognition, but performs less effectively in speech recognition, would provide a more nuanced understanding of its capabilities.

We thank the reviewer for their valuable suggestions to strengthen our claim, which aligns with similar feedback from Reviewer kqQ1. In the general response above, we provide details of the additional experiments conducted. These analytical experiments provide a more nuanced understanding of the features, showing that $Z^c$ perform well on emotion recognition, while the utterance embeddings excel in speaker identification tasks. We hope that this addresses the reviewer's feedback.

The descriptions of the evaluation are unclear. The authors should provide a clear explanation of whether the AR model is utilized for each metric, possibly by referring to section 3.1 of the GSLM paper.

We thank the reviewer for the feedback. In the paper, we evaluate both the reconstruction and continuation capabilities of our model. The AR component is only utilized for continuation. $F_0$-RMSE, MCD, and CER are used to evaluate reconstruction performance, while all other metrics are used to evaluate continuation performance. We update the paper to clarify this distinction as follows:

• Similar to the GSLM paper, we will add a paragraph in Section 4.3 stating that we are evaluating both speech reconstruction and speech continuation, where the reconstruction part doesn't involve the AR model.
• We move N-MOS from Table 1 to Table 2 since the N-MOS is evaluated on speech continuation, not speech reconstruction. As a result, Table 1 includes only reconstruction measures that don't involve the AR model, and Table 2 includes only measures that involve the AR model. We will modify the captions correspondingly for better clarification.

In the main results, the conclusion, "Speech generated from our proposed approach does not sacrifice meaningfulness compared to speech generated from the baselines.", is not strongly supported by the experiments, particularly when considering the observed declines in sWUGGY and sBLIMP. Despite the authors' speculations, the empirical data does not robustly corroborate this claim.

We thank the reviewer for their thoughtful feedback. M-MOS measures subjective meaningfulness, while sWUGGY and sBLIMP measure objective syntactic and grammatical correctness. We believe that M-MOS is the most direct measure of meaningfulness compared to sWUGGY and sBLIMP. Therefore, we support our claim with the observation that our method achieves comparable (and slightly better) M-MOS scores than other methods. We will revise the text to address the feedback. Specifically stating: Speech generated using our proposed approach maintains subjective meaningfulness (as measured by M-MOS) compared to that of the baselines, despite a slight reduction in the objective sWUGGY and sBLIMP scores.

It is very weird that the sWUGGY of (Token + continues features) is worse than that of both Token-LM or continues features-LM. (Proposed vs. Token-LM, Proposed vs. Proposed - tokens, in Table 3).

From Table 2, we observe a trend: the more information encoded in the extracted features, the lower the sWUGGY and sBLIMP scores. This observation makes us believe that this is the cause of the sWUGGY of Proposed is lower than the other methods, as it contains the most information (both tokens and the continuous features). A similar trend is evident in Table 7 in Appendix F, where increasing the number of tokens from $k=200$ to $k=1000$ results in a noticeable decline in both sWUGGY and sBLIMP scores. This supports the idea that the amount of encoded information inversely impacts these metrics.

As the discussion period concludes on December 2, we warmly invite the reviewer to share any remaining concerns or feedback. Your valuable insights are greatly appreciated and will help us further refine our work. If possible, we would be grateful if the reviewer could consider revisiting your score, taking the points we address during the discussion into account.