Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis

2. Section 3.3 is a little bit unclear. For instance, how was this energy function $e_{ij}$ calculated? Does it have something to do with the cross-attention weights acquired by the transformer?

The energy is computed using cross-attention between encoder and decoder features, and monotonic alignment is enforced based on the energy terms. We have added Algorithm 1 in the appendix, which provides a detailed explanation of the energy terms, monotonic alignment process, and decoder functionality.

3. What are the specs for the baseline methods: StyleTTS-2 and HierSpeech++? Are they of similar parameter size?

We used the standard versions of the models in their GitHub repositories. The StyleTTS-2 models have a total of 142M parameters, and HierSpeech++ has 97M parameters. Both models have more parameters than our GMM-LM mini model, which outperforms them in both MOS and WER. We have included the model size information in the revised manuscript.

We sincerely thank the reviewer for taking the time to read our paper and provide thoughtful feedback. We have addressed the reviewer’s comments and concerns as outlined below.

1. Concerns regarding the separation of contributions between the GMM approach and the proposed monotonic alignment.

We understand the reviewer’s concerns and explain below why the proposed monotonic alignment was not included in VALL-E and other models, along with our efforts to separate contributions in the revised manuscript.

Why the proposed monotonic alignment was not included in VALL-E and other models:Our monotonic alignment method refines cross-attentions between the encoder and decoder to enforce monotonic alignment. However, it is unclear how to apply this method to a decoder-only model like VALL-E, which relies solely on self-attention. This is why we did not include the proposed monotonic alignment for VALL-E. As for StyleTTS2 and HierSpeech++, both already incorporate monotonic alignment mechanisms. StyleTTS2 uses a duration prediction model to enforce strict monotonic alignments, while HierSpeech++ employs monotonic alignment search (MAS), as introduced in the VITS paper.

Is monotonic alignment the main or the only reason for WER improvement? We agree with the reviewer that it is important to separate the contribution of continuous tokenization in our GMM approach from the proposed monotonic alignment method. To address this, we conducted a comparison using discrete autoregressive (AR) models with the proposed monotonic alignment method, implemented using the same architecture as the GMM-LM large version (315m), except for the discrete codec embedding layer and softmax layers.

Specifically, we implemented two discrete AR encoder-decoder models:

1. The first model used discrete codes extracted from a VQ-VAE codec model, implemented with a single codebook containing 8192 entries and using the same architecture as the DAC model [1]. This model was trained to predict the next tokens using standard cross-entropy with a single softmax layer.

2. The second model used discrete codes extracted from the DAC model with 8 codebooks, each containing 1024 entries. The model adopted delayed codebook prediction with multiple softmax layers, as proposed in MusicGen [2] and [3].

The WER (%) of these models are presented in Table 10 of the revised manuscript and summarized below.

The results clearly show that monotonic alignment contributes to WER improvement. However, even with monotonic alignment, the discrete models do not achieve the same performance as the proposed GMM-based version. We believe this performance gap can be attributed to two factors:

1. Quantization and discrete representations introduce limitations:
   Discrete representations can result in mispronunciations and artifacts. This is why some researchers use embeddings before the AR model's softmax layer as input to waveform decoders as a workaround [4,5].

2. Challenges of applying monotonic alignment to RVQ-based models:
   RVQ models increase the complexity of TTS systems, as they require multiple softmax heads to predict codes from different codebooks. While it is possible to predict all codebooks in parallel at each timestep, this approach ignores dependencies between codebooks and yields suboptimal results. A better approach, as used in VALL-E and the delayed prediction model in Table 10, is to predict "coarse" codes first, followed by "fine" codes. However, this approach complicates monotonic alignment because alignment must be performed using only the coarse codes at each timestep. This limitation likely contributes to the higher WER observed with RVQ-based models.

[1] Kumar, Rithesh, et al. "High-fidelity audio compression with improved rvqgan." Advances in Neural Information Processing Systems 36 (2024).

[2] Copet, Jade, et al. "Simple and controllable music generation." Advances in Neural Information Processing Systems 36 (2024).

[3] Lyth, Dan, and Simon King. "Natural language guidance of high-fidelity text-to-speech with synthetic annotations." arXiv preprint arXiv:2402.01912 (2024).

[4] Casanova, Edresson, et al. "XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model." arXiv preprint arXiv:2406.04904 (2024).

[5] Betker, James. "Better speech synthesis through scaling." arXiv preprint arXiv:2305.07243 (2023).

4. I am wondering the validity of modeling the autoregressive distribution as a GMM. In your case, even if the KL regularization loss of GMM-VAE makes $q(h|x)$ a mixture of Gaussians, does it in any sense implies that the autoregressive conditional distribution $p(h_t|h_{t-1}.\ldots)$  is also close to a Gaussian mixture?

The validity of interpreting the AR distribution as a GMM can be justified by the fact that GMMs are universal approximators of any smooth distribution [1,2]. Thus, the question is not whether a given distribution is inherently a GMM but rather whether it can be efficiently approximated by a GMM. Our positive results confirm that this is indeed achievable.

Intuitively, the latent space distribution $p(h)$ can be interpreted as a marginalization over all possible phonemes, while the conditional distribution $p(h_t | \text{phoneme})$ represents the distribution conditioned on a particular phoneme. For a specific phoneme, $p(h | \text{phoneme})$ is likely a subset of the GMM. Since this exact subset is unknown, we train a model to predict it.

[1] Carreira-Perpinan, Miguel A. "Mode-finding for mixtures of Gaussian distributions." IEEE Transactions on Pattern Analysis and Machine Intelligence 22.11 (2000): 1318-1323.

[2] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.

I thank the authors for the detailed experiments. This additional experiments of comparing with the Discrete Methods w. monotonic alignment from the authors resolves my main concern about the paper. I am willing to raise the score.

Additionally, I want to clarify on the fifth point of the author's response, I provided that reference to provide grounding for why the the ablation of monotonic alignment is needed, not to suggest inadequate reference of the original submission.

We sincerely thank the reviewer for taking the time to read our paper and provide thoughtful feedback. We have addressed the reviewer’s comments and concerns as outlined below.

1. There is a minor question of motivation in the author's approach: they take the stance that the community views vector quantization approaches as a necessity, but there are a fair amount of approaches in the speech modeling community that have used straight reconstruction approaches.

Thanks for point this out. We have added several literatures using reconstruction and multi-task approaches for learning speech features as the reviewer suggested in literature review section (line 88-91).

2. While their gaussian mixture approach is still suitably novel, this position seems to ignore other considerations that go into VQ approach. Notably that the use of discrete tokens is relatively easy to implement in parallel with text encoding, all while minimizing storage an I/O limitations from audio/image processing.

We have emphasized in the revised manuscript that the VQ approach is more efficient in terms of storage and I/O and enables the direct use of language models (line 100-103).

3. Given the reliance of the model architecture on monte-carlo estimation, how sensitive are results to random seeding during experimentations?

We found that the model is not sensitive to random seeds in our experiments. With a batch size of 680 for GMM-VAE training, we observed no noticeable variations between runs. We have clarified this for readers in the revised manuscript.

6. Could you clarify how the monotonic alignment mechanism in Figure 2 (right) works? It seems to align the encoded text and speech prompts prior to decoding. Additionally, a more comprehensive description of the GMM-LM would be nice, including how speech and text features are fed into each decoder step, and the formulation of $e_{i,j}$.

In the revised manuscript, we have provided a step-by-step explanation in Algorithm 1 detailing how text and speech features are aligned monotonically, integrated into the decoder, and used to compute the negative log-likelihood.

5. Why does the model require relatively few Gaussian mixtures compared to VQ codes in RVQ.

To explain this phenomenon, we consider the distinctions between Vector Quantization (VQ) codes and Gaussian Mixture Models (GMMs) in representing distributions. VQ codes do not directly correspond to Gaussian mixtures; instead, they represent a distribution similarly to fitting a histogram, where probability mass is assigned only to specific embedding points (the codebook vectors). This approach allows for the flexible placement of probability mass exactly where it is needed, unconstrained by the form of continuous distributions like GMMs, which always assign some probability mass around the modes and in the surrounding regions.

VQ works particularly well for distributions with many modes and minimal probability mass between them. It efficiently captures each distinct mode without concern for intermediate regions. However, it can become inefficient if there is significant probability mass between the modes, as it would require a large number of codebook vectors to accurately represent the distribution.

On the other hand, GMMs can model multi-modal distributions effectively by utilizing continuous distributions. They naturally allocate probability mass not only at the modes but also between them, due to the inherent properties of Gaussian functions. If the modes are far apart, a GMM may need as many components as there are modes, potentially making it less efficient than the VQ approach in such scenarios. However, for distributions where the modes are close and there is a smooth transition between them, GMMs are better suited because they can capture the gradual changes in probability density effectively.

Therefore, the choice between VQ and GMM depends on the nature of the latent space distribution. For distributions with distant modes and minimal probability mass between them, VQ or other discrete distributions are more effective. Conversely, for distributions where the modes are close and there is significant probability mass in between, continuous distributions like GMMs are preferable. Based on our experiments, it appears that the latter scenario applies in this case, indicating that GMMs are the more appropriate choice.

We sincerely thank the reviewer for taking the time to read our paper and provide thoughtful feedback. We have addressed the reviewer’s comments and concerns as outlined below.

1. Limited discussion of prior Gaussian mixture VAE work, e.g., "Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders"(Minor: The notation, either GMM-VAE or VAE-GMM, should be consistent.).

Thanks for pointing out the relevant prior work on Gaussian mixture VAEs. We have incorporated the suggested reference, along with several other works on VAEs with learned priors, into the literature review section under the subsection "VAEs with Learned Priors" (lines 146–155). We have made the notation consistent as “GMM-VAE” throughout the manuscript, replacing the two instances of “VAE-GMM.

2. The counterintuitive result where increasing Gaussian mixtures in GMM-VAE leads to worse reconstruction, where a 6-mixture GMM should subsume the modeling capacity of a 3-mixture GMM

The reconstruction loss reported in Table 4 is based on the test set (LibriSpeech test clean). The performance gap in terms of VAE reconstruction is relatively small (0.71 vs. 0.73 when λ=0.1, and for λ=10, the 6-mixture model slightly outperforms the 3-mixture model with 0.98 vs. 0.95). To investigate further, we sampled a subset of the training data (200 randomly selected speakers with 5 utterances each) and computed the reconstruction loss. In this case, the 6-mixture model consistently outperformed the 3-mixture model, suggesting that the 6-mixture model may slightly overfit the data. We have included these additional experiments (Table 9) and discussions in Section A.3 of the revised manuscript.

Additionally, we believe that when the GMMs are parameterized using neural networks, the 6-mixture GMM may not fully subsume the modeling capacity of a 3-mixture GMM. In a traditional GMM, a 6-mixture model typically has twice as many parameters as a 3-mixture model, allowing it to subsume the latter’s modeling capacity. However, in our approach, which utilizes a neural network for parameterization, both the 3- and 6-mixture models have approximately the same total number of parameters. Consequently, the 6-mixture model has fewer parameters per mixture, which may limit its ability to completely subsume the modeling capacity of a 3-mixture GMM.

3. Some modeling details are missing, e.g., GMM-VAE frame rate, which is crucial as it could affect the type of information captured

For GMM-VAE, we used the same model architecture as described in the DAC paper [1] with modifications that removed all quantization-related layers. The encoder rate are decoder rate are [8, 8, 4, 2], for 16khz speech it is 320ms frame rate. We have included the frame rate and the more details of GMM-VAE in section A.5 of the revised paper.

[1] Kumar, Rithesh, et al. "High-fidelity audio compression with improved rvqgan." Advances in Neural Information Processing Systems 36 (2024).

4. Does the frame rate of GMM-VAE features align with the mel spectrogram hop length (240ms) described in 5.2?

They are not the same. The frame rate of GMM-VAE is 320. The mel spectrogram hop length is 240ms. We have included this detail in the revised paper.