A$^2$-Flow: Alignment-Aware Pre-training for Speech Synthesis with Flow Matching

Weakness 1 & Q1. Regarding Alignment-aware pre-training

• Importance of alignment modeling First, discussing the importance of alignment modeling, effective alignment modeling is crucial in the TTS field. Non-autoregressive models, from FastSpeech and Glow-TTS to the most recent VoiceBox, have explicitly modeled phoneme durations using a separate duration predictor. In these models, a phoneme duration predictor predicts the duration for each phoneme, expands the phoneme sequence based on the predicted durations, pre-aligns it with speech, and generates audio from the aligned sequence using a decoder. However, as explained in the official comment, the E2TTS approach, which inherently aligns input text tokens with speech through self-attention without using a separate text-token-based duration predictor, has been shown in the E2TTS paper to outperform models like VoiceBox.

  • Implicit alignment modeling holds greater potential in terms of performance in TTS field.

• Existing models like E2TTS and DiTTO-TTS, which learn alignment jointly, have relied on large amounts of transcribed data for training. However, as shown in Figure 2 of this paper, E2TTS-LT, which uses a smaller amount of transcribed data, significantly underperforms compared to E2TTS in zero-shot TTS performance.

  • Zero-shot TTS models that internally perform alignment modeling typically require a large amount of transcribed data.

• The alignment-aware pre-training method proposed in this paper is designed for generative models like E2TTS that inherently align text sequences with speech. The key idea, inspired by E2TTS, is to train the model to jointly align a de-duplicated unit sequence extracted from unlabeled speech with the speech itself and generate speech. During the fine-tuning process, the unit sequence is replaced with text input, allowing the model to adapt the ability to align unit sequences with speech during pre-training into aligning text with speech.

• This approach enables the construction of models like E2TTS, allowing the model to jointly align text and speech while generating audio with relatively little transcribed data and minimal fine-tuning. Additionally, unlike traditional TTS models, the proposed method offers the added advantage of supporting voice conversion.

• Pre-training methods like SpeechFlow work well with approaches like VoiceBox, which rely on an explicit phoneme duration predictor, but as shown in Figure 2 of this paper, they are not well-suited as initialization for models like E2TTS that jointly perform internal alignment modeling. This highlights the advantages of the proposed method as a pre-training strategy for models that inherently learn alignment.

In summary, E2TTS has recently achieved the best performance in the zero-shot TTS field compared to methods using explicit duration predictors. This paper proposes a pre-training method that enables such a model to be built using as little as 1% of transcribed data, making it significantly more efficient.

Weakness 2.

The ability to perform voice conversion is an additional advantage, but this method does not focus on handling multiple tasks. As mentioned earlier, the primary contribution of the paper is proposing a pre-training method that introduces unit representations from unlabeled data to enable alignment learning within the E2TTS framework. This approach minimizes reliance on transcribed data while building speech generative models like E2TTS capable of alignment modeling. As a note, both VoiceBox and AudioBox require explicit duration predictors and do not perform internal alignment modeling.

Weakness 3.

Using a sentence duration predictor is not our main contribution. The models you mentioned, such as SimpleSpeech, DiTTO-TTS, and E3TTS, all perform implicit alignment modeling, similar to E2TTS. However, the E2TTS model, which is primarily used in our paper, demonstrates significantly better performance compared to these approaches. This paper shows that we can achieve performance comparable to E2TTS using only 1% of transcribed data, leveraging unlabeled speech for alignment-aware pre-training.

Q2.

We provide our core contributions in the official comment above.

Q3.

When we refer to alignment modeling, we mean the alignment between the input sequence and speech. In TTS models that use explicit duration predictors, the role of the duration predictor, which models phoneme durations representing the alignment, is crucial. In contrast, our model internally models this alignment information. While it uses a sentence duration predictor, it inherently models the alignment between unaligned units or text tokens and speech. This is why we call it alignment-aware pre-training. Additionally, in the VC task, since de-duplicated units are used as input, the model can internally align the input source speaker and generate modulated speech with a different alignment, offering an additional advantage.

Weakness 1. Limited Baseline Comparisons & Q2

• We primarily show the effectiveness of A^2-Flow by leveraging a large amount of English untranscribed data and comparing it with state-of-the-art zero-shot TTS models for English. In multilingual settings, the amount of untranscribed data available for other languages in our experiments was relatively limited. As a result, rather than aiming to propose a state-of-the-art model for other languages, we focused on showing whether a model trained on multilingual untranscribed data could extend alignment learning to other languages with only a small amount of transcribed data. Consequently, we did not conduct direct comparisons with existing state-of-the-art multilingual TTS models. Pre-training our approach on significantly larger and more diverse multilingual untranscribed datasets to achieve state-of-the-art performance in multilingual zero-shot TTS is beyond the scope of this paper and is reserved for future work.

Weakness 2. Discussion of Limitations

• A%^2%-Flow relies on self-supervised speech units, leveraging HuBERT models in the speech domain. We have explained that our pre-training approach offers advantages over methods like SpeechFlow, which also operate without any conditions, particularly for tasks requiring alignment modeling in the speech domain. However, when extending generative models beyond the speech domain to general audio domains, challenges arise. For instance, the AudioBox paper utilized SpeechFlow, pre-trained without conditions, across both speech and general audio domains. While our A²-Flow approach could potentially be applied in these scenarios, directly using speech-specific self-supervised representations for general audio may not be intuitively appropriate.

Weakness 3. Ablation Study Depth

• In our experiments, we fixed the architecture design for the reproduced E2TTS and our A²-Flow, and therefore, we did not conduct ablation studies on the architecture itself. However, we provided additional results for A²-Flow-T in Appendix A.3, Table 7, to demonstrate the advantages of the model when using the same amount of transcribed data as E2TTS. Additionally, we included ablation experiments on the impact of different unit representations, showing how the choice of unit representation affects performance in the downstream TTS task.

Weakness 4 Methodology Clarity & Q1

• We added explanations for the total length predictor and details on timestep shifting in Appendix A.2. We used the ground truth length during fine-tuning for TTS, and the total length predictor was employed only during inference.

Q3

The DiTTO-TTS and E2TTS models compared in the paper are alignment-free methods. In the DiTTO-TTS paper, the DiTTO-en-L model, which uses the DiT-L architecture similar in size to ours, requires 1.479 seconds (RTF 0.15) to generate 10 seconds of audio. The DiTTO-en-XL model, which uses the DiT-XL architecture and is used for comparison in Table 1 of our paper, requires 1.616 seconds (RTF 0.16) for the same task. The reproduced E2TTS and A²-Flow models in our paper share the exact same architecture, with the only difference being that A²-Flow utilizes unit-based learning. Consequently, the inference speed for both models is identical, achieving an RTF of 0.23 for generating 10 seconds of audio on an A100 GPU, including computations for the total length predictor, flow matching decoder, and vocoder. Compared to VoiceBox and other models with similar architectures, our method generates audio using 32 sampling steps with classifier-free guidance (computing both conditional and unconditional outputs with a batch size of 2) through the first-order Euler’s method. This ensures that the inference time remains reasonably fast.

Weaknesses 1. Additional Experiments

• In the paper, we provided objective metrics (WER, SECS-O) for each model but did not include a separate evaluation of audio quality. However, in Table 2, we presented the results of A/B tests comparing our model with SpeechFlow and E2TTS based on human evaluations, focusing on similarity to reference audio in terms of "prosody, emotion, and timbre," instead of providing CMOS or SMOS. As the baseline models are not open-source, we compared our model to samples available on their demo pages, consistent with prior works.
• Additionally, we have included results evaluating speech naturalness to supplement the existing subjective evaluation and objective metrics. To assess speech naturalness, we provide UTMOS results for all samples in the LibriSpeech test-clean evaluation. UTMOS, which is known to correlate highly with human subjective ratings, serves as a robust proxy for audio naturalness. The average UTMOS scores are included in Tables 6, 7, and 8 in Appendix A.3.
• Furthermore, as requested, we have added a comparison of model performance over different training iterations in Table 7, directly comparing the results with E2TTS. Additionally, to demonstrate the effectiveness of the pre-training method with different unit representations, we have included experimental results in Table 8, highlighting the impact of various unit representations.

Q1-1) Speech Duration Predictor Design and Robustness

We detailed the total length predictor's design in Appendix A.2.1 and confirmed its robustness for shorter and longer texts. Using samples from LibriSpeech test-clean, we computed the mean and 95% confidence intervals of the ratio between predicted length and ground truth length for various durations:

The results demonstrate that the total length predictor performs robustly across varying text lengths, outputting ratios close to 1. For very short text inputs, we concatenate the prompt text with the target text sequence to predict the duration. Sample results for these cases are provided in the demo page.

Q1-2) WER Performance by Text Length

The WER and its 95% confidence interval for each duration range were computed as follows:

Although WER slightly worsens for shorter or longer durations compared to the 4–10 second range, the alignment remains stable, maintaining good WER performance.

Q2) Impact of More Iterations for Fine-tuning

We extended the fine-tuning process to 300K iterations, adjusting the learning rate decay schedule to reach zero at the end. The results are provided in Table 7 (Appendix A.3). Below is a summary:

Increasing fine-tuning to 300K iterations improves text-speech alignment, reducing WER to 1.9, comparable to E2TTS. However, SECS-O slightly declines from 0.704 (150K FT) to 0.695 (300K FT), likely due to overfitting to LibriTTS. Despite this, A$^2$-Flow’s SECS-O remains comparable to E2TTS, which uses significantly more transcribed data for training. These results demonstrate that A$^2$-Flow effectively maintains strong alignment and TTS performance with extended fine-tuning, even outperforming E2TTS in WER with fewer resources.

Q3) Impact of the De-duplication Step in HuBERT Processing

The de-duplication step in our method is essential for learning the alignment between input sequences and speech. While HuBERT unit sequences are inherently aligned with speech, removing consecutive repetitions creates de-duplicated units that focus on semantic transitions. This enables the model to learn both alignment and duration information internally during pre-training.

Without de-duplication, the model would process repeated tokens, making it difficult to learn alignment or adapt effectively to downstream tasks like TTS, where aligning text and speech is critical. The absence of this step would also hinder fine-tuning in scenarios without explicit duration information, as the model would lack the foundational alignment learned during pre-training.

Thus, de-duplication ensures that the pre-training closely mimics alignment challenges in TTS, making it a core component of our alignment-aware pre-training approach.

Weaknesses 1.

• Regarding ASR Transcribed Data for High-Quality Zero-Shot TTS

While it is possible to construct transcribed datasets for TTS using ASR systems, this approach assumes the ASR model is highly accurate and produces reliable transcriptions. ASR models often struggle with segmentation issues, such as mid-word cuts, which significantly impact performance. Additionally, constructing datasets like the Emilia dataset in F5-TTS requires complex preprocessing steps, such as ASR using Whisper and WhisperX, followed by filtering based on language ID, speech quality, and average character duration.

While effective for English, this process becomes more challenging for languages with less accurate ASR models. Additionally, even for English, Whisper outputs may include hallucinations. According to the mentioned F5-TTS paper, training E2TTS on the Emilia dataset resulted in a WER of 2.95, compared to 1.9 (with $\alpha=3$) and 2.1 (with $\alpha=1$) for our reproduced E2TTS, likely due to transcription inaccuracies. Our proposed A²-Flow offers an alternative by using HuBERT units for pre-training, eliminating the need for ASR or transcriptions. Notably, both the HuBERT units and pre-training data are entirely untranscribed.

This approach enables competitive TTS performance with minimal transcribed data, as demonstrated for English. We believe it offers significant benefits for communities with large-scale untranscribed data but limited transcription resources, enabling high-quality zero-shot TTS for under-resourced languages.

• Regarding Training Resources:

A²-Flow and E2TTS were trained using 32 GPUs for pre-training, while A²-Flow fine-tuning on LibriTTS utilized only 8 GPUs. Calculating GPU hours, 150K fine-tuning iterations on 8 GPUs equates to 37.5K iterations on 32 GPUs. For transparency, we share the results for A²-Flow fine-tuned for 150K iterations (8 GPUs) and E2TTS trained for 700K and 850K iterations (32 GPUs). The experimental results are detailed in Appendix A.3, Table 7.

As shown, training E2TTS for an additional 150K iterations (total 850K) resulted in negligible improvement compared to 700K iterations. This demonstrates that A²-Flow’s results were not achieved simply by training longer. Additionally, A²-Flow fine-tuning required fewer GPUs and less transcribed data than E2TTS while achieving superior SECS-O performance.

Furthermore, we provide results for A²-Flow-T, which refers to A²-Flow fine-tuned on 40K hours of transcribed data (the same amount used for E2TTS) using 32 GPUs. With the same computational resources as E2TTS at 850K iterations, A²-Flow achieved a WER of 2.0 and a SECS-O of 0.711, surpassing E2TTS in SECS-O performance. These results demonstrate that even with fully transcribed datasets, the alignment-aware pre-training of A²-Flow enhances zero-shot TTS learning, creating a synergistic effect that significantly boosts performance.

Additionally, our pre-training method not only demonstrates boosted performance but also shows that the pre-trained model itself enables high-quality zero-shot VC without the need for fine-tuning.

Weaknesses 2.

It is true that a pre-trained English model can be fine-tuned with limited transcribed data for other languages and achieve reasonable results. As shown in the table above (and in Table 7 of the paper), A²-Flow demonstrates superior performance compared to E2TTS under equivalent conditions, particularly in SECS-O metrics, when fine-tuned with the same amount of transcribed data. This underscores the advantage of A²-Flow in scenarios with limited transcribed data.

However, A²-Flow offers a unique benefit beyond the ability to fine-tune from transcribed data: it leverages untranscribed data effectively to enhance zero-shot TTS performance. For languages where transcribed data is scarce but large amounts of untranscribed data are available, A²-Flow provides a robust method to utilize such resources. In contrast, traditional approaches like E2TTS heavily depend on transcriptions and face challenges in incorporating untranscribed data to boost performance.

By employing untranscribed data during the pre-training stage, A²-Flow offers a scalable and practical solution for low-resource languages. This capability significantly enhances zero-shot TTS performance without requiring extensive transcription resources, addressing a critical gap in zero-shot TTS development and broadening its applicability to under-resourced language communities.