F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

We sincerely appreciate your positive review and constructive comments. Below we address the concerns mentioned in the review.

W1,2: Lack of direct comparison with previous works and on LibriSpeech test-clean.

Due to space constraints, we put the comparison in the Appendix. Since the setup of prompts in previous works' test sets remains unknown, we take the LibriSpeech-PC (LS-PC) 4~10s filtered set (which is organized in the same way as in the E2 TTS paper, despite having differently paired reference audios for the reason mentioned above) to compare with reproduced E2 TTS and other open-source models.

Meanwhile, we additionally conduct evaluations on a non-capitalized LS-PC 4~10s test set while removing all punctuations, which can also be viewed as a LibriSpeech test-clean 4~10s subset. A pity is that the LS-PC misses one speaker, though it still serves as a solid set of 1127 samples in total 2 hours, enabling future works to switch from a PC or non-PC version with the same reference audio pairs.

Part of updated evaluation results (Hubert-large-based ASR model to transcribe, results in brackets are with Whisper-large-v3 as before, more results in Appendix B.6), asterisk denotes scores reported in E2 TTS paper (both are English-only models trained on Libriheavy 50K hours):

W3: Redundant (w/o SS) in Tables 1 and 2.

Many thanks for pointing out this, we have handled it in the updated version.

Q1: Worse WER compared to closely related two models in Table 1.

In Table 1, we are making efforts toward a thorough comparison with existing outstanding models. To be specific, we divide the table into 3 subsets to avoid any ambiguity as to the fact of different reference audio pairs posed in each subset (otherwise we could take the scores in the above table to compare).

From the relative result compared to each subset's ground truth score, we are confident that our F5-TTS shows no worse performance. Moreover, since we leveraged a multilingual in-the-wild dataset for training which contains not typically high quality and uncensored speech-text pairs compared to the English-only audiobook corpus leveraged by other models, there will be broader space for F5-TTS to improve on English tasks.

Q2: Will Sway sampling have the advantage but at the same time hurting speaker similarity to some extent?

We appreciate this highly observant question. From the overall objective metric, Sway Sampling always improves the performance (more rigorously, Sway Sampling with (more) negative coefficient). We have designed the function so that the timesteps from the larger side will not vanish (still holds several NFE steps there, not deviate too much from the training pattern) even with the most aggressive Sway Sampling case.

While in practice, we observed a slight increase in the probability of some artifacts (e.g. exaggerated fricative), when using a more aggressive Sway Sampling strategy with the same classifier-free-guidance (CFG) coefficient (2.0 in our paper). Intuitively, a more aggressive Sway Sampling simulates a uniform sampling with more NFE on the small timestep side, while the common practice of previous works is to search for the most suitable CFG coefficient accordingly. Due to main text space limitations, we did not perform meticulous ablation of CFG choice but here we are willing to share this observation.

Q2: The lower limits of the data threshold while still maintaining stable training

We are pleased to share our relevant experimental results here and have updated them in our paper. We train F5-TTS 158M small models on LibriTTS 585 hours and LJSpeech 24-hour English datasets to provide some insights. The model trained on LibriTTS is evaluated on LibriSpeech-PC test-clean. The model trained on LJSpeech is evaluated on in-set 1000 samples since LJSpeech is an up-to-10-second single-speaker dataset.

How much training time is typically needed to achieve stable pronunciation accuracy?

Every 100K update takes around 8 hours on 8 NVIDIA H100 SXM GPUs. Thus 40 hours on LibriTTS to achieve pretty good zero-shot results (an even lower WER than ground truth, and an acceptable SIM with only 585 hours); and 16 hours on the single-speaker LJSpeech to learn character-speech alignment and close voice-cloning.

We sincerely appreciate your careful reading and providing constructive feedback. Below we address the concerns mentioned in the review.

W1: Limited contributions as a DiT structure or text encoder already proposed in previous studies.

We acknowledge that we have not made radical innovations in proposing a new backbone structure, but we believe that our work has the following sufficient contributions in the scope of generative models:

1. We first prove the feasibility of a medium-scale (336M) adaptive layernorm DiT model with effective ConvNeXt for text refinement on the zero-shot text-to-speech task. To be specific, we achieve comparable performance with the in-context-learning pattern compared to the cross-attention structure nor with explicit phoneme-level alignment, and it is with a much smaller model size compared to recent DiT-based works.

2. We also arranged a multilingual training setup with a large-scale in-the-wild but uncensored corpus, which was organized with the Whisper medium for language identification and transcription. On the one hand, our work achieves comparable performance with a relatively small model size compared to other models (many are English-only models trained on high-quality audiobook datasets) on English-only tasks. On the other hand, we conduct evaluations on more languages (here Mandarin) and verify our model's seamless code-switching ability (no cherry-picked samples on the demo page) which are also contributions made promoting and exploring capacities under the new paradigm of using sentence-level duration.

3. The Sway Sampling strategy is proposed to improve sampling efficiency and can be easily applied to other flow matching models, which just fit our simple and effective scheme.

4. We believe that less is more. We contribute and make transparent a simple and effective design pointing out that a modest refinement for text can allow good results while bringing a tidy pipeline, fast training, and fast inference which are valuable qualities within the scope of generative models.

W2,Q1: A richer comparison between Sway Sampling and other sampling strategies.

Thanks for the constructive suggestion. We have updated Appendix B.4 with comparison results of Sway Sampling incorporating

• the Euler ODE solver (first-order; default in our updated version, for F5-TTS base model's 1.2M updates result in Table 1 and 2),
• the midpoint ODE solver (second-order; used by Voicebox, E2 TTS, and our previous version of F5-TTS),
• the Heun-3 ODE solver (third-order improved Heun; to make a difference with second-order solver),

while switching off Sway Sampling equals to uniformly sampled timestep at inference time used by previous TTS models. From the results, Sway Sampling further allows switching from 2nd-order midpoint to 1st-order Euler which brings significant speed up and maintains the performance.

We appreciate your feedback and are very willing to include the results of the UTMOS evaluation. We have also updated the paper with proper citations of UTMOS and the used repository.

The UTMOS results confirm with CMOS in the main text, and the differences between switching on and off Sway Sampling (SS) add to the effectiveness. It is quite interesting that the simplest 1st-order Euler ODE and the most aggressive SS supplement each other better (in terms of performance and efficiency; bold).

We also include UTMOS results for small models trained with limited data scales.

The UTMOS with SIM-o implies promising performance training with a certain amount of high-quality data.

Thanks again for the constructive feedback which also brings discoveries for us. We are always open to further questions and would be happy to discuss.

W2 Quality: 1. Correctness of E2 TTS implementation. 2. Fairness and comparability of the results using Whisper-large-v3. 3. Overstatement to claim state-of-the-art performance 4. Demo samples issues. 5. Duration predictor.

1. We have had discussions with E2 TTS authors and confirmed the correctness of the implementation. Our base model is trained on an in-the-wild multilingual dataset which leverages Whisper-medium as the language identifier and to transcribe, and the comparisons are made with English-only models trained on much higher quality audiobook data, thus the reproduced E2 TTS has rational scores (WER 2.95 to ground truth 2.23, SIM-o 0.69 to 0.69).

2. We are encourage using a unified ASR though we are willing to provide results with the Hubert-large-based ASR model. Since the commonly used latter is finetuned on LibriSpeech, the scores are generally better while having similar relative values. To further ensure fairness, we use BigVGAN as the vocoder. As suggested, we compare with the original E2-TTS reported scores (denoted with asterisk) on LibriSpeech-PC (LS-PC) test-clean: | Model | LS-PC test-clean WER | SIM | |-----|-----:|-----| | Ground Truth| 2.0* | 0.695* | | Voicebox | 2.2* | 0.667* | | E2 TTS | 2.0* | 0.675* | | F5-TTS | 1.81 | 0.67 |

3. In addition to the above results, we also updated more evaluation results in Appendix B.6, and we are encouraging comparisons among the same test sets and with the corresponding ground truth (otherwise distorted, e.g. our F5-TTS achieves a WER of 1.81, but we are only claiming that we are among the leading TTS systems in this case since the prompt pairs used are different).

4. We appreciate your careful listening to our demo samples. We are trying to present the results in the most rigorous way possible, so the numbers we report are the average of multiple runs and there are no cherry-picks for samples you mentioned. Thanks again for pointing out the shortcomings, which shows that there is a lot to explore in the field of TTS, and motivates us to continue to work hard.

5. We adopted the simple estimating way for simplicity, and the duration predictor will surely help.

W3 Clarity: 1. Lack of clear explanations for several hyperparameters' choices. 2. Reasoning with GT duration implies robustness. 3. Unclear explanation with E2 TTS consistent failures. 4. Other wording and referencing.

1. Due to the main text space limits and the expectation of the paper we all agreed, that we are not introducing solely hyperparameter searching ablations. Thanks for pointing out the Sway Sampling coefficient, which is updated in Appendix B.3. More aggressive Sway Sampling keeps improving performance and allows switching from 2nd-order midpoint to 1st-order Euler which brings significant speed up. | F5-TTS 16 NFE | RTF | LS-PC test-clean WER | SIM | UTMOS | Seed-TTS test-en WER | SIM | UTMOS | Seed-TTS test-zh WER | SIM | UTMOS | |----|----|----:|----|----|-----:|----|----|-----:|----|----| | s=-1, Euler | 0.15 | 2.53 | 0.66 | 3.88 | 1.89 | 0.67 | 3.76 | 1.74 | 0.75 | 2.96 | | s=-1, midpoint | 0.26 | 2.43 | 0.66 | 3.87 | 1.88 | 0.66 | 3.70 | 1.61 | 0.75 | 2.87 | | s=-0.8, Euler | 0.15 | 2.82 | 0.65 | 3.73 | 2.14 | 0.65 | 3.70 | 2.28 | 0.72 | 2.74 | | s=-0.8, midpoint | 0.26| 2.58 | 0.65 | 3.86 | 1.86 | 0.65 | 3.68 | 1.70 | 0.73 | 2.83 | (Update 2024/11/21 UTC-12, extra evaluation with UTMOS)

2. Having some gain when providing GT does not imply robustness, but allowing "more" gain implies robustness in alignment which means F5-TTS can leverage better the truth information. At the same time, E2 TTS is relatively dull and insensitive.

3. "Re-ranking" is a commonly put term for retrieving an improved result with multiple sampling (e.g. conduct 5 times the generation and pick the best one based on metrics). The content you have mentioned is an indispensable part of E2 TTS's robustness issue analysis. The depiction of details of E2 TTS's "consistent failures" and ablation results with input condition that drops reference audio corroborate each other, and together have great significance and connection to verify that E2 TTS structure deeply entangles acoustic and semantic features.

4. Yes, F5-TTS still exhibits significant length differences, similar to E2-TTS, but "mitigated" with our refinement.
   Thanks for pointing out the lack of reference to MAS with Glow-TTS, we have updated in main text.
   The cross-sentence term might be phrased first in Voicebox but the task is not introduced by Voicebox (as far as we know, VALL-E is earlier than Voicebox).
   SIM-o is put as "speaker Similarity between generated and the original target speeches", the same as "similarity against the original audio context" in Voicebox, or the reviewer is implying a lack of referencing to this "advocacy" from Voicebox (untypical, not referenced based on this term by NaturalSpeech 3, E2 TTS, etc.).
   The long skip connection is from UNet, and DiTTo-TTS also borrows from it.

First of all, we want to thank the reviewer for carefully reading, spotting unclear points, and providing constructive feedback! Below we address the concerns mentioned in the review.

W1 Orig. & Sign.: 1. Whether sufficient contribution with the proposed simple refinement module; in-context-learning compared with cross-attention; and whether E2-TTS-like architecture has rigid demand to a large dataset. 2. Lack of model structure search with refinement module alternatives. 3. Novelty with Sway Sampling.

1. In-context-learning with Voicebox-like design has shown longstanding impressive performance, and E2 TTS has exactly the same structure but removing phoneme-level force alignment. We have first made efforts to diagnose the slow convergence and low robustness issues with the E2 TTS structure (Sec.5.1, Appendix B.2), and are not just adding to UNet-Transformer but equipping the adaptive layernorm DiT with ConvNeXt modest refinement.
   
   Compared to DiTTo-TTS (740M, English-only trained on 55K hours, cross-attention DiT with extra text encoder and codec joint-finetuning), our F5-TTS (336M, multilingual 100K hours, in-context-learning with simply padded character sequence) has -0.6 WER (-23%), +0.043 SIM (+7%), and -0.012 RTF (-7%) on LibriSpeech test-clean (Table 1 and 8, Hubert-large-based model to transcribe). It is clear that our in-context-learning architecture has overall better performance, and a certain degradation of performance is expected if replacing modules in DiTTo-TTS with a simply padded character sequence, using in-the-wild data than audiobooks, and training with a smaller model size.
   
   We further include experiments in Appendix B.7 to address your concern about dataset scale. It is clear from evaluation results on small models trained on LibriTTS 585 hours and LJSpeech 24 hours, that our F5-TTS enables stable training to learn speech-text alignment (without grapheme-to-phoneme) with varying data amounts. | F5-TTS small | Trained on | LibriTTS | 585 hours | Trained on | LJSpeech | 24 hours | |---|----:|:-----:|------|-------:|:-----:|------| | | WER | SIM | UTMOS | WER | SIM | UTMOS | | Ground Truth | 2.23 | 0.69 | 4.09 | 2.36 | 0.72 | 4.36 | | 100K updates | 29.5 | 0.53 | 3.78 | 5.64 | 0.72 | 4.17 | | 200K updates | 4.58 | 0.59 | 4.07 | 2.93 | 0.72 | 4.18 | | 300K updates | 2.71 | 0.60 | 4.11 | 3.26 | 0.71 | 4.12 | | 400K updates | 2.44 | 0.60 | 4.11 | 3.90 | 0.70 | 4.05 | | 500K updates | 2.20 | 0.60 | 4.10 | 4.68 | 0.70 | 3.99 | | 600K updates | 2.23 | 0.59 | 4.10 | 5.25 | 0.69 | 3.93 | (Update 2024/11/21 UTC-12, extra evaluation with UTMOS)

2. Our work focuses on providing a simple and effective modeling way for force-alignment-free NAR TTS and has put forward that a lightweight and modest refinement for padded character sequence is sufficient facilitating model to learn alignment, rather than exploring sweepingly best DiT or UNet-Transformer variant.

3. We have clearly stated in our main text that our Sway Sampling is an inference-time strategy while a commonly put cosine scheduler is applied for the training stage. While we acknowledge that we are not the first to emphasize the significance of inference sampling strategy, but as far as we know, there are not any previous strategies like our method that explicitly bias to smaller timesteps and can seamlessly incorporate with fixed-step ODE solvers that are closer to the works you have mentioned (just for reference, Voicebox and E2 TTS are using midpoint ODE solver).

Q1: "Fairytaler Fakes Fluent" in the title does not clearly convey the content or main contributions of the paper.

We all know Andersen's fairytale, "Fairytaler" bears our dream to advance TTS techniques that can read us a bedtime story. "Fakes" is a reminder for technical supervision. "Fluent" is for good prosody, and "Faithful" is for nice WER and SIM. "Flow matching" is the basic method.

Q2: Examining whether entanglement is a structural limitation in both E2-TTS and F5-TTS. To include an ablation study where a modified model uses cross-attention for text conditioning instead of adding filler-padded text to speech.

As we have stated in W1-1 and W3-3, it is clear that our F5-TTS enables stable training and inference addressing E2 TTS robustness issues (with concrete analysis on entanglement of E2 TTS), and a cross-attention structure has no clear advantages compared to our design (F5-TTS has globally better performance to DiTTo-TTS which is trained on English-only data with a double model size and more complex design requirement extra training).

Q3: Lack of experiments to demonstrate the effectiveness of two-stage control of CFG. What specific alignment issues arise and how does the two-stage control address these problems?

We appreciate your careful reading. Although not adequate for our main text to extend this experiment since not closely related to our main research problem and out of the scope of scientific research of "generative models", we are willing to share our informal assumption about it.
Since the two-stage control of CFG has a larger potential to let the model predict solely based on text, it intuitively enhances the model alignment ability for text modality which is harder to learn from a uniform observation (see evaluation results in the early stage of training). No "specific alignment issues" are outlined from our side, simply better performance.

Q4: Insufficient information about MOS. Mentioning studies that used similar MOS evaluation standards could provide a basis for comparison.

We appreciate your advice. We have included Appendix C to provide more details of our subjective evaluations. Briefly, in both the CMOS and SMOS tests, 20 natives were invited to evaluate 30 rounds with randomly selected utterances for all test sets and models. To make a comparison as suggested, DiTTo-TTS received and reported 6 and 12 ratings for SMOS and CMOS, respectively, NaturalSpeech 3 invited 12 natives to judge 20 samples for CMOS and 10 samples for SMOS.

Q5: An explanation of why F5-TTS−Conv2Text (a pure DiT adaLN structure) failed.

As the reviewer has mentioned, F5-TTS−Conv2Text is similar to E2-TTS in setup. With the same training configuration, when E2 TTS can produce intelligible speech, F5-TTS−Conv2Text still produces gibberish. Since the pure DiT adaLN structure is even more slimmed without any skip connections compared to E2 TTS (which already has issues with slow convergence and low robustness), it makes sense that F5-TTS−Conv2Text "fails to learn".

typo

Thanks for mentioning. We have made corrections in the updated version.

We sincerely appreciate your feedback and are willing to have a point-to-point discussion.

W1 Orig. & Sign.

• What do the authors consider to be the research question (RQ) of this study?

  • RQ1.1: With an embarrassingly easy scheme and promising results, why E2 TTS has slow convergence and low robustness issues, what is the possible reason?
  • RQ1.2: While keeping an easy scheme, is there a way to make training and inference faster, and stabler?
  • RQ2: Is the new paradigm (no phoneme-level force-alignment) feasible in more general cases? e.g. on in-the-wild data, multiple languages, code-switching, etc.
  • RQ3: Based on the observation both in training and inference, is there a better and explainable sampling strategy?

• The Sway Sampling is one of our main points (RQ3). We have made a lot of efforts to address your concern (also R-mAVA Q4 and R-X8VX Q2) and would like to receive the reviewer's feedback.

• Is this alignment ability a consequence of scaling up the dataset? seems addressed by our previous response while still listed (just to make sure).

• Fundamentally, it remains unclear why E2-TTS-like architecture is capable of learning some degree of alignment without cross-attention mechanisms is the most concerned point and we would like to make it clear since closely related but different to our RQ1.1.

  • To make it clear, Seed-TTS, DiTTo-TTS, and E2 TTS are concurrent works. Although the reviewer is giving superiority to cross-attention, we need to emphasize again that in-context-learning (in Voicebox, etc.) is widely used across variable research fields and cross-attention is not the only way to achieve alignment and has no earliness on the new paradigm. Thus we are not surprised that the alignment can be achieved and it is no less natural to use in-context learning than cross-attention.
  • We clarify the difference between the reviewer's feedback with our RQ1.1. With a scientific perspective, we located and disclosed the deep entanglement of acoustic and semantic features with careful observation and experiments. Thus we propose an entanglement-mitigated adaLN DiT with a lightweight and modest text refinement module (RQ1.2).
  • From the comparison between E2 TTS and DiTTo-TTS, a more complicated cross-attention structure with extra design has no significant advantages to a smaller and simpler in-context learning structure. Thus with RQ1.2, we come up with F5-TTS.
  • While we acknowledge that a sweeping model structure searching will definitely help to further prove the superiority of adaLN DiT with ConvNeXt or find a better combination, sufficient experiments are made to confirm the effectiveness of our proposal to RQ1.2.

• Simply demonstrating improved performance over E2 TTS is an incremental contribution.

  • We personally think a poor contribution judgment for our work is not that fair.
  • Apart from the abovementioned contributions (RQ1 RQ3) and given the background that many zero-shot large-scale models are not open-source and hard to evaluate, we not only made commendable efforts to compare with many current models (including industrial systems) and to demonstrate the feasibility and high performance on in-the-wild data with the new paradigm (RQ2).
  • We also made huge efforts to give completely transparent and reproducible details of our open-sourced system presented in a paper clearly written and easy to follow as the reviewer has mentioned.

W2 Quality

• Glad that we were able to address most of your concerns including the correctness of E2 TTS reproduction, the fairness and comparability of the results, and claiming state-of-the-art performance in the conclusion may be an overstatement.

• We are willing to provide the result with CIs, under the reviewer's recommended evaluation setup: |F5-TTS|WER|SIM| |-|-|-| |16 NFE|1.958±0.077|0.667+0.005| |32 NFE|1.806±0.075|0.669+0.004|

• It is clear that WER 1.81 is fairly low, and we may have some concerns that:

  • If introducing a duration predictor would bring more significance as we have compared the results providing ground truth duration information, we may kindly ask what specific additional insights are expected as we are also delighted to see.
  • Given a WER lower than ground truth, the result is probably distorted given the current ASR model's transcription, while the SIM is maintained the same as we have already mentioned in our paper.

• Find the sample quality somewhat questionable compared to the reported table metrics.

  • We have mentioned that all code and checkpoints will be open-source. We ensure that all results are reproducible.
  • We are pleased that the reviewer has a high expectation of us (we have mentioned in the first response that no cherry-pick is made), and we would like to provide more information if necessary to prove an acceptable quality.

W3 Clarity

1. Could you also clarify how low the value of s can go (e.g., s = -1.5)? In section 3.2, we have mentioned $s ∈ [-1, 2/(π-2)]$.
2. We have removed the phrase "robustness" to avoid ambiguity. While we have the same observation with Seed-TTS that more "standardized" speech is easier for the ASR system to recognize, we believe in a more powerful ASR model in the future to better reflect zero-shot TTS models' nuanced differences in performance.
3. Many thanks for your advice. We have changed from 1 to 2:
   1. And unlike the way of feeding the model with inputs of significant length difference (length with effective information) as E2TTS does, our design mitigates such gap.
   2. Not directly feeding the model with inputs of significant length gap as E2 TTS does, the proposed text refinement mitigates the impact of using inputs with mismatched effective information lengths, despite equal physical length in magnitude as E2 TTS.
4. We fully agree on the essentialness of giving proper credit to all works that have paved the way for our research.
   1. VALL-E and Voicebox are cited for cross-sentence task.
   2. Voicebox is credited for the clarification and advocacy of putting SIM-o.
   3. Surely DiTTo-TTS is the first to explicitly propose "an additional long skip structure connecting the first to the last layer". We have properly referenced DiTTo-TTS with the method description, ensuring no ambiguity with the long skip connections proposed by U-Net.

Q2:

1. Whether entanglement is a structural limitation in both E2-TTS and F5-TTS.
2. Using cross-attention for text conditioning instead of adding filler-padded text to speech seems a straightforward and intuitive modification to reduce entanglement.

• The first part of Q2 is unchanged and we have already made a detailed response. We would appreciate it if the reviewer could clarify whether the comprehensive comparisons we made can confirm that F5-TTS is liberated from the entanglement (RQ1.2) rather than suffers together with E2 TTS.
• The second part of Q2 seems advanced with our text refinement method explicitly involved. From our perspective, DiTTo-TTS has conducted profound experimental analysis with model structure, in which the text encoder and probably joint finetuned codec play relatively important roles. Based on the experiences of DiTTo-TTS, it is hard to say our lightweight text refinement could replace the original text encoder and codec to achieve better performance with cross-attention than DiTTo-TTS. Moreover, in Response to W1 Orig. & Sign., we hope our point regarding the effectiveness and naturalness of using the in-context learning pattern has been clearly conveyed.

Q3: Why not compare the performance of two-stage control of CFG with single-stage one which seems essential for identifying the causes of improvement?

• We acknowledge that it was our oversight. We were assuming that the Voicebox's setting of 0.3 for $p_{drop}$ and a $p_{uncond}$ of 0.2 is commonly put as well as the 70% to 100% random mask. Ensuring closest reproduction, the 0.3/0.2 is the same with E2 TTS (following Voicebox).
• We have updated with referencing Voicebox to split between the configuration setting and our new terminology of two-stage control of CFG. Even though it was for misunderstanding and mainly our oversight, we would like to thank the reviewer again for careful reading and interest in it.
• We hope it is clear now that we are not avoiding answering this question and are definitely not claiming to propose this setting. For the abovementioned reasons, the two-stage CFG does not additionally benefit F5-TTS compared to Voicebox/E2 TTS.

Q5: Review-1: Need to explain why a pure DiT adaLN failed in contrast to E2-TTS to understand the effectiveness and limitations of ConvNeXt. Review-2: If skip connections help stabilize training in DiT but do not improve refinement with ConvNeXt (F5-TTS+LongSkip), what could be the underlying reason? ... would provide valuable insights for improving text-speech alignment in E2 TTS-like architectures.

• We hope the R1-Q5 has been addressed with our first response.
• R2-Q5 is a different but similar question regarding the model design, consistent with R1-Q5, concerning the effectiveness of ConvNeXt for improvement of text-speech alignment.
  • We assume that the reviewer is referring to a singular form skip connection as in F5-TTS+LongSkip. In the ablation study, this structure shows inferior performance to F5-TTS and is not more stable in training. From the comparison of F5-TTS, F5-TTS+LongSkip, and analysis in our response to R1-Q5, a conclusion that the text refinement ConvNeXt to DiT helps stabilize training makes more sense.
  • One of our main points in this paper is with RQ1.2 while the analysis of how ConvNeXt is refining padded text input (e.g. analyze the output of ConvNeXt, probe E2 TTS and F5-TTS transformer first layers) serves as our future research question.

W3 Clarity

1. As we have mentioned in the earlier response: the Sway Sampling function is monotonic with coefficient s∈[-1, 2/(π-2)], it is obvious mathematically that going out of the monotone interval of the given function is mapping a certain uniformly sampled u to negative values which are invalid for conditional flow matching. We hope it is clearly enough now for the reviewer concerning about the performance of a lower value s=-1.5.

2. We are always willing to address your concerns and thanks again for pointing out unclear points.

Q3: Based on the authors' response, it seems the issue was that the original phrasing made it appear as though a new method was being proposed. My understanding is that the explanation has now been updated to cite the existing method (presumably Voicebox's) instead. Could you confirm if this interpretation is correct?

As we have mentioned previously, we assumed that the setup of Voicebox is widely known, and the reviewer has not mentioned other hyperparameters in the same paragraph. The update is to explicitly cite Voicebox for the hyperparameter setup sentence before our new terminology in a one-sentence assumption. Voicebox takes $p_{drop}$ and 70% to 100% random mask together for infilling setup, and $p_{uncond}$ for CFG, while we describe in our paper $p_{drop}$ and $p_{uncond}$ with the new terminology "two-stage control of CFG". All these things belong to the "Experimental Setup" section where a new method is not introduced here.

Q5

The future research questions we mentioned in the previous response are quite different from the ablation of the model structure suggested by the reviewer. We believe the future research questions we mentioned are scientific and are to explore the essence.

\

Thanks again for the reviewer's rigorous comments. Although in some areas our perspectives differ, we fully respect your ideas.

As we have mentioned earlier, some other areas have not been fully discussed but are crucial for both our work and ensuring comprehensiveness of judgment. We would be delighted to receive the reviewer's feedback.

Thank you for the authors' kind and detailed responses. Below is my feedback.

W1 Orig. & Sign.

1. In my previous response, I omitted a paragraph while attempting to meet the character limit. This was clearly an error on my part, and I apologize if it caused the authors to perceive my review as incomprehensive. I want to emphasize that I had no intention of deliberately dismissing contributions that deserve recognition. My intent was to clarify that I recognize and agree with some of the contributions highlighted by the authors. To this end, I wrapped up my previous response with the statement, "I want to explicitly acknowledge that I have recognized the improved results and several key contributions of this work from my initial response." I hope the authors will take this clarification into account when interpreting my previous feedback.

   • I acknowledge that the proposed sampling method is well-explained and contributes to the final performance of F5-TTS, even though, as I noted in my initial review, the importance of sampling methods in diffusion is not a novel insight. However, since it addresses only one of the research questions, I am uncertain whether it is sufficient to establish the paper’s overall contribution. I also recognize the significance of open-source challenges within the TTS research community and appreciate the authors' efforts toward open-source contributions. Furthermore, as highlighted in RQ2, I value the authors' work in addressing underexplored areas such as in-the-wild data and code-switching, though multiple languages have already been investigated in works like Voicebox. Nonetheless, these contributions are of lower importance and priority compared to my concerns regarding the paper’s main contribution.

   At the time I submitted my last response, the announcement about the extension of the discussion period had not yet been made, and only limited time remained. I kindly ask for your understanding, as I was rushing to prepare my response, knowing that continuing the discussion with the authors within the available time was the best way I could assist them.

2. I followed the classification initially provided by the authors to rephrase and elaborate on my response, so I did not consider a citation necessary. However, it is now clear that the authors did not fully acknowledge the pre-existing nature of cross-attention, and I believe this requires correction.

   • Cross-attention: Broadly speaking, cross-attention has been a foundational component in conventional encoder-decoder TTS models, starting with the strong baseline Tacotron series [1, 2], followed by Transformer TTS [3]. More recent works include SPEAR-TTS [4], Simple-TTS [5], E3 TTS [6], CLaM-TTS [7], and T5-TTS [8]. Among concurrent works, Seed-TTS and DiTTo-TTS, mentioned by the authors, also utilize cross-attention.
   • Phoneme-level alignment: This is arguably the most conventional approach, first introduced by the FastSpeech series [9, 10], which used duration predictors based on forced alignment obtained from encoder-decoder teachers or tools like the Montreal Forced Aligner. Many subsequent works have adopted this monotonic alignment modeling for TTS tasks, including Glow-TTS [11] and its numerous variants such as VITS [12] and Your-TTS [13]. Additionally, large-scale architectures like Voicebox and the NaturalSpeech series [14, 15, 16] have continued to build on this approach.

   On the other hand, I was surprised to learn that the authors were unfamiliar with this classification. Given that the authors' work makes its main claims based on text conditioning (and speech-text alignment) with the distinctive E2 TTS framework, this is foundational knowledge that should have been addressed. It appears that the paper lacks a dedicated related work section. I strongly suggest including a separate paragraph to summarize text conditioning methods, incorporating the models I mentioned, to provide proper context and background for the proposed approach.

3. Thank you for providing a summary once again. The points you raised are consistent with the claims already made in your responses, without any notable differences. I will restate the key aspects based on my previous feedback:

   • As mentioned earlier, while I acknowledge that the authors "draw attention" to and "disclose" the issue, the analysis of its causes (e.g., the deep entanglement of acoustic and semantic features) is still insufficient.
   • Similarly, although the final performance comparisons and ablations aim to "prove" the effectiveness of the current approach, the justification lacks deeper insights into its validity. Experiments exploring the underlying causes, which are critical, remain insufficient.

   There is a lack of insight into the question, "What exactly does text refinement entail?" For further clarification on examples and remaining points, please refer to my response to point 1 above and my previous feedback.

References

[1] Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., ... & Saurous, R. A. (2017). Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135.

[2] Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., ... & Wu, Y. (2018, April). Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 4779-4783). IEEE.

[3] Li, N., Liu, S., Liu, Y., Zhao, S., & Liu, M. (2019, July). Neural speech synthesis with transformer network. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 6706-6713).

[4] Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., ... & Zeghidour, N. (2023). Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11, 1703-1718.

[5] Lovelace, J., Ray, S., Kim, K., Weinberger, K. Q., & Wu, F. (2023). Simple-TTS: End-to-End Text-to-Speech Synthesis with Latent Diffusion.

[6] Gao, Y., Morioka, N., Zhang, Y., & Chen, N. (2023, December). E3 tts: Easy end-to-end diffusion-based text to speech. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 1-8). IEEE.

[7] Kim, J., Lee, K., Chung, S., & Cho, J. (2024). CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech. arXiv preprint arXiv:2404.02781.

[8] Neekhara, P., Hussain, S., Ghosh, S., Li, J., Valle, R., Badlani, R., & Ginsburg, B. (2024). Improving robustness of llm-based speech synthesis by learning monotonic alignment. arXiv preprint arXiv:2406.17957.

[9] Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T. Y. (2019). Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32.

[10] Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T. Y. (2020). Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558.

[11] Kim, J., Kim, S., Kong, J., & Yoon, S. (2020). Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33, 8067-8077.

[12] Kim, J., Kong, J., & Son, J. (2021, July). Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning (pp. 5530-5540). PMLR.

[13] Casanova, E., Weber, J., Shulby, C. D., Junior, A. C., Gölge, E., & Ponti, M. A. (2022, June). Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning (pp. 2709-2720). PMLR.

[14] Tan, X., Chen, J., Liu, H., Cong, J., Zhang, C., Liu, Y., ... & Liu, T. Y. (2024). Naturalspeech: End-to-end text-to-speech synthesis with human-level quality. IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15] Shen, K., Ju, Z., Tan, X., Liu, Y., Leng, Y., He, L., ... & Bian, J. (2023). Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116.

[16] Ju, Z., Wang, Y., Shen, K., Tan, X., Xin, D., Yang, D., ... & Zhao, S. (2024). Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100.

Thank you to the authors for their response. Below are my comments.

W1 Orig. & Sign.

1. Thank you for providing a detailed explanation of the claims and rationale behind Sway Sampling. However, the reasons provided to support the novelty of the proposed method are not convincing to me for the following reasons:

   • It is well established that diffusion modeling allows the inference-time schedule to be modified independently of the training-time schedule (i.e., without retraining the model) [1]. Additionally, improving performance through highly interpretable solvers is not a novel concept [2]. Applying a different scheduler to E2 TTS essentially involves using an alternative existing scheduler, and the lack of performance improvement in such cases would be more surprising than its presence.
   • This scalability stems from the inherent properties of diffusion models, not from the proposed methodology itself. Therefore, the claim that the method is "easily applied to other research fields" could not substantiate its novelty. The idea that it would perform well in other domains is a promising hypothesis that needs to be validated through rigorous experimentation in the target domain.
   • The core idea of Sway Sampling lies in dividing diffusion time steps unevenly. However, the concept of defining and optimizing the noise schedule as a time-step optimization problem to improve performance has already been well established [3].
   • The ability to apply various samplers, including Euler, is not a unique advantage of the proposed Sway Sampling but rather a characteristic of FM ODE-solvers, to which Sway Sampling belongs [4].

   For these reasons, I feel that the proposed Sway Sampling demonstrates very limited scientific novelty, and the fact that it can improve final performance is not particularly groundbreaking (independent of the fact that the performance is better than E2 TTS). Separate from novelty, the only contribution I see is that it demonstrates, for the first time, the ability to improve the inference-time scheduler within the newly proposed TTS modeling framework of E2 TTS. However, I remain uncertain whether this finding is significant enough to be shared through ICLR, rather than a speech-specific conference, to reach a broader audience of researchers across various fields.

2. Thank you for the detailed point-to-point explanation.

   • My intent was not to critique the distinction between AR and Non-AR models but to highlight that the authors seem unaware of the older origins of the cross-attention-based textual conditioning method, as well as to point out the lack of a related work section addressing the core issues discussed in this paper.
   • My comment on unfamiliarity with classifications refers to the error of grouping two distinct text conditioning methods—Voicebox and E2 TTS—into the same category and misusing the term "in-context learning" to describe this category. E2 TTS represents a completely new approach, distinct from the two well-established categories (Cross-attention and Phoneme-level alignment) previously clarified in the last responses.
   • Regarding Seed-TTS, the authors previously stated in their response ("Replying to Response to Reviewer abUH [Cont. 1/2; W1,2]" starting with "To make it clear,") that "Seed-TTS, DiTTo-TTS, and E2 TTS are concurrent works." I would like to ask why Seed-TTS, which had not been previously discussed, was suddenly mentioned in that context. Based on the subsequent discussion, I understand it to be cited alongside DiTTo-TTS as an example of models using cross-attention, but clarification on this point would be appreciated.

I feel that the authors' responses, particularly the third point under item 2, deviate somewhat from the core of the discussion that should focus on the most concerned points. As a reviewer, I believe this paper falls short as a scientific paper, and I have raised critical questions that must be addressed to support its claims. To redirect the discussion appropriately, I would like to reiterate the key questions I have raised since the beginning, to refocus on the essential issues.

• What exactly is the definition of "text refinement" as claimed by the authors? By what mechanism does ConvNeXt address the robustness issues of E2 TTS? Is ConvNeXt the only viable solution?
• Why does the "entanglement of acoustic and semantic features in E2 TTS" occur? Could this be a fundamental issue stemming from the newly proposed text conditioning method in E2 TTS (combining speech and text without explicit alignment)?
• By what mechanism does Sway Sampling address the robustness issue?

Through prior discussions with the authors, it is clear that my concerns remain unresolved, and without satisfactory answers to address them, I find no reason to change my evaluation.

In any case, we deeply and sincerely appreciate the reviewer's multiple rounds of discussions. This rebuttal process has greatly enhanced various aspects of our paper and deepened our understanding of the model, which, in our view, embodies the true purpose of ICLR's open rebuttal system.

To summarize our discussion, we aim not only to address the rebuttal score and technique questions but also to provide a broader perspective on the innovation and contributions of our work.

We fully respect and acknowledge the impact of the E2 TTS work, which we have appropriately credited throughout the paper. At the same time, we believe our work represents a substantial step forward based on E2 TTS. Specifically, we leverage DiT with ConvNeXt as the backbone model to improve the robustness of speech-text alignment. Additionally, we proposed Sway Sampling, which significantly enhances the inference performance of the flow matching algorithm in TTS, enabling E2 TTS/F5-TTS to achieve impressive results on in-the-wild data. Furthermore, we will open-source the model, code, and the entire training and inference pipeline. We believe these contributions have the potential to result in a profound influence on TTS modeling.

This mirrors the historical progression of deep neural networks in speech recognition. Before 2011, DNNs demonstrated their effectiveness on small, clean datasets. Subsequent experiments on large-scale datasets like Switchboard and Fisher further convinced the community of deep learning's transformative potential, sparking widespread adoption and innovation in the field. We believe our work will have a similar impact by using transparent and reliable experiments to validate the effectiveness of a simple yet powerful model, facilitating new insights and approaches for TTS researchers.

In light of these contributions, we believe that F5-TTS has the potential to represent a significant advancement in the TTS field.

Q1: Sway Sampling performance on smaller s (e.g. -1, -0.9).

We use $s = -1$ and first-order Euler ode solver as the default values for F5-TTS in the updated main text. From Appendix B.3 Table 5, more aggressive Sway Sampling keeps improving performance and enables using the 1st-order Euler ODE solver that brings significant speed up compared to the 2nd-order midpoint.

Q2: Why ConvNeXt not improving E2-TTS?

As E2 TTS directly takes off phoneme-level force alignment while still keeping the UNet-equipped Transformer structure, it deeply entangles acoustic and semantic features (analyzed with experiment mentioned in response to W2). We speculate that E2 TTS is leveraging the whole network where all layers are skip-connected symmetrically to learn speech-text alignment. In contrast, F5-TTS is with an adaptive layernorm DiT structure free of skip connections in E2 TTS, which is not conflictive with a jointly learned refinement module.

Our work is putting forward that a modest refinement is sufficient for force-alignment-free NAR TTS rather than exploring sweepingly best DiT or UNet-Transformer variant, thus we leave some possible improvements for future works (e.g. asymmetric skip connections excluding the first few layers, a pre-learned refinement plug-and-play, etc.)

Q3: Why ConvNeXt harmful with audio but helpful for text in terms of WER?

With a fairly neat pipeline making the most of trainable parameters used, adding Conv2audio gains slight SIM while losing much WER. It makes sense because speaking style can be extracted with such a module to facilitate voice-cloning while the time series real information is eroded (to an excessive extent from experiment results, thus we state in the main text not ideal for scaling up). In contrast, the padded text sequence is not force-aligned, natural to benefit from a refinement operation.

Q4: Have you tried to observe the mel spectrogram in different sampling stages, to verify your assumptions?

Yes, we have observed the intermediate output mel spectrograms. We have also analyzed the trajectories of the whole sampling process of different samples and seeds. Due to the limited main text space, we decided to phrase with a clear and easy-to-grasp "leak and override" experiment that provides an intuitive explanation for all field researchers while no less concrete verification.

From $t=0.1$ where decoded samples are still of too high SNR for humans to hear the content or distinguish the corresponding mel spectrogram, the model already grasps the general form of mel (for $t>0.25$ we are able to see and hear the consistency between generated outputs in different stages, the later steps focus more on formed intermediate noisy output). The more successful override with Sway Sampling emphasizes that the early flow steps are crucial for sketching the silhouette of target speech based on given prompts faithfully.

Also, from the early training stage performance, when the model is still producing gibberish, leaking some ground truth information the same way as in the above experiment enabling intelligible output. This adds to the verification of our assumption for "later steps".

We sincerely appreciate your constructive feedback and spotting unclear points.

W1: Although this work demonstrates impressive performance, it appears to be primarily an incremental engineering exploration of E2-TTS, rather than a novel contribution.

We acknowledge that we have not come up with a breakthrough of backbone model innovation like Transformer in 2017, while we believe that our work has concrete contributions under the scope of "generative model" and more than simple incremental engineering effort on E2 TTS (which is very nice work simple and effective though in few first glance seems a duplicate of Voicebox just taking out the phoneme-level force-alignment). (Off-topic) we are not actually with the use of engineering as a rigid separation word in this age, as we (probably we all) are apt to see solid work with solid steps forward rather than flighty and boastful work.

With our sincere words put forward, we list the impact of our work:

• Not only prove and make transparent the simple and effective scheme of E2 TTS (we confirmed with strong results and with E2 author's check on the correctness of implementation), which liberating the upper limit of naturalness from previous phoneme force alignment, we also achieve commendable performance with relatively small model size on in-the-wild multilingual dataset compared to English-only models trained on high-quality audiobook corpus.
• To be specific:
  • Addressing slow convergence and deep entanglement of acoustic and semantic features of E2 TTS (which makes sense as a solid step forward to us),
  • trained with limited model size and uncensored data (more general usability with sentence-level duration provided zero-shot TTS paradigm),
  • our F5-TTS allows faster training, faster inference (a key factor for generative model RTF=0.15, fastest in comparison of Table 1, no specific optimization did yet),
  • and comparable or even better performance to leading TTS systems (one or several times the model size).
  • Not to mention that we are releasing all code, checkpoints, and every corner of details with our promising F5-TTS showing seamless multilingual code-switching and stuff to promote the whole community, which we believe are not trivial contributions.
• The inference-time Sway Sampling is a matter of great account which seems ignored from "major contribution". This training-free and easily applicable method greatly improves CFM-based models' performance and efficiency, which is highly desirable for generative models. Since CFM is one of the major trends, our Sway Sampling can help with other fields e.g. Video generation which also takes latency as critical.

W2: Lack of insightful analysis in the discussion section.

We would like to phrase two of our analyses here which are revealing for us and were not provided by related TTS works as far as we know:

• Sec.5.1 Investigation of the models’ behaviors with different input conditions to take a deeper look at the reasons behind the entanglement of acoustic and semantic features of E2 TTS (objective evaluation scores in Appendix B.2). It is clear that E2 TTS design faces the problem of deep entanglement of features (WER 9.63 to 3.48), and F5-TTS succeeds to address this (WER 4.17 to 3.22, a minor gain for more standard output when dropping audio prompt; and we are not applying Sway Sampling here to make fair comparison).
• Sec.5.2 Sway Sampling "leak and override" experiment to provide intuitive and concrete evidence of the effectiveness of our method (will elaborate with Q4).

Also, with R7LV6 Q1 and R-X8VX Q2 along with Appendix B.3 & B.4, we are willing to add more observations and analyses about our Sway Sampling strategy.

An analysis of training stability on varying data scales is further included in Appendix B.7 (R7LV6 Q2).

Again, we appreciate the reviewer's valuable comments. We have carefully addressed the main concerns in detail.

As the paper discussion phase is approaching an end and we truly value the discussion with the reviewer, we would be grateful to hear your feedback. We hope you would find the response satisfactory and we are always open to answering further questions.

Thanks for the reviewer's comments.

As we have already stated in our first response The inference-time Sway Sampling is a matter of great account which seems ignored from "major contribution". We would like to make the discussion if the reviewer has any concerns about the novelty of our proposed sampling strategy.

Furthermore, E2 TTS (which is very nice work simple and effective though in few first glance seems a duplicate of Voicebox just taking out the phoneme-level force-alignment) is our response which is different from the fragment taken out of context. What we are trying to express is that more complex things are not necessarily more valuable or effective, and doing subtraction is not easy and is not of less value.

The modest refinement is not mentioned by Reviewer abUH but text refinement (line235-243). The description modest is to make comparisons with complex or rigid text preprocessing designs (e.g. the way of DiTTo-TTS: pretrained language model as text encoder and with which jointly finetuned the codec; or the phoneme-level force-alignment). The usage of the term refinement is for our jointly learned module solely for the text modality before making concatenation with the speech modality.

We are willing to address your concerns (e.g. which specific question in the reviewer's first comments has not been fully addressed). Looking forward to the reviewer's feedback.

Thank you for the response.

However, the response contains some more unverified assumptions (e.g. what is "modest refinement" as mentioned by Reviewer abUH) over the assumptions in the manuscript itself.

Additionally, the response to W1 does not address the concerns regarding the novelty (to replace backbone, encode text with ConvNeXt, use a sway sampling). Given your remark that E2 seems a duplicate of Voicebox, you can understand why I view your work as merely a simple incremental improvement on E2.

I do not deny that it could be a good technique report (actually, further experiments in other domains are necessary to validate the proposed method), some techniques lack theoretical reasoning, concerning e.g. inconsistencies of ConvNeXt on F5-TTS and E2-TTS.