DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors

We sincerely appreciate your encouraging review and constructive feedback, and we hope our response thoroughly resolves your concerns.

[W1] Concern on novelty with acknowledgment of valuable analysis

Personally, I feel that the novelty is somewhat limited, as LDM-based structures in zero-shot TTS are not entirely new (e.g., NaturalSpeech 2, 3, as you already mentioned in your paper.), and the approach of allowing the model to learn speech-text alignment with cross-attention has been seen in previous works, such as Simple-TTS. Nonetheless, the systematic analysis of each component and the guidance provided to readers on existing design choices and their limitations are indeed valuable contributions.

We are grateful for your kind words and acknowledgment of the significance of our contributions. Regarding the novelty, as you rightly pointed out, LDM-based TTS systems are not entirely new, and we would like to clarify that we are not claiming to be the first to explore that. We also give full credit to previous works like Simple-TTS for demonstrating the possibility of LDM-based TTS without domain-specific factors. However, as you have kindly recognized, our main contribution is an experimental analysis of key aspects that enable such simple modeling to achieve state-of-the-art performance, validated by our final model's strong results, as detailed in our resaerch questions R1 and R2 of Section 5.

[Q1] Plans for code and checkpoint release of DiTTo-TTS

1. Are you plan to release the code and checkpoints of DiTTo-TTS?

We thank reviewer for interest in our open-sourcing plans. As mentioned in Section 9 (REPRODUCIBILITY STATEMENTS) of our paper, we plan to release the inference code, pre-trained weights, and eventually the full training implementation in phases, provided any legal issues are resolved.

[Q2] Robustness of DiTTo-TTS in low-data scenarios

2. Regarding the content in Appendix A.13, I am interested in whether the model performs well in smaller data scenarios, such as a single-speaker setup with less data than LibriTTS. The scale-up trend in performance relative to data quantity seems less pronounced compared to other zero-shot TTS models (e.g., Voicebox Section B.2). I would like to understand the authors' perspective on why DiTTo-TTS remains robust even with limited data.

We are grateful for your thoughtful feedback and for highlighting this important point. The performance improvement trend from 0.5K hours of data for DiTTo-libritts to 5.5K hours for DiTTo-libritts+mls-en-5k aligns with the trend observed in Voicebox from 0.6 to 6K hours (as discussed in Section B.2 of their paper), indicating similar patterns. To address your question, we need to evaluate our model under more restricted conditions, such as the 60 hours used by Voicebox.

We trained a DiTTo-libritts-0.05k model on approximately 50 hours of data, using the same settings described in Appendix A.13, Table 18, by randomly sampling around 10% of the LibriTTS dataset. Due to the limited data, we encountered overfitting and applied early stopping at 40K steps (instead of the planned 200K steps). The results, summarized in the table below, show significantly lower performance. The performance of models other than DiTTo-libritts-0.05k was directly taken from Table 18 in the paper.

Although these results do not allow for a direct comparison with Voicebox’s corresponding setting due to overfitting, we believe that incorporating phoneme-level durations during training, as done in Voicebox, enhances robustness when working with small datasets. This is implicitly supported by our observation in the second paragraph (3) of Appendix A.13, where our WER was worse than that of P-Flow, which incorporates phoneme-level durations, when trained on 0.5K data.

We include this explanation at the end of Appendix A.13 in the revised manuscript. We sincerely appreciate your suggestions, which have helped improve the quality of our work.

We are truly grateful for your thorough feedback and constructive input, which has helped refine the manuscript. Below, we provide detailed responses to each of your points.

[W1] Clarification of "domain-specific factors"

Dependency on Domain-specific factors DiTTo-TTS is claimed to avoid reliance on domain-specific factors like phonemes or durations; however it needs the pre-trained Text Encoder (ByT5, SpeechT5), which aligns the textual and acoustic information into a unified semantic space, and a pre-trained neural audio codec which aligns the speech latents with the linguistic latents of the pre-trained language model during the autoencoding process. Moreover, ..., Could you clarify your definition of "domain-specific factors" and explain how your approach differs from traditional TTS systems in terms of reliance on phonemes and durations, even with the use of pre-trained components?

We thank the reviewer for insightful feedback and for allowing us the opportunity to clarify our work. First, we would like to note that ByT5 is a text-only pre-trained model, and that DiTTo-TTS can be trained using a text-only pre-trained encoder alongside any neural audio codec without requiring semantic injection. The absence of phonemes and phoneme-level durations enables the effective utilization of pre-trained text encoders, similar to text-to-image models that use pre-trained text encoders and image autoencoders. This illustrates that latent diffusion model-based TTS can be implemented using components analogous to frameworks in other domains, such as text-to-image.

Moreover, improving speech-text alignment by enhancing pre-trained models for each modality is not a domain-specific methodology. We have found and empirically demonstrated that cross-attention training is enhanced when the alignment between the speech and text domains is strengthened, and this approach can be applied to any two domains. For example, methods like CLIP [1], which improve alignment between text and images, are not considered specific to the image domain. The Speech Length Predictor is one method for enabling variable-length modeling in diffusion processes and does not rely on phonemes or phoneme-level durations, unlike duration predictors in traditional TTS systems.

We define "domain-specific factors" as phonemes and phoneme-level durations, which traditionally require specialized linguistic preprocessing and detailed alignment in TTS systems. Our approach minimizes reliance on these factors, setting it apart from traditional TTS systems and offering a more scalable and efficient solution.

[1] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.

[W2] Comparison of DiTTo-TTS with Simple-TTS and NaturalSpeech 2

Limited Novelty of the Proposed Approach Simple-TTS and NaturalSpeech 2 has already proposed to , ..., Could you provide a more detailed comparison between DiTTo-TTS and Simple-TTS/NaturalSpeech 2 and highlight the key innovations or improvements in DiTTo-TTS that differentiate it from these prior works?

Thanks for your insightful comments and for the opportunity to clarify the distinctions between DiTTo-TTS and prior works like Simple-TTS and NaturalSpeech 2. Firstly, we would like to note that NaturalSpeech 2 does not utilize a pre-trained language model text encoder, as it relies on phoneme inputs. Additionally, NaturalSpeech 2 does not use cross-attention to align speech with text; instead, it uses phoneme-level durations to pre-align text representations with speech before applying them as conditioning inputs. Beyond this, NaturalSpeech 2 also requires pitch conditioning, which involves preprocessing steps to extract pitch information from the audio and depends on a third-party pitch extractor to obtain ground truth pitch values. In other words, traditional powerful TTS models require such TTS-specific factors, which increase model complexity and preprocessing costs, ultimately hindering scalability.

Simple-TTS established that LDM-based TTS could be achieved without domain-specific factors, marking it as a pioneering effort in this emerging paradigm. However, it lagged significantly behind state-of-the-art performance, lacking insights into the causes of this gap. Our main contribution is an experimental analysis of key aspects that enable such simple modeling to achieve state-of-the-art performance, validated by our final model's strong results, as detailed in our resaerch questions R1 and R2 of Section 5. Notable contributions include methodological advancements such as the semantic injection of Mel-VAE++, a speech length predictor, and architectural ablations involving long-skip connections, offering actionable guidance for optimizing the promising modeling paradigm.

We appreciate the valuable feedback you have provided, which has helped us refine our work. With the discussion period now extended, we hope this provides an opportunity to understand whether our previous responses have adequately addressed the concerns you raised and to engage with any remaining questions or feedback. We deeply value your insights and are happy to provide further clarification or additional details to assist in your evaluation. Thank you for your engagement.

We do appreciate the reviewer's positive feedback and constructive suggestions to further improve our paper. We hope our response thoroughly addresses your questions.

Before we begin, we would like to clarify that the speech length predictor is trained with causal masking but generates the total length in a single forward pass during inference, making it non-autoregressive. Causal masking during training ensures that the predictor calculates the total length based solely on the given speech prompt, as described in the Speech Length Predictor paragraph of Section 3.2. Furthermore, the predictor is designed to simultaneously learn the remaining lengths for all positions within an audio sample, eliminating constraints on prompt length and enhancing training efficiency. This clarification has been added to the Inference paragraph in Appendix A.7 of the revised manuscript.

[Q1,Q2] Table 7: fine-tuning details & adding SIM-o results

• Table 7:
  • Are the DiTTo-Encodec and DiTTo-DAC models also fine-tuned with an LM objective? If not, please include DiTTo-MELVAE for comparison.
  • Please report sim-o when comparing different neural codecs.
• In general, reporting sim-o for other ablations would be beneficial for the community to facilitate comparisons with any results in the paper.

We thank the reviewer for insightful comments and valuable suggestions. We would like to clarify that both DiTTo-encodec and DiTTo-dac models In Table 7 are not fine-tuned with the LM objective. Also, DiTTo-mls (DiTTo-MELVAE) model is not fine-tuned with the LM objective as well.

Initially, we observed that SIM-o and SIM-r followed similar trends in Tables 1 and 2, which led us to use SIM-r as a representative measure in subsequent ablation studies. As per your great suggestion, we have included SIM-o results in Table 7 of the revised manuscript as shown below (only codec-related results are included here).

DiTTo-dac-24k shows slightly higher SIM-o compared to DiTTo-mls. However, our preliminary experiments revealed a trade-off between WER and SIM, as shown in Table 22 of the revised manuscript. Specifically, reducing the scale factor from 0.3 to 0.2 resulted in a WER improvement from 7.21 to 4.70 (though still suboptimal), while SIM-o decreased from 0.5478 to 0.5302. We also include the discussion on this in the Appendix A.19 of the revised manuscript.

We sincerely appreciate your constructive feedback and thoughtful suggestions, which have been invaluable in improving the manuscript. Below, we address your comments point by point.

[W1] Regarding contributions and the novelty of the work

The major weakness of this paper is the novelty. The most contribution as the author also mentioned is removing the domain-specific factors like phoneme and duration, and just giving a total duration estimation is good enough for the task which is even better. However, this is not new, and similar work has been done at SeedTTS https://arxiv.org/pdf/2406.02430#page=6&zoom=100,144,548.

SeedTTS-DiT has no-dependence on the phoneme duration by just giving the total duration for the diffusion model. It already shows that this fully diffusion based speech generation model could achieve superior performance. This integration is just a quite natural way regarding the highly flexibility of noise iteration with DiT structure for a diffusion or flowmatching model. I would believe many folks/peers have applied it in speech generation before or after Seed-TTS-DiT and it indeed works as expected. According to these information, I would give the question on the novelty of this paper.

We thank the reviewer for thoughtful feedback regarding the novelty of our work. We would like to respectfully argue that the removal of domain-specific factors such as phonemes and duration is not positioned as our main contribution. As noted in the paper, we acknowledge prior pioneering works (Simple-TTS and E3 TTS) that have explored these ideas and highlight their limitations to contextualize our study.

We also acknowledge the concurrent work Seed-TTS, which demonstrates impressive performance with a simple modeling approach. However, Seed-TTS does not provide a comprehensive comparison with diverse baselines (RQ1) and the critical factors that enable LDMs to succeed without domain-specific factors (RQ2), which are the central focus of our study as discussed in Section 5:

• RQ1: Can LDM achieve state-of-the-art performance in text-to-speech tasks at scale without relying on domain-specific factors, similar to successes in other domains?
• RQ2: If LDM can achieve this, what are the primary aspects that enable its success?

Our primary contribution lies in systematically addressing these questions through extensive empirical investigations. We demonstrate that LDM-based TTS, even without domain-specific factors, can achieve performance that is superior or comparable to various state-of-the-art autoregressive (AR) and non-autoregressive (Non-AR) TTS baselines, enabling a fair evaluation of its performance within the broader research landscape. Additionally, we identify the key factors driving this success and provide practical insights into a new modeling paradigm for TTS systems. Notable contributions include methodological advancements such as the semantic injection of Mel-VAE++, a speech length predictor, and architectural ablations involving long-skip connections, offering actionable guidance for optimizing LDM-based TTS systems. While the technical novelty in model architecture design may appear incremental, we believe our work makes a significant contribution to advancing the understanding of this emerging paradigm and shaping the future direction of TTS modeling.

We have incorporated this discussion into Appendix A.1 under Related Work in the revised manuscript and sincerely thank you for your valuable contribution to enhancing the clarity and quality of our paper.

Thanks the authors' fruitful discussion and additional experiment for the clarify.

For the Q1, it is quite encouraging to extend more experiments to explore the capability of diffusion model for speech generation. Is fully diffusion model for speech generation better or comparable as LLM? and each model's advantages and limitations for now or then.

For the Q2, I think this is very interesting result. As the direct thought would be modeling mel, however, the result shows even with mel as intermediate feature for vocoder, it is still useful to adopt latent representation for LDM model and decoder as mel then. Have you conducted any subjective test on the quality and similarity to valid the table objective result? It would be quite helpful if these result include in the paper as well.

At last, I would change the total rating score from 5 -> 6 according to authors detail reply especially for the contribution and primary goal of the paper, but still keep the each item score especially for the novelty.