DMDSpeech: Distilled Diffusion Model Surpassing The Teacher in Zero-shot Speech Synthesis via Direct Metric Optimization

We thank the reviewer for acknowledging the strengths of our work, including its clear presentation and comprehensive experiments. These aspects underscore the rigor and reproducibility of our approach. Below, we address the concerns raised regarding novelty and comparisons.

Response to Novelty Concerns

While we agree that individual components like distillation and metric optimization have been explored before, we respectfully emphasize several key innovations in our work:

Regarding Diffusion Model Distillation in TTS: Previous distillation approaches in audio synthesis consistently showed performance degradation compared to their teacher models. For instance, the work cited by the reviewer [2] reported a MOS decline from 4.136 (teacher) to 3.902 (student). In these studies, the gains only focus on improving the inference speed, not performance. In contrast, our work demonstrates that, through direct metric optimization applied on a distilled diffusion model, it is possible to not only accelerate inference but also to surpass the teacher model in both quality (MOS) and speaker similarity, which has not been demonstrated by previous TTS distillation approaches.

In Terms of Metric Optimization:

1. True End-to-End Metric Optimization: While prior works like YourTTS used speaker embedding similarity loss, their approach was fundamentally limited by non-differentiable components (e.g., monotonic alignment search) that prevented gradient flow to crucial components like the text encoder and duration predictor. Our framework enables true end-to-end optimization across all components, resulting in substantial improvements in both objective metrics (SIM, WER) and human evaluations (SMOS-V, MOS). On the other hand, YourTTS did not present consistent and significant improvements over speaker similarity with their proposed SCL loss, suggesting that non-end-to-end optimization is less effective.

2. General Framework for Metric Optimization: Moreover, our framework is more general than just SV loss, as it enables optimization of any differentiable metric. The successful integration of CTC loss demonstrates this generality, something not previously achieved in speech synthesis models due to architectural limitations. This is likely because previous methods without iterative sampling (such as YourTTS) cannot optimize the text encoder and duration predictor simultaneously with the decoder for the CTC loss, components critical for natural and intelligible speech. On the other hand, our use of a distilled diffusion model enables this end-to-end optimization approach, which has proven effective in enhancing both speaker similarity (SIM) and intelligibility (WER) measures, as well as subjective metrics such as SMOS and MOS by human raters.

Response to Concerns Over “Direct Metric Optimization” Claims

We understand the reviewer’s perspective on the indirect nature of metrics like SIM and WER. While these metrics do serve as proxies, they are nonetheless strongly correlated with human judgment, as supported by our subjective evaluations (from 290 raters) and correlations demonstrated in Figure 5. Furthermore, we observed that optimizing SIM and CTC loss improves corresponding human-rated metrics, such as SMOS and MOS, as shown in Table 4. We also encourage the reviewer to listen to the effect of direct metric optimization on our demo page: https://dmdspeech.github.io/demo/#ablation. Since we explicitly state that we are optimizing evaluation metrics rather than directly optimizing for subjective human preference, we believe our description is appropriately cautious and accurate.

Moreover, the essence of our direct metric optimization lies in making previously non-optimizable metrics, such as speaker similarity and WER, directly optimizable within an end-to-end framework. This advantage over existing TTS models is meaningful, resulting in practical improvements in quality and alignment with human perception. Hence, we believe that our experiment designs are fair, as our method revolves around the advantages and gains of directly optimizing evaluation metrics.

Additional Clarifications

In line 412, could you elaborate on the statement, "Table 5 shows that our model achieved the highest speaker similarity score (SIM) to the prompt, even surpassing the ground truth"? It is unclear how Table 5 demonstrates that DMDSpeech exceeds the ground truth.

Thank you for catching the table reference error. This was indeed a typo. We meant to reference Table 3, not Table 5, regarding our model achieving the highest SIM score. We will correct this in the revision.

Does your training dataset exclude the official samples from the baseline models which are used for evaluation?

Yes, our training dataset (LibriLight) contains no speakers and utterances from the LibriSpeech test dataset. As stated by Kahn et al. (2020) in the LibriLight dataset paper: "We then removed corrupted files, files with unknown or multiple speakers, and speakers appearing in LibriSpeech dev and test sets."

Could you clarify why you chose to fine-tune the CTC-based ASR model to derive the latent SV model? From your explanation in Appendix C.4, you use a distillation approach to align the CTC-based ASR model's embeddings with the concatenated embeddings from the WeSpeaker's ResNet-based SV model and EPACA-TDNN with a fine-tuned WavLM Large model. Wouldn't it be more efficient to utilize the WeSpeaker's ResNet-based SV model and EPACA-TDNN with a fine-tuned WavLM Large model to directly extract the concatenated embeddings of the prompt and the concatenated embeddings of the generated speech, then calculate the SV loss in Equation 14 using these two embeddings? Can you provide experiment to evaluate that the latent SV model fine-tuned from CTC-based ASR model performs better in speaker similarity?

We appreciate this insightful question. Our work follows recent developments in diffusion-based TTS models that use the latent diffusion framework, which operates on latent representations rather than waveforms. Since both the WeSpeaker ResNet-based SV model and EPACA-TDNN with WavLM Large were trained on waveform inputs, they cannot be directly applied to our latent representations. Converting these latents back to waveforms during training would be both inefficient and computationally intractable due to GPU memory constraints. Therefore, we distilled these waveform-based SV models into latent SV models for both efficiency and applicability. We will clarify this technical constraint in our revision.

Reference:

Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., X*u, Q., Mazaré, P. E., ... & Dupoux, E. (2020, May). Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7669-7673). IEEE.

We appreciate the reviewer’s feedback for recognizing the strong results and advantages of DMDSpeech, including its superior performance in naturalness, speaker similarity, and inference speed. We appreciate the reviewer’s feedback and questions. Below, we address each point with clarifications and additional details.

Response to Concerns about Limited Novelty

We agree that DMD 2 is an important component of our work, but we respectfully clarify that it is not our only contribution. While we do utilize DMD 2, it serves as a foundational component to achieve our primary objective: creating an efficient TTS model with high-quality output and fast inference.

Our primary innovation lies in combining DMD 2 with direct metric optimization to achieve both faster inference and better performance than existing approaches. As demonstrated in Table 4, applying DMD 2 alone does not improve performance beyond the teacher model except for inference speed. Significant improvements in naturalness and speaker similarity have been made in our novel application of direct metric optimization. We encourage the reviewer to listen to the audio samples on our demo page (https://dmdspeech.github.io/demo/#ablation) to hear the clear quality improvements achieved through direct metric optimization.

Our methodological contribution is significant because:

1. We enable end-to-end direct metric optimization for TTS, which was not previously possible due to non-differentiable components (e.g., duration predictor or monotonic alignment search) or iterative sampling (e.g., autoregressive models or diffusion models) in existing models. By distilling the diffusion model without non-differentiable components into a single-step generator, we overcome these obstacles, making it possible to directly optimize differentiable metrics.

2. We demonstrate that this approach can surpass teacher model performance in both speed and quality.

3. Our framework is generalizable: as evaluation metrics continue to evolve, DMDSpeech’s approach allows for future improvements in human preference alignment. Therefore, we believe our work offers a novel methodology and significant advancement in TTS, as acknowledged by Reviewer i2SD.

Response to Request for Additional Experiments

We appreciate the reviewer’s suggested experiments but wish to clarify a fundamental aspect of diffusion models that limits the possibility of these suggested experiments: Diffusion-based models do not produce the final output directly; instead, they compute intermediate “score” (added noise) estimates at each step of the denoising process. While theoretically possible, retrieving the final output from a high-noise level yields unintelligible audio, making direct metric optimization infeasible.

To illustrate this limitation, we have added samples at various noise levels to our demo page: https://dmdspeech.github.io/demo/#rebuttal. These samples demonstrate why applying direct metric optimization to the teacher model without iterative sampling (which would require tens to hundreds of steps) is impractical. The speech is unintelligible and speaker identity is unclear at higher noise levels, making it impossible to derive useful gradients from SV or CTC losses. While it is possible to sample through multiple steps to generate the final output, backpropagating across multiple steps in a large model also leads to instability and high computational costs, making the suggested experiments impractical.

Thus, while we value the input, we would like the reviewer to understand that the experiments proposed do not align with the operational structure of diffusion-based models in TTS.

We sincerely appreciate your review and valuable feedback on our paper. We are pleased that you found our work technically sound and well-presented, with sufficient implementation details and experiments showcasing the advantages of our approach. We would like to clarify the novelty and distinct contributions of our approach:

1. Direct Metric Optimization in TTS: Our work is the first to enable direct, end-to-end optimization of key metrics such as speaker similarity (SIM) and word error rate (WER) in zero-shot TTS. Traditional models like YourTTS and NaturalSpeech 2 & 3 cannot achieve this due to non-differentiable components (e.g., monotonic alignment search, duration predictors) that block gradient flow, making optimization of text encoder and duration predictor for metrics such as WER with CTC loss ineffective.

2. Challenges in Iterative Sampling Models: Recent approaches (e.g., Vall-E, DiTTo-TTS) rely on iterative sampling, requiring up to 100+ steps in autoregressive models or 16 steps in flow-matching diffusion models. Backpropagation through these steps is computationally prohibitive and unstable.

3. Our Solution: By distilling the diffusion model into a single-step generator, we overcome these limitations, enabling efficient and stable optimization of differentiable metrics. This results in consistent improvements over the teacher model across MOS, SMOS, WER, and SIM. Our originality and novel contribution are also acknowledged by Reviewer i2SD.

We hope this clarifies the meaningful advancements of our work. We also invite any further questions or specific examples of comparable approaches. Thank you for your valuable feedback.

We sincerely appreciate the reviewer's continued engagement and feedback on our work. Below, we address your follow-up concerns and provide further clarification.

Response to "Direct Metric Optimization" as a Straightforward Idea

As far as we know, no prior TTS paper has implemented true end-to-end direct metric optimization with significant improvements across key metrics such as SIM, WER, and MOS. For instance:

• YourTTS included a speaker similarity loss (SCL) but explicitly reported minimal impact, stating: According to the Sim-MOS, the use of SCL did not bring any improvements; however, the confidence intervals of all experiments overlap, making this analysis inconclusive.
• Metrics like WER require optimization of the text encoder and duration predictor simultaneously, which is infeasible in prior TTS frameworks with non-differentiable components such as monotonic alignment search (MAS).

Our method overcomes these challenges by enabling true end-to-end optimization for metrics like WER through CTC loss, a novel contribution in the speech synthesis domain. To our knowledge, this is the first approach that achieves such significant and consistent improvements, which we kindly ask the reviewer to acknowledge or provide references to similar works, if they exist.

Response to "DMD Motivation Similar to Other Distillation Approaches"

We respectfully disagree that the application of DMD to speech synthesis is trivial. Speech synthesis presents unique challenges compared to the image domain, as it requires strong conditional generation rather than the weak conditional generation typical in image synthesis. This fundamental difference is analyzed in detail in Section 3.4. We have also provided new insights into mode shrinkage in strong conditional generation and its potential benefit and drawback in Section 4.4 and Appendix A, a phenomenon not previously explored in speech synthesis with diffusion models. These contributions have also been acknowledged by Reviewer i2SD.

Furthermore, we emphasize that while prior distillation approaches like progressive or consistency distillation focused solely on improving inference speed, our method achieves both speed improvements and performance gains over the teacher model, something not previously reported in the TTS field. This makes our motivation completely different from previous works.

Response to "No Fundamentally New Ideas"

We respectfully contend that our contributions are both novel and impactful:

1. End-to-End Direct Metric Optimization: No previous method has successfully achieved this for metrics like WER or SIM, which depend on optimizing all components used for speech synthesis, and our optimization of WER using CTC loss is the first of its kind. Our approach outperforms state-of-the-art models and even the teacher model, demonstrating the practical utility of this innovation.

2. General Framework: Our method is applicable to any different metric, including MOSNet, and can adapt as evaluation metrics continue to align more closely with human preferences. By contrast, the only prior works that optimize over MOS rely on complex multi-step sampling procedures with reinforcement learning [1,2] to approximate human preference alignment using pre-trained neural networks such as MOSNet, whereas our approach can achieve similar goals with a simpler, more generalizable solution through direct optimization.

Finally, we would like to invite the AC and reviewers to consider this key question: If our method were as straightforward as suggested, why has no prior work proposed and successfully implemented it, especially given the demonstrated challenges of aligning generative models with human preferences and our framework's incredible effectiveness as acknowledged by all reviewers? We hope the AC and reviewers consider this question when evaluating the novelty and impact of our contributions.

We again thank the reviewer for their comments and hope these clarifications will assist in understanding the meaningful advancements presented in our work.

References:

[1] Chen, C., Hu, Y., Wu, W., Wang, H., Chng, E. S., & Zhang, C. (2024). Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback. arXiv preprint arXiv:2406.00654.

[2] Chen, J., Byun, J. S., Elsner, M., & Perrault, A. (2024). Reinforcement Learning for Fine-tuning Text-to-speech Diffusion Models. arXiv preprint arXiv:2405.14632.

We thank the reviewer for appreciating our work, acknowledging its originality and novelty, clarity in presentation and significance it provides to the industry and relevant research. Below are our responses to your questions.

1. For the subtitle of Fig 1., we will make sure all four subfigures have the same location in the subtitle. We will make this change in the revised submission.

2. The reason we chose 4 steps is the original DMD 2 paper uses 4 steps, and coincidentally this is also the minimal step number for the state of the art performance. We agree that any step number lower than the teacher can improve the inference speed, but from our experiments we found 4 steps to be the minimum required to obtain the best performance. We will add a discussion about this in the revised version of our paper.