DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis

Thank you for your thoughtful review of our paper. We appreciate your feedback and address your concerns below:

Correlation Between Subjective and Objective Metrics

We have indeed analyzed the correlation between subjective and objective metrics in our paper. As shown in Figure 3 and Figure 5 (appendix), the Pearson correlation between speaker embedding similarity (SIM) and human-rated voice similarity is 0.55, while the correlation with style similarity is 0.50. Similarly, word error rate (WER) correlates with naturalness and sound quality at -0.16 for both metrics. All correlations are statistically significant ($p << 0.01$), demonstrating that our optimized objective metrics strongly align with human perception even at the individual utterance level.

Applicability Beyond Speech Synthesis

While our implementation focuses on speech synthesis, the core framework of enabling direct metric optimization through distillation can be applied to other generative domains. For example:

• In music generation, differentiable models like instrument detection models or melody extraction models could optimize text-to-music alignment or ensure generated music matches specified instruments or melodic (MIDI input) constraints.

• In image generation, differentiable models could maxizmize CLIP scores between the prompt text and generated image, or verify the presence of all text-described objects using image segmentation models

• In video generation, similar principles could ensure temporal consistency

The key innovation is creating a direct gradient pathway that enables end-to-end optimization with any differentiable metric within the diffusion model frameworks, which has broad applications beyond speech synthesis.

Comparison with StyleTTS-ZS

We have conducted objective evaluations of StyleTTS-ZS that will be included in the revised manuscript. DMOSpeech significantly outperforms StyleTTS-ZS in speaker similarity (0.69 vs. 0.56) with comparable real-time factor (0.07 vs. 0.04). While StyleTTS-ZS achieves lower WER (1.17 vs. our 1.94), our model delivers better overall performance as confirmed by subjective evaluations in Table 1. Additionally, our training pipeline is more straightforward without requiring aligner training, making it easier to scale across languages and larger datasets.

Parameter Selection Process

We have detailed our parameter selection approach in the paper. The process is intuitive rather than difficult—we observe gradient norms for each loss term and balance them accordingly. As described in Section 3.4:

" We set $\lambda_{\text{adv}} = 10^{-3}$ to ensure the gradient norm of adversarial loss is comparable to that of DMD loss. During early training stage, we observed that the gradient norms of SV loss and CTC loss were significantly higher than DMD loss, likely because $G_\theta$ was still learning to generate intelligible speech from single step. To address this, we set $\lambda_{\text{CTC}} = 0$ for the first 5,000 iterations and $\lambda_{\text{SV}} = 0$ for the first 10,000 iterations. This allows $G_\theta$ to stabilize under the influence of DMD loss before integrating these additional losses. After that, both $\lambda_{\text{CTC}}$ and $\lambda_{\text{SV}}$ are set to 1."

This approach follows established practices in the literature (Yin et. al, 2024) and doesn't require extensive hyperparameter tuning.

Regarding Other Suggestions

1. The description in Section 3.2 should be improved. Most of the contents are not new, so it is unclear which parts are novel.

Section 3.2 provides necessary background on Distribution Matching Distillation, establishing context for our improvements. We acknowledge this is primarily background material and will clarify which aspects represent our specific contributions in the revised manuscript.

2. The MOS values of various methods in Table 1 are almost the same as the ground truth. I am not sure whether the improvement obtained by the proposed method is meaningful enough.

While MOS values are similar to ground truth, our key contribution is achieving comprehensive improvements across multiple metrics simultaneously. Previous models like StyleTTS-ZS achieve high naturalness but lower similarity, while NaturalSpeech 3 achieves high similarity but lower naturalness. DMOSpeech uniquely excels in both dimensions while maintaining significantly faster inference speed (13.7x faster than the teacher model). This balanced performance across all metrics represents a meaningful advancement in the field. We appreciate your constructive feedback and will incorporate these clarifications in our revised manuscript.

We sincerely thank the reviewer for their thorough evaluation and constructive feedback. Below, we address each point raised:

StyleTTS-ZS Comparison

The performance of StyleTTS-ZS in our evaluation aligns with what was reported in their original paper. The fundamental architectural difference is that StyleTTS-ZS employs a specialized decomposition approach, modeling specific prosody components (F0, energy, duration) separately, while DMOSpeech adopts a more holistic end-to-end generation framework.

Our approach offers several advantages:

1. Greater generalizability across diverse speech conditions (such as non-speech vocalizations and multiple speakers in the same utterance).
2. Support for end-to-end optimization with perceptual metrics
3. Robustness to challenging audio conditions (e.g., background noise, processing artifacts, as we presented on our demo page)

StyleTTS-ZS achieves high efficiency through its decomposition strategy (enabling one-step generation, since generating prosodic features only is easier than generating the whole speech), but this same design introduces limitations when faced with complex or noisy audio conditions. DMOSpeech maintains comparable efficiency while providing superior generalizability and performance.

Value of Super-Efficient Generation

While the teacher model achieves real-time generation (RTF < 1) on a high-end GPU (V100), there are compelling reasons to pursue even greater efficiency:

• Device Compatibility: Enabling deployment on resource-constrained environments (CPUs, mobile devices, edge computing)

• Service Scalability: A 13.7× reduction in inference time translates to substantially higher throughput for cloud services supporting millions of users

• Energy Efficiency: Reduced computation requirements lead to lower power consumption and carbon footprint

For industrial applications, these efficiency gains are critical for accessibility, scalability, and sustainability.

Regarding model scaling, while extremely large models (10B-100B parameters) are indeed being explored in production environments (e.g., by ByteDance's Seed-TTS and Amazon's Base-TTS), analyzing scaling behaviors was outside our paper's scope. Our distillation technique remains relevant regardless of teacher model size, as the efficiency benefits become even more pronounced with larger models.

Ablation of Individual Losses

As shown in Table 4, we conducted comprehensive ablation studies applying CTC and SV losses individually:

• CTC Loss Only: Achieved superior word error rate (1.79 vs. 1.94) but significantly lower speaker similarity

• SV Loss Only: Produced slightly higher speaker similarity (0.70 vs. 0.69) but substantially worse WER (6.62 vs. 1.94)

These results demonstrate that combining both losses achieves the optimal balance between intelligibility and speaker similarity, which aligns with human preference as shown in our subjective evaluations.

Loss Optimization and Potential Trade-offs

The reviewer raises an important point about potential conflicts between optimization objectives. In theory, overly aggressive optimization of auxiliary losses (CTC, SV) could indeed harm performance by causing distribution mismatches with the training data, especially when the loss is optimized below that achieved in ground truth data.

To mitigate this risk, we carefully balanced the gradient contributions from each loss component, ensuring the auxiliary loss gradients are comparable in magnitude to the primary DMD loss. This calibrated approach prevents any single objective from dominating and maintains distributional alignment throughout training.

Regarding CLAP regularization, while it's an intriguing direction, we focused specifically on speech synthesis rather than general audio generation in this work. We appreciate the suggestion and will mention this as a potential future direction in our revised manuscript.

Theoretical Contributions

While our paper emphasizes empirical results, we provide some theoretical contributions, especially our Analysis of Mode Shrinkage. Our detailed examination of distributional changes during distillation (Figure 2 and Appendix A) offers novel insights into how distillation affects output diversity in conditional generation tasks. Our analysis shows that although distillation causes a loss in diversity for a fixed prompt and text input, this reduction in diversity is not necessarily negative or could even be beneficial to the performance of conditional generation, and the mode coverage is not compromised when the model processed different prompt and text inputs.

These insights contribute to the theoretical understanding of both diffusion model distillation and perceptual metric optimization in generative models.

We thank the reviewer again for their thoughtful comments and positive assessment. Hope our responses address their concerns satisfactorily.

We appreciate the reviewer's feedback, but we believe there is a fundamental misunderstanding about our paper's contribution. Our work presents several key innovations that have not been explored in prior research, including FlashSpeech:

Clarification on Direct Metric Optimization vs. Adversarial Training

The reviewer conflates direct metric optimization with adversarial (GAN) training, which are fundamentally different approaches: FlashSpeech does not implement direct metric optimization. After careful examination of the FlashSpeech (Ye et. al 2024) paper, we confirm it uses adversarial consistency training but makes no mention of direct metric optimization of perceptual metrics such as speaker similarity or word error rate. Their adversarial training is solely for improving general speech quality.

Our direct metric optimization is novel. DMOSpeech enables true end-to-end optimization of specific perceptual metrics (SV loss for speaker similarity, CTC loss for word error rate) through differentiable pathways - not simply adversarial training. This is a significant advancement for TTS systems.

State of Direct Metric Optimization in TTS

The reviewer suggests that "both diffusion distillation and direct metric optimization in TTS are well studied." This is incorrect. As we explicitly state in our paper:

While optimizing perceptual metrics has shown promise in speech enhancement through approaches like MetricGAN for PESQ and STOI, and recent attempts have explored RLHF for improving naturalness, implementing these approaches in modern TTS systems has remained challenging. Previous attempts (e.g., YourTTS) reported minimal improvements from speaker similarity optimization due to their inability to propagate gradients through all model components. The field has struggled with this problem due to architectural limitations such as non-differentiable components or computationally prohibitive backpropagation through iterative sampling steps.

Relationship to DIFFUSION-GAN

Our work bears limited similarity to DIFFUSION-GAN. Our focus is not on GAN-based techniques but on enabling direct optimization of perceptually relevant metrics through a differentiable pathway created by one-step generation through distillation. This is a fundamentally different approach with different goals.

Novel Contributions

Our paper makes several novel contributions:

1. We present the first distilled TTS model that consistently outperforms its teacher model (not merely matching it), while reducing inference time by over 13×.

2. We introduce a framework enabling true end-to-end optimization of differentiable metrics in TTS, demonstrating substantial improvements in speaker similarity and word error rate.

3. We provide comprehensive analyses establishing correlations between objective metrics and human perceptions, revealing new insights into sampling speed and diversity trade-offs.

These contributions represent significant advancements in the field of speech synthesis that are well-aligned with ICML's focus on machine learning innovations.

Regarding F5TTS and MASKGCT Comparisons

We thank the reviewer for their suggestions for more baseline comparisons. Per the reviewer's suggestion, we have trained a new DMOSpeech model using F5-TTS as the teacher on the Emilia dataset and conducted comprehensive objective evaluations comparing our new model against both F5TTS and MASKGCT models on the SeedTTS-Eval benchmark dataset:

Our model has achieved similar or better performance than both MaskGCT and F5-TTS on SeedTTS-eval in both Chinese and English test sets for both intelligibility and similarity. Moreover, our model is significantly faster than both MaskGCT and F5-TTS.

We will include the complete evaluation results in the appendix of our revised manuscript. This additional analysis provides a more comprehensive understanding of how our approach compares to current state-of-the-art methods.

Regarding FlashSpeech Comparison

We appreciate the suggestion to compare with FlashSpeech. However, despite our best efforts, we have been unable to conduct direct experimental comparisons due to the unavailability of publicly accessible pre-trained checkpoints for this model, as documented in https://github.com/zhenye234/FlashSpeech/issues/3.

In our revised manuscript, we will include a thorough discussion of FlashSpeech, addressing its approach and how it relates to our work. While we cannot provide direct experimental comparisons, this discussion will help contextualize our contributions among other efficient zero-shot TTS systems.

We sincerely thank the reviewer for their positive recommendation for our paper. We appreciate the recognition of our model's ability to enable direct gradient pathways for end-to-end optimization and the acknowledgment of our efficiency improvements. While we agree with many points raised in the review, we would like to address two specific concerns:

On Theoretical Claims and Scientific Contribution:

We respectfully disagree with the assessment that our theoretical claims are "not significant, novel" and that there is "no significant contribution to the broader scientific literature." Our work makes several important theoretical and scientific contributions:

• Novel Distribution Matching Framework: We present the first successful application of distribution matching distillation in speech synthesis that achieves superior quality to the teacher model. This counters the prevailing view in the field that distillation necessarily leads to quality degradation.

• Mode Shrinkage Insight: Our analysis of mode shrinkage during distillation reveals a fundamental insight about conditional generation tasks: in strongly conditional generation, diversity reduction can be beneficial when it emphasizes high-probability regions without compromising output variation across different prompts and text inputs.

• Unified Optimization Framework: By enabling direct metric optimization in a diffusion framework, we bridge the gap between two previously separate research areas: perceptual metric optimization and diffusion models, establishing a foundation for future research on optimizing generative models with human preferences.

These contributions extend beyond speech synthesis and offer valuable insights for any conditional generative modeling task, especially those requiring both quality and efficiency.

We believe our work advances both the theoretical understanding of generative model distillation and provides a practical framework applicable to numerous domains requiring conditional generation with perceptual quality constraints.

Thank you again for your overall positive assessment. We hope this clarification addresses your concerns regarding the broader impact and theoretical novelty of our work.