Continuous Speech Synthesis using per-token Latent Diffusion

1. The trends observed from experimental results are not clear. The Abstract states "both continuous and discrete approaches are highly competent". In addition, for various (tasks, modeling) combinations in Table 1, both discrete and continuous features have their merits. Therefore, it seems a bit unreasonable to use this title in this context, "CONTINUOUS SPEECH SYNTHESIS USING PER-TOKEN LATENT DIFFUSION".
2. There is no comparison of the proposed SALAD model with other open-source models. Such comparisons make the paper more convincing.
3. There is no comparison of the proposed SALAD model with other baselines without pre-token latent diffusion methods, which is very important for verifying motivation.
4. Time consumption is more important for sequence generation tasks, such as speech generation in this paper than image generation. The authors should give the RTF or other metrics on inference time in Table 1 as a reference.
5. I don't think the ablation study of VAE sampling makes much sense, and the authors should include more experiments with "Discrete Q" or "Continuous d" to further illustrate the relationship between them and the quality of generation.

Limited Comparison with Prior Work:

The paper's comparison with previous work is insufficient. It does not thoroughly engage with the existing body of literature on speech synthesis, particularly in terms of how SALAD's performance compares to state-of-the-art models on various metrics, such as the vall-E series including VALLE 2, RALLE and the naturalspeech series. The author should at least compare with their demos. There is a lack of depth in the discussion of how SALAD's approach differs from and improves upon previous methods except for the diffusion head, which is crucial for establishing the novelty and impact of the research.

Insufficient Result Significance:

According to Table 1, it seems that the continuous models cannot make a huge difference, which weakens the contribution of this paper. Moreover, the three variants in paper have been proposed in previous works.

• Comparison to prior work is somewhat limited. The paper compares its model to XTTS-v2 but overlooks other prior work. Additionally, even with XTTS-v2, some metrics are not reported.

  • The authors point that "We did not report objective scores for XTTS due to the sample limit in their demo." While the 200-character limitation exists in the demo, the main repository has an API (tts.tts_to_file(*)) that breaks up long text into sentences for audio synthesis beyond the 200-character limit.
  • While Voicebox and Vall-E are not open-source, the ZS-TTS test setup on LS test-clean dataset and protocol are documented in detail. Voicebox, ClAM-TTs, Audiobox, NaturalSpeech3, and MELLE compare to Vall-E on ZS-TTS; is there a reason the protocol described in the Vall-E ZS-TTS protocol not followed here?
  • VoiceCraft (weights and code released), which is based on discrete codec and auto-regressive modeling would be another valuable comparison.

• The paper lacks the some ablations and misses discussion of some design choices:

  • In NAR modeling, the paper extends MaskGIT with diffusion loss which couples diffusion and MaskGIT. Ablations for (a) diffusion only and (b) MaskGIT only would be informative.
  • The paper use BPE for semantic tokens but its impact on performance is unclear. It would be make a comment on this if this was found to useful in prior work.
  • The paper describes design choices for training/inference, such as $64$ MaskGIT steps for NAR, $20$ diffusion steps, $4$ codebooks (for discrete), and $d=8$ for continuous VAE. A brief discussion of these choices would be valuable. For example, while the choice of fewer codebooks is discussed, $d=32$ appears to be more effective for continuous VAE. Similarly, Figure 6(c) indicates that $5$ diffusion steps yield better UTMOS with minor drop in the similarity scores from 0.528 to 0.520.

• Benchmarks/discussion on inference speed.

  • The combination of continuous features with a diffusion head would incur a significant inference speed cost, as each timestep now requires diffusion. Even for the NAR model, MaskGIT with a diffusion head will be slower than using either diffusion or MaskGIT alone. This is relevant for practical applications and should be discussed in the paper along with the results.
  • Section 3.3 states "Given $K$ MaskGIT steps, SoundStorm requires passes $QK$ through the transformer, as it employs a MaskGIT procedure per RVQ layer, unlike SALAD-NAR, which requires $K$ transformer passes." This should be updated to also include the effect of diffusion steps.

References:

• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (NeurIPS 2023)
• CLAM-TTS: Improving Neural Codec Language Modeling for Zero-shot Text-to-speech (ICLR 2024)
• VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild (ACL 2024)
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (Arxiv 2023)
• Autoregressive Speech Synthesis without Vector Quantization (Arxiv 2024)
• Audiobox: Unified Audio Generation with Natural Language Prompts (Arxiv 2023)

• The performance improvements presented are not consistently better than those achieved by discrete methods. While the paper hypothesizes that quantizing continuous latent representations is suboptimal and introduces a per-token latent diffusion approach as an alternative, the continuous VAE model (with bottleneck dimension d=8 ) shows lower reconstruction quality, as measured by PESQ, compared to the 4-codebook Residual VQ model. Furthermore, in key generative modeling experiments, the proposed method often underperforms, as shown in Table 1 and Figure 5. I recommend that the authors clarify their rationale for choosing such a small bottleneck dimension for generative modeling. Additionally, a more in-depth investigation of such as when and why quantization of continuous latent representations leads to suboptimal performance, whether in generative modeling or reconstruction quality, would enhance the contribution and understanding of this work.
• Although the continuous model shows superior intelligibility, the paper lacks a detailed qualitative analysis or ablation study that convincingly demonstrates why continuous methods should be preferred over discrete ones. This analysis is crucial to strengthen the claims about the advantages of continuous modeling. For instance, exploring how continous methods might achieve better trade-offs between reconstruction quality of autoencoders and generation performance, or between generation performance and sampling efficiency. Demonstrating these trade-offs would offer valuable insights into the advantages of continuous representations.
• The work primarily extends existing generative methods from the image domain to audio, with the primary distinction being the parallel prediction of semantic and acoustic tokens for handling variable-length outputs. This extension, while useful, limits the scientific novelty and originality of the contribution. Emphasizing the unique challenges involved in adapting these methods to audio for variable-length modeling would be valuable. For instance, the authors could highlight how parallel semantic and acoustic token prediction is non-trivial and explain its effectiveness compared to alternative methods, such as binary stop prediction. Providing evidence that the proposed stopping condition using semantic tokens performs better than simpler binary classifiers would strengthen the originality of this work.
• The proposed method is inefficient in terms of sampling, as it requires more generation steps due to the iterative denoising process of diffusion models. Although Figure 6 (c) and (d) hint at potential advantages in sampling speed compared to discrete methods, further explanation is needed. Specifically, the authors should discuss why the generation quality degrades when the number of diffusion steps exceeds 20 and whether MaskGIT steps could be reduced while maintaining quality, possibly by applying fewer iteration steps at deeper quantization layers, as in SoundStorm. Additionally, the use of a 12-block MLP for noise estimation appears significantly larger compared to the 3-block architectures used in prior work. The authors should provide a detailed justification for using a 12-block MLP, including its impact on overall performance and whether smaller architectures were considered.

The paper lacks a clear narrative about what authors want to achieve. It focuses on the implementation of expressive diffusion head and related TTS architecture, that makes the paper sounds a technical report than a scientific paper.