Controllable Text-to-Speech Synthesis with Masked-Autoencoded Style Representation

Clarity Issues:

• For Equations (2) (3) (4), several notations are left unexplained. For example, what is $I$? What does $s_{<t}, s^{<i}_t$ mean? What is $\theta_s$. The authors provide these formulations but I don't feel like they are well defined. The authors should either remove those formulations or provide more explanation on them.
• The methods in Table 2 are not well-explained. There is nowhere in the main text that explicitly explain what are "GT. + Codec" and "Acoustic LM + GT Style". While I can infer from the text that "Acoustic LM + GT Style" is passing the ground-truth style tokens and use the Acoustic LM to infer the codec tokens, I have no idea what is "GT. + Codec" though.
• Figure 3 is super-hard to read. It is too small. Please consider enlarge it.

I think most of my comments about the paper come from the experiments. Experiments:

• In Section 4.3.1, you are comparing existing TTS systems (YourTTS, XTTS-V2) that conditioned only on phonemes, with your system that conditioned on phonemes and ground truth style tokens. This is misleading as your system has an inherent advantage to access some ground truth attributes of the target speech. Furthermore, this is the only experiment you compare with the existing systems. My suggestion is to run experiments comparing your method with the whole two-stage inference and the existing TTS systems. Ideally, you should also compare with these methods trained with the similar datasets and number of parameters, where in this paper, the information is not given.
• Also, in Section 4.3.1, you mentioned that "when we swap the phonemes and style tokens from different samples, the model fails to generate meaningful speech. This demonstrates that our style representation does not result in significant content information leakage." But to me, this sounds like a content leakage issue. As the syntheses of the swapped version should still be producing the same content as the given phoneme sequence.
• You claimed that "training the first-stage model on extensive datasets enhances the robustness of the two-stage model in terms of quality and content accuracy". I don't think I saw experiments that support this. I thought that you will compare a model that the first-stage is trained on extensive datasets v.s. a ablated model that the first-stage is not trained on extensive datasets.
• See more in the questions...

1. The style tokens combine speaker timbre, prosody, and acoustic environment information into a single representation, primarily through a reconstruction task. However, the contributions of each attribute are not well disentangled. Additional experiments showing how modifying one attribute affects others would strengthen the analysis.
2. The experiments and demos focus heavily on the acoustic LM + ground truth (GT) style reconstruction results. However, the main contribution should emphasize style token generation rather than reconstruction.
3. The formatting and presentation of tables could be improved. For example, using only one CFG parameter for inference, except in the ablation study of CFG hyperparameters, would provide a clearer understanding for readers.

The most egregious weakness of this paper is in the core experimental results on style-controllable TTS. Specifically:

• The paper does not compare against a single baseline model from the literature for this task, when numerous open source models already exist given the current popularity of the task. It only compares against a 1-stage version of the proposed model. This fact alone is sufficient to warrant a "Reject" for ICLR.
• Only automatic metrics are used, no human listening tests at all are conducted. Again, for a TTS paper this fact alone is sufficient to warrant a "Reject" for ICLR.

I also believe that the reconstruction results are flawed, specifically the comparison against voice cloning TTS models like YourTTS or XTTSv2. The proposed model is being given as input frame-level embeddings extracted from the speech signal used as a reconstruction target (in the form of the style tokens), whereas the voice cloning TTS baselines are not. Despite the authors' claims that the style tokens do not contain information about the phonetic content of the speech signal, I strongly believe they do (given that similar MLM encoders - without the extra phonetic input - such as HuBERT or WavLM represent the current SotA in the speech field for self-supervised learning of information like phonetic content). The authors would need to provide some experimental justification for their claim, but such evidence doesn't exist in the paper.

The model is very complicated, with 3 separately trained components (4 if you count the RVQ stack which is trained separately from the Style MAE. However, the paper does not conduct enough ablation studies to verify the importance of all of these components. For example, what happens if you simply replace the style tokens with off-the-shelf HuBERT units?

The paper is difficult to follow and missing critical details. For example:

• The paper does not describe which neural codec model is used, it only ever refers to the output of the Acoustic LM as "codec tokens"
• The paper does not describe how the discrete control signals are extracted during training, it only states "These labels are denoted by extracting attribute values with some tools and binning them to different levels."
• The training of the RVQ stack is never explained (which loss functions, codebook size, what specific type of RVQ, etc).

• The review of fine-grained control in TTS systems is somewhat basic. The shortcomings of existing approaches and the rationale for using this particular style-LM in LM-based methods are not clearly addressed. To my knowledge, there are many relevant papers on this topic; please cite and discuss the pros and cons accordingly.

• The style-LM relies on external embedding or attribute extractors, whose quality, I believe, significantly impacts performance. Since perfect embeddings or attributes cannot always be extracted, it would be beneficial to analyze how these extractors affect the final output, especially given the issues with the arousal-valence-dominance extractor mentioned by the authors. Including an analysis on ground-truth audio would strengthen the paper’s conclusions.

• There is limited comparison between the proposed approach and existing models. For instance, it would be helpful to show the level of improvement over models like VALL-E (a fundamental LM-based approach) or NaturaSpeech3.

1. The paper does not compare to existing baselines; it should do so, especially for easily available open-source ones that support some of the same controls used by this work. For example, Parler-TTS (https://github.com/huggingface/parler-tts) supports gender, pitch, snr and c50 control. Closed-source models (e.g. SpeechCraft https://www.arxiv.org/abs/2408.13608 , TextrolSpeech https://arxiv.org/abs/2308.14430) will require retraining to reproduce, but if possible, one of these should be compared against for emotion control.
2. There is no human evaluation in any of the experiments, which is the gold standard for evaluation; it would be good to have human speech-controllabel consistency metrics and human speech naturalness metrics.