MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer

Q3: Is the output length of the text-to-semantic stage equal to that from the semantic-to-acoustic stage? From Figure 2, the former seems shorter than the latter?

Yes, they are the same. More precisely, the total length of the prompt semantic tokens and target semantic tokens is consistent with the total length of the prompt acoustic tokens and target acoustic tokens.

Q4: The terms "coarse-grained" and "fine-grained" do not seem to be mentioned in the SpearTTS paper, what do the authors mean by them?

In our paper, "coarse-grained" refers to the first layer of acoustic tokens, while "fine-grained" refers to the remaining layers. This terminology aligns with AudioLM [2], which SpearTTS extends. AudioLM explicitly uses the terms "Coarse acoustic modeling" and "Fine acoustic modeling" in its framework. To clarify the connection: SpearTTS employs an autoregressive (AR) model to predict semantic codes from text, followed by another AR model for acoustic code prediction. As stated in the SpearTTS paper: "Similar to AudioLM, it is possible to add an optional third stage, with the goal of improving the quality of the synthesized speech by predicting acoustic tokens corresponding to fine residual vector quantization levels."

Q5: What's the length of prompt audio? Does it affect the quality of the generated speech?

Our model offers considerable flexibility in prompt handling:

1. Training Strategy: We randomly select 0-50% of the input speech as the prompt during training, enabling the model to adapt to varying prompt lengths.

2. Inference Guidelines: Since our training data consists of samples under 30 seconds, we recommend keeping prompt lengths below 20 seconds for optimal performance in practical applications.

Thanks again for your constructive comments. We would be grateful if we could hear your feedback regarding our answers to the reviews. We would be happy to answer and discuss if you have further comments.

[1] Lee K, Kim D W, Kim J, et al. DiTTo-TTS: Efficient and Scalable Zero-Shot Text-to-Speech with Diffusion Transformer[J]. arXiv preprint arXiv:2406.11427, 2024.

[2] Borsos Z, Marinier R, Vincent D, et al. Audiolm: a language modeling approach to audio generation[J]. IEEE/ACM transactions on audio, speech, and language processing, 2023, 31: 2523-2533.

First of all, we want to thank the reviewer for your careful reading and providing a lot of constructive comments! Additionally, we appreciate the acknowledgment of the impact of our work on the community. Below we address the concerns mentioned in the review.

Weaknesses

Weakness 1: The novelty is incremental as it mainly follows SPEAR-TTS's two-stage strategy and SoundStorm/NaturalSpeech3's masked generation approach, though it uniquely applies non-AR masked generation to both stages.

We appreciate your recognition of our engineering contributions and commit to open-sourcing all code and weights to benefit the research community. We would like to highlight MaskGCT's significant contributions at both representation and modeling levels:

1. At the modeling level, MaskGCT innovatively applies masked generative modeling to both text-to-semantic and semantic-to-acoustic generation. While this approach was originally developed for image generation (MaskGIT), its adaptation to TTS presents unique challenges and opportunities. Notably, MaskGCT achieves non-autoregressive TTS without requiring explicit phone-speech alignment supervision or phone-level duration prediction, which significantly simplifies the traditional TTS pipeline while achieving more natural and consistent generation results.

2. At the representation level, MaskGCT introduces VQ-based semantic tokens, which represents a distinct approach compared to previous methods that relied on k-means clustering.

Weakness 2: The claim of not requiring text-speech alignment or duration prediction seems inconsistent, as the method still needs target sequence length prediction based on phone-level durations.

It is noteworthy that there are several methods to determine the total duration length. Our trained duration predictor is solely for providing a rough estimate to facilitate inference and comparison. Alternatively, a simpler approach could be: $*target total duration* = *target phone number* \times \frac{*prompt total duration*}{*prompt phone number*}$. Additionally, one could train a regression-based model to directly predict the total duration using prompt text, target text, and prompt speech as inputs, as proposed in [1]. In Section 4.2.3 and our demo pages, we also demonstrate that our method can generate satisfactory results within a reasonable range of total durations.

The table below illustrates the comparative results of MaskGCT under three different total duration calculation methods. The results indicate that our model, using simple rules to predict total duration, can generate speech with SIM and WER that are essentially comparable to those of the ground truth.

Weakness 3: The claim about VQ-VAE's superiority over k-means clustering for semantic representation needs experimental validation.

Thank you for your valuable suggestion. This question has also been raised by other reviewers, and we believe a thorough investigation of this matter will significantly strengthen our paper. We will first present our empirical findings (which are already included in our paper), followed by additional experimental results. All these analyses will be incorporated into the revised version of our paper.

In our initial experiments, we observed that k-means-based semantic tokens were less effective in predicting acoustic tokens for languages with rich prosodic variations, particularly Chinese, where significant pitch variations were observed. We further support this finding with qualitative analysis.

1. Since k-means can be seen as optimizing the reconstruction loss between input and reconstruction, we present reconstruction loss curves under different k-means and VQ configurations. We compare four configurations: VQ-8192 (which is the same as in our paper), VQ-2048, k-means-8192, and k-means-2048. The loss curves can be found at this link.

2. The information preservation in semantic tokens directly affects the semantic-to-acoustic model's prediction performance. We present the top-10 accuracy (shown in this link) of the semantic-to-acoustic model in predicting the first layer of acoustic tokens. The results demonstrate that VQ-8192 outperforms VQ-2048, which in turn outperforms k-means-8192.

3. We investigate the impact of different semantic representation approaches on acoustic token reconstruction. We train separate semantic-to-acoustic (S2A) models for each configuration and evaluate their performance through speech reconstruction metrics. Across all three test sets, the results reveal a consistent performance ranking in similarity (SIM) scores, with VQ-8192 yielding the highest performance, followed sequentially by VQ-2048, k-means-8192, and k-means-2048. For WER, VQ-based methods also demonstrate superior performance over k-means approaches, though this advantage is less pronounced on LibriSpeech test-clean. Notably, on SeedTTS test-zh (Chinese test set), k-means exhibits a substantial degradation in WER. We attribute this to the stronger coupling between Chinese semantics and prosodic features, where the transition from VQ to k-means results in significant loss of prosodic information in the semantic representations.

We believe these additions will strengthen our argument and offer a clearer understanding of the impact of different semantic token approaches on model performance.

Questions

Q1: In Table 1, does "ZS TTS" denote "zero-shot TTS", and what's "CL TTS"? For "Imp. Dur.", I also don't fully agree that this work implicitly models duration, as it requires a specified length of the target generated speech.

Yes, "ZS TTS" denotes "zero-shot", "CL TTS" denotes cross-lingual TTS. For the latter part of the question, we are primarily referring to the absence of explicit phone-level duration modeling. We appreciate the feedback on the unclear definitions. We will add more detailed explanations in the revised paper.

Q2: In Section 3.2.3, why "the number of frames in the semantic token sequence is equal to the sum of the frames in the prompt acoustic sequence and the target acoustic sequence"?

Because our semantic tokens and acoustic tokens both have a frame rate of 50Hz, while the acoustic tokens consist of multiple layers, each with a frame rate of 50Hz. More accurately, the semantic token sequence input to the semantic-to-acoustic model is a concatenation of the semantic tokens corresponding to the prompt speech and the target speech.

It's worth noting that semantic and acoustic tokens don't necessarily need to share the same frame rate, as long as they maintain a fixed ratio of tokens generated per unit time.

Thank you very much for your valuable feedback!

• I agree that both SoundStorm and NaturalSpeech 3 are codec-based TTS systems. In SoundStorm, it uses an AR (Autoregressive) model as the first stage and an NAR (Non-Autoregressive) model as the second stage. Robustness and inference speed are known issues in AR models. Unlike AR-based models, NaturalSpeech 3 explicitly employs a phone duration predictor and then obtains the frame-level phoneme as a condition to predict the frame-level prosody tokens, followed by predicting the frame-level content tokens and acoustic tokens. This means it requires pre-extracted phone duration for model training, and the performance of phone duration prediction impacts the overall prosody of the generated speech. Unlike SoundStorm and NaturalSpeech 3, MaskGCT does not use phone-level duration. We only need phone-level conditioning to predict frame-level semantic tokens, or directly use text tokens obtained from BPE (we have shown the results in Appendix A.7, which indicate that using BPE does not cause significant performance loss and may even yield better WER for Chinese). Additionally, MaskGCT also explores speech semantic tokens based on VQ. In our supplementary experiments, we also investigate the differences between a two-stage model and direct text-to-acoustic modeling (the results are shown in Appendix K).

• We also illustrate the differences between MaskGCT and traditional NAR (Non-Autoregressive) methods in text-speech alignment modeling through a figure (in this https://raw.githubusercontent.com/maskgct/maskgct/refs/heads/main/aatn_map.png). For traditional NAR TTS systems, a duration predictor and length regulator are first used to obtain frame-level conditions that provide alignment information. For MaskGCT, similar to AR (Autoregressive) methods, we concatenate the text in front of the speech tokens through in-context learning, and the model implicitly learns speech-text alignment through self-attention.

• In addition, we provide attention maps at different stages of inference (steps 1, 10, 20) and across various layers of the model (layer 1, layer 6, layer 16) in this https://raw.githubusercontent.com/maskgct/maskgct/refs/heads/main/ar_vs_nar_vs_maskgct.png. These attention maps demonstrate that our model implicitly learns the alignment between text and speech. It shows that the model handles the alignment between speech tokens and text in the middle layers of the network, while in the deeper layers, the model has already resolved the alignment and focuses on processing the tokens.

• For the second concern, we appreciate the highly constructive comments, and we will incorporate the results with the rule-based duration into Table 1, as we agree that this can better substantiate our claim that text-speech alignment supervision or phone-level duration prediction is unnecessary. We further emphasize and discuss this point in Appendix A.5. Additionally, the absence of phone-level duration means that we can simply adjust the total duration to control the tempo of the generated speech. In Section 4.2.3 Duration Length Analysis, we demonstrate that MaskGCT exhibits robustness across gt duration lengths ranging from gt len * 0.7 to gt len * 1.3, and at the demo page "Speech Tempo Controllability", we showcase the demonstrate of the produced speech across varying total durations (gt len * 0.6 to gt len * 1.2), all of which exhibit good naturalness.

Thanks again for your valuable comments. We have made every effort to address your concerns and enhance our paper, and we kindly hope you can reconsider our paper. If you have any further suggestions or concerns, we are more than willing to provide more discussion.

Weakness 3: should acknowledge e2-tts as the first work using unexpanded text conditioning in NAR TTS.

Thank you for highlighting this important point. We would like to clarify our treatment of this topic:

Thank you for pointing out a very important point. In our related works, we mention that "SimpleSpeech, DiTTo-TTS, and E2-TTS [1] are also NAR-based models that do not require precise alignment information between text and speech, nor do they predict phone-level duration," and discuss the differences between our model and theirs in Appendix K. E2-TTS and its recent open-source version [2] have demonstrated superior performance in large-scale training. However, recent work [2, 3] also points out that it struggles to converge on small datasets. We conduct a similar experiment, directly predicting acoustic codes from text, and observed similar phenomena. We provide some results below. The results show that directly predicting acoustic codes from text is challenging to converge, resulting in lower SIM and significantly higher WER. Of course, this approach differs from E2-TTS in several aspects. We conduct this investigation primarily to demonstrate that our two-stage model potentially reduces the overall modeling complexity.

Thanks again for your constructive comments. We would be grateful if we could hear your feedback regarding our answers to the reviews. We would be happy to answer and discuss if you have further comments.

[1] Eskimez S E, Wang X, Thakker M, et al. E2 TTS: Embarrassingly easy fully non-autoregressive zero-shot TTS[J]. arXiv preprint arXiv:2406.18009, 2024.

[2] Chen Y, Niu Z, Ma Z, et al. F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching[J]. arXiv preprint arXiv:2410.06885, 2024.

[3] A^2-Flow: Alignment-Aware Pre-training for Speech Synthesis with Flow Matching https://openreview.net/attachment?id=e2p1BWR3vq&name=pdf

First of all, we want to thank the reviewer for your careful reading and providing a lot of constructive comments! Below we address the concerns mentioned in the review.

Weaknesses

Weakness 1: The higher WER values in Tables 4 and 5 compared to previous reports need explanation.

We find that one possible reason for the high WER could be that the ASR model has poor recognition capabilities for accents and emotional data. We discovered that the WER of the ground truth is also high. Thanks for your valuable suggestions, we will incorporate these discussions into the revised version of our paper.

Weakness 2: The choice of VQ-VAE over k-means for semantic tokenization needs experimental validation.

Thank you for your valuable suggestion. This question has also been raised by other reviewers, and we believe a thorough investigation of this matter will significantly strengthen our paper. We will first present our empirical findings (which are already included in our paper), followed by additional experimental results. All these analyses will be incorporated into the revised version of our paper.

In our initial experiments, we observed that k-means-based semantic tokens were less effective in predicting acoustic tokens for languages with rich prosodic variations, particularly Chinese, where significant pitch variations were observed. We further support this finding with qualitative analysis.

1. Since k-means can be seen as optimizing the reconstruction loss between input and reconstruction, we present reconstruction loss curves under different k-means and VQ configurations. We compare four configurations: VQ-8192 (which is the same as in our paper), VQ-2048, k-means-8192, and k-means-2048. The loss curves can be found at this link.

2. The information preservation in semantic tokens directly affects the semantic-to-acoustic model's prediction performance. We present the top-10 accuracy (shown in this link) of the semantic-to-acoustic model in predicting the first layer of acoustic tokens. The results demonstrate that VQ-8192 outperforms VQ-2048, which in turn outperforms k-means-8192.

3. We investigate the impact of different semantic representation approaches on acoustic token reconstruction. We train separate semantic-to-acoustic (S2A) models for each configuration and evaluate their performance through speech reconstruction metrics. Across all three test sets, the results reveal a consistent performance ranking in similarity (SIM) scores, with VQ-8192 yielding the highest performance, followed sequentially by VQ-2048, k-means-8192, and k-means-2048. For WER, VQ-based methods also demonstrate superior performance over k-means approaches, though this advantage is less pronounced on LibriSpeech test-clean. Notably, on SeedTTS test-zh (Chinese test set), k-means exhibits a substantial degradation in WER. We attribute this to the stronger coupling between Chinese semantics and prosodic features, where the transition from VQ to k-means results in significant loss of prosodic information in the semantic representations.

These comprehensive analyses demonstrate the advantages of VQ-based semantic tokens over k-means clustering, providing strong empirical evidence to support our design choices in MaskGCT.

Weakness 5: The claim about information loss in k-means-based discrete SSL units compared to VQ-VAE needs better substantiation.

Thank you for your valuable suggestion. This question has also been raised by other reviewers, and we believe a thorough investigation of this matter will significantly strengthen our paper. We will first present our empirical findings (which are already included in our paper), followed by additional experimental results. All these analyses will be incorporated into the revised version of our paper.

In our initial experiments, we observed that k-means-based semantic tokens were less effective in predicting acoustic tokens for languages with rich prosodic variations, particularly Chinese, where significant pitch variations were observed. We further support this finding with qualitative analysis.

1. Since k-means can be seen as optimizing the reconstruction loss between input and reconstruction, we present reconstruction loss curves under different k-means and VQ configurations. We compare four configurations: VQ-8192 (which is the same as in our paper), VQ-2048, k-means-8192, and k-means-2048. The loss curves can be found at this link.

2. The information preservation in semantic tokens directly affects the semantic-to-acoustic model's prediction performance. We present the top-10 accuracy (shown in this link) of the semantic-to-acoustic model in predicting the first layer of acoustic tokens. The results demonstrate that VQ-8192 outperforms VQ-2048, which in turn outperforms k-means-8192.

3. We investigate the impact of different semantic representation approaches on acoustic token reconstruction. We train separate semantic-to-acoustic (S2A) models for each configuration and evaluate their performance through speech reconstruction metrics. Across all three test sets, the results reveal a consistent performance ranking in similarity (SIM) scores, with VQ-8192 yielding the highest performance, followed sequentially by VQ-2048, k-means-8192, and k-means-2048. For WER, VQ-based methods also demonstrate superior performance over k-means approaches, though this advantage is less pronounced on LibriSpeech test-clean. Notably, on SeedTTS test-zh (Chinese test set), k-means exhibits a substantial degradation in WER. We attribute this to the stronger coupling between Chinese semantics and prosodic features, where the transition from VQ to k-means results in significant loss of prosodic information in the semantic representations.

We believe these additions will strengthen our argument and offer a clearer understanding of the impact of different semantic token approaches on model performance.

Minor Point 1: Figure 2 would benefit from additional notations for clarity. Also, what is the difference between prompt semantic tokens and target semantic tokens? Can you give an example?

The prompt serves as a reference for style, prosody, and speaker characteristics, enabling zero-shot TTS through in-context learning. During training, we randomly select a prefix from each sample as the prompt, keeping its semantic tokens unmasked. The model learns to predict masked tokens by conditioning on both the prompt's semantic tokens and the input text. For inference, we construct the input sequence by concatenating [prompt text, target text, prompt semantic tokens], which guides the model in predicting the target tokens.

We sincerely thank the reviewer for their thorough review and valuable constructive feedback. We are also grateful for your recognition of our work's impact on the research community. We address each of your concerns in detail below.

Weaknesses

Weakness 1: Limited novelty as the method mainly combines existing works (MaskGIT, SpearTTS) with no apparent novel contributions in acoustic modeling (using DAC).

We would like to highlight MaskGCT's significant contributions at both representation and modeling levels: (1) At the modeling level, MaskGCT innovatively applies masked generative modeling to both text-to-semantic and semantic-to-acoustic generation. While this approach was originally developed for image generation (MaskGIT), its adaptation to TTS presents unique challenges and opportunities. Notably, MaskGCT achieves non-autoregressive TTS without requiring explicit phone-speech alignment supervision or phone-level duration prediction, which significantly simplifies the traditional TTS pipeline while achieving more natural and consistent generation results. (2) At the representation level, MaskGCT introduces VQ-based semantic tokens, which represents a distinct approach compared to previous methods that relied on k-means clustering.

Weakness 2: The claim about NAR methods being complex due to duration prediction needs better justification, as duration modeling in models like NaturalSpeech3/FastSpeech2 is relatively straightforward.

We would like to clarify that the complexity of phone-level duration prediction arises from several key aspects:

1. Implementation Complexity: Phone-level duration prediction requires pre-extracted phone-level durations as supervision and additional modeling modules, which becomes more challenging in zero-shot scenarios and with in-the-wild or noisy data. For instance, NaturalSpeech 3 employs a Conformer-based duration predictor with discrete diffusion, while [1] relies on diffusion models.

2. Computational Overhead: The lack of efficient tools for phone-level duration extraction makes it computationally intensive, especially when processing large-scale datasets.

3. Accuracy Limitations: Phone-level duration extraction from in-the-wild data often lacks accuracy, potentially compromising model performance. Evidence from [2] shows that VoiceBox's reproduction on the Emilia dataset, which depends on phone-level duration prediction, achieved lower naturalness and similarity scores compared to autoregressive models that avoid such prediction.

4. Scalability Considerations: Our approach aims to reduce dependence on such priors while leveraging more powerful generative models and larger datasets to enhance generation quality. We acknowledge this may not fully apply to traditional small-data TTS scenarios, where phone-level duration prediction might be necessary for model convergence, as demonstrated in flow-matching-based works [3,4]. To validate our approach, we conducted additional experiments training T2S models on smaller datasets (LibriTTS and a 1K-hour Emilia subset) while maintaining the self-supervised S2A model. Results demonstrate robust performance even with limited training data, which we attribute to the effectiveness of semantic token prediction and the strong modeling capabilities of masked generative approaches.

Weakness 3: The Related Work section omits important flow-based methods (VoiceBox, p-flow) that use masked generative models.

We would like to note that VoiceBox was introduced in the first paragraph of our Related Work section and served as a baseline in our experimental comparisons. We acknowledge the oversight in not including the p-flow paper and will incorporate it in the revised version.

Additionally, we want to clarify some potential misconception. VoiceBox is a flow matching-based TTS system, and its mask is designed to serve as a prompt during training, which is different from the "mask" in masked generative models. We believe the "mask" in masked generative models is more akin to the noise in diffusion models, as masked generative models have a discrete diffusion perspective.

Weakness 4: Table 1's prominence suggests multi-task capability as the main contribution, while the focus should be on concrete improvements in performance and efficiency.

Thank you for your suggestions! We will reorganize it in the revised version of the paper.

Minor Point 2: The advantage of using semantic tokens as an intermediate representation needs justification (e.g., direct text2acoustic without semantics) should be justified.

Thank you for this constructive suggestion. We have investigated this issue thoroughly and found several key advantages of our semantic token approach:

1. Performance Superiority: Recent works attempting to model continuous features like Mel-spectrograms directly without phone-level duration [3] show lower similarity compared to MaskGCT. Our experiments demonstrate that directly mapping text to multi-layer acoustic tokens using masked generative models leads to convergence difficulties, resulting in poor intelligibility and high WER.

2. Training Stability: Both [3,5] report convergence issues on small datasets when bypassing intermediate representations. This aligns with our early experimental findings.

3. Empirical Evidence: We conducted a comparative study between MaskGCT and a direct text-to-acoustic approach (implemented by removing semantic token conditioning and adding text conditioning to our semantic-to-acoustic model) on a 10K-hour subset. The results clearly show that direct acoustic token prediction from text faces convergence challenges, yielding lower SIM scores and substantially higher WER. This demonstrates that our two-stage approach effectively reduces the overall modeling complexity.

We are currently extending these experiments to the full dataset and will include comprehensive results in the revised paper.

Minor Point 3: Section 4.2.3's purpose and motivation need clarification, particularly if addressing speed modification, as specialized tools exist for this purpose.

This section aims to demonstrate two key aspects of our model:

1. Generation Capability: MaskGCT can produce high-quality speech across a wide range of durations while maintaining natural prosody. This is fundamentally different from simple speech rate adjustment, which often introduces artifacts in prosody, pitch, and timbre.

2. Control and Diversity: Our approach offers better controllability over speech generation compared to AR models, while ensuring diverse outputs. We have provided audio samples on our demo page that showcase natural prosodic variations across different durations.

Minor Point 4: Additional efficiency metrics beyond inference steps (e.g., latency) should be considered.

Thanks for your suggestion. We provide the real-time factor (RTF) of MaskGCT on an A100 GPU for generating a 20-second speech across various inference steps in Table. Across all configurations presented, there is no significant performance difference. Additionally, we also present the RTF of AR + SoundStorm. For AR + SoundStorm, generating a 20-second speech requires 20 * 50 = 1000 steps for text-to-semantic inference. However, we can leverage kv-cache to accelerate the process.

Thanks again for your constructive comments. We would be grateful if we could hear your feedback regarding our answers to the reviews. We would be happy to answer and discuss if you have further comments.

[1] Li X, Liu S, Lam M W Y, et al. Diverse and expressive speech prosody prediction with denoising diffusion probabilistic model[J]. arXiv preprint arXiv:2305.16749, 2023.

[2] He H, Shang Z, Wang C, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation[J]. arXiv preprint arXiv:2407.05361, 2024.

[3] Chen Y, Niu Z, Ma Z, et al. F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching[J]. arXiv preprint arXiv:2410.06885, 2024.

[4] A^2-Flow: Alignment-Aware Pre-training for Speech Synthesis with Flow Matching https://openreview.net/attachment?id=e2p1BWR3vq&name=pdf

First of all, we want to thank the reviewer for your careful reading and providing a lot of constructive comments! Below we address the concerns mentioned in the review.

Weaknesses

Weakness 1: The main novelty appears to be extending Soundstorm's mask-predict approach from acoustic to semantic token generation.

MaskGCT employs masked generative modeling for both text-to-semantic and semantic-to-acoustic generation. Beyond this, we would like to highlight that MaskGCT's novelty lies in two key aspects: (1) MaskGCT explores non-autoregressive TTS without requiring explicit phone-speech alignment supervision or phone-level duration prediction, which significantly simplifies the traditional TTS pipeline while achieving more natural and consistent generation results. (2) MaskGCT introduces VQ-based semantic tokens, which represents a distinct approach compared to previous methods that relied on k-means clustering.

Weakness 2: The paper lacks experimental comparison between the proposed VQ-VAE and k-means clustering for semantic tokens

Thank you for your valuable suggestion. This question has also been raised by other reviewers, and we believe a thorough investigation of this matter will significantly strengthen our paper. We will first present our empirical findings (which are already included in our paper), followed by additional experimental results. All these analyses will be incorporated into the revised version of our paper.

In our initial experiments, we observed that k-means-based semantic tokens were less effective in predicting acoustic tokens for languages with rich prosodic variations, particularly Chinese, where significant pitch variations were observed. We further support this finding with qualitative analysis.

1. Since k-means can be seen as optimizing the reconstruction loss between input and reconstruction, we present reconstruction loss curves under different k-means and VQ configurations. We compare four configurations: VQ-8192 (which is the same as in our paper), VQ-2048, k-means-8192, and k-means-2048. The loss curves can be found at this link.

2. The information preservation in semantic tokens directly affects the semantic-to-acoustic model's prediction performance. We present the top-10 accuracy (shown in this link) of the semantic-to-acoustic model in predicting the first layer of acoustic tokens. The results demonstrate that VQ-8192 outperforms VQ-2048, which in turn outperforms k-means-8192.

3. We investigate the impact of different semantic representation approaches on acoustic token reconstruction. We train separate semantic-to-acoustic (S2A) models for each configuration and evaluate their performance through speech reconstruction metrics. Across all three test sets, the results reveal a consistent performance ranking in similarity (SIM) scores, with VQ-8192 yielding the highest performance, followed sequentially by VQ-2048, k-means-8192, and k-means-2048. For WER, VQ-based methods also demonstrate superior performance over k-means approaches, though this advantage is less pronounced on LibriSpeech test-clean. Notably, on SeedTTS test-zh (Chinese test set), k-means exhibits a substantial degradation in WER. We attribute this to the stronger coupling between Chinese semantics and prosodic features, where the transition from VQ to k-means results in significant loss of prosodic information in the semantic representations.

These comprehensive analyses demonstrate the advantages of VQ-based semantic tokens over k-means clustering, providing strong empirical evidence to support our design choices in MaskGCT.

Weakness 3: The semantic tokens are not well disentangled from speaker information, making MaskGCT less effective for voice conversion tasks.

Since MaskGCT was primarily designed for Zero-Shot TTS tasks, we did not extensively focus on decoupling speaker information from semantic tokens. However, in our recent research, we have enabled voice conversion capabilities through a simpler approach in the semantic-to-acoustic model. Specifically, we utilize a lightweight yet efficient voice conversion model (such as OpenVoice), to perform real-time voice conversion on the target speech using randomly sampled prompt speech, thereby achieving timbre perturbation. These perturbed semantic tokens are then used as input to predict target acoustic tokens with the prompt in the semantic-to-acoustic model. The results are presented below. We will incorporate these details in the revised version of the paper. We use the VCTK dataset to evaluate our system, randomly selecting 200 samples as source speech. For each sample, we randomly select another sample from the same speaker as the prompt speech.

Weakness 4: MaskGCT shows lower pronunciation accuracy compared to VoiceBox/NaturalSpeech 3 in single-sample generation.

I believe the reason MaskGCT outperforms NaturalSpeech 3 and VoiceBox in terms of WER metrics may be due to ASR's tendency to more accurately recognize standard speech. This is because NS3 and VoiceBox incorporate phone-level duration predictors, and the LibriSpeech dataset, being a more standardized recording test set, allows NS3 and VoiceBox to predict durations more effectively. However, for CMOS, MaskGCT performs better than VoiceBox.

Weakness 5: The objective metrics appear to be worse than seed-TTS baseline.

As far as I know, Seed-TTS utilizes over 2 million hours of training data, which is 20 times the amount used by our model. Consequently, it is challenging for our model to achieve a fair comparison in performance with Seed-TTS.

Questions

Q1: It would be helpful to mention the frame rate of semantic and acoustic tokens.

As detailed in Appendix A.4, our model operates with two types of tokens: semantic tokens (16KHz sampling rate, 320 hopsize) and acoustic tokens (24KHz sampling rate, 480 hopsize). Both tokens effectively maintain a consistent frame rate of 50Hz. We will move this important technical detail to the main text for better clarity.

Q2: I am curious whether only a single sampling step is used from the third layer during acoustic token generation. Is there any difference when using more sampling steps?

Given that the first layer of acoustic tokens carries the majority of information, we allocate more inference steps to this layer while reducing steps for subsequent layers. As demonstrated in Appendix A.3, we have analyzed how varying the number of inference steps in the second-stage model affects overall performance. Our experiments show that even a minimal configuration [10, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] maintains comparable performance, with only marginal decreases in SIM and WER metrics.

Q3: It would be beneficial to also present the overall sampling speed of the model, including the Real-Time Factor.

Thanks for your suggestion. We will also add it to the revised version of the paper. We present the real-time factor (RTF) of MaskGCT on an A100 GPU for generating a 20-second speech across various inference steps in Table. Across all configurations presented, there is no significant performance difference. Additionally, we also present the RTF of AR + SoundStorm. For AR + SoundStorm, generating a 20-second speech requires 20 * 50 = 1000 steps for text-to-semantic inference. However, we can leverage kv-cache to accelerate the process.

Q4: In Table 1, it seems that VoiceBox's Imp. Dur. should be marked as "X."

Thank you for the reminder! We will fix it in the revised version.

Thanks again for your constructive comments. We would be grateful if we could hear your feedback regarding our answers to the reviews. We would be happy to answer and discuss if you have further comments.

[1] Lee S H, Choi H Y, Kim S B, et al. Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis[J]. arXiv preprint arXiv:2311.12454, 2023.

[2] Wang Z, Chen Y, Xie L, et al. Lm-vc: Zero-shot voice conversion via speech generation based on language models[J]. IEEE Signal Processing Letters, 2023.

[3] Yang D, Tian J, Tan X, et al. Uniaudio: An audio foundation model toward universal audio generation[J]. arXiv preprint arXiv:2310.00704, 2023.

Additional Comparisons

We have conducted extensive comparative experiments to provide a more comprehensive understanding of MaskGCT:

1. Efficiency Analysis: Compared RTF (Real-Time Factor) between MaskGCT under various inference parameters and AR models

2. Small Dataset Performance: Evaluated MaskGCT's effectiveness when trained on limited data

3. Architecture Comparison: Conducted comparative studies between MaskGCT and direct text-to-acoustic models

4. Voice Conversion Enhancement: Introduced a novel approach to improve MaskGCT's voice conversion capabilities, with comparative evaluations against baseline methods

Revised Paper

In addition, we have uploaded a revised version of the paper with all changes highlighted in blue. The key updates include:

• Fixed a typo in the system comparison figure and moved it to the Appendix (page 20, start at line 1050)

• Added P-Flow reference in the related work section (page 2, line 101)

• Added analysis of semantic representation codecs (VQ vs. k-means, vocabulary size) in Section 4.4 (page 9, start at line 466)

• Included RTF measurements under different settings in Appendix A.3 (page 13, start at line 938)

• Added performance comparison on small datasets in Appendix K: Discussion about Concurrent Works (page 24, start at line 1290)

• Added performance comparison between MaskGCT and direct text-to-acoustic model in Appendix K: Discussion about Concurrent Works (page 25, start at line 1307)

• Added new voice conversion improvements in Appendix I: Voice Conversion (page 23, start at line 1236)

• Included explanation for WER variations in Section 4.3: Speech Style Imitation (page 9, start at line 441)

• Detailed two approaches for total duration calculation in Appendix A.5 (page 19, start at line 1025)

All modifications are clearly marked in the revised manuscript. We welcome any additional questions or feedback.