Metis: A Foundation Speech Generation Model with Masked Generative Pre-training

Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.

Weakness 1: From an application standpoint, the authors conduct comprehensive experiments and achieve strong results... However, I believe both conditioning methods are not novel and are commonly used in various large-scale models.

Thank you for the thoughtful feedback! While some components of our method (e.g., conditioning strategies) have been explored in prior work, the core novelty of our approach lies in leveraging masked generative modeling for pretraining a unified foundation model for speech generation. This model can be efficiently adapted to a wide range of speech generation tasks using only a small amount of training data and trainable parameters. The fine-tuning strategies across tasks are distinguished only by the type of condition, whether it is frame-level or not. As a result, tasks such as VC, SE, TSE, and lip2speech share the same fine-tuning and model architecture, differing only in the form of input condition, which naturally reflects their task-specific requirements. We also propose a unified fine-tuning and inference mechanism, as discussed in our response to Reviewer zbxf (weakness 1), to further simplify and standardize the framework across tasks.

Our goal is not to solve a single task with masked generative models, but to build a general-purpose foundation model that can flexibly adapt to diverse speech generation scenarios through a shared architecture and training pipeline.

Weakness 2: In the evaluation, the explanation or ablation study regarding architectural choices is limited...

We are happy to provide more clarification regarding the choice of LoRA ranks for different tasks. First, we find that LoRA fine-tuning is effective across all tasks, though some tasks can achieve strong performance with significantly fewer trainable parameters. In general, higher LoRA ranks lead to better performance. However, for certain tasks, even a very small LoRA rank (e.g., 4) achieves satisfactory results. For example, in SE and TSE, we observe that a LoRA rank of 4 is sufficient. We hypothesize this is because these tasks use frame-level conditions that are closely aligned with the target speech tokens, and the model's primary goal is to extract the target speech from the input.

In contrast, for tasks like TTS, the model must implicitly learn the alignment between the condition (text) and the target speech tokens, which requires more modeling capacity. We find that a LoRA rank of 32 provides strong performance in this setting, while lower ranks lead to notable degradation—particularly in WER. For example, using a LoRA rank of 4 for TTS results in a WER of around 10 on the SeedTTS-en test set. Notably, even with a LoRA rank of 32, the number of trainable parameters remains relatively small, only around 32M.

We will add these details to the paper to provide a clearer explanation of our rank selection strategy.

Weakness 3: The purpose of training a general-purpose foundation model is to enable effective and efficient adaptation to a variety of tasks... the authors should fine-tune it with a smaller dataset and showcase its performance under low-resource conditions.

Thank you for the great suggestion! In our experiments, we have explored training with even less than 1000 hours of data. For example, in Section 4.2.1, we fine-tune Metis-TTS LoRA-32 on the LibriSpeech dataset using only 0.58K hours of training data. Despite the small scale, the model achieves a WER of 4.55 on the SeedTTS test-en set, outperforming strong baselines such as VALL-E (trained on 45K hours) and VoiceCraft (9K hours), and approaching the performance of CosyVoice (170K hours). It also achieves a SIM score of 0.72, surpassing most baselines.

Similarly, in the voice conversion (VC) task, we use only 0.4K hours of training data and still outperform the SOTA baseline Vevo (trained on 60K hours) in terms of SIM and quality metrics, with a slightly higher WER (4.49 vs. 3.38).

To further validate the low-data regime setting, we extend our experiments to even smaller training sets. Due to resource and data constraints, we take TTS and VC as representative tasks and randomly sample 0.1K hours of English data from the Emilia dataset for fine-tuning.

SeedTTS en:

VCTK:

Question 1: The proposed method appears to adopt a non-autoregressive generation approach... It would be helpful to know how the inference speed of the proposed method compares to that of other existing approaches.

We are happy to provide more justification for choosing masked generative modeling (MGM) as our modeling approach for speech tokens.

• Previous works have demonstrated that MGM can yield better performance under comparable conditions for speech generation tasks. For instance, [1] shows improvements in TTS, and [2,3] report superior results for speech enhancement compared to AR baselines.

• For most tasks with non-frame-level conditions, such as VC, SE, TSE, the conditioning inputs are typically more dense and locally aligned with the target speech tokens. We observe that the attention matrices in these tasks exhibit strong locality, making MGM a more efficient modeling choice. Additionally, as a NAR method, MGM supports faster inference compared to AR models, as detailed below.

• Compared with AR models, MGM shows advantages in both training and inference efficiency, especially in tasks with frame-level conditions. AR models often require concatenating condition tokens with target tokens or using cross-attention, both of which significantly increase training complexity. In contrast, MGM can model targets independently and requires only a few iterative steps during inference. While AR models can benefit from KV caching, we find that MGM offers better inference speed in most scenarios under the same sequence length. For example, in our TTS experiments, MGM achieves lower RTF (0.35 vs. 0.57), and its advantage becomes even more pronounced as the token rate decreases. Our recent exploration shows that at a token rate of 12.5Hz, MGM achieves a lower RTF (0.18 vs. 0.31) than AR for TTS (The RTF is measured on an H100 GPU using 10-second speech samples).

• MGMs represent a promising and under-explored generative paradigm in speech pretraining. While prior works such as MaskGCT (for TTS) and AnyEnhance (for SE) have adopted MGM, they are task-specific models. In contrast, our goal is to explore the potential of MGM in building a unified pretraining framework for various speech generation tasks. Moreover, recent studies have begun exploring MGM-based pretraining in other domains, such as text generation [4] and text-image multimodal modeling [5], further validating its generality and potential.

Question 2: Why does the model require a prompt during the pre-training stage?...

In this work, introducing the prefix prompt mechanism during pretraining is a design choice to enable the model to learn how to utilize prompts for tasks that require them. This mechanism improves the model's adaptability to downstream tasks involving prompts, such as zero-shot TTS, voice conversion, and target speaker extraction. For tasks that do not require prompts, such as speech enhancement, no prompt is used during fine-tuning. In practice, we adopt a mixed strategy during pretraining: prompts are introduced in 80% of training steps, while the remaining 20% are trained without prompts. This design encourages the model to develop in-context learning abilities, specifically to better support prompt-based downstream tasks.

We are happy to provide an ablation study. We retrained the pre-trained model without using prompts, and then fine-tuned it on the zero-shot TTS task with the prompt mechanism. (Due to time and resource constraints, we will include results for other tasks in future work.) The results show that removing prompts during pretraining leads to a drop in SIM, while WER remains largely unaffected.

SeedTTS en:

SeedTTS zh:

Question 3: ...

We discuss the question in weakness 2.

Question 4: Additionally, what happens when the tasks are trained with a smaller amount of data?...

Thank you for the insightful question! We discuss the question in weakness 3.

Thanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.

Reference:

[1] Wang Y, Zhan H, Liu L, et al. Maskgct: Zero-shot text-to-speech with masked generative codec transformer[J]. arXiv preprint arXiv:2409.00750, 2024.

[2] Zhang J, Yang J, Fang Z, et al. Anyenhance: A unified generative model with prompt-guidance and self-critic for voice enhancement[J]. arXiv preprint arXiv:2501.15417, 2025.

[3] Li X, Wang Q, Liu X. Masksr: Masked language model for full-band speech restoration[J]. arXiv preprint arXiv:2406.02092, 2024.

[4] Nie S, Zhu F, You Z, et al. Large language diffusion models[J]. arXiv preprint arXiv:2502.09992, 2025.

[5] Yang L, Tian Y, Li B, et al. Mmada: Multimodal large diffusion language models[J]. arXiv preprint arXiv:2505.15809, 2025.

I appreciate the authors for their sincere explanation and the additional experiments provided in response to the review. The new results comparing pretraining with and without prompting are particularly interesting. It would be helpful if the authors could include some insights on this point in the manuscript.

Regarding one of my main concerns about the amount of data used for fine-tuning, I still personally believe that 0.1 K hours is quite substantial (for example, the YourTTS was built using around 50 hours of data, even though its performance is much lower than that of current zero-shot TTS systems). The authors argue that other baseline systems require even more data; however, since the proposed model has already been pretrained on a significantly larger dataset, I believe this comparison may not be entirely fair. According to the additional experimental results, the performance degrades considerably when the fine-tuning data is reduced to 0.1K hours, which suggests that the amount of data used in the fine-tuning stage is indeed critical.

As mentioned in the limitations section, NLP foundation models are capable of few-shot learning without any fine-tuning, which indicates that such models capture general domain knowledge. Similarly, I believe that strong performance under low-resource fine-tuning conditions would demonstrate the general knowledge captured by the proposed model.

Given the nature of the task (zero-shot and voice conversion), it might be beneficial to reduce the total amount of fine-tuning data while maintaining the number of speakers. For example, assuming a total of 10K samples, using 10 sentences for each of 1K speakers, rather than 100 sentences for each of 100 speakers, may better leverage the general knowledge acquired during pretraining for the zero-shot TTS task.

I acknowledge that developing a general-purpose speech foundation model is highly challenging, especially given the discrepancies across different tasks, and I believe this work proposes a valuable direction with the use of masked generative modeling (MGM).

Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.

Weakness 1: The idea is too simple and incremental... but many speech LLM-based systems are conditioned on a prefix.

In this work, introducing the prefix prompt mechanism during pretraining is not essential, but rather a design choice to enable the model to learn how to utilize prompts for tasks that require them. This mechanism improves the model's adaptability to downstream tasks involving prompts, such as zero-shot TTS, voice conversion, and target speaker extraction. For tasks that do not require prompts, such as speech enhancement, no prompt is used during fine-tuning.

We also considered removing the prefix during pretraining. 20% of the training steps do not include any prompt, as mentioned on line 216. We emphasize that prefix prompting is not the core focus of our work; it can be introduced solely at the fine-tuning stage for specific tasks if desired. Nonetheless, including it during pretraining does not harm performance, and we adopt the mixed strategy.

Weakness 2: The paper does not have ablation experiments, and its validation is not obvious. E.g., pre-training w/ and w/o prefix would be very informative to show the effectiveness of the proposed method.

We conducted ablation studies to isolate the core contribution of our method and validate the effectiveness of the large-scale pretraining–then–fine-tuning paradigm. Taking zero-shot TTS as an example, Metis-TTS with pretraining significantly outperforms Metis-TTS without pretraining on both SIM and WER metrics using the same training data (WER: 2.28 vs. 4.91; SIM: 0.72 vs. 0.69). We also show that with pretraining, our models can be fine-tuned using significantly fewer trainable parameters via LoRA.

We add the ablation study on with or without prompt for pretraining in the response to reviewer o6KH's question 2.

Question 1: Abstract: The abstract should be revised for clarity and informativeness... the claim is vague and less impactful.

Thank you for your suggestion. The "300 times less" refers to the fact that, compared to the 300K hours of pretraining data, our fine-tuning only requires 1K hours or less. We will revise and clarify this in the paper to ensure clearer expression.

Question 2: Lines 30–33: The discussion of limitations in prior work is unclear...

Thank you for your suggestion. We have provided clearer evidence to support our claims. For task-specific models, state-of-the-art systems in zero-shot TTS typically rely on large-scale training data, ranging from 60K to over 100K hours (e.g., MaskGCT, CosyVoice, F5-TTS, VoiceBox, and NaturalSpeech 3). Similarly, recently proposed state-of-the-art VC models such as Vevo (ICLR 2025) are trained on 60K hours of data. Multi-task models like UniAudio use a total of 165K hours, with 60K/60K/20K/10K hours allocated to TTS, VC, SE, and TSE respectively, and contain up to 1B trainable parameters.

In contrast, our experiments demonstrate that comparable or even better performance can be achieved with significantly less task-specific fine-tuning data with our masked generative pre-training. For example, in the TTS task, our model achieves performance on par with or better than MaskGCT and CosyVoice (both trained on about 100K hours), using only 10K or even 1K hours of data. In the VC task, we obtain better results than Vevo (SIM 0.50 vs. 0.38), using just 0.4K hours of training data compared to their 60K hours.

Our method addresses this challenge more effectively. It is motivated by the observation that models trained on large-scale data can recover masked SSL tokens from prompts and unmasked tokens, even without task-specific conditions, as shown in other domains [1, 2]. Empirically, our pretrained model is able to generate speech that captures the prosodic style and timbre of the prompt. However, without semantic guidance, the generated speech may lack intelligibility, often resulting in random word concatenations. This highlights the importance of task-specific conditioning, which can be efficiently introduced through fine-tuning, allowing the model to adapt flexibly to various speech generation tasks.

Question 3: Section 3.4: Please clarify the properties of the prompt used in the method...

During pretraining, we do not incorporate any task-specific assumptions. For each training sample, we randomly select a prefix of the sequence to serve as a prompt, which remains unmasked with a probability of $p = 0.8$. The prompt length is uniformly sampled from the range [0%, 40%] of the total sequence length.

During finetuning, prompts are used only for tasks that require them (e.g., zero-shot TTS, voice conversion). For tasks that do not require prompts, such as speech enhancement, no prompt is used. In all cases, the prompt mechanism used during fine-tuning is kept consistent with that of pretraining.

Question 4: Section 3.5: The explanation of prompts and conditioning is difficult to follow...

We are happy to provide more details for clearer explanation. For all tasks that require prompts, the prompts refer to the SSL tokens of the prompt speech. Specifically, for zero-shot TTS, VC, and TSE, the prompt corresponds to the SSL tokens of the reference speaker's speech. The condition varies depending on the task: for TTS, it refers to the text embedding derived from the input text; for VC, TSE, and SE, it corresponds to the speech features of the input speech, such as the source speech for VC, the mixed speech for TSE, and the noisy speech for SE. These features can be either mel-spectrograms or SSL features. In this work, we use w2v-BERT 2.0 SSL features, which are temporally aligned with the target SSL tokens, that is, they share the same sequence length along the time axis.

Question 5: Section 4.1: The source of the “200K hours” of training data is not clearly explained...

The additional 200K hours of data come from Emilia-Large [3], which primarily consists of YODAS2 [4] data processed through the Emilia pipeline.

Question 6: Section 4.2: The experimental setup is hard to interpret...

Comparing with open-source models can be challenging due to differences in experimental settings. However, for some tasks such as TTS and VC, certain baselines are trained using the Emilia dataset, and our fine-tuning data is a randomly selected subset of the same dataset, albeit with significantly less data. For the lip-to-speech task, we use the same training data as the baseline models.

For internal comparisons within our method, such as varying the fine-tuning data size or the number of trainable parameters, we ensure that all other variables are held constant. We appreciate your suggestion and will conduct more rigorous and controlled experiments in future work.

Question 7: Section 4.2.1: I could not find a discussion of how the length parameter n...

I would like to clarify your question: does the length parameter n refer to the target speech length during testing? That is also our understanding. Since our method does not rely on explicit alignment between text and speech or a duration predictor, we need to specify a target length at inference time. Following previous NAR TTS models that similarly do not use duration predictors (e.g., MaskGCT, E2TTS, F5-TTS), we adopt a simple rule-based approach to determine the target length:

target len. = (prompt len. / prompt phone count) × target phone count.

Question 8: Section 4 – Evaluation Metrics..

• Thank you for your suggestion! Due to space limitations, we provide subjective evaluation results for zero-shot TTS and speech enhancement in Appendix D.3 Results of Subjective Evaluation, taking these two tasks as representative cases. We evaluate QMOS and SMOS to assess speech quality and speaker similarity, respectively.

• We also provide the results of UTMOS for TTS and VC in response to Reviewer c6Ub's Question 2.

• We do not include signal-level metrics such as PESQ in the main paper, as we find these metrics are not well-suited for evaluating generative-model-based methods. This observation is consistent with prior works [5, 6]. We add these results and will include them in the appendix of the paper.

Thanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.

Reference

[1] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019: 4171-4186.

[2] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.

[3] He H, Shang Z, Wang C, et al. Emilia: A large-scale, extensive, multilingual, and diverse dataset for speech generation[J]. arXiv preprint arXiv:2501.15907, 2025.

[4] Li X, Takamichi S, Saeki T, et al. Yodas: Youtube-oriented dataset for audio and speech[C]//2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023: 1-8.

[5] Li X, Wang Q, Liu X. Masksr: Masked language model for full-band speech restoration[J]. arXiv preprint arXiv:2406.02092, 2024.

[6] Zhang J, Yang J, Fang Z, et al. Anyenhance: A unified generative model with prompt-guidance and self-critic for voice enhancement[J]. arXiv preprint arXiv:2501.15417, 2025.

Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.

Weakness 1: No unified inference pipeline: While pre-training is unified, fine-tuning still requires task-specific configurations (e.g., frame-level vs. non-frame-level conditioning logic).

The unified fine-tuning and inference logic for both frame-level and non-frame-level conditions indeed simplifies and streamlines the overall pipeline. In our work, the non-frame-level conditional task mainly refers to the TTS task, while all remaining tasks (voice conversion, speech enhancement, target speaker extraction, and lip2speech) rely on frame-level conditions.

To handle non-frame-level conditions, we prepend the condition embeddings to the speech representation sequence along the time axis and apply self-attention over the combined sequence. Recent studies [1,2,3] have shown that for non-autoregressive TTS systems, it is effective to directly concatenate text embeddings with speech representations at the feature/channel level without requiring upsampling or alignment. Since text embeddings are much shorter than speech representations in sequence length, the remaining positions can simply be filled with padding embeddings.

This approach allows us to unify both frame-level and non-frame-level conditional tasks: we concatenate the task-specific condition with the speech representations along the feature/channel dimension, rather than the time dimension, and pad the condition to match the sequence length when necessary.

Weakness 2: Lack of cross-task analysis: While Metis-Omni is introduced for multi-task fine-tuning, detailed cross-task interference analysis is not provided (e.g., does speech enhancement degrade TTS?).

Thank you for the suggestion. We will provide more details about the inference process of Metis-Omni.

We observe that Metis-Omni achieves performance that is on par with or even surpasses task-specific fine-tuned models for speech enhancement and target speaker extraction. One exception is the WER metric in zero-shot TTS, where performance is slightly lower. A possible explanation is that, under the same number of training steps, multi-task fine-tuning allocates fewer effective steps to the zero-shot TTS task, resulting in less task-specific optimization.

Additionally, regarding the inference process: All frame-level conditional tasks (i.e., voice conversion, speech enhancement, and target speaker extraction) share the same adapter, and no additional task ID is required during inference. For zero-shot TTS, the inference process is identical to that of Metis-TTS, and the adapter used for frame-level conditional tasks is not activated during TTS inference. Metis-Omni also supports some generalized inference patterns, even if they were not explicitly encountered during training. For example, we can perform text-guided inference for target speaker extraction and speech enhancement by providing the transcript of the target speech (obtained via an ASR system) as an additional condition. As shown in Table 6, this leads to improved performance in some cases.

Question 1: Can the authors further unify inference logic so that new tasks don’t require any architectural changes?

We propose a unified inference logic for both frame-level and non-frame-level conditional tasks, as raised in Weakness 1, by concatenating the task-specific condition with the speech representations along the feature (channel) dimension rather than the time dimension, and padding the condition as needed to match the sequence length.

Question 2: How does Metis handle long-form or streaming generation tasks?

In this work, we primarily focus on non-autoregressive (NAR) speech generation tasks and design our model for non-streaming scenarios. However, recent studies [4] have also demonstrated the potential of masked generative models in streaming inference. Specifically, the speech can be split into chunks, where masked generative modeling is applied within each chunk, and autoregressive generation is performed across chunks. We will explore these directions in future work.

Question 3: How sensitive is the model to the length or content of prompts used in zero-shot or few-shot generation?

Since our models are trained on speech segments shorter than 30 seconds (due to limitations of the Emilia dataset), we recommend that the prompt length not exceed 15 to 20 seconds. During training, the maximum prompt length was set to half of the total input length. However, we believe this constraint is largely dependent on the training data, and longer prompts could be supported if trained accordingly.

Thanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.

Reference

[1] Eskimez S E, Wang X, Thakker M, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts[C]//2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024: 682-689.

[2] Chen Y, Niu Z, Ma Z, et al. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching[J]. arXiv preprint arXiv:2410.06885, 2024.

[3] Ning Z, Chen H, Jiang Y, et al. DiffRhythm: Blazingly fast and embarrassingly simple end-to-end full-length song generation with latent diffusion[J]. arXiv preprint arXiv:2503.01183, 2025.

[4] Arriola M, Gokaslan A, Chiu J T, et al. Block diffusion: Interpolating between autoregressive and diffusion language models[J]. arXiv preprint arXiv:2503.09573, 2025.

Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.

Weakness 1: Quality: The paper (the text) might be lacking details in terms of reproducibility (even though the author checklist suggests that the code will be made available). Some of the evaluation criteria may not be well-suited for the individual task and/or they do not show demonstrate the model differences well.

Thank you for the comments! We have made every effort to improve the reproducibility of our work. We plan to open-source the code for each model, along with detailed configuration files specifying hyperparameter settings and model architecture.

For evaluation, all baseline models are open-source and evaluated on public benchmarks. We will also provide scripts for computing evaluation metrics on each test set.

Weakness 2: Originality: Based on the text, it seems that the current paper follows reference [12] quite closely with some differences on the text embedding component. If the reference [12] has been published in a peer-reviewed venue, it may make the novelty of this paper somewhat limited. Based on the current pdf, reference [12] is an arxiv paper. If that is still the case, this paper might satisfy the novelty criterion.

Reference [12], MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer, has already been published at ICLR 2025. We cite MaskGCT in multiple parts of our paper, as both our work and MaskGCT adopt masked generative modeling as the core generative paradigm.

However, our contributions go beyond MaskGCT in several key aspects:

• First, while MaskGCT focuses solely on zero-shot TTS, our work extends masked generative modeling to a broader set of speech generation tasks, including zero-shot TTS, voice conversion (VC), speech enhancement (SE), target speaker extraction (TSE), and a multimodal task, lip-to-speech generation.

• Our goal is not to apply masked generative models to solve a single task, but rather to build a unified foundation model with large-scale pretraining, capable of efficiently adapting to a wide range of speech generation tasks. To this end, we design a unified pretraining mechanism, which enables efficient fine-tuning across diverse speech generation tasks. In the pretraining stage, we adopt unconditional masked generative modeling as the objective, without relying on any task-specific conditioning.

Masked generative modeling has shown effectiveness across various domains, e.g., MaskGIT (image generation), MaskGVIT (video generation) in vision, and MaskGCT (TTS), NaturalSpeech 3 (TTS), AnyEnhance (SE and TSE), MaskSR (SE), and in audio, but all of these are task-specific models. In contrast, our work aims to build a unified framework that aims to generalize masked generative modeling for diverse speech generation tasks.

Weakness 3: Clarity: It is good to have multiple experiments on various tasks but it might have been better to share more details on the core model and its training details.

We provide the details of the pre-training process in Section 4.1, including the model architecture, dataset, and training strategy. For each downstream task, we describe any differences from the pre-training setup in the corresponding experimental sections. Due to space limitations, we are unable to provide more details in the main text. However, we will do our best to include additional information, including detailed model architecture, in the appendix.

Question 1: Please comment on whether reference [12] has been already published in a peer-reviewed venue.

We discuss this question in Weakness 2.

Question 2: In some experiments, DNSMOS and NISQA evaluations have been used. However, these scores are discriminative enough to compare the quality of various systems. The values themselves may suggest that the system is working but they do not necessarily imply that the proposed model is significantly outperforming another one (for example, Vevo vs. Metis in Table 2). Could you please comment on this? Have you considered using other speech evaluation metrics such as the UTMOSv2 scores?

Yes, since DNSMOS and NISQA are commonly used for evaluating SE and TSE tasks, in our work, we also adopt DNSMOS for evaluating zero-shot TTS and VC, and additionally use NISQA for VC, in order to maintain a relatively streamlined evaluation setup. However, we note that these metrics are not always reliable. For example, DNSMOS and UTMOS are often inaccurate on non-English speech. We present these metrics mainly for reference, and for zero-shot TTS and VC we believe SIM and WER are more meaningful. In addition, based on our experience, SIM computed with WavLM embeddings also correlates with speech quality to some extent. We also provide subjective evaluation results in Appendix D.3 Results of Subjective Evaluation.

We are happy to supplement the UTMOS scores below for reference:

Zero-shot TTS:

VC:

Question 3: Minor mistake: Section 4.2.2. mentions "HireSpeech++" in the text which should be HierSpeech++ as mentioned in Table 2.

Thank you for pointing this out! We will address and fix it.

Thanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.