Unispeaker: A unified speech generation model for multimodality-driven voice control

The main weaknesses of this paper are a lack of clarity in some experimental setups, specifically as follows:

1. In Table 2, comparisons between the UniSpeaker and baselines are not fair enough. The official checkpoints of these baselines may be trained on different datasets. For example, on the TextTTS task, the training and testing sets in this paper adopt academic reading-style datasets like VCTK. However, CosyVoice-Instruct [1] uses an internal commercial dataset, which creates a domain gap with the MVC benchmark used in this paper. Therefore, the experimental results are not convincing. Perhaps a fairer comparison could be made by using the CosyVoice-Instruct model fine-tuned on the same training dataset. I can also understand that many baselines use non-open-source data, making it difficult to fully reproduce them on the same datasets used in this paper. At least, the authors should include the specific training datasets used for these baseline models in Table 2.

There are also some minor weaknesses:

2. In terms of the proposed dataset and benchmark, this paper primarily collects several open-source datasets to form its dataset and establishes the MVC benchmark, so the contribution is relatively modest. But I think the MVC benchmark could serve as a valuable tool for evaluating multimodal-driven voice control. In terms of novelty, this paper seems to integrate existing algorithms to propose a unified model, lacking some insightful algorithms or results.

3. Instead of using the weights $\alpha$ for interpolating the speaker embeddings obtained from face and textual descriptions, the multi-condition classifier free guidance (See Section 3.2.1 in [2]) may be a better choice for multi-modal voice control.

4. Using only a global speaker embedding may limit timbre reconstruction quality, as time-averaged pooling removes some time-varying timbre details. To address this, prior works like NaturalSpeech 3 [3] have incorporated both global speaker embeddings and detailed embeddings for enhanced speech reconstruction. The performance could be further improved by incorporating finer-grained speaker embeddings.

5. Additional details on the inter-modal and intra-modal alignment learning process may be included in Figure 2 to enhance clarity.

6. Typo: in Appendix F, line 1049, Figure 8 -> Table 8; in the title of this submission, Unispeaker -> UniSpeaker.

7. This paper advances the development of multimodal-driven speech synthesis, a technology that raises certain ethical issues for society. Therefore, an ethics statement should be added to the end of the article.

Overall, as mentioned in Weakness 1, the training sets of different models are different, making the experimental results not promising enough. Therefore, I rate it 5. If there are any errors or inappropriate aspects in the points I've raised, please feel free to contact me at any time.

[1] Du, Zhihao, et al. "Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens." arXiv preprint arXiv:2407.05407 (2024).
[2] Brooks, Tim, Aleksander Holynski, and Alexei A. Efros. "Instructpix2pix: Learning to follow image editing instructions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[3] Ju, Zeqian, et al. "Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models." arXiv preprint arXiv:2403.03100 (2024).

First, I believe this work represents a commendable extension to the existing zero-shot TTS framework. However, the paper's most significant weakness lies in its lack of novelty and failure to introduce new insights to the community.

1. All methods employed are merely combinations of existing frameworks, many of which have already been utilized in the TTS domain. Moreover, the conclusions drawn do not offer any groundbreaking insights or provoke thought among peers in the field. For instance, the

2. The use of contrastive learning to align features from other modalities with speaker embeddings, as well as the two-stage approach (AR + diffusion/flow matching), have been employed in numerous TTS works (SeedTTS, CosyVoice, FireRedTTS, Tortoise-TTS, SoundStorm...).

3. At the results level, while the authors do surpass state-of-the-art solutions on their proposed benchmarks, there is no further breakthrough. For example, how well does the model adhere to out-of-domain text descriptions (e.g., generating speech as if spoken by a dragon from a life story)? Or, can the face images be from animations or non-realistic faces? (In fact, I suspect the results may not be satisfactory due to the lack of diversity in the training data).

• Originality: this paper lacks originality. The whole pipeline is just a naive combination of existing work: the backbone is CosyVoice, and the modality-specific encoders are all from previous existing pretrained models. The only module that needs to be trained is the multimodal voice aggregator, which is also based on memory mechanism. And the so-called self-distillation is just to adapt the CFM to condition only on the speaker embeddings, i.e. removing the mel-spectrogram prompt. In addition, the proposed benchmark is also just a repetition of existing evaluation methods, including WER, MOS, and speaker-embedding-based similarity metrics (CosyVoice has the exact same formulation).

• Missing experiments: since this is a model that can handle multiple conditioning modalities, experiments should be conducted to assess the performance of conditioning on different numbers of modalities. For example, how good is the performance of adding additional face information together with text descriptions compared to only text / only face? Besides, the paper fails to point out the number of K-V vectors used in the experiments. This is a very important hyper-parameter, since it directly defines the codebook size. Moreover, experiments should also be conducted to assess the effect of different numbers of K-V vectors and the utilization rate. It would also be more interesting to show if the different learned K-V vectors actually represent different properties of the voice, like pitch, gender, age, etc.

• Writing accuracy:

  • line 290, for the shape of neg-disentangled distribution, instead of 1xN-1, it should be 1x(N-1)
  • line 304, instead of L_{INTRA}, it should be L_{INTER}, i.e. inter-modality loss is the InfoNCE loss
  • line 312, instead of “the CFM is fine-tuned to predict the source speech”, it should be “the CFM is fine-tuned to predict the converted
  • speech”, since the input speaker embedding is from the converted speech *.Table 2, SST, SSC, SSD, and WER should be all in the range of (0, 1). Add (%) in the table or caption

The paper presents a well-rounded method and extensive experimental analysis. While I have only minor concerns (detailed in the question section), no significant issues were identified in the current draft.