EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions

1. The core or central motivation of this paper is a bit unclear to me. Is the motivation to include visual ability in a speech-language model (which is not reflected in its name EMOVA), or to include speech synthesis with emotion in a speech-language model? I agree with the authors that the system is the first to achieve these functionalities in a single system, but I am not sure how those functionalities are connected to be put in a single paper. Why being able to understand an image is crucial to emotional speech synthesis, or why emotional speech synthesis is crucial for understanding an image? Taking the example the authors show, "being able to write a sad poem about the image" and "being able to synthesize sad speech about the image" are not strongly connected in my opinion. Note that this problem also exists in the experiments since none of the test sets used actually reflect the importance of jointly understanding image and interacting in speech with emotions, apart from case studies.
2. Regarding novelty, which is also related to motivation, I need more clarification on which parts of the system haven't been done before. For example, there are full-duplex systems [1,2] that can understand speech and output speech. There exists a lot of work in the TTS community working on disentangling style and text representations for speech synthesis [3,4]. There is work controlling emotion in TTS using natural language prompts [5]. There also exist systems that can understand both speech and visual inputs [6]. I understand that this is the first single speech-LM that achieves all the functionalities above (which I gave credit to in the strength part), but the actual approach for each functionality does not seem to be novel to me.
3. The experiments showed that EMOVA outperforms GPT-4v on various benchmarks, but most of them are in the training set (hence not zero-shot).

[1]. https://arxiv.org/pdf/2408.02622 [2]. https://arxiv.org/abs/2409.06666 [3]. https://arxiv.org/abs/2306.07691 [4]. https://arxiv.org/abs/1810.07217 [5]. https://arxiv.org/abs/2406.06406 [6]. https://arxiv.org/pdf/2406.15704

1. The descriptioins of the proposed method are not clear. For example, as the authors mentioned, the semantic-acoustic disentangled speech tokenizer is one of the contributions of this paper. However, the introduction to the disentanglement in Section 3.2 is very simple and it is difficult for readers to get the key ideas.
2. The experimental results are not convincing enough. Only the overal performances of the proposed model on different tasks were evaluated. There are no ablation studeis on the key modules of the proposed method which makes it difficult to accept the contributions of the proposed method. In Section 5.3, there were no comparative models for evaluating the emotion-rich spoken dialogue, and only the synthetic data was used for evaluation.

1. Since only the semantic embedding of speech is quantified to speech units, due to the semantic-acoustic distengled modeling of speech tokenizer, the emotional information in the input speech is not conveyed to the LLM. The model generates emotion labels only based on vision and text modalties, which is not often the case in real dialogue.
2. On vision-language benchmarks, SOTA VLLMs are compared. While on speech benchmarks, only omni-modal LLMs are compared. SOTA SLLMs such as Qwen-Audio and SALMONN are not compared. So the conclusion "EMOVA obtains state-of-the-art performance on speech benchmarks" is doubtful.

The key negative of this paper is the presentation, which I find a bit cramped. The presentation of key results, intuition and the relation to the literature is good, but the description of the key parts of the architecture (speech tokenizer / detokenizer) and the experimental procedure is delegated into the appendix, calling into question the rigor in the main part of the paper. Also, I find the constant referencing of upcoming sections a bit tedious for the reader.

Secondly, I find the lack of human evaluation for the speech synthesis as an omission. The examples presented in the example page are impressive, but the voice sounds robotic, which calls into question whether the end to end approach is practical in this scenario, or having a separate TTS module which can be independently optimized would be preferable.

Finally, some key information is missing, i.e. whether the model will be released as open-source (and the relevant license) and the amount of compute used for training (what kind of GPUs, how many, for how long). I find this information to be essential when presenting a new model to the community.