Emotional Face-to-Speech

We are grateful for your kind words and appreciating the significance of our contributions, and we try our best to address your questions as follows.

Q1: Impact of expressiveness variation

Thank you for the insightful suggestion. In this paper, we use one-hot emotion labels to learn identity-agnostic emotional embeddings, ensuring that variations in expression do not significantly affect the generation results. In the future, we plan to incorporate emotional intensity as an additional condition to achieve more natural speech synthesis.

Q2: Extra user study

Thank you for the valuable suggestion. Per your suggestion, we have supplemented an extra objective evaluation with larger participants and new MOS metrics for both attributes. Please refer to the Q5 response for Reviewer 7YBQ for the results, which show that DEmoFace outperforms in naturalness, identity timbre, and emotional prosody consistency. We will revise our manuscript accordingly.

Q3: Clarification on one-to-many mapping issue in continuous speech features

The one-to-many mapping of speech generation denotes that multiple speech sequences can possibly correspond to the same text sequence with diverse pitch, duration and prosody, making the synthesis speech distribution multimodal instead of unimodal. The issue arises from attempting to generate continuous speech features, such as mel-spectrograms, which are highly correlated over time and frequency, leading to over-smoothing during frame-level prosody or linguistic predictions [1]. In addition, in this paper, we focus on extracting identity styles and emotions from facial features, rather than semantics from lip motion. We will certainly clarify this in future version.

[1]: FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model

Q4: Clarification on t-SNE experiments

To further compare our method with real speech data and ablation methods, we have added relevant visualizations referring to Figures 1 and 2 in the anonymous link https://anonymous.4open.science/r/demoface. The results demonstrate that our method shows a similar clustering distribution to that of real speech.

Q5: Implementation of baselines

To ensure a fair comparison, we have included detailed implementation details in the appendix. The speech quality differences in some baselines, compared to original demos, stem from their limited dataset size and lenient evaluation, which may lead to overfitting. For example, they relied on the limited-vocabulary GRID dataset or the speaker-limited V2C dataset. In contrast, our method introduces a mixed dataset and is evaluated on unseen speakers, reflecting more challenging real-world scenarios.

Q6: Clarification on related works, references, and writing

We appreciate your valuable feedback, which has helped us refine our work.

1. We will revise the related work section in detail: Compared with acoustic-guided methods, these non-autoregressive (NAR) methods suffer from over-smoothing, less diversity, and complex alignment issues, despite fast inference. Compared with visual-guided methods, they still pose the above issues by introducing NAR for coarse mel-spectrogram generation and diffusion for refinement and lacking an efficient conditioning paradigm. In contrast, our DEmoFace can flexibly leverage both visual and acoustic conditions while dynamically aligning text for higher-quality and more diverse speech generation, offering practical guidance for optimizing the promising modeling paradigm.
2. We will add the suggested references and compare them with ours.
3. We will refine our writing to improve clarity and enhance readability.

Q7: Training cost comparison in Figure 5

Figure 5 of main text shows a comparison of metrics in the same model checkpoint, under the same training steps or costs, which demonstrates that curriculum learning can improve training efficiency.

Q8: Should facial expressions be called the ``most visual indicator''

Our proposed eF2S problem aims to infer timbre and emotional prosody solely from visual cues instead of acoustic cues, with facial expressions serving as the most relevant source of emotional information naturally.

Q9: Why need extra an LRS3 subset for learning more semantic units

The "semantic unit" refers to text-rich units containing contextualized linguistic details. The audiovisual datasets used in this work exhibit limited linguistic diversity in acted emotional speech, which constrains the model to accurately generate speech for unseen text. Therefore, we incorporate a subset of LRS3 to enhance semantic learning from real-world scenario.

Thank you very much for your insightful comments and suggestions, motivating us to rethink a more comprehensive experiment evaluation. We try our best to address your questions as follows.

Q1: Emotion ambiguity between text and face

Thank you for your insightful comment. Determining the dominant and subordinate modalities is crucial for resolving ambiguities in multimodal emotion generation. However, in our eF2S task, we focus on inferring timbre and emotional prosody solely from visual cues, with facial expressions serving as the dominant modality rather than textual cues. In the future, we will explore integrating face, text, and speech for more consistent emotion generation by dynamically adjusting modality impacts through cross-modal shift estimation.

Q2: Dataset-wise experiment for out-of-domain robustness

We appreciate your insight for a more thorough evaluation of our method. To address your concern, we have supplemented dataset-wise experiments on the out-of-domain LRS2 benchmark. Specifically, due to the limited dataset size, we conducted full fine-tuning of the base model for 20 epochs on each of these four datasets to learn dataset-specific knowledge and tested them on LRS2. Please refer to the Q2 response for Reviewer kkEx for quantitative results, and we will include this evaluation and discussion in the future version.

Q3: Clarification on t-SNE experiment

To further compare our method with real speech data and ablation methods, we have added the requested visualizations, which can be found in Figures 1 and 2 in the anonymous link https://anonymous.4open.science/r/demoface. The results demonstrate that our method exhibits a clustering distribution similar to that of real speech.

Q4: Extra ablation study on emotion conditioning

Thank you for helping improve the clarity of our paper. We have conducted an ablation study on the $\boldsymbol{c} _\text{emo}$ condition. The results below show that $\boldsymbol{c} _\text{emo}$ enhances speech naturalness and expressiveness without significantly affecting SpkSim or WER, confirming that DEmoFace effectively decouples different conditions. The associated t-SNE visualizations can be found in the anonymous link provided above.

Q5: Limited subjective evaluation

Following the setup of Face-TTS [1] using 17 evaluators, we initially conducted similar evaluations, and we acknowledge that a larger number of evaluators ensures more reliable results. Per your suggestions, we have expanded the evaluation to 50 evaluators with 15 samples each, and we also introduce new MOS metrics for both timbre （MOS$_\text{id}$） and prosody （MOS$_\text{emo}$） evaluation as Reviewer 4KNc suggested. The new evaluation results with 95% confidence intervals are as follows:

The results show that DEmoFace outperforms EmoSpeech and Face-TTS in naturalness, identity timbre, and emotional prosody consistency. We will revise our manuscript accordingly.

$1$ Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech

Q6: Clarification on existing method limitations

The limitations of existing methods can be categorized into two aspects: 1) task-level, where they cannot jointly model speaker identity and emotion solely from visual cues, and 2) method-level, which involves issues like one-to-many mapping with limited diversity or inefficiencies in continuous NAR or discrete AR frameworks. Building on these insights, we propose a novel discrete diffusion framework for a novel task—emotional face-to-speech generation.

Q7: References to be discussed

Thank you for your suggestion. We will certainly incorporate the relevant discussion and citations [2-3] in the future version.

$2$ : Deep facial expression recognition: A survey \

$3$ : Emotion Recognition and Generation: A Comprehensive Review of Face, Speech, and Text Modalities

Thank you very much for your positive comments and efforts in reviewing our manuscript. We try our best to address your questions as follows.

Q1: Limited novelty

Thank you for the opportunity to clarify the distinctions from previous methods. Although DEmoFace builds on existing discrete diffusion models (DDMs), our work makes key contributions.

1. While DDMs have shown great promise in text generation, their potential for speech generation—particularly with RVQ codec tokens—remains underexplored.
2. Existing DDMs struggle with effective guidance for multi-conditional generation. Our EPFG addresses this challenge by providing more efficient guidance, supported by strong empirical evidence, while also introducing theoretical insights that enhance the understanding of multi-conditional generation in DDMs.
3. DEmoFace is a unified framework for both acoustic- and visual-guided speech generation, with extensive results demonstrating its efficiency.

Q2: Limited data size

To address concerns about model generalization, we randomly sample 1,500 utterances from the LRS2 benchmark to evaluate its performance on real-world out-of-domain data. Results demonstrate that DEmoFace consistently achieves high-quality generation and outperforms the Face-TTS, even for unseen speakers and content.

Furthermore, as suggested by Reviewer 7YBQ, we have conducted dataset-wise experiments on LRS2, performing full fine-tuning of the base model for 20 epochs on four datasets to capture dataset-specific knowledge. The results are as follows:

We have four key findings: 1) limited semantic content (low utterance and word counts) leads to higher WER; 2) limited speaker diversity (a small number of speakers) negatively affects SpkSim; 3) emotion-rich but small datasets like RAVDESS may not accurately reflect real-world distributions, as acted emotions tend to be exaggerated; and 4) mixed-dataset training improves generalization across all aspects on out-of-domain real-world data.

In future, we will expand the dataset to enhance diversity and real-world applicability.

Q3: Clarification on efficiency analysis

We measure latency on 4090 GPU with a mini-batch size of 1 and 32 utterances, and function evaluation numbers (NFE) of 32 and 64. Latency is averaged over the test set utterances, and we report the Real-Time Factor (RTF), which indicates the time (in seconds) required to synthesize one second waveform. The results show that our method with NFE=32 and batch size=1 has the potential to build a real-time TTS system, being 2 times faster than real-time. In future, we will optimize inference efficiency by developing accelerated sampling techniques.

We are grateful for your positive feedback and constructive suggestions, and try our best to address your concerns as follows.

Q1: Facial expression ambiguity

Thank you for your insightful comment. In this paper, we leverage a pre-trained facial expression recognition model to generate one-hot emotion labels for learning emotional embeddings, serving as a plug-and-play component. This allows for the integration of stronger models, such as micro-expression recognition, to enhance expression precision and reduce expression ambiguity. We will incorporate this discussion in the future version.

Q2: Limited data size and multilingual extension

Thank you for your insightful comment. To address concerns about model generalization, our submitted paper ensures that test set speakers are unseen for RAVEDSS and MEAD, demonstrating strong generalizability. In addition, per Reviewer kkEx's suggestion, we have supplemented an out-of-domain evaluation on LRS2 by randomly sampling 1,500 utterances, with results provided in the Q2 response to Reviewer kkEx. Experimental results show that DEmoFace achieves better generalizability and outperforms the Face-TTS even with unseen speakers and semantic content.

Expanding to unseen languages is an interesting challenge that requires multilingual datasets and handling linguistic and phonetic differences. However, this is beyond the scope of our eF2S task, which focuses on generating speech aligned with facial identity and emotional expression. In the future, we plan to explore multilingual speech generation using International Phonetic Alphabet (IPA) embeddings.

Q3: Training computational efficiency with standard TTS model

Thank you for your valuable suggestion. Following DiTAR [1], which evaluates the FLOPs of standard Non-AutoRegressive (NAR) TTS models, we compare DEmoFace with both Continuous (Cont.) and Discrete (Disc.) NAR methods at Number of Function Evaluations (NFE = 32). As shown in the following table, DEmoFace achieves comparable training efficiency to other NAR TTS models. In the future, we will further optimize its efficiency and extend DEmoFace from cross-modal face-to-speech to standard TTS to explore its broader impact.

[1] DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation

Q4: Comparison with acoustic-driven methods

In Table 1 of the main paper, the key difference between acoustic-guided and visual-guided methods lies in the modality used to model timbre and emotional prosody, as reflected in SpkSim and EmoSim. However, both approaches rely on textual modality to guide semantic content generation, as indicated by WER. Our acoustic-guided DEmoFace* outperforms the visual-guided DEmoFace due to a smaller distribution shift from ground-truth (GT) speech. This is because DEmoFace* is trained and tested with speech modality, while DEmoFace is trained with GT speech but tested with vision modality.

Q5: Superiority of EPFG

The key difference between EPFG and vanilla PFG in discrete diffusion lies in multi-condition disentanglement versus aggregation. As stated in the main text (lines 401 and 432), EPFG significantly enhances multi-conditional generation quality, mitigating semantic confusion caused by aggregation. We will clarify this more explicitly in the future version.