EcoFace: Audio-Visual Emotional Co-Disentanglement Speech-Driven 3D Talking Face Generation

Dear reviewer, we sincerely appreciate your careful and accurate understanding of our approach. We hope our response fully resolves your concerns.

Q1: A limitation of the objective metrics, such as LVE, is that they do not account for type of error. A larger LVE in the articulation of, say, /k/ might be insignificant, but a large error in the articulation of, say, /b/ is highly problematic. I realize these are standard metrics, but I still see these as problematic for this reason.

A1: We agree with your insightful comment regarding the limitation of objective metrics like LVE. We chose to use LVE because it is a widely accepted evaluation standard. However, we also recognize the issue you pointed out: LVE does not account for the type of error. Specifically, while LVE may show a large error in the articulation of /k/ due to the mouth's opening being different from the ground truth, this may not necessarily indicate an error in pronunciation. In contrast, LSE-C and LSE-D address this issue more effectively, as they assess the matching of continuous lip movements to the speech in the time domain. These metrics can distinguish cases where a large LVE might be insignificant, like with the articulation of /k/, from cases where a large error is truly indicative of a pronunciation mistake, such as with the articulation of /b/ where the lips are closed.

When both LVE values are large, it can lead to misjudgment, but using LSE-C and LSE-D allows us to avoid such situations. We also see potential improvements for this metric in mesh-driven evaluations. Since the SyncNet expert used in Wav2Lip is trained with both audio and image inputs, a possible next step would be to train a specialized SyncNet expert using 3D lip vertex from the mesh and audio as inputs. This approach, if applied to large-scale datasets, could significantly enhance the evaluation of mesh-based talking faces.

Q2: How is the level of emotion in the training data assigned?

A2: In the RAVDESS dataset, the emotion intensity is divided into two categories. Therefore, we assign emotion intensity labels of 0 and 1 to the training data. These labels are primarily used in the calculation of triplet loss, where we select samples with the same intensity label as the anchor as positives, and samples with different intensity labels as negatives.

Q3: On line 135 — is an element of s \in R^{n} the probability of that expression being present?

A3: In line 135, s is a one-hot embedding vector. Assuming there are n speakers, s is a vector of length n containing only 0 and 1. If we want to condition the model on the speaking style of the i-th person (where 0≤i<n), then S_i=1 and all other elements of s are set to 0.

Q4: In Equation (7) how is alpha set?

A4: In Equation (7), we set α to 0.5. This choice was made after exploring alternative values, such as α=0.25 and α=0.7. When α=0.25, positive and negative samples become too close, reducing the model's ability to effectively separate subtle differences in emotion intensity. On the other hand, with α=0.7, the final loss converges to a larger value compared to α=0.5. Furthermore, t-SNE visualizations of the encoded feature space revealed that the separation of emotion intensities was similar for both α=0.5 and α=0.7. This indicates that at α=0.5, the model has already achieved an upper limit in separating different emotion intensities, and further increasing α does not yield additional benefits. We have included t-SNE visualizations in the appendix, illustrating the distribution of features for different emotion intensities in a single emotion space. These plots demonstrate that the features are well-separated, with distinct clusters for the two intensities within the same emotion class under the α=0.5 setting.

Q5: In the paragraph following Equation (7), should z_l} be z_{i} to match the equation?

A5: Thank you for pointing out this issue. You are correct, and I have revised the notation in the paragraph following Equation (7) to use z_i, ensuring consistency with the equation.

Q4: Although you retrained FaceFormer and CodeTalker on RAVDESS and HDTF, you did not retrain EMOTE which seems to provide better results than your approach on MEAD that the model was trained on. For fair comparison could you perform this experiment by retraining EMOTE on the RAVDESS too?

A4: Thank you for highlighting this important point. We appreciate the opportunity to clarify our experimental choices and to address your suggestion. Initially, we chose not to retrain EMOTE because our aim was to evaluate the generalization capability of our model, which was trained on the RAVDESS dataset, to a new dataset like MEAD. Both RAVDESS and MEAD are emotion-rich datasets featuring audio-visual recordings of actors expressing different emotions. By not retraining EMOTE, we intended to measure how well our model could adapt to a dataset it had not been trained on, compared to EMOTE’s performance on the dataset it was specifically trained on (MEAD).

After receiving your valuable feedback, we revisited this aspect to better understand why our model underperforms on MEAD compared to EMOTE. We observed that, due to the influence of speaker styles, some test samples in the MEAD dataset exhibit relatively subdued emotional expressions in speech. This weakens the performance of our model, which relies on extracting emotional features from speech. In contrast, EMOTE directly specifies emotions through labels for generation, making it immune to such effects.

In response to your suggestion, we have retrained EMOTE on both the RAVDESS and HDTF datasets. The results of this experiment are presented in the table below.

We acknowledge that EMOTE’s use of explicit labels provides a more direct mapping for emotion-driven generation. However, our approach focuses on learning nuanced emotional details from speech sequences, which enhances the granularity of emotional dynamics. These strengths are reflected in the superior performance of our model on RAVDESS.

Q5: Table 3: Can you show the results on each dataset separately and for each emotion? Since your model was trained on RAVDESS it may contain some bias towards that dataset when compared with EMOTE.

A5: Thank you for pointing this out. We have now provided the results for each dataset and emotion separately.

Here are the results of the User study on the Vocaset：

Here are the results of the User study on the RAVDESS：

Here are the results of the User study for each emotion on the RAVDESS:

To further address the potential bias, we conducted the same user study comparing the retrained EMOTE model with our original model on the RAVDESS dataset. The results of this additional user study are presented below.

As shown in the table, our model achieves better performance in both the original and retrained emote. This highlights the generalizability and effectiveness of our approach, despite differences in training datasets.

Q6: Sec. 4.3,”Effect of emotion embeddings”: Have you tried to extract content features from an emotional speech signals? How would the model perform in this more challenging experiment?

A6: Thank you for raising this important question. In our original setup, we chose to extract emotion features from neutral speech because Hubert demonstrates a high similarity in content features extracted from speech signals with identical content but varying emotions.

To evaluate the model's performance under this more challenging scenario, we conducted an additional experiment where content features were extracted from emotional speech signals rather than neutral speech. Specifically, we selected one speech sample from each of the seven emotional categories (excluding neutral) to extract the text content. For the emotion features, we used speech and video samples corresponding to the remaining emotions and combined these with the extracted text content as input to generate results. The generated outputs were then evaluated using LSE-C, LSE-D, and VE-FID metrics. The results are presented in the table below.

As shown, even when extracting content from emotional speech signals and subsequently performing emotion swapping for generation, our model maintained robust performance across the evaluation metrics. This demonstrates the resilience and adaptability of our approach in handling emotionally infused inputs.

Furthermore, we have updated the qualitative results for this experiment in the appendix of the revised paper and on our webpage. We hope these supplementary experiments address your concerns and enrich your understanding of our model's capabilities.

We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.

Q1: Speaker-specific modeling. Generation can only be performed for speakers the model has seen during training.

A1:Thank you for pointing out this insightful weakness. We greatly appreciate your suggestion and agree that the ability to generalize to unseen speakers is an important area for improvement.

In our current work, we focus on emotionally expressive talking face generation, where a key challenge lies in the fact that emotional expressions vary significantly across individuals. For instance, during expressions of happiness, the degree of zygomatic muscle activation (e.g., smiling amplitude) can differ between individuals, leading to varied smile dynamics. The speaker-specific nuances are critical for generating realistic and emotionally expressive talking faces. As such, our current design emphasizes speaker-specific modeling, allowing the system to capture and reproduce these fine-grained variations. While this approach achieves strong performance for known speakers, we acknowledge that it limits the generalization to unseen subjects.

To address this concern, we explored a generalization strategy proposed in the UniTalker[1]. Specifically, we introduced a pivot identity into our model, which acts as a generalized speaker label. During training, we replace the ground truth identity label with this pivot identity label with a probability of 10%. This augmentation encourages the model to learn a generalized representation that is not strictly tied to individual speaker characteristics. We retrained our model and performed quantitative analyses on the test set with pivot identity as a condition, and the results are tabulated below.

We observed that similar results were achieved using the pivot identity. Notably, for unseen subjects, we can use the pivot identity as a condition to quickly fine-tune our EMG decoder, adapting it to the speaking style of new individuals. This approach helps improve the model's generalization capability.

We appreciate your feedback, which motivated us to explore this improvement. We believe this addition strengthens the contribution of our work and opens up exciting future directions, such as further balancing generalization and speaker-specific detail preservation.

[1]Fan X, Li J, Lin Z, et al. UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model[J]. arXiv preprint arXiv:2408.00762, 2024.

Dear Reviewer,

Thank you very much for updating your rating based on the additional results and revisions we provided! We greatly appreciate your recognition of the improvements made to the paper, and your feedback has been invaluable in enhancing the quality of our work. Once again, thank you for your time and thoughtful suggestions!

I would like to thank the authors for thoroughly addressing my concerns and providing the requested additional results. Based on this I have decided to update my rating from 5 to 6.

We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.

Q1: LSE-C Range Consistency There appears to be a discrepancy in the LSE-C values reported in this manuscript compared to those commonly cited in the community. For instance, DINet [https://arxiv.org/pdf/2303.03988] lists an HDTF LSE-C value (as GT) of 8.9931, while the value reported in this work is 0.824. Clarification regarding this disparity is needed, as it may suggest a difference in evaluation methodology or algorithm implementation.

A1: Thank you for pointing out the discrepancy in the LSE-C values. As mentioned in several recent works such as FaceTalk and DeepTalk, the LSE-C and LSE-D metrics have been increasingly used to measure lip synchronization accuracy in 3D mesh representations (e.g. FaceTalk[1], DeepTalk[2]). Follow these studies, we adopted the same metrics to evaluate our model's performance. However, as mentioned in these papers, the image rendered by 3D mesh is a grey model of the head, which is fundamentally different from a real human head image, which inherently differ from real human head images. This difference leads to variations in LSE-C values when compared to results evaluated on real human heads, such as those reported in DINet. Despite these discrepancies, the effectiveness of the LSE-C metric in evaluating lip synchronization remains consistent across these approaches.

[1]Aneja S, Thies J, Dai A, et al. Facetalk: Audio-driven motion diffusion for neural parametric head models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 21263-21273.

[2]Kim J, Cho J, Park J, et al. DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation[J]. arXiv preprint arXiv:2408.06010, 2024.

Q2: Definition Clarity for L_emo and L_sync in Ablation Study. The explanation of L_emo and L_sync in Table 4 is missing, which decreases in interpreting the results and understanding the contributions of each component.

A2: Thank you for raising this important issue. L_emo is the last term of Equation 11 and L_sync is the second-last term of Equation 11. I have now revised the manuscript to clearly specify each component.

Q3: Validity and Interpretation of VE-FID Metric Introducing VE-FID as a new metric to measure emotion expression is a great try. However, some results in Table 1 and Table 4 raise questions. For example, the comparison between EcoFace (21.57) and EmoTalk (51.98) shows a large difference, even when EcoFace lacks L_emo (30.88). Does this suggest that EcoFace may outperform EmoTalk on emotion-related metrics even without the emotion-related modules or guidance?

A3: Thank you for your insightful comment regarding the VE-FID metric. Although we did not use L_emo, our emotion encoder (EDE) already disentangles emotion features into an independent emotion space. This allows the EMG module to still receive clear emotion-related signals, albeit without the explicit emotional guidance from L_emo. Thus, while the absence of L_emo leads to a slight reduction in emotional generation quality, the well-disentangled emotion features in our model still enable it to perform relatively better in terms of emotional expressiveness compared to EmoTalk in this specific metric. This suggests that the disentanglement of emotional features in our model contributes significantly to its improved performance even without explicit emotional guidance (L_emo).

Dear Reviewer kKfU,

Thank you again for your time and effort in reviewing our work! We would appreciate it if you can let us know if our response has addressed your concern. As the end of the rebuttal phase is approaching, we look forward to hearing from you and remain at your disposal for any further clarification that you might require.

Thanks in advance,

Paper 10320 authors

Q2: The EMOTE comparison videos on the webpage don’t match the exemplar videos from other methods, such as FaceTalk, 3DiFACE, EMOTE . Could the authors explain the observed differences?

A2: Thank you for bringing this issue to our attention. We acknowledge that the EMOTE comparison videos on the webpage did not perfectly align with the exemplar videos from other methods, such as FaceTalk, 3DiFACE, and EMOTE. This was a challenge noted by other users as well. We initially attempted several solutions discussed in here for the EMOTE model but were unable to resolve the discrepancies. However, during the rebuttal period, the authors of the EMOTE paper shared a new approach to address this issue. After implementing their suggested solution, we were able to successfully align the comparison results. We have since re-conducted the quantitative experiments, and the updated results are provided below.

The updated metrics show that while EMOTE has improved in various benchmarks, our model still outperforms EMOTE.The webpage and the corresponding sections in the paper have been updated to reflect these revised results.

Q3: Triplet loss in eq 7 operates within a single emotion space. Since it focuses on a single emotion, how does EcoFace ensure that emotion intensities for high arousal, high valence (e.g., surprise) are disentangled from those of high arousal, low valence (e.g., anger or fear)?

A3: Thank you for your insightful questions. Although arousal primarily reflects emotional intensity, it also intrinsically encodes features related to emotional type. This means that there is some correlation between emotion type and emotion intensity on the arousal dimension. For example, emotions such as surprise and anger may exhibit similar levels of arousal, but their differences in emotion type on the valence dimension place them in different quadrants of the VA model.

Building on this idea, we employ contrastive loss to construct independent feature spaces for each emotion. This design ensures that even when emotions like surprise and anger have similar arousal levels, their arousal features belong to distinct distributional spaces, as illustrated in Figure 3(b) of the paper. By effectively disentangling arousal representations based on emotion type, we achieve differentiation in emotion intensity across emotion categories.

Once these independent emotion-specific spaces are established, we further refine intensity distinctions within the same emotion type using triplet loss. This step allows us to model subtle variations in arousal intensity, enabling precise differentiation of emotion intensity levels.

To further support this explanation, we have provided visualizations and detailed analyses of intensity differentiation in the Appendix B.1. We hope this clarifies the rationale behind our approach and demonstrates the effectiveness of combining contrastive and triplet loss for disentangling emotion intensities.

Thank you for your review and constructive feedback. We hope our revisions fully address your concerns.

Q1: The GT videos on the webpage show severe issues, such as unnatural mouth closure and a jerky appearance, which don’t match the original implementation. Since EcoFace is trained on meshes obtained using EMOCA, the authors should clarify how their model produces improved animations compared to the GT animations on which it was trained, as shown in their demo videos.

A1: We appreciate the reviewer’s observation. It is indeed true that the mesh estimations generated by EMOCA occasionally exhibit distortions, especially under exaggerated expressions, such as a surprise emotion with intensity level 2. However, in our demo videos, it is evident that models like FaceFormer and CodeTalker, which retrained on the RAVDESS dataset using the same EMOCA-estimated meshes, also produce smoother results.

Upon further investigation, we identified certain cases in the RAVDESS dataset where estimation anomalies were more prevalent, particularly for Actor 12. When conditioning on this actor, mesh-based autoregressive methods like FaceFormer and CodeTalker occasionally generated distorted outputs due to the influence of these anomalies during training. We provide a vdeo here that illustrates a comparative analysis of GT, FaceFormer, and our generated results for Actor 12 in the validation set. It can be observed that GT meshes for this actor already exhibit noticeable distortions, and FaceFormer’s autoregressive approach is affected by these anomalies, resulting in warped or misaligned outputs. In contrast, our model effectively mitigates such issues. This is primarily because we adopt a latent motion distribution space with flow-based priors for training, rather than relying on a direct autoregressive method like FaceFormer.

Although EMOCA occasionally introduces anomalous distortions, most of its mesh estimations are accurate, forming a distribution with a high density around well-estimated cases and a lower density for extreme outliers. Our approach leverages this property by focusing on fitting the overall motion distribution space. By incorporating flow priors, our model smooths the distribution sampling at each time step. Consequently, during inference, the sampled motion trajectories tend to align closely with the high-probability regions of the distribution, effectively avoiding anomalies like unnatural mouth closures or jerky facial movements.

We hope this explanation clarifies how our model achieves improved animation quality compared to the GT meshes it was trained on.

Q5: Training details are unclear. With a batch size of 30, how are typical forward passes structured to use 2N pairs for computing contrastive loss in Eq. 4, and how is triplet loss computed per emotion in Eq. 7?

A5: Our typical forward pass is structured as follows:

Contrastive Loss Calculation:

1. Feature Extraction and Averaging. In the first step, we extract emotion features from the input (video/audio) along the time dimension. This results in a tensor of shape (batch_size, time_steps, feature_dim), where feature_dim = 1024. Since the emotion features across time steps for the same video/audio are expected to belong to the same emotion space, we take the average along the time dimension to obtain a fixed-size feature vector for each sample. This results in a tensor of shape (batch_size, feature_dim) where each sample represents the averaged emotion feature over time.

2. Constructing 2N Pairs for Contrastive Loss. The Supervised Contrastive Loss involves comparing each sample with every other sample in the batch to form both positive and negative pairs. The loss calculation follows these key steps:

• Masking Positive Pairs: We first create a mask to identify which samples belong to the same class, i.e., positive pairs. The mask matrix has values of 1 for pairs of samples with the same label, and 0 for pairs with different labels. This mask is of shape (batch_size, batch_size).
• Pairwise Similarity Calculation: We compute the pairwise cosine similarities between the features of all samples in the batch. The computation is done using the dot product, and the result is scaled by the temperature parameter: similarity[i, j] = (features[i] · features[j]) / temperature
• Numerical Stability: To ensure numerical stability, we subtract the maximum value in each row of the similarity matrix: similarity[i, j] = similarity[i, j] - max(similarity[i, :])
• Masking Self-Comparisons: To avoid self-comparisons (i.e., comparing a sample to itself), the diagonal of the similarity matrix is set to zero: similarity[i, i] = 0 for all i
• Log-Probability Calculation: The log-probability for each sample is computed using the softmax function applied to the similarity matrix, with the self-comparisons excluded: log_prob[i] = log(softmax(similarity[i, :])) for each sample i
• Mean Log-Probability for Positive Pairs: We then compute the mean log-probability for the positive pairs by taking the sum of log-probabilities weighted by the mask, and dividing by the number of positive pairs for each sample: mean_log_prob_pos[i] = (sum(log_prob[i] * mask[i, :])) / (sum(mask[i, :]))

3. Contrastive Loss Calculation. Finally, the contrastive loss is computed as follows:

• The temperature-scaled mean log-probabilities for the positive pairs are averaged over the batch and scaled by the temperature and base temperature: loss = - (temperature / base_temperature) * mean_log_prob_pos, loss = average(loss) across all samples. This ensures that the model pulls together the features of positive pairs (samples with the same label) and pushes apart the features of negative pairs (samples with different labels).

Triplet Loss Calculation:

4. Emotion-Based Sample Selection. For each feature in the batch calculated in step 1, we first identify the samples that belong to the same emotion category using the emotion category label (labels[i]). These samples are grouped together, excluding the current sample (anchor). Then, within this group of samples, we distinguish between the positive samples (those with the same emotion intensity) and the negative samples (those with different emotion intensity). This step ensures that the positive and negative samples are selected from the same emotion.

5. Generating Triplets. For each sample (anchor), we form triplets by selecting:

• Anchor (a): The feature vector of the current sample.
• Positive (b): A feature vector from samples with the same emotion category and intensity.
• Negative (c): A feature vector from samples with the same emotion category but different intensity.

6. Loss Calculation. For each triplet, we calculate the standard Triplet Loss using the formula:

$L = \max (d(a,p) - d(a,n) + \alpha ,0),d(a,p) = ||{z^a} - {z^p}||_2^2,d(a,n) = ||{z^a} - {z^n}||_2^2$

7. Per Emotion Calculation. The triplet loss for each sample is calculated as described above, but it is done per emotion because we specifically select positive and negative samples based on both the emotion category and emotion intensity. As a result, the loss is computed in a manner that respects the emotional structure of the data. We compute the total loss for the batch by summing the losses from all the triplets and then averaging them over the batch size.

We hope that this explanation clarifies how the Contrastive Loss and Triplet Loss are computed and how they contribute to emotion-based learning in our model.

Quantified explanation in Q4

After training with the pivot identity, the performance metrics under the condition of using the pivot identity are as follows.

As mentioned earlier, the use of a pivot identity allows us to quickly fine-tune our decoder for an unseen person, enabling us to capture the specific speaking style of that person efficiently. For example, when testing on an unseen person, we use Actor 24 in RAVDESS as a case study. The results before and after the fine-tuning (within 20 minutes) are as follows.

We provide a brief demonstration in here, where you can observe that after a short fine-tuning period, we are able to capture the speaker's eye expression features, while maintaining the overall emotional expression and the accuracy of lip-sync.

I hope my clarification helps to address the concern. Once again, thank you for your time and consideration. If there are any additional questions or if further details are needed, we would be more than happy to provide further explanations and address any remaining concerns.

Quantified explanation in Q3

As mentioned in our previous response, when disentangling the feature spaces of different emotions, the intensity of each emotion is already embedded within these features. Therefore, even if two emotions are similar in terms of arousal, their intensities can still be independently distinguished.

To quantitatively analyze whether there is enough distinguishability between the features of different emotions and different intensity levels, we calculated the Euclidean distance matrix for the feature vectors of each emotion and intensity level. The distance between feature vectors for the same emotion but different intensities is computed using the following formula:

$D(i,j) = \frac{1}{{{n_i}{\rm{ \times }}{n_j}}}\sum\limits_{k = 1}^{{n_i}} {\sum\limits_{m = 1}^{{n_j}} {d(x_i^k,x_j^m)} }.$

We consider seven emotions: calm, happy, surprise, angry, disgust, fear and sad, with each emotion having two intensity levels. Therefore, we have a total of 14 categories, where $i,j \in (0,7{\rm{ \times }}2 = 14]$. For each pair of emotion objects, we calculate the Euclidean distance between all possible pairs of feature vectors from both objects. The average of these pairwise Euclidean distances is then taken as the overall Euclidean distance between the two emotion objects.

As can be seen from the results, features of different intensity levels within the same emotion are close but not overly similar, allowing for a certain degree of distinction. On the other hand, the features of different emotions and different intensity levels are far apart. For instance, even in cases like "surprise" and "angry," the features for both the emotion category and the intensity level category show separability.

Dear Reviewer AbM6,

Thank you again for your time and effort in reviewing our work! We would appreciate it if you can let us know if our response has addressed your concern. As the end of the rebuttal phase is approaching, we look forward to hearing from you and remain at your disposal for any further clarification that you might require.

Thanks in advance,

Paper 10320 authors