VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

Response to W1 (CAPP model sharing): We will soon release the CAPP model, which we believe fills the missing piece of audio-pose alignment metric in talking face generation research and will be valuable to the community.

Response to W2 (Voxceleb training data): We used the entire training set of Voxceleb2 for our model training. After filtering out invalid or low-quality videos, we ended up with approximately 170K training clips.

Response to W3 (OneMin-32 data details): It contains 32 one-minute video clips of 17 subjects (Line 229-232). They are mostly educational lectures and coaching sessions sourced from online video sharing platforms. The resolution is 512x512.

Response to W4 (training time): Our face latent model (encoder and decoder) takes around 7 days' training on a 4xA6000 workstation, and the diffusion transformer takes ~3 days. The total data used for training is ~500k clips (about 2-10 seconds each).

Response to W5 (condition usage): Yes, the condition signals are simply concatenated with noise along the temporal dimension as the input to the transformer. We'll clarify this in the paper.

Response to W6 (change to [2]): Our latent model architecture is same to [2]. We did not change the architecture but modified the training loss functions (Line 149-160) which are critical to achieve disentanglement (Figure A.8 and Line 344-350).

Response to W7 (data scale ablation): As mentioned above, our data size is ~500K (not ">10e6"). In the attached one-page PDF, we add a data size ablation study and comparison with other methods, as per the reviewer's suggestion. We trained a model using only 10% of the data (i.e., 50k clips). As shown in Table I, this model achieve comparable audio-lip and audio-pose synchronization to the full-data model, though the FVD and $\Delta$p metrics are not as good. This shows that our method performs well even with much less data, and more data enhances the motion diversity. Moreover, it still significantly outperforms other methods in all metrics assessing synchronization, motion intensity, and video quality.

Response to W8 (novelty): First, our motivation in the first place is, to model the human conversational behavior (facial dynamics and head movement ) holistically using diffusion model in a latent space that is agnostic to ID and appearance. This is our core innovation and, to the best of our knowledge, no previous methods have done this (it differs from the trend of further factor disentanglement and direct image generation; see our discussion in Line 47-55, 103-111). Second, in pursuit of the aforementioned goal, we did find the 3D-aided representation to be promising especially in terms of expressiveness and hence chose to leverage them. However, they can NOT meet our requirement of effective disentanglement. We made some insightful and provably critical modifications (Line 149-160, 344-350, and Figure A.8), without which we can never reach the current generation quality, esp. the liveliness with nuanced emotions. We perhaps have underemphasized the importance and contribution of such modifications, and will revise our presentation in the revision.

Apart from the two main contributions, our paper also offers other ones such as the design of face-factor-conditioned diffusion training, the CAPP model for filling the missing piece of pose-audio alignment metric, etc., which are also novel and valuable to the community.

Response to W9 (limited comparison to sota?): To the best of our knowledge, there are no other published methods which can generate both audio-driven head poses and facial movements with single images. We mentioned some concurrent unpublished works in our paper, and have added the visual comparison with a concurrent work EMO in the one-page PDF. We'd appreciate it if the reviewer can point out some specific papers that we should compare with.

Regarding the added comparison with EMO, we provide our results on some samples from EMO's official website (we are unable to provide video links per the rebuttal policy). As shown in Figure I, our method works consistently well and delivers vivid talking head videos. It is obvious that EMO has smaller head motion compared to us. Also, EMO seems less robust than ours in some cases, with artifacts – such as abrupt expression change, inaccurate lip sync, and subtle texture detail flickering – occasionally appear upon close inspection (note that their reported average lip-sync score is significantly lower than ours). On the other hand, however, EMO's video quality is slightly higher than ours in terms of sharpness, owing to their use of the large and powerful image generation foundation model.

Response to W10 (FVD on Voxceleb2) : As shown in Table I of the attached PDF, we provide the FVD scores of different methods on Voxceleb2. However, it should be noted that the video quality of Voxceleb2 varies widely and is often low (see Figure II of the PDF). Hence the FVD score may not accurately reflect the true generation quality, as mentioned in our paper.

Response to W11 (gaze estimation) : Thanks for your careful reading. We found that we inadvertently cited the wrong paper: we actually used L2CS-Net [a] to extract gaze direction. Will fix this error in our revised paper.

[a] Ahmed A.Abdelrahman, Thorsten Hempel, Aly Khalifa and Ayoub Al-Hamadi, L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments, 2022

Response to Q1: The results of different methods are resized to the same resolution (224x224) for FVD evaluation.

Response to Q2: This is an interesting question. For now we did not conduct in-depth analysis on whether the CAPP score captures the semantic relationship between speech and pose. We'll further explore this in our future work and thank you for the suggestion.

We hope we have addressed your questions. If not, it would be great to let us further know your concerns during discussion.

Thank you for your acknowledgment of our response and the additional comments.

Yes, our method can be easily adapted to generate facial dynamics only. Another easier way to achieve this is to directly replace the generated head poses with predefined ones before face image decoding. However, we shall point out that if the given head poses do not match the emotion or rhythm of the audio, the realism of the generated talking face video could degrade significantly (e.g., a calm head movement with intense speech or a rhythmic nodding with smooth speech would look weird). Generating realistic poses is one of the key contributing factors to achieve our high-quality results. That being said, we will try to add comparisons and more discussions about this type of methods in our revised paper and thank you again for the suggestion.

After reading the extensive rebuttal I see that the authors responded to most of my concerns. I see no reason to reject this paper and change my rating to accept. Some of the explanation of the rebuttal should be included in the final version. I would have been interesting to see comparison with more recent method even if they don't generate head pose. If the head pose is controllable shouldn't it be possible to freeze it to match that of the other methods ?

Response to W1: We achieve real-time efficiency because of our framework design, i.e. (diffusion-based) motion generation in latent space + (CNN-based) decoding in image space. Our diffusion transformer works in the latent space and is small (only 29M parameters), so it runs very fast. The CNN image decoder is also small (55M parameters) and runs efficiently. (Note that we only need to run the face encoder once for generating a video and thus its time can be neglected or counted into the starting latency.)

We simply evaluated the running efficiency with our whole method naively deployed in PyTorch without any special speed-up strategy. We believe there's still room for improvement with sophisticated implementation optimization.

Response to W2: Regarding the scale of video data, we trained our model on approximately 500k clips (2-10 seconds each). In the attached one-page PDF, we also provide an additional ablation study for training data scale. As shown in Table I, the model trained with 10% of the data achieves comparable audio-lip and audio-pose synchronization to the full-data model, though the FVD and $\Delta$p metrics are not as good. This shows that our method performs well even with much less data, and more data enhances the motion diversity.

Response to Q1: We'll try our best to release the source code of our project in the future. However, we hope you understand that due to significant concerns regarding the potential risks, particularly those related to deepfake and fraud, we (as well as the community) do need to be very cautious with releasing a powerful model. In fact, due to RAI considerations, our team faced great difficulties in getting approved by our organization for open source, unlike any other projects we did before. While we are seeking for the possibility of open source, we'll also add more implementation details such as those suggested by the reviewers into the revised paper. Also note that we will soon release the CAPP model, which we believe fills the missing piece of audio-pose alignment metric in talking face generation research and will be valuable to the community.

Response to Q2: EMO is a concurrent work (published Feb 27th on arXiv) at the time of our submission (May 22th) and there's no public implementation, so we did not compare with it. However, we did mention it with some discussions in our paper (Line 41-44, 115-118). EMO uses an image diffusion model based on StableDiffusion to generate talking face videos, which is a significantly different technique. It can generate high-quality videos but suffers from heavy computation and slow generation speed compared to ours.

In the attached one-page PDF, we provide our results including animations on some samples from EMO's official website (we are unable to provide video links per the rebuttal policy). As shown in Figure I, our method works consistently well on EMO's demonstrated cases and delivers vivid talking head videos. It is obvious that EMO has smaller head motion compared to ours, perhaps due to the constraint of the face region mask used by it. Also, EMO seems less robust than ours in some cases, with artifacts – such as abrupt expression change, inaccurate lip sync, and subtle texture detail flickering – occasionally appears upon close inspection (note that their reported average lip-sync score is significantly lower than ours). On the other hand, however, EMO's video quality is slightly higher than ours in terms of sharpness, owing to their use of the large and powerful image generation foundation model.

Response to Limitations: Thank you for the comments. We plan to handle upper-body/full-body and explore more explicit 3D representations in our future work (both projects are on-going).

We hope we have addressed your questions. If not, it would be great to let us further know your concerns during discussion.

Response to W1: Thank you for the comment. We will add more details of the 3D-aided face latent model into our main paper and appendix. Our network architecture follows MegaPortraits [18] where details can be found. To achieve disentanglement, we modified the loss functions and incorporated the cross-transfer consistency loss $L_{consist}$ and ID-disentanglement loss $l_{cross-id}$. (Line 149-160). These loss modifications are critical (Line 344-350 and Figure A.8), without which we can never reach the current generation quality, esp. the liveliness with nuanced emotions. Again, we'll add more details and further improve the clarity as per your suggestion.

Response to W2: Different conditional signals are directly concatenated with noise along the temporal dimension as the input to the transformer.

Response to W3: First, please note that different methods in the literature may have used different data for training. We are unable to compare with these methods using exactly the same data and have simply followed the practice of running the trained models for comparison.

In terms of training data scale, our model is trained on ~500k clips (2-10 seconds each). To validate the data scale influence and compare it with previous methods at similar scales, we additionally trained a model using only 10% of the data (i.e., 50k clips). As shown in Table I in the attached PDF, the model trained with 10% of the data achieves comparable audio-lip and audio-pose synchronization to the full-dataset model, though the FVD and $\Delta$p metrics are not as good. This shows that our method performs well even with much less data, and more data enhances the motion diversity.

Regarding the compared methods, Audio2Head used >70k data clips for training but clearly underperformed compared to our model with 50k clips. MakeItTalk and SadTalker used very small subsets of Voxceleb for training, but there's no clear evidence that increasing their data would improve their performance significantly or even bring any positive consequence - we explain the reasons as follows. MakeItTalk uses an LSTM to map audio features to landmark offsets deterministically, which may struggle with modeling complex data distributions and one-to-many mappings as training data increases. SadTalker assigns a style code for each identity to generate head poses, but more data will introduce more diverse head motion patterns for the same identity, which a shallow VAE with a condition code might not be able to effectively model. Our model with 10% data still significantly outperforms these methods in all metrics assessing synchronization, motion intensity, and video quality.

Response to Q1: We set the transformer layer number to 8 as we found it produces good results while enabling the whole algorithm to run in real time on a consumer-grade GPU. We didn't explore more layers or larger model size because real-time efficiency is a key factor we want to achieve. We presume that a larger model size will further improve the performance because our current model is still small, and we'll further explore this in our future work.

Response to Q2: Yes, we will soon release the CAPP model, which we believe fills the missing piece of audio-pose alignment metric in talking face generation research and will be valuable to the community.

Response to Q3: No, we did not use distillation or any other strategies to speed-up the 3D-aided face encoder and decoder. These models are naturally small and run very fast. Note we only need to run the encoder once so essentially only the decoder needs to run for generating each video frame.

Response to Q4: The parameter counts of our 3D-aided face latent model and diffusion transformer model are about 200M and 29M respectively.

We hope we have addressed your questions. If not, it would be great to let us further know your concerns during discussion.

Response to W1 (contributions): Thank you for the comment. In response to your question on our technical contributions and relationship with MegaPortraits, we'd like to emphasize two aspects. First, our motivation in the first place is to model the human face conversational behavior (facial dynamics and head movement) holistically using a diffusion model, in a latent space that is agnostic to ID and appearance. This is our core innovation and, to the best of our knowledge, no previous methods have done this (it differs from the trends of further factor disentanglement and direct image generation; see our discussion in Lines 47-55, 103-111). Second, in pursuit of the aforementioned goal and building a disentangled latent space, we did find the 3D-aided representation to be promising especially in terms of expressiveness and hence chose to leverage them. However, they can NOT meet our requirement of effective disentanglement. We made some insightful and provably critical modifications (Line 149-160, 344-350, and Figure A.8), without which we can never reach the current generation quality, esp. the liveliness with nuanced emotions. We perhaps have underemphasized the importance and contribution of such modifications, and will revise our presentation in the revision.

Apart from the two main technical contributions, our paper also offers other ones such as the design of face-factor-conditioned diffusion training, the CAPP model for filling the missing piece of pose-audio alignment evaluation, etc., which are also novel and valuable to the community.

Response to Q1: The extra condition signals such as the main eye gaze direction and head-to-camera distance are optional and they are provided by users. If not given, we can either set them to some default parameters (e.g., a forward-looking eye gaze and the average head-to-camera distance of the training data; see Line 220-222), or just leave them blank for unconditional generation.

Response to Q2: Thank you for the suggestion. We have run our method on the images and audios from EMO's official website as per your suggestion. Some visual results including animations can be found in the one-page PDF provided on this page (we are unable to provide video links per the rebuttal policy). EMO is a concurrent work which we mentioned and discussed in the related work section. It uses an image diffusion model based on StableDiffusion to generate talking face videos, which is a significantly different technique. EMO can generate high-quality videos but suffers from heavy computation and slow generation speed compared to ours.

As shown in Figure I of the attached PDF, our method works consistently well on EMO's demonstrated cases and delivers vivid talking head videos. It is obvious that EMO has smaller head motion compared to ours, perhaps due to the constraint of the face region mask used by it. Also, EMO seems less robust than ours in some cases, with artifacts – such as abrupt expression change, inaccurate lip sync, and subtle texture detail flickering – occasionally appear upon close inspection (note that their reported average lip-sync score is significantly lower than ours). On the other hand, however, EMO's video quality is slightly higher than ours in terms of sharpness, owing to their use of the large and powerful image generation foundation model.

We hope we have addressed your questions. If not, it would be great to let us further know your concerns during discussion.