GAIA: Zero-shot Talking Avatar Generation

Dear Reviewer tNS3, thank you for taking the time to review. We appreciate that you found our model simple and scalable with superior performance.

Q: The authors did not provide a large number of generation examples. I hope it is possible to see more visual results.

A: Thank you for your interest. We provide more samples on the anonymous demo page for convenience. More specifically, 1) an example of "long video generation" is provided in this video. Since we focus on demonstrating the effectiveness of our proposed method, the long video generation is not yet well explored in the original manuscript. According to your comments, we present a simple yet effective strategy to generate infinite long videos -- by fixing the first several frames of each generation step during the reverse diffusion process. In this way, we obtain a smooth transition between each sub-video. 2) more examples of "different reference frame but the same driving video" are provided in this video. 3) more examples of "different driving video but the same reference frame" are provided in this video.

We hope the above videos could make our results more clear.

Q: Is it possible to drive cartoon characters (humanoid or non-humanoid)?

A: Sure. GAIA supports cartoon characters well. Examples can be found at this video, this video and also this video, which demonstrates that our method generalizes well to out-of-domain data.

Q: Will the proposed method and the established dataset be publically available?

A: We promise that the code will be released upon publication. For the dataset and pre-trained models, since privacy information is involved (i.e., real human faces), we are going through an internal compliance review and will try our best to mitigate the privacy risks and release them to facilitate future research.

We welcome further discussion and are willing to answer any further questions.

Dear Reviewer tNS3,

We greatly appreciate the time you've invested in reviewing our response. Having submitted our rebuttal, we are eager to know if our response has addressed your concern. As the end of the rebuttal phase is approaching, we look forward to hearing from you for any further clarification that you might require.

Best,

Submission 5722 authors

Dear Reviewer Lkke, thank you for your positive and thoughtful feedback and for looking into every detail of our work.

Q: The comparison with HeadGAN for the video-driven generation.

A: Thanks for reminding us of this important related work! We evaluate this model with the released checkpoint on our test set. To ensure fair comparisons, in addition to the model trained on our dataset, we also train one smaller GAIA model with 80M parameters on the VoxCeleb2 dataset to keep consistent with HeadGAN. We test models on the video-driven self-reconstruction and cross-reenactment settings, and the results are shown below.

Table 5. Quantitative comparisons of self-reconstruction with HeadGAN.

Table 6. Quantitative comparisons of cross-reenactment with HeadGAN.

From the results, GAIA achieves consistent improvements over HeadGAN across different metrics and settings. We have added the comparison with HeadGAN to Table 2 and 7 in the manuscript.

Q: Will the dataset be shared as part of this submission?

A: We promise that the code will be released upon publication. For the dataset and pre-trained models, since privacy information is involved (i.e., real human faces), we are going through an internal compliance review and will try our best to mitigate the privacy risks and release them to facilitate future research.

Q: The ablation on training data size.

A: We have investigated the scaling of training data in Tables 4 & 5 of the manuscript, where #Hours indicates the total training size. It can be observed that, with the same model size (80M VAE and 180M diffusion), our method obtains better results on more training data (e.g., 17.486 vs. 18.353 FID score, 8.913 vs. 9.145 Sync-D score). It is also interesting to see what will happen on larger data size (i.e., more than 1K hours). We leave it as future work.

Q: How does randomness from the diffusion model affect generations?

A: Thank you for your interest! To show the effect of randomness, we conduct three kinds of visualizations in this video: 1) we use different random seeds (i.e., seed 42 & 82) for the generation, from which we observe that different seeds make slight differences for the generated videos; 2) we evaluate the models from different training epochs (i.e., epoch 1599 & 1799), which shows the large difference on the generated motion; 3) we make generation with different reference images, which demonstrates that the reference image affects the appearance of the generated video since the missing information should be predicted by the model.

We welcome further discussion and are willing to answer any further questions.

Dear Reviewer Lkke,

We greatly appreciate the time you've invested in reviewing our response. Having submitted our rebuttal, we are eager to know if our response has addressed your concern. As the end of the rebuttal phase is approaching, we look forward to hearing from you for any further clarification that you might require.

Best,

Submission 5722 authors

Q: About the location of Ethical consideration.

A: Thanks for the suggestion, and we agree that ethical considerations are crucial for the completeness of the paper. We have revised the content of the manuscript and put the ethical considerations in Section 6 (Conclusion).

We hope our rebuttal and paper revision can address your concerns. We welcome further discussion and are willing to answer any further questions.

Q: The experiments are limited, especially comparison with end-to-end trainable models not provided. I suggest enriching the benchmarking with other existing models such as PC-AVS and PD-FGC as they are also end-to-end trainable models.

A: Thanks for your suggestion! Following your advice, we have evaluated PC-AVS [1] and PD-FGC [2] with their released models on the same test set. The results are listed in the table below. The visual comparison is provided in this video and the full comparison is presented in this video.

Table 4. Quantitative comparisons of speech-driven generation with previous end-to-end trainable models.

In this table, * indicates that PD-FGC depends on extra driving videos to provide pose, expression, and eye motions. We use the real (ground-truth) video as its driving video (therefore the score of motion diversity is high). We also note that the Sync-D score for real video is 8.548, which is close to ours. Overall, our method beats the previous end-to-end trainable models in terms of overall naturalness (Nat.), lip-sync quality (Lip.), motion jittering (Jit.), visual quality (Vis.), and motion diversity (Mot.). We have added the comparison in Table 8 in the revised version.

[1] Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation. CVPR 2021.

[2] Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis. CVPR 2023.

Q: Although the writing quality is decent, it is hard to follow as it refers to other sections frequently and other issues (see Questions 1 and 2).

A: Thanks for the suggestion! We have removed the unnecessary referrals to make the manuscript easy to follow. For example, we remove the referral in Section 5.3.1 (Ablation Studies on Scaling) and Section 5.5 (Discussion). In the revised version, apart from the referrals to the tables or figures, most referrals are referred to the Appendix for detailed illustrations.

Q: About the collection of HDTF, CC v1&v2 and AVENA datasets.

A: Thanks for your careful reading of our paper and supplementary materials, and sorry for the confusion. Your understanding is correct. AVENA is an internal dataset collected from our institution, and we utilize the union of AVENA, HDTF, CC v1 and v2 as the training set of our model. We have revised the description in Section 3 (Data Collection and Filtering) and Section 5.1 (Experimental Setups) for better understanding and marked it in blue.

Q: About the discussion section.

A: Thanks for the suggestion! We have reorganized the flow in the revised manuscript and marked the revisions in blue. For example, we moved the improved discussion to the end of the experiments (Section 5.5).

Q: I am not sure the model can be named as zero-shot as it requires one shot for unseen faces. So, could you elaborate on the following '... generates a talking video of an unseen speaker with one portrait image ...'?

A: The term "zero-shot" is potentially ambiguous in literature. In this paper, we use the zero-shot term due to the following reasons:

1. the method is "zero-shot" typically in the sense that no gradient updates are performed;
2. while inference, the difference between zero-/one-/few-shot lies in the number of utilized demonstrations [1], where a demonstration usually indicates an input-output pair of data. In our case, the provided unseen face is not a kind of demonstration, making the inference procedure zero-shot;
3. to be differentiable with the related works where a fine-tuning stage on the given reference portrait image is needed for the unseen identities [2,3,4].

We are open to further discussions on this concern.

[1] Language Models are Few-Shot Learners. NeurIPS 2020.

[2] Few-Shot Adversarial Learning of Realistic Neural Talking Head Models. ICCV 2019.

[3] Neural Head Reenactment with Latent Pose Descriptors. CVPR 2020.

[4] Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation. ICCV 2023.

Due to character limitations, please refer to the next comment for other questions.

Dear Reviewer ft55, thank you for taking the time to review and propose promising extensions. We appreciate that you found our rich ablation study and the good writing. We address your concerns as follows.

Q: Comparisons with other end-to-end trainable talking face synthesis models.

A: Thanks a lot for reminding the related end-to-end trainable works! We take PC-AVS [1] and PD-FGC [2] as examples to make more discussions. PC-AVS [1] and PD-FGC [2] similarly introduced identity space and non-identity space. The identity space is obtained by leveraging the identity labels, while the non-identity space is disentangled from the inputs through random data augmentation [3]. The authors employed contrastive learning to align the non-identity space and speech content space (except pose). However, our method differs in three ways: 1) they need additional driving video to provide motion information like head pose. In contrast, we generate the full motion from the speech at the same time and also provide the option to control the head pose; 2) they use contrastive learning to align speech and visual motion, which may lead to limited diversity due to the one-to-many mapping nature between the audio and visual motion. In contrast, we leverage diffusion models to predict motion from the speech; 3) their identity information is extracted by using identity labels while our method does not need additional labels.

We further highlight our contributions as follows:

• We propose a novel and sensible (as recognized by Reviewer 1S6P & Lkke) framework that eliminates the heuristics and generates the full motion latent at the same time. The method reveals three key insights: 1) the complete disentanglement between the motion (speech-related) and the appearance (speech-agnostic) is the key to success; 2) handling one-to-many mapping with the diffusion model and learning full motion from real data distribution result in natural and diverse generations; 3) less dependence on heuristics and labels makes the method general and scalable.

• We achieve superior performance on naturalness, diversity, lip-sync quality, and visual quality (as recognized by Reviewer 1S6P & Lkke & tNS3).

• We, for the first time, verify the scalability of our method in talking avatar generation (as recognized by Reviewer Lkke & tNS3).

• The method is general and flexible to support many applications: speech-driven only, coarse-grained control (controlled by head pose), fine-grained control (fully controllable), and text-instructed avatar generation (as recognized by Reviewer 1S6P & Lkke & tNS3), which has never been explored in previous literature.

We have added the discussions in the revised manuscript and marked the revisions in blue in Section 2.

[1] Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation. CVPR 2021.

[2] Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis. CVPR 2023.

[3] Neural Head Reenactment with Latent Pose Descriptors. CVPR 2020.

Due to character limitations, please refer to the next comment for other questions.

Dear Reviewer ft55,

We greatly appreciate the time you've invested in reviewing our response. Having submitted our rebuttal, we are eager to know if our response has addressed your concern. As the end of the rebuttal phase is approaching, we look forward to hearing from you for any further clarification that you might require.

Best,

Submission 5722 authors

Dear Authors,

Thank you for the rebuttal and addressing my concerns.

I have read the responses carefully. And I think, with the revisions, the overall quality has been improved which led me to reconsider my score in the final review. I tend to increase my rating.

Dear Reviewer 1S6P, we are grateful for your careful review and the valuable feedback that you provided for our paper. We appreciate that you found our paper well-written and sound with convincing results. We hope the following comments address your concerns.

Q: Missing evaluation of disentanglement between appearance and pose latent codes, i.e., cross-reenactment.

A: We have evaluated the cross-reenactment performance of GAIA and baselines in the right half of Table 2 of the manuscript, denoted as "Cross-Reconstruction" in the original version, where GAIA achieves significant improvements over baselines. Sorry for the confusion about the notations we utilized, we have changed it to "Cross-Reenactment" in this version to keep consistent with baselines.

Q: Missing discussion of some related works.

A: Thank you for pointing out the related works! We make discussions as follows. We have also added the discussions to Section 2 (Related Works) and the revisions are marked in blue.

Latent Pose Descriptors [1] achieved pose-identity disentanglement, where the identity embedding is averaged across multiple frames and the pose embedding is obtained with augmented input. However, the model needs additional fine-tuning for the unseen identities while we can directly generalize to the unseen identities with only one reference portrait image.

MegaPortraits [2] adopted warping-based transformation, which is similar to face-vid2vid [3], but uses latent descriptors to represent expression instead of keypoints like face-vid2vid [3]. However, as we have validated in the subjective comparison and the experiments of the original manuscript, the usage of the heuristics hinders direct learning from data distribution, leading to unnatural results and limited diversity.

In addition, instead of supporting video-driven avatar generation only [1,2], GAIA contains both motion & appearance disentanglement and speech-to-motion modules that support speech-driven, video-driven, pose-controllable, full-controllable and text-instructed avatar generation in a unified framework.

[1] Burkov et al., "Neural Head Reenactment with Latent Pose Descriptors", CVPR 2020

[2] Drobyshev et al., "MegaPortraits: One-shot Megapixel Neural Head Avatars", ACMMM 2022

[3] Wang et al., "One-shot free-view neural talking-head synthesis for video conferencing", CVPR 2021

Q: The proposed method and the baselines were trained on different datasets.

A: The reason we choose different datasets is that we find commonly utilized datasets such as VoxCeleb2 suffer a lot from the jittering content and thus have relatively low quality. However, we agree with the point that training on the same dataset provides fair comparisons. Therefore, we train our model on VoxCeleb2, the training dataset utilized by face-vid2vid, while keeping the test set unchanged. Due to the time limitation, we train a smaller model with 80M parameters and will train a larger one in the next. The results of video-driven self-reconstruction and cross-reenactment are listed as follows.

Table 1. Quantitative comparisons of self-reconstruction when trained on VoxCeleb2 dataset.

Table 2. Quantitative comparisons of cross-reenactment when trained on VoxCeleb2 dataset.

From the results, we can find that when trained on the same dataset, GAIA still outperforms face-vid2vid on most metrics except PSNR (though it is the smallest model), demonstrating the power of the proposed method. We have added the results in Table 7 and more discussions in Section C.1 (More Video-driven Results) of the manuscript.

Due to character limitations, please refer to the next comment for other questions.

Q: Comparisons of the inference time are not provided.

A: Thank you for your interest! According to your suggestions, we have evaluated the inference time of the same speech (8s) for each method. Below are the results, where our diffusion step is set to 150 (the same as the one we used in all experiments).

Table 3. Comparisons of the inference time of the same speech.

From the table above, we observe that our model achieves comparable inference time against the baselines (e.g., 2.639s for 80M VAE, 2.909s for 180M diffusion model). Although the larger models like 700M VAE and 600M diffusion need more inference time, it can be further reduced with advanced techniques (e.g., fast diffusion sampling, model distillation, etc.), yet is not the focus of this work. For the current studies, we have shown that our method is able to achieve substantial improvement over the state-of-the-art and illustrated the scalability and generalizability of the proposed method.

Q: Could the authors clarify if they plan to release the filtered dataset and the pre-trained models?

A: We promise that the code will be released upon publication. For the dataset and pre-trained models, since privacy information is involved (i.e., real human faces), we are going through an internal compliance review and will try our best to mitigate the privacy risks and release them to facilitate future research.

We welcome further discussion and are willing to answer any further questions.

Dear Reviewer 1S6P,

We greatly appreciate the time you've invested in reviewing our response. Having submitted our rebuttal, we are eager to know if our response has addressed your concern. As the end of the rebuttal phase is approaching, we look forward to hearing from you for any further clarification that you might require.

Best,

Submission 5722 authors