GPAvatar: Generalizable and Precise Head Avatar from Image(s)

Q3: About the larger head pose

We agreed that a large change in the view angle is crucial for evaluating a 3D-based method. The animation sequence in our audio-driven demonstration is extracted from SadTalker, which always has smooth and gentle head movements. But in our paper numerous results show significant head movements, even in side views (as illustrated in Figure 7). These results show that our method effectively inherits the advantages of 3D methods in view angle changes.

Q4: The visualization of the attention weights

We appreciate the reviewer's insightful suggestion to show attention visualizations, which can better show how our MTA module works. For now, we have included these visualizations in the revised version (Sec.C and Figure.9). These attention visualization results are very interesting and show that our method can indeed pay attention to and combine different angles of people in different images.

Q5: How we obtain facial expression from audio

We employ a widely used audio-driven motion generator: SadTalker to generate head motion based on audio input, although other similar methods can also be used. SadTalker uses an image and a piece of audio to generate a video, and then we track the 3DMM parameters in that video. We then use these parameters to animate the GPAvatar. We will add descriptions in future demos.

Thank you for your constructive feedback and detailed review. We have taken your suggestions and conducted experiments to improve the identity similarity. Our responses below address each of your points.

W1: About the PEF

Please refer to the Q1.

W2: About the identity similarity

Please refer to the Q2.

W3: About the image quality

Since our model is only trained on VFHQ data (characters from TV interviews) and inferred on unseen IDs, for some inputs like CG characters, there are some cases where we cannot generate clear reconstruction. We hope that we can solve this problem by enhancing data or improving cross-domain robustness in future work. It is worth noting that even though we have not added these improvements, our model still has amazing generalization ability and is able to handle many cross-domain cases.

Q1: How the PEF handles some non-facial parts.

Our feature space mainly consists of two parts, the canonical tri-planes feature space and the expression feature space based on the FLAME point cloud (PEF). Non-face areas are overall modeled by the tri-planes space, which is essentially a rigid body. If we limit the feature search of the expression field to the FLAME area, then parts such as hair and torso will be completely modeled as rigid bodies that change as the camera perspective rotates. In order to be able to solve some non-flame modeling areas related to the expressions, we relax the nearest neighbor search distance of the expression feature field, so that some hints of the current expression can be obtained in the entire tri-planes feature space, which makes some non-FLAME areas more flexible. Especially areas such as hair and glasses that are close to the face but are not modeled by FLAME. The quantitative ablation experiment on global sampling also confirmed that this can improve various metrics including reconstruction quality and expression control fineness.

Q2: About the identity similarity of our method and HideNeRF

We examined the technology and implementation used by HideNeRF in detail, and found that Arcface was used to establish identity loss to improve identity consistency. We also adopted the same method and conducted experiments, and we can observe that our identity consistency has also been improved, but at the same time, the reconstruction quality and expression control accuracy have decreased. We believe that we may need to conduct more experiments to find a better balance between multiple losses to obtain better results. We will continue to work on this part. But it is worth noting that the fine control of expressions and the substantial improvement in reconstruction quality are our main contributions and the features we most want to retain.

In the following is the result when we add Arcface-based identity loss L_ID. Among them, ours one-in w/o L_ID is the result in the original paper version, and w L_ID is the result with identity loss added.

Thank you for your constructive feedback and time to review our paper. We have taken your suggestions and further refined our paper. Please find our detailed responses to each of your points raised below:

W1: Methodology needs several clarifications

Please refer to the responses to the corresponding questions.

W2-3: About the missing citations.

Thanks for pointing this out. We acknowledge the relevance of this paper and include it in our updated related work section. However, it is worth noting that our method is not comparable to these methods. Our approach focuses on oneshot and fewshot settings, meaning the input is only one or a few images, no training on new identities is required, and our model is driven by motion sequences based on FLAME parameters. Among these papers, the paper [1] requires videos as input (may have thousands of images), and each ID-specific model needs to be trained based on the videos. This prevents the model from making inferences on unseen IDs, but requires data from that ID for training. This is different from our setting, where we use unseen IDs during inference to evaluate our method, which is not possible in paper [1]. At the same time, the way it uses FLAME is also different from us. Our motivation is to avoid the loss of expression information, but the paper [1] adopts a method similar to Figure.2(c), by directly inputting the expression vector to control the expression, which will obviously lead to expression Loss of information.

Papers [2] and [3] use audio signals as the driving method, which makes it difficult to accurately control the avatar to make the desired expression, and since we use FLAME parameters to control the avatar to make expressions, the difference in control signals makes we differ from papers [2] and [3] and cannot evaluate in the same setting.

The setting of paper [4] is similar to paper [1], requiring the use of video as training data and cannot infer with an unseen ID. This makes paper [4] not comparable to our method.

Q1: Sec 3.2, how the expression information affects the PEF?

Our feature space mainly consists of two parts, the canonical tri-planes feature space and the expression feature space based on FLAME point cloud (PEF). Expression information is given in the form of FLAME parameters, and the FLAME model dynamically forms a point cloud based on these parameters. That is, different FLAME parameters will generate point clouds with different positions. Since our PEF is based on these point clouds, the feature queried from PEF will also be changed with different expression parameters.

Q2: Sec 3.3, how do we build the canonical encoder?

We adopt the same Style-UNet structure as in GFPGAN with only modification of the input and output size. In short，we obtain the style code through 4 groups of ResBlock down-sampling, use 3 groups of ResBlock up-sampling to obtain the conditions, and then use StyleGAN to output the 3 × 32 × 256 × 256 tri-planes based on style code and conditions. For more implementration details please refer to Sec.A.1, and we will also release the code and checkpoints to facilitate reproducibility and further research.

Q3: Figure 4, missing column title

The last two columns actually correspond to "Ours One-in" and "Ours Two-in". For a fair comparison, we decided to only compare here the driving results of a single image as input. In the revised version we have removed "Ours Two-in" in Figure 4. Results of Ours Two-in or more can be referenced from other Figures (like Figure 6,7,8,9).

Q4-Q5: About the equations.

We appreciate the reviewers for identifying sign errors in our formulas. These mistakes have been rectified in the revised version. The corrected equation is $f_{exp,x} = \sum_{i}^{K} \frac{w_i}{\sum_{j}^{K} w_j} L_p(f_{i}, F_{pos}(p_i-x)), \text{where}~w_i=\frac{1}{p_i-x},$ where $x$ is the coordinate to be queried, $K$ is the number of neighbors, $f_i$ is the corresponding feature, $p_i$ is the point coordinate, $L_p$ is the linear layers and $F_{pos}$ is the frequency positional encoding function. In this equation, the $w_i$ ensures that the nearest point has the highest weight when contributing to the feature.

In Equation (2): $N$ should be referred to the number of input images and we also correct it in the revised version.

Ethic concerns:

We agree with you that there are potential ethical risks with talking head generation. Please refer to the "Response to Ethics Concerns" for details. We have also added more discussion about ethics in Sec.7 of the revised paper.

Thank you for your kind feedback and your pointers and questions. We have refined our revised version paper based on your suggestions. Please find our answers for each of your points below.

W1. About trainable parameters

Given the wide range of parameters in these models, there is a concern regarding fair comparisons. However, it is quite difficult to evaluate these methods at the same parameter level due to different custom structures. Notably, our method does not have the most trainable parameter among all the baselines: our model has 96M parameters but HideNeRF has 569 million parameters.

W2. About the limitations of our approach

We appreciate the reviewer for pointing out this missing part. Specifically, our current FLAME-based model lacks a module to control the shoulders and body, resulting in limited control below the neck (the shoulder position generally aligns with the input image for now). Additionally, for other areas not modeled by FLAME, such as hair and tongue, explicit control is not feasible. Furthermore, while our aspiration is to achieve real-time reenactment of more than 30 fps, our current performance is pre-real-time for now (approximately 15 fps on the A100 GPU). We added the discussion about limitations in our revised paper (Sec.B), and will solve these problems as future work.

Q1. About how we captures the subtle information

We avoid the information loss resulting from over-processing using our PEF (illustrated in Figure 2). Consequently, any subtle details captured by FLAME directly influence the point positions in the PEF, thereby contributing to the rendering results.

Q2-Q3. About the normalization weights in equation 1 and R in equation (Page 3).

We appreciate the reviewers for identifying sign errors in our formulas. These mistakes have been rectified in the revised version. The corrected equation is $f_{exp,x} = \sum_{i}^{K} \frac{w_i}{\sum_{j}^{K} w_j} L_p(f_{i}, F_{pos}(p_i-x)), \text{where}~w_i=\frac{1}{p_i-x},$ where $x$ is the coordinate to be queried, $K$ is the number of neighbors, $f_i$ is the corresponding feature, $p_i$ is the point coordinate, $L_p$ is the linear layers and $F_{pos}$ is the frequency positional encoding function. In this equation, the $w_i$ ensures that the nearest point has the highest weight when contributing to the feature. While for the $R$ in equation (page 3), it refers to the volume rendering process in NeRF, which renders the merged feature space into the result image.