Generalizable and Animatable Gaussian Head Avatar

Thank you for your review and helpful comments, which triggered deeper thinking about our approach. We would like to address your concerns in the following sections:

How could the static points produced by the reconstruction branch model the dynamic expressions?

• Your understanding is mostly correct. In our method, the reconstruction branch generates static Gaussian points, while the expression branch generates dynamic Gaussian points.
• However, it's important to note that our Gaussians are not purely RGB or spherical harmonics Gaussians. Instead, our Gaussians include 32-D features (as described in Sec. 3.3). In Fig. 7 of the paper, we visualize the first 3 dimensions of these features (i.e., the RGB values of the Gaussians) without the neural rendering module. This visualization is intended to intuitively display the functionality of each part and the importance of each branch should not be judged based on RGB values ​​alone. Their importance is determined by the entire 32-D features and the neural renderer module. In fact, the expressions are modeled solely by the expression branch, without expression Gaussians, the lifted Gaussians from the reconstruction branch are static.

How does the proposed method resolve the potential conflicts of these two point sets?

• Thank you for pointing out some missing parts of the paper. We will include this discussion on how to resolve conflicts in the revised version.
• Although we attempt to bring the two point sets closer together, there are inherent conflicts since one set is static and the other is dynamic. We address these conflicts through the neural rendering module. Our Gaussian points have 32-D features, which contain more information than just RGB values. Thus, the neural rendering module can leverage these features to integrate the two point sets. We show some results with conflicts in Fig. 4 on the new supplementary page. It can be seen that when there are significant expression differences, the RGB values of the Gaussian points conflict, but these conflicts are well-resolved after neural rendering. We believe this demonstrates that the neural rendering module acts as a filter, using features from the expression Gaussian in areas where conflicts may arise.

About the novelty of the dual-lifting method.

• We believe our method is novel. The work of [Gabeur et al.] uses two sets of points to model a static human body and then obtains the body mesh. Our method uses two sets of lifted 3D Gaussians to model the human head, which is the first one-shot 3DGS method for head avatars. Additionally, we incorporate expression Gaussians, enabling dynamic expression modeling, which further develops this method. Our method also achieves real-time re-driving, which is a first in one-shot head avatars and is crucial for practical applications. Finally, our work provides a comprehensive evaluation and comparison for one-shot dynamic head avatar reconstruction, which builds a solid baseline for future research. For missing citations and discussion of related papers, we will include and discuss them in the revised version.

The baseline methods and evaluation method.

• We used pre-trained network weights and the official code implementations for evaluation. Since each method has different data requirements, it is challenging to train every method on a unified dataset. For instance, Portrait4D uses single-view head images, Portrait4DV2 uses generated multi-view data, GOHA needs video and single-view head images, Real3DPortrait needs EG3D-generated multi-view head images and pre-train image-to-plane model, ROME did not release their training code.
• Since these methods emphasize their generalization ability, using pre-trained weights for evaluation is acceptable and all compared baseline works also follow the same way for their evaluation.
• Additionally, the HDTF and VFHQ test sets are commonly used benchmark datasets. OTAvatar, GOHA, and GPAvatar have reported results on the HDTF test set, while GPAvatar, Portrait4D, and Portrait4DV2 have reported results on the VFHQ test set, therefore we chose to test on these two datasets.

Side-by-side video comparison with baselines.

• Due to rebuttal restrictions, we are unable to provide additional videos for rebuttal. Instead, we have included some consecutive frames in the new supplementary page and highlighted the areas of interest. We will add side-by-side video comparisons in our future open-source code and demo website.

Thank you for your positive review and helpful comments. We would like to address your concerns in the following sections:

Is it possible to conduct a study by removing the reconstruction branch to better demonstrate the importance of dual-plane lifting?

• Removing the reconstruction branch is possible, but it would result in a lack of detail. Our method integrates global identity features when predicting the expression Gaussians, which provides some identity information to the expression Gaussians. However, the 1D global features are insufficient to reconstruct detailed identity features. We provide qualitative results of the reconstruction branch in Fig. 2 on the new supplementary page and quantitative results here. The qualitative results show a severe lack of details on identity, demonstrating the necessity of the reconstruction branch, and the quantitative results also support this conclusion (the CSIM metric).
• | Method | PSNR↑ | SSIM↑ | LPIPS↓ | CSIM↑ | AED↓ | APD↓ | AKD↓ | CSIM↑ | AED↓ | APD↓ | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | | w/o Recons | 18.006 | 0.756 | 0.261 | 0.454 | 0.203 | 0.223 | 5.324 | 0.230 | 0.246 | 0.279 | | Ours (full) | 21.83 | 0.818 | 0.122 | 0.816 | 0.111 | 0.135 | 3.349 | 0.633 | 0.253 | 0.247 |

The visual results of the method seem to be overly smoothed, making the results less realistic compared to some baselines. The authors may need to provide additional explanation for this and discuss why the method achieves better qualitative results than other methods.

• Currently used quantitative metrics focus more on the correctness of results rather than their realism. For example, AED measures whether subtle expressions are correctly modeled, and PSNR and SSIM measure if identity details are authentically reproduced. Sometimes visually realistic results may be incorrect. To verify the realism of our results, we have provided our quantitative evaluation with FID scores (self-reenactment on VFHQ), which is a commonly used metric for realism. Our model demonstrates competitive performance compared to state-of-the-art methods.

• | Method | StyleHeat | ROME | OTAvatar | HideNeRF | GOHA | CVTHead | GPAvatar | Real3D | P4D | P4D-v2 | Ours | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | | FID↓ | 72.138 | 49.516 | 70.692 | 51.930 | 39.638 |109.054 | 37.610 | 38.999 | 46.965 | 30.573 | 28.938 |

• Our method tends to over-smooth the hair regions. We believe this because our method does not include control of these regions. During training, the hair and upper body will always look like the input image instead of the target image, and this inconsistency leads to over-smoothing. This problem is also observed in other baseline methods since all the baseline methods do not support hair control. Among them, GOHA, Real3DPotrait, and Potrait4D use GAN loss to enhance the realism, but lead to inaccurate expression control and false illusion of details, while Portrait4DV2 improves hair realism by using data generation to obtain correct ground truth of uncontrolled regions. But it's worth noting that the primary contribution of Portrait4DV2 is a new learning paradigm, which is orthogonal to our main contribution. Our method can also benefit from the data generation method of Portrait4DV2.

The approach is highly dependent on 3DMM but lacks a discussion about its impact.

• Thank you for pointing out some missing parts of the paper. We will include this discussion on 3DMM in the revised version.
• 3DMM estimation is important in our method. We use the 3DMM estimation methods provided by GPAvatar (based on EMOCA and MICA) to process our training and inference data. While these methods introduce some errors during our training process, their robustness ensures that our model can still be effectively trained. In the future, our method will also benefit from advancements in 3DMM estimation; the more accurate the estimation, the better our model is expected to perform.
• During inference, if the 3DMM model fails to accurately predict the target expression from the target image, our model will also reproduce these inaccuracies in the driving result. For example, some subtle expressions are not captured by 3DMM (subtle frowns), or the expression is incorrect due to the non-decoupling of identity and expression. Mitigating this issue depends on further developments in the 3DMM estimation field.
• Control without the 3DMM is possible but would require significant modifications to the method. We hope to leave this for future work.

There should be some failure cases to show the limitations of the work.

• We show some limitations in Fig. 3 on the new supplementary page. This includes the results of the tongue that are not modeled by the 3DMM, and the model's lack of details in invisible areas in the input image. These examples will also be integrated into future revisions.

Thank you for your positive review and insightful comments. We are happy to address your questions in the following:

Why not also predict position offsets for the expression Gaussians?

• Our method emphasizes the efficiency of inference rendering. Predicting expression Gaussians offset for each frame will impact the inference efficiency. Our expression Gaussians are controlled by 3DMM vertices now, which can be performed asynchronously with rendering and is very efficient.
• In addition, the offset of the point is also related to the shape and expression. These two parameters determine the position of the point before the offset. At the same time, the number of points is large (5023), which makes it difficult to directly learn the offset of each expression point.
• But we also experimented with adding additional offsets to the expression Gaussians. Our setup is as follows: we extract global features from the driving image, then use the initial positions given by the 3DMM and trainable point features to predict point offsets, and we apply regularization to prevent excessive displacement. The results are shown below.
• | Method | PSNR↑ | SSIM↑ | LPIPS↓ | CSIM↑ | AED↓ | APD↓ | AKD↓ | CSIM↑ | AED↓ | APD↓ | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | | with offsets | 21.93 | 0.814 | 0.138 | 0.797 | 0.121 | 0.168 | 3.732 | 0.584 | 0.263 | 0.294 | | Ours | 21.83 | 0.818 | 0.122 | 0.816 | 0.111 | 0.135 | 3.349 | 0.633 | 0.253 | 0.247 |
• It can be seen that in such a setting, adding offset predictions has little effect or even worse impact on most metrics.

Would the two sets of Gaussians contract each other during expression?

• During the expression (inference) process, the two sets of Gaussians do not contract towards each other. But during training, we use MSE loss to bring the two sets of Gaussians closer together to fully leverage the priors of the 3DMM. So the two sets of Gaussians are close to each other after training.

Why not bind the lifted Gaussians to the 3DMM vertices and animate them together?

• Our primary consideration is efficiency. In our setup, we obtain 175,232 lifted Gaussians and 5,023 expression Gaussians, which require a large matrix to compute their binding relationship. Unlike full-body reconstruction, which involves significant deformations (requiring binding points to a few bones), facial reconstruction doesn't demand such extensive deformations. Therefore, we first attempted to merge the expression Gaussians and enhancement Gaussians directly without binding. The results indicate that this approach is effective for capturing expressions and jaw movements.
• We have further discussed how we resolved the conflict between the lifting Gaussians and the expression Gaussians caused by not binding in our response to reviewer d3t7.

Why do we only use the vertices of the 3DMM model?

• The vertices are naturally compatible with the positions of the Gaussians in 3DGS. Additionally, 3DMM (FLAME) has 5,023 points, providing sufficient information for head expression control. To integrate 3DMM more effectively into 3DGS, we chose not to use edge and face information.
• Moreover, the points in 3DMM contain richer information beyond just shape and expression parameters. Therefore, we opted not to redundantly input the shape and expression parameters of 3DMM into the model.

Side-by-side video comparison with baselines.

• Due to rebuttal restrictions, we are unable to provide additional videos for rebuttal. Instead, we have included some consecutive frames in the new supplementary page and highlighted the areas of interest. We will add side-by-side video comparisons in our future open-source code and demo website.

About the limitations of 3DMM.

• We further discussed the impact of 3DMM in our rebuttal to reviewer 65B2.

Thank you for your positive review and valuable comments. We are pleased to address your concerns in the following sections:

How sensitive is the model to image quality and lighting conditions?

• We present more qualitative results with low-quality images or challenging lighting conditions in Fig. 1 of the new supplementary page. The reconstructed avatars are inevitably affected by image quality and lighting conditions. For example, avatars reconstructed from blurred images lack details, while those from images with challenging lighting conditions have a fixed lighting condition, such as some shadows on the nose. However, these features also demonstrate that our model can faithfully restore details from the input image and handle scenarios with varying image quality and challenging lighting.

How robust is the model to significant occlusions or extreme facial expressions?

• In Fig. 1, we also show the results for inputs with significant occlusions and extreme expressions. For common occlusions such as sunglasses, the model can handle them well. For some uncommon occlusions such as hands, the reconstructed avatar will be affected by the occlusion. For extreme inputs or driving expressions, our method shows good robustness. This shows that our method can produce reasonable results even in extreme cases.

What specific measures were taken to ensure the diversity and quality of the training data?

• We constructed our training data from the VFHQ dataset, a talking head video dataset containing 15,204 video clips. Despite the repeated identities, the dataset still has a large number of unique identities, ensuring sufficient identity diversity in our training data. To ensure diversity in expressions and head poses between frames, we extracted frames from the videos. As described in Sections 4.1 and A.1, we uniformly sampled 25-75 frames from each video based on its length. This temporally sparse sampling ensures that different frames exhibit as much variation in expressions and poses as possible.

• For the ground truth 3DMM parameters in our dataset, we adopted the implementation from GPAvatar [Chu et al., 2024]. This implementation integrates state-of-the-art 3DMM estimators (MICA, EMOCA), resulting in high-quality 3DMM labels.

Implementation complexity and computational resources for training.

• Our method is also highly efficient in training. As shown in the table below, our training costs are lower than those of the baseline methods. In addition, as a generalizable method, our model only needs to be trained once and does not require fine-tuning during inference. In the future, we plan to open-source our code, including implementations of all components, as well as our pre-trained models. We hope this will provide an easy-to-use and solid baseline for future research.

• | Method |StyleHeat | ROME | OTAvatar | HideNeRF | GOHA | CVTHead | GPAvatar | Real3D | P4D | P4D-v2 | Ours| | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | | Time(GPU Hours) |166 | - | 192 | 1667 | 768 | 1200 | 50 | 1424 | 672 | 1536 | 46 |

• Among them, ROME did not provide training details and code. StyleHeat, HideNeRF, and CVTHead did not release official training time, so we conducted limited training and estimated the total time for these methods. All times tested on / reported on /or converted to A100.

About the limitations of 3DMM.

• We further discussed the impact of 3DMM in our rebuttal to reviewer 65B2.