SurFhead: Affine Rig Blending for Geometrically Accurate 2D Gaussian Surfel Head Avatars

We would like to thank the reviewer for acknowledging our work's effectiveness such as Jacobian Blend Skinning and Eyeball regularizations. Following section will be the responses of the reviewer's concerns, fine-detailed reconstruction and reporting time complexity.

Fine-detailed Reconstruction Quality

Since the most bottom row of Fig. 5 shows cross-reenactment results which is not easy to validate the subject-centric deformations such as wrinkles. Therefore, to validate for solid manner, we show additional results which shows wrinkle-related geometric detail in this link. As can be seen in the image file, the addition of Jacobian and Jacobian Blend Skinning shows increase of quality of wrinkles. Although this achievement, we regard there has a room for improving this fine-detailed parts. As we mentioned in the Section A.5 in Appendix, we suggested the future work related to this part. Notably, wrinkles and facial shadows are inherently influenced by ambient occlusion, which highlights the importance of dynamic appearance modeling in achieving realistic representations. Incorporating dynamic appearance features related pose and expression conditioned such as color and opacity, is essential. For related work, please refer to GHA [1], HeadGAS [2], and NPGA [3]. This could be also viable option for realistic rendering and geometry, but we intentionally excluded network-inferred color for potential time-complexity overheads. Exploring ways to effectively represent ambient occlusion while reducing such time-complexity overhead presents a highly interesting direction for future research. Nevertheless, we effectively captured deformations beyond 3DMM using JBS, which we would like to highlight as one of our achievement.

Missing Related Work

Thanks for pointing out these related works and for insightful suggestions. We agree that discussing these methods would enrich the context of our work. We have added the literature mentioned by the reviewer to the revised version under the section Neural Surface Reconstruction in Related Work A.1 of the Appendix. Please refer the revised version of our paper.

Training and Rendering Time Complexity Measurement

Please kindly refer to the answer Training and Rendering Time Complexity Measurement. from the general response.

We sincerely thank the reviewer once again for this valuable suggestion; the provided analysis are already incorporated into the current revised manuscript and appendix.

We would greatly appreciate any further comments or discussions on potential improvements to enhance our manuscript.

Reference

[1] Xu, Yuelang, et al. "Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Dhamo, Helisa, et al. "Headgas: Real-time animatable head avatars via 3d gaussian splatting." European Conference on Computer Vision. Springer, Cham, 2025.

[3] Giebenhain, Simon, et al. "NPGA: Neural Parametric Gaussian Avatars." arXiv preprint arXiv:2405.19331 (2024).

Thank you for the R Cr49's positive feedback on the additional visualization results and for recognizing the key role of JBS in dynamic head reconstruction. We are pleased to hear that our response has addressed your concerns. Based on your comments, it seems that our paper now aligns with your expectations.

If there are any remaining points of discussion or clarifications that could further strengthen your assessment of our work, we would be more than happy to address them. Any questions and discussions are welcome.

Your constructive feedback has been invaluable, and we hope the revisions merit a higher rating reflecting our improvements!

We would like to thank the reviewer for considering our work as exceptionally novel and thoroughly evaluated and ablated, as well as for the insightful comments and questions regarding the proposed method and its performance. In the following, we address the remaining discussion points related to our innovation, the potential effect of hair-level geometric quality of proposed framework, and utilizing off-the-shelf head tracking algorithm with monocular videos.

Innovation of SurFHead

We would like to draw attention to Tab. 3, which highlights the significant improvements over the vanilla 2DGS [1] (eyeball regularization added). Specifically, the increase in PSNR from 22.35 to 23.09 is substantial, given that PSNR is measured on a logarithmic scale. Not only PSNR, SSIM, LPIPS, and NCS shows significant improvement compared with vanilla 2DGS. Beyond this improvement, our method introduces valuable mathematical properties, including Jacobian gradients, normal deformation, and Jacobian Blend Skinning (JBS).

Furthermore, we believe these properties lay a strong foundation for advancing dynamic Gaussian models, extending their applications beyond head avatars to encompass full-body, hands and 4D Gaussian representations. As noted by reviewer D2NV, we believe that these advancements can serve as meaningful cornerstones for future research.

Robustness of Geometric Quality of Hair Strands

Hair reconstruction and simulation are among the most challenging aspects of modeling the human head. We believe the variability in hair reconstruction results is primarily due to the nature of the data. When the head is simultaneously talking and swaying, the dynamic motion of hair strands introduces instability during training, often resulting in an averaged representation of the moving strands. This issue arises from the inherent elasticity and non-rigidity of hair strands, as demonstrated in the attached video. Even after the subject's head stops moving, several strands continue to oscillate due to their elastic properties. This phenomenon is illustrated in Fig. 10 of the Appendix. To further support our hypothesis, we included the results of GaussianAvatars with Ours in this link. This result demonstrates that using 3D Gaussians instead of 2D surfels faces the same issue.

Train with Coarse Mesh (monocular datasets)

To evaluate the extent to which our method relies on preprocessing, as requested by the reviewers, we trained on a monocular dataset using a low-cost keypoint-based tracking approach. The qualitative results, which can be viewed at this link, demonstrate that while monocular datasets inherently face challenges in reconstructing normals for occluded areas (such as below the jaw or side profile) compared to multiview datasets, this limitation is not specific to our method but rather a consequence of the narrow field of view in monocular datasets. Nonetheless, SurFhead achieves superior performance compared to GaussianAvatars in capturing fine details such as wrinkles and provides significantly better normal estimations.

We sincerely thank the reviewer once again for this valuable suggestion; the provided analysis are already incorporated into the current revised manuscript and appendix.

We would greatly appreciate any further comments or discussions on potential improvements to enhance our manuscript.

Reference

[1] Huang, Binbin, et al. "2d gaussian splatting for geometrically accurate radiance fields." ACM SIGGRAPH 2024 Conference Papers. 2024.

We thank the reviewer for acknowledging our work's contribution, superiority and novelty, and that its effectiveness is well studied. In the following, we address the remaining discussion points:

Clarifying again the role of $\mathbf{J}_{b}$

For the first, our final parameterization is usage of $\mathbf{J}\_{b}$, not $\mathbf{J}$, as mentioned in Eq. (3) and (4). To give more understanding, the GaussianAvatars' square root of covariance $\Sigma^{1/2}$is defined as $RS = s_{p}R_{p}R_{c}S_{c}$ and position $\mu = s_{p}R_{p}\mu_{p}$, as mentioned in Eq. (1). To handle the shear and stretch deformations, we introduced Jacobian Gradient $\mathbf{J}$ which replaces $s_{p}R_{p}$ term in square root of covariance and position. This formula is also could be found in 214th line. Finally, to tackle the potential discontinuity of local deformations, we replace the Jacobian Gradient $\mathbf{J}$ to blended Jacobian $\mathbf{J}\_{b}$, again. This is clarified in Eq. (3). Specifically, the $\mathbf{J}$ is broken down in to unitary matrix $\mathbf{U}$ and positive semi-definite matrix $\mathbf{P}$. The $\mathbf{J}\_{b}$ is derived from blending with adjacent triangles $\mathbf{U}$s and $\mathbf{P}$s such as Eq. (2). To summarize, there are two transitions for transformations: $s_{p}R_{p}$ $\rightarrow$ $\mathbf{J}$ $\rightarrow$ $\mathbf{J}_{b}$. Note that this deformation is only applied for Gaussians, not mesh triangles which provide the base deformation such as $\mathbf{J}$.

I hope that this explanation helped the reviewer's comprehension. Also, we augmented these flow of transition to current revised version in Section 2.3. Are there any other questions, please leave comments to let us know. We will reply as soon as possible.

Reliable Metric to Evaluate Geometry Quality.

Thanks for your insightful comment. As suggested, we measured the Chamfer distance on the Facetalk dataset id-8, which includes ground truth meshes. Since it is a monocular dataset, we focused on the frontal region and randomly sampled 30 cameras near the frontal view (both azimuth and elevation sampled from $U\sim(-0.8, 0.8)$ in radian scale) to extract meshes using a Truncated Sign Distance Field (TSDF) method. We compared our approach with both FLARE and GaussianAvatars. FLARE was chosen for comparison because it has the second-highest Normal Cosine Similarity (NCS) after our method as shown in Tab. 1. For FLARE, which is a mesh-based method, we directly used its optimized mesh for comparison. For GaussianAvatars, we extracted the mesh using the same TSDF-based method as our approach. As shown in the table below, the Chamfer Distance of the meshes generated by our proposed method outperformed both FLARE and GaussianAvatars. This indicates that our approach achieves an improvement also in mesh quality compared to these methods.

Additional Qualitative Results in Extreme Scenarios

As demonstrated in Fig. 5, we have showcased the qualitative rendering and geometry quality of our method under extreme pose scenarios. To quantitatively evaluate the robustness of our approach in such scenarios, we selected the top 10 largest jaw poses and expressions per subject, totaling 20 images per subject, and conducted evaluations across 9 subjects.

As shown in the table below, our method performs on par with GaussianAvatars in terms of both rendering and geometry quality, demonstrating its effectiveness under challenging conditions.

Effect of Changing 3D Gaussians to 2D Gaussians

Naively replacing 3D Gaussians with 2D Gaussians improves geometric quality but does not enhance rendering quality, as shown in the table below. This trend has also been observed in 2DGS [1]. However, our method surpasses these limitations, demonstrating superior rendering and geometry quality. This is achieved through the proposed Jacobian-based normal deformation, Jacobian Blend Skinning (JBS), and the incorporation of eyeball constraints. Please refer the Table 3. in the current revised version of the manuscript.

Reference

[1] Huang, Binbin, et al. "2d gaussian splatting for geometrically accurate radiance fields." ACM SIGGRAPH 2024 Conference Papers. 2024.

We sincerely thank the reviewer for recognizing the effectiveness and versatility of our work beyond human head avatars, as well as for the insightful comments and experiment suggestions. Below, we address the feedback by contextualizing the SurFhead approach with the suggested related method, providing quantitative and qualitative comparisons with Gaussian Head Avatars (GHA) [1], training and rendering speed evaluations, and additional experiments with eyeglasses-wearing subjects.

Comparison with GHA.

We acknowledge that GHA serves as a strong baseline in rendering quality, leveraging an additional super-resolution model in screen space. To facilitate comparison, we provide additional quantitative and qualitative results in the table and images below. As shown in the table, GHA outperforms our method in terms of PSNR. However, in other metrics such as SSIM and LPIPS, GHA falls behind.

Please be aware of this link. The attached images reveal potential reasons for this discrepancy. Notably, artifacts such as over-saturation are visible, which we attribute to GHA's screen-space refinement. This refinement struggles when rendering extreme poses that fall outside the distribution expected by the super-resolution model. Additionally, GHA faces challenges in reconstructing high-fidelity geometry and handling extreme expressions, both of which are key strengths of our approach. We would like to emphasize the importance of capturing the specular highlights of the eyes, particularly the cornea. GHA renders the pupils with a matte appearance, neglecting the high-frequency specular reflections in the corneal region. We argue that these specular details are critical for enhancing the realism of the eyes, which play a pivotal role in creating immersive and lifelike head avatars.

We will augment these results with additional subjects in Fig. 5 and Tab. 2 of the manuscript for the camera-ready version.

Train with eyeglasses wearing identity.

We noticed that the dataset provided by GaussianAvatar does not include data for subjects wearing glasses, so we had to preprocess new data accordingly. Additionally, it took some time to gain access to the NeRSemble dataset, and we are currently in the process of preprocessing it. We will share the results with you as soon as possible. Thank you for your understanding and patience!

Training and Rendering Time Complexity Measurement

Please kindly refer to the answer Training and Rendering Time Complexity Measurement. from the general response.

We sincerely thank the reviewer once again for this valuable suggestion; the provided analysis are already incorporated into the current revised manuscript and appendix.

We would greatly appreciate any further comments or discussions on potential improvements to enhance our manuscript.

Reference

[1] Xu, Yuelang, et al. "Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

We appreciate your time and attention.

We have successfully trained the eyeglasses-wearing subject (NeRSemble - 079). As demonstrated in the video and image showcasing the self-reenactment task, our method achieves high-quality geometry reconstruction and rendering. Even for a subject wearing eyeglasses, our approach faithfully reproduces both the convex eyeballs and the eyeglasses with high fidelity.

While our method does not explicitly model eyeglasses, such as capturing the metallic frame's reflectivity or the specular effects of the lenses, these aspects are not the central focus of the dynamic head avatar task. Future work could explore integrating a parametric eyeglasses modeling framework, such as MEGANE [1], to create a unified avatar system.

Reference

[1] Li, Junxuan, et al. "Megane: Morphable eyeglass and avatar network." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Dear Reviewer D2NV,

We wanted to kindly follow up to inquire if you’ve had the chance to review our updated manuscript and the responses addressing your discussion points. Your feedback is extremely valuable to us, and we would greatly appreciate any additional questions or comments you may have.

Thank you for your time and consideration.

Best regards, The Authors

Thanks for the response. After reading the response and other reviews, I would like to maintain my rating.