TimeWalker: Personalized Neural Space for Life-long Head Avatar

Q1: The reconstructed lifelong avatars exhibit significant artifacts beyond those associated with the mouth region, as noted by the authors. However, there is a lack of discussion regarding the origins of these artifacts. For instance, could the limited availability of data during certain life stages be a contributing factor?

A1: Due to the page limitations of the main paper, we briefly address this in Section 5. And yes, data limitations contribute to the observed artifacts, as the mouth region is often occluded, leading to incomplete supervision during training. This results in artifacts during exaggerated expressions, as shown in Figure 6. Incorporating generative priors or curated datasets specifically for the mouth region could help mitigate this limitation.

Q2: It would be beneficial to explore the advantages of incorporating multiple lifetimes within the representation for this task. However, the paper lacks a discussion on this point, particularly concerning the extent of performance improvement attributed to the inclusion of additional lifelong data.

A2: We discussed the benefits in Sec.3.5 Lines 424-430 and will further emphasize these in the revised version. In summary, our model learns a person's personalized space with the support of neural "mean head" and principal head basis, making the space steerable and fostering high-quality results under various settings.

Key advantages include: full-scale animation, reenactment, high-quality rendering, disentanglement of attributes, identity preservation, and enhancement of downstream task performance. Additionally, incorporating multiple lifetimes within the representation helps compensate for data scarcity, particularly in geometry reconstruction and producing side-view rendering outcomes. Refer to the answer to your Q7 for more details on this benefit.

Q3: The use of blendshape concepts within the Gaussian Splatting framework bears similarities to prior work. "3d gaussian blendshapes for head avatar animation.(3DGS-B)"

A3: The concepts differ fundamentally between our work and 3DGS-B in both modeling scope and complexity. 3DGS-B focuses on and can only learn a linear expression space under a single appearance modeling condition. In contrast, our framework, tackles the more complex challenge of lifelong modeling, serving as a first baseline to construct a neural personalized space towards life-long scale. This space captures not only expression variations but also shared and distinct features across multiple life stages. It achieves compelling attribute disentanglement, enabling comprehensive modeling and dynamic, multidimensional animations of personalized head avatars under diverse conditions. Besides, additional by-products (as replied in the last questions) can be inherently generated from our neural personalized space.

One more important point we would like to highlight is that the vanilla blendshape concept has been extensively applied in various studies combined with different representations (e.g., Nersemble [2], NerfBlendshape [4]). However, these approaches focus on modeling momentary-level spaces. The novelty focus for no matther these papers, or our work, is not about simply combining blendshape concepts with Gaussian Splatting or NeRF, but in how the space could be learned and the unique properties of the learned space. Building on such a classic linear combination concept, we have developed a neural personalized space that redefines the scale and adaptability of head avatar modeling.

Q4: The adoption of the multi-resolution Hashgrid has also been utilized in INSTA, further limiting the originality of the approach.

A4: The multi-resolution hashgrid is a foundational representation introduced by instant-ngp [1] and has been widely adopted as a common strategy across various fields. For head modeling, for instance, INSTA utilizes a single multi-resolution hashgrid to directly store head avatar features, while NerSemble [2] employs a fixed number of hashgrids.

In contrast, our approach dynamically adjusts the number of multi-resolution hashgrids during training. This adaptive design allows the model to capture deeper characteristics rather than merely memorizing superficial appearances, enabling more comprehensive and personalized head avatar representations (Refer to Sec. 3.2.2, Lines 297–315).

Q5: What method was employed to obtain the FLAME parameters for the constructed dataset?How accurate are these FLAME parameters? Additionally, how might the accuracy of the preprocessed FLAME parameters impact the fidelity of the final avatars?

A5: We adopted the FLAME fitting preprocessing inspired by INSTA with the following step: (1) MICA[5]-based FLAME parameter initialization; (2) Optimization process based on the Metrical Photometric Tracker. Given the discontinuous nature (camera switching) of the camera within one life stage, this step needs additionally heightened the learning rate for the camera extrinsics. (3) Post-optimized filtering. Owing to the increased noise inherent in in-the-wild data, the accuracy of FLAME parameters tends to be lower compared to data from "standard inputs". To mitigate this, we implemented a filtering rule post-FLAME fitting to eliminate low-quality FLAME parameters.

In general, substandard FLAME parameters can introduce artifacts in rendering results and diminish the performance of expression animation. Refer to Fig. 6 to observe how the less accurate FLAME result impacts the intensity of the animation with exaggerated expressions.

Q6: Is the representation of lifetime spaces continuous?

A6: Yes, the demo on our project page also demonstrated the continuity of personalized space in multiple scales.

Q7: What is the performance of the method from different viewpoints? The results presented primarily focus on frontal views.

A7: The head pose distribution in our Timewalker-1.0 dataset exhibits a predominantly long-tail distribution, as illustrated in Fig. 9. This distribution highlights the insufficiency of side view data for modeling identity features at wide angles. Thus, our emphasis is primarily on rendering viewpoints around views with angles inside 30 degree, to showcase the effectiveness of our method in multi-dimensional reconstructions and animations

Two observations for sideview: (1)our model outperforms other SOTAs even when confronted with missing view data. By constructing a comprehensive personalized space across different lifestages. This can capture the overall representation of the identity. When certain view data is unavailable, our model generates the view renderings that reflect the average representation of the identity, rather than merely introducing artifacts. (2) While effective within moderate view ranges (Refer to Fig. 17 Row 1, the visualization of the novel view rendering), the model struggles to handle extreme side views like 90 degree. In the revised version, we will expand the discussion on the performance of side-view rendering.

Q8: The specific lifetime avatars exhibit noticeable shadowing. Given the videos captured at different times, could it be possible to mitigate these shadows to enhance the visual quality of the results?

A8: To eliminate shadowing from the rendering output, the model must essentially learn features that are independent of shadowing effects. Several potential approaches can be considered: (1) Incorporate the modeling of light effects by including albedo map prediction during model training; (2) Utilize the robust capabilities of pre-trained generative models to address and mitigate shadowing issues effectively.

Citation

[1] Müller T, Evans A, Schied C, et al. Instant neural graphics primitives with a multiresolution hash encoding. In ACM  TOG 2022.
[2] Kirschstein T, Qian S, Giebenhain S, et al. Nersemble: Multi-view radiance field reconstruction of human heads. In ACM  TOG 2023.
[3] Zheng Y, Yifan W, Wetzstein G, et al. Pointavatar: Deformable point-based head avatars from videos. In CVPR 2023.
[4] Gao et al Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video. In SIGGRAPH Asia 2022.
[5] Zielonka W, Bolkart T, Thies J. Towards metrical reconstruction of human faces. In ECCV 2022.

Q1: The specific role of this residual embedding is unclear. Is it because the expressiveness of the head basis is insufficient? There seems to be a lack of ablation studies. Is the I_{res} that was consistent for all age stages?

A1: The residual embedding I_{res} constitutes learnable parameters that serve as global compensators for each lifestage, like adjusting lighting conditions. Through the incorporation of residual embedding in our pipeline, we partially disentangle lighting effects from lifestage-specific attributes, allowing lighting adjustments while preserving the core characteristics of each lifestage. In our revision, we will include an analysis and discussion to clarify the specific role and impact of the residual embedding.

Q2: For example, during inference, how are the blend weights for head basis determined?

A2: After model training, for each lifestage, a set of specific blending weights are learned and assigned to the multiple multi-solution hashgrids for each lifestage. By adjusting these blending weight coefficients, we can exercise precise control over the progression of lifestages.

Q3: When the age is 20, how is the age-to-blend weight mapping obtained?

A3: Not sure if we understand your question correctly. If you are inquiring about the blending weight associated with the specific age of 20 for an identity, it is important to note that our model does not explicitly define the concept of "Age." Instead, it constructs an interpolatable personalized space based on the notion of lifestage (refer to Sec.A.2). However, as a possible solution, an age-to-blend-weight mapping can be approximated for each identity by labeling specific ages with the corresponding lifestages and their blending weights.

Q4: The results contain several artifacts, especially during eye-blinking and around the head edges.

A4: In Sec.5 Lines 527-533, we shortly discuss how data preprocessing inaccuracies and data constraints contribute to rendering artifacts. Here is more concrete unfolding on these two types of artifacts:

(1) Eye-blinking: our dataset, curated from in-the-wild unstructured sources, inherently exhibits a long-tail distribution. While our method addresses data imbalance to some degree, rare occurrences of eye-blinking during training can still result in artifacts during these sequences in inference.

(2) Head edges: the discrepancies observed around the head edges stem from inaccuracies in the image background matting process. The complex and varied backgrounds in in-the-wild data can result in imprecise matting, introducing additional noise in the region around the head edges during training. These challenges underscore the need in the future to develop methods for creating a high-fidelity head avatar that minimizes its reliance on data preprocessing techniques.

Q5: In the video results, different life stages exhibit varying lighting conditions. That means several age-irrelevant attributes, such as lighting, were not decoupled from age.

A5: The diverse lighting conditions present across different lifestages pose significant challenges to fine-grained attribute disentanglement for constructing head avatars. By integrating residual embedding into our pipeline, we can partially separate lighting effects from lifestage attributes. For better disentanglement of lighting effects, some possible solutions could be: (1) incorporating Albedo map modeling into the pipeline to isolate the intrinsic properties of the subject's appearance from external lighting effects; (2) adding lighting augmentations and equivalent representation learning during training, foster invariant representations robust to lighting variations.

Q6: Why not use a general basis instead of having a specific head basis for each person?

A6: There is a trade-off between opting for a general basis vs a person-specific head basis. Employing a general basis such as the 3DMM style enables the generation of a head avatar in a standardized representation, albeit at the cost of losing individual personal characteristics. Conversely, our objective is to create a personalized space capable of accurately and effectively capturing changes across various lifestages while preserving common characteristics and personality. If you want to establish a general basis for multiple individuals, constructing a conditional space with instance-specific designs to compensate for missing details would be a more effective approach.

Q7: How many head bases are learned for each person?

A7: The number of head bases for each person is dynamically learned by the network, and typically ranges from 6 to 10 in our experiments. Refer to Sec.3.2.2 Lines 297-315, in which we answer the question "How to set the number of head basis? " in detail.

Q8: What is the inference speed?

A8: Refer to reviewer TANo Q5.

Q1: Main contributions are two losses: lpips and consistency loss.

A1: At no point in the paper we claim LPIPS or consistency loss as our contributions. These metrics are well-established, and commonly utilized strategies in the field, which are appropriately cited.

Q2: The application demonstrates the Gaussian-based approach's generalization capabilities and flexibility.

A2: Interpreting this evaluation as a demonstration of GS approach’s flexibility misrepresents the paper’s clear purpose. The purpose of this evaluation is clear: to demonstrate the benefits of our neural parametric model, which achieves 3D-consistent and personalized outcomes. It is worth noting that DGE (Chen et al.) itself relies on GS as its base representation. The side-by-side comparison (Figure 17) illustrates the limitations of GS+DGE, such as incoherent geometry and blur, and how our method (Ours+DGE) addresses these issues. Above are all explicitly stated in Sec. A6.

Q3: Comparison with the recent work mentioned by reviewer- PAV: Personalized Head Avatar from Unstructured Video Collection.(ECCV2024)

A3: (1) Note that PAV was released on 22nd, July, 2024, making it a concurrent work with ours. (2) PAV builds on INSTA by extending it to support multiple appearances through the introduction of appearance embeddings. In our paper, we have already implemented and compared INSTA++, an enhanced version of INSTA, which integrates learnable latent factors into the MLP (Lines 1015–1017, Section A.5.2). This extension allows INSTA++ to model transitions across multiple life stages, aligning in spirit with PAV and enabling a fair and meaningful comparison. For a comprehensive analysis of the results, refer to Fig.11 & Tab.5 & Lines 1074-1129.

Q4: Societal negative impact/potentially harmful applications.

A4: Considering the potential societal negative impacts around sourced data, we strictly obey the non-commercial license (Refer to Sec.A.7). While the real and rich human head assets generated by our method could be misused for deceptive applications, such as fake talking head video generation, they also have the potential to be leveraged for positive uses, such as training robust deepfake detection models. To mitigate risks, we will only make the dataset available to individuals or institutions who have reviewed and agreed to our proposed data usage agreement, ensuring ethical and responsible use.

Q1: The visual effects of this method have many artifacts, which can usually be avoided when using a large amount of data (15k-260k per person).

A1:

(1) To clarify, we use 8,000–20,000 images per person, with train-test splits detailed in Section A.5.2 (Lines 1059-1060). For reference, single-appearance head reconstruction paradigms like INSTA [4], I AM Avatar [2], PointAvatar [3] ,and GaussianAvatars [5] typically use 1,700–5,368 images. Our dataset spans all life stages, making the data allocation per stage relatively modest.

(2) Larger in-the-wild datasets often introduce noise, imbalance, and challenges rather than improvements. Similar observation has also been shown in other related human-centric fields (e.g., Li et al, StyleGAN-Human, ECCV 2024). Also, scaling data volume does not directly address entangled issues across dimensions. Concrete evidence supporting the reviewer's claim is needed.

(3) Subjective assessments vary. Our method demonstrates better quantitative and qualitative performance over SOTAs (Sec.5.2, Lines 1074-1144, Tab. 5&6, Fig.11&12). Providing examples with better results than ours would strengthen your review argument. While not perfect, we aim our work as a baseline for this new task, and the remaining challenges could also drive future advancements in lifelong personalized avatar learning.

Q2: The transitions between different time periods are not natural enough, which has a significant impact on achieving the main goal of this method: seamlessly altering their appearance within a predefined set of life stages.

A2: Our primary focus in this paper is the establishment of a neural personalized space spanning various life stages, which could benefit reconstruction and animation in multiple dimensions (e.g., shape/appearance/expression/) across lifestages , rather than solely transitions between different time periods. (as stated in Sec.3.5 Lines 424-430)

Q3: Most importantly, the approach has not been thoroughly compared, making it difficult to assess its contribution. For example, the method lacks discussion of recent Gaussian head methods (Gaussian Head Avatar/SplattingAvatar)

A3: (1) We conduct a thorough comparison (Sec.5.2, Lines 1074-1144, Tab. 5&6, Fig.11&12) based on several critical criteria to reveal the factors that might influence the accomplishment of the task: (a) Base representations. Thus we categorize and select representative SOTA methods into NeRF-based (INSTA, and modified version INSTA++), Gaussian Splatting-based (FlashAvatar), and generative methods (GANAvatar). The selection of Flashavatar as a representative state-of-the-art Gaussian head method is deliberate. (b) Gaussian Splatting variants. To further explore the impact of different design choices within the base Gaussian Splatting (GS) paradigm, we provide an in-depth evaluation of GS variants (Gaussian Surfels, Gaussian Surfels++). This ensures a precise analysis of GS impacts and limits. It would be better if the reviewer could list the necessity and essence to compare with Gaussian Head Avatar/SplattingAvatar, and the reason to support the "most important" claim of comparing them.

Q4: There is also a lack of comparison with one-shot head reconstruction methods.

A4: The references highlighted by the reviewer predominantly concentrate on:

(1) Generalizable On-shot NHA/Portrait4D-v2/GPAVATAR: head reconstruction and animation from single view images/videos.

(2) Real3D-Portrait: primarily designed for talking face generation.

While effective for their tasks, these methods are 3d generative model --EG3D [6]variants, and tasks are momentary. They lack the ability to construct a personalized space for full-scale animation across life stages. We've compared with GANAvatar, a representative work that shares similar 3d generative model benefits (in Sec.5.2 Lines 1130-1144, Tab. 7 & Fig. 14), while the results cannot preserve identity across ages or achieve consistent quality under the same training data conditions.

We encourage the reviewer to show the evidence that proves these methods align more closely with our research goals or can achieve personalized, animatable head modeling as comprehensively as our approach with only using the same training data conditions.

Q5: How fast is the rendering of this method during inference?

A5: Approximately 5-6 FPS on a single A100 GPU. This is improvable through further optimizations, e.g., adopting a C++ implementation of insta-ngp and leveraging libraries like tiny-cuda-nn to accelerate MLP networks. While rendering speed is not the primary focus of this work, we demonstrate superior performance and efficiency compared to recent reconstruction methods (see Sec. A.5.3, Lines 1215-1220).

Q6: The method does not evaluate the identity preservation of the person when controlling for lifestage and shape.

A6: We acknowledge that the current work does not explicitly evaluate identity preservation. ArcFace [1] could be used for obtaining quantitative evaluations. We will incorporate ID preservation assessment in future revisions. Additionally, we expect the reviewer can point out specific visual results where identity preservation appears to be compromised or worse than other methods, as this feedback will help refine our analysis.

Q7: When switching between different life stages, do you consider the order of switching time? Or multiple life stages switched randomly?

A7: Both sequential and random switching between life stages can be supported by our method. After training, each life stage is associated with distinct learnable blending weights assigned to multiple multi-resolution hash grids. By adjusting these blending weight coefficients, we can achieve precise control over the progression of lifestages, ensuring controllability of the lifestage transitions. Alternatively, by setting random blending weights, the model can produce varied outcomes across different life stages.

Q8: It seems that the Gaussian and 3DMM of this method are bound together (during initialization). Why is it that when it is driven by moment (change the expression), it does not directly use 3DMM to deform the Gaussian points, but introduces a warp field instead?

A8: The Gaussian Surfels are aligned with 3D morphable head model at initialization. However, in our personalized space construction, which accommodates varied appearances across life stages, strictly binding during training would limit adaptability. To enable learning features like long hair or other life-stage-specific traits, we relax the Gaussian points from the 3D morphable head model. Fundamentally, our head avatar is still driven by FLAME expression/shape parameters. However, the absence of a direct mapping between Gaussian Surfels and the FLAME template make it unfeasible to directly employ morphable models for deforming the Gaussian points. To address this challenge, we introduce an inverse warping operation adapted from the methodology found in INSTA[4], in which details are unfolded on Sec.3.3.1 Lines 355-368.

Q9: In Table 5 (in 1 vs N), FlashAvatar reports lower results than INSTA. The experiments (1vsN) has a similar setting to FlashAvatar paper, but the results is inconsistent with the statement in the FlashAvatar paper that FlashAvatar performs better than INSTA. What could be the possible reason?

A9: While the experimental setup for Table 5 (1 vs. N comparison) aligns with the FlashAvatar paper, the key difference lies in the dataset: for Table 5, we used Timewalker-1.0, which differs significantly from the datasets used in FlashAvatar.

Timewalker-1.0 is composed of unstructured data, unlike the high-quality, front-view "standard inputs" used in FlashAvatar. Variations in data distribution, head poses, expressions, and noise levels likely explain the differences in results.

To evaluate robustness and generalization, we conducted additional experiments with "standard inputs" data (Table 6). In this scenario, FlashAvatar outperformed INSTA, consistent with the findings reported in the FlashAvatar paper.

Citation

[1] Deng J, Guo J, Xue N, et al. Arcface: Additive angular margin loss for deep face recognition. In CVPR2019.
[2] Zheng Y, Abrevaya V F, Bühler M C, et al. Im avatar: Implicit morphable head avatars from videos. In CVPR2022.
[3] Zheng Y, Yifan W, Wetzstein G, et al. Pointavatar: Deformable point-based head avatars from videos. In CVPR2023.
[4] Zielonka W, Bolkart T, Thies J. Instant volumetric head avatars. In CVPR2023.
[5] Qian S, Kirschstein T, Schoneveld L, et al. Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians. In CVPR2024.
[6] Chan E R, Lin C Z, Chan M A, et al. Efficient geometry-aware 3d generative adversarial networks. In CVPR2022.