Low-Rank Head Avatar Personalization with Registers

We thank the reviewer for their thoughtful feedback. We are encouraged that they found our Register Module to be a key technical contribution. We are glad the reviewer recognized the clarity of the paper, the perceptual and identity improvements over the baseline (GAGAvatar), and the value of releasing our collected dataset. We also appreciate their observation that LoRA alone produces blurry results, highlighting the importance of improved personalization strategies. We address the reviewer comments below and will incorporate all feedback in the revised manuscript.

Weakness 1 & Question 1: Ablations. We thank the reviewer for their comments. We will incorporate the LoRA baseline results in our ablations table (Table 1) to highlight the difference in the ablations study. We present those results in the table below. We would like to mention that these results are present in Table 2 of the main paper in row 2 (LoRA variant) and row 4 (LoRA w/ register variant) on the VFHQ Test and RareFace-50 datasets. We also present those results in the table below.

The ablation to use xyz coordinates instead of learnable embeddings for vertices in the register module is an interesting approach. Prior approaches, such as NeRF [1] and mip-NeRF [2], use multi-layer perceptrons (MLPs) and positional encoding (PE) to inflate the dimensions of the 3-dimensional input. Using an MLP+PE approach will give us a learnable embedding for each vertex, thus making it similar to our proposed approach.

We present the results for the suggested ablation of replacing learnable embeddings with Gaussian noise below. We observe that using Gaussian noise embeddings leads to an improvement over variant (a) LoRA. However, they still perform worse than variant (b) LoRA w/ Reg (Ours). Qualitatively, we see higher detail preservation as compared to variant (a), but worse detail retention than our method.

We also present the results with DoRA: (c) DoRA [3] and (d) our register module with DoRA. We note that the performance of (c) DoRA is inferior to (a) LoRA. However, introducing our register module during training improves its performance. We also see a similar trend qualitatively.

Note that the constructed dense feature $f_{S}$ has $D=512$ channel dimensions (refer to Ln 288 onwards). Since $f_{dense}^{src}$ features have $256$ channel dimensions, we use $E_{proc-1}$ to scale down the dimensions of $f_{S}$ using $f_{proc-1} = E_{proc-1}(f_{S})$ to support the operation $f_{proc-1}+f_{dense}^{src}$. Hence, using $E_{proc-1}$ is necessary. Note: Best and second-best results highlighted in bold and italics respectively.

Weakness 2 & Question 2: Missing Comparisons. We thank the reviewers for their suggestions. We note that the papers referred to by the reviewer, SplattingAvatar [4], FlashAvatar [5], and MonoGaussianAvatar [6], are head reconstruction methods, which have a different setting from our method. Head reconstruction methods either take multi-view or single-view videos to train a NeRF or variant (e.g., Tri-Planes, Gaussian Splatting, etc), in order to learn the 3D representation of a single identity. In other words, they require separate from-scratch training for each identity on long videos (5-10 mins) and are sensitive to camera calibration errors. We, on the other hand, take a generative 3D-aware one-shot avatar model that is trained on a large-scale in-the-wild dataset and can generate any identity. We propose to personalize it using a short in-the-wild video (5-10 seconds) to preserve identity-specific and high-frequency details.

Weakness 3: Missing Citations. We thank the reviewer for the relevant citations and will gladly incorporate them into our revised manuscript. Cafca [7] uses data-driven priors for novel-view synthesis of expressive faces; however, it does not generate face animations. Tancik et al. [8] use meta-learning to initialize weights in coordinate-based neural networks, with interesting applications, including image regression (but not face animation as in our paper). DoRA [3] is an extension to LoRA, which we have incorporated into our method and compared with.

References

[1] Mildenhall et al., ‘NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis’, in ECCV, 2020.

[2] Barron et al., ‘Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields’, ICCV, 2021.

[3] Liu et al., ‘DoRA: Weight-Decomposed Low-Rank Adaptation’, in ICML, 2024.

[4] Shao et al., ‘SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting’, in CVPR, 2024.

[5] Xiang et al., ‘FlashAvatar: High-fidelity Head Avatar with Efficient Gaussian Embedding’, in CVPR, 2024.

[6] Chen et al., ‘Monogaussianavatar: Monocular gaussian point-based head avatar’, in ACM SIGGRAPH 2024 Conference Papers, 2024, pp. 1–9.

[7] M. C. Buehler et al., ‘Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures’, in ACM SIGGRAPH Asia 2024 Conference Paper, 2024. [8] M. Tancik et al., ‘Learned Initializations for Optimizing Coordinate-Based Neural Representations’, in CVPR, 2021.

We thank the reviewer for their detailed and encouraging feedback. We are pleased they found our paper well-written and easy to follow, and appreciated the structure and clarity of our presentation. We are glad the effectiveness of our 3D Register Module for enhancing identity-specific facial details was recognized, as well as the meaningful role it plays in improving personalization. We also appreciate the acknowledgment of RareFace-50 as a valuable contribution that helps address the underrepresentation of distinctive facial features in current benchmarks. We address the reviewer comments below and will incorporate all feedback in the revised manuscript.

Weakness 1: Limited Novelty Due to Similarity with LAM and Reuse of LoRA. LAM [1] is a one-shot head avatar model that produces an animatable Gaussian head avatar from a single image. Our goal in this paper is to personalize a generalized head-avatar model to a particular identity, such that high-frequency details are preserved. LAM focuses on animating a single image, while we personalize a single image animation model using a short clip.

Furthermore, our model design is entirely different from LAM's, which rigs embeddings to the vertices of a 3DMM's mesh with cross-attention layers to learn from DINOv2 features. In contrast, our register module operates in a learnable 3D feature space and projects back to the 2D space for facilitating LoRA adaptation, and is only used during training.

Weakness 2, Weakness 3 & Question 2: Limited dataset and baseline evaluation. We believe that our approach of low-rank personalization is extendable to other approaches since we use DINOv2 features, which are general domain features and commonly used in various works. We specifically chose GAGAvatar to demonstrate our method because it is a very competitive method, having been trained on a very large dataset (VFHQ [1]), and producing state-of-the-art results in the general avatar domain.

VFHQ is a large-scale and high-quality video face dataset. It includes diverse identities, facial expressions, head poses, and settings, and thus it is used in recent works. On the other hand, we collected RareFace-50 to include unique facial details that are underrepresented or do not even appear in existing datasets. We also provide the evaluation metrics on the HDTF dataset for all baselines and our method below. Note: Best and second-best results highlighted in bold and italics respectively.

Note that the NeRSemble [2] is a multi-view high-quality lab-setting video dataset used for the 3D multi-view face reconstruction task. This dataset is used when the method needs multi-view images and videos. We used other datasets that have in-the-wild monocular videos, which are suitable for our setting.

Question 1: Rasterization instead of visible vertices → projection → interpolation. We appreciate this interesting suggestion as one implementation option. Since this pipeline is not used during inference, it does not affect the speed or the scalability of the method. While we believe it does not have a critical effect on our results, we will explore this option as future work.

References

[1] He et al., ‘LAM: Large Avatar Model for One-shot Animatable Gaussian Head’, in SIGGRAPH, 2025.

[2] Kirschstein et al., ‘NeRSemble: Multi-View Radiance Field Reconstruction of Human Heads’, ACM Trans. Graph., vol. 42, no. 4, Jul. 2023.

We sincerely thank the reviewer for their thoughtful and detailed feedback. We are encouraged that they found our problem motivation strong and practical, particularly the need for high-fidelity personalization beyond what standard LoRA provides. We are pleased the reviewer appreciated the simplicity and intuitiveness of the Register Module, and found the adaptation of Transformer-style registers to a 3D feature space to be a novel and insightful idea. We are also glad they recognized the effectiveness of our method in capturing identity-specific high-frequency details and outperforming both standard LoRA and a meta-learning baseline. We address the reviewer comments below and will incorporate all feedback in the revised manuscript.

Weakness 1 & Question 1: Lack of Architectural Detail. We thank the reviewer for their suggestion. We will highlight the architectural details and the "forward" process in the revised manuscript. We would like to note that we explain the entire "forward" process in Section 3 of the main paper and give additional architectural details of the register module in Section B.3 of the supplementary material. Furthermore, we illustrate our register module in Figure 3. Each vertex in a 3D face mesh is rigged with an embedding, which gets projected to the screen space according to the camera pose, which gives us the 3D feature space on a 2D screen space. These embeddings capture identity-specific information and elicit such information from downstream trainable layers, similar to how registers work in Vision Transformers [3]. We remove the register module during inference, after it has served its purpose of "teaching" LoRA layers to retain fine details.

Weakness 2 & Question 2: Unsubstantiated Efficiency Claims. We thank the reviewer for their comment. We will incorporate changes in the revised manuscript to highlight the fact that the following two points: (a) The register module is only used during training, as stated in L211-212, and (b) The trained LoRA layers are merged with the original parameters for inference using Eq. 1 in Section 3.2. As a consequence of this implementation, our method retains the exact efficiency of the original model during inference, as does the vanilla LoRA variant.

We provide a table of parameter counts below:

Note that during inference, all variant baselines have the same efficiency, due to the reasons explained above. The FPS numbers are measured based on an Nvidia A6000 GPU.

Weakness 3: Limited Discussion on Generalizability. We believe that our approach of low-rank personalization is extendable to other approaches because of the use of DINOv2 features, which are general domain features and commonly used in various works. We specifically chose GAGAvatar to demonstrate our method because it is a very competitive method, having been trained on a very large dataset (VFHQ [1]), and producing state-of-the-art results in the general avatar domain.

Question 3: Temporal Consistency. The video-level prediction is done frame-by-frame. We focus on personalizing GAGAvatar [2] to capture identity-specific details, and thus we do not change its basic architecture. We process the predicted 3DMM parameters using exponential smoothing as in GAGAvatar [2], thereby ensuring temporal consistency in output frames.

References

[1] Xie et al., Vfhq: A high-quality dataset and benchmark for video face super-resolution. In CVPR Workshops (CVPRW), 2022.

[2] Chu et al., Generalizable and animatable gaussian head avatar. In NeurIPS 2024.

[3] Darcet et al., ‘Vision Transformers Need Registers’, In ICLR 2024.

We thank the reviewer for their thoughtful and constructive feedback. We are encouraged that they find the logic of our paper and idea clear, and recognize that our proposed method effectively addresses the limitations of traditional generalized avatar models and existing fine-tuning methods in detail preservation. We address the reviewer comments below and will incorporate all feedback in the revised manuscript.

Weakness 1: Dependency on pose estimation; cascading errors. We have not observed any significant degradation for the vast majority of driving frames. This is because we use the state-of-the-art 3DMM parameter estimation algorithms from Inferno:FaceReconstruction [4]. These estimated parameters do well for our purposes. However, the problem the reviewer pointed out might still exist in extreme poses and expressions, as we pointed out in Section 5 of the main paper and Section F.1 in the supplementary paper. Improving pose estimation is an interesting future work. Given the quality of our results, it is out of scope for this paper.

Clothing details. Cloth modeling is typically considered as a separate modeling problem, independent from face modeling. While FOMM might still be useful for cloth modeling, it is a 2D key-point method that is not as accurate as 3DMM parameters for fine facial motion control. Since our method aims to personalize 3D-aware generic avatar face generation models, cloth modeling is out of our scope. We regret the poor quality of any cloth visualizations; they are presented only for completeness of results. Instead, we point the reviewer to the quality of our results in detailed face features such as wrinkles, tattoos, freckles, etc., which is the focus of our research.

Weakness 2: Bakes source lighting into appearance, harming realism and generalization under new lighting. Our results achieve preservation of specific textural details on faces (such as wrinkles, tattoos, freckles, etc.) in animatable avatars of a generalized avatar generation model. Our work is in line with lighting-agnostic methods (e.g., GAGAvatar[1], GPAvatar [6], Portrait4D [7], etc.) to model avatars. While modeling light for avatar relighting is interesting, it is a separate line of work (e.g., Portrait Video Relighting [8], Holo-Relighting [9]). We leave the task of incorporating lighting effects for the future.

Weakness 3: Training–inference feature mismatch. As described in L127-132, the Register Module is designed to learn a 3D feature space that captures identity-specific information and elicits such information from downstream trainable layers, similar to how registers work in Vision Transformers [10]. We remove the Register Module during inference, after it has served its purpose of "teaching" LoRA layers to retain fine facial details. This is similar to the way registers were proposed for ViTs [10]. They help store and repurpose information during training and are removed during inference. For completeness, we also tried the case when our Register Module is used during inference; however it harms the output quality - we see an increase in the ACD metric by around 5%.

Question 1: Regularization Loss for Register Embeddings. Thank you for pointing out a typo in the text. We would like to bring to attention that while equation 9 in the main paper is correct, the text in line 200 is incorrect. As the equation suggests, minimizing the objective $L_{reg} = \frac{`pcos`(e)}{n(V)(n(V)-1)}$ where $`pcos`(X) = \sum_i \sum_j \frac{X_i \cdot X_j}{||X_i||||X_j||} - n(V)$ is the sum of the non-diagonal elements of a self-pairwise cosine distance, will result in embeddings $e_i$ to be dissimilar to each other. We will make a correction in the revised manuscript.

We experimentally found that $\lambda_{reg} = 20$ performs best. We present the ablation study for this hyperparameter below on the VFHQTest dataset.

Question 2: Qualitative Conclusions on Register Features. As in most current deep-learning methods that use latent spaces, it is impossible to tease out the effect of specific elements on the learnt features. This is the case of Figure 4 in the main paper and Figure 4-11 in the supplementary, where we visualize the projected embeddings. We posit that these features learn information in the 3D space that helps improve learning signals for LoRA, in order to capture fine identity-specific details. Approximating such black-box deep-learning features with networks that use interpretable features, while maintaining accuracy (as in SI-MIL [12]), is interesting future work.

Question 3 & Question 4: Rationale for Register Pipeline. The Register Module learns a 3D feature space that stores and repurposes information for a human face during training, as explained in L127-132. Inspired by registers for ViT [10] that store global information of an image, our Register Module stores the distinctive details of an identity, given different views. We sum the source features with features projected by the Register Module, in order to inject identity-specific information into the representation.

Question 5: Evaluation Metrics. We found that for our purpose of computing how well the fine details are preserved and the avatar is personalized to the input identity, metrics such as PSNR and SSIM were too unreliable. For example, for the case of the VFHQ Test dataset, we get the following results:

Note: Best and second best results highlighted in bold and italics respectively.

MetaPortrait (meta-learning on LoRA weights) gives us the worst performance qualitatively, among all these methods, as can be seen in Figure 6 of the main paper, Figures 2-3 in the supplementary material, and the supplementary video. However, it fares 2nd best in PSNR, and best in SSIM. We believe this is because these metrics do not focus on specific features in faces, but pixel-level features. SSIM specifically is not necessarily designed with human perception in mind (Understanding SSIM [11]). Hence, we rely on the ACD metric and LPIPS, which rely on the perceptual similarity of identity and images, respectively.

References

[1] Chu et al., Generalizable and animatable Gaussian head avatar. In NeurIPS, 2024

[2] Qian et al., Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians. In CVPR, 2024.

[3] Dhamo et al., Headgas: Real-time animatable head avatars via 3d gaussian splatting. In ECCV, 2024.

[4] Daněček et al., Inferno. https://github.com/radekd91/inferno?tab=readme-ov-file

[5] Siarohin et al., ‘First Order Motion Model for Image Animation’, In NeurIPS, 2019.

[6] Chu et al., ‘GPAvatar: Generalizable and Precise Head Avatar from Image(s)’, In ICLR 2024.

[7] Deng,et al., ‘Learning One-Shot 4D Head Avatar Synthesis using Synthetic Data’, In CVPR 2024.

[8] Rao et al., ‘Lite2Relight: 3D-aware Single Image Portrait Relighting’, in Proceedings of ACM SIGGRAPH 2024.

[9] Mei et al., ‘Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image’, arXiv preprint arXiv:2403. 09632, 2024.

[10] Darcet, et al., ‘Vision Transformers Need Registers’, in ICLR, 2024.

[11] Nilsson, et al., ‘Understanding SSIM’, arXiv [eess.IV]. 2020.

[12] S. Kapse et al., ‘SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology’, arXiv preprint arXiv:2312. 15010, 2023.