AniHead: Efficient and Animatable 3D Head Avatars Generation

Thank you for your valueable comments. Please refer to the general response for information about 4K resolution results. Here are the answers to the other questions:

Q1: skin color

A1: Besides texture resolution, reviewers are also concerned about the skin color generated by our method. This issue is partly caused by the lighting settings in rendering software, such as Blender. We have updated the relevant images in the main paper and the appendix to better highlight the skin color variations.

Q2: Why selection of training examples is needed

A2: We use only 50 samples because, generally, LoRA training requires just a few dozen samples. Besides, due to the inductive bias of SD, the generated data by SDS is not very fair in terms of gender, ethnicity, and age. Therefore we perform a simple selection from 600 sample to form our training set. Moreover, we empirically find that using more sample for LoRA finetuning not only leads to longer finetuning time, but also results in overfitting problem. The finetuned model will concentrate on generating normal-looking humans, while having poor generation of contents that are not part of the dataset and require large areas of color blocks, such as The Joker and Batman.

Q3: Comparison with SDS

A3: We have updated the results in our appendix. As shown in Fig.10 in the appendix, the data generated by SDS does contain some blurry samples, but the final generated results are clearer, thus showing the effectiveness and necessity of our method.

We thank reviewer 9Wmk for the valuable time and constructive feedback.

Q1: synthesize 3D hair for "Taylor Swift."

A1: Thank you for your comment. As mentioned in Sec 3.2, the addition of accessories such as hats and hair are solved by post-processing steps. This procedure is similar to DreamFace, where candidate hair is selected and added to the generated avatars. Specifically, we put the generated 3D head into Blender and add a default hair accessory for it. We have updated the caption of Fig.1 for better understanding.

Q2: difference in the input for SD during data preparation

A2: We utilize SDS to generate both shape and texture training data. Originally, SDS optimizes latent 3D parameters like NeRF for generation. Since we want to generate decoupled shape and texture instead of single 3D shape, we have to optimize two sets of parameters controling shape and texture respectively. Specifically, shape paramter $S\in R^{1 \times 100}$ is optimized with SDS. $S$ can be mapped to 3D head shape by fixing the pose and expression parameters to 0 in the FLAME model. Formally, $T, M_{normal} = FLAME(S)$, where $T$ is mesh shape and $M_{normal}$ is the according normal map. Then, we use a differentiable renderer $R$ with a random camera pose $C$ to obtain the shader image $I = R(T, C, M_{normal})$. We then encode $I$ into the latent space to obtain $z_t$, and we use the SDS loss to get the optimized shape $S'$.

For texture, we fix the shape parameters $\hat{S} = S'$ and set the parameter to be optimized as texture map $M_{\text{tex}}$, which is initialized as the mean texture for four different skin colors. Similarly, we obtain $I = R(T, C, M_{\text{tex}})$, which reflects the effect of mapping the texture map to be optimized onto the 3D shape.

We have updated this detail in the appendix.

Q3: quality constrained by the use of FLAME

A3: It is true that FLAME may have limitations in representing certain accessories such as hair or hats. However, using a parametric model has its advantages:

(1) The parametric model incorporates a human head prior, providing constraints that enable the generation of more reasonable results, while implicit representation methods struggle or fail to generate reasonable results (as shown in Fig.3). Both shape and texture are more controlable and reasonable, which had also mentioned in our overall respose.

(2) Our approach is animatable, allowing for easy integration with off-the-shelf driving methods to create animations, as demonstrated in the gif in Fig.5. In contrast, implicit methods, although capable of representing hair, are not animatable and lack downstream applications.

(3) For static 3D head avatars, adding hairs by post-processing, as adopted by us and DreamFace, would not lead to significant performance gap with latent-based methods.

We have updated the discussion in our appendix.

We thank reviewer ggv2 for the valuable time and constructive feedback.

Q1: Novelty in terms of pretrained CLIP and LDM

A1: Thank you. We would like to stress that using pretrained foundation models like CLIP and LDM is common in the current vision tasks due to their strong capability. We have not listed such techniques as our contribution. Instead, as mentioned in the general response, our novel pipeline for 3D head avatar generation along with the data-free training strategy and the usage of mean texture token for representing general human facial details are the key contribuion of our paper.

Q2: Mean texture token v.s. DreamBooth and domain-specific prompt tuning

A2: We have explained the difference between our mean texture token and identifier in DreamBooth in Sec.3.2. We quote the original content here: "It is noteworthy that while the proposed mean-texture token is similar to the unique subject identifier utilized in DreamBooth, the actual effect is indeed different. The identifier in DreamBooth is aimed to represent a specific subject. In contrast, our mean-texture token helps complement text prompts by representing common texture-wise features shared by human beings." Moreover, our method differs from DreamBooth in implementation. DreamBooth finetunes the entire large model and requires a prior preservation loss. In contrast, our approach employs the LoRA method for training, which adds low-rank layers to the freeze original Stable Diffusion, reducing overfitting and training consumption due to fewer parameters.

As for the domain-specific prompt tuning in DreamFace, we think it has essential difference with our mean texture token. The domain-specific prompt tuning was proposed to help model identify the unwanted domain data resulted from the collection. In short, the method is used for data filtering. On the contrary, our mean texture token is proposed to represent the prior knowledge of human face textures, as mentioned in Sec.3.2.

Q3: ICT-FaceKit v.s. FLAME

A3: Thank you. The advantage of FLAME introduced in our paper is in comparison to latent representation methods like Nerf and DMTet. With this linear shape parametric model, we can easily control the shape using 100-dimensional parameters and easily integrate with a CG pipeline to achieve downstream tasks, such as video-driven animation.

Both our FLAME model and DreamFace's ICT-FaceKit belong to the 3DMM parametric model category and share the advantages mentioned above. The differences lie in:

(1) The texture coverage of ICT-FaceKit does not include the entire face, covering only the ears and the frontal face without eyes, while our FLAME model's texture covers the entire head, which can be seen in the videos in our supplementary material.

(2) Our usage of the 3DMM model also differs from DreamFace. DreamFace samples shape parameters from a multivariate normal distribution N(0,1) to obtain one million candidates, followed by 300 carving steps. In contrast, we design the AniHead model pipeline to directly learn shape parameters, inferring the shape parameter vector directly from the prompt. This not only results in higher shape-text matching (as shown in Tab.1 and Fig.3) but also reduces inference time (ours can generate shape within seconds).

Q4: more generation results demonstrating varying age, skin colors

A4: As both our method and DreamFace use parametric models as shape model, they can share a CG pipeline to achieve effects such as hair selection and video-driven animation, as demonstrated in the gif in Fig.5. We have updated Fig.4 in the main paper to demonstrate the editing capabilities for age and skin colors. As can be seen from the results, our model is versatile to multiple types of editing on different people.

Q5: Unclear writing and missing citations

A5: Thanks for pointing out. We have updated these problems in the new version.

Q6: 3D view

A6: We have provided the required videos in the supplementary material. The results are consistent with the original results in our paper, supporting the efficacy of our method.

Q7: data-free training strategy

A7: Thank you. We would like to kindly point out that 'data-free' refers to the fact that manual data collection is not required. In the SDS generation process, the only preparation needed is providing a prompt, such as "a portrait of {name} with neck and shoulder, without hair", which can be easily created. Moreover, the generalization ability has been shown in results in Fig.4. Given limited training data generated by SDS, these results further prove the effectiveness of our proposed pipeline.

We would like to clarify that the Stable Diffusion model is freeze during the SDS optimization process, because SDS only needs to leverage the prior information from Stable Diffusion.

The major changes are as follows:

1. We've updated Fig.1-3 in the main paper and Fig.8-9 in the appendix with more suitable material and lighting settings, to better show the skin color variety as suggested by 3kTT, and with 4K textures, as mentioned by ggv2 and 3kTT.

2. We've added more editing results in Fig.4 to demonstrate the editing capabilities for age and skin colors, as suggested by ggv2.

3. We've added SDS Training Details and in appendix as suggested by 9Wmk.

4. We've further discussed our choice of FLAME model in appendix according by 9Wmk.

5. We've updated the caption of Fig.1 for better understanding of the CG pipeline postprocessing effect, according by 9Wmk.

6. We've added Fig.11 in appendix to show the effect of mean texture token, as suggested by ggv2.

7. We've corrected the unclear writing and missing citations as pointed out by ggv2.

8. We've provided 3D view videos in the supplementary material, as suggested by ggv2.

9. We've added Fig.7 in appendix to better demonstrate our Classifier-free guidance discussion, as suggested by ggv2.

10. We've added Fig.10 in appendix to compare the result from SDS optimization and inference, as suggested by 3kTT.