MI-NeRF: Learning a Single Face NeRF from Multiple Identities

We are thankful to the reviewer x1A2 for their effort and time to review our paper. We address their concerns below:

Q1: Learned latent codes.

We included in the appendix C a qualitative comparison of our model trained with and without the latent codes (see Fig. 10). In the revised manuscript, we denote with red color the changes for visual distinction. As the indicative frame illustrates, our learned latent codes only capture very small variations in appearance and high-frequency details, while the expression and the identity remain the same (see also ablation study in Sec. 4.2). Thus, our experimental analysis exhibits that there is no overlap of these codes with identity and expression.

In practice, we ensure this by adding a higher L2 regularization on the latent codes (see $\lambda_{l}$ in Sec. 3.3). This results in latent codes only obtaining small values and capturing very small per-frame variations, that are not captured by the powerful representation of the expression parameters in the PCA space, or the time-invariant identity code.

Q2: Comparison with GAN-based methods.

Earlier works that propose GAN-based methods for expression transfer, like GANimation [1], operate in the 2D image space. Similarly with Wav2Lip, these 2D-based approaches frequently produce artifacts, since they cannot handle 3D motion (see Fig. 4, Sec. 4.4, and appendix C).

Recent state-of-the-art GANs propose 3D-aware approaches, like Next3D [2], that are based on EG3D [3] and use StyleGAN2 [4] priors. The output identity depends on the input latent code and therefore on the random seed. In order to render a specific person, they require to run a time-consuming GAN inversion (e.g. PTI [5]). This process is required in order to project the true identity to the closest identity the model can express. To achieve that, they use the closest latent codes and fine-tune the generator weights, in order to approximate an identity, given their image. In contrast, in our case, we can directly learn a code for each identity and render them with a high fidelity. In addition, these 3D-aware GANs require very long training times (e.g. training EG3D for 8.5 days on 8 Tesla V100 GPUs [3] and another 4 days training for Next3D on 4 3090 GPUs [2]).

Q3: Dataset details.

We included in the pdf an additional section in the appendix, Sec. F, where we provide additional details for the dataset. As Sec. F clarifies, we collected talking face videos from publicly available datasets, commonly used by other published works. We provide a detailed list of the identities and videos used, as well as the corresponding links that include the origin of the datasets and the license. A visualization of the learned identity codes is given in Fig. 8.

As mentioned in Sec. 3.1 and Sec. F, we run standard pre-processing techniques to fit a 3DMM per video frame and extract the corresponding head pose and expression parameters. We refer the reviewer to [6] for more details in the data pre-processing.

If the reviewer has any remaining concerns, we would be happy to address them.

Q4. Limitations of the multiplicative module.

Our proposed multiplicative module learns non-linear interactions between identity and expression. To learn these interactions, we use talking face videos that include a variety of facial expressions from different identities. We believe that a limitation would arise if there is no overlap between the expressions of different identities. As shown in [7], multiplicative interactions can be successfully captured if there is some overlap between different attributes, in order to learn all possible combinations. However, the case of no overlap would not easily appear in a real-world scenario, since it is very probable that different subjects will show some similar expressions, e.g. neutral expression or a smile, while talking.

---

References

[1] Pumarola et al., GANimation: One-Shot Anatomically Consistent Facial Animation, IJCV, 2019.

[2] Sun et al., Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars, CVPR, 2023.

[3] Chan et al., Efficient Geometry-aware 3D Generative Adversarial Networks, CVPR, 2022.

[4] Karras et al., Analyzing and improving the image quality of StyleGAN, CVPR, 2020.

[5] Roich et al., Pivotal tuning for latent-based editing of real images, ACM Trans. Graph., 2021.

[6] Guo et al., Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images, IEEE transactions on pattern analysis and machine intelligence, 2018.

[7] Georgopoulos et al., Multilinear latent conditioning for generating unseen attribute combinations, ICML, 2020.

We are thankful to the reviewer ud4c for their time and effort to review our paper.

Q1: Comparison with other multi-identity NeRF-based methods.

We believe that we have included all the relevant qualitative and quantitative comparisons with other multi-identity NeRF methods. As mentioned in our paper, to the best of our knowledge, MI-NeRF is the first multi-identity NeRF for faces, learned from monocular videos of multiple subjects. For facial expression transfer, we show comparisons with the single-identity NeRFace, its immediate extension "Baseline NeRF" trained on multiple identities, and the NeRF-based parametric model HeadNeRF (see Sec. 4.3, Fig. 2, Table 2). However, if the reviewer ud4c is aware of other multi-identity NeRF-based methods and would like to see a comparison with those, please let us know.

---

Q2: Fast training time is a known benefit for multi-identity NeRF-based methods. For the training time evaluation, it misses the comparison with other multi-identity methods, like HeadNeRF.

According to the respective paper, HeadNeRF requires about 3 days of training on a single NVIDIA 3090 GPU, which is similar to our reported time for training of 100 identities.

If the reviewer has any remaining questions, we are happy to address them. We would appreciate it if the reviewer provides any concrete multi-identity NeRFs they believe we should compare with.

We are thankful to the reviewer TTon for their time and effort to review our paper. We address their concern below:

Q1: High-degree module.

We conduct experiments using the high-degree module H, please see Table 1. The results indicate that it performs on par with the module M that we propose and visually we also observe a similar visual quality to the module M.

In addition, this module can be even more useful when interactions between additional variables should be captured, e.g., if we include lighting conditions in the future. Therefore, we do believe that this method is useful in our methodological part.

We are thankful to the reviewer d6cv for their time and effort to review our paper. We address their concerns below:

Q1: How the face expressions are aligned or handled.

We perform standard processing techniques. Please let us elaborate on those. As described in Sec. 3.1, we fit a 3D morphable model (3DMM) for each video frame, and extract the corresponding head pose and facial expression parameters. A 3DMM [1] is a parametric model that represents the human face as a linear combination of principle axes for shape, texture, and expression, learned by principal components analysis (PCA). Thus, the extracted expression coefficients are in a common space for different identities. The extracted head pose corresponds to rotation and translation matrices that are used to transform the sampled 3D points to the canonical space (see Sec. 3.1). We also refer the reviewer to the cited work of [2] for more details.

Q2: Lighting conditions.

Handling different lighting conditions is an interesting research problem, but beyond the scope of this paper. We refer the reviewer to related works that address this problem [3, 4]. In our case, in addition to expression and identity codes, we learn per-frame latent codes (embeddings) that capture time-varying information (see Sec. 3.1). These codes are used to capture small per-frame variations in appearance and illumination, and reconstruct them in the generated videos, in order to enhance the final visual quality (see ablation study in Sec. 4.2). Since our training videos do not include any significant lighting changes, these latent codes only give a very small increase in performance. Handling larger lighting changes is an interesting problem for future research.

Q3: Quantitative comparisons.

We provide extensive quantitative comparisons with the relevant state-of-the-art methods in Sec. 4.3 and 4.4, and in Table 2. If the reviewer d6cv is aware of any other relevant state-of-the-art work and would like to see a comparison with those, please let us know.

Q4: How to deal with different identities.

As described in Sec. 3.1, we learn an identity code per video. This is a learnable embedding that is optimized during training (see torch.nn.Embedding from PyTorch). By enforcing it to be the same for all the frames of a subject, we encourage it to capture time-invariant information. These identity codes are used in the proposed multiplicative module (see Sec. 3.2).

Q5: With respect to ethics review.

We included in the pdf an additional section in the appendix, Sec. F, where we provide additional details for the dataset. The changes in the manuscript are highlighted with red for visual clarity. As Sec. F clarifies, we collected talking face videos from publicly available datasets, commonly used by other published works [5, 6, 7, 8, 9]. We provide a detailed list of the identities and videos used, as well as the corresponding links that include the origin of the datasets and the license.

In addition, our original submission does include an ethics statement as encouraged by the ICLR guidelines. If the reviewer believes something else should be added there, we would be happy to include it.

If the reviewer has any remaining concerns, we would be happy to address them.

---

References

[1] Blanz et al., A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on computer graphics and interactive techniques, 1999.

[2] Guo et al., Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images, IEEE transactions on pattern analysis and machine intelligence, 2018.

[3] Wang et al., Portrait Reconstruction and Relighting using the Sun as a Light Stage, CVPR, 2023.

[4] Jiang et al., NeRFFaceLighting: Implicit and Disentangled Face Lighting Representation Leveraging Generative Prior in Neural Radiance Fields, ACM Transactions on Graphics, 2023.

[5] Guo et al., AD-NeRF: Audio driven neural radiance fields for talking head synthesis, CVPR, 2021.

[6] Lu et al., Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation, ACM Transactions on Graphics, 2021.

[7] Chatziagapi et al., LipNeRF: What is the right feature space to lip-sync a NeRF, International Conference on Automatic Face and Gesture Recognition, 2023.

[8] Zhang et al., Flow-Guided One-Shot Talking Face Generation With a High-Resolution Audio-Visual Dataset, CVPR, 2021.

[9] Liu et al., Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation, ECCV, 2022.