Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

Dear reviewer, we sincerely appreciate your careful and accurate understanding of our approach. We hope our response fully resolves your concerns.

Q1: Can authors clarify which 3DMM they use? In 3.4, authors say: "3DMM basis is a nonhomogeneous linear equation and has multiple solutions". At least for PCA-based 3DMMs like BFM2009 and [Blanz and Vetter 1999], this is incorrect. Because of this misunderstanding, the paper may have an over-complex setup in its audio-to-motion model.

• A1: We use BFM2009 in all our experiments, though we have also cited [Blanz and Vetter 1999] in B.2. Following your comment, we checked the details in BFM2009 and found that our previous statement "3DMM basis is a nonhomogeneous linear equation and multiple solutions" is incorrect since the expression bases are orthogonal. Due to this problem, it is not necessary for the audio-to-motion model to avoid the MSE loss of expression (in the previous version to prevent over-smoothness, we designed an MSE loss of landmark). To handle this mistake, we have updated the discussion in the corresponding part of the revised manuscript. Specifically, in the blue-colored part in Section 3.4 of the revised manuscript, we replace the incorrect statement with:

  • [Revised Part 1] We choose BFM 2009 \citep{bfm09} as the 3DMM model. Since all expression bases are orthogonal, given the same identity code, the reconstructed 3D face meshes in a video are uniquely determined by the expression code. Hence an L2 error on the expression code, L_ExpRecon, is feasible to be the reconstruction term in training the VAE. To encourage the model to better reconstruct the facial landmark (instead of only the 3DMM parameters), we additionally introduce the L2 reconstruction error of 468 key points of the reconstructed 3DMM vertex, L_LdmRecon, as an auxiliary supervision signal. The training loss of the generic audio-to-motion model is as follows:

• Further, to fully investigate the influence of using MSE loss on expression and landmark, we compare three settings: (1) only use Expression MSE, (2) only use Landmark MSE, and (3) use both Expression / Landmark MSE. We compute the reconstructed landmark L1 error and SyncScore, and the results are shown as follows, we found that using Expression MSE loss alone achieves slightly better reconstruction performance than only using landmark MSE loss but leads to a slightly degraded sync score. we suspect that it is because for our model that predicts the expression code given audio, updating on an Expression MSE is straightforward and easier to optimize than on a Landmark MSE so the model would more focus on the reconstruction. Another finding is that combined usage of these two losses achieves the best performance in terms of reconstruction and sync score. Therefore, in the revised manuscript, we turn to using a combination of Expression MSE and Landmark MSE as the reconstruction loss of the VAE. We sincerely thank you for your careful reading and in-depth understanding of 3DMM which helps improve the correctness and soundness of this paper.

  • | Loss Setting | L1 error on pred. Ldms $\downarrow$ | Sync Score $\uparrow$| | ----------------------------------------------- | --------------------------- | ---------- | | MSE_Landmark (in the original manuscript) | 0.0113 | 6.565 | | MSE_Expression | 0.0104 | 6.438 | | MSE_Expression + MSE_Landmark (revised version) | $**0.0102**$ | $**6.617**$ |

Q4: Can authors provide results from multiple views to show how good the 3D reconstruction is?

• A4: Sure, please refer to this link for a demo of multi-view synthesis produced by our I2P model (Which is extracted from the Real3D-Portrait's final checkpoint). We can observe that the I2P could reconstruct accurate 3D shape and texture given the input image.

Q5: (1) Comparison to Next3D. (2) Also, add discussion on "Generalizable One-shot Neural Head Avatar" to help understand the novelty of this work.

• A5: (1) As for the Next3D, we provide a comparison video in this link. Note that since Next3D has not released their GAN inversion code, we reproduce a Next3D + PTI to achieve face reenactment by combining the official Next3D repo and an unofficial PTI repository. In the demo video, we can see that our method has the following advantages: rendering uncropped full frames, natural torso, better identity preservation, more accurate expression control, and more efficient (ours is one-shot while PTI requires 1000 iterations of tuning). As for qualitative results, we found Next3D+PTI is worse than ours in terms of CSIM, FID, AED, and APD.

  • | Method | CSIM$\uparrow$ | FID$\downarrow$ | AED$\downarrow$ | APD$\downarrow$ | | ---------- | ---------------- | ---------------- | ---------------- | ---------------- | | Next3D+PTI | 0.645 | 58.66 | 0.183 | 0.039 | | Ours | $\mathbf{0.758}$ | $\mathbf{42.37}$ | $\mathbf{0.138}$ | $\mathbf{0.022}$ |

• (2) As for the discussion on GOS-Avatar, as we discussed in $\mathbf{A2}$, our most related improvement is to design a motion adapter that efficiently performs the "source-to-target" transform, instead of GOS-Avatar's "source-to-neutral-to-target" transform that requires additional supervision signals and may bring information loss to the model (We suspect this might be reason that GOS-Avatar has to train an additional appearance branch to help the model to "remind" these texture details). Besides, we adopt a pretraining process to the I2P model, which is not performed by GOS-Avatar. Further, we are the first to consider the head-torso-background individual modeling, which is necessary to achieve realistic video results. Finally, we are also the first work that supports the end-to-end audio-driven task. We have added a discussion on the difference between our method and GOS-Avatar in the blue-colored part in Appendix A of the revised manuscript.

Q6: (minor typo) "3.1 Network Design comprises a stack of SegFormer bock." bock -> block

• A6: Thanks for your careful reading of this manuscript! We have fixed this typo in the revised manuscript accordingly.s

Summary

Following your comments, we have modified the manuscript and we found the correctness and soundness of the paper have been improved. Again, we thank the reviewer for the insightful review and positive recommendation for our paper.

Dear reviewer Kntd,

Thanks for your acknowledgment on our rebuttal. To answer the remaining problems:

(1) About the clarifying the updated Eq.2. The Eq.2 is updated according to our clarification on the input feature of Motion adapter (as discussed in [Revised Part 2] in the revised PDF): we exactly feed the concatenation of source PNCC and driving (target) PNCC into motion adapter, so the input condition of motion adapter is (PNCC_drv, PNCC_src) instead of only (PNCC_drv). By perceiving the information of source expression and target expression, the MA could learn the source-to-target expression transformation. Thanks for pointing out this update, and we have also blue-colored the eq.2 to highlight the difference.

(2) As for update the figure. We have updated the figure 1,2,3,4 in the manuscript with the correct inputs to the motion adapter.

Again, we'd like to thank you for your precious time and valuable comments, which have improved the manuscript by a large margin.

The revision on the 3DMM and the A2M model looks good to me.

I honestly think the expression loss should be sufficient because the landmark positions can be linearly computed from the expression coefficients. But as long as the authors do not claim a huge technical contribution here, this is minor.

We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.

Q1: Should citing important related methods of audio-driven methods, other methods using PNCC, and some work that solve the same problem. Compare with AUD ("Audio-driven talking face video generation with learning-based personalized head pose") and DFA-NeRF.

• A1: Thanks for your insightful comment, we have cited the mentioned related works in the revised manuscript, which we found has improved the soundness of the manuscript. We also compare with DFA-NeRF in this link and AUD in this link. Note that both DFA-NeRF and AUD are person-specific methods that over-fit a target identity with a two-minute-long video. We tried our best to reproduce these results, but since DFA-NeRF seems incomplete, it could only produce reasonable results when driven by the ground truth expression. So the comparison is mainly performed with AUD. Firstly, we can perform a qualitative comparison from the demo videos. We can see that both DFA-NeRF and AUD produce temporal jittering results, and AUD has a slightly worse identity-preserving ability. By contrast, our method is one-shot, temporally stable, identity-preserving, and lip-synchronized. As for the quantitative comparison with AUD, for a fair comparison, we crop and downsample our method into 256x256 resolution to align with AUD. The results are as follows, we can observe better Sync Score, AED, CSIM, and FID over AUD.
  • | Method | CSIM$\uparrow$ | FID$\downarrow$ | AED$\downarrow$ | Sync$\uparrow$ | | ------ | ---------------- | ---------------- | --------------------------- | ---------------- | | AUD | 0.742 | 47.59 | 0.160 | 5.863 | | Ours | $\mathbf{0.775}$ | $\mathbf{42.63}$ | $\mathbf{0.141}$ | $\mathbf{6.630}$ |

Q2: In the methodology, the paper does not clarify how the per-frame head pose is predicted from the audio, in the case of audio-driven talking face generation. More details and clarifications about that should be provided.

• A2: Thanks for pointing out this problem. In this paper, since the prediction of head pose is not our main interest, we devise a naive strategy to obtain head pose sequences from GT videos of CelebV-HQ. To be specific, (1) in the same-identity setting, the driving head pose is exactly the same as the GT video of the source image (source image is the first frame of the test video); (2) in the cross-identity setting, the driving head pose is extracted from the driving video; (3) then, in the audio-driven setting, most importantly, we first estimate the head pose in the source image, then we query 10 videos in the CelebV-HQ that has nearest distance between its first frame's head pose and the source image. Once the candidate videos that provide head pose are sampled, we randomly choose one of them to drive the source image. We acknowledge that an explanation of the head pose selection strategy is beneficial for the soundness of the manuscript, and have added the discussion above in the blue-colored part in Appendix D.1 of the revised manuscript. Thanks again for improving the soundness of this paper!

Dear Reviewer MF4S,

Thank you again for your time and effort in reviewing our work! We would appreciate it if you can let us know if our response has addressed your concern. As the end of the rebuttal phase is approaching, we look forward to hearing from you and remain at your disposal for any further clarification that you might require.

Dear Reviewer MF4S.

As the discussion period is closing in several hours, we would like to know if there are any additional questions. We are glad to answer them.

Again, we sincerely appreciate your insightful review and "Accept" recommendation for our paper.

We thank the reviewer for the constructive feedback and the positive remarks on our qualitative/quantitative results. We acknowledge that your concerns are mainly about the novelty of this paper, and hope our response resolves your concerns fully.

Q1: (1) Given that the overall architecture is very similar to HiDe-NeRF, it is unclear where the improvement of the proposed method is coming from. (2) Is it because a pretrained Tri-plane is a better representation than the multi-resolution tri-plane features of HiDe-NeRF? It would be great if the authors could clarify this

• A1: (1) Actually, there are several significant differences between our method and HiDe-NeRF. Firstly, we pretrain the I2P model while HiDe-NeRF doesn't; secondly, we use the motion adapter to predict a motion diff-plane while HiDe-NeRF resorts to the deformation field; thirdly, we first propose to individually model the head-torso-background parts; fourthly, we propose a generic audio-to-motion model to additionally support audio-driven tasks while HiDe-NeRF only support video-driven.
• (2) As for the performance improvement of ours over the HiDe-NeRF, we think the reasons are three-fold. Firstly, as pointed out by the reviewer, pre-training the I2P model is an important reason since it utilizes the prior knowledge about 3D geometry from the 3D GAN. We have ablated this in Figure 15 of the original manuscript. Secondly, and most importantly, we think replacing the deformation field with our proposed motion adapter is the key reason. The deformation field aims to learn to deform the ray in the canonical space during inference to achieve dynamic scene synthesis. However, based on our experiment (please kindly refer to this link, in which we compare our motion adapter versus the deformation field by HiDe-NeRF), we find deformation field destroys the good 3D mesh reconstructed by our I2P model and tends to predict an ill-posed face surface (instead of a 3D mesh), which results in bad robustness and bad visual quality. By contrast, our motion adapter only edits the minimal part of the 3D face representation corresponding to the input PNCC, hence maintaining the most of geometry and texture information stored in the reconstructed canonical tri-plane. As a result, our method achieves better identity preserving and geometry realness than HiDe-NeRF. The third reason for our better performance over HiDe-NeRF is that we individually model the torso part. By contrast, in HiDe-NeRF the torso part will rotate along with the head part, which is unrealistic. We sincerely thank the reviewer for sharing the valuable feedback, and we have added the discussion above in the appendix of the revised manuscript (Please refer to the blue-colored part in Appendix A of the revised manuscript for more details), which we suspect could better clarify our novelty and the intrinsic mechanism of our better quality than HiDe-NeRF.

Q2: The addition of background and torso modelling, while important, is relatively incremental.

A2: We appreciate it for acknowledging the importance of individually modeling the torso and background segment in the field of one-shot 3D talking generation. We believe that the proposed head-torso-background individual modeling paradigm could pave the way for future works to achieve more realistic results (since most existing works only focus on the head part or jointly model the head and torso). As for the "incremental", we admit that our HTB-SR has adopted many known techniques to guarantee robustness and performance, e.g., the Torso branch is based on the warping-based method. However, we have also come up with several necessary modifications to achieve more realistic results. For instance, in the warping-based torso branch, we propose to replace the implicit key points in the warping-based method with our predefined projected facial landmark, which is necessary to guarantee temporal stability. Also, it is the first attempt that only warps a specific torso segment of the talking person frame, as the previous methods typically warp the whole frame, which results in distortion artifacts in the background part. Besides, when integrating the head-torso-background segments, we propose an alpha-blending-style fusion mechanism, which could handle the robustness issue caused by the naive channel-wise concatenation. We hope the above clarification could better highlight our contribution to the addition of background and torso modeling.

Summary

In summary, following the given comments, we have performed in-depth analysis and experiment to clarify the intrinsic mechanism of our method's advantage over HiDe-NeRF. We also rethink the novelty of the addition of background and torso modelling. With these revision, we find that the soundness and readability of the paper has been improved. Again, we would like to appreciate the reviewer’s valuable review. We sincerely hope the reviewer will reconsider their rating in light of the rebuttal.

Dear Reviewer 8dWt,

Thanks for your acknowledgment on our clarification regarding the contributions. As for the remaining problem "visualization of the normals". We admit that depth alone is not adequate to prove the geometry quality, so a mesh visualization, which could reflect both depth and surface normals information is necessary.

To this end, we have updated a demo video in this link on our demo page, which compares the visualized mesh from Real3D-Portrait and Deformation Field. The visualized mesh contains the geometry information of depth and surface normals. We can see that our method with I2P model and Motion Adapter could generate mesh with good geometry while the Deformation cannot.

As for the details in our visualization proces, we mainly follow the instruction of EG3D in this link. Specifically, we first sample the density in a $[-1,1]^3$ cube of the world coordinate, then export a .mrc file, which is then visualized by a software named ChimeraX.

Finally, we'd like to thank you for your precious time and valuable comments, which have improved the soundness and clarity of this manuscript. We will be very happy to clarify any remaining points (if any).

We are grateful for your positive review and valuable comments, and we hope our response fully resolves your concerns.

Q1: (1) Doubt about the motion adapter's capability of expression canonicalization (or say, neutralization), as without any further stronger constraints, the proposed method cannot fully remove the source image's original facial expression. (2) Can the author show examples of these canonicalized 3D face reconstructions of the input images without applying the target expression? (3) Does it work for expressions where the source image has a wide open mouth and lowered jaw or nearly closed eyes, for example?

• A1: (1) Thanks for pointing out this problem! We acknowledge your concerns that if the motion adapter is only provided with the target PNCC, it cannot work well with the I2P model which could have arbitrary expression in the cano_planes. We apologize for not clarifying this issue in the previous manuscript. Actually, our key motivation in inventing the motion adapter is to directly perform the source-to-target transform in the cano_planes (skipping an intermediate expression canonicalization process that is required by a traditional source-to-neutral-to-target pipeline). In practice, instead of only feeding the target PNCC into the motion adapter, the input is a channel-wise concatenated source-target PNCC image of shape $(C_{src}+C_{tgt})\times H\times W$. So the motion adapter could learn the source-to-target transform only related to the 3DMM geometry prior while eliminating any other texture-related information. We sincerely thank you for your insightful comments and we have added the technical details above to the revised manuscript (Please refer to the blue-colored part in Section 3.2 of the revised manuscript for more details).
• (2) Sure, in this link we provide a demo in which the I2P model is extracted from Real3D-Portrait's final checkpoint and perform the multi-view synthesis task. We can observe that the rendered canonical plane indeed is of the source expression. (3) Yes, as we discussed in the first point, although without an extra expression neutralization process since our motion adapter learns the source-to-target transform by utilizing the input source-target-concatenated PNCC, it could support arbitrary target expression given a source image of arbitrary source expression. To demonstrate this, we provide a demo video in this link, in which we show our method performs well given the critical source images that have a wide open mouth, lowered jaw, or totally closed eyes. To summarize, in this QA, we discussed Real3D-Portrait's generalizability of 3D reconstruction and animation. We'd like to thank you for providing constructive comments to help us enrich the technical details in the manuscript and motivate us to produce these intuitive demos, which we believe are helpful for new readers to perceive the intrinsic mechanism of this paper.

Q2: About novelty. The proposed methods combines several existing ideas from prior works to culminate in a successful large engineered end-to-end solution.

• A2: Thanks for your comments. We'd like to explain the major novelty of this paper as follows. We first propose an efficient motion adapter that achieves high-quality and geometry-preserving 3D face source-to-target animation, without an extra expression neutralization process. Secondly, we are the first work that points out the necessity of head-torso-background individual modeling and come up with an effective HTB-SR model (with several necessary modifications, such as explicit keypoint in the torso branch and alpha-blending-style fusion mechanism in combining the three segments) to handle this issue. Thirdly, we are the first work that achieves audio-driven ont-shot 3D talking face. Besides, the proposed four-stage framework, which supports audio/video-driven applications, is easy to be expanded to support other sub-topics, such as stylized talking faces, etc. We expect Real3D-Portrait to provide a new paradigm for the new generation of one-shot 3D talking face methods to produce realistic video results.

Q3: Specify the dataset/protocol that are used for evaluation.

• A3: Thanks for pointing out this issue. Due to space limitations, in the original manuscript, we only briefly explain the dataset/protocol used for evaluation. To be specific, there are three groups of evaluated data: (1) for the Same-Identity Reenactment, we randomly chose 100 videos in our preserved validation split of CelebV-HQ. (2) for the Cross-Identity Reenactment, we use the first frame of the previously selected videos to obtain 100 identities, then use the expression-pose sequence from a randomly selected video to construct the cross-identity reenactment data pair. (3) for the audio-driven scenario, we use 10 out-of-domain images downloaded from the internet (which are exactly the 10 identities shown in the demo video) and choose 10 audios from different languages to form the data pair (so each method has 100 videos as the test samples). We apologize for the missed dataset/protocol specification in the original manuscript, and we have added it to the blue-colored part in Appendix D.1 of the revised version. Thanks for your suggestion that improves the technical soundness of the manuscript.

Q4: Add comparison to newer (than Face-Vid2Vid) and better performing video-driven 2D facial animation methods.

• A4: We admit that more recent 2D methods that are newer Face-Vid2Vid should be compared for better soundness. We have tested 5 of the 6 mentioned baselines (We failed to reproduce FNeVR since the checkpoint link provided in the official repo is broken). We provide a qualitative comparison demo video in this link, in which we can observe that (1) both the advanced 2D methods and our real3d-portrait achieve realistic results when the head motion is small (refer to the part of Obama in the demo video); (2) most of the tested 2D methods failed to achieve naturalness when given a large head pose motion (refer to the part of KFC in the demo) while our Real3D-Portrait keeps performing well. As for more quantitative results, due to the time limitation, we have come up with the quantitative result of "Thin-plate spline" (TPS) and DPE, which are widely used 2D methods that are considered the successor of Face-Vid2Vid. The tested results are as follows, we can see that our Real3D-Portrait outperforms TPS in terms of. We plan to add the quantitative results of the other 4 additional baselines in the camera-ready version.

  • | Method | CSIM$\uparrow$ | FID$\downarrow$ | AED$\downarrow$ | APD$\downarrow$ | | ------ | ---------------- | ---------------- | ---------------- | ---------------- | | TPS | 0.745 | 43.54 | 0.137 | 0.028 | | DPE | 0.731 | 44.07 | 0.135 | 0.021 | | Ours | $\mathbf{0.764}$ | $\mathbf{41.58}$ | $\mathbf{0.129}$ | $\mathbf{0.017}$ | Note that in the above table, we resample our method from 512x512 to 256x256 resolution for fair comparison with the baselines since they produce 256x256 results.

Summary

Following your comments, we have modified the manuscript and we found the soundness and completion of the paper have been improved and the novelty has been more highlighted. Again, we thank the reviewer for the insightful review and positive recommendation for our paper.

Dear Reviewer yTCz,

Thank you again for your time and effort in reviewing our work! We would appreciate it if you can let us know if our response has addressed your concern. As the end of the rebuttal phase is approaching, we look forward to hearing from you and remain at your disposal for any further clarification that you might require.

For the lowered jaw expression in the source image, I meant an expression like this: https://www.shutterstock.com/image-photo/close-portrait-happy-young-girl-dressed-772513501. The results that you've provided are for an "face look down" expression. Can the authors provide results for a source image like the one shared?

Dear Reviewer yTCz,

We acknowledge that it is necessary to quantitatively compare our 3D-based method with the most recent 2D-based SOTA, MCNet (ICCV 2023). We have updated the quantitative results of MCNet (ICCV 2023) as follows. We can see that MCNet achieves better CSIM and FID than the other 2D baselines, but is slightly inferior to our method. We suspect the hard large-pose cases in the test samples are the reason, as the APD (average pose distance) is relatively worse.

We find the comparison with these state-of-the-art 2D methods helpful to demonstrate the performance of our method. We have added the comparison table in Appendix D.5 of the latest manuscript, and have cited the mentioned remarkable works in our discussion on related work (The paragraph named "2D/3D Face Animation" of Sec. 2 in the revised manuscript).

Finally, we'd like to thank you for your expertise and valuable comments, with which we found have improved the completeness of the related work part and the soundness of the experiment part by a large margin.