TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction

Thank you for your valuable comments and acknowledgement of our effectiveness. We hope our response fully resolve your concerns.

Q1: The rendered portrait clips still suffer from over-smoothed textures.

To clarify, our motivation is to provide more accurate photometric supervision for 3D expression reconstruction. Reconstructing high-frequency details for each image using a general model is inherently challenging. As shown in Fig. 4, our method can better reconstruct and capture highlights and shadows compared to other methods. However, rendering high specular regions and heavily occluded areas while maintaining image sharpness remains challenging, as discussed in our limitations Sec.. We will investigate this further in future work. Thank you.

Q2: Lack of an analysis of the concept represented by the multi-scale appearance tokenizer.

Thank you for your meaningful suggestion. We have demonstrated in Table 2 that using multi-scale tokens improves our method's performance. To further analyze how different scales of appearance tokenizer affect face reconstruction, we designed an experiment where we sequentially swap appearance tokens of different scales between two individuals. Please refer to Fig A-5 in the revised paper for the details. We found that 1）Shallow-level appearance tokens relate to facial texture details (e.g., skin tone) and 2) Deep-level appearance tokens capture high-level semantics (e.g., gender, shape of eyebrow, etc.) This observation aligns with findings from previous work (VGG loss, ResNet). Additionally, our renderer generates facial motions conditioned on the input face mesh. Since multi-scale appearance tokens don't contain facial geometry information, our reconstructed face images maintain consistency with the input mesh geometry. We demonstrate this capability for various facial expressions in our supplementary video(at 1min 7s).

Q3: Running some experiments on a dataset with ground-truth 3D geometry can possibly support it.

Thank you very much for pointing out these. In the Appendix Table A-1, we compare our method with additional approaches on the Stirling benchmark for 3D geometry reconstruction, where we achieve comparable or better results. We have also validated our method on the NoW benchmark and FaceWareHouse dataset, demonstrating superior performance on both. More details please refer to Reviewer N9UC Q1.

Q4: What contributes to the better temporal consistency?

Our method achieves better temporal consistency through two main factors:

1. Better reconstruction of facial details
2. Stable appearance token estimation

Additionally, we use only one token (first frame) for expression-driven face video generation.

Q5: How to measure the matching between the geometry and the rendered face?

We evaluate the matching in following ways:

1. We employ various 2D metrics to measure shape and expression consistency between reconstructed and input faces.
2. We validate using datasets with ground truth 3D facial scans.

Q6: Can we say that the contributions to better shape-matched facial rendering all come from the neural facial renderer, rather than the geometry shape estimator?

No, it cannot. Our renderer primarily conditions on the facial mesh and appearance tokens, which do not include geometric information about the face (for more details, please refer to Reviewer N9UC Q3).

The facial shape and expressions in the renderer come entirely from the facial mesh, which is obtained by rendering the FLAME mesh produced by the encoder. Its accuracy of facial geometry directly impacts the results of the renderer. In addition, we adopt a token-cycle loss and an iterative training strategy to reduce the effect of the neural renderer compensating for the encoders. Furthermore, our experimental results on several 3D benchmarks also validate that our method improves geometry shape.

Summary

Based on your suggestions, we have added more analysis of the different level appearance tokens to Fig. A-5 and corresponding analysis to Sec. E. We think this enhances the paper and better explains the method. Thank you again for your valuable comments.

Q4: Imagine the SMIRK renderer with more parameters would be able to learn them without explicit inductive bias.

Thanks for your excellent comment. Our renderer has only a modest increase in parameters compared to SMIRK's renderer. We experimented with increasing SMIRK's renderer parameters to match or exceed our model size. We found that increasing the learnable parameters of the renderer did not significantly improve the performance of SMIRK. Overall, our method achieves significantly better results, as shown in the table below:

Summary

Thank you for your valuable comments. Following your advices, we have added the quantitative comparisons on three 3D face reconstruction benchmarks and discussions to Sec. D in Appendix. We have also added number of network parameters analysis to Sec. D in appendix. We believe such discussions and updates makes the paper more complete and better explains the method. Thank you again for your valuable feedback.

Thank you for your valuable comments and acknowledgement of our novelty and effectiveness. We hope our response fully resolve your concerns.

Q1: Contains no measure of ground-truth reconstruction accuracy.

Thank you for your reminder. First, we have provided comparison results on the Stirling benchmark in our Appendix, where our method achieves comparable results with SOTA methods. Second, following the reviewer's suggestion, we evaluated our method on the NoW benchmark, where it also performs well (results shown in the table below). Note that both the Stirling and NoW benchmarks primarily focus on face shape reconstruction accuracy, which is not our main contribution, our method still shows competitive performance. Finally, we compared our method on FaceWareHouse, a dataset that provides 3D geometry meshes with different expressions across 150 subjects. We used the now_evaluation codebase with standard (non-metrical) evaluation to calculate median/mean/std metrics. All images from FaceWareHouse are used for benchmarking. Methods marked with * in below table used additional 3D datasets during training, making direct comparisons with our method unfair. Nevertheless, our method still achieves comparable results.

Q2: This would make sure the renderer is actually learning the appearance tokens and not just copying the input pixels.

A2: We have designed our method specifically to prevent the renderer from simply copying input pixels, specifically:

1. Our token features are spatially flattened, which eliminates spatial information from the input images.
2. We design token cycle loss to enforce the learned token robust to different facial expressions.
3. We alternately train the encoders (including the geometry encoder and MFAT) and TFS during the training process. This design helps reduce the effect of the neural renderer compensating for the encoders.
4. In our supplementary video (at 40s), we have demonstrated that our method can generate stable facial video with one fixed token, which are extracted from the first frame.

For more training details and loss details, please refer to Sec. B.1.

Q3: In general the sensitivity of the appearance tokens to different expressions/views.

In fact, we considered this situation. To reduce the sensitivity of the appearance token to different expressions, we designed the following approach and conducted validation.

1. We introduced a cycle token loss with expression augmentation to ensure tokens remain consistent for the same person across different expressions.
2. To enhance appearance token stability and prevent tokens from simply copying input pixels, we spatially flatten the token features, which eliminates spatial information from input images.
3. Our experiments validate the disentanglement capabilities, showing that tokens can be used for downstream tasks, including generating facial videos with new expressions and views.
4. For better understanding, we visualized the features in Fig. 8, demonstrating that our learned appearance tokens cluster by identity and are minimally affected by different expressions/views.

We are grateful for your positive review and valuable comments, and we hope our response fully address your concerns.

Q1: The paper is heavily based on the architecture / methodology of SMIRK

A1: We acknowledge SMIRK's valuable insight that analysis-by-neural-synthesis supervision is essential for accurate 3D expression reconstruction. However, our method differs significantly in several technical aspects:

1. We designed a hybrid representation for 3D expression reconstruction and introduced a more accurate facial renderer.
2. We introduce our method with several novel components, including 1) multi-scale tokens, 2) token cycle loss, 3) pose-dependent keypoint loss.
3. Our approach achieves both 1) Accurate 3D expression mesh extraction and 2) Implicit and interpretable facial appearance representation.
4. Compared to SMIRK, our method not only supports multiple downstream tasks but also generates more stable and high-quality portrait reconstructions. Furthermore, our method enables better video driving results

Therefore, we believe these differences clearly distinguish our work from SMIRK including architecture and methodology.

Q2: What is the inference execution time of the proposed approach?

A2: On an RTX 3090, our method runs at an overall speed of 20.43 FPS. Both the Encoder and Face Renderer operate above real-time, with the Encoder at 43.94 FPS and the Token-guided Face Synthesizer at 45.77 FPS. Our model can be easily converted to a ONNX model, after which the overall speed increases to 29.77 FPS.

Q3: How is the parameter $\epsilon$ tuned in equation 13?

A3: We compared the keypoints obtained from the FLAME mesh with the pseudo keypoints estimated by InsightFace to determine at what head pose there is a significant difference. We empirically set the parameter $\epsilon = 0.05$ by observing face images with various image with different headposes.

Summary

Regarding your comments, we add detailed inference time to Section D, and add $\epsilon$ settings to the main paper. Please find the updated contents in the revision. We highlight new contents with blue color. We think such updates enhance the paper and better explain the method. Thank you again for your positive rating and valuable feedback.

Q5: More discussion about ethics risks.

Thanks for your constructive suggestions. The proposed method for 3D reconstruction of facial expressions, while innovative, raises significant ethical concerns due to its potential misuse. Specifically, the ability to edit faces and transfer expressions could facilitate the creation of deepfake videos, which might be used to manipulate the likeness of individuals without their consent. This includes generating highly realistic but fabricated content, such as making individuals appear to say or do things they never did or placing their likenesses in misleading or harmful contexts. Such misuse poses risks to privacy, consent, and trust in digital media, as well as broader societal implications, such as spreading misinformation or damaging reputations. Although Section D.2 in the Appendix addresses these concerns, the discussion is insufficiently detailed given the gravity of the ethical risks. Future work should prioritize an in-depth exploration of these issues, including robust mitigation strategies, transparent usage guidelines, and technical safeguards to prevent misuse.

Summary

Following your comments, we add more design details and discussions to the pose-guided landmark loss. Due to page limitations, we have included more quantitative comparisons with DECA, 3DDFA-v2 to Table A-2 in Appendix. Finally, we add more discussions about ethics risks to the ethical considerations. Please find more details in the revised paper, in which the updated contents are highlighted with blue color. Thank you again for your suggestions and positive feedback for our paper.

We are grateful for your positive review and valuable comments. We hope our response fully resolves your concerns.

Q1: About the novelty over the closely-related method of SMIRK.

We first apologize for any lack of clarity in our previous explanation. We acknowledge that SMIRK indeed inspired our work. However, there are many significant differences in several technical aspects:

1. We designed a hybrid representation for 3D expression reconstruction and introduced a more accurate facial renderer.
2. We introduce our method with several novel components: a) Multi-scale appearance tokens b) Expression-independent token cycle loss c) Pose-dependent keypoint loss
3. Our approach achieves both a) Accurate 3D expression mesh extraction and b) Implicit and interpretable facial appearance representation
4. Compared to SMIRK, our method: a) Supports multiple downstream tasks b) Generates more stable and high-quality portrait reconstructions c) Enables better video driving results

We believe these differences clearly distinguish our work from SMIRK. In addition, our experimental results demonstrate better performance, including:

• Significant improvements in 3D face expression reconstruction
• Enhanced stability in video portrait reconstruction (7-38s)
• Better performance on the Stirling benchmark. We have also demonstrated more competitive performance on NoW benchmark and FaceWareHouse benchmark (Please refer to Reviewer N9UC-Q1). We appreciate that other reviewers have also acknowledged these improvements.

Q2: About the pose-dependent masking of eq. (13).

Thanks your for the suggestion. Our initial focus was on demonstrating that this loss term brings significant improvements to the model. Our experiments show that it enhances both 1) overall model performance, and 2) local facial details, particularly around the mouth corners and eyes. Since our selected facial keypoints focus on the eyes, nose, and mouth regions, we observed that:

• Yaw angle significantly affects the accuracy of nose and jawline keypoints, please see Fig. A-2 for more details.
• Pitch angle has minimal impact on keypoint loss.

Therefore, we think such design is effective. We agree that using a smoothly-varying confidence score is a reasonable suggestion. We will investigate this further in future work. Thank you.

Q3: About experimental protocol.

We first apologize for the misunderstanding. We clarify that we did not use videos of the same individuals for both training and testing. Following previous methods, our training and testing samples come from different videos with no overlap between them. So we believe the comparison is fair and adequate. We will make this clearer in the revised paper.

Q4: The paper includes quantitative as well as qualitative comparisons with only two recent SOTA methods of 3D face reconstruction. Other methods could have been included

Thanks for your advise. We have provided visual comparisons with several SOTA methods in Fig. 3-4. Following your suggestion, we have included more quantitative comparisons with other methods, shown below. Our method consistently achieves better performance across metrics:

We have also present comparisons with additional methods on three 3D facial benchmark in Appendix Table A-1, where our method achieves overall better performance in 3D face shape reconstruction.

Q4: About comparison with SOTA methods for downstream tasks.

Thanks for your suggestion. The main purpose of our downstream tasks is to demonstrate the disentanglement capabilities of our learned hybrid representation. Specifically:

• Face mesh can explicitly drive facial movements
• Appearance tokens can accurately swap facial appearances

More comparisons with SOTA methods specifically designed for these downstream tasks will be our future work.