A Large-Scale 3D Face Mesh Video Dataset via Neural Re-parameterized Optimization

We thank the reviewer for the time and constructive comments that strengthen our paper. We discuss the concerns and questions of the reviewer below and in the revision (please check pdf, highlighted in pink).

Please let us know if our answers satisfy the reviewer’s concerns. We would be happy to provide further discussions and clarifications.

Weaknesses

W1. A previous work [C1] has already introduced this technique to solve very relevant (actually more challenging) tasks; they adopt a neural network to re-parameterize the per-vertex displacement of a FLAME mesh. However, the submission does not discuss these very relevant works.

We appreciate the reviewer for recommending the reference. We have revised the paper to include the discussion about the suggested reference in the 4th paragraph of Sec. 2 of the revision, highlighted in pink.

However, please note that our work and [C1] are focus on very different perspectives as:

1. We first remark that, in contrast to ours, Neural Head Avatars [C1] does not re-parameterize FLAME parameters with neural networks, but directly optimizes FLAME parameters which corresponds to the baseline we presented.
2. In contrast, we provided the empirical analysis about the gradient densification (Sec. 3.3, p. 5 in the initial submission and Sec. A.1, p. 17 in the Appendix), which can be attributed to the proposed neural re-parameterization, and hasn’t been explored in the fields. While [C2] also tries to make the dense gradient for geometry optimization, it uses a different technique, pre-conditioning, which is different from our neural re-parameterization. Also, we briefly showed the robustness and high-probability convergence guarantee of the optimization to global optima in the theoretical proof sketch of the initial submission (Sec. A.2, p. 18 in the Appendix).
3. They parameterize residual fine-detail geometries of face using an MLP, which is not our scope for reconstructing details of face shapes. Our scope is to reconstruct facial geometries, well complying with input facial gestures and motions as discussed in the response to W1 of Reviewer axFo.

Given these, [C1,C2] do not harm our contribution. Thanks for suggesting references.

[C1] Grassal et al., Neural head avatars from monocular rgb videos. In CVPR 2022.

[C2] Nicolet et al., Large Steps in Inverse Rendering of Geometry. In SIGGRAPH ASIA 2021.

W2. The video FLAME fitting algorithm is not well-designed.

• W2-1. Why not use a photometric loss term?

As requested, we report the effect of the photometric loss ($\mathcal{L}\_{photo}$) in the metrics for the MEAD dataset we used in the main paper. We compare the optimization that uses three different loss configurations as below:

We tried this in our preliminary experiment, and even with the tuning of balancing, we obtain negligible effects of using $\mathcal{L}\_{photo}.$ Using $\mathcal{L}\_{photo}$ rather degrades the performance, and the larger $\mathcal{L}\_{photo}$ results in more degradation.

We postulate that, since $\mathcal{L}\_{photo}$ is designed to enforce the consistency of colors, i.e., albedo assumption, thus it has been known to be very prone to self-shadows, and non-Lambertian reflection by lighting, and noise existing in videos, which distract the optimization. Furthermore, it takes more than twice the GPU memory than the original configuration, due to the requirement of the differentiable rendering. Thus, we claim our design choice is sufficient to reconstruct dynamic 3D faces despite its compact combination.

We have newly added the results and discussion in the revision (Sec. A.4, p. 21 in the Appendix).

• W2-2. Why not use a shared identity code?

We tried this in our preliminary experiment, but found negligible differences in our early test. In our newly conducted investigation, we found that the standard deviation of the identity codes ($\boldsymbol{\beta}$) of all frames, gradually decreases and converges during NeuFace optimization.

This means, the identity codes across video frames and views automatically converge, although we do not manually force the shared identity codes with the losses. Accordingly, in the visual results, we obtain identical results in both 1) original NeuFace optimization and 2) NeuFace optimization with consistent $\boldsymbol{\beta}$.

We have revised the paper and added this discussion and results in the revision (Sec. A.4, p. 21 in the Appendix)

• W2-3. More advanced strategies beyond the temporal moving average method should be exploited, like the common-used optical flow loss, or stabilize the detected landmarks [C3].

We thank the reviewer for suggesting diverse methods for improving each component in our optimization. As discussed in the response to [W2-1], our design choice is sufficient to reconstruct dynamic 3D faces despite its compact combination. It is also sufficient to validate the idea of neural re-parameterization with bootstrapping. Thus, adding more losses does not change our message and finding of our work we conclude (This is the goal of a piece of the academic paper).

On the other hand, we agree with the reviewer’s point that using advanced losses would increase the accuracy. Note that we can readily add more advanced losses for our proposed optimization. Given the limited rebuttal period, we’ll investigate other advanced combinations of methods later, and add the relevant discussion in the camera-ready version.

[C3] Naruniec et al., High-Resolution Neural Face Swapping for Visual Effects. In Eurographics Symposium on Rendering 2020.

W3. Why not compare with these more strong baselines specialized in video fitting? In the supp. material, the method is compared to MICA-T, but I think more competitors are expected.

We compared with MICA+Tracker as the representative competitor because of the following reasons:

1. MICA+Tracker [C4] is the most competitive video fitting work, which already demonstrate the surpassing performance over Face2Face [C5], and other previous baselines [C6,C7]. Ours better performs than MICA+Tracker, which implies ours performs also better than [C5,C6,C7].
2. For [C8,C9], their codes are not publicly available. During the discussion period, we reached out to the authors of [C8,C9] and requested to share the code, but we couldn’t get a reply. We cannot even quote their quantitative results, because they did not present the benchmarks shared with both MICA and ours.

Their quantitative results cannot be directly quoted, because the benchmarks they presented are not compatible with ours. Despite our efforts, we could not manage to add them. Instead, we have newly added the reference in the revision (Sec. C.2, p. 24 in the Appendix).

[C4] Zielonka et al., MICA - Towards Metrical Reconstruction of Human Faces. In ECCV 2022.

[C5] Thies et al., Face2Face: Real-time face capture and reenactment of rgb videos. In CVPR 2016.

[C6] Deng et al., Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In CVPRW 2019.

[C7] Feng et al., Learning an animatable detailed 3D face model from in-the-wild images. In SIGGRAPH 2021.

[C8] Cao et al., 3D Shape Regression for Real-time Facial Animation. In SIGGRAPH 2013.

[C9] Cao et al., Real-time high-fidelity facial performance capture. In SIGGRAPH 2015.

W3. Effectiveness of the multi-view consistency loss. In the training set, only the MEAD dataset consists of multi-view video data, while VoxCeleb2 and CelebV-HQ have only single-view video data. Consequently, it appears that only the MEAD dataset can effectively leverage the multi-view consistency loss.

We’d like to remind that our work focuses on the effectiveness of our proposal, the neural re-parameterization with bootstrapping (temporal and multi-view consistency).

Thus, we show that

1. Even with the temporal bootstrapping alone in the majority of data, we demonstrate the effectiveness of our NeuFace dataset by significantly enhancing the 3D reconstruction quality in the public benchmark (Table 3a, Sec.5.1. in the initial submission).
2. Also, Table 3a, Sec.5.1. in the initial submission shows that the model trained with our NeuFace dataset significantly improves 3D reconstruction quality for extreme side view cases, where the temporal consistency alone is weak while the multi-view consistency effectively works. This suggests that even with the small portion of data involving multi-view consistency as the reviewer mentioned, the multi-view consistency helps.

Regardless of which bootstrapping (temporal or multi-view consistency) is, it does not alter our conclusion that the neural re-parameterization with bootstrapping is effective. Also, while the multi-view consistency is important, for scalability of the dataset, it is a practical consideration to leverage large-scale monocular video data.

We will consider processing and enlarging the portion of multi-view datasets, e.g., HUMBI [C5], to provide FLAME parameters when releasing the dataset.

[C5] Yu et al., HUMBI: A Large Multiview Dataset of Human Body Expressions. In CVPR 2020.

Questions

Q1. The proposed dataset contains a large amount of data, and the preliminary 3D mesh results generated based on DECA (EMOCA) may have errors. Have the authors considered how to filter out failed reconstruction results?

Our NeuFace optimization is robust to the initial 3D face estimation so that small number of failure does not affect the performance. In the newly conducted experiment, we have compared the multi-view consistency of the final reconstruction on two scenarios: 1) Standard cases (use original images) and 2) Extreme cases (use perturbated images). For extreme cases, we randomly apply perturbation with large black boxes to the facial areas for 2~3 views in the multi-view videos to mimic significant corruption scenarios.

The results show our optimization is robust to initial failure that occurs in the extreme cases.

Furthermore, to filter out the potentially remaining erroneous samples or outliers, we conducted 1) automatic filtering and 2) manual human verification when we constructed our NeuFace dataset. Please refer to Sec. B in the appendix of the initial submission.

Q2. The quality of the reconstructed results for extreme facial expressions appears suboptimal.

The failure cases stem from the limited performance of the off-the-shelf 2D landmark detection algorithm for extreme facial expressions. Since 2D landmark human annotations are relatively cheaper than any other signals, we think using better 2D landmarks can mitigate this limitation. We newly added this discussion in the revision (Sec. E, p. 24 in the Appendix).

Also note that these failure cases are rare, and we showed our annotation method’s effectiveness even in the presence of these rare cases, by enhancing the existing 3D reconstruction model with our dataset (Table 3. in Sec. 5.1.). We hope the reviewer acknowledges this perspective.

Q3-1. Room for improvement in reconstruction accuracy; It might be worthwhile for the authors to consider utilizing such data as annotations for 3D landmarks rather than 3D mesh data.

Thank you for the suggestion regarding the use of our dataset for improving the another task, 3D landmark reconstruction. As our dataset includes accurate pseudo 3D mesh data along with corresponding 3D landmarks, which indeed makes it a valuable resource for this purpose. Due to the limited time of the rebuttal, we aim to include the results of these experiments in the camera-ready version of our paper.

Q3-2. Have the authors explored the possibility of applying their proposed method to a different face model, such as the Basel Face Model (BFM), or investigating alternative pre-trained models instead of DECA or EMOCA?

We think our losses and optimization are applicable to other 3DMMs and neural models. We chose FLAME since it is the most recent and popular 3D linear face model.

For the alternative pre-trained models, we can use our method and the same benefits would hold as long as the model predicts low dimensional parameters and the neural network is overparameterized (from Proposition 1 in Sec. 3.3.). Also, using a neural model with better accuracy would provide better initial points that can lead to the faster convergence of NeuFace optimization.

We thank the reviewer for the time and the thorough review, which helped us improve our paper. We address the concerns and the questions below and in the revision (please check pdf, highlighted in pink).

Please let us know if our answers satisfy the reviewer’s concerns. We would be happy to provide further discussions and clarifications.

Weaknesses

W1. Unfair Comparison with DECA/EMOCA, which operates on single-view data.

We confirm that the experiments were fair.

Our NeuFace optimization is a test-time optimization method. The test-time optimization is a method that updates the model to fit into the deployment environment at test time, without access to training data [C1]. Test-time optimization methods utilize the losses with self-supervision or pseudo supervision, available at test-time. Prior arts [C1,C2,C3,C4] also compared their proposed test-time optimization methods and test-time losses with the baseline methods that do not involve test-time optimization. It is regarded as fair and valid comparison.

When evaluating the test-time optimization methods, the configuration of the pre-trained backbone network is important. Our NeuFace optimization starts from the pre-trained DECA/EMOCA checkpoint same as the compared baselines, which makes our settings and comparison fair.

Also, we would have tried if there was a fair setting that the reviewer specifically thinks. Please leave the comment if there is a specific fair setting to compare with DECA/EMOCA. We’d like to further discuss and willing to include it.

[C1] Yi et al., Temporal Coherent Test-Time Optimization for Robust Video Classification. In ICLR 2023.

[C2] Zeng et al., Test-Time Optimization for Video Depth Estimation Using Pseudo Reference Depth. In Computer Graphics Forum 2023.

[C3] Luo et al., Consistent Video Depth Estimation. In SIGGRAPH 2020.

[C4] Schneider et al., Improving robustness against common corruptions by covariate shift adaptation. In NeurIPS 2020.

W2. Novelty; The proposed temporal-consistency-loss and multi-view-consistency-loss seem more like separate regularizations (or averages) to achieve reduced jitter in the reconstructed videos.

We haven’t claimed the novelty of our losses. While our loss configurations are simple, we obtain a sufficiently good quality dataset, thanks to our neural re-parameterized optimization and its favorable properties. We can readily add more advanced losses for our proposed optimization.

Although our loss configuration is sufficient, we share the experimental result that reviewer B8S8 asked: NeuFace optimization + photometric loss ($\mathcal{L}_{photo}$).

Below we report the metrics for the MEAD dataset we used in the main paper. We compare the optimization that uses three different loss configurations as follows:

We obtain negligible effects of using $\mathcal{L}_{photo}$. The results show our minimal loss combination is sufficient and essential to reconstruct dynamic 3D faces. We have added the results and discussion in the revision (Sec. A.4, p. 21 in the Appendix).

Dear reviewer C76P,

Thank you for reviewing our work to enhance the quality of the paper. Please check our rebuttal. We think our rebuttal addressed all of the reviewer's comments. If there is anything more we can do to improve the paper, please leave a comment. We'd be happy to discuss further.

Best regards, Authors of 12

Thank you for your thoughtful and encouraging feedback. We think that all the comments by the reviewer have been addressed. If there is no other concern, we would like to respectfully ask the reviewer to re-assess our work and increase the rating.

We thank the reviewer for the interest in our work and the valuable feedback that strengthens our paper. We address the concern below and in the revision (please check pdf, highlighted in pink).

Weaknesses

W1. The theoretical novelty of face reconstruction using Neural network parameterization is incremental.

We haven’t claimed the theoretical novelty itself.

The theoretical context in Sec. A.2 helps the deep understanding of the analysis and algorithmic behavior of our proposed optimization system. We respectfully request the reviewer to consider our contributions below:

• Proposing the first and insightful concept of neural re-parameterized 3D face optimization, which mitigates the undesirable sparse gradient for face optimization (acknowledged by Reviewer B8S8).
• Providing a NeuFace-dataset, the first large-scale 3D face mesh pseudo-labels for existing large-scale 2D face video datasets. The dataset would benefit future research in this field (acknowledged by all the other reviewers).
• Extensive experiments to compare the quality and reliability of the proposed optimization and the dataset, and some empirical analysis.

Questions

Q1. What are the typical failure cases of the proposed method?

The failure cases could occur when the 2D video itself contains extreme degradations, e.g., motion blur, low resolution, extremely (> 50%) occluded, so that the 2D keypoint detection fails. Please note that, when we construct the NeuFace-dataset, we tackle these cases with automatic filtering followed by human verification, discussed in Appendix Sec. B of the initial submssion, which guarantees the reliability of the dataset.

Q2. In the EM-like optimization, does the reconstruction always go to the right optional result?

The NeuFace optimization and its losses are designed in a self-improving manner. Although the initial estimate of DECA could be noisy, the strong measurement of detected 2D landmarks and robust target supervision at each iteration gradually corrects the initial noisy predictions. The theoretical analysis shows that our neural re-parameterized optimization is highly likely to converge to global optima (Appendix Sec. A.2 of the initial submission). This hints that, with our neural parameterization, the optimization is robust to noisy initials and guarantees to exhibit at least stabler optimization behaviors than the compared baseline methods.

Dear reviewer GPao,

We would like to clarify the reviewer's last comment:

"However, the authors mentioned they specifically did QA to review challenging examples in the data set which is a bit concerning."

If the reviewer meant our 'human verification process' for the word QA, we'd like to correct and clarify that:

• We didn't involve the human verification process at all, when assessing our dataset's quality in experiments (Secs. 3.4, 4, and A.3, Fig 4, Table 2, and Table S1).
• We conduct the human verification process only for making the release version of our dataset.

We hope this addresses your concern about QA. If there is no other concern, we would like to respectfully ask the reviewer to re-assess our work and consider increasing the rating.

Dear Reviewer GPao,

We wish to highlight that Reviewer B8S8 has acknowledged the value of our work and updated the rating to a weak accept. We have thoroughly addressed all the comments raised by Reviewer GPao, thus we kindly request Reviewer GPao to review our revision and the responses of the rebuttal once more. We hope this might prompt a reconsideration and adjustment of your rating.

Best regards, Authors of 12.

Q3. Would an alternative robust operation, e.g., median, improve the results?

Following the reviewer's suggestion, we have compared the mean, median, and weighted average (ours) in the multi-view loss. We have compared the results on two scenarios: 1) Standard cases (use original images) and 2) Extreme cases (use perturbated images). For extreme cases, we randomly apply perturbation with large black boxes to the facial areas for 2~3 views in the multi-view videos to mimic significant corruption scenarios.

• Results of standard cases: All three methods (mean, median, and weighted average) showed similar performance, with no marked difference in optimization results evaluated on the MEAD subset.
• Results of extreme cases: The median outperforms the mean in perturbed MEAD data.
• Our Method: Our methodology, which employs a weighted average grounded in the confidence scores from multiple view vertices, not only shows favorable performance over the simple average but also aligns closely with the median's results. This indicates that our approach maintains robustness in extreme cases, akin to the median operation highlighted by the reviewer.

The reviewer's suggestion about the median was insightful. Both our weighted average and the median work for extreme scenarios. One difference is that our visibility-based weighted average has a control parameter to adjust the robustness through hyperparameters (as in Q2-2.).

Q4. Ethics Concerns: Some of these datasets were automatically gathered from the internet. So, it is unclear whether the new dataset is legally compliant.

We will not release the video dataset itself, but will release only the optimized 3DMM parameters obtained by our method without the video frames that might have been gathered from the internet.

The optimized 3DMM parameter does not contain identity-specific metadata or facial texture maps. Also, we will release the code that can generate pseudo ground-truth datasets like the NeuFace dataset for generic applications.

We thank the reviewer for the time and the thorough review. By addressing the reviewer’s questions and comments, we could strengthen our paper. We address the concerns and the questions below and in the revision (please check pdf, highlighted in pink).

We’d like to ask the reviewer to re-assess the value of our work with the following clarification.

Please let us know if our answers address the reviewer’s concerns. We would be happy to provide further discussions and clarifications.

Weaknesses

W1. Claims to investigate facial details. The approach is based on 3DMM with limitations of a linear model to represent fine facial details.

We would like to clarify the scope of our approach. This is a misleading point due to the term we used, which we have revised and tone-downed the terms across the paper in this revision.

Our work aims to facial details of facial gestures and motions, not mesoscopic facial geometry of facial skin, which is the point the misleading happens at. Thus, we do not reconstruct the displacement-level facial details which is not the target of our work. We focus on 3D facial geometries complying with input facial gestures and motions.

We revised the overall paper and tone-down our claim of “reconstructing image-aligned facial details” into “reconstructing facial geometries, well complying with input facial gestures and motions”, highlighted in pink. Thanks for the comment for specifying our scope better.

W2. Concerns on fair comparison and missing descriptions of experiments (Sec. 3.4, Fig. 4).

We respectfully note that this concern misled by the reviewer's misunderstanding of the premise of our work. We hope the explanation below clearly addresses the points leading to the reviewer's misconcern.

First of all, the evaluation of our NeuFace test-time optimization is indeed fair, which we carefully designed, as all the other test-time approaches are. Specifically,

1. Our NeuFace optimization, as elaborated in Sec. 3.2, is a TEST-TIME approach and does not involve training with train/validation/test splits. Our NeuFace step itself only exploits the input samples at test-time, but no additional data is used for the NeuFace itself.
2. We use the the pre-trained base model in NeuFace, e.g., DECA, for test-time fine-tuning on the input samples at test-time. Thus, the comparison with the pre-trained base model and our NeuFace (base model + NeuFace test-time optimization) is indeed in the fair setting; they have seen the exactly same data).
3. Our NeuFace is evaluated on each sample sequence independently. In Sec.3.4 and Fig. 4, for NeuFace optimization, we do not split the VOCASET into train/validation/test splits. We performed the test-time optimization for each sequence of VOCASET, starting from the pre-trained DECA checkpoint (pre-trained on VGGFace2 [C1], BUPT-Balancedface [C2], VoxCeleb2 [C3]), and detected 2D landmarks (we didn't use any GT information including mesh or 2D landmarks from VOCASET). Note that 2D landmark detections are not the ground-truth, as well. Thus, we claim that both DECA and ours do not have the chance to see any test dataset.
4. Likewise, our method is evaluated and compared for each test samples in each dataset (including VOCASET, MEAD, Voxceleb, and Celebv-HQ) independently without using their ground-truth mesh.

Our goal of experiments in Sec 3.4 and Fig. 4 was to compare the quality of pseudo 3D mesh annotations obtained from existing methods. In other words, the experiment in Sec 3.4 and Fig. 4 did not aim to compare the performance of each trained model on test datasets.

In the 3D face community, recent works [C4,C5] still naively use a pre-trained DECA or FLAME fitting (baseline) for annotator. In our experiments, we wanted to show how accurate and reliable NeuFace optimization is in generating pseudo 3D face annotations compared to existing methods.

To conclude, NeuFace is not a learned model but a test-time optimization method; thus, it does not require training/validation/test sets. Therefore, we confirm that our experiments were correct and fair, which we carefully designed.

[C1] Cao et al., VGGFace2: A dataset for recognising faces across pose and age. In International Conference on Automatic Face & Gesture Recognition (FG) 2018.

[C2] Wang et al., Racial Faces in the Wild: Reducing Racial Bias by Information Maximization Adaptation Network. In ICCV 2019.

[C3] Chung et al., VoxCeleb2: Deep Speaker Recognition. In INTERSPEECH 2018.

[C4] Ng et al., Learning to listen: Modeling non-deterministic dyadic facial motion. In CVPR 2022.

[C5] Paraperas et al., Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos. In CVPR 2022.

W3. Unfair comparison in Table 2, Sec A.3., & Table S1.

Our goal in Table 2 was to compare the quality of our pseudo 3D mesh annotations (NeuFace-D-dataset, NeuFace-E-dataset) with those obtained from existing methods (Base-dataset, DECA-dataset, EMOCA-dataset).

We emphasize that both DECA and ours do not have the chance to see any test dataset, given the fact that 1) for Table 2, we do NOT train models using MEAD, VoxCeleb2, and CelebV-HQ at all, and 2) 2D landmark detections are not the ground-truth but estimated from the off-the-shelf module.

In Sec. A.3 and Table S1, we have compared the dataset quality obtained with our different TEST-TIME loss configurations. When we use only $\mathcal{L}_\text{2D}$ to refine the DECA estimations, the NME indeed improves than DECA’s initial prediction. However, as mentioned in [W2], NeuFace optimization is not a learned nor a fine-tuned model on MEAD, VoxCeleb2, and CelebV-HQ datasets. NeuFace optimization is a test-time optimization method for fitting 3D face for each video. Also, the 2D landmark detections itself is a pseudo ground-truth. Therefore, we conclude that our experiment for Table S1 was correct and fair as well.

Questions

Q1-1. It would be good if you extended the explanation by adding the dimension of each FLAME parameter.

Thanks for the thoughtful comment. We have added the detailed explanation in the revision (Sec. 3.1), highlighted in pink, as:

"We use FLAME, a renowned 3DMM, as a 3D face representation. 3D face mesh vertices $\mathbf{M}$ and facial landmarks $\mathbf{J}$ for $F$ frame videos can be acquired with the differentiable skinning: $\mathbf{M}, \mathbf{J}{=}`FLAME`(\boldsymbol{\mathbf{r}, \boldsymbol{\theta}, \boldsymbol{\beta}, \boldsymbol{\psi}})$, where $\mathbf{r}\in{\mathbb{R}^{3}}$, $\boldsymbol{\theta}\in{\mathbb R}^{12}$, $\boldsymbol{\beta}\in{\mathbb R}^{100}$ and $\boldsymbol{\psi}\in{\mathbb R}^{50}$ denote the head orientation, face poses, face shape and expression coefficients, respectively."

Q1-2. Considering the displacement map in addition to the FLAME parameter?

We think optimizing the mesoscopic detailed face geometry by re-parameterizing the displacement map is definitely an interesting future direction.

As discussed in [W1], the face skin detail reconstruction is not our scope of this work, but “reconstructing facial geometries, well complying with input facial gestures and motions.” More importantly, we are motivated by the fact that there are lack of existing large-scale in-the-wild or multi-view video datasets that contain high-level face geometry, head motion, identity, and expressions that comply with input videos, in the community.

We believe our NeuFace dataset, which contains large-scale, diverse, natural, high-level human 3D face motion, would invigorate the 3D face community.

Q2-1. Sec. 3.2: how were the ground-truth landmarks for equation 2 obtained?

We used the off-the-shelf 2D landmark detection model, FAN [C6]. We also performed manual human verifications to reject the failure cases when constructing the NeuFace-dataset.

We have added this to the revision (Sec 3.2, second paragraph, page 4), highlighted in pink. Thanks for checking the details.

[C6] Face Recognition (FAN), https://github.com/1adrianb/face-alignment

Q2-2. In the Multi-view consistency loss, where do the confidence values come from?

We assign the confidence score per vertex by measuring the angle between the vertex normal and the camera ray. We set the vertices as invisible if the angle is larger than the threshold $\tau_a$, and the vertex has a deeper depth than $\tau_{z}$ (i.e., $z<\tau_z$). We empirically choose $\tau_a = 72^\circ$, $\tau_z=-0.08$. We have revised the paper accordingly (p5, Sec 3.2), highlighted in pink.

Dear reviewer aXFo,

Thank you for reviewing our work. We sincerely ask reviewer axFo to check our rebuttal.
We believe that we have thoroughly addressed all the comments and suggestions reviewer axFo had raised. Considering the remaining time, we welcome any further comments or feedback that may help to enhance our work. We'd be happy to further discuss.

Best regards, Authors of 12

Thank you for your thoughtful and encouraging feedback. We think that all the comments by the reviewer have been addressed, and the reviewer also confirmed the reviewer's misled concerns being corrected. If there is no other concern, we would like to respectfully ask the reviewer to re-assess our work and increase the rating.

Dear reviewer axFo,

We kindly note that the reviewer's comments were at the top (not linked to this thread), and we already answered above.

1) Answer about the experiments

[22 Nov 2023, 16:31] Reply link: https://openreview.net/forum?id=E6EbeJR20o&noteId=bsAfVVwlqg

2) Answer about temporal consistency loss

Also, please check our response on temporal consistency loss at above (the reviewer's comment), as well.

[22 Nov 2023, 17:53] Reply link: https://openreview.net/forum?id=E6EbeJR20o&noteId=n5KZS1btNz

We respectfully request the reviewer axFo to have a look at our responses, re-assess our work, and increase the rating, if the reviewer's concerns are addressed.

Dear Reviewer axFo,

We believe that all the comments by the reviewer have been addressed. If there is no other concern, we respectfully request Reviewer axFo to review our revision and the responses of the rebuttal once more. We hope this might prompt a reconsideration and adjustment of Reviewer axFo's rating.

Best regards,

Authors of 12.