Learning to Decouple the Lights for 3D Face Texture Modeling

Q1: Line 119 says the n lighting coefficients are initialized differently. Can you provide some insights behind these design choices? Is the method sensitive to the initialization of lighting?

A1: Sure. The lighting coefficients are just simply initialized as a uniform distribution between -1 to 1 as illustrated in Line 119, Sec. 3.3 of the paper. Such operation can provide $n$ initialized local illuminations ranging from dark to light. During the later optimization of Stage 2, the $n$ groups of lighting coefficients are optimized towards actual illumination. Coefficients that deviate significantly from the existing illumination in the images will have masks with smaller areas, as shown in the visualized $M_N$ of Fig.2 of the main paper. These coefficients with smaller masks are deemed less important and are removed according to a predefined threshold $\epsilon$ in ACE.

Here, we also provide a ablation study for the number of $n$ used for initialization of lighting conditions below. Single images from VoxCeleb2 are used for evaluation.

We observe that $n=3$ produces sub-optimal results, likely because the number of lighting condition candidates is insufficient to model images with complex illuminations. In contrast, $n=3, 5, 9$ yield good and similar results as there are enough initial lighting conditions, and any redundant ones are removed. The result for $n=5$ is slightly better. While introducing larger $n$ and further adjusting the hyper-parameters might improve performance, it would also increase the optimization burden. Therefore, we use $n=5$ in this work.

Q2: In the proposed Human Prior Constraint, the rendered face is encouraged to be similar to a specific face in the FaceNet. Why not use the common perceptual loss as D3DFR?

A2: Please note that the rendered faces used to calculate the Human Prior Constraint (HP) are $I_{Rs}$. As illustrated in Fig. 2 of the main paper, $I_{Rs}$ includes rendered faces under multiple predicted lighting conditions. We cannot apply perceptual loss to $I_{Rs}$ because we do not have corresponding ground truth images under the multiple decoupled lighting conditions. However, HP does not require such ground truths. Perceptual loss can indeed be applied between the final rendered result $I_{out}$ and input image $I_{in}$. Below, we present a comparison between the perceptual loss implementation and the HP we proposed. We can see that HP still has better performances, which can confirm it provides more effective constraints for the textures through rendered faces under multiple lighting conditions.

Q3: The texture of D3DFR in Figure 3 is missing. In addition, the FFHQ-UV's texture is presented in UV space while NextFace and the proposed methods' texture are presented in image space, I suggest presenting them all in UV space or all in image space.

A3: Thank you for your suggestion. We agree that the texture presentation space needs to be consistent. To achieve this, we transform all UV-textures to the image space. This allows textures from D3DFR and CPEM to be visualized as colored faces, even if they do not have UV-textures. Fig. 16 and 17 in the rebuttal PDF are provided in this space. We will also update the texture images in the main paper accordingly in the revised version.

Q1: The authors have not compared against optimizing in haar-wavelets[1] which are designed specifically to model sharp illumination dependent effects.

[1] Triple Product Wavelet Integrals for All-Frequency Relighting

A1: Thank you for your suggestion. Please note that our contribution lies not in proposing a universal neural illumination model for all scenes, but in decoupling lights to aid in recovering more accurate textures in 3D face reconstruction.

Reference [1] offers a solution for modeling environment illumination using haar wavelets for relighting. However, [1] focuses on efficient rendering, with all geometrical and texture information pre-processed and fixed. This differs significantly from the 3D face reconstruction setting, where geometrical, texture, and illumination parameters are co-optimized, making illumination optimization more challenging.

To our knowledge, haar-wavelet modeling of illumination has not yet been applied to 3D face reconstruction, and no open-source codes are available. To validate this idea, we create a baseline by replacing the environment map modeling in NextFace and NextFace* using Spherical Harmonics (SH) with haar wavelets.

The optimization of wavelets is considerably slower than SH, making it difficult to conduct quantitative comparisons on the full datasets. Therefore, we conducted a comparison on a subset of 5 pairs of single images from VoxCeleb2. The quantitative comparisons are presented below:

We observe that modeling global illumination with haar wavelets has limited improvements over the original SH on PSNR for our face reconstruction task. Although performance might be enhanced by modifying the wavelets or increasing the size of the environment map, such modifications are beyond the scope of this paper and would incur significant time costs. Optimizing such a 64 × 64 environment map from wavelets takes approximately 28 minutes on a 256 × 256 image, whereas our method takes about 340 seconds. Therefore, our method remains more efficient and effective at this time.

Q2: There are no results shown of the initial AlbedoMM texture the texture map is initialized from. AlbedoMM already gives a relatively uniform texture map that is free from lighting artifacts, it is unclear what additional benefits texture optimization yields.

A2: As demonstrated in Fig. 2 and Alg. 1 of the main paper, we use AlbedoMM to initialize the textures in Stage 2, while the textures are directly optimized in Stage 3. Therefore, the textures from Stage 2 are precisely the AlbedoMM textures.

In Section A.4 of the appendix, on page 12, we have provided a comparison between the AlbedoMM textures (Stage 2) and the final textures (Stage 3). As shown in Fig. 10 of the appendix, the texture optimization adds more details, such as beards, to the AlbedoMM textures, resulting in a more realistic reconstruction.

Q3: How many bands of SH are optimized? Did you investigate optimizing higher number of bands instead of generating the masks again?

A3: In this work, we follow NextFace [1] by using 9-band spherical harmonics (SH) to model local illumination. Additionally, we provide quantitative results of our method modeled with a single global SH using 9, 12, 15, and 18 bands, by removing the $f(\cdot)$ and $g(\cdot)$.

We observe that increasing the number of bands in a single global SH yields quite limited improvements. A possible reason is that the external occluded shadows on human faces represent drastic illumination changes in relatively small areas, which may not be appropriately modeled as a single global SH during optimization.

Q1: Quantitative evaluations for ablations.

A1: Thank you for your suggestion. Here, we provide the quantitative ablation study of our proposed components below, following the same setting as Sec. 4.4 of the main paper. GP, LP, and HP denote our proposed constraints $L_{GP}$, $L_{LP}$, and $L_{HP}$ mentioned in Sec. 3.4, where Light and Occlusion denote the $f(\cdot)$ for combining multiple lighting conditions, and $g(\cdot)$ to remove the direct external occlusion, as claimed in Sec. 3.3, respectively.

Here is the ablation study for constraints:

Here is the ablation study for the proposed $f(\cdot)$ and $g(\cdot)$.

We can see that each component has contribution to the final performances.

Q2: Missing citations and Discussions about relighting works [1, 2, 3].

[1]SfSNet : Learning Shape, Reflectance and Illuminance of Faces in the Wild (CVPR 2018)

[2]Face Relighting with Geometrically Consistent Shadows (CVPR 2022)

[3]Learning Physics-guided Face Relighting under Directional Light (CVPR 2020)

A2: [1] proposes a framework for decomposing face attributes from in-the-wild images through physics-based rendering, while [3] predicts residuals to enhance diffuse relighting performance. [2] introduces ray-tracing to model geometrically consistent self-occlusion shadows for relighting. However, these methods primarily focus on self-occlusion shadows rather than external shadows. Due to character limitations, we will discuss these approaches in more detail in the revised version.

Q3: Evaluations on images with more diverse foreign (external) shadows.

[4] Portrait Shadow Manipulation (SIGGRAPH 2020)

A3: Thank you for your suggestion, we introduce 15 pairs of data (30 images) from [4] for evaluation under more diverse external shadows. The quantitative results are presented below, where some qualitative results are presented in Fig. 16 of the rebuttal PDF. We can see that our method still outperforms other methods under faces with diverse shadows.

Q4: Comparing with the simple baseline of running a foreign shadow removal method such as [4, 5, 6] on the images beforehand.

[5] Unsupervised Portrait Shadow Removal via Generative Priors (MM 2021)

[6] Blind Removal of Facial Foreign Shadows (BMVC 2022)

A4: It is an interesting idea. To validate its effectiveness, we introduce the shadow removal work [5] to pre-process the face images with shadows because it is the most recent shadow-removal work with a pre-trained model. Here are the quantitative results evaluated on data from [4].

Here are quantitative the results evaluated on single images from VoxCeleb2.

We also provide qualitative comparisons in Fig. 17 of the rebuttal PDF. The pre-processing with the shadow removal method indeed improves the performances of baselines, while our method still outperforms them.

From the qualitative results, we observe that the shadow removal model cannot fully remove the external shadows in cases where the external shadows cover relatively large regions. Although modifying the shadow removal model to be more powerful may further improve the final performances, it goes beyond the range of this work. We can explore it in the future.

Q5: More analysis on failure cases. For example, does the method fail when the external shadow covers most of the face?

A5: As shown in Fig. 16 of the rebuttal PDF, our method performs well even under external shadows covering most of the face. We present visualizations of our inferior cases in Fig. 19 of the rebuttal PDF. As mentioned in the Limitations section of the main paper, our primary limitation is the initialization with AlbedoMM. For faces with many high-frequency details, such as wrinkles, our method may lose these details during reconstruction. Replacing AlbedoMM with more powerful face representations could address this issue. We plan to explore this further in future work.

Q6: There is no section on potential negative societal impact.

A6: Sorry for the lack of such discussion. Similar as existing human face reconstruction works, our method, focusing on recovering more realistic face textures under shadows from self and external occlusions, may also lead to the ethical concerns about privacy, consent, and the spread of misinformation. We will discuss about it carefully in the revised version.

Q1: Lack of discussion and comparisons with 3D face reconstruction works dealing with de-occlusion, such as [1, 2].

[1] Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation (CVPR'23)

[2] Occlusion-aware 3D Morphable Models and an Illumination Prior for Face Image Analysis (IJCV'18)

A1: Thank for your reminding. References [1] and [2] propose different methods to predict masks to remove external occlusions, where they optimize 3D Morphable Model(3DMM) according to the remained regions to reconstruct de-occluded faces. We will discuss these methods in more detail in the revised version.

However, de-occlusion focuses on removing external occlusions, while the shadows are not just occlusions. A shadow region is actually the face under another illumination, which may include useful texture information. Simply treating shadowed regions as occlusions and removing them using de-occlusion methods may lead to the loss of face texture information in these areas. Our method, which integrates multiple lighting conditions, is more suitable for recovering textures from faces with shadows.

Here, we present a comparison with the recent open-sourced de-occlusion method [1] by extracting its 3D textures and evaluating them using the texture quality metrics as mentioned in Section 4.1 of the main paper. The quantitative results, evaluated on single images from VoxCeleb2, are presented below.

We also conduct comparisons on Video sequences from VoxCeleb2:

Our method demonstrates superior performances. We also provide qualitative comparisons in Fig. 18 of the rebuttal PDF. Our method constructs more realistic results. For example, results from [1] miss details such as beards, while our method preserves them.

Q2: Can we use the de-occluded results of generative de-occlusion model such as [3] on NextFace for high quality reconstruction?

[3] Face De-occlusion using 3D Morphable Model and Generative Adversarial Network (ICCV'19)

A2: Reference [3] trains a generative model to produce de-occluded face images from input and initial 3DMM synthesis images. We will discuss this in more detail in the revised version.

However, [3] does not provide its source code, making a direct comparison unavailable at this time. Additionally, as stated in Sec. 5.1 of [3], the model is trained on synthetic data constructed by layering common non-face objects onto faces. This approach focuses on learning how to remove these objects, without any consideration about shadows.

As we mainly discuss about the influence of shadows from self and external occlusions in this work, a closer idea is using 2D shadow removal methods mentioned by Q3-Reviewer SyuP. Such works desire to introduce generative model to recover high quality images free from shadows.

We conducted experiments using the output of the shadow removal method [4] as input to CPEM, D3DFR, NextFace, NextFace*, and FFHQ-UV to evaluate their ability to acquire accurate textures without shadows. The quantitative comparisons are presented in Q3-Reviewer SyuP, and we also provide qualitative comparisons in Fig. 17 of the rebuttal PDF.

Our results show that introducing the output of such generative models to the texture modeling baselines still yields inferior results compared to our method.

[4] Unsupervised Portrait Shadow Removal via Generative Priors (MM 2021)

Q3: The technical details of the paper were not easy to read. I did not understand how many number of masks are used, what 't' refers to in L128, how many such 't' are used. How are all the hyper-parameters, such as 'n', n_L', m, epsilon, etc., chosen?

A3: Sorry about the confusing parts. The number of masks are actually changing according to the illumination complexity of the image. As illustrated in Line 117~124 of the paper, we initialize $n$ separate lighting conditions, where $f(\cdot)$ predicts one mask for each Lighting condition. The $n$ masks make up $M_N$ in Fig.2. Then, the parameters of $n$ lighting conditions and $f(\cdot)$ are optimized to construct the face image, where some lighting conditions not existing in the face image will get smaller mask areas as illustrated in Fig.2 of the paper.

Then, in ACE, we drop such lighting conditions and masks with small mask areas, as explained in Line 142~147 of the paper. In this way, $n_L$ masks in $M_N$ are remained, which make up $M_L$. We can see that $n_L$ is a adaptive number less than $n$.

We select $n=5$ in this work. It means there would be 5 masks in $M_N$ at the beginning. Where $n_L<=n$ would be decided according to the subsequent optimization. Corresponding discussions could be found in Q1-Reviewer NbDK.

$m$ mentioned in Line 142 to 157 is not a hyper-parameter, it represents the mask value between 0 ~ 1 at a pixel position, e.g., $m=0$ at the black regions of the masks. $\epsilon$ is a threshold to filter out the masks with quite small areas. We set it as 0.17 in this work.

$t$ is decided by the number of frames in the input image/video sequence. For the single image reconstruction, $t$ is set as a fixed 0 value, where it would be $i/k$ for the $i_{th}$ frame of a $k$ frames video sequences. In this work, we mainly conduct comparisons on 8-frame sequences as claimed in Line 203, page 7 of the paper. We will add more details about the mentioned contents in the revised version.

Thank you for your response! We apologize for the confusion regarding the qualitative results. We agree that the work "[1] Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation [CVPR23]" can avoid shadow effects on the textures. But please note that it is different from our method and limited by texture details. [1] predicts a binary mask to remove these shadow regions, while our method reconstructs them with different illuminations.

The details, e.g. beards, may be difficult to be captured by [1] because it fully relies on a pure linear parametric 3D Morphable Model (3DMM) to model the face textures and distinguish the occlusion regions. As claimed in Sec.3.1 of [1], it models face textures with 3DMM parameters predicted by networks. Although 3DMM does not have explicit shadows on its textures as shown in the rebuttal PDF, it is hard to construct details, e.g. mentioned beards, as claimed in “Parametric Face Models”, Sec. 2 of [2]. Therefore, even lowering the weight of the regularizer in its Eq.8 may not be able to capture the details due to the limitation of the 3DMM model itself. It is also mentioned in the Limitation, Sec. 4.5 of [1] that a face model capable of depicting facial details, makeup, and beard is required. While incorporating such a model could potentially address this issue, doing so requires substantial effort and falls outside the scope of our paper, which we leave for future work.

[2] Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz

In our method, although we also initialize face textures using a parametric AlbedoMM in Stage 2, our direct texture optimization in Stage 3 allows us to add fine details to the textures, as shown in Figure 10 of the appendix in our paper. This can be achieved because we model the shadows with illumination to avoid their influence on textures. Directly optimizing the textures in [1] may not achieve this. As explained in Sec. 3.2 of [1], it treats regions which cannot be well described with 3DMM under a global illumination, e.g. the beard regions, as occlusions and learns to predict a mask to remove them. Since these regions are removed, they cannot be further used to refine the 3DMM textures through optimization.

Since we are not allowed to include more qualitative results in this comment, we will provide additional visualizations of the predicted masks from [1] in the revised version to support our claim. Furthermore, the quantitative comparisons in Q1 also confirm the effectiveness of our method, which will be discussed in greater detail in the revised version.

Thank you once again for your feedback. Please feel free to reach out if you have any further questions.