HairFree: Compositional 2D Head Prior for Text-Driven 360° Bald Texture Synthesis

Thank you for providing constructive feedback on how to improve the presentation and writing of the paper.

We are grateful to hear that the compositional 2D head prior will be useful for the community (yes, we will release the model weights, see line 19) and that the experiments are solid, showing that the proposed method obtains the best results.

In the following, we will provide the additional requested details which we will also add to the revised paper.

(1) Implementation / Usage of the inpainting network:

• Does this inpainting need training? What is the training data?

  • The inpainting LDM prior is functionally equivalent to the locked (frozen) LDM prior, with the key distinction that the inpainting model was additionally conditioned on masks during training. In our setup, we do not retrain the inpainting LDM; rather, we use a pre-trained version and replace the original LDM prior with it. At inference time, we attach ControlNet to the inpainting LDM instead of the original LDM, allowing us to perform inpainting using both mask-based conditioning and ControlNet guidance. For further details on the training of prior LDMs, we refer the reader to Rombach et al. [51] (Stable Diffusion).

• What's the role of the inpainting network in the 3D texturing pipeline?

  • At inference time, we use the inpainting LDM prior in combination with our ControlNet to generate head images conditioned on view-dependent 3D head mesh and skin/ear appearance cues. As the camera gradually rotates around the head, previously unseen head regions become visible and are inpainted by the model. The visible regions that have already been synthesized are stored in UV space and later re-rendered and reused for inpainting as the camera continues to move, ensuring spatial and temporal consistency across views. (See Figure 2 (system figure) and its description, and Sec. 4.2).

• What is the baseline called "the model without inpainting"?

  • The "model without inpainting" generates all head views at the same time, without accumulating previously seen regions in UV space or inpainting newly visible areas. In contrast, our progressive texturing approach is iterative: views are generated one after another and depend on previously generated content. As the camera rotates, visible regions are stored in UV space, and newly revealed areas are inpainted based on both the current view and prior results. This approach better handles extreme angles outside the fine-tuning distribution.

(2) It seems that there is a gap between the training and texturing phase of the 2D prior. In the training phase, there is always a hair mask, but in the texturing phase, the hair mask is set to zero. What about the effects of this gap?

Yes, the mask distribution in the training and texturing phase is different. During training, ControlNet learns hair appearance conditioned on the hair mask. At test time, by setting the hair mask to zero, reducing the ControlNet strength to rely more on the inpainting prior, and specifying via text that the head should be bald, the model efficiently removes hair. Combined with our progressive texturing approach, this enables plausible bald head generation mitigating artifacts coming from the training/inference conditioning gap.

(3) The training dataset, i.e. FFHQ and CelebA-HQ, does not contain the top view of bald people. How does the method synthesize the top view of the bald head? In addition, how does the method synthesize the texture of the back head?

Since the training data lacks top and back views of bald heads, we reduce the ControlNet strength to enable the generation of these views. This allows the model to rely more on the inpainting LDM prior, which was trained on larger, more generic datasets, making the fine-tuned dataset less critical. Combined with our progressive texturing approach and a text prompt specifying a bald head, this helps produce plausible textures for extreme angles (See lines 237-238 and Figure 6(C) for Controlnet strength; lines 140-143 for the inpainting LDM; lines 168-189 and Figures 6(A) and Figure 8 for progressive texturing and related ablations).

(4) Dataset contribution:

We would like to clarify that the dataset itself is not the primary contribution of our work. Rather, our key contribution lies in the methodology that enables the generation of high-quality, 360-degree bald head textures. This method can serve as a powerful tool for data generation in downstream tasks. Photorealistic examples from the resulting dataset are presented throughout the paper and in the Appendix. Importantly, our dataset is generated entirely automatically using our method and can be scaled or reproduced using only text prompts and 3D head meshes. This makes it fundamentally different from datasets such as FFHQ-UV, which rely on GAN-based models, produce lower-quality textures, and include baked-in hair in the generated outputs. Additionally, FFHQ-UV requires a reference image and does not allow control over the underlying head geometry.

Thank you again for your time and constructive feedback. We hope our responses have clarified the key points, and we would be glad to provide further details if needed.

Thank you for your valuable feedback and highlighting the strengths of our work!

In the following, we give additional insights and clarify questions raised:

(1) Our contributions:

As mentioned, our approach is able to generate diverse textures without any major seams. This is achieved through our proposed compositional 2D prior and a progressive texture merging module that ensures consistency across views. While both components build on prior ideas, they are combined and adapted in a novel way to enable fully automatic, 3D-consistent 360° head texturing for both human and non-human skin—without requiring reference images, multi-view data, or bald human head data.

• The ControlNet-based 2D head prior is trained to generate semantically aligned RGB views, conditioned on 3D geometry and face parsing. As noted by Reviewer ZFQT, this component acts as a strong foundation model for face generation, supporting a wide range of skin types and enabling robust hair-skin separation for bald texture synthesis.

• The progressive texture merging module incrementally completes the UV texture by re-rendering and infilling texels from new viewpoints, using previously generated texels as a reference.

These design choices are key to the robustness and generality of our method across identities, viewpoints, and texture styles. Moreover, quantitative and qualitative evaluations show that our method outperforms all recent state-of-the-art techniques in terms of texture quality, consistency, and realism.

(2) What do Person 1 and Person 2 refer to in the user study? How were these examples chosen? Are these two samples representative of the output distribution?

Person 1 and Person 2 refer to the individuals shown in Figure 3. We will clarify this in the paper. These examples represent photorealistic results and were selected as a random male and female celebrity. For additional examples of this type, please refer to Figure 4. For a broader set of celebrity texture maps, see Appendix Figure 23.

(3) Why does the proposed texture aggregation handle seams better than existing techniques?

• The related methods mentioned are based on fixed inputs, i.e., multiple captured images that might have variations in shading or even facial expressions. In contrast, HairFree is generating its views with a technique that ensures consistency in image space which results in consistent and seam-less uv textures. Specifically, our proposed inpainting technique ensures to use the already existing colors from the texture and generates new color values for regions that are becoming visible when rotating the 3D head in image space. In the paper, we are visualizing the views that are iteratively generated with a major order of front, sides, back. This order also contributes to the consistent merging of the left and the right sides.

• Note that we have conducted qualitative ablation studies on inpainting in UV space (inpainting in UV space leads to the seams/discontinuities at the back/top of the head, see Figure 8) and inpainting in RGB space (without our progressive re-rendering approach, it is not possible to generate 3D head-consistent extreme side views and back view; notably, Janus effect will appear at the back, see Figure 6(A)).

• We will add additional zoom-ins to the figures that show the head from the back. Note that larger examples, additional texture maps, and more extreme viewpoints are provided in the Appendix. We recommend zooming in on the back view samples for a clearer inspection.

(4) Influence of guidance scale:

Ho and Salimans do not measure the influence of guidance scale when the large diffusion prior is fine-tuned on a smaller dataset. In our case, we fine-tune our prior on photorealistic frontal views of faces only, and at inference time, we show that we can generate back-views, bald samples, and non-photorealistic examples that are out of the fine-tuning distribution, emphasizing strong generalization properties of our approach.

Thank you again for your time and constructive feedback. We hope our responses have clarified the key points, and we would be glad to provide further details if needed.

Thank you for the constructive feedback which we are happy to incorporate in the revised paper. Please find detailed responses and clarifications in the following paragraphs.

(1.1) Explanation of the necessity and unique role of the conditioning signals:

The overall goal of the proposed conditioning strategy is to expose the model to both desired and undesired features during training. This enables the preservation of desired elements and the removal of unwanted ones at inference time. We will add a detailed description of the conditionings to the paper and will show the effect of removing each condition signal in an ablation study (thank you for the suggestion).

- Skin and face mask: During training, the diffusion model is conditioned to learn the appearance of skin and facial regions where specified. At inference time, high-precision 3D skin and face masks are obtained using the FLAME model.

- Hair mask: The model is trained to recognize and predict hair regions when provided. However, no hair mask is applied during inference. When combined with text prompts indicating a bald head, this approach enables the removal of hair and the generation of clean skin. Without this, hair would be baked into the skin texture.

- Ear mask: Ear masks are used both during training and inference. During training, masks are generated using an off-the-shelf face parsing model. At inference, accurate 3D ear masks are obtained from FLAME. This ensures consistency between generated and FLAME ears, improving texture fidelity.

- Accessory mask: Similar to hair masks, accessory masks (e.g., eyeglasses, earrings, clothing) are used during training to guide the model’s learning. These are omitted during inference, allowing for the removal of such accessories and the synthesis of unobstructed skin regions.

- Rendered 3D head mesh: Conditioning on a rendered 3D head mesh guides the model to generate images consistent with the target geometry and ensures accurate pixel-to-UV space mapping. Training meshes are predicted using an off-the-shelf model, while inference-time meshes can be explicitly defined for texturing. Appendix Figures 21 and 22 (the “LDM Inpaint” row) demonstrate the impact of removing the 3D head mesh conditioning, which leads to reduced geometric consistency and poorer UV mapping.

- RGB background: Without conditioning on the background, generated images often include undesired elements (e.g., hands, microphones, miscellaneous artifacts) or unnatural shadows. At inference time, setting the background to a uniform color helps prevent such artifacts.

(1.2) Explanation of the necessity and unique role of the different modules:

The presented texture generation approach consists mainly of two modules:

(i) a compositional, 3D-controllable prior which can be used for inpainting in image space, and

(ii) a progressive texture merging/fusion module.

Both components are key contributions of our method to produce bald human textures that can be combined with other 3D assets. The progressive texture generation (ii) iteratively generates a complete texture, by leveraging the prior (i) using the conditionings explained above. The conditioning allows for a good alignment between the 3D model and the generated images which is important to project the colors to the texture space of the FLAME model accurately. The iterative inpainting in image space guarantees that the resulting texture will be seam-less (as the generated images will not have seams) - this is in contrast to methods that do inpainting in texture space. Additionally, Figure 8 shows why inpainting in image space using (i) is important, while Figure 6(A) shows why (ii) is important.

(2) Generalization of the method:

The goal of our method is a text-based generation of bald human head textures for a given 3D FLAME mesh. Based on this mesh and the generated texture, additional assets can be added by the user. For generating the texture, we condition our approach on the given 3D FLAME mesh with a neutral expression. In the paper, we show that our method is able to generate textures with a large variation of different attributes and that it also generalizes to the non-photorealistic setting. In the photorealistic setting, Figure 4 presents generated results across a range of visual attributes, including gender, age, and ethnicity. Additional examples highlighting diversity in skin tone and nationality are provided in the Appendix (Figures 9 and 10). Figure 4 also includes several celebrity cases, while Appendix Figure 23 offers an extended collection of 360-degree texture maps for various celebrities. In the non-photorealistic setting, representative results are shown in Figure 5, with further diverse examples included in the Appendix (Figures 11-20).

(3) Usage of the proposed method for reconstruction:

The proposed method is a generative model that based on a text prompt generates a complete texture of a human head. However, we believe that some of our contributions can also benefit the reconstruction given a reference image. Specifically, our 2D diffusion prior is capable of removing hair from a reference image, enabling it to serve as a bald proxy as shown in Figures 21 and 22 in the Appendix. These generated images could be used as input to a face reconstruction method.

(4) Implementation and training details:

Our method is based on Stable Diffusion 2.1 LDM, which we fine-tune using ControlNet. At inference time, we use the corresponding inpainting prior from Stability AI, both of which are publicly available. The model is fine-tuned for 1500 GPU-hours on a single H100 GPU (see line 144). As mentioned in the submission, we will release the code, pre-trained model, and a dataset of generated samples.

Thank you again for your time and constructive feedback. We hope our responses have clarified the key points, and we would be glad to provide further details if needed.

We are thankful for the positive and constructive feedback! In the following, we will clarify the questions raised under ‘weaknesses’ and we will incorporate the suggestions and answers into the revised paper.

(1) Miscellaneous issues:

• In Figure 2(A), why is the label for the last row "RGB"? Please clarify.

  • R, G, B refers to the three usual RGB channels. We will rename it to Channel 1, Channel 2, and Channel 3 for clarity.

• In Figure 2(B), does $\beta$ denote ControlNet strength? Also, what does $\epsilon$ represent?

  • Yes, $\beta$ is ControlNet strength. $\epsilon$ is a small neural network through which the input condition downsamping is learned (Zhang et al. [68]).

• In Figures 21 and 22, the legend labels appear to be flipped.

  • Thank you for noticing this, we will put them back on top!

• In Figure 6(B), the figure lacks a legend indicating that the first row corresponds to higher CS values, and the second to lower ones.

  • Yes, we will add the legend for clarity, thank you!

(2) An ablation study on the number of viewpoints would strengthen the justification.

We empirically found that reducing the number of side-views may lead to the Janus artifact at the back of the head (similar to Figure 6(A)). We will add examples of this when fewer views are used to the revised version.

(3) One-to-many issues:

We use a splatting-based approach to set the texel values given a specific pixel and corresponding UV coordinate. Thus, two pixels could be mapped to the same UV texels. Here, a distance based blending scheme could be used, but we found that assigning one of the values without blending yields good results.

(4) What is "ControlNet Strength (CS)"?

ControlNet is a copy of the original LDM U-Net that processes the control (conditioning) image separately. Its outputs are then added to the original U-Net’s outputs at corresponding blocks during the denoising process. This allows the model to integrate external structural guidance (like pose, edges, or depth) without modifying the base U-Net. The degree to which this control input influences the final image can be adjusted using the ControlNet strength parameter. We will describe this in more detail in the revised paper. Specifically, let $F_i$ be the output from the original LDM U-Net at block $i$, $C_i$ be the corresponding ControlNet feature map at the same stage, and $\beta$ be the ControlNet strength parameter. Then, the combined feature at each block during the denoising process is computed as: $F_i^{\text{new}} = F_i + \beta \cdot C_i$

(5) In lines 204–205: "By reducing CS, we can generate more diverse examples, enabling the synthesis of novel head views that do not exist in the training data (Table 2)." However, Table 2 shows that all metrics degrade as CS decreases. Therefore, the claim about improved diversity lacks sufficient evidence.

Table 2 shows that all metrics degrade as CS decreases, considering the data distribution that the model was fine-tuned on. To remember, this data contains mostly frontal and some side-views of photo-realistic human faces with hair. In our case, we want to generate samples that are different from the original data distribution in three aspects (1) no hair (bald examples), (2) various camera views including top and back-views), (3) non-photorealistic (including non-human) examples. Therefore, the goal is for the metrics to decrease as ControlNet strength decreases which is what is shown in the table (because data distribution that we want to generate differs from the one that is present in data). In other words, metrics degrading as CS reduces (there is a trend, lower CS leads to worse metrics’ scores) signifies that the distribution of generated data shifts more and more from the original distribution, which is what we want.

(6) The content in Section 5.1 "Quantitative Evaluation" and Section 5.2 "Ablation - ControlNet Strength (CS)" is somewhat redundant. Moreover, the conclusion of Ablation - ControlNet Strength (CS) to be more relevant to Section 5.1.

In Section 5.1 we were referring to quantitative evaluation of the CS parameter, while in section 5.2 this was shown qualitatively. We will merge those two in Section 5.1 in the revised version.

(7) In Section 5.2 "Comparison with Inpainting Methods": What does Content-Aware Fill (CAF) refer to? Please provide more technical details or cite relevant literature.

CAF is an “industrial standard” for inpainting in professional tools like PhotoShop.

Thank you again for your time and constructive feedback. We hope our responses have clarified the key points, and we would be glad to provide further details if needed.

Thank you for the rebuttal; most of my concerns have been addressed. However, I still have some questions that I want to discuss with you.

Considering the response (3), does the distance-based blending scheme used follow the original strategy in 3D Gaussian splatting? Since there has been research showing that the 3D Gaussian splatting presents the view-dependent appearance issues that may introduce flicker artifacts when the view camera shifts around certain angles. Did this phenomenon occur in your experiment?

For response (5), it is reasonable that reducing CS leads to a degradation of quantitative results regarding the data distribution inconsistency. However, it cannot be said that this observation exactly follows what the author thought. Since inferior synthesized results can also make the metrics worse, could the authors provide more evidence (such as visualizations or more appropriate metrics) to illustrate this finding? Or just found out some examples where the bald texture of the back and side-head is available but not used for training, and calculate the metrics on this constructed evaluation set?