HairDiffusion: Vivid Multi-Colored Hair Editing via Latent Diffusion

Thank you for your suggestions. We have included additional failure cases in Figure 1 of the PDF, where we comprehensively demonstrate the effects of multi-view hair color transfer and discuss the reasons for poor performance.

In Figure 1 of the PDF, we have added discussions on the limitations of the warping module in extreme cases of hair color transfer, including significant pose differences, complex textures, and large discrepancies in hairstyle regions. To visually illustrate the limitations of the warping module method, the warped results in the figure have not undergone the post-processing mentioned in the paper. The first and second rows demonstrate cases with significant differences in hair length. While the hairline region aligns well, the hair ends do not; however。the second row on the left, with similar hair lengths, performs well, which is related to not randomly removing hair end information when generating the paired hair dataset. The third, fourth, and fifth rows show cases with significant pose differences. The warped result on the right side of the second row shows that the hair orientation aligns well with the facial orientation of the Source Image, indicating some robustness in the model across different poses. However, the right side of the second row and the third row still show misalignment in hair parting, leading to color inconsistencies in the Output. The left side of the fourth row shows the model discarding the hair ends while supplementing the bangs area of the Source Image, though there is still some deviation in the hair color's centerline. The last row depicts complex textures scenarios where the Output hair color does not match the Target Image, due to the compression of images required for diffusion input and the bilateral filtering operation removing high-frequency color details. For cases with missing target region hair color, post-processing with PatchMatch can fill in the blank areas, as shown in Figure 4. The effectiveness is demonstrated in the ablation experiments in Table 1.

We will correct typos and missing text in the future version. Thank you for your assistance!

Due to the upcoming deadline for the discussion, I would like to confirm if you have any further questions or if there are areas where you need further clarification. If there are no more issues with my submission, I hope to receive your feedback and proceed.

Thank you very much for taking the time to review my work amidst your busy schedule. I greatly value your opinions and hope to improve my research under your guidance.

Thank you for your assistance, and I look forward to your reply.

Thank you to the reviewers for pointing out the weaknesses.

W1: This study is the first to propose a diffusion-based pipeline in the hair editing field, addressing the issue of multi-color hair structure in text2img and img2img scenarios. Previously, no methods specifically focused on the transfer of multi-color hairstyles, and our visual effects, whether in text2img or img2img, are among the most effective to date. We introduced the Warping module into the hair editing domain to align hair colors, resulting in a Color Proxy that enables the model to faithfully align the target hair color while maintaining the hairstyle structure. To achieve color alignment through the warping module, we undertook the following work:

1. Addressing the lack of paired datasets: we used semantic segmentation and data augmentation to obtain the hair region deviating from the original face.
2. Preserving hairstyle structure by removing low-frequency details: bilateral filtering was employed to remove low-frequency details, enhancing the quality of hairstyle generation. Overall, the Warping Module can provide guidance for future researchers applying it in the hair editing field. Our proposed hair-agnostic mask is also crucial for enabling hair editing tasks using diffusion methods, as it maximizes the retention of hairstyle-independent attributes while preserving the editability of the hairstyle region. Although our pipeline incorporates components from existing technologies, we focus on integrating and optimizing these techniques to address specific issues in practical applications. For instance, how to blend hair color and hairstyle to achieve balance while reducing artifacts, filling blank areas without a hairstyle, and improving the quality of generated hairstyles. Additionally, how to achieve more flexible editing by decoupling hair color and hairstyle through multiple steps are all considerations for practical application.

W2:1. Color discrepancy in facial images

The issue of color discrepancy outside the inpainting region is a common challenge in diffusion-based inpainting tasks. For example, in the ControlNet-Inpainting model used in our paper, the initial convolution layer of the UNet architecture merges the noisy latent, masked image latent, and mask, all of which are collectively influenced by the text embedding. Consequently, subsequent layers in the UNet model struggle to obtain pure masked image features due to the text's influence. The paper "BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion" proposed a blending operation to address this issue. It presents a simple pixel space solution by first blurring the mask and then performing copy-and-paste using the blurred mask. It is important to emphasize that this is a common challenge for diffusion inpainting and is beyond the scope of this paper.

2. Preserving hairstyle details

There are indeed slight changes in the generated hairstyle structure compared to the original, but the overall structural control is good. HairCLIPv2 does not faithfully reproduce the target image's hair color at all. This paper primarily focuses on multi-color transfer. ControlNet, as a sparse control matrix, relies on canny maps for hairstyle structure retention. However, Canny maps cannot achieve pixel-level control, resulting in minor differences between the generated and original hairstyles. These differences do not affect the overall hairstyle structure. Training a more refined canny map with a hairstyle dataset could potentially improve the quality of hairstyle retention, which is a direction for future improvements. As shown in Figure 3 and Table 2 of the supplementary PDF, we supplemented qualitative and quantitative experiments to demonstrate our method's ability to preserve hairstyle details.

W3: When the target hairstyle is the one generated by the Hairstyle Editing Stage,I_c is P^s, while retaining the original hairstyle means I_i. This aspect was not clearly explained. The style proxy is intended to guide images with the target hairstyle, as mentioned in the introduction. Therefore, when only changing the hair color while retaining the original hairstyle, P^s in Figure 2 will be replaced by I_i. We will address this in future versions. Thank you for your help!

Due to the upcoming deadline for the discussion, I would like to confirm if you have any further questions or if there are areas where you need further clarification. If there are no more issues with my submission, I hope to receive your feedback and proceed.

Thank you very much for taking the time to review my work amidst your busy schedule. I greatly value your opinions and hope to improve my research under your guidance.

Thank you for your assistance, and I look forward to your reply.

Thank you to the reviewers for pointing out the weaknesses.

1. We have reviewed the HairNet video and paper. The issue of transferring hair across diverse/different poses is not the problem addressed in our paper. HairNet can effectively control different facial angles, which would be highly beneficial for our method to achieve multi-angle and multi-color hair transfer in the future. We will cite this work in our revised version. However, it is challenging for GAN-based methods to excel in both editability and retention of original features. In Figure 5 of our parper, we compare various space mapping methods based on StyleGAN (including Barbershop based on FS space), and it is evident that they struggle to reconstruct such niche features as multi-color hairstyles. In the global response, we present some examples of our Warping module handling cases with significantly different poses.
2. Advantages of GAN-based methods:
3. The speed of image generation is very fast. Benefiting from the decoupled facial features in StyleGAN, it is almost possible to change certain feature vectors and quickly generate the corresponding images.
4. They retain editability, allowing operations in the latent space, where feature vectors in the StyleGAN latent space can be edited to control different features such as hair length and face orientation.
5. Barbershop cannot control multi-color hair transfer. Although the FS space based on optimization can retain facial feature details, it is difficult to decouple the two features of hair color and hair color structure, because these two features are highly coupled (Figure 5 in the parper compares Barbershop).

Below are the responses to the questions:

A1: The global response section G1 discussion on the Warping module's ability to transfer hair color under extreme conditions, including significant posture differences, complex textures, and large differences in hair regions.

A2: Diffusion models generate images through a multi-step denoising process, from pure noise to clear images. This allows the model to better capture image details and semantic information at different levels during the generation process, rather than generating the entire image at once. This gradual generation approach helps avoid overfitting, as each step improves and adjusts different levels of the image. This editing capability makes diffusion models more flexible and controllable for different editing tasks. Our paper adopts several common Diffusion Inpainting methods to ensure the masked region remains unchanged. For example, we input the Hair Agnostic Mask and Source Image together into the model, allowing it to distinguish between known and unknown areas, thus only filling in the unknown areas without affecting the known ones. During the generation process, noise can be injected only into the masked areas while keeping the unmasked areas unaffected. This means the unmasked areas always retain their original pixel values during the diffusion process, ensuring these areas are not influenced by the generation process. For the hairstyle region needing editing, we remove information from the original image other than the face, use sparse Controlnet's openpose and textual information to control the hairstyle generation StyleProxy, then further obtain the hairstyle region mask of StyleProxy. As mentioned above, we input these together into the model, thereby preserving both the editability of the hairstyle and the high fidelity of the unrelated regions.

A3: Our method does not support additional edits post-hairstyle transfer. Using StyleGAN, direct operations in the latent space can achieve quick and direct feature editing. However, the process of generating images via Diffusion is a multi-step gradual denoising process, where the latent space is not directly editable but involves a series of denoising steps to achieve image generation. Therefore, direct editing in the latent space is more challenging. The generation process of diffusion models requires multiple iterative steps, making real-time editing as in StyleGAN's HairNet impractical. Although some degree of editing can be achieved through conditional generation, such editing usually needs pre-set conditions before generation, making it difficult to achieve real-time, dynamic feature adjustments.

A4:

• Editability:

1. For text-based hairstyle editing, compared to the current HairCLIP and HairCLIPv2 based on the combination of CLIP and StyleGAN, their editing capabilities are limited by the dimensions of the StyleGAN latent space and the GAN training process. They cannot capture niche and complex facial features or information outside the face, such as background and arms. In contrast, diffusion models can combine text conditions multiple times during the generation process, ensuring that each step of image generation aligns with the text description. This method better captures the details and complexity in the text description.
2. Our method can integrate multiple conditions for additional control, such as stroke maps to control hair color, controlnet to introduce canny maps to control hair structure, and denspose maps to ensure the hairstyle generation aligns with facial posture. These capabilities significantly enhance the editability of our method.

• Handling pose diversity: Compared to methods specifically dealing with multi-pose hair transfer, our paper focuses more on color transfer while maintaining the original hairstyle. The warping module aligning the target hair color and region with different facial poses is a conditional GAN method, as it is currently challenging to decouple hairstyle and hair color directly in the latent space of diffusion models. There are also related articles exploring multi-view generation in diffusion models, such as DiffPortrait3D, which achieves multi-view facial effects.

Thank you for your valuable feedback.

1. We will correct the typos and missing text, and supplement the finetune hyperparameter settings in the future version.
2. We have showed the limitations of our method's dependency on masks in the supplementary materials. We have supplemented the limitations and ablation experiments proving the effectiveness of the warping module (Figure 1, Table 2).
3. The results of the ablation experiments in our paper clearly demonstrate the considerations at each step of our pipeline, and the reasons have been analyzed.
4. User Study strictly follows HairCLIPv2.
5. We have supplemented the experiments with HairFastGAN metrics and detailed visual comparisons with HairFastGAN and StyleGANSalon in the global response sections G3, G4 and G5. The following outlines reasons for not making certain comparisons:
   • Quantitative comparisons with StyleGANSalon: StyleGANSalon is based on EG3D pre-trained on the FFHQ dataset for pose transfer, and EG3D does not provide pre-trained models on CelebA-HQ. Hence, we do not conduct quantitative metric comparisons with StyleGANSalon. Moreover, StyleGANSalon performs comparisons on two subsets of the FFHQ dataset without disclosing the partitioning method.
   • Quantitative self-transfer comparisons with StyleGANSalon: Unlike HairFastGAN and our method, which can decouple and transfer hair color and hairstyle, StyleGANSalon transfers the entire hairstyle. Thus, we believe that comparisons would be unfair.
   • Qualitative color transfer comparisons with StyleGANSalon: StyleGANSalon transfers the entire hairstyle and cannot decouple and transfer hair color independently, making hair color transfer visual comparisons infeasible.

Regarding the concern that "the Control-SD method produces hairstyles that are virtually identical to those generated by HairDiffusion": The advantage of our method over diffusion-based text2img lies in multi-color text control and the maintenance of the original hair color when only using text to control the hairstyle. Our work is the first to introduce a warping module for color alignment in the field of hairstyle editing, differing from previous StyleGAN-based methods. This approach faithfully aligns with the target hair color without being constrained by the limitations of the StyleGAN latent space. To achieve color alignment through the warping module, we undertook the following steps:

1. Addressing the lack of paired datasets: We used semantic segmentation and data augmentation to obtain hair regions deviating from the original face.
2. Preserving hairstyle details: We employed bilateral filtering to eliminate low-frequency details, improving the quality of hairstyle generation. Overall, our approach can guide future researchers in applying the warping module to hairstyle editing. The hair-agnostic masks we proposed is crucial for enabling hairstyle editing tasks with diffusion methods. It maximizes the preservation of hairstyle-independent attributes while retaining the editability of the hairstyle region. Although our pipeline borrows certain components from existing techniques, we focus on integrating and optimizing these techniques to address specific issues in practical applications. For example, achieving a balance between hair color and hairstyle blending while minimizing artifacts, filling in blank regions without hair, improving the quality of generated hairstyles, and achieving more flexible multi-condition editing through the multi-step decoupling of hairstyle and hair color are all considerations in practical applications.

Responses to specific questions are as follows:

A1: In the global response sections G3, G4 and G5, we have supplemented experiments with HairFastGAN and StyleGANSalon, including detailed visual comparisons in self-transfer methods with HairFastGAN and StyleGANSalon. As shown in Figure 3, our method demonstrates better visual effects in preserving multi-color hairstyles and hairstyle details under facial detail attributes and occlusion conditions. Since whole hair transfer is not the focus of our work and StyleGANSalon is a pose transfer model pre-trained on EG3D with the FFHQ dataset, we do not conduct quantitative metric comparisons with StyleGANSalon.

A2:

• 1. Limitations of the Warping Module: We have supplemented discussions on the limitations of the warping module in extreme cases of hair color transfer in Figure 1 of the supplementary PDF, including significant pose differences, complex textures, and large differences in hairstyle regions.
• 2. Effectiveness of the Warping Module: We have added an ablation experiment where 2000 images from the CelebA-HQ validation set are divided into source images and target hair color images for mutual color transfer experiments. The results are shown in Table 1 of the supplementary PDF. As for Effectiveness of CLIP Fine-tuning, the results are shown in section G6 of the supplementary PDF.

A3: Dataset Details: Number : 4,625;

Average size: 224.93 x 264.25;

Categories:

• Hair colors: purple, red, green, blue, brown, silver, blonde, pink, burgundy;
• Color structures: ombre, dip dyed, streaked, half and half, split color.

Data acquisition: We crawled tens of thousands of multi-color hair images using keywords, cleaned the data, and obtained approximately six thousand multi-color hair images. These images were input into GPT with predefined hair color and structure categories to generate text annotations like "a photo of {colors}, {color structure} hairstyle." We then had 10 professional annotators rate the annotations in multiple rounds, discarding low-rated images, resulting in 4,625 multi-color text-image pairs. If the paper is accepted, we will make this dataset publicly available.