FuseAnyPart: Diffusion-Driven Facial Parts Swapping via Multiple Reference Images

[W1. The problem of skin color change.] FuseAnyPart may encounter skin color changes, particularly when there is a significant difference between the skin colors of the source images and the target image. This issue arises because the source and target images are fused in the latent space, and the global attention mechanism diffuses the color features. However, this of skin color changes issue can be effectively resolved by replacing the generated skin regions with the inverted latent representations of the original skin color using DDIM inversion. This ensures the skin color region remain unchanged. The improved results are illustrated in Figure 2 of the newly submitted PDF. The pseudocode is as follows.

Algorithm: Merging with Ground Truth Latent
Input: latents_ref_img, init_noise, pixel_mask
Output: latents

1. Initialize thres_i
2. If i < thres_i
   a. Retrieve noise_timestep from scheduler:
      - noise_timestep <- scheduler.timesteps[i + 1]
   b. Add noise to the latent of the reference image to obtain ground truth noised latent:
      - gt_latent <- scheduler.add_noise(latents_ref_img, init_noise, noise_timestep)
   c. Update latents using pixel mask:
      - latents <- (1 - pixel_mask) * gt_latent + pixel_mask * latents

[W2. Feature injection difference to IPAdapter and ControlNet.] IPAdapter is not an addition-based injection method. It employs a decoupled cross-attention strategy, where an additional cross-attention layer is added to each original cross-attention layer to inject image features. And our method could perform better in facial parts swapping tasks. In Section 4.3 of our paper, titled "Cross-attention vs. Addition," we compared our feature injection method with cross-attention. The results indicate that our injection method is superior. ControlNet is an addition-based injection method. We have not yet compared it in our experiments. However, for face parts swapping tasks, the majority of the target image remains unchanged, and the features needing injection are only those corresponding to the parts requiring replacement, thus eliminating the need for a complex network structure. Our method involves the addition of only linear layers, whereas ControlNet requires replicating the parameters of UNet. Therefore, our approach has a clear advantage regarding the increase in parameter count. For this task, our method is simple but effective.

[W3. The training strategy.] FuseAnyPart uses reconstruction as a proxy task and does not require swapped paired faces, which draws inspiration from E4S and DiffSwap. For instance, the target face image and the source face images, which provide different facial parts, are sampled from different images of the same identity. Our paper details the training and inference pipeline in Lines 60-62. A visual grounding model is employed to detect bounding boxes for feature decoupling across different regions.

[W4. The term 'one-step'.] The 'one-step' mentioned in Line 65 means that the eyes, mouth, and nose can be swapped in a single inference, unlike traditional inpainting methods, where they are swapped sequentially. Therefore, it does not contradict the 50 steps of the DDIM sampler. We appreciate the reviewer's careful reading and will revise the paper in the final edition to clarify this point. The term 'one-step' in Line 65 will be replaced with 'simultaneous'.

[Q. Evaluate the performances quantitatively.] As described in Section 4.2, three metrics, including Fréchet Inception Distance (FID), Organ Similarity (Osim), and Reconstruction Mean Squared Error (MSE), are adopted to evaluate the performance quantitatively. FID is used to evaluate the generated faces' overall image quality and visual realism. Osim is used to evaluate how well individual features such as the eyes, nose, or mouth match between the generated image and the sources, focusing on the fidelity of the swap within localized regions. The MSE metric measures the pixel-wise accuracy of the reconstructed image against the original. It helps quantify the preservation of the non-swapped parts of the face and the seamless integration of swapped parts. Regarding the visual plausibility, we provided some qualitative results in the paper, and additionally, we have included more qualitative results in the newly submitted PDF. The results showcase the efficacy of our method in generating plausible faces even when there are significant differences between the source and target images, including differences in age and race.

[W1. Details about OSim.] Osim measures the similarity between the swapped facial parts (eyes, nose, mouth) in the generated image and those in the reference images. For example, we utilized the CelebA-HQ dataset and employed the grounding-dino detection model to identify and extract bounding boxes for the eyes. Each cropped eye region (the eye image) was then labeled with the identity (ID) of the corresponding individual. Then, we train the ResNet50 with arcface loss, which maximizes the intra-class similarity (among the same IDs) and minimizes the inter-class similarity (across different IDs). After training, we use the ResNet50 model to extract feature vectors from the eye regions. Osim is computed using cosine similarity between these feature vectors: $Osim = \frac{f(a)\cdot f(b)}{\Vert f(a) \Vert \Vert f(b) \Vert}$, where f represents the feature extraction function implemented by our trained ResNet50 model and a and b are the input images of the specific organ.

[W2. Alternatives to the Addition-based Injection Module.] The Addition-based Injection Module beats multiple attention-based baselines in Table 2 and Figure 7. Although simple, the Addition-based Injection Module is the most suitable for the facial parts swapping task and validated by extensive experiments.

[Q1. The visualization results.] This point refers to the baseline method, IP-Adapter. IP-Adapter depends on both text and image prompts. If the text prompt is inaccurate or even conflicts with the image prompt, it may have little effect on the results, leading to outcomes that do not align with the text prompt. We will clarify this point to ensure it is clear in the final edition. Please check the Figure 5 in the newly submitted PDF.

[Q2. The target image for training.] FuseAnyPart uses reconstruction as a proxy task for training and does not require swapped paired faces, which draws inspiration from E4S and DiffSwap. For instance, the target face image and the source face images, which provide different facial parts, are sampled from different images of the same identity. Our paper details the training and inference pipeline in Lines 60-62.

[Q3. Code available.] Once the paper is accepted, the training and inference code, model weights, and essential documentation will be released to the public. The authors aim to make a valuable contribution to the NeurIPS community.

[Q4. The VAE version.] FuseAnyPart is based on the widely used SD-v1.5, and the VAE model utilized in this work is the same as that in SD-v1.5. The authors acknowledge that a lightweight yet effective pre-trained VAE may exist, but exploring this is beyond the scope of FuseAnyPart.

[L. The landmark detection network.] A visual grounding model is used to detect bounding boxes around faces, rather than relying on a landmark detection network, as described in Line 216.

[W1. The primary challenge lies in the fusion mechanism.] Traditional face-swapping methods first perform face reenactment and then paste it onto the target image in pixel space, as illustrated in FSGAN and DiffFace. However, operations in pixel space often result in unnatural images with visible seams, leading to a low success rate in practical applications and producing many poor outcomes as shown in Figure 3 of the newly submitted PDF. The current mainstream face-swapping techniques now perform the fusion of source and target images in the latent space, resulting in more harmonious generated images. Therefore, the feature fusion mechanism becomes critical in affecting the quality of the generated images. In the facial parts swapping task, the number of source images increases from one to multiple, further complicating the fusion process and making this issue more prominent. As shown in Table 2 and Figure 1 of this paper, the authors conducted extensive experiments to validate the superiority of the proposed fusion mechanism quantitatively and qualitatively. We thank the reviewer for highlighting this issue and suggesting that the task description be more precise. However, we hope the unique aspects and challenges of the task are adequately appreciated.

[W2. The details.]

1. The Initial Point of the Denoising process. During the training phase, the initial point consists of concatenating the target image with noise in latent space, the masked image in latent space, and the mask itself, as illustrated by z_t' in Figure 2. In the inference phase, the target image with noise in latent space is replaced with Gaussian noise at the initial point. The above is discussed in Lines 186-189 of the paper.
2. The training Objective. FuseAnyPart uses reconstruction as a proxy task and does not require swapped paired faces. For instance, the target face image and the source face images, which provide different facial parts, are sampled from different images of the same identity. Therefore, reconstructing the target image allows the FuseAnyPart to gain the ability to generate natural images by merging different organs. Our paper details the training and inference pipeline in Lines 60-62.
3. The dimension of the extracted feature. The feature $f$ from the CLIP image encoder has a shape of $(h\times w) \times c$, which corresponds to $h\times w$ visual tokens. This feature $f$ can be input into an MLP to align with the dimensions of the latent features in UNet.

[W3. Questions on the CelebA dataset.] The CelebA dataset now includes identity annotations in the “Anno/Identity_CelebA.txt file,” which can be downloaded from the official CelebA website. During training with FuseAnyPart, source and target images are sampled from different images of the same identity. More qualitative results are added in Figure 4 in the newly submitted PDF, and they will be added in the final edition of the paper.

[W4. More comparisons.] The state-of-the-art face-swapping methods supporting face parts swapping are all included in Table 1 of our paper. According to their papers, FuseAnyPart beats Diffswap in FID and OSim, and Diffswap beats SimSwap(2020) and FSGAN(2019). A qualitative comparison of FuseAnyPart with DiffFace is shown in Figure 3 of the newly submitted PDF, demonstrating a significant advantage of FuseAnyPart.

[W. The weakness of FuseAnyPart.]

1. Diffusion models typically have high computational complexity due to the need for recursive iterations, which limits their applications to real-time or on lower-end devices.
2. More diverse testing results are illustrated in Figure 1 in the newly submitted PDF to validate the robustness and generalization of FuseAnyPart.
3. Most models rely on high-quality training datasets. The quality of images can be improved using super-resolution methods. For example, the dataset used in this work is CelebA-HQ, which is reconstructed with super-resolution through GANs.

[Q1. Performance of low quality inputs.] The performance of FuseAnyPart may be affected by lower-quality or varying lighting conditions input images. However, some preprocessing techniques, such as facial alignment and super-resolution, can be adopted to mitigate these effects.

[Q2. Optimizations for real-time applications.] Algorithms like Latent Consistency Models (LCM) can overcome the slow iterative sampling process of Diffusion Models, enabling fast inference with minimal steps instead of the usual dozens or hundreds. In engineering, techniques like int8 model quantization can significantly reduce computational load. Together, these strategies can speed up FuseAnyPart.

[Q3. Results of significantly different racial or age groups.] These results are shown in Figure 1 in the newly submitted PDF.