WarpGAN: Warping-Guided 3D GAN Inversion with Style-Based Novel View Inpainting

Q1: Incremental Innovation over HFGI3D.

R1: WarpGAN is NOT an incremental innovation over HFGI3D.

1. HFGI3D is NOT a combination of the "warping-and-inpainting" strategy and the 3D GAN inversion. The core idea of HFGI3D is to use warping operations to synthesize pseudo-multi-view images, where the occlusion regions of the warped images are simply filled by using the results of PTI. However, such a filling operation is coarse. Since PTI is optimized by using a single-view image and has low fidelity, it leaves obvious discontinuities when combined with the warped image (similar to the coarse result $\hat{\mathbf{I}}_{novel}^{\text{initial}}$ from 3D GAN inversion, as shown in Fig. 2 of our paper). We believe that such a way is merely a simple fill rather than true inpainting, and the synthesized pseudo novel-view images cannot be regarded as ideal results. Hence, they still require a final optimization. In contrast, our SVINet can truly inpaint the warped image and directly generate natural and plausible occlusion regions.
2. Our method is NOT a simple replacement of optimization-based approaches with feed-forward methods. The purpose of synthesizing pseudo-multi-view images in HFGI3D is to assist optimization. However, since PTI optimizes solely based on a single-view image, it is prone to generating novel-view images with poor geometry quality. Using PTI to fill in the warped image results in pseudo-multi-view images with inferior geometry quality. Moreover, inevitable errors in the depth map can make warped images unreliable under large view changes, thereby affecting the quality of pseudo-multi-view images. Consequently, low-quality pseudo-multi-view images can have a negative impact on the final optimization. In comparison, the occlusion regions synthesized by our SVINet are of higher quality than those filled by PTI and have the ability to correct unreliable warped images to some extent (as shown in Fig. 5(b)). As shown in Fig. 5(c), the geometry quality of the novel-view images generated by WarpGAN-Opt is better than that of the results from HFGI3D. This indicates that the novel-view images synthesized by our WarpGAN have better quality than the pseudo-multi-view images generated by HFGI3D.

Q2: Unvalidated Generalizability.

R2: The extreme cases you mentioned, such as extreme poses, exaggerated expressions, complex occluders, or non-face objects, are indeed challenging for methods based on 3D GAN inversion. Existing methods generally struggle to perform well under these circumstances. We discuss the generalizability of our method to these extreme cases here in principle and will supplement the performance of our method under these extreme conditions in the Appendix.

For extreme poses, although our method can generate high-quality novel-view images, there are certain limitations in the range of pose changes. We mentioned the limitations of our method in the Conclusion section of the paper. Due to inevitable errors in the depth map, the quality of the images generated by our WarpGAN will decrease when the camera pose changes significantly. More seriously, when the camera pose changes significantly, leaving no visible area in the novel view, our method will degrade to a typical encoder-based 3D GAN inversion (as mentioned in lines 76-79 of the Appendix).

For exaggerated expressions and complex occluders, which can be collectively referred to as out-of-domain regions in the GAN field, existing GAN inversion methods, limited by the low-dimensional latent space of GANs, struggle to reconstruct these out-of-domain regions effectively. However, the visible regions of the novel-view images synthesized by our method come from the original image, and the occlusion regions come from SVINet, which can bypass the information loss caused by the low-dimensional latent space and thus better preserve these out-of-domain regions.

For non-face objects, EG3D provides a pre-trained model for cats, and we will evaluate our method on cats in the Appendix. Faces are more complex than cats. Thus, in principle, our method should be able to generalize well to novel-view synthesis of cats.

Regarding symmetry-aware feature extraction, as shown in Eq. (4), we do not directly add the features of the input image and its mirrored image. Instead, we use featurewise linear modulation (FiLM) to refine the features, which can be considered as a weighted fusion. When the input face image is significantly asymmetric, the symmetric weight will decrease. Similarly, Triplanenet (WACV'24), when using symmetry priors for loss calculation, introduces a per-pixel Gaussian density with a pixelwise uncertainty map that assigns lower confidence to the region in the mirrored image where the symmetry assumption fails. Therefore, our explanation is justified in theory, and we will also supplement our performance when the face prior does not work (such as in the case of ``hand covering face'' that you mentioned) in the Appendix.

Q3: Typo.

R3: Thank you for your attention to detail. We will correct the typo in the revision.

Q1: Novelty.

R1: Although the technical components are established techniques, we do not simply stack them. Instead, we are the first to successfully apply the "warping-and-inpainting" paradigm to 3D GAN inversion, by innovatively designing a warping-back strategy and introducing a symmetry prior along with latent codes from a 3D-aware GAN, greatly alleviating multi-view inconsistency. Our work clearly showcases the great potential of the "warping-and-inpainting" paradigm in 3D GAN inversion. Qualitative and quantitative results confirm the effectiveness of our design. We believe that the fast inference time, together with the good performance of the framework, will benefit and ease future research on novel-view synthesis of human faces.

Q2: Poor Performance of 3D GAN Inversion.

R2: Entry A in Table 2 reports results obtained directly from the initial novel-view images produced by 3D GAN inversion. Recent 3D GAN inversion methods fall into two categories: optimization-based and encoder-based. Optimization methods are time-consuming, whereas encoder-based methods suffer from low fidelity. To boost the fidelity of encoder-based methods, some works complement the high-dimensional $\mathcal{F}$ space, yet GOAE (ICCV'23) shows that such a way introduces distortion in occluded regions. Because the primary role of 3D GAN inversion in our method is to produce a depth map and provide an initial fill for occluded regions in the warped image, we adopt the most basic approach (Entry A): simply projecting the image into the low-dimensional $\mathcal{W}$ space of EG3D. This inevitably incurs substantial information loss, which in turn leads to lower fidelity in the novel-view images. In contrast, our full model retains the visible regions from the original image and generates occlusions with SVINet, thereby bypassing the information loss inherent to the low-dimensional latent space and achieving much better results.

Q3: Comparison with Diffusion-Based Methods.

R3: The "warping-and-inpainting" strategy in diffusion models (such as GenWarp(NeurIPS'24) and LucidDreamer(ICCV'24)) is primarily designed for novel-view synthesis of scenery. In contrast, our work is object-centric, focusing on novel-view synthesis of human faces. Moreover, their training datasets and problem settings differ from ours. Hence, these methods cannot be directly transferred to our task.

Following the suggestion of Reviewer bp2G, we have also added the performance comparison with a representative diffusion-based method, DiffPortrait3D [1], on the CelebA-HQ dataset. The FID value is 19.26 and the ID value is 0.7964 for DiffPortrait3D, while our WarpGAN achieves an FID value of 19.12 and an ID value of 0.7882. The metrics indicate that our method can achieve performance comparable to diffusion-based methods in the real human face domain.

Moreover, due to the uncertainty in the generation of 2D diffusion models and their lack of an explicit 3D representation, multi-view inconsistency is a common issue. To address this problem, DiffPortrait3D proposes a view-consistency module that processes multiple novel-view images simultaneously to improve the consistency of the generated novel-view images. However, such a way requires a significant amount of GPU memory. The authors of DiffPortrait3D set the number of novel-view images to 8 in implementation, which results in the need for inference on at least a single A100. Due to the limitation on the number of images processed in parallel, flickering artifacts can still be observed in the multi-view rendering animations provided in the DiffPortrait3D demo. In contrast, our WarpGAN, benefiting from the multi-view consistency prior of 3D GANs as well as the symmetry prior and latent code from the 3D-aware GAN that we introduce, can easily generate multi-view consistent rendering videos. We will provide a demo alongside our open-source code.

It is worth noting that our method can train and infer on consumer-grade GPUs (such as the 3090 and 4090), with inference times on a single 4090 GPU within milliseconds. In contrast, DiffPortrait3D requires training on multiple professional-grade GPUs and inference on a single professional-grade GPU, with inference times exceeding 1 minute on a single A100 GPU.

Q4: No Diffusion Used.

R4: We discuss this from three perspectives:

1. Our work is a study of 3D GAN inversion, so we primarily consider using GAN-based methods to achieve novel-view synthesis of human faces from a single image, as done in SOTA 3D GAN inversion methods (e.g., Triplanenet (WACV'24), In-N-Out (CVPR'24), Dual Encoder (NeurIPS'24)). Diffusion models are trained on massive image corpora that inevitably cover the celebrities present in CelebA-HQ. Hence, it would be unfair to compare them with 3D GAN-inversion methods.
2. 3D-aware GANs can provide strong facial priors and multi-view-consistent generation, together with a latent space that is semantically rich and highly disentangled. We thus believe that advancing 3D GAN inversion itself is worthwhile. Extensive experiments demonstrate that our WarpGAN delivers high-quality novel-view face synthesis, runs faster, and consumes fewer resources than diffusion-based alternatives.
3. In future work, we believe that incorporating diffusion models into our framework is a promising research direction, such as attempting to use diffusion-based methods to replace SVINet for inpainting occlusion regions.

References:

[1] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis. CVPR24.

After reading the rebuttal, my concerns are basically addressed. I still maintain my viewpoint that the main efforts of this paper are devoted to improving the quality of synthesis results.

Q1: Impact of Occlusion.

R1: We evaluate our method against baseline methods from both quantitative and qualitative perspectives to clearly reveal how occlusion affects each method. Owing to space limitations, the original paper offers only a brief discussion. Here we provide a more complete analysis.

1. Quantitatively, FID reflects the overall quality of the synthesized novel views, while ID measures the identity preservation and the plausibility of the generated occluded regions to some extent. The LPIPS metric on MEAD further reflects the reconstruction fidelity of novel views and, indirectly, the accuracy of the occluded areas. Across these metrics, our method achieves the best results, suggesting that baselines produce less realistic occlusions.
2. Qualitatively, extensive visual comparisons are presented in Fig. 3 and Fig. 4 of the main paper, as well as Fig. 4 in the supplementary materials. They demonstrate that for regions unobserved in the input (e.g., ears or side profiles), baselines tend to yield blurry artifacts or unnaturally flattened heads, whereas our WarpGAN generates occluded facial regions that are both more coherent and richer in detail.

Q2: Sampling Strategy for Synthetic Samples.

R2: To facilitate training and enhance multi-view consistency, we leverage the powerful generative capacity of EG3D to generate multi-view-consistent synthetic images as additional supervision. G-NeRF claims that regions of lower density in the latent space of StyleGAN may be inadequately represented, leading to synthetic images with degraded geometry quality. After checking the synthetic images used in our training, we did not observe the fidelity or visual-appeal issues reported by G-NeRF. We attribute this to two key reasons.

1. EG3D checkpoint: As G-NeRF does not release the code for generating synthetic images, the exact details of their sampling strategy remain unknown. However, the EG3D checkpoint they provide was trained on the standard FFHQ dataset, whereas we use a checkpoint fine-tuned on a rebalanced FFHQ with a more uniform pose distribution. According to the EG3D authors, such a fine-tuned model produces better 3D shapes and better renderings from steep angles than the one trained on the standard FFHQ. We therefore consider this to be a reason for our observations.
2. Sampling ranges: For latent codes, we sample from a Gaussian distribution and then map to the $\mathcal{W}$ space (as mentioned in the G-NeRF paper). For camera poses, because the FFHQ dataset has a limited pose distribution, sampling over a wider range can introduce artifacts. Therefore, our sampling range primarily focuses around the frontal view of the face: yaw and pitch are both drawn within approximately $\pm 30^{\circ}$ of the frontal view. Poor geometry quality typically appears at extreme angles, so our constrained sampling avoids the issues highlighted by G-NeRF.

Q3: Organization of the Paper.

R3: Thank you for your valuable feedback. We agree that introducing warping as a Preliminary at the beginning of the Methodology section will make our Overview more concise and improve the readability of the paper. We will revise our paper according to your suggestions.

Q1: Comparison with Diffusion.

R1: Our work is a study of 3D GAN inversion, so we primarily compare our method with other 3D GAN inversion-based approaches, as done in SOTA 3D GAN inversion methods (e.g., Triplanenet (WACV'24), In-N-Out (CVPR'24), Dual Encoder (NeurIPS'24)). Following your comments, we have also added the performance comparison with the diffusion-based method, DiffPortrait3D [1], on the CelebA-HQ dataset. The FID value is 19.26 and the ID value is 0.7964 for DiffPortrait3D, while our WarpGAN achieves an FID value of 19.12 and an ID value of 0.7882. The metrics indicate that our method can achieve comparable performance to diffusion-based methods on the real human face domain.

Moreover, due to the uncertainty in the generation of 2D diffusion models and their lack of an explicit 3D representation, multi-view inconsistency is a common issue. To address this problem, DiffPortrait3D proposes a view-consistency module that processes multiple novel-view images simultaneously to improve the consistency of the generated novel-view images. However, such a way requires a significant amount of GPU memory. The authors of DiffPortrait3D set the number of novel-view images to 8 in implementation, which results in the need for inference on at least a single A100. Due to the limitation on the number of images processed in parallel, flickering artifacts can still be observed in the multi-view rendering animations provided in the DiffPortrait3D demo. In contrast, our WarpGAN, benefiting from the multi-view consistency prior of 3D GANs as well as the symmetry prior and latent code from the 3D-aware GAN that we introduce, can easily generate multi-view consistent rendering videos. We will provide a demo alongside our open-source code.

It is worth noting that our method can train and infer on consumer-grade GPUs (such as the 3090 and 4090), with inference times on a single 4090 GPU within milliseconds. In contrast, DiffPortrait3D requires training on multiple professional-grade GPUs and inference on a single professional-grade GPU, with inference times exceeding 1 minute on a single A100 GPU.

Q2: Measures of Spread.

R2: Statistical significance is indeed crucial for reliable scientific conclusions. However, the calculation of confidence intervals or standard deviations is applied mainly to prediction tasks such as object detection and classification, where the ground truth is available. In contrast, in the generative domain, there is usually no ground truth of the generated images. The goal is to synthesize new images that conform to certain data distributions, rather than making a decision on a single input. As a result, most studies in the generative domain do not analyze statistical significance.

Q3: Small Gains.

R3: Although our method shows only minor improvements over optimization-based baselines in certain metrics, our inference speed is significantly faster (over 500 times faster), demonstrating the efficiency of our method.

Q4: Comparison of the Full Pipeline.

R4: Our method consists of two steps: warping and inpainting. We need to warp the input image to the target view first before using SVINet for inpainting. Therefore, it is NOT feasible to conduct ablation studies separately for these two steps.
Regarding the "rerendering" you mentioned, if you are referring to WarpGAN-Opt used in editing operations, we provide extensive visual comparisons and analyses in Fig. 4 of the Appendix.

Q5: Comparison Between SVINet and Diffusion Models.

R5: We experimented with the diffusion-based RePaint [2] to inpaint our warped images. Unfortunately, since the holes in our warped images are often large and complex (as shown in Fig. 2), RePaint, which was not trained in our scenario, struggles to generalize to our task and frequently fails to inpaint the holes. We also provide metrics on CelebA-HQ, with an FID of 60.79 and an ID score of 0.7390, indicating poor performance. Additionally, it takes about 5 minutes per image, which is much slower than our SVINet. However, we believe that modifying diffusion-based methods to replace our SVINet is a promising research direction, and we plan to explore this in the future.

Q6: Outdated GAN.

R6: Due to the powerful generative capabilities of diffusion models, which overcome issues such as domain limitations faced by GANs, an increasing number of works in the generative domain adopt diffusion-based methods.

However, thanks to the strong priors for human heads embedded in 2D/3D/4D GANs, they are capable of generating human heads with rich details. Although the low-dimensional latent space of GANs imposes domain limitations, it also endows GANs with rich semantic information and powerful disentanglement capabilities within the latent space, enabling finer-grained control over human heads. Given the limitations of diffusion models, such as slower inference speeds and higher computational resource requirements, a significant number of researchers continue to explore GAN-based methods.

Additionally, in the generative domain, diffusion models do not completely outperform GAN-based methods. Both GANs and diffusion models have their own strengths and weaknesses. Given the powerful generative capabilities of GANs and the strong generalization abilities of diffusion models, many researchers have attempted to combine the two to achieve complementarity. Here are a few examples:

1. $\mathcal{W}_+$ Adapter [3] utilizes StyleGAN (2D GAN) to enhance the disentanglement and facial information retention of diffusion models.
2. DiffPortrait3D [1] leverages EG3D (3D GAN) to control portrait novel-view synthesis and employs PanoHead (3D GAN) to generate multi-view images that assist in the training of the diffusion model.
3. AvatarArtist [4] uses Next3D (4D GAN) to bridge 2D images and triplanes, and employs a robust 2D diffusion prior to handle diverse data distributions.

Therefore, we believe that research on GANs remains highly meaningful. Combining the strengths of GANs and diffusion models to exploit the best of both worlds is a promising research direction. We will also explore integrating diffusion models into our framework in the future.

Q7: Artificial Single-View Datasets.

R7: Although multi-view datasets of human heads exist, the requirements for capturing multi-view human heads are quite stringent. Tasks based on single-view images are more aligned with real-world scenarios. Therefore, research on single-view images is meaningful and has attracted considerable attention [1][4].

Q8: Comparison with RenderDiffusion.

R8: The denoising process of RenderDiffusion resembles EG3D, mapping latent space to 2D images, while its diffusing process parallels 3D GAN inversion, mapping images to latent space. Both learn 3D-consistent representations from 2D images. RenderDiffusion also performs single-image 3D reconstruction. Although unreleased code prevents direct comparison, the facial reconstruction examples in its paper appear far inferior to our method.

Q9: Limited to Tightly-Cropped Human Heads.

R9: Our method is primarily applied to the task of single-view image reconstruction of human faces. Due to the limitations of the pre-trained EG3D, our model requires aligned and cropped images as input and can only work within the pre-trained domain of EG3D (e.g., human faces and cats). Under this setup, we are able to achieve high-quality novel-view synthesis of human faces. Additionally, we can provide the experiments on cats in the Appendix.

Q10: Two-Stage Inpainting.

R10: Since warped images often contain large areas of holes, we first use 3D GAN inversion to roughly fill in the occlusion regions (as shown in Fig. 2). Although the fidelity of the results from 3D GAN inversion is low (corresponding to entry A in Table 2), it can outline the facial contours in the occlusion regions, thereby alleviating the inpainting burden on SVINet. Given that warped images are somewhat distorted and the initial filling is coarse (corresponding to entry B in Table 2), it is necessary to refine the inpainting in the second stage using SVINet. To clearly demonstrate the role of the first stage, we conducted an ablation study by directly feeding the warped image into SVINet for inpainting. The resulting FID was 19.98 and the ID score was 0.7876, indicating that the initial coarse filling in the first stage is indeed effective.

Q11: Symmetric Prior.

R11: The facial symmetry prior we employ does NOT assume that the input image is symmetric, but rather that the face structure itself is symmetric. By incorporating this prior, we extract information from the visible part of the face to inpaint the face on the other side of the occlusion region. As can be seen from the visualizations we provide, none of our input images are symmetric.

Q12: Blurring Outside of the Face Region.

R12: The input images are real images from CelebA-HQ, but they need to be preprocessed with an alignment and cropping script. Since using the EG3D official preprocessing script to process CelebA-HQ images can result in black borders (as shown in the Triplanenet (WACV'24) paper), we follow the same preprocessing method of HFGI3D. Specifically, we first pad the image edges using the "reflect" mode, then perform blurring and other smoothing operations, and finally conduct alignment and cropping.

References:

[1] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis. CVPR 2024.

[2] RePaint: Inpainting using Denoising Diffusion Probabilistic Models. CVPR 2022.

[3] When StyleGAN Meets Stable Diffusion: a $\mathcal{W}_+$ Adapter for Personalized Image Generation. CVPR 2024.

[4] AvatarArtist: Open-Domain 4D Avatarization. CVPR 2025.

Q1: Novelty.

R1: Existing "warping-and-inpainting" methods cannot be directly used in 3D GAN inversion (lines 40-42). We are the first to successfully apply such a paradigm to 3D GAN inversion. We innovatively design a warping-back strategy and introduce a symmetry prior along with latent codes from a 3D-aware GAN, greatly alleviating multi-view inconsistency. This enables novel-view synthesis of a human face from a single image, achieving performance superior to optimization-based methods with the inference time comparable to encoder-based methods. Hence, our work clearly showcases the great potential of the "warping-and-inpainting" paradigm in 3D GAN inversion. We believe that the fast inference time, together with the good performance of the framework, will benefit and ease future research on novel-view synthesis of human faces.

Q2: Comparisons with SOTA Methods.

R2: Our method is intrinsically different from GenWarp (NeurIPS'24) and LucidDreamer (ICCV'24), which are specifically designed for novel-view synthesis of scenery; in contrast, our work is object-centric, focusing on novel-view synthesis of human faces. Moreover, their training datasets and problem settings differ from ours. Hence, these methods cannot be directly transferred to our task.

We provide an extra comparison with In‑N‑Out (CVPR'24) on the CelebA-HQ dataset. The authors of In‑N‑Out evaluated their method on a custom dataset that includes the mask information marking out-of-domain regions. To make a fair comparison with our method and other baselines, we apply In‑N‑Out to the CelebA-HQ dataset, where it achieves an FID value of 60.41 and an ID value of 0.6744, indicating poor performance. One possible reason is that CelebA-HQ does not have annotations for marking out-of-domain regions. Therefore, during optimization, we use the method (provided by the authors) that does not include the mask in the loss calculation. This may directly cause the additionally optimized tri-plane to fail to reconstruct out-of-domain regions well, leading to a performance drop.

In addition, following the suggestion of Reviewer bp2G, we have also added the performance comparison with a representative diffusion-based method, DiffPortrait3D [1], on the CelebA-HQ dataset. The FID value is 19.26 and the ID value is 0.7964 for DiffPortrait3D, while our WarpGAN achieves an FID value of 19.12 and an ID value of 0.7882. The metrics indicate that our method can achieve performance comparable to diffusion-based methods in the real human face domain.

Moreover, due to the uncertainty in the generation of 2D diffusion models and their lack of an explicit 3D representation, multi-view inconsistency is a common issue. To address this problem, DiffPortrait3D proposes a view-consistency module that processes multiple novel-view images simultaneously to improve the consistency of the generated novel-view images. However, such a way requires a significant amount of GPU memory. The authors of DiffPortrait3D set the number of novel-view images to 8 in implementation, which results in the need for inference on at least a single A100. Due to the limitation on the number of images processed in parallel, flickering artifacts can still be observed in the multi-view rendering animations provided in the DiffPortrait3D demo. In contrast, our WarpGAN, benefiting from the multi-view consistency prior of 3D GANs as well as the symmetry prior and latent code from the 3D-aware GAN that we introduce, can easily generate multi-view consistent rendering videos. We will provide a demo alongside our open-source code.

It is worth noting that our method can train and infer on consumer-grade GPUs (such as the 3090 and 4090), with inference times on a single 4090 GPU within milliseconds. In contrast, DiffPortrait3D requires training on multiple professional-grade GPUs and inference on a single professional-grade GPU, with inference times exceeding 1 minute on a single A100 GPU.

Q3: Limited Evaluation Domain.

R3: We follow state-of-the-art 3D GAN inversion methods (e.g., Triplanenet (WACV'24), In-N-Out (CVPR'24), Dual Encoder (NeurIPS'24)) to validate our method in the human face domain. We agree with you that evaluating our method in other domains is a worthwhile endeavor. Note that EG3D provides a pre-trained model for cats. Compared with faces, which contain rich details, cat reconstruction is relatively simpler. Therefore, we believe our method, in theory, can perform comparably well on cats as it does on faces. We will supplement this evaluation in the Appendix.

References:

[1] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis. CVPR 2024.