Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images

We would like to thank the reviewer for their feedback and for highlighting several strengths of our paper.

1. (W.1) For the head geometry after editing, we realized that the example we share in the paper does not do justice; the reference images have more flat-looking hairs. We added more examples to rebuttal Fig. 7. It can be seen in those examples that the back of the head is as bumpy as the others. We will include more results in the revised paper. Thank you for bringing this to our attention.

2. (W.2) Based on the reviewers' suggestions, we added visualizations of the geometry in rebuttal Fig. 6. Our model outputs better geometry compared to the two best-competing methods. In many examples, PTI struggles; the most obvious is the last row results, where PTI outputs a flat backside of the head. TriplaneNetv2 also struggles with many realistic details and especially on the side and back-views.

3. (Q.1) The reviewer makes a great point. It is true that the reconstruction quality of the unobserved part is bounded by the capacity of Panohead since we rely on its generations while training a discriminator. This is also meaningful since we use the latent space of Panohead for our inversion, and input images can only provide faithful information on the visible parts. We will add this discussion to the revised version.

4. (Q.2) The reviewer asks about the comparisons with optimization-based methods provided by Panohead. Our main paper already included these comparisons as Panohead uses PTI optimization. We observe that PTI sometimes may achieve good results, as given in the Panohead paper; however, the method often fails for the novel views even though it takes significantly longer to run inference. It can also be seen in the geometry in rebuttal Fig. 6. Even though it is still worse than ours, PTI may achieve reasonable results, as shown in the first example. On the other hand, in many examples, it outputs very unrealistic results, as given in the last example in Fig. 6. Similar results exist in the main paper and Supplementary.

5. (L.2) We appreciate the reviewer asking for future directions. Reviewer AnnS mentioned the transformer-based architecture. We also believe using attention techniques can be a valuable research direction to distribute the knowledge of visible regions to others. Additionally, combining this method with diffusion-based techniques is another direction we would like to pursue. We will include these ideas in the revised manuscript to further stimulate research in this area.

We would like to thank the reviewer for their feedback and finding our paper interesting and technically solid.

In the training process, E1 is trained with visible parts by using input-output paired losses of LPIPS, L2, and identity losses. On the other hand, E2 is trained to learn the occluded regions by learning from the discriminator. The occlusion-aware E2 is not initialized with a pre-trained encoder. E2 and D are trained jointly from scratch. We will elaborate on the training procedures. We would like to thank the reviewer for bringing this to our attention.

Additionally, we would like to highlight that Rebuttal Table 1 and Fig. 1 demonstrate our method's significant superiority over competing approaches on the multi-view MEAD dataset evaluation. We also show our method’s ability to generalize across diverse ethnicities, heavy makeup, and extreme camera viewpoints in Rebuttal Fig. 2, 3, and 4, respectively. The geometry of the generated results is detailed in Fig. 6. These findings will be included in the final paper, which we believe will further strengthen our work.

We would like to thank the reviewer for their feedback and highlighting our paper’s several strengths.

1. About the claim on lines 168-169: We acknowledge that not receiving any gradients for the visible region may be theoretically incorrect and will make the necessary modifications to avoid confusion. Thank you for pointing this out. Meanwhile, although the occlusion masks do not 100% eliminate information flow between occluded and visible regions because of the volumetric rendering, we emphasize that our approach performs adequate disentanglement between two regions to produce realistic and coherent samples. This is evident by the occlusion-aware dual encoder combination. Figure 4 in the main paper makes this very visible, too. The dual encoder is able to take the non-occluded parts from Encoder 1 and the rest from Encoder 2. This is also shown by the occlusion-aware / non-occlusion-aware triplane D ablation (Table 2 on main paper, last two rows, specifically the improvements on ID). The occlusion masks can mask a significant portion, and experimental results show they are effective.

We emphasize that this does not hinder the overall claim of our work. Still, they must be acknowledged in the main paper to avoid confusion. Again, we thank the reviewer for such valuable feedback and will improve the relevant sections.

Answers to the questions:

1. We extract the depth maps from the NeRF model.
2. The figure has an unfortunate typo, and the E1 and E2 labels need to be switched. Thank you for your attention.
3. The competitive methods are based on PanoHead’s generator backbone; hence, they are compared against the same set of images synthesized by the same method. We will make this more clear in the revised paper. The reason we use these synthesized images is to ensure comprehensive coverage of all angles in our evaluation.

In the rebuttal, we also provide evaluations on the multi-view MEAD dataset as suggested by Reviewer AnnS. In this setting, we use paired images, where the input is at one angle, and the output is $\pm$60 degrees rotated. In this setting, for the FID evaluation, we use the ground-truth images. These results are given in rebuttal Table 1 and Fig. 1, which will be added to the main paper.

We would like to thank the reviewer for their feedback and for considering our rebuttal. We appreciate their willingness to discuss our responses with the other reviewers.

We would like to thank the reviewer for their feedback and suggestions, which further helped us improve the paper.

1. (W.1) The reviewer wanted more visual results from extreme viewpoints. We did not notice the visual results from the paper were all more from the frontal view. We apologize for that. LPFF dataset is a large-pose Flickr dataset; therefore, our quantitative results included such evaluation. In rebuttal Fig.4, we also provide visual images with challenging viewspoints. We will incorporate more examples into our final paper. Thank you for bringing this to our attention.

2. (W.2) Based on the reviewer’s suggestions, we include heavy make-up input results in rebuttal Fig. 3; additionally, diverse ethnicities are represented in Fig. 2 as suggested by Reviewer 9tZp.

3. (W.3) Thank you for the neat idea. We have generated a multi-view dataset from videos in MEAD (https://github.com/uniBruce/Mead) and quantitatively evaluated ours and competing methods. Specifically, we fed front MEAD images (0 yaw degree) to all methods, and rendered them from novel views of MEAD (from $\pm$60 to 0 yaw degrees). We compared them with corresponding ground truths. Our method performs better quantitatively for novel views (rebuttal Table 1 and Fig. 1). Figure 1 shows ground-truth input and output pairs from MEAD dataset. These experiments will be added to the revised paper.

4. (Q.2) About the choice of CNN versus transformers, in our experiments, TriplaneNetv2 (CNN-based) and GOAE (transformer-based) inversion methods performed similarly when evaluated on challenging samples. That being said, self and cross-attention-based approaches to capture similarities may be a good idea to investigate further. We thank the reviewer for encouraging further study directions.

5. (L.1,2) Regarding the limitations, in Rebuttal Fig. 5, we find that our method struggles when input images have head accessories and are treated as hair from the back-view. We will add those additional analyses to our final manuscript. Symbols in formulas and figures will be rewritten with clearer notation.

We would like to thank the reviewer for their feedback, and we will improve the writing further based on the reviewer’s suggestion. The reviewer mentions that the innovation of the paper may be a little lacking, and there is a lack of in-depth research on the proposed method. However, it is crucial to note that our study addresses the novel challenge of GAN inversion for 360-degree GANs, a topic that has not been previously explored. We demonstrate that existing methods fail to address the intricacies of this problem. The dual-encoder design coupled with an occlusion-aware triplane discriminator proves to be intuitive, and the ablation study effectively demonstrates the impact of each design choice, offering valuable insights, as noted by other Reviewers (AnnS, jtFz, Dd7h, XyTr). Furthermore, many triplane-based models have been emerging lately [Yang et al., Dreamcomposer, CVPR 2024], and the occlusion-aware discriminator approach may enable a new, unsupervised training direction for generating coherent 3D samples from single images. This is also appreciated by Reviewer jtFz (Strength-3).

About limitations and failure cases; we have tested different scenarios. Firstly, as suggested by Reviewer AnnS, to include multi-view evaluation, we set up the MEAD dataset, which also includes more diverse ethnicities than the CelebA dataset, as shown in rebuttal Fig. 2. We show that the method also handles heavy make-up inputs, as given in Fig. 3. We also visualize results when the input views are extreme side-views, given in Fig. 4. Finally, in Fig. 5, we find that our method struggles when input images have head accessories and treats them as hair from the back-view. We will add those additional analyses to our final manuscript.

Answers to the questions:

1. In our experiments, removing Encoder 1 resulted in a significant decrease in the ID scores, as given in Table 4. Encoder 1 provides high fidelity to the input, a must in reconstruction. On the other hand, Encoder 2 excels in generating better representations of invisible regions. Thus, we observed a trade-off between these two critical requirements and found the best-achieving method to be a dual-framework. That said, we provide analysis, methods, and baselines for this new and challenging problem, which can pave the way for more elegant frameworks.

2. We combined FFHQ and LPFF for training (140k) and test (14k) sets and applied pose rebalancing proposed in LPFF. We also used the same dataset to train competitive methods for fair evaluation. We will detail the relevant section and elaborate more on the Supplementary.

3. Our method achieves 3D reconstruction on diverse ethnicities, as included in rebuttal Fig.2. More visual results will be added to the main paper and the Supplementary. Thank you for the suggestion.

4. The head information is implicitly encoded in PanoHead’s generator backbone, which we keep frozen. Following the reviewer's suggestion, we examined how Encoder 2 performs when given a silhouette of a face as input, as shown in rebuttal Fig. 8. Our observations confirm that, even with this out-of-domain input, Encoder 2 generates latent representations that produce a realistic head shape. This indicates that Encoder 2 retains an implicit representation. We appreciate the reviewer’s insightful suggestion and will include this finding in the revised paper.

I have read the author's response, as well as the comments and discussions with other reviewers. The author has addressed my concerns. I will improve my score.