Instax3D: Creating 3D Portrait from a single-view image in Minutes

1、Data is the upper bound of any algorithms. The author says that they utilize a pre-trained 3D-ware generator Panohead (An et al., 2023) to produce 1, 000 synthetic turn-around videos for training. We know that the turning around videos generated by 3D-Aware GAN-based methods sometimes may not be that 3D-consistent. I wonder whether the author manually filter out those samples which is distorted? Was 1000 ids sufficient for algorithm training to make it generalize well? I'd like to see more cases.

2、Image-to-3D methods like Wonder3D、ImageDream was not trained on specific human head dataset. It will be more fair to finetune them on your specific dataset.

3、Some talking head methods can also turn around the face, and drive the face with any facial animation. However, the proposed method can only produce a static head model. I can not figure out the strength of the proposal when compared with the following works. Since in most application cases, we only need to see the front face of the virtual portrait.

1. Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis
2. X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention

• Novelty of the Pipeline: This paper presents a "generation-then-reconstruction" pipeline for single-view 3D portrait generation. However, several existing works, such as CAT3D [1], ReconX [2], ViewCrafter [3], and MagicMan [4], have explored similar schemes across various domains, including single-view 3D object, scene, and human generation. Furthermore, initializing with a parametric mesh is a common practice for 3D Gaussian initialization. For instance, methods like HumanGaussian [5] and DreamWaltz [6] use the SMPL-X template for this purpose.

• Unsatisfactory Quantitative Results and Incomplete Baselines: The authors selected baselines such as LGM, LRM, ImageDream, IPAdapter, and Magic123, but overlooked critical baselines relevant to human head avatar generation. Both feed-forward methods (LRM/LGM) and multi-view generators (Wonder3D, ImageDream, Magic123) were initially designed for general 3D object generation, relying on Objaverse for training. Finetuning these models on the portrait dataset is essential if the authors intend to compare their method against these general-purpose approaches. Additionally, for portrait generation, Rodin [7] and RodinHD [8] could serve as valuable baselines. The qualitative results presented in the video demo do not appear to match the quality of RodinHD.

• More: I do not believe the pixel-wise optimization formulation is the main reason for efficiency. Moreover, SDS-based methods are not inherently slow if the 3D representation is built on 3D-GS, as evidenced by GaussianDreamer [9] and DreamGaussian [10]. A major concern is the quality associated with SDS-based methods, particularly issues like texture oversaturation. The challenge of reconstructing from generated multi-view images is also noteworthy, given the potential for multi-view inconsistencies.

[1] Gao et al., CAT3D: Create Anything in 3D with Multi-View Diffusion Models, arXiv 2024.

[2] Liu et al., ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model, arXiv 2024.

[3] Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis, arXiv 2024.

[4] MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement, arXiv 2024.

[5] Liu et al., Text-Driven 3D Human Generation with Gaussian Splatting, CVPR 2024.

[6] Huang et al., Make a Scene with Complex 3D Animatable Avatars, NeurIPS 2023.

[7] Wang et al., Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion, CVPR 2023.

[8] Zhang et al., High-Fidelity 3D Avatar Generation with Diffusion Models, ECCV 2024.

[9] Yi et al., GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models, CVPR 2024.

[10] Tang et al., DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation, ICLR 2024.

1. Although the authors wish to generate multi-view images from a video diffusion model and subsequently apply a multi-view reconstruction method to create a static human avatar from the generated images, there is a core issue that they do not address. The main problem is that the images produced by the multi-view generation model are view inconsistent. How can one reconstruct a static Gaussian portrait from multi-view inconsistent images? More importantly, what do the outputs of your video diffusion model look like? Are the rendering results of your 3D Gaussian model overfitting the video image output? Please show me more novel view synthesis results.

2. The novelty is limited. The author straightforwardly fine-tunes a video diffusion model using the generated head model. This strategy is somewhat Uncanny. Does it imply that the upper bound of the generated multi-view image is the rendered view of the generated portrait model? Additionally, we could not find any interesting designs in the multi-view 3DGS reconstruction.

3. Their results do not capture my attention. Honestly, the quality of the generated heads is not satisfactory.

4. The comparison results are unfair. The authors employ a pre-trained 3D-aware generator to produce 1,000 synthetic self-rotating videos to fine-tune their multi-view video model. For a fairer comparison, I suggest that the authors fine-tune the authors' baseline models, such as Wonder3D and LGM, to obtain their results instead of simply resuming their weights to generate their results.

The paper has the following shortcomings: Effectiveness: While it is true that the proposed method outperforms techniques like LRM in image reconstruction, it remains unclear whether the improved results stem from the use of SVD rather than the reconstruction algorithm itself.

Shape and Texture Issues: Observations from the 360-view rendering on the webpage show that the side-view renderings have significant shape inaccuracies. Additionally, the textures in novel views lack realism.

Lack of Comparisons: The paper does not include comparisons with 3D GAN-based methods such as EG3D, Next3D, LP3D(Live 3D Portrait), and Panohead, which could provide valuable context for its findings.

• Video Diffusion Model

The quality of the 3D reconstruction produced by Instax3D is inherently limited by the quality of the synthesized 360-degree selfie generated by the video diffusion model. Any artifacts or inaccuracies in this initial synthetic sequence are carried over into the final 3D reconstruction, which may impact the realism and fidelity of the result. While the authors claim that the 3D reconstruction benefits from the identity and shape priors of the video diffusion model, they do not provide any examples of the synthesized 360-degree video. Including sample views for each column shown in Figure 2 would better illustrate the effectiveness of these priors and the quality of the synthetic sequence generated.

• Comparisons

In the evaluation section, the paper primarily compares Instax3D with recent generative models for 3D generation but lacks comparisons with methods focused specifically on portrait reconstruction. Given that the video generation model produces a 360-degree video, Instax3D could also be evaluated against fully optimization-based models or fine-tuned 3D GAN-based methods for synthesizing novel views. Including these comparisons would provide a clearer perspective on Instax3D’s relative performance and help determine whether its major performance gains come from the synthesized 360 video or from the 3D Gaussian reconstruction.

• Convexity in side-views

In several examples shown on the project page, the convexity of the reconstructed head appears inaccurate, especially in side views where the face can look concave rather than convex. This issue is not present in the FLAME model, suggesting that it may come from the 3D Gaussian optimization process. To better understand the source of this issue, it would be helpful to include side-view portraits from the synthesized 360-degree video, providing additional insight into whether the problem originates in the video generation or during the optimization stage.