WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space

Thank you for reviewing our work. We appreciate that you consider the soundness, presentation, and contribution of our work “good” (hence, we were surprised by the overall “rejection” rating despite these good scores on all three subcategories). We address your concerns below.

For the first stage of 3D-aware VAE, there are previous methods that propose very similar method... I would expect authors to give a more detailed explanation of how the proposed method differs from these methods (VQ3D, GINA-3D).

Our initial submission includes a discussion on VQ3D in Appendix A. The main difference to VQ3D is that their autoencoder yields sequence-like latent variables on which an autoregressive transformer is trained while we train a latent diffusion model on feature maps. We extended the discussion of VQ3D in the Appendix of our paper and will move this part of the discussion to the main paper in the camera-ready version. Moreover, it is worth mentioning that VQ3D is a ICCV’23 paper, which took place in early October. According to the ICLR reviewer guidelines (https://iclr.cc/Conferences/2024/ReviewerGuide, Reviewer FAQ, last Q) it is therefore considered concurrent work (anything published at a peer-reviewed venue on or after May 28, 2023 is concurrent).

According to our understanding, GINA-3D assumes known camera and LIDAR sensors, which are not given in our setting. Hence, GINA-3D would not be directly applicable in its current form. Also architecturally GINA-3D differs from WildFusion in multiple ways. For instance, it uses a vision transformer encoder and defines its latent space where ta discrete token generative model is trained directly on triplanes, while WildFusion trains a diffusion model on 2D spatial feature maps. GINA-3D only applies losses to the reconstructed image, while our approach uses a discriminator to get guidance on novel views. We included GINA-3D and a discussion in Appendix A in the extended related work section.

To continue on the above comment, I would also like to see the performance comparison with VQ3D on both VAE and generator

As discussed above, the concurrent VQ3D was only recently published at ICCV’23. Moreover, it does not have any code available for a comparison and a faithful reimplementation is far beyond the scope of this rebuttal. A meaningful comparison to VQ3D is hence impossible and without available code or checkpoints we also cannot evaluate it on all relevant metrics (FID, FID_CLIP, Precision, Recall, NFS).

I would recommend using 50k.

We evaluated FID50k for WildFusion on ImageNet, which is 25.3. We will include this number in the main paper in the final version.

Visual comparison with other methods is not enough.

Our initial submission includes 2 figures (Figure 1 and 2) that illustrate the mode collapse for the strongest GAN-based baseline, 3DGP. We hope that it is understandable that with a limit of 9 pages in total, we cannot add visual comparisons to all baselines in the main paper.

However, we added many more samples of the strongest 3DGP baseline in Appendix D in Figure 14 for more classes. The other GAN-based baselines are known to perform significantly worse than 3DGP (see 3DGP paper, as well as many samples and comparisons on their project page, https://snap-research.github.io/3dgp/ and https://u2wjb9xxz9q.github.io/), hence we did not bother to add even more samples in our paper. However, the quantitative results speak for themselves: Both EG3D and StyleNeRF have very low recall scores (Table 3), indicating similar mode collapse and lack of sample diversity.

360 degree novel view synthesis

In theory we could train WildFusion with more drastic camera angle changes, potentially full 360 degree. However, not all ImageNet classes even show objects from all different angles. IVID can likely also only achieve this for some specific classes, and with limited quality and a high inference time compute cost. Consequently, we focussed on the most common setting of most previous work (like the important 3DGP baseline, or EG3D and StyleNeRF), which tackles 3D-aware image generation within a limited camera angle range (IVID is a concurrent work, also published at ICCV’23, with a very different approach, after all. Also note that IVID is discussed in Appendix A). That said, we agree with you that extending WildFusion to full 360 degree 3D synthesis would be an exciting avenue for future work.

We hope that we were able to address your concerns and questions and are happy to discuss should you have remaining questions. If you feel that our rebuttal sufficiently addresses your main concerns, we would like to kindly ask you to consider raising your score accordingly.

We would like to thank you for your positive review of our work (“excellent exposure and clarity”, “well-motivated”, “appropriate baselines”, “clear design choices”, “insightful and detailed ablations”). We also appreciate your constructive feedback with respect to the additional experiment you suggested.

We implemented and ran the additional baseline experiment you suggested, building on top of the public implementation of SceneScape (https://github.com/RafailFridman/SceneScape.git).

Discussion: We generate images using the ImageNet checkpoint from LDM (https://github.com/CompVis/latent-diffusion.git). Next, we predict the corresponding depth using ZoeDepth, i.e. using the same pretrained depth estimator as in our approach, and warp the image to a novel view. Lastly, an inpainting model fills the holes that result from warping. We use an inpainting variant of Stable Diffusion (https://huggingface.co/docs/diffusers/using-diffusers/inpaint#stable-diffusion-inpainting) and provide the warped image, its mask, and the text prompt “a photo of a <class name>” as input.

For quantitative analysis, we sample novel views similar to our evaluation by sampling yaw and pitch angles from Gaussian distributions with $\sigma = 0.3$ and $0.15$ radians, using the depth at the center of the image to define the rotation center. With this approach, we get an FID of 12.3 on ImageNet, compared to 35.4 for WildFusion. However, as discussed in the main paper, FID only measures image quality and does not consider all important aspects of 3D-aware image synthesis, e.g. 3D consistency. In fact, we observe that the inpainting often adds texture-like artifacts or more instances of the given ImageNet class. We provide some examples in Appendix D, Figure 20.

To enforce consistency between novel views, we follow your suggestion and run the full pipeline of SceneScape to fuse the mesh for multiple generated images. For this setting, we sample 9 camera poses by combining yaw angles of $[-0.3, 0., 0.3]$ and pitch angles of $[-0.15, 0., 0.15]$ and iteratively update the mesh by finetuning the depth estimator. We show the final meshes in Appendix D, Figure 20 (bottom two rows). For all samples we evaluated, we observe severe intersections in the mesh and generally inferior geometry to our approach. We remark that SceneScape's test time optimization takes multiple minutes per sample and a large-scale quantitative evaluation was out of the scope of this rebuttal.

Our rotating camera movements around a single object are much more challenging, e.g. due to larger occlusions, than the backward motion shown in SceneScape. We hypothesize that this causes SceneScape to struggle more in our setting.

We included this discussion in the appendix of our paper, at the very end (Appendix D). For the camera-ready version, we may move this to earlier and add pointers to this experiment in the main paper. We hope that we were able to address your concerns. If you have further questions about the implementation of the baseline or any remaining concerns, we are happy to discuss. Otherwise, we would like to kindly ask you to consider raising your score accordingly.

Thank you for reading and reviewing our work. We appreciate that you consider our work well-written and our approach interesting and reasonable. We address your questions in the following.

An important work (VQ3D) is missing in discussion/comparison.

Our initial submission includes a discussion on VQ3D in Appendix A. The main difference to VQ3D is that their autoencoder yields sequence-like latent variables on which an autoregressive transformer is trained while we train a latent diffusion model on feature maps. Another difference is that VQ3D applies two discriminators on the generated images, one that distinguishes between reconstruction and training image, and another one that discriminates between reconstruction and novel view. We only apply a single discriminator to supervise the novel views and instead have an additional discriminator on the depth. We would like to note that VQ3D was published at ICCV in early October this year and is therefore concurrent work with ours (https://iclr.cc/Conferences/2024/ReviewerGuide, Reviewer FAQ, last Q: anything published at a peer-reviewed venue on or after May 28, 2023 is concurrent) and has no code available for comparison. We extended the discussion of VQ3D in the Appendix of our paper. For the camera-ready version of the paper, we will move this part of the discussion of concurrent work from the Appendix to the main paper.

I did not observe a significant visual improvement over the EG3D

The strength of diffusion models over GANs becomes most apparent on large, diverse datasets, where GANs often struggle with diversity, coverage of the data distribution and mode collapse. On ImageNet, we clearly outperform EG3D by a very large margin, which is best highlighted in our quantitative metrics (FID: 35.4(ours) vs 111.6(EG3D), recall: 0.19(ours) vs 0.01(EG3D) – this low recall score of 0.01 indicates mode collapse).

So what is the benefit of introducing the second stage? Easy sampling?

The benefit of introducing the second stage is to be able to use a diffusion model as a generative model. Diffusion models have shown much better coverage and training stability on large, diverse datasets than GANs, which is supported by the recall scores in Table 3. Also note that the first stage autoencoder cannot generate diverse, novel samples on its own, as it always relies on input images for encoding. With the diffusion model in latent space, however, we can generate novel samples from scratch.

Strictly speaking, the training involves a large amount of 3D data, which comes from the pre-trained single view depth estimator.

Yes, we agree. This is why in the abstract we wrote, “without any direct supervision from multiview images or 3D geometry”. We did indeed not use direct 3D supervision, but only leveraged pre-trained depth estimators. We will add the following sentence to the introduction in the camera ready version to further clarify this:

To prevent generating flat 3D representations, we leverage cues from monocular depth prediction. While monocular depth estimators are typically trained with multi-view data, we leverage an off-the-shelf pretrained model, such that our approach does not require any direct multi-view supervision for training.

We remark that such pre-trained depth estimation models are today commonly available and do not represent a major limitation of our method.

We hope that we were able to address your questions and concerns. If so, we would like to kindly ask you to consider raising your score accordingly.

Thank you for reading and reviewing our work and for appreciating our novel approach. While we agree that a comparison with recent diffusion-based methods is interesting, at time of submission no code was available for GenVS, IVID and VQ3D. Also please note that GenVS requires multi-view data for training, which is not available in our setting, so a direct comparison is not possible. Moreover, it is worth mentioning that IVID and VQ3D are ICCV’23 papers, which took place in early October. According to the ICLR reviewer guidelines (https://iclr.cc/Conferences/2024/ReviewerGuide, Reviewer FAQ, last Q) these two papers are therefore considered concurrent work (anything published at a peer-reviewed venue on or after May 28, 2023 is concurrent).