Denoising Diffusion via Image-Based Rendering

Two type of baselines is lacked for 3D generation / reconstruction: (1) Optimization-based pipeline with diffusion model as priors (2) Diffusion models trained on purely 3D data

We have already made significant extensions to state-of-the-art works (e.g. RenderDiffusion) to make them support our challenging in-the-wild setting. Nevertheless, we agree it is valuable to include an “Optimization-based pipeline with diffusion model as priors”. We therefore now include results from SparseFusion on the CO3D Hydrant dataset (we will add results on other datasets soon). We see that SparseFusion performs significantly worse than our approach, and is also far slower than ours. Specifically, SparseFusion achieves PSNR of 12.06 (vs 16.07 for ours) and LPIPS of 0.63 (vs 0.456 ours). As we explain in sec. 2, this is because score-distillation is not a good objective for reconstruction due to its mode-seeking behaviour – this fails when the conditioning image does not align with the dominant mode. We also already include an experiment with generating multi-view images using only 2D multi-view diffusion. However, in contrast to Zero-1-to-3, which samples multi-view images independently, we train this 2D baseline to sample multi-view images jointly, using cross-view attention so the model is expressive enough to learn 3D consistency. Our experiments in Table 3 (row “No 3D”) demonstrate that resulting reconstructed images score poorly according to PSNR & SSIM, since they fail to show a coherent and accurate 3D scene (LPIPS is slightly better since it is less sensitive to fine details in the images).

Regarding Rodin as an additional baseline, we have already included an experiment with triplanes instead of our IB-planes representation (see “No image-based representation” paragraph in A.1). Note that Rodin itself would require very substantial modifications and extensions to work in our setting. In particular, Rodin is designed to work with 1) masked objects 2) a bounded scene volume 3) canonically aligned camera poses, e.g. all faces in the dataset point north in a shared coordinate frame 4) uniform scales (i.e. facial features of all people in the dataset are of nearly identical size). These assumptions are reflected in their use of canonically-aligned triplanes (in world-space) as a 3D representation; we already note (in App. D) that this representation performs very poorly in our more challenging setting. In contrast, our in-the-wild datasets (e.g. MVImgnet) contain arbitrary scene scales and orientations, diverse camera poses, and large-scale outdoor scenes. Nevertheless, if you still feel that we should include a baseline of this kind (regardless of the very different setting and of our existing “No image-based representation” experiment results), we are happy to implement and test a suitable variant of our model (since there is no public implementation of Rodin) if you could let us know what you would consider a fair comparison:

• Which representation to use – e.g. triplanes in canonical world-space orientation or in view-space?
• How to condition the diffusion model on multi-view images and their camera poses?
• How to rescale the scenes (which have arbitrary scales from COLMAP) so that they are modelled by a fixed-size triplane/voxel-grid?

Missing baseline qualitative results in the supplementary

Thank you for noting this omission! Today, we will include visualisations in the supplementary material for pixelNeRF, RenderDiffusion and Viewset-Diffusion. We will also release corresponding video visualisations on the project website upon acceptance.

“inference results do not provide input images.”

We do already show the input images in fig. 3, in the leftmost column titled "conditioning". However, we will add an explanation in the caption.

Is the (polar) feature map shared across all views? How to use the angles to interpolate this feature map?

No, one feature map is predicted for each view, as a second output from the U-Net, in the same way as we predict feature planes in each view frustum. This can be thought as warping a 2D feature plane into a sphere using equirectangular projection, i.e. we use the (normalised) polar coordinates theta and phi to choose where to take these features from.

How does performance vary w.r.t number of views?

We already include detailed experiments to answer this question. On one hand, increasing the number of noisy views during training leads to better performance across both generation and reconstruction metrics (see Table 4 and sec. A.2). However, increasing the number of views has diminishing returns. Specifically, increasing from 3 to 5 improves FID by 58.4, while increasing from 6 to 8 improves FID by 20.0. On the other hand, increasing the number of conditioning views during test-time reconstruction naturally increases the fidelity of the scene as there is less uncertainty. Specifically, increasing the number of input views from 1 to 6 increases the PSNR from 18.54 to ​​22.09 on MVImgNet (see Table 1).

Real multi-view datasets usually contain camera calibration errors (as they are usually estimated via some algorithm). How does the robustness of proposed method w.r.t camera calibrations?

Indeed, 4 out of our 5 datasets contain significant camera calibration errors. For example, the creators of CO3D state that around 18% of camera poses are inaccurate. We do not filter these as we can see that our model is robust to these errors. However, we do notice increased quality for more accurate camera poses (e.g. classes with accurate camera poses perform better than without; however, we have not quantified this error as it is difficult to come up with a measure).

It seems the experiment directly using 2D U-Net to output tri-plane features. Yet, as argued in [Q1], there exists an incompatibility between the tri-plane representation and the naive 2D U-Net method. In [Q1] a 3D-aware convolution for output tri-plane features from 2D U-Net is proposed and achieved good quality. How would this variant fit into the proposed method in this paper?

Thanks for pointing this out. We too conducted some initial experiments with a more sophisticated unprojection operation. However, we observed that our IB-planes representation performed well without the additional compute and memory requirements for a 3D-aware convolution. We concluded that such techniques are only needed when the mapping from image pixels into triplane features is difficult (as also shown by recent works), which is not the case for our IB-planes representation as it instead defines and exploits a simple mapping from the image to IBR features, which is both fast and expressive.

Thank you for the detailed and constructive review. Below and in the updated paper, we address your concerns and suggestions.

The proposed representation of 3D scenes is not totally new.

Though image-based rendering has been studied in computer graphics, our approach makes three new technical contributions (which we now describe in section 3.1). The first contribution (as you noted) is joint prediction of IBR features across images, via the multi-view-attentive U-Net. This is strictly more expressive than prior IBR approaches (pixelNeRF, IBRNet, FVS) that calculate features independently for each image – our multi-view U-Net can arrange different IBR features for the i-th viewpoint depending on other input images. Specifically, it can remove the ambiguity of the 3D scene that is present when given only one image (e.g. pixelNeRF, IBRNet, FVS). Prior approaches (e.g. pixelNeRF, IBRNet, FVS) had to alleviate this problem by having a large (hence expensive) model to fuse features (e.g. FVS has RNN as you mentioned whilst IBRNet has a deep attention network over point features and nearby 3D points in contrast to our simple max-pooling). Intuitively, our representation may be seen as an improvement of K-planes (https://sarafridov.github.io/K-Planes/) as (i) we arrange by placing them at camera position instead of canonical position and (ii) projecting to them using camera intrinsics instead of perpendicular projection. We have now named our representation "IB-planes" to better highlight this distinction.

Second, we use a learnable embedding for sample depths, by projecting the depth of a sample point wrt each view to a learnable feature vector (in contrast to embedding global canonical position with fourier positional encoding as PixelNeRF). This was mentioned very briefly in sec. 3.1, but we will expand on this point. It allows the model to better express different features for all points along the ray, yet without requiring scenes to have a canonical orientation (as depths are relative to the camera).

Third, we also define features even outside of all camera-view frustums, by including "spherical features" (i.e. polar unprojection of all points into an additional feature map for each camera). This significantly improves FID at heldout views from 126.5 to 118.1 (see “No spherical” experiment in Table 3).

Together, this means our representation is more expressive than existing methods, and can represent complex in-the-wild scenes, including outdoors (see figure 4), and allows incorporating information from multi-view images for IBR-based rendering. Our experiments with triplanes (“No image-based representation” in table 3), pixelNeRF representation and ablations (e.g. w/o multi-view attention in table 3), demonstrate the superiority of our representation.

However, the most important technical contribution of our work is that it is the first principled approach for integrating image-based rendering of 3D scenes into diffusion models. Mathematically, the diffusion formulation requires that all dimensions (e.g. pixels) of a noisy datapoint must be denoised by the model; changing any noise variable then correspondingly changes the sample. However, the difficulty of using diffusion with an IBR representation is that using all IBR frames, including of the (noisy) i-th image when rendering (i.e. denoising) the i-th camera pose is that it leads to pathological 3D solutions (e.g. flat planes in front of the camera). We demonstrate this with an experiment in Table 3 (“No repr. drop”) where we see that the DRC (depth rank correlation) metric worsens from 0.821 to 0.586. One naive approach to avoid trivial 3D solutions is to use held-out views for supervision. However, this approach is mathematically incorrect as the uncertainty of (or latent information about) held-out images has no connection to the noise distribution. In practice, this prevents the diffusion model from sampling plausible details in these withheld views, instead merely approximating the average observation, much like older non-generative techniques. In contrast, our model can use dropout without this limitation as the noise from each view can influence all latent 3D scene representation via multi-view attention in the UNet. We add this explanation to section 3.4.

We do already show the input images in fig. 3, in the leftmost column titled "conditioning". However, we will add an explanation in the caption.

Thanks for the clarification.

No, one feature map is predicted for each view, as a second output from the U-Net, in the same way as we predict feature planes in each view frustum. This can be thought as warping a 2D feature plane into a sphere using equirectangular projection, i.e. we use the (normalised) polar coordinates theta and phi to choose where to take these features from.

Does it mean that the features are re-sampled by assuming it is on an infinite-far plane?

We already include detailed experiments to answer this question (performance vary w.r.t number of views).

Now I see the Table 4 and Sec A.2. Thanks for the clarification.

For example, the creators of CO3D state that around 18% of camera poses are inaccurate. We do not filter these as we can see that our model is robust to these errors.

Thanks for the clarification.

The multi-view version of diffusion models inputs the camera poses as embeddings. If allowing the arbitrary viewpoints to reconstruction, I thought it is not enough just to input (and map) camera matrices to achieve reasonable multi-view conditioning on diffusion models. […] Do they use any strong assumptions, e.g., camera locations are almost the same or less variety for training datasets?

Indeed, our experiments (called “No 3D” in Table 3) show that it is not sufficient to condition the diffusion model on camera poses without geometric priors. Hence, one of our technical contributions is to resolve this by leveraging the geometric correspondence between image-space features and 3D points. We have now emphasised this more in section 3.1 and 3.2. We do not make any strong assumptions on the diversity of camera poses in the training set – our model is trained on challenging real-world collections of images (e.g. MVImgNet), without using any masks or depths; this we now emphasise at the start of experiments section.

Also related to the above, a potential drawback of using diffusion models for 3D reconstruction may be that they change the content of given multi-view images. I could see some discussions (and potential failure cases, if any).

One of the major strengths of our work is that the model faithfully represents the input/conditioning images -- this is ensured by our neural 3D scene representation (section 3.1) which projects rendered 3D points to all images to query image features, which are then fused to output radiances. This is in contrast to prior works, such as RenderDiffusion, which requires black-box CNN to map to 3D features. We now emphasised this point further in section 3.1.

The contribution statement is not clear: "first denoising diffusion model that can generate and reconstruct large-scale and detailed 3D scenes." Which part is actually the "first"? Which adjective corresponds to which technical part? For example, readers may think RenderDiffusion met the above stuff. Related to the above, it is pretty helpful to state the technical difference and practical merit compared with recent methods like RenderDiffusion and PixelNeRF.

Thanks, we have now clarified these points (including our contributions) in sec. 1 & 2. Specifically, we refer to the first method capable of reconstructing large in-the-wild 3D scenes from a single image. As we explain in the Related Work section, prior work, including pixelNeRF, cannot achieve plausible reconstruction in unobserved/ambiguous regions as this requires a generative capability that they lack. On the other hand, generative models, e.g. RenderDiffusion, also had some major limiting assumptions that prevented them from achieving this task. For example RenderDiffusion has a limited 3D representation that can only represent objects but not scenes and requires canonically oriented camera poses only available in synthetic datasets. This statement is verified by reconstruction experiments in 4.1, which show very significant improvement over all prior works. We now include specific contributions in the introduction as well as specify additional technical contributions in sections 3.1 and 3.2.

As the paper mentions, the proposed 3D representation is similar to PixelNeRF. To the 3D representation, which part is the key technical difference from PixelNeRF?

Thanks to this question, we have now rewrote section 3.1 to include specific technical contributions. To summarise, our approach makes three new technical contributions.

The first contribution is joint prediction of IBR features across images, via the multi-view-attentive U-Net. This is strictly more expressive than prior IBR approaches (pixelNeRF, IBRNet) that calculate features independently for each image – our multi-view U-Net can arrange different IBR features for the i-th viewpoint depending on other input images. Specifically, it can remove the ambiguity of the 3D scene that is present when given only one image (e.g. pixelNeRF, IBRNet). Prior approaches (e.g. pixelNeRF, IBRNet) had to alleviate this problem by having a large (hence expensive) model to fuse features (e.g. IBRNet has a deep attention network over point features and nearby 3D points in contrast to our simple max-pooling). Intuitively, our representation may be seen as an improvement of K-planes (https://sarafridov.github.io/K-Planes/) as (i) we arrange by placing them at camera position instead of canonical position and (ii) projecting to them using camera intrinsics instead of perpendicular projection. We have now named our representation "IB-planes" to better highlight this distinction.

Second, we use a learnable embedding for sample depths, by projecting the depth of a sample point wrt each view to a learnable feature vector (in contrast to embedding global canonical position with fourier positional encoding as PixelNeRF). This was mentioned very briefly in sec. 3.1, but we will expand on this point. It allows the model to better express different features for all points along the ray, without requiring scenes to have a canonical orientation (as depths are relative to the camera).

Third, we also define features even outside of all camera-view frustums, by including "spherical features" (i.e. polar unprojection of all points into an additional feature map for each camera). This significantly improves FID at heldout views from 126.5 to 118.1 (see “No spherical” experiment in Table 3).

Together, this means our representation is more expressive than existing methods, and can represent complex in-the-wild scenes, including outdoors (see figure 4), and allows incorporating information from multi-view images for IBR-based rendering. Our experiments with triplanes (“No image-based representation” in table 3), pixelNeRF representation and ablations (e.g. w/o multi-view attention in table 3), demonstrate the superiority of our representation.

We are grateful for your constructive feedback, thanks to which we improved the clarity of the paper by incorporating your suggestions and performing additional experiments. Below we address one misunderstanding (and underestimation) of our work and answer your other questions.

the proposed method basically targets the multi-view contexts. For multi-view 3D reconstruction, usual MVS-based methods (or neural-MVS-based methods, perhaps) may still achieve much better performance for large-scale and detailed reconstruction.

This is incorrect, we specifically target the more challenging tasks of (i) unconditional 3D generation (ii) 3D reconstruction from a single view and (iii) 3D reconstruction from very few (6) views. Even during training we only require at most 8 views per scene. In contrast, MVS-based methods (and NeRFs) typically require more than 20 images. Another crucial disadvantage of MVS-based methods is that they are not generative, so they cannot generate plausible details in parts of the scene that are unobserved (e.g. due to occlusion). In contrast, our approach can sample multiple plausible completions of unobserved regions, that are compatible with the input image. We now added further clarifications in sec. 1 (1st paragraph) and sec. 2 (1st paragraph).

A fundamental drawback of using multi-view-consistent generative models for multi-view 3D reconstruction may be in the limitation of input image resolution (due to the size of graphics memory), as well as the generative models change the image content that can degrade the faithfulness of the 3D reconstruction.

Regarding faithfulness of the reconstruction, a particular benefit of our IBR-based scene representation is that it can preserve fine details in image-space, avoiding a mapping to a smaller latent space that might indeed remove such details. This is evidence in fig. 3, where very fine details (e.g. narrow chair legs) are preserved in the reconstructions. Regarding resolution, our model estimates a continuous 3D scene, and we can render its samples at any resolution. Indeed, we have now experimented with rendering reconstructions at 1024x1024 resolution. We see that it performs well, predicting additional details not visible in the input 96x96 image. Hence, our model can also perform superresolution by first predicting a 3D asset from a conditioning image and then rendering the 3D at a higher resolution. It is noteworthy that this is the highest resolution supported by any generative 3D work to date; we now state this contribution explicitly in the introduction.

a potential merit of using generative models for 3D reconstruction is the occlusion recovery, as written in the introduction. In such senses, I could see experiments assessing the results of 3D reconstruction under severe occlusions.

Again, we already report extensive experiments on single-image 3D reconstruction, where the model is tasked to generate occluded regions, notably the reverse of the object and volumes outside of the input image (both visible in 360deg spin views around the object). We always see that our model samples plausible 3D completions of occluded volumes, for example, in Figure 3 row 4, you can see that our model can take as input the single image on the left (column name ‘conditioning’) and output a full 3D scene, whose renderings we visualise as “Reconstruction”. We see that our model correctly predicts the shape of the hydrant (including the back), as it has learned a prior over 3D scenes (e.g. that hydrants tend to be cylindrical with pipes at the top). Also, note that the model also samples a plausible background (flat, with green grass around it) for the region behind the original camera, despite it not being observed in the original conditioning image. We now emphasise this further in the experiments section 4.1.

Thanks for the authors' detailed responses. I think most concerns of mine were addressed; these would be beneficial for clarifying the contribution of this work. Especially, the description of the proposed method as an extension from K-planes might give a good intuition for readers.

Thank you for the positive reception of our work! We have now fixed the weaknesses you mentioned.

After reviewing the supplementary material, I noticed that the generated images from different viewpoints don't seem very consistent. The videos display noticeable bouncing.

This is purely a visualisation problem, which arose due to inconsistent cropping. We have now fixed this to stay consistent throughout the video. We invite you to see the updated supplementary videos, which clearly show the multi-view 3D consistency of the generated images.

The paper presentation could be enhanced. While Figure 1 demonstrates the adaptive neural scene representation, there is no figure of the structure or pipeline of the multi-view diffusion model. Given that this diffusion model plays a crucial role in this paper and contains modules like the setwise multi-view encoder, it deserve a figure.

Thank you, we now added an additional figure to ​​supplementary section B to also show our diffusion architecture and its connection with the scene representation.

have the authors undertaken both qualitative and quantitative evaluations to figure out if their method indeed preserves consistency? (or at least an ablation study). While I acknowledge that perfection might not be attainable due to the presence of the black-box diffusion model, an examination in this regard is necessary to substantiate the authors' assertion that their method ensure 3D consistency while retaining expressiveness.

We emphasise that our neural scene representation defines an explicit 3D scene (i.e. a density and color at every point in 3D space), that is rendered directly to the output images as the last stage of the denoiser model. Thus, 3D scenes generated by our method are guaranteed to be 3D-consistent when rendered from any viewpoint. This is in contrast to neural rendering methods that use implicit features and black-box postprocessing – it is a particular benefit of our model that the denoiser is not a black box. We have now further clarified this in section 3.1 and 3.2.

Thank you for the positive reception of our work and for the constructive and specific suggestions – we hope we have taken all of them into account in the revised version. Below we address some remaining concerns you raised. We also invite you to look at the additional points we now emphasise thanks to the other reviewers, including specific novelties of our neural representation and the ability to render 3D scenes at 1024x1024 resolution.

No qualitative comparison was made with baselines, particularly VSD.

Thanks for noticing this omission! We will today include qualitative comparison with VSD, pixelNeRF++ and RenderDiffusion++ in the supplementary.

No quantitative and qualitative comparison was made with VSD in the task of single view reconstruction.

Again, thanks for noting this omission. We are running this evaluation now, and will add the results in a couple of days.

The experiments section lacks clarity, as the pixelnerf++ and renderdiff++ are not explained in the main paper but rather in the appendix. Additionally, although the authors revised VSD, the table still refers to it as VSD instead of VSD*, and this should be explicitly noted.

Thanks, we now denote VSD as VSD* -- though note that we only relaxed their assumption of masks and increased the voxel grid size. We also have now moved the explanation of baselines to the main paper, in addition to the explanation in the supplementary.

It is crucial to include the ablation study in the main paper rather than relegating it to the appendix.

Thank you, we have now moved the ablation study to the main paper.