DMV3D: Denoising Multi-view Diffusion Using 3D Large Reconstruction Model

We thank the reviewer for your appreciation of our method’s capability to handle diverse objects, our novelty to use a reconstructor as a multi-view denoiser, and our high-quality results. We address your questions below.

1. Complex prompts: our text-to-3D model has limited capability to handle complex prompts due to the limited scale of multi-view data. But as the reviewer points out, we can use Stable Diffusion to generate a single image from a complex prompt, and then use our image-conditioned model to lift it into 3D. We show the results here (https://dmv3d.github.io/rebuttal_sdcond.html).

2. Noise in reference image: we tried adding noise to reference images, but found that this is essentially equivalent to unconditional generation, causing the model to less respect the respect image. Hence, we choose to use the noise-free reference image across all timesteps.

Dear Reviewer,

We sincerely thank the reviewer for your feedback and your appreciation of our methods and results. And we would like to express our gratitude for raising the score of our paper!

We thank the reviewer for your appreciation of our clear presentation, method’s elegance and high-quality results. We address your concerns and questions below.

1. Runtime efficiency: we agree with the reviewer that it might be a good idea to interleave geometry-based multi-view denoising and geometry-free multi-view denoising for best runtime efficiency, as rendering 4 views from a triplane at each denoising step does have a negative influence on the inference speed. We leave this point for future exploration. We also visualize the intermediate reconstructed results during denoising process at this link (https://dmv3d.github.io/rebuttal_x0vis.html).

2. Blurry texture: we agree with the reviewer that our generated textures for unseen parts are a bit blurry at its current stage, although we outperform baselines. For future work, it’s interesting to explore increasing image and triplane resolution and decoupling geometry and texture modeling to address this issue.

3. Viewpoint sampling: we randomly sample 4 views during training to use as our input views, and another randomly selected 4 novel views as supervision. During inference, we sample 4 structured viewpoints around an object to denoise.

4. Degenerate solution: we use novel view supervision to prevent the degenerate solution of generating 4 flat planes to explain the input views (see table 3). The results without novel view supervision can be found at this link (https://dmv3d.github.io/rebuttal_wonovelview.html).

Dear reviewer,

We kindly request your feedback on whether our response has addressed your concerns. If you have any remaining questions or concerns, we are more than happy to address them. Thank you for your time and consideration!

We thank the reviewer for your appreciation of our impressive results, our method’s great versatility and clear presentation. Please see our responses to your concerns below.

1. DMV3D vs LRM: We would like to clarify the different contributions of LRM and DMV3D. Please note that, in DMV3D, we acknowledge LRM as prior work and have explicitly cited LRM in section 3.2. Our work makes orthogonal contributions, focusing on different challenges that are not covered by LRM. In particular, LRM focuses on the task of single-image reconstruction only and introduces the first scalable transformer model trained on massive multi-view data. However, in essence, LRM is a deterministic model that cannot model the inherent uncertainty (multiple plausible solutions exist for unseen parts of the 3D objects) in single-image 3D reconstruction. In contrast, DMV3D aims to build a general probabilistic diffusion model for 3D generation. Our framework is not only applicable to single-image reconstruction but also text-to-3D generation – a task completely uncovered by LRM.

   To do so, we propose a novel single-stage multi-view diffusion framework and adopt a transformer model to achieve multi-view denoising. Again, please note that we have cited LRM (by anonymously putting it in the supplemental material) and clearly stated in Sec.3.2 that our transformer architecture is inspired by LRM to avoid any misconception about dual submission. We emphasize that the whole story of our paper, including our entire method section (Sec. 3), does not focus on the original transformer contribution made by LRM. Instead, our focus is mainly on: 1) how to build a multi-view diffusion framework for 3D generation (Sec. 3.1), 2) how to improve/extend the transformer model to support multi-view denoising (Sec. 3.2), and 3) how to enable both text and image conditioning (Sec. 3.3). These aspects are distinct from the scope of LRM. Furthermore, our transformer architecture is also different from the one in LRM in taking noisy multi-view images as input, being conditioned by the diffusion time step, utilizing a new ray-conditioning technique to better handle the diverse camera intrinsics, and supporting text conditioning.

   As a result, even for the same single-image reconstruction problem, DMV3D and LRM address it in very different ways. DMV3D leverages multi-view diffusion denoising, whereas LRM performs single-view image-to-NeRF translation. The transformer inputs (multi-view noisy images vs a single-view clean image) and the inference processes (DDIM-based multi-step diffusion denoising vs single-step transformer inference) are highly different. Moreover, because of being a probabilistic generative model, DMV3D can generate multiple high-quality 3D assets given the same single image input (as shown in Fig. 1 and our project page), while LRM can only generate one single result. The probabilistic modeling introduced by DMV3D also enhances the realism and reduces the blurriness of generated textures for unseen parts; in contrast, LRM tends to generate blurred results due to mode averaging.

   In sum, different from the deterministic model in LRM, our DMV3D introduces a novel probabilistic model for 3D generation and contributes 1) a novel multi-view diffusion framework 2) the general support for both text and image conditioning and 3) new architecture designs to support multi-view denoising and text/image conditioning. The contributions are made orthogonal to those of LRM and we believe these are valuable contributions to the field towards a general, practical, and high-quality 3D generative model.

2. Novelty: As discussed in the related work, we are inspired by the RenderDiffusion work for using a reconstructor as a denosier. Concurrent work, Viewset Diffusion, adopts the same idea from RenderDiffusion, but was only demonstrated to work on limited data lacking diversity. Our novel transformer-based 3D denoiser architecture is different from the reconstructor architecture in viewset diffusion and enables state-of-the-art results for scalable, diverse, and high-quality 3D generation.

3. Limitations: one limitation with our text-to-3D model is that we cannot handle complex text prompts as well as 2D diffusion models like stable diffusion, as a result of the limited scale of multi-view data compared with 2D data. Another limitation is that the generated textures at unseen views are still a bit blurry compared with input view. It will be interesting to further boost the quality of texture sharpness in future work.

4. Training time: despite our method being a diffusion model, we only sample a single time-step at a time to train our multi-view denoiser. This single-step setting is similar to what LRM does except our input images are noisy; hence our model trains as fast as LRM during training.

We thank the reviewer for your appreciation of our method and result. We address your questions below.

1. Train from scratch: all of our models are trained from scratch without using pretrained diffusion models. Generalization of our model is backed up by large-scale training on the massive multi-view data including Objaverse and MvImgNet (See Fig. 7).

2. Number of views: we use 4 views during training and inference (see section 4.3).

Dear reviewer,

We kindly request your feedback on whether our response has addressed your concerns. If you have any remaining questions or concerns, we are more than happy to address them. Thank you for your time and consideration!