MVDream: Multi-view Diffusion for 3D Generation

We thank the reviewer for the detailed and helpful comments. The belows are our responses to reviewer's concerns:

1. Janus Problem
   In the anonymous project page https://mvdream-anonymous.github.io/test_0.html, we have shown the qualitative comparison over 40 prompts without cherry picking nor prompt-wise parameter tuning. It could be seen that our model almost eliminates the multi-face Janus problem by a 100% chance (with a fixed configuration). We have also conducted an experiment for the same prompt but different random seeds. However, we found the difference to be rather subtle because of the large CFG guidance used during SDS: https://github.com/mvdream-anonymous/mvdream-anonymous.github.io/blob/master/static/seeds.mp4

2. Hyper-parameters

• number of views Since we are proposing a new type of neural networks, it will inevitably comes with a few new hyper-parameters. For the number of views, as shown in Figure 11 in Appendix A.2, we found 4 views to a minimum number of views to ensure multi-view consistency for SDS. Therefore, we choose to stop at 4 views to balance the tradeoff between performance and resource cost.
• camera parameters In our experiments, we have tried different types of camera inputs (16-dim extrinsics and 25-dim intrinsics+extrinsics) and different ways to embed them (cross attention vs. time embedding). All of these choices work to some extent. So we choose our current design due to its simpleness and empirical robustness.
• Overall, our pipeline already has much less hyper-parameters compared with prior studies such as Dreamfusion and Magic3D. In particular, our model can in fact use SDS loss alone to generate 3D content while prior studies have many regularization losses to ensure 3D quality. Similar to the response to Question 1, we believe that the stableness of our method can be evaluated from the visual examples in the project page.

3. View-dependent Prompts
   Yes, the view dependent prompting are completely removed in MVDream. We only give the original text descriptions, e.g. "an astronaut riding a horse", to the text encoder.

We thank the reviewer for the careful and helpful comments. Here are our responses to the reviewer's concerns:

1. point of experiments
   we have added an overview paragraph in the revised paper to explain the experiments. In particular, we evaluated our method on three tasks: (1) multi-view image generation for evaluating image quality and consistency (Sec. 4.1), (2) 3D (NeRF) generation with multi-view score distillation as a main downstream task (Sec. 4.2), and (3) DreamBooth for personalized 3D generation (Sec. 4.3).

2. Sec 4.1 and 4.2
   Sec 4.1 (multi-view image generation) Sec 4.2 (multi-view score distillation) are both important contributions of our work. Since our model is essentially an image generation model, we put Section 4.1 before 4.2. We will explain this in the overview paragraph.

3. Direct 3D Generation
   In order to generate 3D content directly without score distillation (optimization), a few-shot reconstruction model will be needed to generate 3D content from 4 views. However, we do not have access to such a few-shot reconstruction model for the moment. We believe this is achievable by training such a model or extending multi-view diffusion to more views. But this is out of the scope of this paper and will be part of the future work.

4. Experiment Details
   We will try to add more explanations to the experiments, including the attention part. The main motivation of the attention experiment is to compare the multi-view consistency, for which we are not aware of any good quantitative metrics for the moment. So we choose to present qualitative results instead.

5. Table 1
   We have updated the results on a held-out validation set.

• We have removed the batch size comparison part and there are now two models trained with the same batch size.
• Yes, "2D data" refers to the LAION dataset. We apologize for the confusion and will make it more clear. Figure 12 in Appendix shows the results between using 3D data alone and 2D+3D data. We observe a clear improvement in image quality by combining 2D and 3D data for training.

5. Figure 5
   These are the images rather than the 3D models generated by our model. Here, we show both in-domain data (training) and out-of-domain examples (conterfactual text descriptions). We will explain this Figure more clearly.
6. negative prompts and CFG rescaling
   The motivation of negative prompts and CFG rescaling were mentioned Sec 3.3. We will add a reference in the experiment back to this section.
7. Dreambooth
   Dreambooth is an extended experiment of our multi-view diffusion model to evaluate its generalizability. It shows the capability how a multi-view diffusion model connects the knowledge from 2D and 3D domain. The explanation of why DreamBooth would work is a tricky problem by itself. Some of the explanations could be found in the original DreamBooth paper. In terms of why it would work for multi-view generation, we think this is because our model is essentially sharing the same modules and weights for 2D and multi-view generation. Thus, the knowledge learned in 2D domain could be potentially transferred to multi-view generation.

We thank the reviewer for the helpful and careful comments. Here are the response to the reviewer's concerns:

1. Larger elevation angles
   In our initial experiments, we have tried to render camera angles with a wider elevation range. But we found that such diverse camera range leads to more learning difficulty and consequently lower image quality. Therefore, we limit our training data to 0-30 elevations for now. With more data, larger networks and more hyper-parameter tuning, we believe it could be possible to further increase the viewpoint range in the future.

2. Zero123+SDS
   Yes, we do have compared with Zero123XL+SDS, the results are shown in the appendix A.1 and Figure 10 in the original submission. The basic conclusion is that Zero123+SDS tends to blur and flatten the back of objects if the input image has complex poses or details. In contrast, MVDream can synthesize self-consistent 3D content from the text directly.

3. Combining with SJC/VSD
   Theoretically our model is combinable with other SDS variants. For the time limit, we do not have a chance to explore this direction since we think the current SDS results are good enough. We will added such a discussion to the revised paper (Sec 3.2)

4. Descriptions\

• 3D-attention: yes, that's what it is. We will try to explain it more clearly.
• Yes. We have explained the sampling of frames in implementation details.
• We will add an algorithm block

5. The time comparison are as follows
   We have mentioned a time comparison in the revised paper (Sec 4.2). In short, our method has a similar time consumption as dreamfusion-IF (2 hours) and faster than Magic3D-IF (4h) and ProlificDreamer (10hours).

6. Necessity of optimization
   We agree with the reviewer that the current optimization process is still taking too long for many use cases. However, as discussed in Section 3.2, we identify two ways of utilizing a multi-view diffusion model for 3D generation: (1) score distillation and (2) use generated multi-view images for few-shot reconstruction. The latter could be a good strategy for accelerating the overall generation process. But since we do not have such a robust 4-view reconstruction model for the moment, we are not able to demonstrate the results of such a pipeline.

We thank the reviewer for the time and helpful comments. Here are our responses to the reviewer's concerns:

1. Multi-view consistency
   It is true that MVDream does not include any architecture design to inject induct bias to ensure multi-view consistency. The key insight here is that 3D-specific architecture design prohibits the model from share the architecture and weights with 2D image generation models, and consequently limited in terms of generalizability.
   Therefore, in MVDream, we seek a pure data-driven approach, where we simply re-use image diffusion models for multi-view generation. We believe that this is achievable because dense multi-view generation can be regarded as a special case of video generation, which has already been shown successful. Besides, Zero123 does not have any inductive bias to ensure view consistency either but it is able to generate promising novel views.
   Once our data-drive approach is proved feasible (which we believe is proved by our generated images), it will be more flexible to be combine with different 2D diffusion models to extend 2D vision prior to 3D.

2. Re-using 2D attention
   This means (1) we inherit the weights of 2D self-attention layers from Image Diffusion UNet, and (2) during reshaping BxFxHxWxC to Bx(FxHxW)xC for self-attention. (for original 2D models, it was Bx(HxW)xC for self-attention).

3. Training Set \ We will add the training set description into the main paper.

4. 3D dataset only
   We have compared with such a model trained on 3D data only, which is discussed in Table 1 in Sec 4.1 and Figure 12 in Appendix A.3.

5. Quantitative Evaluation
   The evaluation was conducted on a subset from the training set because it was a common practice to calculate FID on training set[1][2]. This being said, we have updated the revised paper by recalculating the metrics on a held-out validation set. We then also evaluated Zero123-XL on the same validation set. But it is worth noting that Zero-123 is an image conditioned model, so the comparison will not be fair. The results are as follows: | Method | Condition | FID | IS | CLIP | | --- | --- | --- | --- | --- | | Zero123-XL | Image | 13.91 | 11.77±0.66 | 26.89±3.25 | | Ours | Text | 39.04 | 12.77±0.60 | 30.38±3.50 | To evaluate the Zero123 model, we feed one of the ground-truth view (image) of each sample to generate 4 novel views. Because of this, Zero123's results will have a smaller FID, which measures semantic difference between the generated and ground-truth distributions. In contrast, our models shows higher Inception Score and CLIP, which are metrics to evaluate image quality and text-image consistency.

[1] Karras, Tero, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. "Analyzing and improving the image quality of stylegan." In CVPR 2020. [2] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." In NeurIPS 2020.