DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior

Thanks for your effort invested in reviewing our work. We are grateful for your recognition of the novelty and advancements our research brings to the field. In response to your questions, please find our clarifications below.

Q1: Runtime of the method.

A: The training time comparison against recently published works is shown below.

Below we report the detailed runtime profile. These numbers are measured on 30 objects. The training is conducted on 8 A100 GPUs and the inference on 512 $\times$ 512 resolution is measured using a single A100 GPU. We did not fully optimize the training time, and we expect significant speedup with further engineering optimization.

Q2: The difference between the proposed bootstrapped diffusion prior and the LoRA updates in ProlificDreamer.

A: Thank you for raising the point. Below we clarify their difference.

ProlificDreamer regards the 3D solution corresponding to the text as a distribution. Thus, ProlificDreamer aims to learn a distribution $\mu$ to match the distribution characterized by the pretrained diffusion model. Mathematically, this is achieved by minimizing the score function of the two distributions. The score function of the pretrained model is represented by its noise prediction $\epsilon_{\text{pretrain}}(x_t,t,y)$. On the other hand, the score function of the optimized 3D scene is estimated by the LoRA model, i.e.,\epsilon_\\phi\left(x_t, t, c, y, \\{x\\}\right), using the multi-view image renderings $\\{x\\}$. Hence, the VSD loss used in ProlificDreamer is:

$$\nabla_\theta \mathcal{L}_{\mathrm{VSD}}(\theta) \triangleq \mathbb{E}\_{t, \epsilon, c}\left[\omega(t)(\epsilon\_{\mathrm{pretrain}}\left(x_t; t, y\right)-\underbrace{\epsilon\_\phi\left(x_t; t, c, y, \\{x\\}\right)}\_{\text {LoRA }}) \frac{\partial g(\theta, c)}{\partial \theta}\right].$$

The proposed bootstrapped score distillation differs in the motivation fundamentally. We claim that a fixed 2D diffusion prior is not 3D aware and cannot provide sufficient guidance that ensures multi-view consistency. Instead, we propose to learn from a moving target that also updates based on the 3D scene being optimized. In the implementation, we render the multi-view images of the scene and augment their quality (i.e., adding the Gaussian noise and then denoising them) using a diffusion model, forming high-quality augmented renderings denoted as $\\{\tilde{x}\\}$. These augmented images are used to compute the LoRA model that serves as the training target (the first term in the following formula):

$$\nabla_\theta \mathcal{L}_{\mathrm{BSD}}(\theta) \triangleq \mathbb{E}\_{t, \epsilon, c}\left[\omega(t)(\underbrace{\epsilon\_{\mathrm{DreamBooth}}\left(x_t; t, c, y, \\{\tilde{x}\\}_k\right)}\_{\text{BSD}}-\underbrace{\epsilon\_\phi\left(x_t; t, c, y, \\{x\\}\right)}\_{\text {LoRA }}) \frac{\partial g(\theta, c)}{\partial \theta}\right]$$

We iterate the above process and apply attenuated augmentation strength to derive augmented views, $\\{\tilde{x}\\}_k$, where $k$ denotes the iteration steps. As training proceeds, both the diffusion prior and the 3D scene get improved, which bootstraps the 3D optimization. We have better formulated the above process during revision.

Q3: Further explanation of the evaluation protocol.

A: Thanks for the question. We compute the CLIP score to evaluate the semantic similarity between the image renderings at novel views and the reference image. A higher CLIP score indicates better semantic consistency around different views. The contextual loss is computed to measure the style similarity between the novel views and the reference image. The contextual metric can be regarded as an improved LPIPS without requiring pixel-wise alignment. A higher contextual metric means improved style consistency around the generated 3D asset. We use 120 rendered views to compute both the CLIP and the contextual score. To better reveal the subjective quality, we also conducted a user study which gathered 480 responses from a total of 32 participants. We have included these details in the paper revision.

We are grateful for your acknowledgment of the technical novelty and results presented in our work. Please find our response to your questions as follows.

Q1: The use of multiple diffusion models.

A: Thanks for your comment. This paper presents two key components for high-fidelity 3D generation: hierarchical generation and bootstrapped score distillation (BSD).

Our hierarchical approach involves decomposing the challenging 3D generation into multiple well-designed stages. The coarse geometry sculpting stage aims to learn a coherent geometry, so we introduce a 3D-aware diffusion prior, along with several effective techniques such as progressive viewing and diffusion timestep annealing, to ensure the creation of a coherent geometry. The subsequent stage further refines the texture, demonstrating how a meticulously designed hierarchical framework can generate high-fidelity 3D assets of unprecedented quality.

Second, we propose a novel bootstrapped score distillation (BSD), which fundamentally differs from prior SDS-based methods in that we optimize against a dynamically evolving target. This bootstrapped diffusion prior substantially improves the realism of the 3D texture. To the best of our knowledge, this technique has not been explored before.

Therefore, we assert that our method is not merely a trivial combination of existing approaches. Rather, this work builds on the merits of prior works while presenting novel insights, setting a new benchmark in the field.

Q2: How does the personalized diffusion model trained on initial texture boost the texture quality?

A: Thanks for raising the point. Indeed, the texture from the previous stage has limited quality and suffers from blurriness. Therefore, we use a pretrained diffusion model to enhance their quality.

We incorporate Gaussian noise into multi-view images, allowing a diffusion model to reconstruct high-quality visuals. Despite affecting view consistency, this process yields impressively realistic images. The enhanced multi-view images introduce scene aspects to form a personalized diffusion model, which significantly improves texture quality.

We iterate the above process several times and each step leads to improved 3D texture. We progressively decrease the strength to augment the multi-view rendering to better retain their view consistency. As we iterate the above process, we bootstrap the texture quality for the 3D instance being optimized.

Q3: Writing.

A: Thanks for your comment. We have conducted careful proofreading and fixed some typos in the revision.

Q4: More ablation studies.

A: Per your suggestion, we provide additional qualitative and quantitative ablation studies for two stages respectively as follows. The techniques of time annealing and progressive training improve the view consistency relative to the reference image, as measured by the CLIP and contextual score. We have included these results in the appendix.

Q5: Training and inference time costs.

A: The training time comparison against recently published works is shown below.

We report the detailed runtime profile below. These numbers are measured on 30 objects. The training is conducted on 8 A100 GPUs and the inference on 512 $\times$ 512 resolution is measured using a single A100 GPU. We did not fully optimize the training time and we expect significant speedup with further engineering optimization.

We are grateful for your recognition to our paper results and the effectiveness of our approach. Please find our point-to-point response to the review comments.

Q1: Technical contribution.

A: Thanks for your question. 3D generation is particularly challenging due to the lack of a large quantity of 3D data. This paper aims to present two key components that lead to high-fidelity 3D content creation.

First, we demonstrate how a meticulously designed hierarchical framework can achieve high-fidelity 3D generation. We decompose the whole generation process into multiple steps: 2D image creation from the text prompt, the geometry sculpting stage and texture boosting stage, with each stage having different priorities. The geometry sculpting stage focuses on obtaining a coherent geometry. we show that a 3D-aware diffusion prior, accompanied with multiple effective techniques like progressive view and diffusion timestep annealing, can lead to consistent 3D geometry. While this stage often compromises the texture, we resort to the following texture boosting stage which is dedicated to improving the texture quality. We believe this hierarchical framework with thoughtful design choices sets up a good starting point for the following research works.

Second, we propose a bootstrapped score distillation (BSD) scheme that considerably enhances the texture quality. The key motivation is that a fixed 2D diffusion model does not suffice to provide view-consistent guidance. We propose to improve this diffusion prior and the 3D mesh in the cyclic optimization: steadily improved multiview renderings lead to an increasingly view-consistent diffusion prior, which in turn further improves the 3D texture. In this way, the 3D optimization is bootstrapped, ultimately leading to highly detailed textures. In the implementation, we render the multi-view images of the scene and augment their quality (i.e., adding the Gaussian noise and then denoising them) using a diffusion model, forming high-quality augmented renderings denoted as {$\{\tilde{x}\}$}. These augmented images are used to compute the LoRA model that serves as the training target: $\nabla\_\theta \mathcal{L}\_{\mathrm{BSD}}(\theta) \triangleq \mathbb{E}\_{t, \epsilon, c}\left[\omega(t)(\underbrace{\epsilon\_{\mathrm{DreamBooth}}\left(x_t; t, c, y, \\{\tilde{x}\\}_k\right)}\_{\text{BSD}}-\underbrace{\epsilon\_\phi\left(x_t; t, c, y, \\{x\\}\right)}\_{\text {LoRA }}) \frac{\partial g(\theta, c)}{\partial \theta}\right]$

This technique substantially differs from score distillation methods in that it optimizes against a continuously evolving target. This technique is vital to compelling visual quality and we ablate its effectiveness through Figure 5, Figure 6, Figure 10 and Table 2. To the best of our knowledge, this method has not been explored before.

Hence, we assert this paper is not a trivial combination of previous works. Rather, we believe it represents a significant contribution to the field through the introduction of the aforementioned key components. We hope that these insights will be valuable to the community.

Q2: Performance of out-of-domain generation.

A: That's a good point! As illustrated in the paper, our method demonstrates good imaginative and combinational power and can produce fantasy 3D assets. We have provided more qualitative results in the appendix. We welcome the reviewer to check them.

To better explain that our results greatly differ from the existing 3D data, we retrieve the top-1 similar image from the Objectverse dataset. The retrieval results are shown in Figure 13 in the appendix. This retrieval experiment shows that our results are far different from the instance in the 3D dataset, showing good out-of-domain generalization capability for our approach.

Q3: Relative weights of loss terms.

A: Thanks for figuring it out. We have included this implementation detail in Section 4.2 during revision.

Q4: Do all the examples share the same parameters?

A: Thanks for raising the point. Most cases in our paper (roughly 90%) share the same hyper-parameters. However, some adjustments might be needed. When the reference is captured at an oblique angle, we may change the evaluation angle for the view-conditioned diffusion model. Additionally, we might need to properly set the camera FOV to suit the object size in the reference image.

Q4: Prompt engineering clarity.

A: We are thankful for your careful examination. We use the prompt "a [V] astronaut" rather than "an astronaut in a sand beach" because we focus on hallucinating the 3D for the foreground object exclusively. The unique identifier “[v]” is used to characterize subject-specific attributes such as lighting, color, pose, etc.

Q5: Training and inference time.

A: We compare the training time against recent leading methods:

The profile of training and inference time is reported in the following table. We report the average runtime on 30 objects. The 3D asset is optimized on 8 A100 GPUs and the inference is done using a single GPU. We render at 512 $\times$ 512 resolution during inference. We haven't carefully optimized the optimization speed we consider efficient 3D generation as future work.

Q6: Robustness of the system.

A: This is a good point! Most cases in our paper (roughly 90%) share the same hyper-parameters. However, some adjustments might be needed. When the reference is captured from an oblique angle, we may change the evaluation angle for the view-conditioned diffusion model. Additionally, we might need to properly set the camera FOV to suit the object size in the reference image.

Q7: The rationale of using Deepfloyd IF.

A: We employ the DeepFloyd IF model as it provides enhanced view-awareness and reduces the likelihood of encountering the “Janus (multi-face) problem”. This is because the DeepFloyd IF model has a stronger language encoder (T5-XXL) that better encodes the pose information denoted in the text prompt. Moreover, the DeepFloyd base model offers gradient on a coarse resolution (64x64), which promotes a coherent 3D geometry rather than introducing high-frequency signals that distract the geometry sculpting. The ablation of this design choice is shown in Table 3 in the revised version.

Q8: Miscellaneous.

A: We have improved the order of figures and added text descriptions of the corresponding stages in Figure 6, as per your suggestion.

Thank you for the kind words for recognizing the impact of the paper. Please find our point-to-point response to your questions below.

Q1: Clarity of approach section.

A: Thanks for the comment. As suggested, we summarize the loss functions in the two stages in Eq.7 and Eq.9, respectively. The algorithm for the bootstrapped score distillation is detailed in the appendix. Furthermore, we have supplemented more implementation details. We hope these changes have improved the clarity of our method.

Q2: Technical contribution clarification.

A: Our method addresses 3D generation through a two-tiered process: a geometry sculpting stage and a texture boosting stage.

In the geometry sculpting stage, we strive to construct a coarse but 3D-coherent geometry. This is achieved through a 3D-aware diffusion prior, augmented by several effective training strategies. This stage aims to create robust and coherent geometry.

For the texture boosting stage, we introduce a novel bootstrapped score distillation loss. This approach enables the generation of detailed and visually stunning textures, contributing to a more realistic and high-quality 3D output.

Hence, the two stages focus on different aspects: the former prioritizes geometry while the latter adds meticulous texture. To explain the relative importance of different techniques, we conduct a more thorough ablation study, both qualitatively and quantitatively, in Figure 11, Table 2 and Table 3, as shown in the appendix. For your convenience, we provide quantitative results for the BSD loss as follows:

Q3: Further validation, especially for $L_{\textnormal{RGB}}$, $L_{\textnormal{mask}}$ and the use of NeuS representation.

A: We add both quantitative and qualitative ablation studies of loss functions ($\mathcal{L}\_{\textnormal{RGB}}$ and $\mathcal{L}_{\textnormal{mask}}$), training strategies (timestep annealing and progressive training), and model choice (DeepFloyd vs. Stable Diffusion) in the revision (See Appendix A.4), showing the effectiveness of individual techniques.

We choose to optimize NeuS instead of NeRF in the geometry sculpting stage because NeRF learns a volume field which is hard to yield a high-quality surface for the following stage, whereas NeuS parameterizes the scene with neural SDF representation, yielding a high-quality surface for the following stage. Please refer to Figure 12 in the Appendix for visual examination.

For progressive training, we linearly increase the sampling range of camera positions with elevation angle ($\phi_\textnormal{cam}$) from $0^{\circ}$ to $[-10^{\circ}, 45^{\circ}]$, and azimuth angle ($\theta_\textnormal{cam}$) from $0^{\circ}$ to $[-180^{\circ}, 180^{\circ}]$. The progress viewing training takes 200 iterations. We have included this implementation detail in the Appendix following your suggestion.