DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation

Thank you very much for the insightful comments and questions.

W1. Timestep schedule adaptive to the current renderings.

We totally agree with the reviewer’s comment that the optimal schedule might be different for different prompts. In this work, we aim to provide a general schedule that works reasonably well across most prompts. Designing a schedule to be adaptive to the renderings of the current optimization state is very exciting future work to be explored.

W2. More explanation of the VSD and TP-VSD results.

Thank you very much for pointing this out. We now add a dedicated section G in the appendix, explaining the details of TP-VSD with more quantitative examples demonstrating finer details and clarity after applying our proposed time schedule.

Q1. Any idea on how to design an adaptive schedule.

Designing an adaptive schedule instead of a fixed one is challenging (in terms of training time efficiency) but a very exciting direction, we can start with a simple adaptive strategy and see how to make it better. A solution inspired by the greedy algorithm is that, based on the rendered image at each iteration, we exhaustively enumerate all timesteps to determine which $t$ can bring the greatest quality or information gain (e.g. measured by CLIP score), and select the optimal one for the current iteration. More delicate design remains to be explored for future work.

Thank you very much for the constructive questions. We’ve provided more results according to your comments in the new revision.

W1. Limited quality compared to Magic3D, DreamBooth3d, or Fantasia3D.

We acknowledge that the quality of the results presented in the paper is limited because most results are based on a basic DreamFusion implementation (64x64 NeRF), without using more advanced 3D representations or higher resolutions as in the original papers.

As the proposed timestep schedule improves over general score distillation, our method is orthogonal to these works and can further improve their quality. To demonstrate this, we conduct more evaluations based on Magic3D and Fantasia3D, including quantitative and qualitative results, as shown in Figure 15, Figure 18, and Figure 21 (the fourth row) in the appendix. DreamBooth3D is not included because it is not open-sourced yet. Specifically, in Figure 18, our method improves the quality of Fantasia3D by avoiding semantic and geometric defects; in the fourth row of Figure 21, our method improves the quality of Magic3D by correcting color distortion.

W2. Comprehensive evaluations on diverse 3D representations.

Thank you for the suggestion. We further conduct comprehensive evaluations of our proposed timestep schedule on different 3D representations, including: NeRF, NeuS, and DMTet.

• For quantitative evaluations shown in Figure 15, DreamFusion and Zero-1-to-3 are NeRF-based methods, TextMesh is a NeuS-based method, Fantasia3D is a DMTet-based method, Magic3D is a hybrid (NeRF + DMTet) method.
• For qualitative evaluations, we provide NeRF-based results in Figure 7, Figure 9, Figure 10, Figure 17, and Figure 21 (the first to third rows), NeuS-based results in Figure 19 and Figure 22, DMTet-based results in Figure 18 and Figure 21 (the fourth row).

Q1. Training processes of different neural implicit fields under SDS guidance.

Thank you for the question. We follow the reviewer's comment and further unfold the optimization processes of NeuS and initialized NeRF under SDS guidance, and visualize the gradients across different timesteps, as shown in Section I Figure 20.

From the results, we observe that: (a) neural implicit fields follow a coarse-to-fine generation process; (b) gradients at different timesteps provide different granularity of information. Based on the above observations, we can conclude that the supervision misalignment issue described in Section 3.2 also exists in the optimization processes of NeuS and initialized NeRF under SDS guidance. As a result, the proposed $t$-annealing schedule should be applicable to these different neural implicit fields. Specially, for NeRF initialized with a rudimentary object, SDS optimization with large timesteps (w.r.t large noise injection) should be avoided, because it could be detrimental to the initialization prior.

Q2. Clarification of the diffusion models used.

Our apologies for the unclear presentation, we will revise the manuscript to add information about the diffusion models used. To clarify, only the Zero123 results (in Figure 1 and Figure 15) are derived from the SDS loss in tandem with the image based Zero123 model. The remaining results are based on DreamFusion and TextMesh using the text based DeepFloyd-IF or Stable-Diffusion model.

Most of our experiments are conducted with text prompts instead of image prompts because we believe text-to-3D generation is more fundamental and challenging. Even though our method is applicable to Image-to-3D generation, using pre-trained 3D-aware Zero123 model only requires a few hundred iterations to converge, which makes the analysis and evaluation of timestep schedule less illustrative.

Q3. Evidence to address blurriness and color distortion is lacking.

Thank you for the the constructive comment. We additionally provide examples to demonstrate the effectiveness of our approach in alleviating blurriness and color distortion issues, as shown in Figure 21.

Q4. Missing citations for AvatarCraft.

Thank you for pointing this out. We now add qualitative examples of applying DreamTime to AvatarCraft in Figure 22. We'll add more citations on text-to-avatar generation in later revisions.

Thank you very much for the intriguing questions and comments.

W1. Existing works ProlificDreamer and HiFA have proposed timestep schedules.

We acknowledge that ProlificDreamer and HiFA are concurrent works to ours, and would like to emphasize that such existing works only treat time schedules as a practical technique, while our work carefully studies the impact of timestep on diffusion-guided 3D generation with detailed analysis.

Nonetheless, we agree with the reviewer's comment and compare our method with the timestep schedules proposed by HiFA and ProlificDreamer, as shown in Section F (Figure 16) in the Appendix. Compared with our results, both competitors fail to generate geometric details. We argue that the reasons for their failures are different:

• HiFA’s timestep schedule favors training with small t, where high gradient variance leads to color bias and simplified geometry. We also mentioned these effects in Figure 11 and Figure 13. As an exception, the original HiFA paper uses an iterative SDS optimization that may avoid the high variance of small t, which is orthogonal to our work.
• ProlificDreamer’s timestep schedule could be regarded as a compromise between random t-sampling and linear t-annealing, Consequently, it retains the drawbacks associated with random t-sampling, as discussed in Section 3 of this paper.

W2. More results of TP-VSD.

Thank you for the suggestion. We now provide more results of TP-VSD on a wider range of objects and scenes in Section G (Figure 17) in the Appendix.

W3. Contribution as an ICLR paper may be limited.

With all due respect, we would like to argue differently. Improving sampling quality and efficiency has been an important subject for diffusion-guided image generation. However, optimization-based 3D generation is known to be time-consuming with sub-optimal generation results, yet understanding and addressing the inefficiencies stemming from the formulation of random t-sampling has been overlooked, which is the focus of our work. We would like to quote reviewer DnQ9 that "It studies an aspect of using diffusion model for 3D diffusion model -- how to schedule the timestep used to condition the diffusion model, which is not systematically explored in prior literature... The authors also show the generality of this technique by showing results across many state-of-the-art 3D generation methods so it can be easily adopted by the community in any future work along the line.", reviewer 5Rpq that "It provides a visualization of NeRF's training steps guided by the SDS loss, uncovering a refined pathway to mitigate the training diversity and challenges associated with SDS loss.", and reviewer iLxc that "The method is well motivated." We hope this would clarify our contributions more concretely.

Q1. Concrete theoretical deduction for timestep annealing scheme.

Thank you for the question. In this paper, we aim to provide a general guideline for timestep sampling, which should a) be monotonically non-increasing for coarse-to-fine generation and to avoid inefficiency b) suppress the sampling of small timesteps to avoid high-variance gradients c) encourage the sampling of medium (content-related) timesteps for most informative guidance.

The proposed timestep annealing scheme is an empirically robust one across prompts, but as pointed out by reviewer iLxc, different prompts may work best under different schedules. While it's challenging to deduct a theoretically optimal schedule to work for all prompts, another possible direction would be to design an adaptive schedule which remains for future work.

Thank you very much for the insightful suggestions. We’ve provided more results according to your comments.

W1. Scope of this work.

Thank you very much for the insightful suggestion. Improving sampling quality and efficiency has been an important aspect for 2D generative models. However, while optimization-based 3D generation is known to be time-consuming with sub-optimal generation results, understanding and addressing the inefficiencies stemming from the formulation with random t-sampling has been overlooked, which is the focus of our work. We totally agree that an unified study on 2D and 3D sampling strategy would be very exciting future work.

W2. More quantitative experiments.

We now provide comprehensive user study results in Section E of the appendix.

Q1. The paper should discuss the findings in the Score Jacobian Chaining paper.

Thank you very much for the suggestion. SJC did experiment with annealed variance scheduling on 2D image generation, but the observations were inconclusive: annealed t benefits sampling for unconditional distribution (FFHQ and LSUN) but not so much for sampling from stable diffusion. We hypothesize that the difference lies in implementation and time scheduling details. Furthermore, our analysis on both 2D and 3D generation reveals more details of the optimization process for designing an effective time schedule. We'll add more discussions on findings in SJC.