DreamFlow: High-quality text-to-3D generation by Approximating Probability Flow

We are glad to hear that we have addressed your concerns. Regarding your additional questions, we respond to your questions one-by-one in what follows:

---

[Q1] Loss weights

The hyperparameters are not that sensitive in our framework. We use APFO loss with orientation regularization loss (see Appendix D), i.e., $L = \lambda_{APFO} L_{APFO}+\lambda_{ori}L_{ori}$, where $\lambda_{APFO}$ and $\lambda_{ori}$ are loss weights. We use $\lambda_{APFO}=1.0$ and $\lambda_{ori}=5.0$ throughout all experiments (unless stated otherwise). Here, we found that using $\lambda_{ori}\in [0.0, 10.0]$ works well, while too large orientation loss causes the shape to be over-smoothed. In 3D mesh tuning, we do not use any regularization loss. The details on the optimizers and learning rates are described in Appendix D.

---

[Q2] Janus’ problem

Since we use pre trained text-to-image diffusion models that do not have 3D knowledge, Janus' problem is still present in our method. One approach in handling Janus’ problem would be using 3D-aware multi-view diffusion models (e.g., concurrent work MVDream [1]), where our proposed method is also applicable to such diffusion models for 3D optimization.

---

[Q3] CLIP R-precision

Yes, we use 397 prompts as reference captions in computation of CLIP R-precision metric.

---

[Q4] Providing additional results

Upon your request, we provide 100 3D contents by using different prompts, which we used for evaluation. Please check out on the following link:

https://drive.google.com/drive/folders/1H1QRb2R0y-gKuAP0LUYd-Px3nL6uAlun?usp=drive_link

---

If you have any further questions/concerns, please do not hesitate to let us know !

Thank you very much, Authors

Reference

[1] Shi, Yichun, et al. "Mvdream: Multi-view diffusion for 3d generation." arXiv preprint arXiv:2308.16512 (2023).

Dear reviewer paN1,

We sincerely appreciate your thoughtful comments and suggestions in reviewing our manuscript. We address each of your questions and concerns individually below. Please let us know if you have any comments/concerns that we have not addressed to your satisfaction.

---

[W1, Q2, Q4] Technical Contribution: Difference vs. VSD

To clarify the difference between APFO and VSD, we provide a detailed explanation in Common Response 1. We also show how APFO differs from VSD with a time-step annealing schedule, either using a linearly decreasing schedule or using weighting priors such as DreamTime [1].

---

[W2, Q1] Motivation and Technical Details

We provide additional information on how the loss and gradient norms are implied in Common Response 2. At a high level, the convergence of the loss implies the convergence of the distribution to the data distribution, and the low-variance gradient norm implies the stability of the 3D optimization. We also show why high-variance loss and gradient norm in VSD could be problematic in Common Response 2, as this can lead to inconsistent 3D optimization and content shifting during optimization.

---

[Q3] Broader application

We demonstrate the advantage of our method that it achieves faster 3D optimization when using text-to-image diffusion priors. Our approach could be further applied to various applications in using the diffusion model priors to optimize visual generators (e.g., neural fields), such as text-to-video generation by optimizing video neural fields with video diffusion models, or text-to-4D generation using image and video diffusion models. We believe that exploring APFO for these tasks are interesting directions that we leave for future work.

Furthermore, since our approach makes use of probability flow ODE, it could be more advantageous when using fast-sampling diffusion models such as distilled diffusion models (e.g., guided distillation [2] or consistency models [3]). On the other hand, the current score distillation method cannot achieve full advantage when used with these models.

---

[Q5] Why is the proposed method superior to VSD?

As we discussed in Common Response 1, our approach allows for faster optimization as we approximate the probability flow ODE, which is a straight path towards data distribution, while VSD utilizes random timesteps in optimization. Also, in Common Response 2, we show that APFO achieves convergence to the data distribution with more stable 3D optimization, while VSD does not. Therefore, those explanations demonstrate how APFO presents faster and higher-quality text-to-3D optimization. Those claims are empirically supported in Tables 2 and 3 of our manuscript, where we provide experiments to compare APFO and VSD under the same setup and show that our approach presents better quality in the CLIP precision metric.

---

Reference

[1] Huang, Yukun, et al. "DreamTime: An Improved Optimization Strategy for Text-to-3D Content Creation." arXiv preprint arXiv:2306.12422 (2023).

[2] Meng, Chenlin, et al. "On distillation of guided diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Song, Yang, et al. "Consistency models." (2023).

Dear Reviewer paN1,

Once again, thank you for taking the time to review our paper. We appreciate your efforts in helping us improve our work. As we approach the end of the discussion period in two days, we wanted to inquire if you have any remaining concerns that we could address. Kindly let us know.

Thank you.

Dear Reviewer ZxmY,

We sincerely appreciate your thoughtful comments and suggestions in reviewing our manuscript. We address each of your questions and concerns individually below. Please let us know if you have any comments/concerns that we have not addressed to your satisfaction.

---

[W1] Comparison to VSD and VSD to other timestep annealing methods

In Common Response 1, we detail the difference between VSD and APFO. Also, we compare APFO and VSD with timestep annealing methods including DreamTime [1]. As you understood, we remark that APFO is a more naturally motivated approach in using a decreasing timestep schedule, and presents faster optimization with higher fidelity.

---

Reference

[1] Huang, Yukun, et al. "DreamTime: An Improved Optimization Strategy for Text-to-3D Content Creation." arXiv preprint arXiv:2306.12422 (2023).

Dear Reviewer ZxmY,

Once again, thank you for taking the time to review our paper. We appreciate your efforts in helping us improve our work. As we approach the end of the discussion period in two days, we wanted to inquire if you have any remaining concerns that we could address. Kindly let us know.

Thank you.

Dear reviewer pJwX,

We sincerely appreciate your thoughtful comments and suggestions in reviewing our manuscript. We address each of your questions and concerns individually below. Please let us know if you have any comments/concerns that we have not addressed to your satisfaction.

---

[W1] Comparison to VSD

We clarify the difference between APFO and VSD in Common Response 1. We remark that the APFO leverages the time-derivative of noise scale derived from probability flow ODE in leveraging the diffusion model prior, as opposed to the weighting function in VSD. This leads to faster convergence and better fidelity as shown in 2D experiments.

---

[W2] Rationale for Color Bias

Note that the color bias does not exist in our method, and different colors or textures can be generated due to the diversity of generation. However, as we have shown in the limitation part in Section 6, some undesirable results could be obtained due to the use of pre-trained text-to-image diffusion models.

---

[Q1] CFG Scale

For your information, we use a classifier-free guidance scale of 7.5 for each stage.

---

[Q2] Memory consumption

For your information, our approach requires >30GB for stage 1 NeRF optimization, and less than 20GB for stages 2 and 3 3D mesh tuning. Note that since we use instant NGP for stage 1 NeRF optimization, the GPU memory may vary due to sparsity during optimization. We tested with an A100 40GB GPU.

---

[Q3] 2D experiments

Following your suggestion, we provide 2D experiments to compare APFO with VSD and VSD with timestep annealing methods (see Common Response 1 and Appendix B of revised manuscripts for details). Note that APFO can produce high-fidelity 2D images with fewer steps than VSD without blurry artifacts.

Dear Reviewer pJwX,

Once again, thank you for taking the time to review our paper. We appreciate your efforts in helping us improve our work. As we approach the end of the discussion period in two days, we wanted to inquire if you have any remaining concerns that we could address. Kindly let us know.

Thank you.

Dear Reviewer 9SLV,

We sincerely appreciate your thoughtful comments and suggestions in reviewing our manuscript. We address each of your questions and concerns individually below. Please let us know if you have any comments/concerns that we have not addressed to your satisfaction.

---

[W1] Difference between VSD and comparison with previous works on decreasing noise schedule

In Common Response 1, we clarify that APFO and VSD have different optimization objectives, and they are different in implementation as APFO is entailed with a predetermined decreasing timestep schedule. In addition, APFO not only differs from VSD together with a decreasing timestep schedule, but also is a better approach in leveraging the diffusion model prior. We also compare APFO with VSD + DreamTime [1], which uses priors on the weighting function. Our explanation with 2D experiments shows that APFO is not only different from VSD with timestep annealing, but also faster (in terms of optimization speed) and better at generating high-fidelity images.

---

[W2] Regarding the Diversity

Our method does not suffer from diversity since we use small CFG scales (e.g. $\omega=7.5$) throughout the experiment.

---

[W3] Experimental Validation

We conducted a human evaluation with 20 text prompts to compare with other baselines, where 10 pairwise comparisons were made for each prompt, thus 200 comparisons were made for each baseline. Our human evaluation experiment suggests that our method shows better photorealism than previous work. In fact, in Table 2 and Table 3, we perform an additional comparison with 100 text prompts taken from the DreamFusion gallery, which shows that our method outperforms others in the CLIP R-precision metric.

---

[W4] Additional ablation study on the effect of using SDXL

To further demonstrate the effectiveness of Stage 3 mesh refinement, we provide more qualitative results in Appendix Figure 13 of our revised manuscript. Remark that Stage 3 mesh refinement results in more high-contrast 3D contents. Also, for quantitative evaluation, we conduct an ablation study by computing CLIP aesthetic score [2]. Similar to evaluation of CLIP R-precision, we render views and compute CLIP aesthetic scores and report average scores. Note that the average CLIP aesthetic score has improved after stage 3 mesh refinement from 4.82 to 5.02, which indicates the advantage of stage 3 mesh refinement. Nevertheless, as we have shown in the limitation part in Section 6, some undesirable results could be obtained due to the use of pre-trained text-to-image diffusion models.

---

[Q1] Gradient variance and the quality of 3D representation

To clarify our points, in Common Response 2, we provide detailed analysis on how the convergence of loss and gradient norm indicates in 3D optimization. Indeed, the convergence of loss denotes the convergence of 3D views to the data distribution and the low-variance gradient norm indicates the stability of 3D optimization, where both implies superiority of APFO compared to VSD.

---

[Q2-1] Comparison with VSD + timestep annealing

Please refer to [W1] and Common Response 1 for detailed comparison on APFO with VSD and its variants with timestep annealing methods including DreamTime [1].

[Q2-2] Timestep selection To clarify, we choose the time steps $t_1,\cdots,t_N$ such that they have decreasing variance $\sigma(t_1) >\cdots > \sigma(t_N)$. In general, diffusion models are trained with 1000 timesteps, and we denote the timestep as the ratio divided by 1000 (e.g., $t_1=1.0$ denotes the timestep at 1000, and $t_N=0.2$ denotes the timestep at 200). While we use $t_1=1.0$ when optimizing NeRF from scratch (i.e., step 1), we choose a smaller value of $t_1$ for fine-tuning the 3D mesh (i.e., steps 2 and 3). This is because starting from a large time step may alter the textures due to the addition of large amounts of noise and lose fidelity to the original 3D content from the previous stage. Empirically, we have found that $t_1=0.5$ is sufficient for stage 2 mesh refinement and $t_1=0.3$ is sufficient for stage 3 mesh refinement.

---

[Miscellaneous]

Thank you for pointing out the typos and missing details. We will address these before the discussion period closes. Regarding the details in Eq. (12) to Eq. (13), it simply computes a gradient $dx/dt$ by inserting discrete time steps, and the loss is given by the difference between the generator $g(\theta,c)$ and the target generated by $x+\Delta t \cdot dx/dt$.

---

Reference

[1] Huang, Yukun, et al. "DreamTime: An Improved Optimization Strategy for Text-to-3D Content Creation." arXiv preprint arXiv:2306.12422 (2023).

[2] C Schuhmann. Laoin aesthetic predictor. 2022. URL https://laion.ai/blog/ laion-aesthetics/.

Dear Reviewer 9SLV,

Once again, thank you for taking the time to review our paper. We appreciate your efforts in helping us improve our work. As we approach the end of the discussion period in two days, we wanted to inquire if you have any remaining concerns that we could address. Kindly let us know.

Thank you.