Noise-free Score Distillation

Thank you for your time and constructive questions. We address them below:

How did the authors separate the small and large timestep values based on $t=200$? Why not $t=100, 300$, or $400$?

We added ablation studies in Appendix A.5, including an ablation on this threshold. As can be seen in the results presented in Figure 12, $t\_s=200$ provides the best tradeoff between fine details and artifacts.

Is it valid to assume $\delta\_{C=p\_\text{neg}} ≈ −\delta\_\text{D}$? Did the choice of the negative prompt affect the performance?

We added a discussion that motivates this choice in Appendix A.4. As explained there, there is a close relation between this assumption and the common technique of using negative prompts in ancestral sampling to improve image quality. The negative prompt was kept fixed during all experiments (both images and NeRFs). Our method is not sensitive to the exact choice of words in the negative prompt, and other combinations of negative words do not have much impact.

How about changing (6) into just using the second part of (6) for all the time steps (i.e., unconditional term - negative-prompt-induced term, for all the time steps)?

We added ablation studies in Appendix A.5. In the time threshold ablation presented in Figure 12 we included results for $t\_s=0$, in which the negative prompt term is applied for all timesteps.

Lack of any quantitative results concerns me a lot. Is it possible to include the comparison using CLIP R-Precision of Table 1 in the DreamFusion paper? Furthermore, leveraging the MS-COCO text-to-image benchmark with FID/IS/CLIP score metrics may be worth trying to justify the results of 2D image generation in Figure

As mentioned in our conclusion section, quantitatively comparing NeRFs generated from text prompt remains a challenge due to the absence of suitable metrics and benchmarks. Additionally, most works do not provide an official implementation which makes the quantitative evaluation even more tricky. Therefore, we chose to provide a large gallery of results (with prompts taken from the baseline works) to allow the reader to get an impression and to directly assess the quality of our results compared to other works. Yet, we add quantitative evaluation in Appendix A.6, measuring FID on generated images with NFSD and SDS-100, and presenting user study results, following Magic3D and ProlificDreamer. As can be seen in Table 1, our method achieves lower (better) FID compared to SDS-100, confirming the better quality achieved by our method. As can be seen in Table 2, our method also achieved the best results in both prompt alignment and image quality in the user study. We suspect that in the prompt alignment question responders were biased by the superior image quality of our results. Note that our results are aligned with the results shown in the user study in ProlificDreamer, in the sense that in both studies ProlificDreamer got better results than Fantasia3D, which, in turn, got better results than Magic3D.

We thank the reviewer for their time and the concern they raised. We have addressed it below.

I'm concerned about whether $p\_\text{neg}$ = “unrealistic, blurry, low quality, out of focus, ugly, low contrast, dull, dark, low-resolution, gloomy” is generalizable across situations and able to cancel out $\delta\_\text{N}$. Would a better generator $g(\theta)$ be able to achieve the same effect, or train a model to estimate the bias $\delta\_\text{N}$?

We evaluated our approach with two generators $g(\theta)$, one that renders NeRFs, and the other is defined by $g(\theta) = \theta$. We used the same $p\_\text{neg}$ in both cases and observed that it generalizes across them. It is worth mentioning that we observed that incorporating $\delta\_\text{D}$ for the directly optimized image was more significant compared with adding it to the optimized NeRF. This is related to the NeRF representation which naturally provides regularization on the rendered images. Therefore, it is true that our implementation for $\delta\_\text{D}$ affects differently on different generators $g$. As mentioned in Section 5, training a model that estimates $\delta\_\text{N}$ is exactly the method introduced in ProlificDreamer. We believe that presenting ProlificDreamer with this formulation helps to gain more intuition over their method, and see this as one of our contributions.

We thank the reviewer for their detailed comments and questions. We address them below:

It remains somewhat challenging to discern the fundamental distinction from the negative prompt trick.

Stemming from the presented score decomposition, our fundamental distinction is that $\delta\_\text{C}$ and $\delta\_\text{D}$ should be used to optimize the parameters $\theta$. While it is clear how to calculate $\delta\_\text{C}$, separating $\delta\_\text{D}$ from $\delta\_\text{N}$ remains a challenge. We added a discussion about negative prompts in Appendix A.4, motivating our definition of $\delta\_\text{D}$ by relating it to the common practice of negative prompts. Please note that we see the exact definition of $\delta\_\text{D}$ as an implementation detail for applying our main observation.

In Sec. 5, the paper asserts that NFSD is notably more efficient than VSD, despite sharing a similar working mechanism... I would recommend providing quantitative evidence. It is conceivable that dropping the noise term could even speed up the convergence of ancestral sampling by using fewer optimization steps.

For 2D-image generation, trained on a single RTX2080 for 1K iterations, on average:

SDS - 2.9 [min] NFSD - 4.3 [min] and Prolific - 8 [min]

For NeRF generation, trained on a single A100 for 25K iterations, on average:

SDS - 3.1 [hr], NFSD - 3.5 [hr] and Prolific - 4.5 [hr]

Please note that we did not optimize our implementation. The noise term is undesirable for score distillation sampling. In contrast, for ancestral sampling, the score (comprising the noise term) is inherent to the step-by-step stochastic progression.

Further ablation studies are needed to validate the assertions put forth in this paper. In comparison to SDS, two terms have been omitted according to Eqs. 5 and 7: the noise prediction $\delta\_\text{N}$ and the noise ground truth $\epsilon$. However, it remains unclear which of these terms plays the most pivotal role in improving the final results.

In practice, the difference between SDS and our NFSD is that for $t < t\_s$ we do not subtract the added noise $\epsilon$, where $t\_s=200$, and for $t >= t\_s$ we subtract $\epsilon(z\_t, p\_\text{neg}, t)$ (instead of subtracting $\epsilon$). We added ablation studies in Appendix A.5, where we ablate the following:

• Subtracting $\epsilon(\mathbf{z}\_t, p\_\text{neg}, t)$ in all timesteps (instead of subtracting $\epsilon$ in SDS) ($t\_s=0$ in Figure 12)
• Not subtracting $\epsilon$ from SDS loss in all timesteps ($t\_s=1000$ in Figure 12)
• Change the threshold $t\_s$ (Figure 12)
• Using only $\delta\_\text{C}$ as the loss (leftmost column in Figure 13)

It is not evident how steering the update of SDS could alter the optimization objective. Providing a more rigorous and formal argument would deepen the contribution of this work.

We developed NFSD with the perspective of score function distillation, aiming to distill the score of a diffusion model for optimizing the parameters of rendered images. We observed that the score distilled in SDS includes a noise component, resulting in an averaging effect, leading to a local optimum that is overly smooth and lacks detail. NFSD attempts to avoid this noise component, allowing the optimization process to converge to a better local optimum. The optimization objective for diffusion time $t < t_s$ is similar to SDS, while for $t \geq t\_s$ it can be shown that $\nabla\_\theta \mathcal{L}\_{\textrm{NFSD}} =\nabla\_\theta \mathbb{E}\_{t}\left[w(t)\frac{\sigma\_t}{\alpha\_t}\textrm{KL}\left(q(\mathbf{z}\_t|g(\theta);y,t)\parallel \frac{1}{Z}\frac{p\_\phi(\mathbf{z}\_t;y,t)}{p\_\phi(\mathbf{z}\_t;y\_{neg},t)}\right)\right]$, where $Z$ is a normliazation factor. However, we find the optimized objective to lack intuitiveness and offer only little additional insight into NFSD. Therefore, we choose to focus on the score decomposition formulation.

The authors introduce Eq. 6 to estimate $\delta\_\text{D}$ Can the authors offer a rationale or justification for this approximation? Additionally, including visualizations that align with Fig. 3 would enhance the clarity and understanding of this proposal.

Separating $\delta\_\text{D}$ from $\delta\_\text{N}$ is challenging. We found that for small enough timesteps (e.g., $t<200$) $\delta\_\text{N}$ is negligible and therefore $\epsilon(\mathbf{z}\_t, \varnothing, t) = \delta\_\text{N} + \delta\_\text{D}$ provides a good enough approximation for $\delta\_\text{D}$. For other timesteps, we approximate $\delta\_\text{D}$ by $-\delta\_{C=p\_\text{neg}}$. The discussion we added in Appendix A.4 motivates the use of a negative prompt to define $\delta\_\text{D}$. Additionally, to show the effect of our defined $\delta\_\text{D}$ in a similar manner to Figure 3, we added Figure 10 where we extract $\delta\_\text{D}$ from the definition in Eq. (6) (two rightmost images on the bottom row).

Q6. Can the LoRA of ProlificDreamer be replaced with NFSD, while variational particle optimization is used?

The variational particle optimization proposed in ProlificDreamer is entangled with the LoRA variational model, as it is designed to model possible 3D representations (rather than one). Hence, applying NFSD for multiple particles can not be directly transferred from ProlificDreamer to our method. A possible approach for multiple particles was proposed by Kim et al. This is a general multi-particle approach for SDS-based methods, which can be combined with NFSD, but is out-of-the scope of our work.

Q7. Comparing the results of NFSD with those of SDS in terms of the number of function evaluations (NFEs) of diffusion models.

To the best of our knowledge, there isn't a method available for automatically determining the optimal number of iterations for SDS-based methods. In all our experiments, we use the same fixed number of iterations both for SDS and NFSD, resulting in an equal count of parameter updates for both methodologies. The term "NFEs" of diffusion models might be misleading here, given that the outcome of the diffusion model functions merely as a guiding factor without involving subsequent backpropagation through it. Theoretically, NFSD necessitates an additional evaluation of the negative prompt (in the worst-case scenario, as the negative prompt isn't utilized for diffusion times below $t\_s = 200$). Consequently, NFEs(NFSD) equate to 1.5 times NFEs(SDS). Increasing the iteration count did not yield improvements in the SDS results; On the contrary, it tended to introduce artifacts.

Q8. In Section 4, how can we assume that the score prediction on text conditions is also composed of $\delta\_\text{D} + \delta\_\text{N} + \delta\_\text{C}$, where $\delta\_\text{D} + \delta\_\text{N}$ is equal to the unconditional prediction?

Thank you for the clarification. Indeed, $\delta\_\text{C}$ is defined as $\epsilon\_\phi(\mathbf{z}\_t;y,t) - \epsilon\_\phi(\mathbf{z}\_t;\varnothing,t)$. Plugging $y=p\_{\text{neg}}$ into the definition directly gives $\epsilon\_\phi(\mathbf{z}\_t;\varnothing,t) - \epsilon\_\phi(\mathbf{z}\_t;y=p\_\text{neg},t) = -\delta\_{c=p\_\text{neg}}$. Therefore, applying the score decomposition on the terms before Equation (6) is unnecessary and we fixed it in the revision.

Q9. How is the negative prompt to estimate the domain term defined? How can we say that the image is from out-of-distribution, when the image can be estimated by text prompts of text-to-image models?

We use the same prompt for all experiments (3D and 2D). The prompt is simply a collection of synonym words for “low quality” image. Given a specific prompt, e.g. “a photo of a panda”, the score of the diffusion model will be in the direction that maximizes the likelihood of the sample given the prompt, i.e. it will point to a dense region of the conditioned distribution. The term out-of-distribution refers to this conditioned distribution. While low quality images with noticeable artifacts can be generated by a diffusion model (simply input the negative prompt), under a “normal” condition e.g. “a photo of a panda”, this is highly unlikely and thus we term it as out-of-distribution.

Q10. Can be simply using $s\delta\_\text{C}$ for the score distillation possible without $\delta\_\text{D}$? That is, is using $\delta\_\text{D}$ is necessary?

We added ablation studies in Appendix A.5, including an experiment for using $s\delta\_\text{C}$ only for 2D-latent in Figure 13. While the generated image is consistent with the prompt it contains artifacts, indicating that $\delta\_\text{D}$ term is necessary for obtaining high-quality results. We do stress that the effect of $\delta\_\text{D}$ can be achieved by other means of regularization, not only the diffusion model. For example, using additional losses for smoothness regularization can be used. Specifically in NeRF optimization, the parameter space and other losses (e.g. sprasity and opacity losses) can be used to regularize the results, serving as $\delta\_\text{D}$.

We thank the reviewer for their detailed comments and questions. We address them below:

W2. Thorough experiments to validate the effectiveness of NFSD are absent…There is no ablation study and quantitative result.

We added ablation studies in Appendix A.5 and quantitative evaluation in Appendix A.6.

Q1.Why the reduced CFG scale cannot produce diverse visual contents. In addition, can the authors provide the samples with different seeds and the same text prompts to show the diversity of generated contents?

Our approach actually sheds some light on the lack of diversity - it is not due to the high CFG. We hypothesize that the reason for the low diversity is rooted deeply in the process of score distillation. Unlike ancestral sampling, at each SDS iteration we add random noise to a rendered image, and we update the rendered image (or the generator parameters) independently of previous noises . Therefore, for a specific noised rendered image we follow some direction to optimize $p\_t(\mathbf{x}\_t)$. At the next step we again add (different) random noise and optimize $p\_t’(\mathbf{x}\_t’)$, which can yield a completely different direction. This results in an averaging effect (unlike ancestral sampling where each step corrects the previous as it is conditioned on it). Moreover, in our work, as in most score sampling methods, even the same diffusion time (i.e. similar noise level) can be used with different noises during the optimization process, yielding a different direction each time. We added a discussion alongside results obtained from different seeds in Appendix A.3.

Q2. In Figure 3, what is the diffusion timestep? ... show the results of $x\_{OOD}+\delta\_D + \delta\_N^{OOD}$ where $ֿ\delta\_\text{N}^{\text{OOD}}$ is the denoising score of $x\_{\text{OOD}}$ not $x\_{\text{ID}}$. I also suggest clarifying the notation of $\delta\_\text{D}$ and $\delta\_\text{N}$ in Figure 3, since the two scores are from different samples

The diffusion timestep used is $t=400$. Indeed, this should be specified in the paper. In Figure 3 we propose to visualize $\delta\_\text{N}$ and $\delta\_\text{D}$ by an engineered pair of $\mathbf{x}\_\text{ID}$ and $\mathbf{x}\_\text{OOD}$ images. We set $\delta\_\text{N}$ to be the denoising score of $\mathbf{x}\_\text{ID}$ as the image is in-domain and the prediction mostly just removes noise. We assume that $ֿ\delta\_\text{N}$ is shared between the in-domain and out-of-domain predictions, thus $\delta\_\text{D}$ is obtained as the difference between predictions of $\mathbf{x}\_\text{OOD}$ and $\mathbf{x}\_\text{ID}$. We explain this more clearly in the revised version. Regarding the term $\mathbf{x}\_\text{OOD}+\delta\_\text{D} + \delta\_\text{N}$, please note that this includes a noise direction term, thus the resulting image is noisy, as we show in Figure 10 in our revision.

Q3. Why do the prediction errors in Figure 4 (the second row) show a less-noisy map at t=1000?

Although the initial impression may appear counterintuitive, it's important to bear in mind that these are unnormalized disparities between the noise $\epsilon$ and the network's prediction $\epsilon\_{\phi}$—meaning, they haven't been adjusted in relation to the diffusion noise scheduler. Predicting the precise added noise to the image becomes notably more challenging when a small amount of noise is introduced. In contrast, at $t=1000$, the noised image predominantly consists of noise ($\epsilon \approx \mathbf{x}\_t$), simplifying the prediction process. as the image is heavily influenced by the noise itself.

Q4. How can we assume that the scale of the domain score is preserved? Is there any rationale that only $\delta\_N$ decreases over the backward process, while $\delta\_\text{D}$ preserves its scale?

You are correct that there is no guarantee that $\delta\_\text{D}$ will preserve its scale. Nonetheless, our main point is that since the noise to be removed is monotonically decreasing in the backward process (according to the noise-scheduler), the noise prediction $\delta\_\text{N}$ part of the score (corresponding to $\epsilon\_{\phi}$) is becoming increasingly negligible as diffusion time becomes smaller. In contrast, the magnitude of the $\delta\_D$ component depends both on the diffusion time and the current stage of the optimization process. Thank you for this important comment, and we have revised our paper accordingly.

Q5. How about the results of SDS, where its CFG adopts the same negative prompts as NFSD, described in Section 4?

We added a discussion about negative prompts in Appendix A.4. As we explain there, using a negative prompt in the CFG term mimics the effect of $\delta\_\text{D}$. Additionally, in Figure 13 we perform an experiment where we add a negative prompt to the CFG term of both SDS-100 and NFSD losses. As can be seen, the negative prompt improves the results of both SDS and NFSD.

(Continued in next reply)

I carefully read the authors' responses and the revised version of the paper including the Appendix.

Regarding to Q1, I encourage the authors to further revise the paper. In the last paragraph in Section 3, “However, high CFG coefficients are known to yield less realistic and less diverse results, as demonstrated in Figure 5, typically leading to over-saturated images and NeRFs.” can lead to misunderstanding of authors, high CFG coefficients are the root cause of low diversity even in SDS.

I have recognized the contributions of this paper, but the major concerns to reject this paper were some unclear explanations under assumptions and the lack of ablation study. However, the authors' responses clearly resolve most of my concerns, so I readily revise my final score to give an opinion to accept this paper.