Thank you for the thorough review and recommendations. We provide clarifications below.

Numerical error in choices of \kappa

Thank you for highlighting this effect in the review, we believe a more detailed explanation will indeed benefit the paper. We touch on the intuition around this question in the general response: it is not exactly true that \kappa with a lower error in eq.8 leads to a better 3D generation. It is true, however, that a more precise solution to eq.8 aligns score distillation more closely with DDIM. Other factors like initialization, view consistency, and DDIM’s generative ability also impact 3D quality. Below we address each of the particular questions in detail.

1. The green configuration in Figure 10 ( $\gamma\_\text{fwd}=1, \gamma_\text{inv}=1$) indeed has the lowest error in eq.8. This is consistent with [1, 2], which claim that DDIM inversion accumulates numerical errors in a conditional generation [1] while achieving almost perfect inversion in an unconditional generation [2]. When we use unconditional generation together with unconditional inversion, we succeed at closely matching the generation guidance to a DDIM trajectory. This trajectory, however, is unconditional. Thus the final 3D shape does not correspond to the desired (or any) prompt: exactly what we see in Figure 10.

2. As it was correctly noted, the purple and red configurations in Fig.10 (conditional vs. unconditional inversion) exhibit the same approximation error. In practice, both configurations yield detailed 3D shapes (also Fig.10). However, as described in the general reply (see “Guidance Term”), a combination of unconditional inversion and conditional generation leads to the accumulation of saturation in the final shape. Please also refer to Fig.5 of the rebuttal PDF for the visualization of this effect.

3. Regarding the error of SGD and Random noise in Figure 9, the two lines are mostly within one sigma of each other (please note the translucent areas behind each line signifying standard deviation). The numerical error on the right is computed and averaged across 10 prompts. We did not notice any improvement in 3D generation quality nor in the error induced in eq.8 by performing the stochastic gradient descent on the noise term. The SGD optimization of 10 steps was initialized to a randomly sampled noise image, which might explain the resemblance of the generated shapes to the randomly sampled noise.

Comparison with StableSD

Thank you for providing a useful reference. We would like to point out that we already have a qualitative comparison with Stable Score Distillation in the appendix (StableSD in section E3).

Mathematically, the idea behind this line of work is that the mode-seeking term in score distillation ($\\epsilon\_\theta^t$) exhibits high variance. Thus the second term ($- \epsilon$ in SDS or the LoRA in VSD[3]) reduces the variance of the first. A similar idea was also introduced in SteinDreamer [4], where authors numerically analyze variance in SDS and VSD. Both StableSD and SteinDreamer propose to use control variates to minimize the noisiness of the guidance. Please note, that in both of these works, the excessive variance is compensated by subtracting another noise term by finding coefficients that allow for a better correlation.

In our work, however, we explain the second term in SDS not as a control variates, but rather as a projection of the previous step in a re-parametrized DDIM trajectory. Moreover, our variance reduction comes not from using control variates, but from finding a better noise to add to the rendering in the first place. We show that the excessive variance in $\epsilon\_\theta^t(x_t)$ comes from the fact that $x_t$ was obtained using a randomly sampled noise, while our analysis reveals that it should follow a very particular structure (eq.8). By finding a better approximation of the desired noise term we eliminate the root cause of the excessive variance instead of compensating for it later.

Unfortunately, the official implementation of Stable Score Distillation is not yet publicly available (nor for SteinDreamer) and we were not able to reproduce their reported results. We will be happy to augment our Table 1 with a quantitative comparison with StableSD as soon as an official implementation gets published.

[1] Ron Mokady et al. “Null-text inversion for editing real images using guided diffusion models”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 6038–6047.

[2] Daiki Miyake et al. “Negative-prompt inversion: Fast image inversion for editing with text guided diffusion models”. In: arXiv:2305.16807 (2023).

[3] Wang, Zhengyi, et al. "Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation." Advances in Neural Information Processing Systems 36 (2024).

[4] Peihao Wang et al. “Steindreamer: Variance reduction for text-to-3d score distillation via stein identity”. In: arXiv:2401.00604 (2023).

Thank you for the thoughtful and constructive review. Below we address the points raised in the review:

Inversion with negative CFG

Thank you for pointing this out. We agree that adding this intuition to the main paper will benefit the clarity. Below we support the intuition described in the general response with theoretical and empirical analysis.

Experimental analysis of negative CFG

In the general responce, we base our intuition on the fact that negative CFG attracts generation to the mean of the training dataset. To demonstrate this, we perform a simple experiment of 2D generation with DDIM for different prompts and CFG values of +/-7.5 for each (Fig.3 of the rebuttal PDF). Instead of generating images with opposite prompts, negative CFG directs images to the same class (real estate). This happens because the “real estate” happen to be in the middle of the embedding space, where the process is attracted to.

Theoretical justification

The effect of attraction to the mean can also be shown more formally. Let’s denote:

$X-\text{full dataset},X_y-\text{prompt conditioned dataset}$

From the optimal denoiser perspective and notations from [1]:

$\hat{\epsilon}\_\theta^t(x_t,\varnothing)=\sum_{i\in X}w_i(x_t)x_i$

$\hat{\epsilon}\_\theta^t(x_t,y)=\sum_{j\in X_y}w_j(x_t)x_j$

Now, rewriting the CFG definition:

$\epsilon_\theta^t(x_t,y)=(1-\gamma)\sum_{i\in X}w_i(x_t)x_i+\gamma \sum_{j\in X_y}w_j(x_t)x_j=$

$=(1-\gamma)[\sum_{i\in X/X_y}w_i(x_t)x_i+\sum_{k\in X_y}w_k(x_t)x_k] + \sum_{j\in X_y}w_j(x_t)x_j=$

$=(1-\gamma)\sum_{i\in X/X_y}w_i(x_t)x_i+\sum_{j\in X_y}w_j(x_t)x_j$

In the final formula, image generation is attracted to the images in the dataset relevant to the prompt (independently from CFG), and also is repulsed or attracted to the mean of the rest of the dataset, depending on the sign of the CFG. Consequently, negative CFG values attract the image to the mean of the dataset and repulse it from the mean on DDIM inversion.

As mentioned in the general response, our intuition is that the repulsion from the mean of the dataset in the case of negative CFG serves as a regularization for the noise term and allows to compensate for the bad initialization.

Comparison with ISM

Please see the general response above for a detailed comparison between our algorithm and ISM.

Additional Metrics

Thank you for the useful suggestion. We agree that reporting a metric that reflects human preference will make the paper stronger. We adopted the suggested ImageReward. We are using the official implementation and report the rewards directly. Following the same protocol as in Table 1 of the main paper we average the metric across 50 views for each of the 43 prompts. We report the obtained metrics below and plan to augment Table 1 with it.

Additional comparisons

NFSD

While we compare our method with NFSD qualitatively, we omit quantitative comparison. The official implementation of NFSD is not yet available and we were not able to reproduce its results. We will be happy to add the numerical comparison to Table 1 as soon as an official implementation becomes publicly available.

Additional qualitative comparisons

In response to more qualitative comparisons apart from the two prompts in the main paper, we would like to kindly point out the numerous qualitative comparisons in the appendix. For this, we follow a protocol adopted across the field: for each baseline, we provide renderings reported in the corresponding papers. To the best of our knowledge, we have reported all prompts and images reported in the corresponding baselines. If considered necessary, we will add visual comparisons on all baselines, using open-source implementations.

Evaluation of diversity

We agree that additional evaluation of diversity will enhance our paper. Please find examples of generations for different seeds and prompts with our method in Fig.4 of the rebuttal PDF. Our generations exhibit a certain degree of diversity but are mostly the same at a coarse level. We did not notice lower diversity compared to SDS[2] or VSD[3].

Additionally, Fig.6 of the rebuttal PDF illustrates interesting behavior, where different human-related prompts generate similar faces. We are not sure if this is a limitation of the underlying diffusion model or our distillation algorithm and plan to address this in future work.

Noise in the background of x_t

We agree with the intuition about initial images being out-of-distribution. Indeed, it is rare to find perfectly monotonic backgrounds in natural images. It is unclear if the intermediate $x_t$ is unlikely under the marginal distributions or $x_t$ is close to a mode (in the same way as a completely uniform image has the highest probability density in the i.i.d Gaussian).

In our experiments, however, we did not notice any issues caused by this behavior. Moreover, as we mention in Appendix A.2, we adopt a technique from [4] to fight the Janus problem: we increase the entropy of the process by mixing a small amount of noise during inversion. This helps increase the flexibility of the model to generate non-frontal views. At the same time, it removes the monotonic regions in the noise samples.

[1] Permenter, Frank, and Chenyang Yuan. "Interpreting and Improving Diffusion Models from an Optimization Perspective." arXiv preprint arXiv:2306.04848 (2023). [2] Poole, Ben, et al. "Dreamfusion: Text-to-3d using 2d diffusion." arXiv preprint arXiv:2209.14988 (2022).

[3] Wang, Zhengyi, et al. "Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation." Advances in Neural Information Processing Systems 36 (2024).

[4] Peihao Wang et al. “Taming Mode Collapse in Score Distillation for Text-to-3D Generation”. In: arXiv:2401.00909 (2023).

We thank r6QS for the helpful review. We will address their comments in the final version and clarify some points below:

Gray color shift

We agree that results in the main paper might seem gray or sometimes have a red/green tone. We explain this by the dark background color that is generated in the NeRF. Our algorithm is implemented on top of threestudio’s SDS, where apart from the object-level NeRF, the shapes are also allowed to have a background. Typically the generation converges to darker tones of the background, which biases the diffusion process to reflect illumination in the rendered 3D shapes. For visualization in the original submission, we simply cropped the object from the final renderings. When on a white background, this removes the illumination context and makes the shapes look grayer.

To further illustrate this we provide renderings with their original background in Fig.7 of the rebuttal PDF. Additionally, Fig. 1 (bottom row) of the rebuttal PDF shows that when our algorithm is reimplemented in the ISM’s code base, where the background is fixed to be white, our final renderings have brighter, white colors reflecting the global illumination.

In summary, the diffusion model bakes in the background illumination into the texture of the shape. One possible solution that was suggested in prior work consists of randomly changing the background colors to avoid overfitting to a single one. In this case, however, we observed a decrease in the quality of the renderings.

Computational Cost

We agree that, unfortunately, DDIM inversion requires additional forward passes with the denoiser, which increases computational cost. However, we would like to clarify a few points:

1. Despite the additional cost required for obtaining the noise sample, our algorithm is typically more stable and requires fewer steps to converge than other state-of-the-art methods, all while having similar quality in the results. The time per generation in Table 1 shows that our algorithm is still 2-3x times faster than state-of-the-art since it takes fewer steps.

2. Our ablation in Fig.11 demonstrates that 10 steps are enough for DDIM inversion to obtain good-quality 3D generations. In practice, one NeRF update is so slow that after the 10 additional inferences with the denoiser per optimization step we see only a 2x slow-down. This effect will be more noticeable in future work involving Gaussian Splatting, where 3D shape update is not a bottleneck anymore.

Differences with ISM

For a detailed comparison of our algorithm and ISM, please refer to the general response above and to the rebuttal PDF attached to the response.

Janus problem

Thank you for the question. Without additional regularization, our algorithm has a stronger tendency towards the Janus problem compared to SDS. SDS uses prompt augmentation to avoid this problem, adding phrases like “front view” and “back view.” As in Appendix A2, our algorithm’s stronger bias toward the Janus problem can be explained by the property of the diffusion models to ignore parts of the prompt when CFG is not high enough (see Fig.3 in the main paper). The proposed improvements of finding a better noise term allow us to reduce the CFG values from 100 in SDS to the typical 7.5, which leads to a weaker view-dependent prompt augmentation.

To additionally prevent the Janus problem, we add entropy as in [1], and orthogonalize the augmented prompts as in [2]. In rare cases, our algorithm still might produce multiple faces in one generation. This limitation is reported and illustrated in the appendix E1.

[1] Peihao Wang et al. “Taming Mode Collapse in Score Distillation for Text-to-3D Generation”. In: arXiv:2401.00909 (2023).

[2] Mohammadreza Armandpour et al. “Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate Janus problem and beyond”. In: arXiv:2304.04968 (2023).