Rethinking Score Distillation as a Bridge Between Image Distributions

1. The formulas of DDS and the proposed method are the same up to the source prompt.

A small yet fundamental difference is that DDS is a specialized method for editing that computes the source distribution direction based on a reference image instead of the current optimized image. That is, DDS computes the direction to the source distribution as $\epsilon_\phi (x_\text{ref}; y_\text{src}, t)$ as shown in the equation above line 155, while we compute it as $\epsilon_\phi (x_{θ,t}; y_\text{src}, t)$ as shown in the equation above line 186. This makes DDS incompatible with generation tasks since it requires a reference image. In addition, when there are reference images, it induces additional source distribution error as we explained in line 155.

2. The idea of using a negative prompt is likely not a substantial scientific

We emphasize in the global response that our main contribution is to provide a new view of the sources of errors in SDS. To demonstrate that improving the source distribution approximation error can improve generation quality, we propose a two-stage optimization process. Although using negative prompt is a common practice, the proper way to use it with SDS is not well explored. For example, as shown by the two-stage ablation experiment in Figure A1, always using negative prompt in SDS leads to divergence and poor geometry.

3. The authors identify two sources of errors - the 1st order approximation and the source distribution match - but they mostly only analyze the latter.

Please find the qualitative results of ISM in Figure A6. We notice that ISM generally outputs sharper results than SDS. However, it actually still uses single-step estimation to approximate the bridge. Although it uses multi-step inversion to produce a noisy sample instead of adding randomly sampled noise, it computes two epsilon directions using diffusion models to estimate the gradient, similar to SDS.

Instead, here we propose to resolve first order approximation error by solving the entire PF-ODE path to recover the dual bridge and estimate the endpoint of the bridge $\psi_{0, \text{tgt}}$ that is coupled with$\psi_{0, \text{src}}$. In this way, we obtain the most accurate gradient direction with minimal approximation error $\psi_{0, \text{tgt}} - \psi_{0, \text{src}}$. We refer to this approach as “full path”. However, solving the inversion ODE is not trivial. We noticed that the inversion can exaggerate the distribution mismatch error and cause the optimization to get stuck at a local optimum at the beginning of the optimization. Instead, the high variance of the single-step methods often demonstrates more robustness to different initializations. Therefore, we first perform the single-step score distillation optimization to obtain reasonable results before moving to solving the full bridge.

With this approach, we can now explore addressing both the first and second source of error. First source (linear approximation) with “full-path” and second source (source distribution mismatch error) with 2-stage. As shown in the table below, we find that using the “full-path”multi-step (mitigating error source 1) always outperforms the single-step methods, achieving a lower FID. However, the same trend does not fully transfer to the text-to-3D experiments. We observe that it typically introduces additional artifacts and makes the optimization less stable. We leave the best way of leveraging this gradient as a future research exploration.

4.a) The analysis of SDS does not explain the poor performance with $s = 1$.

SDS requires $s>>1$ to make $(\epsilon_\text{tgt}-\epsilon_\text{uncond})$ dominate the gradient direction. The function of the $\epsilon$ term effectively averages the optimized image by adding different Gaussian noise. As shown in Figure A5 in the uploaded PDF, a small value of $s$ would make the generation over-smoothed and lacking in details.

4.b) Unlike for SDS, the analysis for DDS omits the effect of CFG.

We believe that the effect of changing s is equivalent to changing the learning rate of the optimization, which controls the strength of the editing in DDS without changing the gradient direction. Therefore, we omit it in our gradient analysis.

5, 6. The related work is deferred to the appendix. Minor issues.

Thank you, We will address these in the revised version.

Q1. Why use CFG strength $s=40$ or 100 which is more than the range 3-15 in DDS?

s does not affect the direction of the gradient and can be absorbed into the learning rate. In their experiment, learning rate of 0.1 is used while we mostly use a learning rate of 0.01. Equivalently, we can use $w=4$ and learning rate=0.1 which results in a similar scale to the DDS hyperparameters.

Q2. Can you please clarify the definition of the score function in Eq. 2, compared with [25]?

We follow the notation in the DDPM paper. Suppose that $\mathbf{x}_t \sim \mathcal{N}(\sqrt{\alpha_t}\mathbf{x}_0; (1-\alpha_t)\mathbf{I})$, then the score can be computed as: $p(\mathbf{x}_t) = \frac{1}{\sqrt{2\pi (1-\alpha_t)}} \exp{(-\frac{1}{2(1-\alpha_t)}\cdot (\mathbf{x}_t - \sqrt{\alpha_t}\mathbf{x}_0)^T(\mathbf{x}_t - \sqrt{\alpha_t}\mathbf{x}_0))}$

$\nabla \log p(\mathbf{x}_t) = - \frac{1}{2(1-\alpha_t)}\cdot 2(\mathbf{x}_t - \sqrt{\alpha_t}\mathbf{x}_0)$

$= - \frac{1}{(1-\alpha_t)}\cdot (\sqrt{\alpha_t}\mathbf{x}_0 + \sqrt{1-\alpha_t}\mathbf{\epsilon}_t - \sqrt{\alpha_t}\mathbf{x}_0) \text{(Reparametrization trick with } \mathbf{\epsilon}_t \sim \mathcal{N}(\mathbf{0}; \mathbf{I}) \text{)}$

$= - \frac{1}{\sqrt{1-\alpha_t}} \mathbf{\epsilon}_t$

[25] uses a different noising function, thus a different score function.

1. The solution in section 2.4 was quite naive and heuristic. A major drawback is whether other models (MVDream, SDXL, PixArt, SD3, etc.) can understand the descriptions "oversaturated, smooth,…" well.

Although using negative prompt is a common practice in text-based diffusion models, how to use it with SDS is not well explored. As shown in the global response and Figure 1A, simply using it all the time during the optimization process leads to inferior results. Instead, the two-stage process consistently outperforms the single-stage baselines.

We show that the prompt generally works well with other models like MVDream and SDXL.

We perform our 2D generation experiment with MVDream using SDS and our two-stage optimization process. We use the same negative descriptors for the source prompt as proposed in our experiments with Stable Diffusion 2.1. As shown in Figure A4-a, we show that the two-stage optimization (bottom) produces more convincing colors and additional realistic high frequency details compared to an SDS baseline (top) for the same MVDream model.

We do the same with the SDXL base model, again using the same negative descriptors proposed for Stable Diffusion 2.1. In Figure A4-b, the two-stage optimization process (below) produces less saturation artifacts and more high frequency details than the SDS baseline (above). SDXL is known to perform poorly in the SDS setting and is therefore not commonly used, but we include these results to demonstrate the universality of the proposed optimization.

Despite the fact that these diffusion base models are trained on multi-view or high-quality images whose captions may not contain the proposed negative descriptors, we argue that the powerful pretrained text encoders that embed their prompts represent these artifacts well. For example, the embedding of “oversmoothed” is likely far from the embedding of “detailed” and close to other negative descriptors like “blurry”. Empirically, we find that the same negative descriptors work across base models without needing retuning.

2. The paper does not consider the Janus problem.

Our proposed framework is focused on analyzing and improving the diffusion gradient in SDS. MVDream addresses the Janus problem, which requires data priors on objects by training on multi-view data and conditioning generations with camera pose. These problems are orthogonal, and we show that our two-stage optimization process works well with MVDream to address both issues in Figure A4-a.

3. VSD has the strength of not only ensuring the quality of rendered images but also achieving sample diversity as the number of particles increases. Therefore, line 282 is not true, and the proposed method falls behind VSD in terms of sample diversity.

Thanks, we will make clear the advantage VSD has on sample diversity. In line 282, we mentioned, “...neither approaches have yet to achieve the quality and diversity of images generated by the reverse process.“ We did not intentionally mean that our two-stage optimization achieves better diversity than VSD, but to say, all SDS variants still induce lower diversity than the reverse process. The reason for that is still unclear. We hope that our analysis may inspire future research to better understand this problem.

For instance, to analyze the diversity issue within our framework, we notice that the ending point of the bridge is deterministically decided by the initial conditions and the ODE processes. When training a LoRA on all the particles, the loss encourages different ODE processes on individual particles. As a result, the LoRA module assigns slightly different directions to each particle and improves diversity. A recent study [1] reinforces this point by introducing a repulsive ensemble method to VSD. In general, this is beyond the scope of our paper. We will add this discussion in the paper.

[1] https://arxiv.org/abs/2406.16683

Question 1: What pre-trained diffusion model did you use in this paper?

We use stable-diffusion-v2-1-base for our experiments. We will make this clear in our revised version.

Question 2: I personally tried text-to-3D with SDS using Diffusion-DPO [A, B], which is a post-trained diffusion model for aesthetic quality, and failed to generate plausible geometry. Can this phenomenon be explained from the perspective of the Schrödinger Bridge problem?

Thanks for bringing up this interesting observation. We have also observed that models like SDXL fail to generate reasonable geometry in practice. Similar to Diffusion-DPO, SDXL filters its data using the aesthetic scores. Our hypothesis is that the images with high aesthetic scores overrepresent canonical views of the object. For example, the front view of a dog is often deemed to be more aesthetic than its back view. As a result, this induces an issue that the target distribution of SB heavily biases toward this canonical view. When applying these models to 3D generation, there could be more inconsistency across different views, which makes the optimization less stable.

Since most text-to-image models use a frozen text encoder, untrained negative descriptor embeddings are likely to be far from positive descriptor embeddings, as the authors mentioned in the rebuttal. However, a text-to-image model would not be able to output a good score or gradient for an untrained negative embedding. I believe this is why SDXL performs poorly in the SDS setting.

Considering that recent text-to-image models are increasingly trained on high-quality images, the proposed method seems difficult to apply beyond stable-diffusion-v2-1-base and MVDream, which is a post-trained version of stable-diffusion-v2-1-base.

While I see value in this paper's explanation of the behavior of SDS and its variants (including VSD as addressed in the rebuttal) from the perspective of the SB problem, I feel that the proposed method for addressing the issues with SDS needs further improvement and is not yet ready for publication in NeurIPS.

Therefore, I will maintain my current score. I remain open to further discussion.

Thank you for the detailed response. I apologize for the late additional question, but I have one more.

In the experiments conducted in the paper and rebuttal, including those with SDXL, within what interval were the diffusion timesteps sampled, and what was the weighting scheme for each timestep?

For example, DreamFusion uses sigma-weighted SDS, and VSD uses timesteps only within the range [0.5, 0] during the refinement stage.

Since the sampling distribution and weights of timesteps in diffusion models are known to be important [A], I am curious about this aspect. I’m curious if the effect of negative prompts might manifest at specific diffusion timesteps and weighting schemes.

[A] Kingma et al., Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation, NeurIPS 2023.

1. Some technical parts lack enough rationales, such as the negative prompt and two-stage optimization process. How do you choose these negative prompts?

We propose a two-stage optimization process motivated by our SB framework. To reduce the effect of the distribution mismatch error, we aim to more accurately describe the current distribution across different stages of optimization. While we experimented with more complicated approaches to achieve this, we found prompting to be unreasonably effective and much simpler. The negative prompts were selected based on the characteristic SDS artifacts, e.g., "oversaturated, smooth, pixelated,.." (rest in paper). This description does not require hand-crafting based on problem domains—it is kept fixed across all shown examples and use-cases.

We do not claim that these specific prompts are optimal or even of particular interest. To demonstrate this, we prompted GPT-4 through ChatGPT a single time to generate alternative negative prompts and ablated the results in Figure A2 with otherwise identical hyperparameters.

ChatGPT Prompt:

Here's a set of "negative prompts" to append to a text-to-image prompt that describe undesirable image characteristics: ", oversaturated, smooth, pixelated, cartoon, foggy, hazy, blurry, bad structure, noisy, malformed" I want to try a variety of them, please brainstorm many of roughly the same length

Generated Variations:

1. ", washed out, grainy, distorted, flat, smeared, overexposed, undefined, choppy, glitchy, dull"

2. ", low contrast, jumbled, faint, abstract, over-sharpened, muddy, cluttered, vague, jagged, poor detail"

3. ", soft focus, muffled, streaky, patchy, ghosted, murky, unbalanced, skewed, mismatched, overcrowded"

4. ", overbright, scrambled, bleary, blocky, misshapen, uneven, fragmented, obscured, chaotic, messy"

5. ", dull tones, compressed, smeary, out of focus, unrefined, lopsided, erratic, irregular, spotty, stark"

In addition, we show that the same negative descriptors work across different base models, such as MVDream. Since MVDream denoises four camera-conditioned images jointly, we treat the canvas of four images as a single optimization variable for the SDS gradient. In Figure A4-a, we compare the SDS baseline (top) to the proposed two-stage optimization (bottom), in which we generate more natural colors and detail. This is especially noticeable in the grass around the crocodile. Due to the space limit, we will add more results in the revised version.

We also ablate the proposed two-stage optimization process in the global response. As the source distribution keeps changing along with the optimization process, it is necessary to update the source distribution. We effectively achieve this by first running SDS (stage 1) then updating our source distribution with negative prompts to steer the optimization away from the artifacts (stage 2). In the global response, we show that if we start with such “negative” prompts from the beginning, which do not accurately describe the rendered images at initialization, it causes additional distribution approximation error and fails to generate a plausible object.

2. How do you choose the value of w? Is large w value similar with large cfg that causes artifacts?

We choose $w=40$ to produce a gradient on a similar scale as SDS with CFG scale $=100$. This is because, in text-to-3D, it is crucial to balance many other regularization losses on sparsity, opacity, and so on. Using a similar scale allows us to adopt the same hyperparameters to compare fairly with other SDS variants. In addition, w is different from CFG since SDS also incorporates a term with sampled Gaussian noise, and the CFG term needs to be large enough to make the $(\epsilon_\text{tgt}-\epsilon_\text{uncond})$ dominant. Instead, $w$ simply scales the gradient, unlike the CFG scale in SDS, which changes the direction of the gradient.

We also notice that $w$ is relatively robust and gives similar results in a range. Instead, when the CFG scale is small in SDS, the averaging effect of Gaussian noise dominates and creates oversimplified 3D objects. When $w$ or CFG scales are large, with other loss weights intact, the optimization becomes unstable and may diverge. See Figure A5 in the uploaded PDF for this qualitative comparison.

3. Missing comparison in text-to-3D with more competitive baselines.

Thank you for the suggestion. We ran a comparison with Fantasia3D, Magic3D, and CSD through a drop-in replacement of SDS with our method. We did not compare with NFSD as they did not release the official code and we empirically found that its results resemble SDS results. Specifically, all three methods optimize a textured DMTet, which is initialized from an SDS-optimized NeRF, using SDS or CSD for 5k or 10k iterations. We replace the SDS or CSD stage of these approaches with the two-stage optimization motivated by our framework. Just like our text-to-3D NeRF experiment, we perform the first stage for 60% of iterations and the second stage for 40% of iterations. Note that we keep all the other hyperparameters the same, which were tuned for the baselines, not our method. This replacement leads to the same optimization time as the original methods. For Fantaisia3D and Magic3D, we use threestudio for fair comparison (Magic3D does not have code available) and the default prompts, which are generally believed to work the best with this reimplementation. For CSD, we use the official implementation. As shown in Figure A3, our method improves the visual quality of all the methods by reducing the oversaturated artifacts of SDS and improving the details.