Mean-Shift Distillation for Diffusion Mode Seeking

We thank the reviewer for the positive evaluation and for recommending acceptance. We are glad they recognize our contributions. Below, we answer questions raised in the review:

While the results of SDS are well represented in the paper, SDI is not as widely shown. E.g. it is not displayed in tables 1 and 2 and Fig 3.

In our rebuttal to Reviewer cVvL we extend Table 1, 2, and 3 with comparisons to SDI and a new baseline VSD [1], where missing. We qualitatively compare with SDI on the fractal dataset in Fig. 2. We will also extend it, along with VSD, to the spiral and pinwheel dataset shown in Fig 3 in our revision.

The z_t in eq. 9 is understood to be the noisy version of x_t (or just noise in the limit) but I do not think it is properly defined in the paper.

Thank you for highlighting this. We will define the term in our revision.

Is the mode seeking behavior always desirable?

The desirability of mode seeking varies between applications. When trying to directly sample images from the trained model, it is true that we wish to sample from the full variety of the distribution instead of getting only the mode. Methods like DDIM aim for this. On the other hand, when we are optimizing an image (or using the image as a proxy to optimize, e.g. NeRF parameters), any gradient-based optimization will converge to a set of sparse points - local extrema - where the gradients are zero (if it converges at all). This is the intended use-case for SDS, SDI, and our method, and in this case, it is not possible in general to have the optimization process converge to a distribution of points. Given that, the best we can guarantee is that the points the process converges to are aligned with the distribution. Mode-seeking is our proposed way of achieving that.

Highly saturated results in Figure 5

We have observed this “washed out” look in the image and 3D results, despite using a low guidance scale (CFG=7.5). We believe this to be an artifact of using CFG in the pipeline, because these effects appear in both our method and the baseline and are consistent with what has been observed in previous literature [2][3]. The analysis on the fractal toy dataset by Karras et al., which we also use in our paper, suggests that this is the visual effect of “sharpening” the modes of the distribution that can also be observed in these 2D test cases.

Figure 8 shows that the guidance in limited interval is important for the results. Would a similar trick also be effective for SDI?

SDI only performs one denoising step in each optimization iteration. It is not straightforward to apply guidance in a limited interval without substantial modifications to their algorithm.

How many samples per prompt were used in the experiments

For each method, we generate 500 samples per prompt. With a total of 10 prompts, this gives us a total of 5,000 images per method.

References:

[1] Wang Z, Lu C, Wang Y, et al. “Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation”. Advances in Neural Information Processing Systems, 2023.

[2] Ho and Salimans, “Classifier-Free Diffusion Guidance”, 2021.

[3] Saharia et al., “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding”, 2022.

We thank the reviewer for the thoughtful and constructive review. We are glad they recognize our contributions. Below, we answer questions raised in the review:

Comparisons with other guidance techniques and variance-reduction methods.

Guidance. In low-dimensional settings (eg, our toy experiments), our method can recover the modes and reconstruct the data distribution well without any guidance (See Fig. 1). This is aided by the fact that the conditional score estimates parameterized as $\epsilon_\theta(z_t, c)$ (predicted noise from the pre-trained network) is good by itself, without guidance i.e. $\tilde{\epsilon}_\theta (z_t, c)$. Empirically, we observe that without guidance, ancestral sampling techniques like DDIM produce samples that lie on the data manifold, albeit with few outliers (See Fig. 2).

This is not the case in the high-dimensional setting with experiments on Stable Diffusion. Here $\epsilon_\theta(z_t, c)$ samples are noticeably bad and are predominantly outliers. Currently, the best fix is to augment these noise estimates with guidance to produce $\tilde{\epsilon}_\theta (z_t, c)$, the strategy prevalent in sampling algorithms. We inherit these practices when performing distillation.

We recognize the existence of alternative guidance techniques to CFG, like Autoguidance (Karras et al., 2024). As these methods pair well with DDIM (and other ancestral sampling techniques), we believe the benefits will extend to distillation-based methods like ours. Ultimately from the perspective of distillation, different guidance simply changes the shape of the output distribution but does not fundamentally change the mechanics of diffusion. We used CFG in all our experiments as it is more widely used, has hyperparameters (guidance scale) that have been more rigorously tested by the community, and was used in all our baselines (SDS and SDI).

We extend Table 2 with comparisons between two guidance techniques, CFG vs Autoguidance (Karras et al., 2024). We add a new baseline VSD [1] (suggested by Reviewer cVvL) and our original baseline SDI. We show just the fractal dataset.

Please see the rebuttal to Reviewer cVvL for additional comparisons with VSD.

Left: learned denoiser with CFG / Right: learned denoiser with Autoguidance. Best performance among distillation-based methods is highlighted in bold.

Variance reduction. One of the variance reduction methods we cite, SteinDreamer [2], propose using control variates to minimize the excessive variance present in SDS. This excessive variance comes from the randomly sampled noise added to the target (or rendered images). Unlike SteinDreamer, SDI finds a better approximation of this desired noise term, eliminating one of the root causes of the excessive variance instead of compensating for it later. We believe this makes SDI a better candidate among previous variance reduction methods. Moreover, the official implementation for SteinDreamer is not yet publicly available. We will be happy to make comparisons as soon as an official implementation gets published. We would be happy to add comparisons to any other specific work the reviewer has in mind.

The impact of integration heuristics and their ad-hoc nature, additional exploration of robust integrators or dynamic bandwidth strategies.

We tried robust integrators in toy settings, and while they do indeed improve numerical stability, the additional score model evaluations they require make them impractical in a higher-dimensional setting. Higher-order numerical solvers like PNDM [3] did not improve numerical stability in our setting. We observed limited guidance to be the simplest to implement and most impactful. Adaptive bandwidth strategies have been proposed for Kernel Density Estimation in the past [4]. We hope to extend these techniques for our use case in the future.

Does the improved stability substantially reduce the required number of iterations, or does it primarily improve final fidelity?

Yes, the improved stability reduces the number of optimization iterations while maintaining the fidelity of the output.

References:

[1] Wang Z, Lu C, Wang Y, et al. “Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation”, 2023.

[2] P. Wang et al. “Steindreamer: Variance reduction for text-to-3d score distillation via stein identity”, 2023.

[3] L. Liu et al. “Pseudo Numerical Methods for Diffusion Models on Manifolds”, 2022.

[4] D. Comaniciu et al. “The Variable Bandwidth Mean Shift and Data-Driven Scale Selection”, 2001.

We thank the reviewer for the feedback. We hope the following will resolve any confusion regarding our contribution and evaluations:

• We would like to emphasize that the goal of our work is to improve mode seeking and achieve better distillation for diffusion models, not to improve metrics for image generation, such as FID, or metrics for evaluating image-text alignment, such as CLIP similarity. Neither FID nor CLIP-based similarity are baselines for our method, but rather the metrics by which we compare our method to the baselines (SDS, SDI, VSD), as in e.g. Table 3. FID and CLIP-based similarity are widely used evaluation metrics for image generative models in all state-of-the-art research works. We also cite the library we used (torchmetrics from PyTorch Lightning).

• We would also like to point out that we evaluate on four datasets: three synthetic datasets demonstrating that current distillation methods fail even in toy distributions and the pre-trained text-to-image StableDiffusion-XL model for the practical setting. The latter was used in all our baselines (SDS, VSD, and SDI). Our text prompts are borrowed from DreamFusion (Poole et al., 2022).

• Regarding the three points raised in the theoretical claim part of the review, we will clarify in the paper:

(1) $t$ is the time step used in denoising diffusion, and $\alpha(t)$ is the time-dependent weighting function. Both these terms are standard notation in the denoising diffusion literature [1]. Eq. 1 is the SDS gradient from DreamFusion, where $t$ is randomly sampled.

(2) $p(y)$ is the data density, and $p(x)$ is the smoothed data density after convolving the data density with the Gaussian Kernel.

(3) $K_\lambda$ is the kernel used in mean-shift clustering, which determines the weight of nearby points for re-estimation of the mean. There have been various choices for kernels in mean-shift clustering, e.g., Gaussian, Epanechnikov, flat kernel [2]. Our analysis focuses on the case $K_\lambda (x)$ is the Gaussian kernel used in Eq. 2. In Eq. 7, we write the general expression for mean-shift updates for any kernel function.

References:

[1] Denoising Diffusion Probabilistic Models. J. Ho, A, Jain, P. Abbeel. NeurIPS 2020.

[2] https://en.wikipedia.org/wiki/Mean_shift

We thank the reviewer for recognising our contributions. We acknowledge the feedback regarding our notations.

1 Writing clarification.

As we cannot update the submission, we provide them below. We believe these are minor corrections and will incorporate them in our revision.

How are losses in Algorithms 1 and 2 derived?

Score distillation methods (SDS, SDI, Ours) provide gradients of the loss w.r.t target parameters but not the loss itself (see Appendix A.4 in DreamFusion (Poole et al., 2022)). Losses are given implicitly. In our toy experiments, we reconstruct the loss function by integrating finite differences and visualizing them in Fig. 1, 2, and 3. Section 4.2 provides more details.

2 Additional baselines.

We compare with VSD (PriolificDreamer, 2023) below. Note the following about VSD:

• VSD claims (Appendix C.3) SDS's mode-seeking causes over-saturated, low-diversity results. We find SDS is not mode-seeking; its high variance and bias from modes are the reasons for poor results. See our rebuttal to Reviewer 6KVp on why mode-seeking is desirable for distillation.
• VSD needs task-specific fine-tuning (via LoRA or training U-Net) to estimate the variational score $\nabla_{z_t} \log q^{x}_t$. Our method needs no fine-tuning, and it’s unclear how VSD’s approach generalizes across domains.
• SDI outperforms prior works like VSD in text-to-3D generation. SDI is also easily extensible to any domain, making it a strong baseline.

We extend Table 1 comparing ideal and learned denoiser. We add new baseline VSD and original baseline SDI.

Left: ideal denoiser / Right: learned denoiser.

Efficiency ($\uparrow$).

Similarly, we extend Table 2. Best performance among distillation-based methods in bold.

Note: With ideal denoiser, the efficiency of our method is comparable to SDI. With learned denoiser, SDI has better efficiency. Despite this, we produce better samples.

Similarly, we extend Table 3.

We accompany Table 3 with additional qualitative comparison for text-to-2D: https://imgur.com/a/BWpHde5.