Closed-Form Diffusion Models

We thank the reviewer for their insightful comments, which will improve the quality of our manuscript, and are glad they share our interest in closed-form diffusion models that generalize. We have uploaded a revision to OpenReview, where the reviewer will find that we have incorporated many of their suggestions. In this revision, we have typeset our relevant changes in teal. We respond to the reviewer's comments below.

Inductive bias of our method.

As the reviewer points out, our method assumes that the barycenters of nearby training points lie within the support of the data distribution. This is a reasonable prior if the data distribution is supported on a manifold whose curvature isn’t too large relative to the number of training samples. As a classical example, Roweis and Saul’s “locally linear embedding” imposes a very similar prior on its inputs.

While this assumption often isn’t true of the image manifold in pixel space (barycenters of images typically don’t look like natural images), by mapping our training data to a latent space with an appropriately-regularized autoencoder, we obtain a latent data manifold whose structure is more favorable to our method. This explains why our method is able to generate plausible CIFAR samples when operating in the latent space of an autoencoder with a tight bottleneck. We have updated our discussion in Section 6.4 to clarify this point.

For future work, we will explore training an autoencoder that enforces local linearity in the latent space; see e.g. Section 4.2 in “Learning Signal-Agnostic Manifolds of Neural Fields” by Du et al. (2021). We hope this strategy will yield a latent data manifold with a locally-linear structure that is highly favorable to our method and produce natural-looking decoded samples. We have updated our conclusion to include this future direction.

Alternative sampling processes.

Naively-implemented “generative mixup” would draw pairs of training points uniformly from the training set and then output some data-independent convex combination of these points. This amounts to sampling from the convex hull of the training set, and is not a suitable inductive bias for most target distributions. One exception is when the target distribution is log-concave; in this case, convex combinations of pairs of target samples have log-likelihood at least as large as one of the endpoint samples. For example, all Gaussian distributions are log-concave. Our fitted Gaussian baseline in Section 6.4 (which performs very poorly) demonstrates that this log-concavity prior is unsuitable for image generation.

On the other hand, directly sampling tuples of $M$ training points with probability depending on their variance and returning their barycenters is challenging, because there are $N^M$ such tuples for training sets of size $N$.

Instead, we have implemented and experimented with a new sampler in Appendix D.4 of the revision, which implements the one-step sampling procedure described in Proposition B.1. This is an extreme case of our existing strategy for reducing the number of sampling steps using a warm start from an unsmoothed CFDM, and uses Gumbel perturbations $u_{i,m}$ to the distance weights in Equation (4) for analytical convenience. Proposition B.1 shows that this method is equivalent to drawing a batch of query points $z_{S-1}$ lying near the support of the data distribution $\rho_1$, drawing $M$ training points $x_i$ with probabilities $\pi^i_\sigma = \textrm{softmax}\left(-\frac{1}{\sigma} \|z_{S-1} - x_i \|^2 \right)$, and then returning their barycenters.

As expected, this method achieves significantly higher sample throughput than our existing approach ($\sim 4$ times faster for CIFAR generation in latent space). While the decoded samples are reasonable, the speedup comes at a cost to sample quality: Many of our decoded samples retain the appearance of superpositions of training samples.

Effect of $M$ on CFDM samples.

We have included an experiment in Appendix E of the revised manuscript demonstrating the impact of $M$ on our method’s samples.

Extension to conditional generation.

In future work, we intend to explore using our method for conditional generation by using Dhariwal and Nichol (2021)'s classifier guidance, which would amount to augmenting our velocity field (7) with the gradient of a pretrained classifier. We have updated our conclusion to include this future direction.

Conclusion.

We thank the reviewer for their comments and hope that we have adequately addressed their concerns. If they are satisfied with our answers and our revision, we respectfully request that they raise their score for this paper. Otherwise, we would be pleased to continue this discussion during the reviewer-author discussion period.

Dear Reviewer pdyz -- as the discussion period will close in a few days, we would greatly appreciate if you would take a look at our response to your review soon and let us know if you would like to see further changes to our manuscript. We look forward to addressing your remaining concerns before the end of discussion period. If our response was satisfactory, we ask that you consider raising your score for our submission. Thank you for your time.

We thank the reviewer for their comments, which we believe will improve the quality of our manuscript. We have uploaded a revision to OpenReview, where the reviewer will find that we have incorporated many of their suggestions. In this revision, we have typeset our relevant changes in teal. We respond to the reviewer's comments below.

Relationship to the generalization of diffusion models.

Our goal is to design a diffusion model that generalizes controllably while requiring no training and being efficient to sample. While our method is indeed inspired by recent work studying the generalization of diffusion models, our contribution is not a rigorous study of the generalization of existing (neural) diffusion models, but rather introducing a new class of diffusion models that do not require trained approximations to the score function.

In particular: Pidstrigach (2022) shows that a score-based generative model (SGM) can generalize only if the model score incurs unbounded approximation error relative to the true score; otherwise, the model memorizes its training data. We combine this strong result with the observation that neural networks tend to learn overly smooth approximations to their target function and propose a simple, training-free method to induce the error that Pidstrigach (2022) shows to be necessary, in a form similar to that of neural SGMs.

We have updated our introduction and our lead-in to Section 4 to clarify these points.

Clarifying our contribution.

In addition to smoothing the closed-form score and showing that this promotes generalization, we introduce two approximation techniques to render this method tractable: Initializing our sampler with unsmoothed CFDM samples at $T>0$ and a nearest-neighbor-based score approximation. These approximations enable our method to achieve very fast inference speeds (for example, 138 latents/sec on our CIFAR experiments) while running on a consumer-grade laptop without a dedicated GPU.

We have updated our introduction to clarify this point.

References for closed-form score formula in Section 3.

We have updated our discussion in Section 3 to include several references for this formula. In particular, Appendix B.3 of “Elucidating the Design Space of Diffusion-Based Generative Models” by Karras et al. (2022) includes a full derivation of the closed-form score formula; their “ideal denoiser” corresponds to our function $k_t$.

Additional benchmarks and metrics.

In our revised manuscript (Table 1), we have also reported our image generation results in terms of the kernel inception distance (KID). Furthermore, in Appendix D.4 of the revision, we implement and experiment with an alternative sampler. This sampler implements the one-step procedure described in Proposition B.1 and is an extreme case of our existing strategy for reducing the number of sampling steps, using a warm start from an unsmoothed CFDM and Gumbel perturbations to the distance weights in Equation (4) for analytical convenience. Proposition B.1 shows that this equivalent to drawing a batch of query points $z_{S-1}$ lying near the support of the data distribution $\rho_1$, drawing $M$ training points $x_i$ with probabilities $\pi^i_\sigma = \textrm{softmax}\left(-\frac{1}{\sigma} \|z_{S-1} - x_i \|^2 \right)$, and then returning their barycenters.

As expected, this method achieves significantly higher sample throughput than our existing approach ($\sim 4$ times faster for CIFAR generation in latent space). While the resulting outputs are reasonable, the speedup comes at a cost to sample quality: Many of our decoded samples retain the appearance of superpositions of training samples.

Conclusion.

We thank the reviewer for their comments and hope that we have adequately addressed their concerns. If they are satisfied with our answers and our revision, we respectfully request that they raise their score for this paper. Otherwise, we would be pleased to continue this discussion during the reviewer-author discussion period.

Dear Reviewer P7kJ -- as the discussion period will close in a few days, we would greatly appreciate if you would take a look at our response to your review soon and let us know if you would like to see further changes to our manuscript. We look forward to addressing your remaining concerns before the end of discussion period. If our response was satisfactory, we ask that you consider raising your score for our submission. Thank you for your time.

We thank the reviewer for their comments, which we believe will improve the quality of our manuscript. We have uploaded a revision to OpenReview, where the reviewer will find that we have incorporated many of their suggestions. In this revision, we have typeset our relevant changes in teal. We respond to their comments below.

Additional baselines.

In Appendix D.4 of the revision, we implement and experiment with an alternative sampler which serves as an additional baseline for our model. This sampler implements the one-step procedure described in Proposition B.1 and is an extreme case of our existing strategy for reducing the number of sampling steps using a warm start from an unsmoothed CFDM and Gumbel perturbations to the distance weights in Equation (4) for analytical convenience. Proposition B.1 shows that this equivalent to drawing a batch of query points $z_{S-1}$ lying near the support of the data distribution $\rho_1$, drawing $M$ training points $x_i$ with probabilities $\pi^i_\sigma = \textrm{softmax}\left(-\frac{1}{\sigma} \|z_{S-1} - x_i \|^2 \right)$, and then returning their barycenters.

As expected, this method achieves significantly higher sample throughput than our existing approach ($\sim 4$ times faster for CIFAR generation in latent space). While the resulting outputs are reasonable, the speedup comes at a cost to sample quality: Many of our decoded samples retain the appearance of superpositions of training samples.

Additional metrics.

In our revised manuscript, we have also reported our image generation results in terms of the kernel inception distance (KID).

Conclusion.

We thank the reviewer for their comments and hope that we have adequately addressed their concerns. If they are satisfied with our answers and our revision, we respectfully request that they raise their score for this paper. Otherwise, we would be pleased to continue this discussion during the reviewer-author discussion period.

We thank the reviewer for their thoughtful comments, which will improve the quality of our manuscript. We have uploaded a revision to OpenReview, where the reviewer will find that we have incorporated many of their suggestions. In this revision, we have typeset our relevant changes in teal. We respond to the reviewer's comments below.

Positioning in the literature.

The closed-form expression for the score of $p_t$ (Equations (1) and (2) in our manuscript) is indeed well-known, with similar formulas having appeared in the empirical Bayes literature as early as 1961 in Miyasawa’s “An empirical Bayes estimator of the mean of a normal population.”

We have updated Sections 1 and 3 to highlight this point and include several references to appearances of this formula in the statistics and ML literature.

However, we emphasize that this formula is not one of our contributions. Rather, we use this formula as a starting point to develop a score-based generative model (SGM) that generalizes without requiring trainable neural approximations, and we introduce two strategies in Sections 5.2 and 5.3 (starting sampling at $T>0$ and the NN-based score estimator) that greatly accelerate our sampler.

We have updated our statement of contributions in Section 1 to further emphasize this point.

Definition of notation $\bar{c}_k$ for barycenters.

We thank the reviewer for pointing out this oversight and have defined this notation in Section 4.2 of the revised manuscript.

Exponential dependence of the sampling error on $T$.

As the reviewer notes, the error bound that in Theorem 5.2 is pessimistic in its dependence on $T$. Given that our experiments in Section 6.2 show little accuracy loss for practical values of $\sigma$, we believe this bound to be loose as a consequence of the elementary techniques used to prove it. We would be interested in alternative methods that lead to an error bound which reflects the small error that we observe in practice.

However, the bound in Theorem 5.2 also indicates that the error incurred by starting at $T>0$ scales linearly with $\sigma$. When $\sigma=0$, the smoothed and unsmoothed CFDMs are identical, and one can sample the unsmoothed CFDM (i.e. mixture of Gaussians) at any time without incurring error. For $\sigma>0$, this is no longer true – but it is not obvious how the error incurred by sampling the unsmoothed CFDM at $T>0$ depends on $\sigma$. Our result confirms that this dependence is linear, so small values of $\sigma$ lead to small errors.

We have updated our discussion in Section 5.2 to clarify this point, namely that the interest of the theorem is the $\sigma$ dependence rather than the $T$ dependence.

Generality of conclusion that under $\sigma<0.4$, a large starting $T$ is harmless.

This was true across all datasets considered in this paper—not just the checkerboard dataset used in Section 6.2.

For example, in Sections 6.3 and 6.4, we used a start time of $T=0.98$ for all image generation experiments with little degradation in sample quality. The fact that $[0,0.4]$ is the appropriate range for $\sigma$ is partially a consequence of having rescaled the training images/latents to lie within the unit ball before sampling and then reversed this rescaling on the generated samples. We have updated our discussion in Section 6.2 to clarify this point.

Subject to time constraints within the discussion period, we would be happy to run any further experiments the reviewer has in mind to further confirm this observation.

Filling in gaps between sparse manifold samples in the 3D point cloud experiment.

We have updated our discussion of this experiment in Section 6.1 to clarify why our method fills in gaps between sparse samples from the surface.

Practical utility of CFDMs.

The benefits of our method over a neural SGM are twofold:

1. Computationally: Our method requires no training and can be sampled very quickly while running on consumer-grade CPUs.
2. Theoretically: Unlike with neural SGMs, we are able to characterize the support of our model’s samples (Theorem 5.1) and can read off the weights of the training points that contributed to each generated sample.

As the reviewer highlights, our model samples barycenters of training points. In light of this fact, sampling in an appropriately-structured latent space (one where the data manifold is “locally linear”) is likely the most realistic path towards obtaining sample quality comparable to neural SGMs on high-resolution image generation tasks. Our CIFAR results in Section 6.4 indicate that this is a promising direction, and we continue to work toward scaling our method up to larger and higher-resolution datasets.

Conclusion.

We thank the reviewer for their comments and hope that we have addressed their concerns. If they are satisfied with our answers and our revision, we respectfully request that they raise their score. Otherwise, we would be pleased to continue this discussion.

Dear Reviewer t4Fr -- as the discussion period will close in a few days, we would greatly appreciate if you would take a look at our response to your review soon and let us know if you would like to see further changes to our manuscript. We look forward to addressing your remaining concerns before the end of discussion period. If our response was satisfactory, we ask that you consider raising your score for our submission. Thank you for your time.